LLenny's Podcast
← All episodes
23 October 2025

Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)

8Frameworks
15Insights

Frameworks in this episode

Communication4 steps

Emotional Journey Design for Content

Content is predicting reader reactions: hook them, pace the emotion, make people likable.

Read →
Innovation4 steps

Frustration-Log Micro-Tool Ideation

Beat the idea crisis: log a week of frustrations, then build tiny AI tools to kill them.

Read →
Innovation5 steps

RAG Data Preparation Over Database Tuning

The biggest RAG quality wins come from preparing data for retrieval, not picking a database.

Read →
Leadership3 steps

Randomized Tiered Trial for AI Productivity

Measure whether AI tools help by running a randomized trial split across performance tiers.

Read →
Innovation3 steps

Step-Wise Eval Design for Multi-Step AI Apps

Don't evaluate agents end-to-end; put an eval on every step until you hit coverage.

Read →
Strategy3 steps

The Eval ROI Decision

Build evals where failure is catastrophic or you must win; vibe-check the rest.

Read →
Strategy3 steps

Two-Question New Technology Adoption Test

Before adopting any new AI tool, ask: how big is the gain, and how painful is the exit?

Read →
Strategy3 steps

What Actually Improves AI Apps

Stop chasing AI news and vector DBs; the real levers are users, data, and prompts.

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 4

Hot Take05:30

What People Think Improves AI Apps vs. What Actually Does

Chip Huyen unpacks her viral table contrasting perceived AI-app improvements (keeping up with AI news, newest agentic framework, agonizing over vector databases) against what actually moves the needle (talking to users, better data, better prompts). She argues most debated technology choices barely change performance, so obsessing over them is wasted effort, and warns against overcommitting to unproven tech that's hard to swap out later.

  • Staying current on AI news, chasing the newest framework, and debating vector databases feel productive but rarely improve the app
  • Talking to users, preparing better data, and writing better prompts are what actually improve AI apps
  • Before debating two technologies, ask how much performance you'd actually gain from the optimal choice
  • Avoid overcommitting to a new, un-battle-tested technology that would be costly to switch out

“how do we keep up to date with the latest AI news? And I'm like why why do you need to keep up to date…”

Chip Huyen · 05:30

“let's say here's a new technology. It hasn't been tested by a lot of people. And if you adopt it, you would be like stuck…”

Chip Huyen · 06:00
#ai products#prompting#strategy#tooling
Hot Take20:30

The Lopsided Economics of Data-Labeling Startups

Chip flags a structural risk in the fast-growing data-labeling sector: a handful of frontier labs buy from many startups, so revenue is concentrated on two or three customers with huge pricing power. From a lab's perspective, keeping many competing providers is rational because it drives prices down, leaving the startups dependent and exposed.

  • Only a few frontier labs demand data while many startups supply it, an extremely lopsided market
  • Some labeling startups have massive revenue but only a tiny number of customers
  • Frontier labs benefit from keeping many providers competing to lower prices
  • Chip frames herself as curious rather than outright bearish, noting things often work out unexpectedly

“I thought it's very interesting to think about because it's very lopsided, right? because like there only like a very small numbers of frontier labs…”

Chip Huyen · 20:30

“if I if I was this company Frontier Labs what could be the right economical things for me to do right now I want a…”

Chip Huyen · 21:00
#data labeling#economics#startups
Hot Take24:30

When You Don't Actually Need Evals

Chip gives a pragmatic, ROI-based case for skipping evals on some features. If a use case already works and users seem happy, spending two engineers on evals to squeeze 80% to 82-85% may be worse than shipping a new feature. But at scale or where failures are catastrophic, evals become essential and you should be tyrannical about what reaches users.

  • You don't have to be perfect to win, just good enough and consistent
  • Weigh eval effort against expected gain versus the upside of shipping a new feature instead
  • Without a clear metric you lose visibility and something dumb or costly can happen
  • At scale or where failures are catastrophic, evals are essential

“You don't have to be like absolutely perfect at things to win. You just need to be like good enough and being consistent about it.”

Chip Huyen · 24:30

“you have if you operate at scale and where like failures can have like catastrophic consequences then you do need to be very tyrannical about…”

Chip Huyen · 26:30
#evals#roi#product
Hot Take51:30

Computer Science Is System Thinking, Not Coding

Chip relays an insight from Stanford's Mehran Sahami (with Andrew Ng) that CS is not really about coding, which is just a means to an end, but about system thinking and problem solving. As AI automates individual skills, the hard, durable skill is knowing how to combine them to solve problems, and problems only get bigger as more gets automated.

  • Coding is a means to an end; CS is about system thinking and problem solving
  • AI can automate discrete skills, but combining them to solve problems remains hard
  • As AI automates more, the problems themselves get bigger, not fewer
  • Brett Taylor shared the same view: learning CS is about how systems work, not Java or Python

“he said like a lot of people think that CS is about coding but it's not like coding is just a means to an end…”

Chip Huyen · 52:00
#system thinking#cs education#problem solving

Explainer· 5

Explainer09:30

Why a Language Model Is Just Encoded Statistics of a Language

Chip explains pre-training as a way of encoding statistical information about a language: after seeing enough text, a model can predict the most likely next token given the preceding words. She traces the idea back to Shannon's 1951 work on English entropy and uses a Sherlock Holmes cipher story, where the most common stick figure must map to the most common English letter E, to illustrate statistical prediction.

  • Language modeling encodes statistical information, e.g. after 'my favorite color is' the word 'blue' is far likelier than 'table'
  • The idea is old, going back to Shannon's 1951 paper on the entropy of English
  • Sherlock Holmes solved a stick-figure cipher by mapping the most frequent figure to E, the most common English letter
  • Tokens sit between characters and words, hitting a sweet spot that keeps vocabulary manageable (e.g. splitting 'podcasting' into 'podcast' + 'ing')

“So I think of language modeling as a way of encoding statistical information about a language, right?”

Chip Huyen · 09:30
#pre-training#tokens#fundamentals
Explainer16:00

How Reinforcement Learning and RLHF Actually Work

Chip explains post-training reinforcement learning as reinforcing better model outputs using a signal for good vs. bad. Because humans struggle to give absolute scores but find comparisons easy, human preference between two responses trains a reward model. She also describes AI feedback and verifiable rewards (e.g. math problems with known answers) as alternative signal sources.

  • Humans find absolute scoring hard but pairwise comparisons easy, so preference data trains a reward model
  • The reward model then scores new responses to bias the model toward better outputs
  • AI feedback can replace human feedback as the signal source
  • Verifiable rewards give math problems with known answers (e.g. expected 42) that can be checked automatically

“as humans uh we tend to it's very hard to give like concrete score but it's easier to do comparisons right”

Chip Huyen · 16:30

“people are very big on nowadays like verifiable rewards which is like natural um so basically they give it a math problems and then math…”

Chip Huyen · 17:30
#reinforcement learning#rlhf#post-training
Explainer32:00

What RAG Really Is (and Where the Idea Came From)

Chip explains RAG (retrieval augmented generation) as simply giving a model relevant context so it can answer better. She notes the core idea predates generative AI, tracing to a 2017 paper where retrieving Wikipedia context for question-answering benchmarks improved answers, and that in its simplest sense RAG is a no-brainer.

  • RAG stands for retrieval augmented generation and is not specific to generative AI
  • The idea traces to a paper around 2017 that retrieved Wikipedia context for QA benchmarks
  • Providing relevant context lets the model answer questions much better
  • In its simplest form the concept is almost obvious

“So rag is stand for retrieval augmented generations.”

Chip Huyen · 32:00

“providing the model with a relevant context so so that it can answer the questions”

Chip Huyen · 32:30
#rag#retrieval#fundamentals
Explainer55:30

ML Engineer vs. AI Engineer: The Real Difference

Chip distinguishes ML engineers, who build models themselves, from AI engineers, who use existing models as a service to build products. Because base model performance is already strong, the entry barrier to building with AI is very low, while capabilities are strong enough to open up a much larger space of possible applications.

  • ML engineers build models; AI engineers use existing models to build products
  • AI-as-a-service means you can integrate AI without learning to train models
  • The entry barrier to building with AI is now very low
  • Strong capabilities plus low barriers expand both the demand and range of applications

“it makes the entry barrier really low for people who want to use AI to build product”

Chip Huyen · 56:30
#ai engineering#roles#products
Explainer1:06:00

Test-Time Compute: Better Answers Without a Better Model

Chip explains test-time compute as allocating more compute at inference to improve results without changing the base model. Techniques include generating multiple answers and taking the majority (e.g. three of four say 42) or generating more reasoning tokens to think longer before answering. The perceived performance improves even though the underlying model is unchanged.

  • Test-time compute spends more resources at inference rather than training
  • Generating several answers and voting (majority says 42) improves reliability
  • Producing more reasoning/thinking tokens yields better final answers
  • Perceived performance rises while the base model itself stays the same

“when the model spend more time exploring different potential answers thinking longer it can give you much better final answers but the base model itself…”

Chip Huyen · 1:08:00
#test-time compute#inference#reasoning

Story· 3

Story44:00

Managers Choose Headcount, Executives Choose AI Tools

Chip shares a revealing pattern: asked whether they'd rather give the team expensive coding-agent subscriptions or one extra headcount, almost all managers pick headcount, while VPs managing many teams pick the AI assistant. Her read is that managers value more people reporting to them as a growth signal, whereas executives optimize for business and productivity metrics, and the deeper problem is that productivity is genuinely hard to measure.

  • Given the choice, most managers pick an extra headcount over AI coding subscriptions
  • VPs and executives managing many teams instead pick the AI assistant
  • Managers value headcount partly as personal growth, not pure productivity
  • The underlying issue is that productivity gains are very hard to measure

“almost everyone could say the managers could say headcount but if you ask VP level or like someone who manage a lot of teams they…”

Chip Huyen · 44:30
#productivity#management#ai adoption
Story45:30

The Randomized Cursor Trial: Who Actually Benefits from AI Coding

Chip describes a friend's company that ran a randomized trial, splitting its 30-40 engineers into top, average, and lowest performers and giving half of each group access to Cursor. The highest performers got the biggest productivity boost, followed by average performers. But she notes a contradicting report from another company where senior engineers were the most resistant to AI because they're more opinionated and hold higher standards.

  • A ~30-40 person team split engineers into three performance buckets and randomly gave half Cursor
  • Highest-performing engineers got the biggest boost, then average performers
  • Chip's read: top performers are proactive and know how to direct AI to solve problems
  • A different company found senior engineers most resistant to AI, calling its code low-quality

“So the highest performing engineer get the biggest boost out of it”

Chip Huyen · 46:30

“actually senior engineers are the one most resistant to like using AI as tooling”

Chip Huyen · 47:00
#coding agents#engineering#productivity
Story52:30

Where AI Coding Fails: The Hosting-Tier Debugging Story

Chip tells how she used an AI coding tool to deploy to an unfamiliar hosting service and hit a persistent bug. The AI kept changing environment variables, functions, and languages without success. When she finally read the docs herself, the real cause was that the feature she wanted simply wasn't available on her pricing tier, illustrating that AI struggles when the true source of an issue is in a different component than the one it's fixing.

  • AI coding is good at clear, well-defined tasks like docs, specific fixes, or apps from scratch
  • It struggles with problems spanning many interacting components
  • The AI kept editing code and config while the real cause was a plan-tier limitation
  • Diagnosing which component causes an issue needs a holistic, system-level view

“it turns out it's like I'm on another tier like the fish I want did not is not available in this tier.”

Chip Huyen · 54:00

“the issue with clon is trying to focus on fixing things from a very a different component where the issue is from a different component.”

Chip Huyen · 54:00
#debugging#coding agents#system thinking

Takeaway· 3

Takeaway12:00

Sampling Strategy Is a Hugely Underrated Performance Lever

Chip points out that a model outputs a probability distribution over next tokens, not a single answer, and the sampling strategy determines whether it always picks the most likely token or something more creative. She argues this choice can boost performance dramatically and is badly underrated by most builders.

  • A model produces a distribution over possible next tokens, not one fixed answer
  • Sampling strategy decides whether output is the safest token or something more creative
  • Tuning sampling can boost performance substantially yet is very underrated

“so I think sampling strategy I think is something extremely important can have you boost up performance in a in a huge way and very…”

Chip Huyen · 12:30
#sampling#inference#performance
Takeaway34:00

Data Preparation Beats Agonizing Over Vector Databases

Chip argues the biggest quality gains in RAG come from better data preparation, not from choosing the perfect vector database. Concrete techniques include tuning chunk size, adding contextual metadata and hypothetical questions, rewriting content into question-answer format, and adding annotation layers so AI (which lacks human common sense) can interpret documentation correctly.

  • The largest RAG performance gains come from data preparation, not database choice
  • Chunk size is a real design tradeoff: too long retrieves fewer chunks, too short lacks context
  • Rewriting content into question-answer format gave one team a big performance boost
  • Docs written for humans need extra annotation for AI, which lacks the human expert's context

“that's like the biggest performance in their rack solutions coming from like better data preparations not agonizing over like what better databases to use”

Chip Huyen · 34:30
#rag#data preparation#chunking
Takeaway1:08:30

The Idea Crisis: Build From What Frustrates You

Chip observes an 'idea crisis': with powerful tools that can build anything, many people are stuck on what to build, partly because specialization has cost us the big-picture view. Her tactical fix is to pay attention for a week to what frustrates you in daily work, look for common frustrations, and build a small tool to address them, a great way to learn and adopt AI.

  • Despite powerful build-anything tools, many people don't know what to build
  • Over-specialization has eroded the big-picture view that sparks ideas
  • For a week, notice what you do and what frustrates you
  • Build a micro-tool around common frustrations as a way to learn and adopt AI

“I feel like we are in some kind like an idea crisis right now we have all this really cool tools”

Chip Huyen · 1:09:00

“go look from the last week right like for a week just like pay attention to what you do and what frustrate you”

Chip Huyen · 1:10:00
#ideas#vibe coding#micro tools