LLenny's Podcast
← All episodes
23 October 2025

Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)

8Frameworks
15Insights

Episode overview

Chip Huyen explains the foundations of AI engineering, including pre-training, post-training, reinforcement learning, evals, RAG, and inference-time compute. She argues that successful AI products improve more through user understanding, reliable systems, better-prepared data, workflow optimization, and stronger prompts than through reflexively adopting new technology. The conversation also examines uneven productivity gains from coding agents, organizational restructuring around AI, multimodal product challenges, and system thinking as a durable skill.

Key ideas

  • Prioritize user feedback, reliable platforms, prepared data, end-to-end workflows, and prompt quality over chasing every new AI tool or model.
  • Post-training shapes model behavior through supervised fine-tuning and reinforcement signals derived from humans, AI feedback, or verifiable outcomes.
  • Evals should guide product development and expose failure modes, but their depth should reflect business value, scale, risk, and expected return on investment.
  • RAG quality often depends more on data preparation—such as chunk design, contextual metadata, hypothetical questions, and AI-readable documentation—than on vector-database selection.
  • AI productivity gains are difficult to measure and vary by employee, task, incentives, experience, and organizational context.
  • System thinking and holistic problem solving remain important because coding agents can struggle to diagnose failures spanning multiple components.
  • Future gains may increasingly come from post-training, inference-time compute, application engineering, and multimodal experiences rather than dramatic base-model improvements alone.
  • Useful AI product ideas can emerge by observing recurring frustrations in everyday work and building small tools to remove them.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Communication4 steps

Emotional Journey Design for Content

Content is predicting reader reactions: hook them, pace the emotion, make people likable.

Read →
Innovation4 steps

Frustration-Log Micro-Tool Ideation

Beat the idea crisis: log a week of frustrations, then build tiny AI tools to kill them.

Read →
Innovation5 steps

RAG Data Preparation Over Database Tuning

The biggest RAG quality wins come from preparing data for retrieval, not picking a database.

Read →
Leadership3 steps

Randomized Tiered Trial for AI Productivity

Measure whether AI tools help by running a randomized trial split across performance tiers.

Read →
Innovation3 steps

Step-Wise Eval Design for Multi-Step AI Apps

Don't evaluate agents end-to-end; put an eval on every step until you hit coverage.

Read →
Strategy3 steps

The Eval ROI Decision

Build evals where failure is catastrophic or you must win; vibe-check the rest.

Read →
Strategy3 steps

Two-Question New Technology Adoption Test

Before adopting any new AI tool, ask: how big is the gain, and how painful is the exit?

Read →
Strategy3 steps

What Actually Improves AI Apps

Stop chasing AI news and vector DBs; the real levers are users, data, and prompts.

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 10

  • Chip HuyenMentionsChip was a core developer on NVIDIA's Nemo platform, an AI researcher at Netflix. She taught machine learning at Stanford.

    “Today my guest is Chip Hen.”

    ▶ Watch at ~1:00approx.
  • Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter

    “Hi, Lenny. I've been a big fan of a podcast for a while.”

    ▶ Watch at ~4:30approx.
  • Claude ShannonMentionsthe idea comes very very old like from the 1951 papers um um the like English entropy I think it's like called Shannon

    “the idea comes very very old like from the 1951 papers um um the like English entropy I think it's like called Shannon”

    ▶ Watch at ~10:00approx.
  • Shreya ShankarMentionsThey've now taught over 2,000 PMs and engineers across 500 companies

    “we had a whole podcast any vals with HML Haml and Shrea”

    ▶ Watch at ~24:00approx.
  • Hamel HusainMentionsHamill and Shrea have played a major role in shifting evals from being an obscure mysterious subject

    “we had a whole podcast any vals with HML Haml and Shrea”

    ▶ Watch at ~24:00approx.
  • Mehran SahamiMentionsMer Sami who was my one of my favorite professors. He was a chair of the curriculum of the CS department at Stford.

    “Mer Sami who was my one of my favorite professors. He was a chair of the curriculum of the CS department at Stford.”

    ▶ Watch at ~52:00approx.
  • Andrew NgMentionsthe other person is like Andrew which is of course it's like a legend in the AI space

    “the other person is like Andrew which is of course it's like a legend in the AI space”

    ▶ Watch at ~52:00approx.
  • Bret TaylorMentionsHe co-created Google Maps, co-founded FriendFeed and Quip, held senior roles at Facebook and Salesforce, and co-founded Sierra.

    “That's exactly the same insight Brett Taylor shared on the podcast. He's the co-founder of Sierra. He created Google Maps.”

    ▶ Watch at ~55:00approx.
  • Lee Kuan YewMentionshe was the one who led Singapore from uh he changed uh Singapore from a third country to a forceful country within 25 years

    “the book from um Singaporean um previous um I think he's known as the father of Singapore. I know like Limi Guango”

    ▶ Watch at ~1:13:00approx.
  • Steve JobsMentionsSteve Jobs is just uh the bar he held for the company and for technical talent and for excellence was not uh wavering.

    “This is what Steve Jobs shared too in one of his most famous speeches just we will all die someday”

    ▶ Watch at ~1:17:00approx.

Resources · 33

  • Stanford UniversityUsesplace

    “She taught machine learning at Stanford.”

    ▶ Watch at ~1:00approx.
  • NvidiaMentionscompany

    “Chip was a core developer on NVIDIA's Nemo platform, an AI researcher at Netflix.”

    ▶ Watch at ~1:00approx.
  • NVIDIA NeMoUsessoftware · NVIDIA

    “Chip was a core developer on NVIDIA's Nemo platform, an AI researcher at Netflix.”

    ▶ Watch at ~1:00approx.
  • NetflixUsescompany · Netflix

    “Chip was a core developer on NVIDIA's Nemo platform, an AI researcher at Netflix.”

    ▶ Watch at ~1:00approx.
  • O'ReillyMentionswebsite

    “AI Engineering, which has been the most read book on the O'Reilly platform since its launch”

    ▶ Watch at ~1:30approx.
  • AI EngineeringCoinedbook · Chip Huyen

    “her most recent book called AI Engineering, which has been the most read book on the O'Reilly platform since its launch”

    ▶ Watch at ~1:30approx.
  • Lenny's NewsletterCoinednewsletter · Lenny Rachitsky

    “if you become an annual subscriber of my newsletter, you get a year free of 16 incredible products”

    ▶ Watch at ~2:00approx.
  • dscoutMentionssoftware · dscout

    “Dout is the all-in-one research platform built for modern product and design teams.”

    ▶ Watch at ~2:30approx.
  • FigmaUsessoftware · Figma

    “You can even test your Figma prototypes directly inside the platform.”

    ▶ Watch at ~3:00approx.
  • JustworksUsescompany

    “But with Just Works, it's super easy.”

    ▶ Watch at ~4:00approx.
  • LinkedInUseswebsite · LinkedIn

    “this table slashchart that you shared on LinkedIn a while ago that went super viral”

    ▶ Watch at ~4:30approx.
  • Model Context ProtocolMentionssoftware · Anthropic

    “recently like MCP versus like Asian Asians, right? like protocol and it was like which one is better”

    ▶ Watch at ~5:30approx.
  • A Mathematical Theory of CommunicationMentionspaper · Claude Shannon

    “the idea comes very very old like from the 1951 papers um um the like English entropy I think it's like called Shannon”

    ▶ Watch at ~10:00approx.
  • Sherlock HolmesMentionsbook · Arthur Conan Doyle

    “did you read Sherlock Holm by the way”

    ▶ Watch at ~10:30approx.
  • GPT-5Mentionssoftware · OpenAI

    “fine-tuning is someone taking GPT5 and doing the same sort of thing”

    ▶ Watch at ~13:00approx.
  • AnthropicMentionscompany

    “we had the c of anthropic on the podcast and he talked about their version of RHF”

    ▶ Watch at ~18:30approx.
  • QuickBooksMentionssoftware · Intuit

    “for editing app or like Quickbooks or like Google Excel like they have very specific like tune specific um expert expertise”

    ▶ Watch at ~20:00approx.
  • Google SheetsMentionssoftware · Google

    “for editing app or like Quickbooks or like Google Excel like they have very specific like tune specific um expert expertise”

    ▶ Watch at ~20:00approx.
  • WikipediaUseswebsite · Wikimedia Foundation

    “What they do was they try to retrieve information from Wikipedia.”

    ▶ Watch at ~32:30approx.
  • PersonaMentionscompany · Persona

    “this episode is brought to you by persona the verified identity platform helping organizations onboard users fight fraud and build trust”

    ▶ Watch at ~37:30approx.
  • Claude CodeMentionssoftware · Anthropic

    “giving people like touchd subscriptions uh cloud code subscriptions like to get the employees to like to to be more AI literate”

    ▶ Watch at ~42:30approx.
  • CursorUsessoftware · Anysphere

    “gave some of them access to say cursor. Was it cursor or what did they give them access to?”

    ▶ Watch at ~48:00approx.
  • SierraMentionscompany · Bret Taylor and Clay Bavor

    “He's the co-founder of Sierra. He created Google Maps. He was CEO of Salesforce, Quip, a few other things.”

    ▶ Watch at ~55:00approx.
  • QuipMentionssoftware · Bret Taylor and Kevin Gibbs

    “He was CEO of Salesforce, Quip, a few other things.”

    ▶ Watch at ~55:00approx.
  • Google MapsMentionssoftware · Google

    “He created Google Maps. He was CEO of Salesforce, Quip, a few other things.”

    ▶ Watch at ~55:00approx.
  • SalesforceMentionscompany · Salesforce

    “He created Google Maps. He was CEO of Salesforce, Quip, a few other things.”

    ▶ Watch at ~55:00approx.
  • Google DocsUsessoftware · Google

    “I was working on a newsletter post inside Google Docs and I I pasted all these images into the Google Doc”

    ▶ Watch at ~1:11:00approx.
  • The Selfish GeneRecommendsbook · Richard Dawkins

    “there's several books that I do think is really change the way I think and see the world. So one thing is a selfish gene”

    ▶ Watch at ~1:12:00approx.
  • From Third World to FirstRecommendsbook · Lee Kuan Yew

    “Uh it's called like from third to first world fashion. I think I have it somewhere here.”

    ▶ Watch at ~1:14:00approx.
  • Story of Yanxi PalaceMentionstv show

    “I think it like uh Yami Palace is a Chinese TV show.”

    ▶ Watch at ~1:15:30approx.
  • Lenny's PodcastCoinedpodcast · Lenny Rachitsky

    “You can find all past episodes or learn more about the show at lennispodcast.com.”

    ▶ Watch at ~1:22:00approx.
  • Apple PodcastsRecommendssoftware · Apple Inc.

    “you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app”

    ▶ Watch at ~1:22:00approx.
  • SpotifyRecommendssoftware · Spotify

    “you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app”

    ▶ Watch at ~1:22:00approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 4

Hot Take05:30

What People Think Improves AI Apps vs. What Actually Does

Chip Huyen unpacks her viral table contrasting perceived AI-app improvements (keeping up with AI news, newest agentic framework, agonizing over vector databases) against what actually moves the needle (talking to users, better data, better prompts). She argues most debated technology choices barely change performance, so obsessing over them is wasted effort, and warns against overcommitting to unproven tech that's hard to swap out later.

  • Staying current on AI news, chasing the newest framework, and debating vector databases feel productive but rarely improve the app
  • Talking to users, preparing better data, and writing better prompts are what actually improve AI apps
  • Before debating two technologies, ask how much performance you'd actually gain from the optimal choice
  • Avoid overcommitting to a new, un-battle-tested technology that would be costly to switch out

“how do we keep up to date with the latest AI news? And I'm like why why do you need to keep up to date…”

Chip Huyen · 05:30

“let's say here's a new technology. It hasn't been tested by a lot of people. And if you adopt it, you would be like stuck…”

Chip Huyen · 06:00
#ai products#prompting#strategy#tooling
Hot Take20:30

The Lopsided Economics of Data-Labeling Startups

Chip flags a structural risk in the fast-growing data-labeling sector: a handful of frontier labs buy from many startups, so revenue is concentrated on two or three customers with huge pricing power. From a lab's perspective, keeping many competing providers is rational because it drives prices down, leaving the startups dependent and exposed.

  • Only a few frontier labs demand data while many startups supply it, an extremely lopsided market
  • Some labeling startups have massive revenue but only a tiny number of customers
  • Frontier labs benefit from keeping many providers competing to lower prices
  • Chip frames herself as curious rather than outright bearish, noting things often work out unexpectedly

“I thought it's very interesting to think about because it's very lopsided, right? because like there only like a very small numbers of frontier labs…”

Chip Huyen · 20:30

“if I if I was this company Frontier Labs what could be the right economical things for me to do right now I want a…”

Chip Huyen · 21:00
#data labeling#economics#startups
Hot Take24:30

When You Don't Actually Need Evals

Chip gives a pragmatic, ROI-based case for skipping evals on some features. If a use case already works and users seem happy, spending two engineers on evals to squeeze 80% to 82-85% may be worse than shipping a new feature. But at scale or where failures are catastrophic, evals become essential and you should be tyrannical about what reaches users.

  • You don't have to be perfect to win, just good enough and consistent
  • Weigh eval effort against expected gain versus the upside of shipping a new feature instead
  • Without a clear metric you lose visibility and something dumb or costly can happen
  • At scale or where failures are catastrophic, evals are essential

“You don't have to be like absolutely perfect at things to win. You just need to be like good enough and being consistent about it.”

Chip Huyen · 24:30

“you have if you operate at scale and where like failures can have like catastrophic consequences then you do need to be very tyrannical about…”

Chip Huyen · 26:30
#evals#roi#product
Hot Take51:30

Computer Science Is System Thinking, Not Coding

Chip relays an insight from Stanford's Mehran Sahami (with Andrew Ng) that CS is not really about coding, which is just a means to an end, but about system thinking and problem solving. As AI automates individual skills, the hard, durable skill is knowing how to combine them to solve problems, and problems only get bigger as more gets automated.

  • Coding is a means to an end; CS is about system thinking and problem solving
  • AI can automate discrete skills, but combining them to solve problems remains hard
  • As AI automates more, the problems themselves get bigger, not fewer
  • Brett Taylor shared the same view: learning CS is about how systems work, not Java or Python

“he said like a lot of people think that CS is about coding but it's not like coding is just a means to an end…”

Chip Huyen · 52:00
#system thinking#cs education#problem solving

Explainer· 5

Explainer09:30

Why a Language Model Is Just Encoded Statistics of a Language

Chip explains pre-training as a way of encoding statistical information about a language: after seeing enough text, a model can predict the most likely next token given the preceding words. She traces the idea back to Shannon's 1951 work on English entropy and uses a Sherlock Holmes cipher story, where the most common stick figure must map to the most common English letter E, to illustrate statistical prediction.

  • Language modeling encodes statistical information, e.g. after 'my favorite color is' the word 'blue' is far likelier than 'table'
  • The idea is old, going back to Shannon's 1951 paper on the entropy of English
  • Sherlock Holmes solved a stick-figure cipher by mapping the most frequent figure to E, the most common English letter
  • Tokens sit between characters and words, hitting a sweet spot that keeps vocabulary manageable (e.g. splitting 'podcasting' into 'podcast' + 'ing')

“So I think of language modeling as a way of encoding statistical information about a language, right?”

Chip Huyen · 09:30
#pre-training#tokens#fundamentals
Explainer16:00

How Reinforcement Learning and RLHF Actually Work

Chip explains post-training reinforcement learning as reinforcing better model outputs using a signal for good vs. bad. Because humans struggle to give absolute scores but find comparisons easy, human preference between two responses trains a reward model. She also describes AI feedback and verifiable rewards (e.g. math problems with known answers) as alternative signal sources.

  • Humans find absolute scoring hard but pairwise comparisons easy, so preference data trains a reward model
  • The reward model then scores new responses to bias the model toward better outputs
  • AI feedback can replace human feedback as the signal source
  • Verifiable rewards give math problems with known answers (e.g. expected 42) that can be checked automatically

“as humans uh we tend to it's very hard to give like concrete score but it's easier to do comparisons right”

Chip Huyen · 16:30

“people are very big on nowadays like verifiable rewards which is like natural um so basically they give it a math problems and then math…”

Chip Huyen · 17:30
#reinforcement learning#rlhf#post-training
Explainer32:00

What RAG Really Is (and Where the Idea Came From)

Chip explains RAG (retrieval augmented generation) as simply giving a model relevant context so it can answer better. She notes the core idea predates generative AI, tracing to a 2017 paper where retrieving Wikipedia context for question-answering benchmarks improved answers, and that in its simplest sense RAG is a no-brainer.

  • RAG stands for retrieval augmented generation and is not specific to generative AI
  • The idea traces to a paper around 2017 that retrieved Wikipedia context for QA benchmarks
  • Providing relevant context lets the model answer questions much better
  • In its simplest form the concept is almost obvious

“So rag is stand for retrieval augmented generations.”

Chip Huyen · 32:00

“providing the model with a relevant context so so that it can answer the questions”

Chip Huyen · 32:30
#rag#retrieval#fundamentals
Explainer55:30

ML Engineer vs. AI Engineer: The Real Difference

Chip distinguishes ML engineers, who build models themselves, from AI engineers, who use existing models as a service to build products. Because base model performance is already strong, the entry barrier to building with AI is very low, while capabilities are strong enough to open up a much larger space of possible applications.

  • ML engineers build models; AI engineers use existing models to build products
  • AI-as-a-service means you can integrate AI without learning to train models
  • The entry barrier to building with AI is now very low
  • Strong capabilities plus low barriers expand both the demand and range of applications

“it makes the entry barrier really low for people who want to use AI to build product”

Chip Huyen · 56:30
#ai engineering#roles#products
Explainer1:06:00

Test-Time Compute: Better Answers Without a Better Model

Chip explains test-time compute as allocating more compute at inference to improve results without changing the base model. Techniques include generating multiple answers and taking the majority (e.g. three of four say 42) or generating more reasoning tokens to think longer before answering. The perceived performance improves even though the underlying model is unchanged.

  • Test-time compute spends more resources at inference rather than training
  • Generating several answers and voting (majority says 42) improves reliability
  • Producing more reasoning/thinking tokens yields better final answers
  • Perceived performance rises while the base model itself stays the same

“when the model spend more time exploring different potential answers thinking longer it can give you much better final answers but the base model itself…”

Chip Huyen · 1:08:00
#test-time compute#inference#reasoning

Story· 3

Story44:00

Managers Choose Headcount, Executives Choose AI Tools

Chip shares a revealing pattern: asked whether they'd rather give the team expensive coding-agent subscriptions or one extra headcount, almost all managers pick headcount, while VPs managing many teams pick the AI assistant. Her read is that managers value more people reporting to them as a growth signal, whereas executives optimize for business and productivity metrics, and the deeper problem is that productivity is genuinely hard to measure.

  • Given the choice, most managers pick an extra headcount over AI coding subscriptions
  • VPs and executives managing many teams instead pick the AI assistant
  • Managers value headcount partly as personal growth, not pure productivity
  • The underlying issue is that productivity gains are very hard to measure

“almost everyone could say the managers could say headcount but if you ask VP level or like someone who manage a lot of teams they…”

Chip Huyen · 44:30
#productivity#management#ai adoption
Story45:30

The Randomized Cursor Trial: Who Actually Benefits from AI Coding

Chip describes a friend's company that ran a randomized trial, splitting its 30-40 engineers into top, average, and lowest performers and giving half of each group access to Cursor. The highest performers got the biggest productivity boost, followed by average performers. But she notes a contradicting report from another company where senior engineers were the most resistant to AI because they're more opinionated and hold higher standards.

  • A ~30-40 person team split engineers into three performance buckets and randomly gave half Cursor
  • Highest-performing engineers got the biggest boost, then average performers
  • Chip's read: top performers are proactive and know how to direct AI to solve problems
  • A different company found senior engineers most resistant to AI, calling its code low-quality

“So the highest performing engineer get the biggest boost out of it”

Chip Huyen · 46:30

“actually senior engineers are the one most resistant to like using AI as tooling”

Chip Huyen · 47:00
#coding agents#engineering#productivity
Story52:30

Where AI Coding Fails: The Hosting-Tier Debugging Story

Chip tells how she used an AI coding tool to deploy to an unfamiliar hosting service and hit a persistent bug. The AI kept changing environment variables, functions, and languages without success. When she finally read the docs herself, the real cause was that the feature she wanted simply wasn't available on her pricing tier, illustrating that AI struggles when the true source of an issue is in a different component than the one it's fixing.

  • AI coding is good at clear, well-defined tasks like docs, specific fixes, or apps from scratch
  • It struggles with problems spanning many interacting components
  • The AI kept editing code and config while the real cause was a plan-tier limitation
  • Diagnosing which component causes an issue needs a holistic, system-level view

“it turns out it's like I'm on another tier like the fish I want did not is not available in this tier.”

Chip Huyen · 54:00

“the issue with clon is trying to focus on fixing things from a very a different component where the issue is from a different component.”

Chip Huyen · 54:00
#debugging#coding agents#system thinking

Takeaway· 3

Takeaway12:00

Sampling Strategy Is a Hugely Underrated Performance Lever

Chip points out that a model outputs a probability distribution over next tokens, not a single answer, and the sampling strategy determines whether it always picks the most likely token or something more creative. She argues this choice can boost performance dramatically and is badly underrated by most builders.

  • A model produces a distribution over possible next tokens, not one fixed answer
  • Sampling strategy decides whether output is the safest token or something more creative
  • Tuning sampling can boost performance substantially yet is very underrated

“so I think sampling strategy I think is something extremely important can have you boost up performance in a in a huge way and very…”

Chip Huyen · 12:30
#sampling#inference#performance
Takeaway34:00

Data Preparation Beats Agonizing Over Vector Databases

Chip argues the biggest quality gains in RAG come from better data preparation, not from choosing the perfect vector database. Concrete techniques include tuning chunk size, adding contextual metadata and hypothetical questions, rewriting content into question-answer format, and adding annotation layers so AI (which lacks human common sense) can interpret documentation correctly.

  • The largest RAG performance gains come from data preparation, not database choice
  • Chunk size is a real design tradeoff: too long retrieves fewer chunks, too short lacks context
  • Rewriting content into question-answer format gave one team a big performance boost
  • Docs written for humans need extra annotation for AI, which lacks the human expert's context

“that's like the biggest performance in their rack solutions coming from like better data preparations not agonizing over like what better databases to use”

Chip Huyen · 34:30
#rag#data preparation#chunking
Takeaway1:08:30

The Idea Crisis: Build From What Frustrates You

Chip observes an 'idea crisis': with powerful tools that can build anything, many people are stuck on what to build, partly because specialization has cost us the big-picture view. Her tactical fix is to pay attention for a week to what frustrates you in daily work, look for common frustrations, and build a small tool to address them, a great way to learn and adopt AI.

  • Despite powerful build-anything tools, many people don't know what to build
  • Over-specialization has eroded the big-picture view that sparks ideas
  • For a week, notice what you do and what frustrates you
  • Build a micro-tool around common frustrations as a way to learn and adopt AI

“I feel like we are in some kind like an idea crisis right now we have all this really cool tools”

Chip Huyen · 1:09:00

“go look from the last week right like for a week just like pay attention to what you do and what frustrate you”

Chip Huyen · 1:10:00
#ideas#vibe coding#micro tools