LLenny's Podcast
← All episodes
Hamel Husain & Shreya Shankar (creators of the #1 eval cours25 September 2025

Why AI evals are the hottest new skill for product builders

4Frameworks
14Insights

Episode overview

Hamel Husain and Shreya Shankar explain evals as systematic measurement and improvement of AI applications, then demonstrate the workflow using anonymized Nurture Boss traces. They show how teams can manually open-code errors, consolidate them into actionable categories, count and prioritize failure modes, and build narrowly scoped binary evaluators. They stress validating LLM judges against human decisions, grounding product requirements in real production data, and treating error analysis, monitoring, unit tests, and experiments as parts of one improvement loop.

Key ideas

  • Evals are a broad system for measuring and improving AI-product quality, not merely unit tests.
  • Begin with manual error analysis of sampled production traces so domain experts can uncover failures they could not specify in advance.
  • Record the first meaningful upstream error in each trace, continuing until additional review stops revealing materially new failure modes.
  • Use LLMs to synthesize detailed open codes into axial categories, but keep humans responsible for reviewing and refining the taxonomy.
  • Count categorized failures and prioritize fixes by prevalence, business risk, actionability, and the cost of maintaining an evaluator.
  • Prefer inexpensive code-based evaluators when possible; use narrowly scoped, binary LLM judges for subjective failure modes.
  • Validate every LLM judge against human labels and inspect both false-positive and false-negative behavior rather than relying on aggregate agreement.
  • Reuse validated evaluators in pre-release tests and production monitoring, while continually updating criteria as new behavior appears.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Innovation6 steps

Error Analysis: Open Coding to Axial Coding to Count

Turn messy LLM logs into a prioritized list of failures before you write a single test.

Read →
Strategy3 steps

Eval Triage: Decide What Actually Deserves an Eval

After counting failures, route each one to a prompt fix, a code check, or an LLM judge — not all three.

Read →
Innovation4 steps

The Aligned Binary LLM-as-Judge

Build a one-failure, pass/fail judge and align it to a human with a confusion matrix before you trust it.

Read →
Leadership3 steps

The Benevolent Dictator Labeling Model

Appoint one trusted domain expert to own eval judgments instead of running it by committee.

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 5

  • Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter

    “Today my guests are Hamill Hussein and Shrea Shankar.”

    ▶ Watch at ~1:08approx.
  • Hamel HusainMentionsHamill and Shrea have played a major role in shifting evals from being an obscure mysterious subject

    “Hamill and Shrea have played a major role in shifting evals from being an obscure mysterious subject”

    ▶ Watch at ~1:34approx.
  • Hamel HusainCoinedHamill and Shrea have played a major role in shifting evals from being an obscure mysterious subject

    “They teach the definitive online course on evals, which happens to be the number one course on Maven.”

    ▶ Watch at ~1:43approx.
  • Shreya ShankarMentionsThey've now taught over 2,000 PMs and engineers across 500 companies

    “They've now taught over 2,000 PMs and engineers across 500 companies”

    ▶ Watch at ~1:49approx.
  • Tom M. MitchellMentionsmentioned

    “Machine learning by Mitchell.”

    ▶ Watch at ~1:39:03approx.

Resources · 35

  • OpenAIMentionscompany

    “Both the chief product officers of Anthropic and OpenAI shared that eval are becoming the most important new skill for product builders.”

    ▶ Watch at ~1:17approx.
  • AnthropicMentionscompany

    “Both the chief product officers of Anthropic and OpenAI shared that eval are becoming the most important new skill for product builders.”

    ▶ Watch at ~1:17approx.
  • MavenUseswebsite · Maven

    “They teach the definitive online course on evals, which happens to be the number one course on Maven.”

    ▶ Watch at ~1:43approx.
  • AI Evals for Engineers and Product ManagersCoinedcourse · Hamel Husain and Shreya Shankar

    “They teach the definitive online course on evals, which happens to be the number one course on Maven.”

    ▶ Watch at ~1:43approx.
  • AI Evals for Engineers and Product ManagersRecommendscourse · Hamel Husain and Shreya Shankar

    “If this conversation gets you excited, definitely check out Hamill and Shreas's course on Maven.”

    ▶ Watch at ~2:35approx.
  • FinRecommendsproduct · Intercom

    “So, if you're ready to transform your customer service and scale your support, give Finn a try for only 99 cents per resolution.”

    ▶ Watch at ~3:43approx.
  • FigmaUsessoftware · Figma

    “You can even test your Figma prototypes directly inside the platform.”

    ▶ Watch at ~4:23approx.
  • dscoutRecommendssoftware · dscout

    “So if you're ready to streamline your research, speed of decisions, and design with impact, head to dscout.com to learn more.”

    ▶ Watch at ~4:40approx.
  • Nurture BossUsescompany

    “It's uh from a company called Nurture Boss.”

    ▶ Watch at ~10:43approx.
  • BraintrustUsescompany · Braintrust

    “I just happen to have this data loaded in a tool called brain trust”

    ▶ Watch at ~12:52approx.
  • PhoenixUsessoftware · Arize AI

    “we had the same example but in Phoenix Arise”

    ▶ Watch at ~13:06approx.
  • LangSmithMentionssoftware · LangChain

    “and there's also Langmith so these are kind of like different tools that you can use”

    ▶ Watch at ~13:13approx.
  • ChatGPTUsessoftware · OpenAI

    “I would bet money on this if I put that into chat GBT and asked is there an error”

    ▶ Watch at ~24:12approx.
  • ClaudeUsessoftware · Anthropic

    “So one is okay, I uploaded into a cloud project. I uploaded a CSV of these notes”

    ▶ Watch at ~32:48approx.
  • Julius AIRecommendssoftware · Julius AI

    “I really like using Julius AI. Um it's one of my favorite tools.”

    ▶ Watch at ~39:39approx.
  • Jupyter NotebookUsessoftware · Project Jupyter

    “I personally like Jupyter notebooks a lot”

    ▶ Watch at ~39:47approx.
  • GeminiUsessoftware · Google

    “And this is using Gemini and AI just for simplicity.”

    ▶ Watch at ~41:41approx.
  • Google SheetsUsessoftware · Google

    “This is like the Google Sheets AI AI prompt.”

    ▶ Watch at ~41:55approx.
  • MercuryRecommendscompany · Mercury

    “See what over 200,000 entrepreneurs love about Mercury. Visit mercury.com to apply online in 10 minutes.”

    ▶ Watch at ~54:25approx.
  • Claude CodeUsessoftware · Anthropic

    “we're clawed code or you know whatever other famous product and we don't do evals”

    ▶ Watch at ~1:11:40approx.
  • StatsigMentionssoftware · Vijaye Raji

    “Do you have thoughts on what stats sig going to do at OpenAI?”

    ▶ Watch at ~1:18:20approx.
  • MMLUMentionsother

    “focused on general benchmarks like MMLU score, human eval which are very important for foundation models”

    ▶ Watch at ~1:21:40approx.
  • HumanEvalMentionsother · OpenAI

    “focused on general benchmarks like MMLU score, human eval which are very important for foundation models”

    ▶ Watch at ~1:21:40approx.
  • CodexMentionssoftware · OpenAI

    “More recently, there's all this talk about codecs, open AI codecs being better and everyone's switching”

    ▶ Watch at ~1:23:19approx.
  • LennybotCoinedwebsite · Lenny Rachitsky

    “just like there's the Lennybot that you have”

    ▶ Watch at ~1:36:07approx.
  • DiscordUsessoftware · Discord Inc.

    “every office hours, every Discord chat, any blogs, papers, anything that we've ever said publicly and within our course”

    ▶ Watch at ~1:36:25approx.
  • ElevenLabsUsescompany · ElevenLabs

    “It's 11 laps powered. It's so good.”

    ▶ Watch at ~1:37:26approx.
  • Apple in ChinaRecommendsbook · Patrick McGee

    “I also am currently reading Apple in China”

    ▶ Watch at ~1:38:31approx.
  • Machine LearningRecommendsbook · Tom M. Mitchell

    “Machine learning by Mitchell.”

    ▶ Watch at ~1:39:03approx.
  • FrozenMentionsfilm · Walt Disney Animation Studios

    “I've watched Frozen like three times in the last week.”

    ▶ Watch at ~1:40:15approx.
  • The WireRecommendstv show · David Simon

    “So my husband and I have been watching The Wire recently. We never actually saw it growing up.”

    ▶ Watch at ~1:40:33approx.
  • CursorRecommendssoftware · Anysphere

    “Yeah, I really like using cursor.”

    ▶ Watch at ~1:41:16approx.
  • Claude CodeRecommendssoftware · Anthropic

    “Yeah, I really like cloud code and I like it because I feel like the UX is outstanding.”

    ▶ Watch at ~1:41:46approx.
  • AI Evals for Engineers and Product ManagersMentionscourse · Hamel Husain and Shreya Shankar

    “If you Google AI evals for engineers and product managers or just AI evals course, you'll find it.”

    ▶ Watch at ~1:44:36approx.
  • Lenny's PodcastCoinedpodcast · Lenny Rachitsky

    “You can find all past episodes or learn more about the show at lennispodcast.com.”

    ▶ Watch at ~1:46:10approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 2

Myth Buster08:00

Evals Are Not Just Unit Tests

Lenny offers the intuitive metaphor that evals are like unit tests for code, and Shreya pushes back. Unit tests only check non-negotiable, well-defined behaviors; evals are a much broader spectrum that includes measuring open-ended, ambiguous quality, spotting new user cohorts, and tracking product signals over time.

  • Unit tests only cover clear-cut, non-negotiable functionality
  • AI assistants do open-ended tasks that need fuzzier measurement
  • Evals can surface new user cohorts and distribution shifts in your data
  • Unit tests are a very small part of a much bigger evals puzzle

“Uh the way I think about eval and tell me if this is wrong just simply is like unit tests for for code and then…”

Lenny Rachitsky · 08:00

“So I would say on the end on overall right unit tests are a very small part of that very big puzzle.”

Shreya Shankar · 09:30
#evals#unit-tests#misconceptions
Myth Buster24:00

No, the AI Can't Just Eval Itself

The single most common misconception is that you can point an LLM at your traces and have it find the errors for you. Shreya explains it doesn't work: the LLM lacks the product context to know that, say, offering a nonexistent virtual tour is a failure. Automating error analysis with an LLM at this stage is the number one pitfall.

  • Ask an LLM to review a trace and it usually says 'looks good'
  • The model lacks the product context to spot bad product smells like hallucinated features
  • The initial error analysis must be done manually by a human
  • Trying to automate this step with an LLM is the number-one pitfall

“what we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because…”

Shreya Shankar · 24:00

“number one pitfall right here is people are like, let me automate this with an LLM.”

Shreya Shankar · 24:30
#evals#error-analysis#misconceptions#llm

Hot Take· 4

Hot Take37:00

Be Wary of Anyone Who 'Invents' New Techniques

Shreya stresses that they did not invent error analysis — and treats claiming to invent things as a bad signal. Open coding, axial coding, and error analysis are grounded in decades of social science and machine learning. Hamel backs it up with an 8-year-old Andrew Ng video teaching the same technique for stochastic systems.

  • They explicitly did not invent error analysis or open/axial coding
  • Be wary of methods presented as brand-new and ungrounded in theory
  • Open and axial coding come from social science
  • Andrew Ng was teaching error analysis for stochastic systems years ago

“We didn't invent error analysis. We don't actually want to invent things. That's a bad that's bad signal.”

Shreya Shankar · 37:00

“this is a technique that's been used to analyze stochastic systems for ages.”

Hamel Husain · 38:30
#error-analysis#evals#methodology
Hot Take52:00

Use Binary Pass/Fail, Never a 1–5 Scale

When building an LLM-as-judge, Hamel insists on a binary pass/fail output rather than a 1-to-5 or 1-to-7 rating. Numeric scales are a 'weasel way' of avoiding a real decision, and nobody can interpret what a 3.2 versus a 3.7 actually means when you report the metric.

  • LLM judges should output binary pass/fail, not a numeric rating
  • Rating scales dodge the hard decision of good-enough-or-not
  • Reported averages like 3.2 vs 3.7 are meaningless to readers
  • Forcing a yes/no keeps the whole process tractable

“We don't want hey like score this on a rating of one to five like how good is it? That's just mostly in most cases…”

Hamel Husain · 52:00

“when you report these metrics, no one knows what 3.2 versus 3.7 means.”

Hamel Husain · 52:30
#llm-judge#evals#metrics
Hot Take1:00:30

Evals Are the New PRDs

Lenny notes recent guests calling evals 'the new PRDs,' and the LLM-judge prompt is exactly that — a product requirements document that specifies how the agent should behave and runs constantly. Hamel adds a twist: unlike a static PRD, evals are derived from real data, so you uncover expectations you could never have dreamed up upfront.

  • An LLM-judge prompt encodes product requirements and runs continuously
  • It's a living version of a PRD tied to real behavior
  • Evals are derived from your own data, not just prior assumptions
  • Looking at data surfaces expectations you couldn't have predicted

“I've had some guests on the podcast recently who've been saying eval are the new PRDS. And if you look at this, this is exactly…”

Lenny Rachitsky · 1:00:30

“as we look at our data, we uncover more expectations that we couldn't have dreamed up in the first place.”

Hamel Husain · 1:01:30
#evals#prd#product-management#llm-judge
Hot Take1:12:00

Does Claude Code Really 'Just Vibe'? The Evals Debate

A viral claim held that Claude Code's team 'doesn't do evals, they just vibe.' Shreya argues that's misleading: coding agents stand on the shoulders of heavily-benchmarked foundation models, and the team is almost certainly doing implicit error analysis by dogfooding and monitoring usage. Hamel adds that coding agents are a special case because the developer is the domain expert.

  • Coding agents rely on foundation models already benchmarked on coding
  • Monitoring usage and dogfooding is itself a form of error analysis
  • There's no world where a top product ships without looking at anything
  • Coding agents are unusual because the developer is the domain expert

“he's like oh we don't do evals we just vibe we just look at vibes”

Lenny Rachitsky · 1:12:00

“All of this is eval right there's no world in which they are just being like I made cloud code I'm never looking at anything”

Shreya Shankar · 1:13:00
#evals#claude-code#coding-agents#debate

Explainer· 3

Explainer05:30

What an Eval Actually Is (No Jargon)

Hamel defines evals in plain terms: a systematic way to measure and improve an AI application. At its core it's just data analytics on your LLM's outputs, plus metrics where needed, so you can iterate and run experiments with confidence instead of guessing.

  • An eval is a systematic way to measure and improve an AI application
  • It doesn't have to be scary or unapproachable
  • At its core it's data analytics on your LLM application
  • You create metrics only where necessary so you can measure and iterate

“Evals is a way to systematically measure and improve an AI application. And it really doesn't have to be scary or unapproachable at all.”

Hamel Husain · 05:30
#evals#ai-products#definitions
Explainer29:30

How Many Traces to Review: Theoretical Saturation

There's no magic number of traces to review. They suggest 100 mainly to unblock people mentally — but the real stopping rule, borrowed from qualitative analysis, is 'theoretical saturation': keep going until you stop uncovering new types of problems. In practice people often find they only need 15–60.

  • They suggest ~100 traces mainly to remove the intimidation
  • After ~20 you'll find the process so useful you keep going
  • The real rule is 'theoretical saturation' — stop when no new notes appear
  • Actual number varies by application; some need only 15

“we say 100 is because we know that as soon as you start doing this once you do 20 of these you will automatically find…”

Hamel Husain · 29:30

“the right answer is keep looking at traces until you feel like you're not learning anything new”

Hamel Husain · 30:00
#error-analysis#evals#data-analysis
Explainer58:30

Why '90% Agreement' Can Be a Trap

Hamel warns product managers about a dangerous metric: raw agreement between the LLM judge and the human. Because errors often occur only in the long tail, a judge that always says 'pass' can look 90% accurate simply because the error is rare. If someone reports an agreement number, dig in and look at the confusion matrix.

  • Errors often happen only in the long tail, at low frequency
  • A judge that always passes can score 90% agreement on a 10%-error problem
  • High agreement can be completely misleading on paper
  • If someone reports agreement, ask for more and look at each error type

“if you only have the error 10% of the time, then you can easily have 90% agreement by just having a judge say, uh, it…”

Hamel Husain · 58:30
#evals#metrics#llm-judge#data-analysis

Q&A· 2

Q&A1:05:00

How Many Evals Do You End Up With? Four to Seven

Asked how many LLM-judge evals a typical product needs, Shreya says between four and seven. Most failure modes get fixed simply by adding the missing instruction to your prompt; you only build a dedicated eval for the pesky cases where you've described the ideal behavior but the agent still fails.

  • A typical product ends up with roughly four to seven evals
  • Most failure modes are fixed by just fixing the prompt
  • Reserve evals for the persistent failures that survive prompt fixes
  • The game is prioritizing finite time toward the pesky, risky areas

“for me like between four and seven”

Shreya Shankar · 1:05:00

“a lot of the failure modes as Haml said earlier can be fixed by just fixing your prompt”

Shreya Shankar · 1:05:00
#evals#prompting#prioritization
Q&A1:30:30

How Long Evals Take: A Week Up Front, 30 Min After

Shreya breaks down the real time cost: for a new application she spends three to four days doing initial rounds of error analysis and labeling with the team, building the spreadsheet and a few LLM-judge evaluators. But that's a one-time cost — after that, keeping the suite running takes only about 30 minutes a week.

  • Initial error analysis takes roughly three to four days of hands-on work
  • The heavy up-front effort is a one-time cost
  • Ongoing maintenance is about 30 minutes a week
  • People overestimate the effort because they assume it never ends

“usually I'll spend three to four days really working with whoever to do initial rounds of error analysis”

Shreya Shankar · 1:30:30

“maybe 30 minutes a week after that.”

Shreya Shankar · 1:31:00
#evals#error-analysis#process#time

Takeaway· 3

Takeaway25:30

Appoint a 'Benevolent Dictator' for Labeling

Teams get bogged down trying to run eval labeling by committee. Hamel's fix is a 'benevolent dictator': appoint one person whose taste you trust — ideally the domain expert, often the product manager — to do the note-taking. The goal is to keep the process cheap and tractable, not perfect.

  • Labeling by committee is usually unnecessary and slows everything down
  • Appoint one trusted person to own the note-taking
  • That person should be the domain expert — often the PM
  • Don't make the process so expensive you can't afford to do it

“You don't want to make this process so expensive that you can't do it.”

Hamel Husain · 26:00

“it should be the person with domain expertise.”

Shreya Shankar · 27:00
#evals#process#product-management#error-analysis
Takeaway56:30

Never Ship a Judge Without Aligning It to a Human

A common mistake is writing an LLM-as-judge prompt and accepting its verdicts as gospel. Hamel warns this is the fastest way to end up with evals that don't match reality — and once people lose trust in your evals, they lose trust in you. Always measure the judge's agreement against your own human labels first.

  • Don't treat the LLM judge's verdict as automatically correct
  • Measure the judge's agreement against your human labels before shipping
  • Misaligned judges are the fastest route to untrustworthy evals
  • Lost trust in evals becomes lost trust in you

“when people lose trust in your evals, they'll lose trust in you.”

Hamel Husain · 56:30
#llm-judge#evals#alignment#trust
Takeaway1:28:30

Looking at Your Data Is the Highest-ROI Activity

If there's one phrase to take from the episode, it's 'look at your data.' Hamel calls it the most powerful, highest-ROI activity you can do when building an AI product — and urges teams to build (or vibe-code) their own custom tools to remove all friction from reviewing traces, which is cheaper than ever with AI.

  • 'Look at your data' is the most repeated lesson of the episode
  • It's the highest-ROI activity in AI product development
  • Build your own tools to remove friction from reviewing traces
  • AI makes building simple data-viewing tools cheap and fast

“if you heard any phrase in this podcast, you've probably heard look at your data more than anything else.”

Hamel Husain · 1:28:00

“it's the highest ROI activity you can engage in.”

Hamel Husain · 1:29:30
#data-analysis#evals#tooling#roi