Lenny offers the intuitive metaphor that evals are like unit tests for code, and Shreya pushes back. Unit tests only check non-negotiable, well-defined behaviors; evals are a much broader spectrum that includes measuring open-ended, ambiguous quality, spotting new user cohorts, and tracking product signals over time.

Unit tests only cover clear-cut, non-negotiable functionality
AI assistants do open-ended tasks that need fuzzier measurement
Evals can surface new user cohorts and distribution shifts in your data
Unit tests are a very small part of a much bigger evals puzzle

“Uh the way I think about eval and tell me if this is wrong just simply is like unit tests for for code and then…”
Lenny Rachitsky · 08:00

“So I would say on the end on overall right unit tests are a very small part of that very big puzzle.”
Shreya Shankar · 09:30

#evals#unit-tests#misconceptions

Myth Buster24:00

No, the AI Can't Just Eval Itself

The single most common misconception is that you can point an LLM at your traces and have it find the errors for you. Shreya explains it doesn't work: the LLM lacks the product context to know that, say, offering a nonexistent virtual tour is a failure. Automating error analysis with an LLM at this stage is the number one pitfall.

Ask an LLM to review a trace and it usually says 'looks good'
The model lacks the product context to spot bad product smells like hallucinated features
The initial error analysis must be done manually by a human
Trying to automate this step with an LLM is the number-one pitfall

“what we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because…”
Shreya Shankar · 24:00

“number one pitfall right here is people are like, let me automate this with an LLM.”
Shreya Shankar · 24:30

#evals#error-analysis#misconceptions#llm

Hot Take· 4

Hot Take37:00

Be Wary of Anyone Who 'Invents' New Techniques

Shreya stresses that they did not invent error analysis — and treats claiming to invent things as a bad signal. Open coding, axial coding, and error analysis are grounded in decades of social science and machine learning. Hamel backs it up with an 8-year-old Andrew Ng video teaching the same technique for stochastic systems.

They explicitly did not invent error analysis or open/axial coding
Be wary of methods presented as brand-new and ungrounded in theory
Open and axial coding come from social science
Andrew Ng was teaching error analysis for stochastic systems years ago

“We didn't invent error analysis. We don't actually want to invent things. That's a bad that's bad signal.”
Shreya Shankar · 37:00

“this is a technique that's been used to analyze stochastic systems for ages.”
Hamel Husain · 38:30

#error-analysis#evals#methodology

Hot Take52:00

Use Binary Pass/Fail, Never a 1–5 Scale

When building an LLM-as-judge, Hamel insists on a binary pass/fail output rather than a 1-to-5 or 1-to-7 rating. Numeric scales are a 'weasel way' of avoiding a real decision, and nobody can interpret what a 3.2 versus a 3.7 actually means when you report the metric.

LLM judges should output binary pass/fail, not a numeric rating
Rating scales dodge the hard decision of good-enough-or-not
Reported averages like 3.2 vs 3.7 are meaningless to readers
Forcing a yes/no keeps the whole process tractable

“We don't want hey like score this on a rating of one to five like how good is it? That's just mostly in most cases…”
Hamel Husain · 52:00

“when you report these metrics, no one knows what 3.2 versus 3.7 means.”
Hamel Husain · 52:30

#llm-judge#evals#metrics

Hot Take1:00:30

Evals Are the New PRDs

Lenny notes recent guests calling evals 'the new PRDs,' and the LLM-judge prompt is exactly that — a product requirements document that specifies how the agent should behave and runs constantly. Hamel adds a twist: unlike a static PRD, evals are derived from real data, so you uncover expectations you could never have dreamed up upfront.

An LLM-judge prompt encodes product requirements and runs continuously
It's a living version of a PRD tied to real behavior
Evals are derived from your own data, not just prior assumptions
Looking at data surfaces expectations you couldn't have predicted

“I've had some guests on the podcast recently who've been saying eval are the new PRDS. And if you look at this, this is exactly…”
Lenny Rachitsky · 1:00:30

“as we look at our data, we uncover more expectations that we couldn't have dreamed up in the first place.”
Hamel Husain · 1:01:30

#evals#prd#product-management#llm-judge

Hot Take1:12:00

Does Claude Code Really 'Just Vibe'? The Evals Debate

A viral claim held that Claude Code's team 'doesn't do evals, they just vibe.' Shreya argues that's misleading: coding agents stand on the shoulders of heavily-benchmarked foundation models, and the team is almost certainly doing implicit error analysis by dogfooding and monitoring usage. Hamel adds that coding agents are a special case because the developer is the domain expert.

Coding agents rely on foundation models already benchmarked on coding
Monitoring usage and dogfooding is itself a form of error analysis
There's no world where a top product ships without looking at anything
Coding agents are unusual because the developer is the domain expert

“he's like oh we don't do evals we just vibe we just look at vibes”
Lenny Rachitsky · 1:12:00

“All of this is eval right there's no world in which they are just being like I made cloud code I'm never looking at anything”
Shreya Shankar · 1:13:00

#evals#claude-code#coding-agents#debate

Explainer· 3

Explainer05:30

What an Eval Actually Is (No Jargon)

Hamel defines evals in plain terms: a systematic way to measure and improve an AI application. At its core it's just data analytics on your LLM's outputs, plus metrics where needed, so you can iterate and run experiments with confidence instead of guessing.

An eval is a systematic way to measure and improve an AI application
It doesn't have to be scary or unapproachable
At its core it's data analytics on your LLM application
You create metrics only where necessary so you can measure and iterate

“Evals is a way to systematically measure and improve an AI application. And it really doesn't have to be scary or unapproachable at all.”
Hamel Husain · 05:30

#evals#ai-products#definitions

Explainer29:30

How Many Traces to Review: Theoretical Saturation

There's no magic number of traces to review. They suggest 100 mainly to unblock people mentally — but the real stopping rule, borrowed from qualitative analysis, is 'theoretical saturation': keep going until you stop uncovering new types of problems. In practice people often find they only need 15–60.

They suggest ~100 traces mainly to remove the intimidation
After ~20 you'll find the process so useful you keep going
The real rule is 'theoretical saturation' — stop when no new notes appear
Actual number varies by application; some need only 15

“we say 100 is because we know that as soon as you start doing this once you do 20 of these you will automatically find…”
Hamel Husain · 29:30

“the right answer is keep looking at traces until you feel like you're not learning anything new”
Hamel Husain · 30:00

#error-analysis#evals#data-analysis

Explainer58:30

Why '90% Agreement' Can Be a Trap

Hamel warns product managers about a dangerous metric: raw agreement between the LLM judge and the human. Because errors often occur only in the long tail, a judge that always says 'pass' can look 90% accurate simply because the error is rare. If someone reports an agreement number, dig in and look at the confusion matrix.

Errors often happen only in the long tail, at low frequency
A judge that always passes can score 90% agreement on a 10%-error problem
High agreement can be completely misleading on paper
If someone reports agreement, ask for more and look at each error type

“if you only have the error 10% of the time, then you can easily have 90% agreement by just having a judge say, uh, it…”
Hamel Husain · 58:30

#evals#metrics#llm-judge#data-analysis

Q&A· 2

Q&A1:05:00

How Many Evals Do You End Up With? Four to Seven

Asked how many LLM-judge evals a typical product needs, Shreya says between four and seven. Most failure modes get fixed simply by adding the missing instruction to your prompt; you only build a dedicated eval for the pesky cases where you've described the ideal behavior but the agent still fails.

A typical product ends up with roughly four to seven evals
Most failure modes are fixed by just fixing the prompt
Reserve evals for the persistent failures that survive prompt fixes
The game is prioritizing finite time toward the pesky, risky areas

“for me like between four and seven”
Shreya Shankar · 1:05:00

“a lot of the failure modes as Haml said earlier can be fixed by just fixing your prompt”
Shreya Shankar · 1:05:00

#evals#prompting#prioritization

Q&A1:30:30

How Long Evals Take: A Week Up Front, 30 Min After

Shreya breaks down the real time cost: for a new application she spends three to four days doing initial rounds of error analysis and labeling with the team, building the spreadsheet and a few LLM-judge evaluators. But that's a one-time cost — after that, keeping the suite running takes only about 30 minutes a week.

Initial error analysis takes roughly three to four days of hands-on work
The heavy up-front effort is a one-time cost
Ongoing maintenance is about 30 minutes a week
People overestimate the effort because they assume it never ends

“usually I'll spend three to four days really working with whoever to do initial rounds of error analysis”
Shreya Shankar · 1:30:30

“maybe 30 minutes a week after that.”
Shreya Shankar · 1:31:00

#evals#error-analysis#process#time

Takeaway· 3

Takeaway25:30

Appoint a 'Benevolent Dictator' for Labeling

Teams get bogged down trying to run eval labeling by committee. Hamel's fix is a 'benevolent dictator': appoint one person whose taste you trust — ideally the domain expert, often the product manager — to do the note-taking. The goal is to keep the process cheap and tractable, not perfect.

Labeling by committee is usually unnecessary and slows everything down
Appoint one trusted person to own the note-taking
That person should be the domain expert — often the PM
Don't make the process so expensive you can't afford to do it

“You don't want to make this process so expensive that you can't do it.”
Hamel Husain · 26:00

“it should be the person with domain expertise.”
Shreya Shankar · 27:00

#evals#process#product-management#error-analysis

Takeaway56:30

Never Ship a Judge Without Aligning It to a Human

A common mistake is writing an LLM-as-judge prompt and accepting its verdicts as gospel. Hamel warns this is the fastest way to end up with evals that don't match reality — and once people lose trust in your evals, they lose trust in you. Always measure the judge's agreement against your own human labels first.

Don't treat the LLM judge's verdict as automatically correct
Measure the judge's agreement against your human labels before shipping
Misaligned judges are the fastest route to untrustworthy evals
Lost trust in evals becomes lost trust in you

“when people lose trust in your evals, they'll lose trust in you.”
Hamel Husain · 56:30

#llm-judge#evals#alignment#trust

Takeaway1:28:30

Looking at Your Data Is the Highest-ROI Activity

If there's one phrase to take from the episode, it's 'look at your data.' Hamel calls it the most powerful, highest-ROI activity you can do when building an AI product — and urges teams to build (or vibe-code) their own custom tools to remove all friction from reviewing traces, which is cheaper than ever with AI.

'Look at your data' is the most repeated lesson of the episode
It's the highest-ROI activity in AI product development
Build your own tools to remove friction from reviewing traces
AI makes building simple data-viewing tools cheap and fast

“if you heard any phrase in this podcast, you've probably heard look at your data more than anything else.”
Hamel Husain · 1:28:00

“it's the highest ROI activity you can engage in.”
Hamel Husain · 1:29:30

#data-analysis#evals#tooling#roi

Frameworks in this episode

Error Analysis: Open Coding to Axial Coding to Count

Eval Triage: Decide What Actually Deserves an Eval

The Aligned Binary LLM-as-Judge

The Benevolent Dictator Labeling Model

Insights & moments

⚡Myth Buster· 2

Evals Are Not Just Unit Tests