Step-Wise Eval Design for Multi-Step AI Apps
Don't evaluate agents end-to-end; put an eval on every step until you hit coverage.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 88%
A method for evaluating multi-step or agentic AI applications (like deep research) where a single end-to-end metric is too costly and too coarse. Instead of asking 'how many evals do I need,' you ask how many you need to get good coverage and high confidence, placing an eval at each stage of the pipeline. Use it for any app that chains retrieval, reasoning, and generation.
Origin
Discussing evals for a deep-research app, Huyen walks through decomposing the workflow into search-query generation, result gathering, aggregation, and summarization, arguing each step needs its own evaluation rather than one end-to-end judgment.
Core principles
- 01Eval count is set by coverage and confidence, not a fixed magic number.
- 02Every step in the pipeline needs its own evaluation.
- 03Evals exist to uncover where the product underperforms so you can fix it, not just to score it.
How to run it
- 1
Decompose the workflow
Break the app into its actual stages, e.g. generate search queries, gather results, aggregate, identify gaps, summarize.
- 2
Attach an eval to each step
Evaluate query quality (are the queries diverse rather than near-duplicates?), result relevance and overlap (breadth and depth without redundancy), and each downstream transformation.
Pro tip Check search queries for diversity: five near-identical queries waste the retrieval budget.
Watch out An end-to-end-only eval is extremely costly and slow and hides which step is failing.
- 3
Aim for coverage, not a count
Keep adding evals until you have good coverage and high confidence in performance, prioritizing the most common user path.
Pro tip Weight eval effort toward the core, most-traveled use case first.
In the wild
For a 'research Lenny's podcast and report topics' app, Huyen evaluates each stage: are the five generated search queries diverse or redundant, do the returned pages overlap too much, are results relevant to the original prompt, and only then the final summary.
→ Failures get localized to a specific step (e.g., bad query diversity) instead of a vague end-to-end fail.
Common mistakes
One number for a pipeline
A single end-to-end metric can't tell you which stage broke, so you can't fix it efficiently.
Chasing a fixed eval count
Asking 'do I need five or seven evals' ignores that the right number is whatever gives coverage for your product's complexity.
Is it for you?
Best for
Engineers building agentic or multi-step RAG apps who need to know where quality breaks down.
Not ideal for
Simple single-shot chatbots where a handful of top-level evals suffice.
From the transcript
“so every step of the way you need evaluations right you don't need to end to end”
“how many evolve should do I need to get a good coverage a high confidence in my application's performance and also to help me understand…”
From the episode
Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)