Eval Triage: Decide What Actually Deserves an Eval
After counting failures, route each one to a prompt fix, a code check, or an LLM judge — not all three.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 3
- Confidence
- 88%
A prioritization decision tree applied once you have counted failure modes. Not every failure needs an automated eval; evals have a cost-benefit tradeoff and you have finite time. For each failure you decide: is it an obvious prompt fix (just fix it), can it be checked deterministically in code (cheap code-based eval), or is it a subjective, pesky, high-risk failure (build an LLM-as-judge)? Experts typically end up with only four to seven LLM judges.
Origin
Articulated by Hamel Husain and Shreya Shankar from their consulting and course experience helping teams operationalize evals without over-engineering.
Core principles
- 01There is a real cost-benefit tradeoff to every eval — don't get carried away
- 02The whole game is prioritization against finite resources and time
- 03Prefer code-based checks over LLM judges when possible because they're cheaper
- 04If a prompt fix obviously solves it, just fix it — don't build an eval to prove it
- 05Reserve LLM judges for the pesky failures where you've already described the desired behavior but it still fails
- 06Weight toward failures most risky to the business
How to run it
- 1
Ask: is this an obvious prompt fix?
Many failure modes exist simply because you never told the LLM what you wanted (e.g. a formatting error you never specified in the prompt). If the fix is obvious, just change the prompt and move on — no eval required.
Pro tip The recommended default is to fix things as you find them: the goal is a better product, not a large eval suite you can point at.
- 2
Ask: can code check this deterministically?
If the failure can be verified with a Python function or string check — is the output valid JSON, is it markdown, is it short enough — write a cheap code-based automated evaluator instead of using an LLM.
Pro tip Even a fixed formatting issue can warrant a code check if you can test the string cheaply without running an LLM.
- 3
Otherwise: build an LLM-as-judge for the pesky, risky ones
For subjective, complex failures you can't capture in code — like whether the agent should have handed off to a human — build an LLM judge. Reserve these for failures you've already tried to fix in the prompt but that keep happening, and prioritize the ones most damaging to the business.
Pro tip You'll typically only need four to seven judges total, because most failure modes get resolved earlier by fixing the prompt.
Watch out Don't try to write an eval for everything — that's an unbounded cost. Prioritize the pesky and high-risk failures.
In the wild
Among Nurture Boss failures, a formatting error might just mean you forgot to tell the LLM the desired format — a prompt fix, or at most a cheap code string-check. But the human-handoff failure was a subjective judgment call ('should we have handed off?') with no obvious fix, making it a good candidate for an LLM-as-judge.
→ Each failure was routed to the cheapest sufficient mechanism, avoiding the trap of writing expensive LLM evals for problems solvable with a one-line prompt change.
Common mistakes
Building an eval for every failure
Treating evals like traditional unit tests (one per bug fixed) ignores the cost-benefit tradeoff and burns finite time. Prioritize; most failures don't need a standing eval.
Reaching for an LLM judge when code would do
LLM judges are more expensive and require their own alignment work. If the check is deterministic (JSON, length, format), a code-based eval is cheaper and more reliable.
Is it for you?
Best for
Teams with a counted list of failure modes who need to decide where to spend limited eval-building effort.
Not ideal for
Very early exploration before any error analysis has been done — you can't triage failures you haven't found yet.
From the transcript
“there's a costbenefit trade-off to eval. You don't want to get carried away with it.”
“You can't write an eval for everything. So prioritize the ones that are the more pesky areas”
“One is codebased which you should try to do if you can because they're cheaper.”
“it's not that many because a lot of the failure modes as Haml said earlier can be fixed by just fixing your prompt”
From the episode
Why AI evals are the hottest new skill for product builders
Hamel Husain & Shreya Shankar (creators of the #1 eval cours