Evals Are the New PRDs
Turn ambiguous user failures into measurable model-improvement targets.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 99%
Begin with a real user pain point, then inspect the complete interaction rather than accepting a broad label such as hallucination or poor instruction following. Determine the precise mechanism of failure: the model may have skipped a tool, retrieved the wrong source, synthesized the wrong fact, violated a schema, or behaved with misplaced confidence. Collect enough examples to establish that the problem is reproducible and important, then convert them into an eval containing representative prompts, expected behavior, and cases where the model should not exhibit the failure. Researchers and engineers can use that eval as an actionable target, while product teams rerun it against later models to measure improvement and catch regressions. PRDs remain useful for alignment and vision, but evals shorten the distance between user evidence and technical action.
Origin
Dianne Penn described how Anthropic transformed complaints that early Claude models were bad at following instructions into a JSON-schema-following eval that became part of its recurring model evaluation repositories.
Core principles
- 01Start with the user pain point, not a familiar product artifact.
- 02Inspect complete model trajectories and sweat the tokens.
- 03Translate vague complaints into specific failure modes.
- 04Represent the failure with reproducible positive and negative examples.
- 05Measure improvements because you cannot improve what you cannot measure.
How to run it
- 1
Capture the real failure
Gather consented user interactions that demonstrate the pain point. Preserve the prompt, model response, context, and tool trajectory rather than relying on a summary.
- 2
Inspect the trajectory
Read the interaction token by token and determine where behavior diverged from the desired outcome. Separate tool-use, retrieval, synthesis, instruction-following, and alignment failures.
- 3
Validate the pattern
Collect additional examples and determine whether the failure is reproducible, frequent, and important enough to prioritize.
- 4
Build a balanced eval
Create representative prompts, responses, expected outcomes, and grading criteria. Include both situations where the behavior should occur and cases where it should not.
- 5
Connect it to action
Give the eval and failure taxonomy to the researchers or engineers who can change the model, harness, retrieval system, or product.
- 6
Measure and retain
Run the eval against candidate improvements and future releases. Keep it in the recurring suite after the immediate problem is solved to detect regressions.
In the wild
Users said early Claude models were poor at following instructions. Penn's team inspected exact interactions and found that roughly 80% of the cited failures involved producing incorrect JSON. They assembled approximately 30 to 40 examples, standardized them as an eval set, and added the eval to Anthropic's repositories so future Claude versions could be tested against the same user need.
→ Later models reached approximately 99.9% to 100% performance on a failure that had previously been a recurring user pain point.
A user reports that an assistant hallucinated while answering from a document. The PM reads the trajectory and discovers that the assistant opened the correct document but selected the wrong facts. Instead of creating a generic hallucination initiative, the team builds an eval for document search and synthesis with examples that distinguish retrieval failures from factual synthesis failures.
→ The responsible team receives a specific target and can measure whether the next release improves the actual behavior.
Common mistakes
Treating the complaint as the diagnosis
User language describes the experience but rarely identifies the technical mechanism. Investigate the trajectory before assigning the failure to a team or solution.
Writing an eval from one example
A single case may be an outlier or omit important variations. Establish a pattern and cover both desired and undesired behavior.
Discarding the eval after a fix
Resolved failures can return as models, prompts, and harnesses change. Preserve valuable evals as regression protection.
Is it for you?
Best for
It is best for PMs and teams building products whose behavior depends on probabilistic models, prompts, tools, or retrieval systems.
Not ideal for
It is not ideal as a complete replacement for vision documents or cross-functional PRDs on broad, ambiguous initiatives.
From the transcript
“we actually have a saying on the team of evals are the new PRDs, right?”
“Here, you have to sweat the tokens as much as you sweat the pixels.”
“because you can't improve what you can't measure.”
From the episode
Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future