The Aligned Binary LLM-as-Judge
Build a one-failure, pass/fail judge and align it to a human with a confusion matrix before you trust it.
- Difficulty
- Advanced
- Time to result
- ~days to results
- Steps
- 4
- Confidence
- 93%
A method for building an automated evaluator that uses an LLM to grade one narrowly-scoped failure mode as pass/fail, then proves it agrees with human judgment before shipping it. The key discipline is scope (one failure, binary output) and alignment (measuring the judge against human labels via a confusion matrix, not a headline agreement percentage). Once aligned, the same judge runs in unit tests, CI, and continuous production monitoring.
Origin
Taught by Hamel Husain and Shreya Shankar. The alignment insight draws on Shreya Shankar's research paper 'Who Validates the Validators?' (started late 2023), which identified 'criteria drift' — that people can't specify their rubrics upfront and discover failure modes only after reviewing many outputs.
Core principles
- 01Scope each judge to exactly one failure mode — the problem is small even when the base task is hard
- 02Output must be binary pass/fail, never a 1-5 or 1-7 rating scale
- 03Never accept the judge's verdict as gospel without aligning it to human labels first
- 04Raw agreement percentage is a dangerous, misleading metric on rare failures
- 05You cannot define your rubric fully upfront — criteria drift means new failure modes emerge as you review data
- 06Reuse the aligned judge everywhere: unit tests, CI, and online production monitoring
How to run it
- 1
Write a binary judge prompt for one failure
Take a single failure mode from your error analysis (e.g. human-handoff) and write a prompt that outputs true/false for whether that specific error occurred, enumerating the concrete conditions that define it (explicit human request ignored, policy-mandated transfer, sensitive issue, tool-data unavailable, etc.).
Pro tip Because you're asking the judge to evaluate just one narrow failure with a pass/fail output, it's a tightly-scoped task LLMs do very reliably — it is not as hard as building the original agent.
Watch out Never use a 1-5 or 1-7 rating scale. It's a 'weasel way of not making a decision,' and nobody can interpret whether 3.2 vs 3.7 is meaningful.
- 2
Align the judge to human labels with a confusion matrix
Before releasing the judge, run it over data you've already labeled by hand and build a confusion matrix (human-says-error vs judge-says-error, true/false on each axis). Look at the off-diagonal cells — where human and judge disagree — and iterate the prompt to shrink those disagreements toward zero.
Pro tip You already have the human labels for free from your axial-coding pass, so you often don't need to re-label from scratch. Split your data properly so you're not cheating on the measurement.
Watch out Never accept the judge's output as truth without this step — it's the fastest way to get evals that don't match reality, and when people lose trust in your evals they lose trust in you.
- 3
Distrust headline agreement percentages
Do not judge alignment by a single agreement number. If a failure only occurs 10% of the time, a judge that always says 'pass' scores 90% agreement while catching nothing. Always demand the confusion matrix instead.
Pro tip As a PM, if someone reports 'the judge agrees 75% of the time,' immediately ask to see the per-cell breakdown — a reported agreement number with no matrix is a bad smell.
Watch out High agreement on rare failures is a mirage; the metric hides that the judge never detects the error you care about.
- 4
Deploy the aligned judge everywhere
Run the validated judge as pre-ship unit tests / CI gates on known failure traces, and also online — sampling e.g. a thousand production traces daily to track the real-world failure rate on a live dashboard.
Pro tip Online monitoring on real production traces gives an extremely specific, ongoing measure of application quality — treat the judge as a living instrument, not just a pre-release gate.
In the wild
Hamel built a binary LLM judge for the handoff failure mode, listing the exact conditions that should trigger a handoff (explicit human request ignored, policy-mandated transfer, sensitive resident issue, tool-data unavailability, same-day walk-in). He then measured it in a spreadsheet against his own prior human labels using a pivot-table confusion matrix to check agreement.
→ A trustworthy, aligned judge that can be run both on existing data and on live production traces to measure how often handoff failures actually happen and to catch regressions.
Shreya's user study of developers writing LLM judges found people's definitions of good and bad shifted as they reviewed more outputs; they thought of failure modes only after seeing 10+ examples, even seasoned experts who'd built many pipelines.
→ Established that rubrics can't be fixed upfront — judges and PRDs must evolve from real data, reinforcing why error analysis (not imagined requirements) must seed the judge.
Common mistakes
Shipping the judge prompt without aligning it
Writing the judge prompt and treating its verdicts as gospel is the most common failure. Unaligned judges silently disagree with reality, and the loss of trust contaminates your entire eval effort.
Using a Likert / 1-7 rating judge
Even 'expert' internet templates ship 1-7 scale judges. They produce uninterpretable averages (a '4.2') and dodge the actual pass/fail decision the judge exists to make.
Reporting agreement without a confusion matrix
A high agreement percentage on a rare failure can be achieved by a judge that always passes, detecting nothing. Without the matrix and driven-down off-diagonals, the number is meaningless.
Is it for you?
Best for
Builders who've identified a subjective, recurring failure mode via error analysis and need an automated, trustworthy way to measure it at scale and in production.
Not ideal for
Failures that can be checked deterministically in code (JSON validity, length, format) — use a cheaper code-based eval instead of an LLM judge.
From the transcript
“you need a binary score. You don't want to think about is this like a one, two, three, four, five”
“So the scope of the problem is very small and the output of this LLM judge is like pass or fail.”
“Don't do that because that's the fastest way that you can have evals that don't match what's going on. And when people lose trust in…”
“then you can easily have 90% agreement by just having a judge say, uh, it passes all the time.”
“you could also use it online for monitoring right like I can sample like thousand traces every day run my LLM judge real production traces…”
“you can't figure out your rubrics up front. People's opinions of good and bad change as they review more”
From the episode
Why AI evals are the hottest new skill for product builders
Hamel Husain & Shreya Shankar (creators of the #1 eval cours