Make the Implicit Explicit: Decision Rubrics and Mediating Judgments
Excavate your gut into a rubric, then score the rubric against reality to learn where your gut is wrong
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 94%
Experts defend intuition with 'I know a great PM / founder when I see one.' Duke doesn't attack intuition — she demands it be written down. Excavate the implicit model into named criteria, define each criterion with mediating judgments (sub-judgments made before the overall rating), score on a 1–7 scale rather than good/bad, and record forecasts. Once the ratings exist alongside outcomes, you can tell each individual which of their judgments are actually predictive — and, critically, which things they pound the table about are predictive of nothing.
Origin
Builds directly on Daniel Kahneman's structured-hiring work (unstructured interviews vs. a decision rubric, which raised hiring hit rate from ~50% to ~65%) and his mediating-assessments protocol. Duke operationalized it inside investment partnerships — five years at First Round Capital, plus Renegade Partners.
Core principles
- 01Intuition is not crap — it is sometimes right and sometimes wrong, and if you never make it explicit you never find out which
- 02You are making the forecast implicitly anyway; making it explicit costs nothing and buys a feedback loop
- 03Ratings need range (1–7), not binary good/bad, so you get precision and spread across raters
- 04Shared definitions are not optional — your 'market quality' and mine are different things until we define them
- 05Use intuition after the structured process, not before it
- 06Table-pounding confidence is uncorrelated with predictiveness, even among true experts
How to run it
- 1
Excavate the implicit model
Interview the experts: when you say you 'just know', what specifically are you looking for? Convert those into a named set of decision criteria (e.g. market, team, founder, product).
Watch out Expect 'we don't need this, we know a good founder when we see one'. That sentence is the reason to do it.
- 2
Define each criterion with mediating judgments
For each top-level criterion, specify the sub-judgments you make before judging it overall — e.g. competitive landscape as a mediating judgment for market. These sub-judgments become the criterion's implied shared definition.
Pro tip Mediating judgments double as the alignment mechanism: they force everyone to be rating the same construct.
- 3
Score on a scale, independently
Have each decision-maker rate every criterion 1–7 privately, plus record explicit forecasts of the key intermediate outcomes, with written rationale.
Pro tip Combine with the nominal-group protocol so ratings aren't contaminated by other raters.
Watch out Binary good/bad ratings destroy the spread you need to detect who is actually predictive.
- 4
Score the raters against outcomes
Once outcomes accumulate, analyse per-person: when this partner rates market highly, does the company actually do well? Are their forecasts better than random? Feed each person their own calibration.
Pro tip Strengths and weaknesses across partners are real and not perfectly overlapping — this is the actual argument for group input.
- 5
Evolve the rubric from evidence, not opinion
The first version of the rubric is just the partners' intuitions made explicit. As data lands, drop criteria that turn out to predict nothing for anyone and strengthen those that do.
Watch out Don't over-index on criteria people feel strongly about until the data says they predict something.
In the wild
Before Duke, First Round recorded only who voted yes or no. She built a rubric across market, team, founder and product, with mediating judgments and 1–7 ratings, plus an explicit forecast of the probability each company would fund a Series A. Five years later they can look across hundreds of companies and see, per partner, which of their judgments actually map onto how the company unfolded.
→ Partners now get individual calibration feedback, the rubric is being revised based on evidence, and the firm discovered that criteria partners pounded the table about were sometimes highly predictive and sometimes predictive of nothing at all — even for the person pounding the table.
Kahneman's approach: take an unstructured hiring judgment, excavate what the hirer is implicitly looking for, turn it into a decision rubric and a structured interview process, and apply intuition only after the structured process — not before.
→ Hit rate on hiring rose from roughly 50% to 65% — an enormous improvement in a noisy domain. The catch, as Duke notes, is that almost nobody actually does it.
Common mistakes
Applying intuition before the structure instead of after
The gain comes from doing the structured scoring first and letting intuition operate on top of it. Leading with the gut just reintroduces the noise the rubric exists to remove.
Assuming expert confidence implies expert accuracy
We have an intuition about intuition — that brilliant people's instincts must be good. Duke's data shows equally vehement table-pounding, and sometimes the factor is genuinely predictive and sometimes it predicts nothing, even for that person.
Recording only the vote
If you keep the yes/no but not the reasoning, ratings and forecasts, you have no material to refine decisions with later. You literally cannot audit decision quality.
Is it for you?
Best for
Partnerships and leadership teams making repeated, high-stakes, noisy judgment calls (investments, hiring, bets) who are willing to be measured
Not ideal for
One-off, novel decisions with no repetition, where there will never be enough instances to calibrate against
From the transcript
“it's so incredibly necessary in improving decision quality to take what's implicit and make it explicit”
“you went from 50% hit rate to 65% hit rate which is huge”
“mediating judgments which are uh judgments that you make related to Market prior to actually judging what you think of the market in general”
“let's make it explicit because you're doing it implicitly anyway”
“your intuition is sometimes right if you don't make it explicit then you don't get to find out when it's wrong”
From the episode
This will make you a better decision-maker
Annie Duke (author of “Thinking in Bets” and “Quit,” former