The Hiring Signal Stack Rank
Reference check beats work sample beats interview — and run the reference check first.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 8
- Confidence
- 93%
Mehrotra ranks hiring signals in a strict order: reference check first, work-product presentation second, interviews last — and when they disagree, that order decides. The method includes a specific reference-check technique for extracting honest PSHE-level judgement despite the perverse incentives references face, plus a home-court / away-court / neutral-court model for balancing the interview loop itself.
Origin
Shishir Mehrotra's own practice, developed across Microsoft, YouTube/Google and Coda, and built directly on top of his PSHE talent ladder — the reference check exists primarily to determine where someone actually sits on P-S-H-E, which he argues is nearly impossible to establish in an interview.
Core principles
- 01A reference has worked with the person for years; an interview is 30 minutes of artificial scenarios.
- 02References carry perverse incentives — they don't like criticising people, they judge themselves by the answer, and they fear legal blowback.
- 03The fix is contrast, not pressure: give them four personas and let them choose, rather than asking for a verdict.
- 04Never disclose which end of the scale you value; if they know you want a P, you'll be told the candidate is a P.
- 05Interview questions live in three courts — home (about your company), away (about their past), neutral (a made-up scenario) — and a loop should balance them.
- 06Interviews lower-bound a candidate; the work-product presentation is the candidate's chance to upper-bound themselves.
- 07Run the reference check as early as practical — being dinged at the reference stage after a full loop is a terrible candidate experience.
How to run it
- 1
Write the reference guide before the interview guide
The most important document is not the interview guide — it's what you ask when you call this person's references, designed to locate them on P-S-H-E.
- 2
Run the reference check as early in the process as practical
Front-load it. Inside large companies (Google, Facebook) informal back-channels are an accepted expectation and can be done before the loop even starts.
Watch out Be careful not to jeopardise the candidate's current position. You can't always do it early — but when you can, do.
- 3
Ideal mode — don't name the candidate at all
With a reference you know well, ask blind: 'when you think of your teams, who is best at X?' You learn whether your candidate is even the name that comes to mind.
Watch out Only works with a deep existing relationship. Don't force it.
- 4
Fallback mode — offer four personas and let them place the person
Describe four archetypes without hierarchy: the one who regularly identifies which problems the team should focus on; the one who solves given problems in creative ways; the one who is really good at getting a team moving; the one who takes a playbook and executes with precision and quality. Ask where this person sits. Explicitly tell them you have no qualitative preference among them.
Pro tip The point of the four-persona framing is to give the reference an out — an honest answer that doesn't feel like a criticism, so they'll actually talk.
Watch out If you leak that you value P over E, the reference will simply place the candidate at P. Do not give away your scale.
- 5
Close with the two forcing questions
Ask 'would you hire this person again?' — or better, 'how excitedly would you hire this person again?' — and always finish with 'what questions should I have asked that I didn't?'
Pro tip Once people start talking, they reveal what they really feel. Silence and open-ended closers do more work than sharper questions.
- 6
Open the loop with a work-product presentation
At the start of the interview loop the candidate presents to everyone on the loop, with other colleagues invited. Roughly half is theirs to use however they want — the 'brag session' — and half responds to a prompt you supply.
Pro tip Tell candidates explicitly: every other question is us trying to lower-bound you, to see how bad this could be. This is your chance to upper-bound us. Don't leave the day wishing we'd learned something about you.
Watch out Some skill sets don't compress into a timed exercise. Design the prompt to fit the role, not the format.
- 7
Balance the interview loop across home, away and neutral court
Home court ('you've joined our company, what would you do about X?') mostly tests whether the candidate guessed your answer. Away court ('how did you solve this problem at your last job?') can't tell you whether the problem was hard, who really did it, or what got left out. Neutral court — a fabricated scenario like the teleportation device — puts you both on level ground. Make most questions neutral, use the presentation as the away-court slot, and use home court sparingly.
Watch out A loop made entirely of home-court questions grades candidates on whether they arrived at the answer you already have — a poor way to judge anyone.
- 8
Resolve conflicts by the stack rank
When signals disagree, the reference check wins over the work-product presentation, which wins over the interviews. Decide this in advance so a charismatic interview loop can't override a lukewarm reference.
In the wild
References rave about a PM: incredibly efficient meetings, always-clear communication, everything buttoned up, everyone knows what's happening. Mehrotra then asks the placement questions — who decides which problem to solve, and who reliably produces the best solution to the hardest problems?
→ 'An amazing number of people' answer that the designer picks the problems, or the CEO tells them what to do. The candidate is a genuinely strong H, not the P the interviews implied — a distinction that is very hard to see in an interview and easy to see in a reference check.
Asked how often a reference check leads to not hiring someone, Mehrotra answers: all the time. Which is precisely why he pushes them as early in the process as is practical.
→ Candidates are spared the worst experience in hiring — completing a full loop and then being dinged at the reference stage — and the company spends far fewer interviewer-hours on candidates the references were always going to fail.
Common mistakes
Treating the reference check as a formality at the end
It's your highest-quality signal — people who worked with the candidate for years versus 30 minutes of artificial scenarios — and running it last both wastes the loop and creates a brutal candidate experience when it fails.
Telling the reference what you value
References want to help the candidate. Reveal that you prize problem-spotting and you will be told the candidate is a brilliant problem-spotter. Present the personas as equals.
Building a loop entirely out of home-court questions
'You've joined us, what would you do about X' just tests whether the candidate reached the same conclusion you did after months of thinking. It grades agreement, not ability.
Is it for you?
Best for
Hiring managers and founders making senior product, engineering or go-to-market hires where the cost of a mis-level is high and interviews keep producing false positives.
Not ideal for
High-volume junior hiring where references are thin or unavailable, or roles where execution against a fixed playbook is genuinely the whole job.
From the transcript
“i generally value the reference check over interview signals”
“those people you know they work with this person for you know sometimes for years like their knowledge what you're gonna get out of 30…”
“reference check at the top you know work product presentation next and then all the interviews”
“for this particular thing if i want to know where they stand on this axis don't tell people what you value”
“a common mistake people make is they do all questions in home court”
“all our questions are are in some senses like we're trying to lower bound you like how bad could this be i want you to…”
From the episode
The rituals of great teams
Shishir Mehrotra of Coda, YouTube, Microsoft