Build Your Own Senior-Engineer Benchmark
Measure AI honestly by scoring new models against real human experts rewriting your actual broken work.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 85%
A method for cutting through hyped benchmarks by building your own, grounded in real human expert work. Shipper took a real app that failed in production, had two senior engineers independently rewrite it, and used those rewrites as the ground-truth answer key. He then scores each new model by giving it the same broken code and a prompt, comparing its rewrite against the human baseline. The deeper insight: benchmarks only rise on problems humans have already framed and scored — the framing itself is invisible, unmeasured human work.
Origin
Dan Shipper's 'senior engineer benchmark,' built after his vibe-coded app Proof kept crashing in production and two senior engineers rewrote it independently.
Core principles
- 01Benchmarks rise only on problems humans have already articulated and scored; the act of framing the problem is unmeasured human work
- 02AI-vs-human comparisons are really one human-using-AI vs another human-using-AI — AI doesn't use itself
- 03A good benchmark uses real expert output as ground truth, not synthetic tasks
- 04Prompts must reveal capability without giving away the answer
How to run it
- 1
Capture a real failure and expert fixes
Take an actual piece of your own work that broke, and have two independent human experts fix or rewrite it. Their solutions become your ground-truth answer key.
Pro tip Let the human experts use AI however they like — you're benchmarking human-plus-AI against human-plus-AI, which is the only realistic comparison.
Watch out Don't hand-write the human baseline without AI; a human who refuses AI isn't a fair or relevant comparison point.
- 2
Craft a prompt that reveals capability without leaking the answer
Write a prompt that lets the model show what it can really do — e.g. 'this is vibe-coded slop, rewrite it from first principles' — but iterate until the prompt no longer hands the model the solution.
Pro tip Test the model's agency: good prompts reveal whether it will rip out and rewrite bad code versus papering over the edges with a small patch.
Watch out A naive prompt like 'here are the issues, go fix them' measures compliance, not judgment — models take it literally and patch instead of confronting the real problem.
- 3
Score each new model against the human baseline
Run every new model on the same task and score its output out of 100 against the senior-engineer rewrites, tracking the gap and the trajectory over model generations.
Pro tip Re-run on every release — turning the rock over again is how you catch a capability jump (his score moved from ~30 to 62 across a model generation).
Watch out A rising score doesn't mean full human replacement; be ready to redesign the benchmark to a higher frame once a model saturates it, because there's always a higher frame of judgment humans add.
In the wild
Shipper's vibe-coded app Proof crashed every few minutes after launch and he couldn't fix it, so he had two senior engineers rewrite it independently. He then fed the broken code plus a first-principles prompt to each new model. Models scored ~30/100 until one generation jumped to 62, versus human seniors scoring high-80s/low-90s.
→ A concrete, honest measure of the model-vs-human gap and its rate of closing, grounded in real work rather than a synthetic leaderboard.
Common mistakes
Trusting a benchmark's name over what it measures
A benchmark called 'senior engineer' primes you to read saturation as full replacement, but it only captures pre-framed, scorable problems — the human work of deciding what to build and whether it's good goes unmeasured.
Using a prompt that measures compliance, not judgment
Telling a model 'here are the bugs, fix them' makes it dutifully patch symptoms; you never see whether it has the judgment to recognize the codebase needs a first-principles rewrite.
Is it for you?
Best for
Operators and technical leaders who want an honest, situation-specific read on model capability instead of relying on public leaderboards
Not ideal for
People who need a standardized, comparable benchmark across organizations rather than a bespoke one tied to their own work
From the transcript
“I made this senior It's called a the senior engineer benchmark, and it's like, "How good is AI versus a human engineer?"”
“I got actually two different senior engineers to fix it independently. So, I have two different rewrites of the code base”
“benchmarks rise on problems that we've framed that we can articulate, that we can score”
“we're actually really always talking about one human using AI versus another human using AI cuz AI doesn't use itself”
From the episode
The AI paradox: More automation, more humans, more work
Dan Shipper