Imperfect Metrics as Education
Report an imperfect engineering metric on purpose, then use every question about it to educate upward.
- Difficulty
- Moderate
- Time to result
- ~ongoing to results
- Steps
- 5
- Confidence
- 90%
Larson's answer to 'how do I prove engineering is productive' has two layers. Structurally: make engineering wholly accountable to product and business goals, and show a roadmap of meaty, impactful things shipped in the last six months. Tactically: refuse the expert's trap of ruling out every imperfect metric until you report nothing. Ship an imperfect metric, and treat each objection to it as a chance to teach the consumer about the rich reality underneath.
Origin
Larson's own practice as an engineering executive at Carta, Calm, Stripe and Uber. The four metrics he points to as a starting place are DORA, from 'Accelerate' by Nicole Forsgren, Jez Humble and Gene Kim — Larson recommends them explicitly as diagnosis instruments rather than scorecards.
Core principles
- 01Benchmarking spend against your funding stage gets you a defensible answer for the board; it does not help you run the org.
- 02Talk to engineers — they know whether their team is effective, and their diagnosis, even when wrong, is a crumb you can trace.
- 03'Trust my intuition' is unusable to a board that has no way to assess your intuition.
- 04DORA metrics are diagnosis instruments: slow deploys tell you where to invest, not whether you're a good company.
- 05Measuring nothing because nothing is measurable perfectly makes you look like you don't know what you're doing.
- 06Metrics are about educating the people consuming them about the reality underneath — not about a perfect dataset.
How to run it
- 1
Anchor engineering to product and business goals
Make engineering wholly accountable to the product goals rather than carving out a separate scorecard where engineering can succeed while the product fails. Engineering exists to support the product and the customer, not to build novel systems for their own sake.
Watch out Many companies find comfort in engineering excellence measured independently of product outcomes. That comfort is the failure.
- 2
Show the six-month roadmap of meaty work
Present the list of meaningful, impactful things engineering has actually delivered in the last six months, with the impact explained. If the list is full, people step back and give you space. If you can't populate it, that is itself the finding — and their concern is warranted.
Pro tip Build this list before anyone asks. Its emptiness is a leading indicator you want to catch yourself.
- 3
Talk to the engineers, continuously
On an ongoing basis, ask the team whether they're effective and why not. They will tell you. Their diagnosis may be wrong, but it's a crumb you can trace with your greater experience to find the actual contributing causes.
Watch out This can't be your answer to the board. 'I talked to the teams and my intuition is spot on' is unverifiable — the board deals with a portfolio of leaders, some with terrible intuition, and has no way to tell which you are.
- 4
Pick an imperfect metric and report it anyway
Get comfortable reporting something measurable but imperfect. DORA (lead time, incident remediation time, change failure rate, deploy frequency) is a good enough starting place for the board or CEO even though its real value is diagnostic.
Pro tip Say the limitations out loud when you present it. Pre-empting the objection is what turns the metric into a teaching instrument rather than a hostage.
Watch out Don't let the metric become the evaluation. A slow lead time tells you where to invest; it does not tell you to fire engineers. And Sprint points reported to a board are a totally fake thing to report on, whatever comfort they provide.
- 5
Use each question to educate upward
When people ask questions about the imperfect measure, treat it as the opportunity: explain why it's imperfect, what it misses, where it lies. Over time your executives and board become more sophisticated readers of engineering — which is impossible if you refuse to give them anything to hold.
Pro tip Start mediocre and improve the audience, not just the metric. Sophistication in the consumer is the actual deliverable.
In the wild
Larson's archetype: an expert asked to measure productivity can explain why every measure is wrong or inaccurate, rules them all out one by one, and ends up measuring nothing. They then go to a non-expert and say there's no accurate measure to give.
→ The non-expert concludes the expert doesn't know what they're doing. Reporting an admittedly imperfect number would have preserved credibility and created a channel to educate them.
The standard first move is to benchmark R&D, engineering and infrastructure spend against a dataset from your VC funds and confirm you're within the expected band. It's mechanical and produces a defensible answer.
→ It makes the board less angry, which is useful because it's hard to do good work with an angry board. It does nothing to help you actually run the organisation effectively.
Common mistakes
Turning DORA into a scorecard
At least fifty startups sell dashboards that instrument DORA and invite you to evaluate teams on them. Deploy speed doesn't make you a good or bad company — it tells you where to focus improvement. Grading people on a diagnosis instrument corrupts both the metric and the diagnosis.
Refusing to measure because nothing is perfect
The expert who rules out every metric ends up reporting nothing, and reads to non-experts as incompetent rather than rigorous. Perfection is not on the menu; a channel for educating your stakeholders is.
Reporting Sprint points to the board
Larson calls these a totally fake thing to report. They provide comfort without information, and they train the board to ask for more of the wrong number.
Is it for you?
Best for
Engineering executives and eng leaders under board or CEO pressure to prove engineering productivity, especially during efficiency-driven headcount scrutiny.
Not ideal for
Small teams where the leader already has direct visibility into all work, and situations where a metric would be used punitively rather than diagnostically.
From the transcript
“engineering evaluation to the the business and product goals so I want us to be wholly accountable to the product goals”
“second I think just showing the road map of the valuable things we've done in the last six months is really powerful”
“challenge is these are really good diagn diagnosis metrics”
“you just have to get comfortable measuring something that's not perfect but you can actually measure and Reporting on it”
From the episode
The engineering mindset
Will Larson (Carta, Stripe, Uber, Calm, Digg)