The Measurement Journey
Start measuring from words and people, then scale into system data — don't wait for perfect.
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 85%
A staged approach to building a developer-productivity (or any complex) measurement practice from nothing. You begin by getting crisp on the problem in words, then lean heavily on subjective data from people (surveys, interviews) because it's fast to collect, and progressively shift toward objective system data as you instrument, because it's scalable. Subjective and objective data are permanent complements, not stages you graduate out of.
Origin
Drawn from Nicole Forsgren's forthcoming book and a paper she co-wrote with Mik Kersten (circa 2016-2017) on combining data from people and data from systems in a DevOps context.
Core principles
- 0180% of teams' biggest problem is not being crisp on what problem or goal they're actually solving.
- 02Start with words: is the challenge inner/outer loop, friction, or culture? These lead in completely different directions.
- 03Early on you rely more on data from people because you can get it relatively quickly.
- 04As you mature you shift toward system data because it's scalable and engineerable — but never drop people data.
- 05Don't let the perfect be the enemy of the good.
- 06When surveys and instrumentation disagree, the surveys are almost always the ones that are correct.
How to run it
- 1
Get crisp on the problem in words
Before measuring anything, write down and clearly define the challenge. Confirm whether you mean inner vs outer loop, friction in tool chains, or culture — these are all valid but point in completely different directions.
Pro tip You can do the 'is it written down and clear?' check in a day; hunting down whether any related signal exists takes about a week depending on how scattered things are.
Watch out If stakeholders are on different pages about what 'improve developer experience' means, teams go off for months and come back with the wrong thing — 'that wasn't what I meant.'
- 2
Start with data from people
Collect subjective data quickly via surveys and interviews — ask a handful of developers how they feel about their tools and process and what their biggest barriers to productivity are.
Pro tip Use example interview scripts and survey scripts, screen and select participants deliberately; you don't need a data scientist to start.
- 3
Transition toward system data
As the practice matures, add objective instrumentation from your systems because it's scalable and can be engineered, shifting the proportion of measurement over time.
Watch out Objective data has holes too — code outside version control, incomplete system data — so acknowledge and hunt for gaps rather than trusting instrumentation blindly.
- 4
Keep triangulating both sources permanently
Even the most advanced teams keep surveying developers at least once a year because people surface insights systems never will. When the two disagree, trust the survey.
Pro tip Google is a strong model: systematic phased measurement, heavy telemetry, and continued developer-experience surveys that they triangulate against instrumentation.
In the wild
On paper a team's commit-to-deploy speed looked fine, but developers reported it took absolute heroics via a Rube Goldberg machine of a pipeline — something the system data would never reveal.
→ Demonstrates why people data remains essential even with full instrumentation.
Working with a company, Forsgren found a significant portion of code that was simply not going into any version control system. Because it wasn't in the systems, no amount of system data could have surfaced it.
→ A mission-critical blind spot was caught only because they combined people data with system data.
Common mistakes
Skipping the crisp problem definition
Leaders say 'improve developer experience' without defining whether they mean friction, culture, or inner/outer loop; teams then work for months in the wrong direction — this is the single biggest and most under-recognized failure, affecting 80% of teams.
Dismissing survey data in favor of instrumentation
Assuming people lie or that only system data is 'real' discards the source that is almost always correct when the two disagree, and blinds you to problems (heroics, untracked code) that never appear in system telemetry.
Is it for you?
Best for
Teams and leaders starting a developer-productivity or developer-experience measurement practice from scratch who need a pragmatic path from zero to instrumented.
Not ideal for
Organizations demanding a fully automated, purely objective dashboard from day one, or those unwilling to invest in ongoing developer surveys.
From the transcript
“starting with what is your problem or what is your goal and being super super crisp on it”
“when you first start off you'll be relying much more on data from people because you can get it relatively quickly but as you kind…”
“even teams that I've worked with that are the most advanced they have absolute instrumentation in every possible scenario in the most detailed way they…”
“if there is ever a disagreement between the surveys and the instrumentation which is incredibly Advanced almost every time every time that I've ever heard…”
From the episode
How to measure and improve developer productivity
Nicole Forsgren (Microsoft Research, GitHub, Google)