The Input-Metrics Stack for Developer Productivity
No single metric rules them all — combine per-component input metrics that ladder up to time-to-value.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 94%
Shani rejects the search for one number that proves an AI tool works. Instead she builds a stack: each component of the software lifecycle gets the metric that fits IT (security tooling is measured in leaked secrets prevented, not productivity), those component metrics combine into quality, collaboration, and time-gained, which ladder up to developer happiness, which the business converts into time-to-value. Time alone is explicitly disqualified — you can write really bad code really fast.
Origin
Developed by Shani and the GitHub product org while defining Copilot's success metrics, in dialogue with enterprise customers running their own measurement experiments. She credits Nicole Forsgren's DORA framework (discussed on the same podcast) as a related way of measuring developer experience.
Core principles
- 01There is no one metric to rule them all
- 02Productivity is not the right metric for every component — match the metric to the job the component does
- 03Time is the easiest metric and the most misleading: fast bad code is still bad code
- 04Lines of code is an anti-metric — volume says nothing about quality
- 05Input metrics combine into developer happiness; developer happiness converts into business time-to-value
How to run it
- 1
Refuse the single-metric mandate
Push back on any request for the one number that proves the AI investment. Establish upfront that the answer is a combination of input metrics, not a scalar.
Watch out Accepting a single headline metric invites gaming — the org optimizes the proxy and loses the outcome.
- 2
Assign each lifecycle component the metric that fits its job
Ask what this specific component is FOR, then measure that. For AI in code security, the right measures are how many secrets were prevented from leaking and how many issues were detected and fixed before shipping — not developer productivity.
Pro tip Write the component's purpose in one sentence first; the metric usually falls out of the verb in that sentence.
Watch out Applying a blanket 'productivity' metric across every AI component is the specific error Shani names.
- 3
Group the inputs into quality, collaboration, and time-gained
Roll the component metrics into intermediate buckets — code quality, security, collaboration, time returned to the developer — rather than jumping straight to a single output number.
- 4
Treat developer happiness as the composite outcome
Happiness is the hard-to-measure outcome the input stack ladders up to. Track it through surveys on confidence, frustration, and focus rather than pretending it is directly instrumentable.
Pro tip GitHub's survey battery is a usable template: % who feel more confident in code quality, % less frustrated, % more focused, code-review completion speed.
- 5
Translate to time-to-value for the business
Replace 'time saved' with time-to-value: from the moment you put a developer on a task, how long until the full value is realized — revenue generated, adoption seen, time to market. This is the number the business side actually wants.
Pro tip Let each business define time-to-value in its own terms; the discipline is picking a value event, not standardizing it across companies.
In the wild
When GitHub applied AI to Advanced Security, it deliberately did not measure developer productivity. The job of that component is writing more secure code, so the metrics became secrets prevented from leaking and issues detected and fixed pre-ship.
→ The component is judged on the outcome it exists to produce, avoiding the trap of a productivity number that would look good while security regressed.
Rather than a single productivity figure, GitHub reported a bundle: code written 55% faster, 85% of respondents more confident in code quality, code reviews completed 15% faster, 88% less frustrated and more focused.
→ A composite picture that supports the happiness thesis and survives the 'you can write bad code fast' objection, because quality and confidence are measured alongside speed.
Common mistakes
Using time as the success metric
Time is the easiest thing to measure and therefore the default choice, but it is not quantifiable as success — you can write really bad code really fast. Time must be translated into efficiency, quality, and value.
Counting lines of code
Volume tells you nothing about whether those are good lines. It is the classic way to measure engineers badly, and AI code generation makes it worse by inflating the count.
Reusing one metric across every AI component
Productivity is the right frame for an in-editor assistant and the wrong frame for a security scanner. A blanket metric guarantees at least one component is being judged on the wrong thing.
Is it for you?
Best for
Engineering and product leaders who must justify AI tooling spend to a CFO and are being pushed toward a single 'productivity uplift' number
Not ideal for
Small teams where the tooling decision is cheap and reversible — the measurement overhead will exceed the value of the answer
From the transcript
“we are in a world that there are no right metrics uh there is no one uh one metric to roll them all”
“productivity is not the right metrics against each one of these components when we're implementing AI to get up Advan security writing more secure code…”
“the most easiest one is time but time is it's funny what I'm going to say but time is not quantifiable as a success metrix…”
“instead of time can we talk about time to Value so from the moment you put a developer on a task how long did it…”
From the episode
The future of AI in software development
Inbal Shani (CPO of GitHub)