LLenny's Podcast
← All episodes
Nicole Forsgren (Microsoft Research, GitHub, Google)30 July 2023

How to measure and improve developer productivity

4Frameworks
15Insights

Frameworks in this episode

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 4

Myth Buster11:00

The Two-Week Change-Approval Wait Doesn't Buy You Stability

Old ITIL/ITSM change management held that you needed a mandatory multi-week wait for change approvals to keep systems stable. Forsgren's research found the opposite: that batching-up of changes actively causes instability and merge conflicts. Longer waits (a month, three months) only make it worse.

  • The received wisdom of a two-week wait for change approvals was an old wives' tale
  • Forcing a pause batches up changes and creates instability, not safety
  • Assuming 'if two weeks is good, a month is better' compounds merge conflicts and integration pain

the common knowledge was that you had to have at least a two-week wait for change approvals in order to get that stability

Nicole Forsgren · 11:00

if two weeks is good a month must be better or three months must be better or six months must be better and I mean…

Nicole Forsgren · 16:30
#devops#change-management#stability#itil
Myth Buster15:00

Moving Faster Makes You More Stable, Not Less

The intuition that speed trades off against reliability is wrong. Forsgren's data shows speed and stability move together with strong statistical significance. Shipping smaller changes more often shrinks the blast radius, making bugs easier to debug and recovery faster.

  • Speed and stability move in tandem with strong statistical significance
  • Frequent pushes mean very small changes and a smaller blast radius
  • Small changes are easier to debug and recover from; large batched changes are a 'ball of mud'
  • The counterintuitive answer to low quality is also to ship smaller things more often

now we say speed and stability move together most people only think about this from the speed standpoint which means when you move faster you…

Nicole Forsgren · 15:00

quality comes from Shipping faster and and also to ship faster and move help your team move faster it's Chip smaller things

Lenny Rachitsky · 18:30
#devops#deployment#quality#speed
Myth Buster22:30

Company Size Doesn't Change the Rules (Retail Excepted)

Both small and large companies claim the benchmarks are unfair to them for opposite reasons, but the research found no statistically significant difference by size. The one outlier was retail, which performed better, most likely because the 'retail apocalypse' selected for companies with strong systems.

  • No statistical difference between small and large companies
  • Large firms blame complexity; small firms blame lack of resources; both are excuses
  • Retail was the lone outlier, and it performed better
  • Likely cause: natural selection eliminated retailers without high-performing systems

on days when I was feeling real snarky I'd be like pick your excuse you've got your like drop down

Nicole Forsgren · 23:30

if you're not already a high performer in the retail space natural selection got rid of you

Nicole Forsgren · 24:30
#research#company-size#retail#benchmarks
Myth Buster36:00

Don't Dismiss Survey Data Because 'People Lie'

A common objection to developer surveys is that people lie. Forsgren pushes back: developers have no incentive to lie about a bad system they want fixed. And the 'systems don't lie' assumption is false too, since system data is routinely incomplete or missing.

  • Developers have no incentive to lie about a bad system they want fixed
  • If they do hide problems, you have a bigger hostile-workplace issue
  • System data is also frequently incomplete or wrong, which is its own kind of lie

what is their incentive to lie why would they lie about having a bad system uh because it's bad and they want it fixed

Nicole Forsgren · 36:00
#metrics#surveys#measurement

Hot Take· 2

Hot Take52:30

AI Isn't a License to Cut Your Team in Half

Forsgren pushes back on executives who see AI coding tools as a way to halve headcount. The point isn't cutting task time in half; it's freeing up cognitive space so developers can tackle harder problems.

  • AI isn't about taking a task and cutting its time in half
  • The real value is freeing cognitive space to do harder work
  • Framing it as a layoff tool misreads what the technology enables

dear CEOs or whoever who are like now I can lay off half my Workforce that's not what this is about

Nicole Forsgren · 52:30
#ai#leadership#workforce#productivity
Hot Take1:07:30

Good Strategy Is Knowing What Not to Do

Forsgren's favorite strategy quote: the key to a good strategy is knowing what not to do, and the key to executing it is actually not doing those things. As a leader you have many options but can only fund some; funding everything guarantees failures.

  • A good strategy is defined by what you choose not to do
  • Execution means actually declining the things you cut
  • Leaders can't fund everything; funding everything makes things fail

the key to having a good strategy is knowing what not to do and the key to executing a good strategy is actually not doing…

Nicole Forsgren · 1:07:30
#strategy#leadership#prioritization

Explainer· 2

Explainer36:30

Why You Always Need Both People Data and System Data

Forsgren explains that survey data and instrumented system data are complements, not substitutes. Systems can tell you lead time is fine while people reveal it takes heroics; and systems can't surface things that never enter them, like mission-critical code sitting outside version control.

  • People and system data reveal different, non-overlapping insights
  • Systems show speed is fine; people reveal it takes a Rube Goldberg machine of heroics
  • Systems can't tell you about code that isn't in the systems at all
  • One company had mission-critical code in no version control system

data from people and data from systems are really important complements because we can get certain insights from people that we'll never get from systems

Nicole Forsgren · 36:30

there was a significant portion code that was just not going into any Version Control System you're never going to find that out from your…

Nicole Forsgren · 37:30
#measurement#surveys#instrumentation#data
Explainer51:30

AI Turns Developers Into Reviewers, Not Writers

Forsgren describes how AI-enabled tools fundamentally change how developers work: instead of writing code, you spend more time reviewing generated code. An MSR study using the CUPS model found roughly half of a developer's time is now spent reviewing versus writing.

  • Working with AI tools shifts effort from writing code to reviewing code
  • An MSR paper using the CUPS model found ~50% of time is now spent reviewing
  • This changes the friction model and cognitive load developers expect

instead of just writing code or like having a short like autocomplete you spend more time reviewing code than writing code

Nicole Forsgren · 51:30

about 50 of your time now is spent reviewing versus writing

Nicole Forsgren · 52:00
#ai#productivity#code-review#research

Story· 2

Story50:00

Even '10x Engineers' Say AI Makes Them Better

Countering the assumption that top engineers won't bother with AI coding tools, Lenny recounts a dinner with several so-called 10x engineers. Rather than dismissing copilots, they said the tools made them dramatically more effective.

  • Assumption was that elite engineers wouldn't need copilot tools
  • In reality they said the tools made them 100% more effective and efficient
  • Signals AI's value isn't limited to junior developers

they were the opposite they're like this is making me 100 more effective and efficient and I love it

Lenny Rachitsky · 50:30
#ai#productivity#engineering#copilot
Story55:30

When Data Disagrees, the Survey Is Usually Right

Forsgren praises Google as a model for its systematic, phased measurement approach and heavy investment in developer surveys alongside advanced telemetry. Notably, when Google's surveys and instrumentation disagree, the surveys have essentially always turned out to be correct.

  • Google takes a systematic, phased measurement approach even when rolling out in new areas
  • They triangulate advanced instrumentation with recurring developer surveys
  • When surveys and instrumentation conflict, the surveys have been correct nearly every time

almost every time every time that I've ever heard of the surveys are correct and not the instrumentation

Nicole Forsgren · 56:00
#measurement#google#surveys#instrumentation

Tool· 2

Tool39:30

Books Nicole Forsgren Recommends Most

For measuring intangibles and starting from zero data, Forsgren recommends Douglas Hubbard's 'How to Measure Anything.' In the lightning round she names her most-recommended books overall: 'Good Strategy Bad Strategy,' 'Designing Your Life,' and 'Ender's Game.'

  • 'How to Measure Anything' by Hubbard covers intangibles and hunching data when you have none
  • 'Good Strategy Bad Strategy' by Richard Rumelt is her most-recommended book
  • 'Designing Your Life' by Bill Burnett and Dave Evans
  • 'Ender's Game' by Orson Scott Card as an easy fun read

how to measure anything and it's by Hubbard

Nicole Forsgren · 40:00

the book I've recommended absolutely the most is called good strategy bad strategy by Richard rumelt

Nicole Forsgren · 1:09:00
#books#recommendations#measurement#strategy
Tool1:11:00

Nicole's Favorite Products: Eight Sleep and Korean Face Masks

In the lightning round, Forsgren names two recently loved products: her 'big one' is the Eight Sleep, which cools the bed and gives her sleep data, and her 'little one' is inexpensive Korean face masks, specifically the COSRX brand, as end-of-day self-care.

  • Eight Sleep cools the bed and provides sleep data (she lives in hot Arizona)
  • Korean face masks as cheap, fun self-care
  • COSRX is her pick; you can get 10 for about $15

my big one is probably sleep eight

Nicole Forsgren · 1:11:00

one of my favorites is cosrx you can get 10 for like 15

Nicole Forsgren · 1:11:30
#products#sleep#self-care#recommendations

Takeaway· 3

Takeaway17:00

Progress Matters More Than Your Performance Label

Forsgren argues the benchmark tiers (low/medium/high/elite) are interesting but not the point. What actually matters is knowing where you currently stand and whether you're making progress, not chasing an 'elite' badge.

  • Being 'elite' vs 'high' performer matters less than knowing your baseline
  • The real question: can you deploy on demand, or only twice a year?
  • Distinguish whether a constraint is a business decision or a technical capability

what's most important is knowing where you are and the progress that you're making right it doesn't matter if frankly you're a high performer or…

Nicole Forsgren · 17:00
#metrics#benchmarks#improvement
Takeaway19:00

The Numbers That Define an Elite Engineering Team

Forsgren shares the DORA elite-performance thresholds, which have stayed roughly consistent since 2019. She stresses precision doesn't matter here: broad categories are fine because 'less than a day is less than a day' from a business perspective.

  • Deployment frequency: on demand
  • Lead time for changes: less than a day
  • Time to restore: less than an hour
  • Change fail rate: between zero and fifteen percent
  • Precision isn't important; general categories are enough

I'll share the Elite Performance so deployment frequency you can deploy on demand lead time for changes takes less than a day time to restore…

Nicole Forsgren · 19:00
#benchmarks#metrics#dora#elite
Takeaway1:05:00

Be Data-Informed, Not Data-Driven

On making decisions, Forsgren describes a weighted-criteria spreadsheet she shares with mentees, but her deeper point is that the exercise itself usually surfaces the answer you already hold. She calls herself data-informed rather than data-driven, admitting she'll sometimes fudge the numbers to reveal what she actually wants.

  • Just walking through the decision exercise often reveals the answer before you finish it
  • Identifying and weighting your criteria is frequently enough to decide
  • She is data-informed, not data-driven, and the data gives a class of options, not a mandate

walking through the spreadsheet is often all you need to do in order to know what the decision is

Nicole Forsgren · 1:05:00

this is where I'm data informed and I'm not data driven

Nicole Forsgren · 1:06:30
#decision-making#mentoring#data