Nicole Forsgren (Microsoft Research, GitHub, Google)30 July 2023

How to measure and improve developer productivity

4Frameworks

15Insights

Episode overview

Lenny Rachitsky talks with developer-productivity researcher Nicole Forsgren about measuring engineering performance through the DORA and SPACE frameworks. Forsgren explains why speed and stability tend to improve together, arguing for small, frequent deployments supported by sound technical, architectural, and cultural capabilities. They also discuss balancing system telemetry with developer-reported data, avoiding misleading activity metrics, and defining the problem clearly before launching an improvement program. The conversation closes with AI-assisted development, practical decision frameworks, and tactical ways to investigate barriers to developer productivity.

Key ideas

Define the specific developer-productivity or experience problem before choosing metrics or interventions.
DORA measures software-delivery speed and stability through lead time, deployment frequency, time to restore, and change-failure rate.
Smaller, more frequent deployments reduce blast radius and make failures easier to diagnose and recover from.
SPACE encourages balanced measurement across satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow.
Use at least three SPACE dimensions so optimizing an easily counted activity does not distort the wider system.
Combine telemetry from systems with qualitative data from developers because each exposes gaps the other cannot.
AI coding tools shift work toward reviewing generated code, making trust, learning, cognitive load, and overreliance important measurement questions.
A practical first step is to write down the challenge, inspect existing signals, and ask developers about barriers in their tools and processes.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Productivity4 steps

The DORA Four Keys

Measure software delivery with two speed and two stability metrics that move in tandem.

Read →

Strategy5 steps

The Four-Box Framework

Write your hypothesis in words before you touch data, so you can debug the idea and the measurement separately.

Read →

Strategy4 steps

The Measurement Journey

Start measuring from words and people, then scale into system data — don't wait for perfect.

Read →

Self-Mastery5 steps

The Weighted Decision Spreadsheet

List your options, weight your criteria to 100%, score and multiply — then let the reaction tell you the answer.

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 15

Nicole ForsgrenMentionsNicole is the developer productivity expert having written the award-winning book accelerate
“today my guest is Nicole forsgren”
▶ Watch at ~0:30approx.
Nicole ForsgrenMentionsNicole is the developer productivity expert having written the award-winning book accelerate
“Nicole is the developer productivity expert having written the award-winning book accelerate”
▶ Watch at ~0:30approx.
Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter
“welcome to Lenny's podcast where I interview world-class product leaders and growth experts”
▶ Watch at ~0:30approx.
Jez HumbleMentionsI was doing with Dora devops research and assessment uh in collaboration with just humble and Gene Kim
“I was doing with Dora devops research and assessment uh in collaboration with just humble and Gene Kim”
▶ Watch at ~5:00approx.
Gene KimMentionsI was doing with Dora devops research and assessment uh in collaboration with just humble and Gene Kim
“I was doing with Dora devops research and assessment uh in collaboration with just humble and Gene Kim”
▶ Watch at ~5:00approx.
Alanna BrownMentionswe started that work with puppet so shout out to Alana Brown for starting that
“we started that work with puppet so shout out to Alana Brown for starting that”
▶ Watch at ~5:00approx.
Nigel KerstenMentionsshout out to Alana Brown for starting that Nigel Karsten and the the team there
“shout out to Alana Brown for starting that Nigel Karsten and the the team there”
▶ Watch at ~5:00approx.
Dustin SmithMentionsit's been led by uh Dr Dustin Smith Nathan Harvey continues the work
“it's been led by uh Dr Dustin Smith Nathan Harvey continues the work”
▶ Watch at ~19:00approx.
Nathen HarveyMentionsit's been led by uh Dr Dustin Smith Nathan Harvey continues the work
“it's been led by uh Dr Dustin Smith Nathan Harvey continues the work”
▶ Watch at ~19:00approx.
Mik KerstenMentionsI wrote a paper with Mick Kirsten on this several years ago
“I wrote a paper with Mick Kirsten on this several years ago”
▶ Watch at ~36:30approx.
Orson Scott CardCoinedthe last one is probably Ender's Game horses got card
“the last one is probably Ender's Game horses got card”
▶ Watch at ~1:09:00approx.
Dave EvansCoineddesigning your life by uh Bill Burnett and Dave Abbott Dave Evans
“designing your life by uh Bill Burnett and Dave Abbott Dave Evans”
▶ Watch at ~1:09:00approx.
Bill BurnettCoinedthis book called designing your life which is written by uh these actually um you know one of the founders of Ido
“designing your life by uh Bill Burnett and Dave Abbott Dave Evans”
▶ Watch at ~1:09:00approx.
Richard RumeltCoinedRichard was a longtime professor at UCLA Anderson School of Management was on the faculty of Harvard Business School
“good strategy bad strategy by Richard rumelt”
▶ Watch at ~1:09:00approx.
John McEnroeMentionsnever have I ever which is fun because John Mcenroe narrates it
“never have I ever which is fun because John Mcenroe narrates it”
▶ Watch at ~1:10:00approx.

Resources · 38

Lenny's PodcastCoinedpodcast · Lenny Rachitsky
“welcome to Lenny's podcast where I interview world-class product leaders and growth experts”
▶ Watch at ~0:30approx.
State of DevOps ReportCoinedpaper · DORA and collaborators
“she's been the co-author of the state of devops report year after year”
▶ Watch at ~1:00approx.
Microsoft ResearchMentionscompany
“she's currently a partner at Microsoft research leading developer productivity research and strategy”
▶ Watch at ~1:00approx.
DXMentionssoftware · DX
“today's entire episode is brought to you by DX a platform for measuring and improving developer productivity”
▶ Watch at ~1:30approx.
getdx.com/LennyRecommendswebsite · DX
“visit their website at getdx.com Lenny”
▶ Watch at ~2:30approx.
IBMMentionscompany
“I started as a software engineer at IBM”
▶ Watch at ~3:00approx.
University of ArizonaMentionsplace
“I went to University of Arizona which has a very very technical degree”
▶ Watch at ~4:00approx.
PuppetMentionscompany
“we started that work with puppet”
▶ Watch at ~5:00approx.
ChefMentionscompany
“Chef uh the little configuration management startup at the time uh hired me”
▶ Watch at ~5:30approx.
GitHubMentionssoftware · GitHub
“after that point uh I joined GitHub which is you know this the largest uh developer Network”
▶ Watch at ~6:30approx.
GoogleMentionscompany · Google
“we were then acquired by Google”
▶ Watch at ~6:30approx.
DORAMentionsother · Nicole Forsgren and collaborators
“Dora is it's an entire research program”
▶ Watch at ~14:00approx.
dora.devRecommendswebsite · Google DORA team
“you can go to dora.dev and find all of the continued reports”
▶ Watch at ~19:00approx.
AccelerateCoinedbook · Nicole Forsgren, Jez Humble, and Gene Kim
“there's Dora there I wrote a book called accelerate”
▶ Watch at ~25:30approx.
DORA Quick CheckRecommendstool · Google DORA team
“if you go to dora.dev the the team at Google was lovely”
▶ Watch at ~27:30approx.
DORA Quick CheckUsestool · Google DORA team
“you can take a quick check there's a button there that says quick check”
▶ Watch at ~28:00approx.
SPACE FrameworkCoinedother · Nicole Forsgren and collaborators
“we called it space because it stands for the five Dimensions that you want to measure”
▶ Watch at ~30:30approx.
DORAUsesother · Nicole Forsgren and collaborators
“Dora would be space four mostly that outer loop”
▶ Watch at ~32:30approx.
SPACE FrameworkRecommendsother · Nicole Forsgren and collaborators
“this space paper for sure”
▶ Watch at ~38:30approx.
The SPACE of Developer ProductivityRecommendspaper · Nicole Forsgren and collaborators
“this space paper for sure it's at ACM”
▶ Watch at ~38:30approx.
How to Measure AnythingRecommendsbook · Douglas Hubbard
“one book that I love it's it's a little dense but it's really interesting that I love is how to measure anything”
▶ Watch at ~39:30approx.
GitHub CopilotUsessoftware · GitHub
“people really fundamentally shift the way they work when they work with an AI enabled tool like GitHub copilot”
▶ Watch at ~51:30approx.
TabnineUsessoftware · Tabnine
“an AI enabled tool like GitHub copilot or like you know tab 9 or others”
▶ Watch at ~51:30approx.
GoogleRecommendscompany · Google
“I think Google does this incredibly well”
▶ Watch at ~55:00approx.
Lenny's NewsletterMentionsnewsletter · Lenny Rachitsky
“I have a newsletter post a guest post on how to do a correlation analysis Center regression analysis”
▶ Watch at ~1:03:30approx.
Good Strategy/Bad StrategyRecommendsbook · Richard Rumelt
“the book I've recommended absolutely the most is called good strategy bad strategy by Richard rumelt”
▶ Watch at ~1:09:00approx.
Designing Your LifeRecommendsbook · Bill Burnett and Dave Evans
“another one is designing your life by uh Bill Burnett and Dave Abbott Dave Evans”
▶ Watch at ~1:09:00approx.
Ender's GameRecommendsbook · Orson Scott Card
“the last one is probably Ender's Game horses got card”
▶ Watch at ~1:09:00approx.
Ender's GameMentionsbook · Orson Scott Card
“they made a movie and I was afraid I wasn't gonna like it”
▶ Watch at ~1:09:30approx.
The Three-Body ProblemRecommendsbook · Liu Cixin
“they're making a movie of three body problem I don't know if you've read that”
▶ Watch at ~1:10:00approx.
SuitsUsestv show · Aaron Korsh
“I'm re-watching suits again”
▶ Watch at ~1:10:00approx.
Ted LassoRecommendstv show · Bill Lawrence, Jason Sudeikis, Brendan Hunt, and Joe Kelly
“Ted lasso is a favorite”
▶ Watch at ~1:10:00approx.
Never Have I EverRecommendstv show
“I just tore through never have I ever which is fun because John Mcenroe narrates it”
▶ Watch at ~1:10:00approx.
Eight SleepRecommendsproduct · Eight Sleep
“my big one is probably sleep eight”
▶ Watch at ~1:11:00approx.
Eight SleepUsesproduct · Eight Sleep
“yeah ate sleep yeah the other way around um yeah so that one's fun because it cool it makes the bed cold”
▶ Watch at ~1:11:00approx.
COSRXRecommendscompany
“one of my favorites is cosrx”
▶ Watch at ~1:12:00approx.
AccelerateRecommendsbook · Nicole Forsgren, Jez Humble, and Gene Kim
“also pick up a copy of accelerate on Amazon or your local retail establishment”
▶ Watch at ~1:14:30approx.
lennyspodcast.comRecommendswebsite · Lenny Rachitsky
“you can find all past episodes or learn more about the show at lennyspodcast.com”
▶ Watch at ~1:16:00approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 4

Myth Buster11:00

The Two-Week Change-Approval Wait Doesn't Buy You Stability

Old ITIL/ITSM change management held that you needed a mandatory multi-week wait for change approvals to keep systems stable. Forsgren's research found the opposite: that batching-up of changes actively causes instability and merge conflicts. Longer waits (a month, three months) only make it worse.

The received wisdom of a two-week wait for change approvals was an old wives' tale
Forcing a pause batches up changes and creates instability, not safety
Assuming 'if two weeks is good, a month is better' compounds merge conflicts and integration pain

“the common knowledge was that you had to have at least a two-week wait for change approvals in order to get that stability”
Nicole Forsgren · 11:00

“if two weeks is good a month must be better or three months must be better or six months must be better and I mean…”
Nicole Forsgren · 16:30

#devops#change-management#stability#itil

Myth Buster15:00

Moving Faster Makes You More Stable, Not Less

The intuition that speed trades off against reliability is wrong. Forsgren's data shows speed and stability move together with strong statistical significance. Shipping smaller changes more often shrinks the blast radius, making bugs easier to debug and recovery faster.

Speed and stability move in tandem with strong statistical significance
Frequent pushes mean very small changes and a smaller blast radius
Small changes are easier to debug and recover from; large batched changes are a 'ball of mud'
The counterintuitive answer to low quality is also to ship smaller things more often

“now we say speed and stability move together most people only think about this from the speed standpoint which means when you move faster you…”
Nicole Forsgren · 15:00

“quality comes from Shipping faster and and also to ship faster and move help your team move faster it's Chip smaller things”
Lenny Rachitsky · 18:30

#devops#deployment#quality#speed

Myth Buster22:30

Company Size Doesn't Change the Rules (Retail Excepted)

Both small and large companies claim the benchmarks are unfair to them for opposite reasons, but the research found no statistically significant difference by size. The one outlier was retail, which performed better, most likely because the 'retail apocalypse' selected for companies with strong systems.

No statistical difference between small and large companies
Large firms blame complexity; small firms blame lack of resources; both are excuses
Retail was the lone outlier, and it performed better
Likely cause: natural selection eliminated retailers without high-performing systems

“on days when I was feeling real snarky I'd be like pick your excuse you've got your like drop down”
Nicole Forsgren · 23:30

“if you're not already a high performer in the retail space natural selection got rid of you”
Nicole Forsgren · 24:30

#research#company-size#retail#benchmarks

Myth Buster36:00

Don't Dismiss Survey Data Because 'People Lie'

A common objection to developer surveys is that people lie. Forsgren pushes back: developers have no incentive to lie about a bad system they want fixed. And the 'systems don't lie' assumption is false too, since system data is routinely incomplete or missing.

Developers have no incentive to lie about a bad system they want fixed
If they do hide problems, you have a bigger hostile-workplace issue
System data is also frequently incomplete or wrong, which is its own kind of lie

“what is their incentive to lie why would they lie about having a bad system uh because it's bad and they want it fixed”
Nicole Forsgren · 36:00

#metrics#surveys#measurement

Hot Take· 2

Hot Take52:30

AI Isn't a License to Cut Your Team in Half

Forsgren pushes back on executives who see AI coding tools as a way to halve headcount. The point isn't cutting task time in half; it's freeing up cognitive space so developers can tackle harder problems.

AI isn't about taking a task and cutting its time in half
The real value is freeing cognitive space to do harder work
Framing it as a layoff tool misreads what the technology enables

“dear CEOs or whoever who are like now I can lay off half my Workforce that's not what this is about”
Nicole Forsgren · 52:30

#ai#leadership#workforce#productivity

Hot Take1:07:30

Good Strategy Is Knowing What Not to Do

Forsgren's favorite strategy quote: the key to a good strategy is knowing what not to do, and the key to executing it is actually not doing those things. As a leader you have many options but can only fund some; funding everything guarantees failures.

A good strategy is defined by what you choose not to do
Execution means actually declining the things you cut
Leaders can't fund everything; funding everything makes things fail

“the key to having a good strategy is knowing what not to do and the key to executing a good strategy is actually not doing…”
Nicole Forsgren · 1:07:30

#strategy#leadership#prioritization

Explainer· 2

Explainer36:30

Why You Always Need Both People Data and System Data

Forsgren explains that survey data and instrumented system data are complements, not substitutes. Systems can tell you lead time is fine while people reveal it takes heroics; and systems can't surface things that never enter them, like mission-critical code sitting outside version control.

People and system data reveal different, non-overlapping insights
Systems show speed is fine; people reveal it takes a Rube Goldberg machine of heroics
Systems can't tell you about code that isn't in the systems at all
One company had mission-critical code in no version control system

“data from people and data from systems are really important complements because we can get certain insights from people that we'll never get from systems”
Nicole Forsgren · 36:30

“there was a significant portion code that was just not going into any Version Control System you're never going to find that out from your…”
Nicole Forsgren · 37:30

#measurement#surveys#instrumentation#data

Explainer51:30

AI Turns Developers Into Reviewers, Not Writers

Forsgren describes how AI-enabled tools fundamentally change how developers work: instead of writing code, you spend more time reviewing generated code. An MSR study using the CUPS model found roughly half of a developer's time is now spent reviewing versus writing.

Working with AI tools shifts effort from writing code to reviewing code
An MSR paper using the CUPS model found ~50% of time is now spent reviewing
This changes the friction model and cognitive load developers expect

“instead of just writing code or like having a short like autocomplete you spend more time reviewing code than writing code”
Nicole Forsgren · 51:30

“about 50 of your time now is spent reviewing versus writing”
Nicole Forsgren · 52:00

#ai#productivity#code-review#research

Story· 2

Story50:00

Even '10x Engineers' Say AI Makes Them Better

Countering the assumption that top engineers won't bother with AI coding tools, Lenny recounts a dinner with several so-called 10x engineers. Rather than dismissing copilots, they said the tools made them dramatically more effective.

Assumption was that elite engineers wouldn't need copilot tools
In reality they said the tools made them 100% more effective and efficient
Signals AI's value isn't limited to junior developers

“they were the opposite they're like this is making me 100 more effective and efficient and I love it”
Lenny Rachitsky · 50:30

#ai#productivity#engineering#copilot

Story55:30

When Data Disagrees, the Survey Is Usually Right

Forsgren praises Google as a model for its systematic, phased measurement approach and heavy investment in developer surveys alongside advanced telemetry. Notably, when Google's surveys and instrumentation disagree, the surveys have essentially always turned out to be correct.

Google takes a systematic, phased measurement approach even when rolling out in new areas
They triangulate advanced instrumentation with recurring developer surveys
When surveys and instrumentation conflict, the surveys have been correct nearly every time

“almost every time every time that I've ever heard of the surveys are correct and not the instrumentation”
Nicole Forsgren · 56:00

#measurement#google#surveys#instrumentation

Tool· 2

Tool39:30

Books Nicole Forsgren Recommends Most

For measuring intangibles and starting from zero data, Forsgren recommends Douglas Hubbard's 'How to Measure Anything.' In the lightning round she names her most-recommended books overall: 'Good Strategy Bad Strategy,' 'Designing Your Life,' and 'Ender's Game.'

'How to Measure Anything' by Hubbard covers intangibles and hunching data when you have none
'Good Strategy Bad Strategy' by Richard Rumelt is her most-recommended book
'Designing Your Life' by Bill Burnett and Dave Evans
'Ender's Game' by Orson Scott Card as an easy fun read

“how to measure anything and it's by Hubbard”
Nicole Forsgren · 40:00

“the book I've recommended absolutely the most is called good strategy bad strategy by Richard rumelt”
Nicole Forsgren · 1:09:00

#books#recommendations#measurement#strategy

Tool1:11:00

Nicole's Favorite Products: Eight Sleep and Korean Face Masks

In the lightning round, Forsgren names two recently loved products: her 'big one' is the Eight Sleep, which cools the bed and gives her sleep data, and her 'little one' is inexpensive Korean face masks, specifically the COSRX brand, as end-of-day self-care.

Eight Sleep cools the bed and provides sleep data (she lives in hot Arizona)
Korean face masks as cheap, fun self-care
COSRX is her pick; you can get 10 for about $15

“my big one is probably sleep eight”
Nicole Forsgren · 1:11:00

“one of my favorites is cosrx you can get 10 for like 15”
Nicole Forsgren · 1:11:30

#products#sleep#self-care#recommendations

Takeaway· 3

Takeaway17:00

Progress Matters More Than Your Performance Label

Forsgren argues the benchmark tiers (low/medium/high/elite) are interesting but not the point. What actually matters is knowing where you currently stand and whether you're making progress, not chasing an 'elite' badge.

Being 'elite' vs 'high' performer matters less than knowing your baseline
The real question: can you deploy on demand, or only twice a year?
Distinguish whether a constraint is a business decision or a technical capability

“what's most important is knowing where you are and the progress that you're making right it doesn't matter if frankly you're a high performer or…”
Nicole Forsgren · 17:00

#metrics#benchmarks#improvement

Takeaway19:00

The Numbers That Define an Elite Engineering Team

Forsgren shares the DORA elite-performance thresholds, which have stayed roughly consistent since 2019. She stresses precision doesn't matter here: broad categories are fine because 'less than a day is less than a day' from a business perspective.

Deployment frequency: on demand
Lead time for changes: less than a day
Time to restore: less than an hour
Change fail rate: between zero and fifteen percent
Precision isn't important; general categories are enough

“I'll share the Elite Performance so deployment frequency you can deploy on demand lead time for changes takes less than a day time to restore…”
Nicole Forsgren · 19:00

#benchmarks#metrics#dora#elite

Takeaway1:05:00

Be Data-Informed, Not Data-Driven

On making decisions, Forsgren describes a weighted-criteria spreadsheet she shares with mentees, but her deeper point is that the exercise itself usually surfaces the answer you already hold. She calls herself data-informed rather than data-driven, admitting she'll sometimes fudge the numbers to reveal what she actually wants.

Just walking through the decision exercise often reveals the answer before you finish it
Identifying and weighting your criteria is frequently enough to decide
She is data-informed, not data-driven, and the data gives a class of options, not a mandate

“walking through the spreadsheet is often all you need to do in order to know what the decision is”
Nicole Forsgren · 1:05:00

“this is where I'm data informed and I'm not data driven”
Nicole Forsgren · 1:06:30

#decision-making#mentoring#data

Key ideas

Frameworks in this episode

The DORA Four Keys

The Four-Box Framework

The Measurement Journey

The Weighted Decision Spreadsheet

People & resources mentioned

People · 15

Resources · 38

Insights & moments

⚡Myth Buster· 4

The Two-Week Change-Approval Wait Doesn't Buy You Stability

Moving Faster Makes You More Stable, Not Less

Company Size Doesn't Change the Rules (Retail Excepted)

Don't Dismiss Survey Data Because 'People Lie'

◆Hot Take· 2

AI Isn't a License to Cut Your Team in Half

Good Strategy Is Knowing What Not to Do

✶Explainer· 2

Why You Always Need Both People Data and System Data

AI Turns Developers Into Reviewers, Not Writers

❝Story· 2

Even '10x Engineers' Say AI Makes Them Better

When Data Disagrees, the Survey Is Usually Right

✦Tool· 2

Books Nicole Forsgren Recommends Most

Nicole's Favorite Products: Eight Sleep and Korean Face Masks

▲Takeaway· 3

Progress Matters More Than Your Performance Label

The Numbers That Define an Elite Engineering Team

Be Data-Informed, Not Data-Driven

Myth Buster· 4

Hot Take· 2

Explainer· 2

Story· 2

Tool· 2

Takeaway· 3