LLenny's Podcast
← All episodes
Nicole Forsgren19 October 2025

How to measure AI developer productivity in 2025

8Frameworks
15Insights

Frameworks in this episode

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 2

Myth Buster12:00

"Most Productivity Metrics Are A Lie" — Why Lines Of Code Is Broken

Nicole is blunt that most productivity metrics are a lie, and lines of code has always been bad — now catastrophically so. Because LLMs are verbose by definition, you can prompt for the longest possible code, gaming the metric while injecting complexity and technical debt.

  • Lines of code was always a poor proxy for output or complexity
  • AI blows the metric out of the water because agents and LLMs are verbose
  • Gaming it introduces complexity and technical debt across your work
  • It becomes more useful only if you can separate human-written from AI-generated code

I will say most productivity metrics are a lie.

Nicole Forsgren · 12:00

if the goal is more lines of code, I can prompt something to write you the longest piece of code ever and add tons of…

Nicole Forsgren · 12:30
#metrics#lines-of-code#measurement
Myth Buster56:30

Why Nicole Doesn't Trust Happiness Surveys For Engineers

Nicole pushes back on happiness surveys: happiness is driven by too many things — work, family, hobbies, weekends — to be a useful engineering signal. Satisfaction is different and measurable: you can ask whether someone is satisfied with a specific tool and follow up. Happy devs do write better code, but you influence that through satisfaction, not by measuring happiness directly.

  • Happiness is too all-encompassing to be a useful metric (work, family, hobbies, weekends)
  • Satisfaction is narrower and answerable: 'Are you satisfied with this tool?'
  • Satisfaction and happiness are related but not the same
  • Happy devs write better code, but target satisfaction to get there

I don't love a happiness survey because there are too many things that contribute to happiness.

Nicole Forsgren · 57:00

happy devs make happy code.

Nicole Forsgren · 57:30
#surveys#satisfaction#measurement

Hot Take· 1

Hot Take29:00

"Faster For What?" — You Can Ship Trash Faster Every Day

Nicole's counterpoint to speed obsession: you can always move faster, but faster for what? Without strategy and smart decisions about what to ship, teams just ship trash faster every single day. Speed only compounds value when the strategy underneath it is right — garbage in, garbage out.

  • You can always move faster, but the real question is 'faster for what?'
  • Without strategy you can ship trash faster every single day
  • Strategy is the core piece; speed multiplies whatever decisions feed it
  • This is where PMs matter — deciding what to build and in what order

you can always move faster, but faster for what?

Nicole Forsgren · 29:00

we can ship trash faster every single day.

Nicole Forsgren · 29:00
#strategy#speed#product

Explainer· 6

Explainer07:00

What "DevX" Actually Means And Why It Gates Everything Else

Nicole defines developer experience (DevX) as what it's like to build software day-to-day: the friction developers face, the workflows they navigate, and the support they have. Her core point is that DevX is foundational: when it's poor, no amount of great tooling or process compensates.

  • DevX = the day-to-day reality of building software (friction, workflows, support)
  • When DevX is bad, the best processes and tools won't save you
  • Productivity sits inside DevX, alongside engineering happiness
  • Output achieved through high toil or high cognitive load eventually burns developers out

when DevX is poor, everything else just isn't going to help, right? The best processes, the best tools, the best whatever magic you have, right?

Nicole Forsgren · 07:00
#devx#developer-experience#productivity
Explainer09:00

How AI Coding Breaks — And Can Rebuild — Developer Flow State

Nicole explains that writing code with AI is now interrupt-driven: instead of locking in and typing for hours, you prompt, get code back, and review it. That fragments flow, but a well-built tool chain can also hand flow-recovery off to the machine, which reminds you of context and regenerates system diagrams.

  • The three reinforcing components of DevX: flow state, cognitive load, feedback loops
  • You no longer lock in and type large blocks of code the way you used to
  • Prompt-review-integrate cycles constantly interrupt flow
  • AI can also restore flow by reminding you of context and generating diagrams

you don't just get in the flow and lock down and write a whole bunch of code and do like the typing of a whole…

Nicole Forsgren · 09:30
#flow-state#ai-coding#cognitive-load
Explainer14:00

Why Dora And Space Can't Be Applied Blindly In The AI Era

Nicole, who created Dora and Space, warns her own frameworks can't be applied blindly now. Dora's prescriptive speed and stability metrics still assess the pipeline, but AI has changed feedback loops so they run much earlier and throughout. Space, being a framework rather than a prescription, actually adapts well to new AI contexts.

  • Prescriptive metrics like Dora must be used only the way they were prescribed
  • AI creates feedback loops much earlier and throughout the pipeline, not just at build/test
  • Space is a framework, not a prescription, so it transfers to emerging AI contexts
  • Nicole would add a 'trust' dimension since LLMs are nondeterministic

we can't just blindly apply the exist existing metrics we've used before because we'll miss super important phenomenon and and changes in the way people…

Nicole Forsgren · 14:00
#dora#space#metrics
Explainer18:30

Four Hours Of Deep Work — And Why 45-Minute Blocks Now Count

Citing Gloria Mark's research, Nicole notes humans get roughly four hours of good deep work a day. Deep work historically needed long uninterrupted chunks (two hours minimum) to reach flow. With AI handing flow-recovery partly to the machine, she asks whether a 45-minute block can now become productive too.

  • People get about four hours of good deep work per day, per Gloria Mark
  • Reaching flow historically needed ~two-hour minimum uninterrupted blocks
  • AI coding is inherently interrupt-driven now
  • Because the machine helps you re-enter flow, shorter blocks may become useful

Cubans can get about four hours of good deep work a day and like that's about it.

Nicole Forsgren · 18:30

does that mean that now we can also make a 45minute work block useful?

Nicole Forsgren · 20:00
#deep-work#flow-state#time-management
Explainer27:00

The Smells That Say Your Engineering Team Could Move Faster

Nicole says most teams can move faster and she doesn't know a single team near the diminishing-returns ceiling. The tells: constant build breakages, flaky tests, overly long processes, painful environment provisioning, and — a strong smell — people refusing to switch teams because the switching cost and friction are so high.

  • Most teams can move faster; almost none are near the point of diminishing returns
  • Listen for builds breaking, flaky tests, and overly long processes
  • Hard-to-provision environments and painful task switching are friction signals
  • Some companies made switching orgs cost the same 'tax' as a brand-new hire

Most teams can move faster, right?

Nicole Forsgren · 27:00

if you're always hearing about bills breaking, flaky tests, uh, overly long processes

Nicole Forsgren · 27:30
#engineering-velocity#friction#diagnostics
Explainer33:30

AI's Productivity Gains Are Real — But We Can't Measure Them Yet

Nicole says the gains are real but we lack good measures. The strongest signal is end-to-end velocity from idea to experiment, though it's hard to attribute to a specific tool or developer. One study she saw found AI generated significantly more code for regular users — and the engineers' own output increase was double what the coding agent contributed, suggesting AI mainly unblocks work you'd already do.

  • Gains are real but there are no great measures yet
  • Best signal is velocity through the whole system, idea to experiment
  • Attribution to a specific AI tool or developer is hard
  • One finding: engineers' output increase was double the coding agent's own contribution
  • AI's main effect may be unblocking work you'd already do

I'd say it's real. And I would also say we don't have great measures for it yet.

Nicole Forsgren · 33:30

AI was generating like significantly more code for the people who were using it regularly.

Nicole Forsgren · 34:30
#ai-productivity#measurement#velocity

Story· 2

Story10:30

The Senior Engineer Who Orchestrates Parallel AI Agents Like Production Code

Nicole describes senior engineers she's watched set up an incredible workflow: they prime the AI with the architecture, stack, and conventions up front, have it design the plan, then assign an agent to each piece to work in parallel. Because they were systematic early, the output lands much closer to production code than something vibe-coded.

  • Prime the model with intended architecture, stack, and workflow before building
  • Let the AI design it, then assign an agent per piece to run in parallel
  • Specify up front that the pieces must work together with correct APIs and conventions
  • Front-loaded systematic planning yields near-production code, not vibe-coded output

And then for each piece, it'll assign an agent to go work on each piece in parallel.

Nicole Forsgren · 11:00

they're much closer to something that looks like production code

Nicole Forsgren · 11:30
#ai-agents#workflow#parallelization
Story23:30

The Mainframe Fix That Was Just Sending An Email

Nicole recounts a company that avoided an ancient, hated mainframe process because they assumed fixing it meant a full replatform. In reality, someone had to physically print a document, walk it down several flights for approval, then walk it back up. The fix wasn't a rebuild — they just had it send an email.

  • The team dodged the process because they thought it required replatforming
  • The actual bottleneck was manual: print, walk down floors, get approval, walk back up
  • The fix was changing the process, not redesigning anything major
  • Often the highest-impact improvement is a low-lift process change

Sometimes all you have to do is change a process

Nicole Forsgren · 24:00

They just had it send an email.

Nicole Forsgren · 24:30
#process-improvement#friction#case-study

Tool· 1

Tool58:00

Claude Code As A "Little Mini God" For Non-Engineering Tasks

Nicole and Lenny riff on Claude Code beyond coding. Nicole loves it and has a post coming on non-engineering use cases. Lenny describes using it to clean up laptop storage — like ChatGPT running on your computer — and quotes Dan Shipper calling it the most underrated AI tool because people don't realize it's not just for coding.

  • Claude Code works well for non-engineering tasks, not just coding
  • Example: asking it to find and clean up storage on your laptop
  • It's like ChatGPT running directly on your computer with system access
  • Dan Shipper called it the most underrated AI tool out there

I love claw code.

Nicole Forsgren · 58:00

clock code is the most underrated AI tool out there because people don't realize what it's capable of.

Lenny Rachitsky · 59:00
#claude-code#ai-tools#productivity

Takeaway· 3

Takeaway22:30

The One Thing To Do This Week: Talk To Developers, Don't Build A Tool

Asked for one tactical move a team can make this week, Nicole says go talk to people and listen — start with listening, not tools and automation. Teams too often build a tool for a problem they personally had; instead, ask developers to walk through yesterday and name where they got frustrated or slowed down.

  • Start with listening, not tools and automation
  • Companies over-build tools for problems that are just easy to automate
  • Ask developers to walk through yesterday: what was delightful, what caused friction
  • A handful of conversations surfaces low-lift, high-impact fixes

Honestly, I think the best thing you can do is go talk to people and listen.

Nicole Forsgren · 22:30

And I would say start with listening and not with tools and automation.

Nicole Forsgren · 22:30
#listening-tour#developer-experience#tactics
Takeaway49:00

To Measure AI Impact, Start With What Leadership Actually Cares About

Nicole's advice for measuring AI's productivity impact: it depends on what your leadership chain cares about. Listen to the recurring theme — market share, profit margin, velocity, transformation — then pick two or three metrics that map to it, and reframe your work in their language rather than making them work to understand it.

  • Start from the theme leadership keeps repeating
  • Market share → measure speed from feature to production/customer/experiment
  • Profit margin → measure money saved, then translate to recouped headcount or vendor spend
  • Reframe in their words; don't make them work to understand the value

it depends on what your leadership chain really cares about, right?

Nicole Forsgren · 49:30

We don't want to make them work to understand what it is that we're doing and the value that we provide.

Nicole Forsgren · 51:00
#measurement#leadership#communication
Takeaway55:30

Don't Ask Four Questions At Once — And Let Claude Draft Your Survey

Nicole warns that a question like "Were the build and test systems slow or complicated last week?" secretly asks four things, making the signal impossible to untangle. She recommends working with someone who knows survey design — or having a couple of rounds with Claude, Gemini, or ChatGPT to refine questions — and a rule of thumb: if you can't answer a question with the data, don't collect it.

  • Compound survey questions bundle multiple variables and muddy the signal
  • Talk to someone familiar with survey design
  • Use Claude, Gemini, or ChatGPT across a couple of rounds to refine questions
  • If you can't answer a question with the data, don't collect it

Were the build and test system slow or complicated in the last week?

Nicole Forsgren · 55:30

If you can't answer a question with data, don't get it.

Nicole Forsgren · 56:00
#survey-design#ai-tools#data