LLenny's Podcast
← All episodes
Inbal Shani (CPO of GitHub)01 December 2023

The future of AI in software development

6Frameworks
14Insights

Episode overview

GitHub CPO Inbal Shani argues that AI coding assistants will augment developers rather than replace them, shifting attention toward systems, architecture, products, and collaboration. She explains why AI-assisted testing is underhyped, how GitHub evaluates Copilot through quality, security, productivity, time-to-value, and developer happiness, and why adoption requires thoughtful change management. The conversation also examines GitHub’s customer-led experimentation culture, GitHub Next, and Shani’s leadership lesson that change succeeds only when people understand why it is necessary.

Key ideas

  • AI coding tools still require a human in the loop and should be treated as copilots, not autonomous pilots.
  • AI assistance can let junior developers spend earlier career time understanding systems, architecture, environments, and products instead of only learning basic code construction.
  • AI-driven testing—from unit and functional tests to load, performance, security, and penetration testing—is underhyped as code generation accelerates.
  • Companies should begin with a customer or workflow problem and determine how AI can help, rather than adding AI merely because it is fashionable.
  • Copilot’s design minimizes friction by fitting naturally into developers’ existing workflows instead of demanding another separate tool or complex interaction.
  • No single metric captures AI’s value; GitHub combines code quality, security, collaboration, productivity, time-to-value, and developer happiness.
  • GitHub creates innovation through customer feedback, freedom to experiment and fail forward, and close links between future-focused researchers and product engineering.
  • Leaders driving change must explain why it matters and bring people along, especially when entering an established team or a different culture.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Self-Mastery5 steps

Choose Your Abstraction Layer Deliberately

Delegate the work you don't enjoy to a higher abstraction; keep the parts that make you good and happy.

Read →
Innovation4 steps

Problem-First AI Adoption

Start from the customer problem and ask where AI helps — never from 'what do we do with AI?'

Read →
Leadership5 steps

Take People With You Before You Fix Anything

A new leader's toolbox is worthless until the team agrees a problem exists — build the why first.

Read →
Innovation6 steps

The Horizon-3 Research Team That Ships

Fund a 3-5 year research team, then bolt it to product so ideas actually reach production.

Read →
Leadership5 steps

The Input-Metrics Stack for Developer Productivity

No single metric rules them all — combine per-component input metrics that ladder up to time-to-value.

Read →
Innovation5 steps

The Zero-Ask Assistant (Copilot Design Philosophy)

If the user has to ask for it or wait for it, they won't adopt it — build assistance that infers.

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 5

  • Inbal ShaniMentionsInbal is Chief product officer at GitHub formerly a general manager at AWS and at Microsoft

    “Inbal is Chief product officer at GitHub formerly a general manager at AWS and at Microsoft”

    ▶ Watch at ~0:30approx.
  • Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter

    “today my guest is Inbal Shani”

    ▶ Watch at ~0:30approx.
  • Dalia FeldheimCoinedmentioned

    “dare to lead like a Girl by D Dalia feldheim”

    ▶ Watch at ~46:30approx.
  • Jim CollinsCoinedmentioned

    “Good to Great by Jim Collins”

    ▶ Watch at ~46:30approx.
  • John C. MaxwellCoinedmentioned

    “failing forward by John Maxwell”

    ▶ Watch at ~46:30approx.

Resources · 32

  • AWSMentionscompany · Amazon

    “formerly a general manager at AWS and at Microsoft”

    ▶ Watch at ~0:30approx.
  • Amazon RoboticsMentionscompany

    “also senior software developer on Amazon robotics”

    ▶ Watch at ~0:30approx.
  • MicrosoftMentionscompany · Bill Gates and Paul Allen

    “formerly a general manager at AWS and at Microsoft”

    ▶ Watch at ~0:30approx.
  • AtlassianMentionscompany

    “the new prioritization and Road mapping tool built for product teams by atlassian”

    ▶ Watch at ~1:30approx.
  • JiraUsessoftware · Atlassian

    “it's all built on jira where your engineering teams already working”

    ▶ Watch at ~2:00approx.
  • Jira Product DiscoveryRecommendssoftware · Atlassian

    “try it for free at atlassian dcom Lenny”

    ▶ Watch at ~2:30approx.
  • SanityRecommendssoftware · Sanity

    “head over to sanity.io Lenny to get started for free”

    ▶ Watch at ~4:00approx.
  • GitHub CopilotMentionssoftware · GitHub

    “co-pilot is a co-pilot it's not a pilot”

    ▶ Watch at ~8:00approx.
  • ShopifyUsescompany · Shopify

    “at Shopify over a million lines of code and their code base was written by co-pilot”

    ▶ Watch at ~10:00approx.
  • AccentureUsescompany

    “Accenture is a recent one that we got where they had 88% of suggested code was retained”

    ▶ Watch at ~11:30approx.
  • ChatGPTUsessoftware · OpenAI

    “let's incorporate open let's incorporate chat GPT”

    ▶ Watch at ~15:30approx.
  • OpenAIUsescompany

    “let's incorporate open let's incorporate chat GPT”

    ▶ Watch at ~15:30approx.
  • GitHubUsessoftware · GitHub

    “we're using GitHub to run GitHub”

    ▶ Watch at ~18:00approx.
  • GitHub Advanced SecurityMentionssoftware · GitHub

    “when we're implementing AI to get up Advan security writing more secure code is the right element”

    ▶ Watch at ~21:00approx.
  • DORAMentionsother · Nicole Forsgren and collaborators

    “this framework Dora that's an interesting way of just measuring developer experience developer happiness”

    ▶ Watch at ~22:30approx.
  • ChameleonMentionscompany

    “this episode is brought to you by help bar by chameleon”

    ▶ Watch at ~23:30approx.
  • HelpBarRecommendssoftware · Chameleon

    “sign up for help bar today it's free and easy to set up in minutes”

    ▶ Watch at ~24:30approx.
  • JavaMentionssoftware · James Gosling

    “then Java became a thing”

    ▶ Watch at ~27:30approx.
  • PythonMentionssoftware · Guido van Rossum

    “then python came which is another abstraction layer”

    ▶ Watch at ~27:30approx.
  • GitHub Copilot WorkspaceMentionssoftware · GitHub

    “we basically shared our next vision for for co-pilot with co-pilot workspace”

    ▶ Watch at ~34:00approx.
  • GitHub NextMentionsother · GitHub

    “that's our GitHub next it's basically our research team”

    ▶ Watch at ~37:00approx.
  • TomTomMentionscompany

    “in TomTom when I was leading the the location based Services team”

    ▶ Watch at ~43:30approx.
  • Failing ForwardRecommendsbook · John C. Maxwell

    “failing forward by John Maxwell”

    ▶ Watch at ~46:30approx.
  • Good to GreatRecommendsbook · Jim Collins

    “Good to Great by Jim Collins”

    ▶ Watch at ~46:30approx.
  • Dare to Lead Like a GirlRecommendsbook · Dalia Feldheim

    “a recent book that I read that is called dare to lead like a Girl”

    ▶ Watch at ~46:30approx.
  • All the Light We Cannot SeeRecommendsbook · Anthony Doerr

    “all the lights we cannot see amazing so well produced”

    ▶ Watch at ~46:30approx.
  • NetflixMentionscompany · Netflix

    “there is a recent one that came up on Netflix”

    ▶ Watch at ~46:30approx.
  • Amazon Prime VideoMentionsproduct · Amazon

    “have you seen Wheel of Time on Amazon Prime if you like fantasy”

    ▶ Watch at ~47:00approx.
  • The Wheel of TimeRecommendsbook · Robert Jordan

    “have you seen Wheel of Time on Amazon Prime if you like fantasy”

    ▶ Watch at ~47:00approx.
  • Lenny's Podcast websiteRecommendswebsite · Lenny Rachitsky

    “you can find all past episodes or learn more about the show at Lenny podcast.com”

    ▶ Watch at ~49:30approx.
  • Apple PodcastsRecommendssoftware · Apple Inc.

    “you can subscribe to the show on Apple podcast Spotify or your favorite podcast app”

    ▶ Watch at ~49:30approx.
  • SpotifyRecommendssoftware · Spotify

    “you can subscribe to the show on Apple podcast Spotify or your favorite podcast app”

    ▶ Watch at ~49:30approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster05:00

The Most Overhyped Idea: That Generative AI Will Replace Humans

Inbal Shani names the biggest overhype in software development as the belief that generative AI will replace humans and solve everything. She argues you always need a human in the loop because AI cannot replace innovation and creative thinking. To her, generative AI is a tool, not the pilot.

  • About 92% of developers already use AI tools, so AI is now table stakes
  • The overhype is that generative AI will replace humanity and be the solution for everything
  • AI cannot replace the creative spark and innovation at the center of humanity
  • Generating good AI output requires humans acting with tools to produce the data

“you always need that human in the loop because AI cannot replace Innovation right that that creative spark that creative thinking that is the center…”

Inbal Shani · 05:30

“the overhype that is happening right now is that generative V will replace Humanity it's going to be the solution for everything and I think…”

Inbal Shani · 06:00
#ai#generative-ai#hype#future-of-work

Hot Take· 4

Hot Take12:00

You Cannot Cut Your Engineers Because AI Made Them Faster

Shani pushes back hard on the idea that efficiency gains mean fewer engineers. Because most developers spend less than 20-25% of their time actually writing code, giving them time back should free them for breaks, collaboration, and creative thinking, not layoffs. Copilot is a co-pilot, not a pilot.

  • You cannot cut people; you must keep a human in the loop
  • Developers have accumulated tasks: testing, meetings, collaboration, waiting on builds, legacy code
  • Most developers spend less than 20-25% of their time writing code
  • Time returned should fuel breaks, collaboration, and innovation, not headcount cuts

“no let let me be very clear you cannot cut your people you have to have a human in the Lo pilot is a co-pilot…”

Inbal Shani · 12:00

“most developers spend less than 25 some say less than 20% of their time writing code”

Inbal Shani · 13:00
#copilot#jobs#productivity#management
Hot Take20:30

There's No One Metric to Rule Them All — And Time Is a Trap

Shani argues there is no single right metric for AI productivity; you need a combination measuring quality, security, collaboration, and ultimately developer happiness. She warns that time is seductive but misleading because you can write really bad code really fast. GitHub is shifting toward 'time to value' as a better business measure.

  • No single metric captures AI's impact; use a combination
  • Productivity is not the right metric for every component (e.g. secure code)
  • Time is not quantifiable as success because bad code can be written fast
  • Developer happiness is the ultimate metric; 'time to value' is the business framing

“there is no one uh one metric to roll them all”

Inbal Shani · 20:30

“time is not quantifiable as a success metrix because you can write really bad code really fast”

Inbal Shani · 22:00
#metrics#productivity#measurement#developer-happiness
Hot Take25:30

Sketch-to-App Is a Collaboration Tool, Not a Production Tool

Asked about demos that turn a sketch into a working website, Shani reframes them as collaboration tools rather than production tools. The real bottleneck is clarity of communication between people building a product. She sees Copilot becoming a universal conversation language, the way math once was.

  • Sketch-to-app is better seen as collaboration than production
  • The core challenge is clarity of thought and communication between personas
  • A drawing fed to Copilot shortens back-and-forth clarification cycles
  • Copilot can become a universal translation layer, like math

“co-pilot becoming that next natural language conversation it's that translator that helps bring everyone on the same page”

Inbal Shani · 26:00
#collaboration#copilot#communication#future
Hot Take30:30

The Future Is Hybrid Models, Not One Giant Generative AI

Shani predicts the industry has pivoted too hard toward a single generalized AI that solves everything, and that the future will scale back toward niche, specialized models. She expects a hybrid, multi-model world where general LLMs coexist with tuned models for high-stakes domains. Self-driving cars and aerospace are examples she doesn't see running on ChatGPT.

  • The pivot to one generalized generative AI has gone too far
  • Future is hybrid: general LLMs plus niche models for specific problems
  • Generative AI is limited by its training set and statistical guessing
  • High-safety domains like self-driving and aerospace need dedicated tuned models

“I believe that the future is in going back to scale back a bit to the Nichi so we will live in a hybrid world…”

Inbal Shani · 31:00
#ai#llms#future#hybrid-models

Explainer· 4

Explainer06:00

The Underhyped Frontier: AI-Driven Testing

Shani argues that AI-driven testing is the most underhyped area in software development. As AI generates far more code, testing becomes a far more critical part of the journey, spanning unit, functional, load, performance, security, and penetration testing. She wants to hear more about how companies use AI to generate testing suites at speed.

  • More AI-generated code means the importance of testing rises sharply
  • Testing spans unit, functional, load, performance, and security/penetration testing
  • Writing all these tests by hand needs many humans with different thinking
  • As every company becomes a software company, testing becomes more critical

“I'm starting hearing a little bit more on AI driven testing but there's not lots of mention on it”

Inbal Shani · 06:00
#testing#ai#software-quality
Explainer17:00

GitHub Runs on GitHub: Dogfooding Before Anything Ships

Shani explains that GitHub is the first to use every feature it builds, across engineering, finance, legal, and HR, not just the product team. Nothing ships to customers until it has spent months 'cooking' internally. The logic: if GitHub can't use its own tools, customers can't either.

  • GitHub runs on GitHub across finance, legal, and HR, not just engineering
  • The product team was among the first to test Copilot chat, search, and PR summaries
  • Nothing ships until it spends months being used inside GitHub first
  • If GitHub can't use a tool, customers won't be able to

“we eat our own du F and that means that we are the first to try every feature every capability that we're developing”

Inbal Shani · 17:00

“nothing is shipped before it spends enough months cooking inside GitHub”

Inbal Shani · 18:30
#dogfooding#product#github#culture
Explainer18:30

Copilot's Design Philosophy: If You Have to Ask For It, You've Lost

Shani describes Copilot's design philosophy as working backwards from the developer's shoes. If a tool is an extra step you have to ask for or wait for, developers won't adopt it. Given how much developers already juggle, the integration had to be seamless and intuitive or it would fail.

  • Design starts by putting yourself in the developer's shoes
  • If you have to ask for or wait for the tool, developers won't adopt it
  • The 2020 effort paired GitHub engineers, design, and OpenAI GPT
  • Added friction, chat, or complexity kills adoption

“the more you add friction the more you add CH the more you add complexity developers will not want to use their tool”

Inbal Shani · 20:00
#design#copilot#developer-experience#adoption
Explainer35:30

GitHub Next: The Research Team That Invents the Future

Shani describes GitHub Next, the applied and research scientist team whose job is to invent the future of software development on a three-to-five-year horizon, and where Copilot emerged from. She contrasts it with famous but unsuccessful innovation teams at other companies. The keys are the right people with freedom to innovate, plus a strong synergy that forces ideas into production instead of becoming an internal 'university.'

  • GitHub Next are applied/research scientists thinking on a 3-5 year horizon
  • Their job is to invent the future; Copilot emerged from this team
  • Innovation teams fail when they become an internal 'university' that ships nothing
  • They also fail when used too tactically, becoming just another Horizon-1 team
  • Success comes from right people plus tight synergy with product and engineering

“their job is to invent the future”

Inbal Shani · 37:00

“we're focusing on making things real so we're not keeping them far or in this connect from the product and Engineering”

Inbal Shani · 38:00
#innovation#research#github-next#product

Story· 1

Story42:00

Her Biggest Learning: Driving Change Too Fast Without the Why

In the podcast's 'failure corner,' Shani reframes failures as learnings and shares her biggest one: early as a leader at TomTom, her go-go-go energy meant she drove change without building understanding of the why. The team told her she was moving too fast, and she learned to slow down, explain the why, and take people on the journey. She notes that people inside a system stop seeing the gaps a newcomer sees.

  • She reframes failures as learnings and opportunities
  • Early as a leader she drove change without explaining why, losing the team
  • It happened leading the location-based services team at TomTom
  • Feedback was to slow down, explain the why, and take people with you
  • Long-tenured teams stop seeing gaps a newcomer immediately notices

“you're moving too fast slow down explain the why why are we doing that how should we move forward”

Inbal Shani · 43:30

“you often don't see these gaps anymore because you got used to living with them”

Inbal Shani · 45:00
#leadership#change-management#failure#career

Q&A· 1

Q&A47:00

The Two Interview Questions Inbal Shani Loves to Ask

Shani shares two favorite interview questions that reveal a candidate's character. The first asks for the most innovative thing they've done and why, which separates people who cite big inventions from those who share something personal and vulnerable. The second asks about a time they disagreed with their manager and what they did, testing whether they'll stand their ground and their influencing skills.

  • Q1: What is the most innovative thing you've done and why do you think it's innovative?
  • Answers reveal whether people cite big inventions or something personal/vulnerable
  • Q2: Describe a time you disagreed with your manager and what you did about it
  • Q2 reveals whether they stand their ground and their communication/influence skills

“what is the most innovative you have done and why do you think it's Innovative”

Inbal Shani · 47:00

“give me an example in a time where you had a disagreement with your manager”

Inbal Shani · 47:30
#hiring#interviewing#leadership

Tool· 1

Tool46:30

The Three Books Inbal Shani Recommends Most

In the lightning round, Shani names the books she recommends most to others. Two are leadership classics and one is a newer read on leadership. It's the first mention of the newest title on the show.

  • Failing Forward by John Maxwell
  • Good to Great by Jim Collins
  • Dare to Lead Like a Girl by Dalia Feldheim (a new mention on the show)

“failing forward by John Maxwell the fly Will from Good to Great by Jim Collins and a recent book that I read that is called…”

Inbal Shani · 46:30
#books#leadership#recommendations

Takeaway· 2

Takeaway10:30

The Copilot Numbers: 1.5M Developers, 55% Faster Coding

Shani shares Copilot adoption and productivity data. Over 37,000 organizations and more than 1.5 million developers use Copilot, and survey data shows large gains in speed, confidence, and satisfaction. She cites Accenture retaining 88% of suggested code.

  • Over 37,000 organizations and more than 1.5 million developers use Copilot
  • Developers write code 55% faster per surveys
  • 85% felt more confident in their code quality; code reviews completed 15% faster
  • 88% felt less frustrated and more focused; Accenture retained 88% of suggested code

“they're writing code based on our surveys 55% faster”

Inbal Shani · 10:30

“then 88% felt less frustrated and more focused on their ability for coding”

Inbal Shani · 11:30
#copilot#productivity#adoption#metrics
Takeaway14:00

Stop Asking 'What Do We Do With AI' — Work Backwards From the Problem

Shani says the biggest mistake teams make is treating AI as something they must 'do something with' because of hype. The better approach is to work backwards from the customer problem and then ask whether AI is the best tool to solve it. She describes customers having lightbulb moments when they hear how GitHub reasoned from developer workflows rather than plastering AI on everything.

  • Companies expect change to happen magically just by handing out a tool
  • Hype creates pressure to 'do something with AI' rather than solve a real problem
  • Start from the customer problem, then choose whether AI is the best tool
  • GitHub started from wanting developers to spend more time coding, then added AI

“what is that problem that we're trying to solve and how can we leverage AI better to help solve the problem versus what do we…”

Inbal Shani · 14:30
#ai-strategy#product#working-backwards