LLenny's Podcast
← All episodes
Inbal Shani (CPO of GitHub)01 December 2023

The future of AI in software development

6Frameworks
14Insights

Frameworks in this episode

Self-Mastery5 steps

Choose Your Abstraction Layer Deliberately

Delegate the work you don't enjoy to a higher abstraction; keep the parts that make you good and happy.

Read →
Innovation4 steps

Problem-First AI Adoption

Start from the customer problem and ask where AI helps — never from 'what do we do with AI?'

Read →
Leadership5 steps

Take People With You Before You Fix Anything

A new leader's toolbox is worthless until the team agrees a problem exists — build the why first.

Read →
Innovation6 steps

The Horizon-3 Research Team That Ships

Fund a 3-5 year research team, then bolt it to product so ideas actually reach production.

Read →
Leadership5 steps

The Input-Metrics Stack for Developer Productivity

No single metric rules them all — combine per-component input metrics that ladder up to time-to-value.

Read →
Innovation5 steps

The Zero-Ask Assistant (Copilot Design Philosophy)

If the user has to ask for it or wait for it, they won't adopt it — build assistance that infers.

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster05:00

The Most Overhyped Idea: That Generative AI Will Replace Humans

Inbal Shani names the biggest overhype in software development as the belief that generative AI will replace humans and solve everything. She argues you always need a human in the loop because AI cannot replace innovation and creative thinking. To her, generative AI is a tool, not the pilot.

  • About 92% of developers already use AI tools, so AI is now table stakes
  • The overhype is that generative AI will replace humanity and be the solution for everything
  • AI cannot replace the creative spark and innovation at the center of humanity
  • Generating good AI output requires humans acting with tools to produce the data

“you always need that human in the loop because AI cannot replace Innovation right that that creative spark that creative thinking that is the center…”

Inbal Shani · 05:30

“the overhype that is happening right now is that generative V will replace Humanity it's going to be the solution for everything and I think…”

Inbal Shani · 06:00
#ai#generative-ai#hype#future-of-work

Hot Take· 4

Hot Take12:00

You Cannot Cut Your Engineers Because AI Made Them Faster

Shani pushes back hard on the idea that efficiency gains mean fewer engineers. Because most developers spend less than 20-25% of their time actually writing code, giving them time back should free them for breaks, collaboration, and creative thinking, not layoffs. Copilot is a co-pilot, not a pilot.

  • You cannot cut people; you must keep a human in the loop
  • Developers have accumulated tasks: testing, meetings, collaboration, waiting on builds, legacy code
  • Most developers spend less than 20-25% of their time writing code
  • Time returned should fuel breaks, collaboration, and innovation, not headcount cuts

“no let let me be very clear you cannot cut your people you have to have a human in the Lo pilot is a co-pilot…”

Inbal Shani · 12:00

“most developers spend less than 25 some say less than 20% of their time writing code”

Inbal Shani · 13:00
#copilot#jobs#productivity#management
Hot Take20:30

There's No One Metric to Rule Them All — And Time Is a Trap

Shani argues there is no single right metric for AI productivity; you need a combination measuring quality, security, collaboration, and ultimately developer happiness. She warns that time is seductive but misleading because you can write really bad code really fast. GitHub is shifting toward 'time to value' as a better business measure.

  • No single metric captures AI's impact; use a combination
  • Productivity is not the right metric for every component (e.g. secure code)
  • Time is not quantifiable as success because bad code can be written fast
  • Developer happiness is the ultimate metric; 'time to value' is the business framing

“there is no one uh one metric to roll them all”

Inbal Shani · 20:30

“time is not quantifiable as a success metrix because you can write really bad code really fast”

Inbal Shani · 22:00
#metrics#productivity#measurement#developer-happiness
Hot Take25:30

Sketch-to-App Is a Collaboration Tool, Not a Production Tool

Asked about demos that turn a sketch into a working website, Shani reframes them as collaboration tools rather than production tools. The real bottleneck is clarity of communication between people building a product. She sees Copilot becoming a universal conversation language, the way math once was.

  • Sketch-to-app is better seen as collaboration than production
  • The core challenge is clarity of thought and communication between personas
  • A drawing fed to Copilot shortens back-and-forth clarification cycles
  • Copilot can become a universal translation layer, like math

“co-pilot becoming that next natural language conversation it's that translator that helps bring everyone on the same page”

Inbal Shani · 26:00
#collaboration#copilot#communication#future
Hot Take30:30

The Future Is Hybrid Models, Not One Giant Generative AI

Shani predicts the industry has pivoted too hard toward a single generalized AI that solves everything, and that the future will scale back toward niche, specialized models. She expects a hybrid, multi-model world where general LLMs coexist with tuned models for high-stakes domains. Self-driving cars and aerospace are examples she doesn't see running on ChatGPT.

  • The pivot to one generalized generative AI has gone too far
  • Future is hybrid: general LLMs plus niche models for specific problems
  • Generative AI is limited by its training set and statistical guessing
  • High-safety domains like self-driving and aerospace need dedicated tuned models

“I believe that the future is in going back to scale back a bit to the Nichi so we will live in a hybrid world…”

Inbal Shani · 31:00
#ai#llms#future#hybrid-models

Explainer· 4

Explainer06:00

The Underhyped Frontier: AI-Driven Testing

Shani argues that AI-driven testing is the most underhyped area in software development. As AI generates far more code, testing becomes a far more critical part of the journey, spanning unit, functional, load, performance, security, and penetration testing. She wants to hear more about how companies use AI to generate testing suites at speed.

  • More AI-generated code means the importance of testing rises sharply
  • Testing spans unit, functional, load, performance, and security/penetration testing
  • Writing all these tests by hand needs many humans with different thinking
  • As every company becomes a software company, testing becomes more critical

“I'm starting hearing a little bit more on AI driven testing but there's not lots of mention on it”

Inbal Shani · 06:00
#testing#ai#software-quality
Explainer17:00

GitHub Runs on GitHub: Dogfooding Before Anything Ships

Shani explains that GitHub is the first to use every feature it builds, across engineering, finance, legal, and HR, not just the product team. Nothing ships to customers until it has spent months 'cooking' internally. The logic: if GitHub can't use its own tools, customers can't either.

  • GitHub runs on GitHub across finance, legal, and HR, not just engineering
  • The product team was among the first to test Copilot chat, search, and PR summaries
  • Nothing ships until it spends months being used inside GitHub first
  • If GitHub can't use a tool, customers won't be able to

“we eat our own du F and that means that we are the first to try every feature every capability that we're developing”

Inbal Shani · 17:00

“nothing is shipped before it spends enough months cooking inside GitHub”

Inbal Shani · 18:30
#dogfooding#product#github#culture
Explainer18:30

Copilot's Design Philosophy: If You Have to Ask For It, You've Lost

Shani describes Copilot's design philosophy as working backwards from the developer's shoes. If a tool is an extra step you have to ask for or wait for, developers won't adopt it. Given how much developers already juggle, the integration had to be seamless and intuitive or it would fail.

  • Design starts by putting yourself in the developer's shoes
  • If you have to ask for or wait for the tool, developers won't adopt it
  • The 2020 effort paired GitHub engineers, design, and OpenAI GPT
  • Added friction, chat, or complexity kills adoption

“the more you add friction the more you add CH the more you add complexity developers will not want to use their tool”

Inbal Shani · 20:00
#design#copilot#developer-experience#adoption
Explainer35:30

GitHub Next: The Research Team That Invents the Future

Shani describes GitHub Next, the applied and research scientist team whose job is to invent the future of software development on a three-to-five-year horizon, and where Copilot emerged from. She contrasts it with famous but unsuccessful innovation teams at other companies. The keys are the right people with freedom to innovate, plus a strong synergy that forces ideas into production instead of becoming an internal 'university.'

  • GitHub Next are applied/research scientists thinking on a 3-5 year horizon
  • Their job is to invent the future; Copilot emerged from this team
  • Innovation teams fail when they become an internal 'university' that ships nothing
  • They also fail when used too tactically, becoming just another Horizon-1 team
  • Success comes from right people plus tight synergy with product and engineering

“their job is to invent the future”

Inbal Shani · 37:00

“we're focusing on making things real so we're not keeping them far or in this connect from the product and Engineering”

Inbal Shani · 38:00
#innovation#research#github-next#product

Story· 1

Story42:00

Her Biggest Learning: Driving Change Too Fast Without the Why

In the podcast's 'failure corner,' Shani reframes failures as learnings and shares her biggest one: early as a leader at TomTom, her go-go-go energy meant she drove change without building understanding of the why. The team told her she was moving too fast, and she learned to slow down, explain the why, and take people on the journey. She notes that people inside a system stop seeing the gaps a newcomer sees.

  • She reframes failures as learnings and opportunities
  • Early as a leader she drove change without explaining why, losing the team
  • It happened leading the location-based services team at TomTom
  • Feedback was to slow down, explain the why, and take people with you
  • Long-tenured teams stop seeing gaps a newcomer immediately notices

“you're moving too fast slow down explain the why why are we doing that how should we move forward”

Inbal Shani · 43:30

“you often don't see these gaps anymore because you got used to living with them”

Inbal Shani · 45:00
#leadership#change-management#failure#career

Q&A· 1

Q&A47:00

The Two Interview Questions Inbal Shani Loves to Ask

Shani shares two favorite interview questions that reveal a candidate's character. The first asks for the most innovative thing they've done and why, which separates people who cite big inventions from those who share something personal and vulnerable. The second asks about a time they disagreed with their manager and what they did, testing whether they'll stand their ground and their influencing skills.

  • Q1: What is the most innovative thing you've done and why do you think it's innovative?
  • Answers reveal whether people cite big inventions or something personal/vulnerable
  • Q2: Describe a time you disagreed with your manager and what you did about it
  • Q2 reveals whether they stand their ground and their communication/influence skills

“what is the most innovative you have done and why do you think it's Innovative”

Inbal Shani · 47:00

“give me an example in a time where you had a disagreement with your manager”

Inbal Shani · 47:30
#hiring#interviewing#leadership

Tool· 1

Tool46:30

The Three Books Inbal Shani Recommends Most

In the lightning round, Shani names the books she recommends most to others. Two are leadership classics and one is a newer read on leadership. It's the first mention of the newest title on the show.

  • Failing Forward by John Maxwell
  • Good to Great by Jim Collins
  • Dare to Lead Like a Girl by Dalia Feldheim (a new mention on the show)

“failing forward by John Maxwell the fly Will from Good to Great by Jim Collins and a recent book that I read that is called…”

Inbal Shani · 46:30
#books#leadership#recommendations

Takeaway· 2

Takeaway10:30

The Copilot Numbers: 1.5M Developers, 55% Faster Coding

Shani shares Copilot adoption and productivity data. Over 37,000 organizations and more than 1.5 million developers use Copilot, and survey data shows large gains in speed, confidence, and satisfaction. She cites Accenture retaining 88% of suggested code.

  • Over 37,000 organizations and more than 1.5 million developers use Copilot
  • Developers write code 55% faster per surveys
  • 85% felt more confident in their code quality; code reviews completed 15% faster
  • 88% felt less frustrated and more focused; Accenture retained 88% of suggested code

“they're writing code based on our surveys 55% faster”

Inbal Shani · 10:30

“then 88% felt less frustrated and more focused on their ability for coding”

Inbal Shani · 11:30
#copilot#productivity#adoption#metrics
Takeaway14:00

Stop Asking 'What Do We Do With AI' — Work Backwards From the Problem

Shani says the biggest mistake teams make is treating AI as something they must 'do something with' because of hype. The better approach is to work backwards from the customer problem and then ask whether AI is the best tool to solve it. She describes customers having lightbulb moments when they hear how GitHub reasoned from developer workflows rather than plastering AI on everything.

  • Companies expect change to happen magically just by handing out a tool
  • Hype creates pressure to 'do something with AI' rather than solve a real problem
  • Start from the customer problem, then choose whether AI is the best tool
  • GitHub started from wanting developers to spend more time coding, then added AI

“what is that problem that we're trying to solve and how can we leverage AI better to help solve the problem versus what do we…”

Inbal Shani · 14:30
#ai-strategy#product#working-backwards