LLenny's Podcast
← All episodes
Ryan J. Salva (VP of Product at GitHub, Copilot)04 September 2022

The role of AI in product development

5Frameworks
14Insights

Frameworks in this episode

Productivity5 steps

Flow-Preserving AI Assistance Design

Design AI suggestions around the user's flow state: no panel switches, no waiting, ephemeral by default

Read →
Innovation5 steps

Persona-Framing for AI Product Behavior

Pick a human metaphor for your AI, then derive its behavioral guardrails from that role

Read →
Innovation7 steps

Research-to-Product Graduation

Move an incubated moonshot from the research lab to a product team without killing it or trapping the researchers

Read →
Strategy5 steps

The 60/30/10 Portfolio Capacity Split

Allocate team capacity across incremental wins, operations, and audacious bets before priorities compete

Read →
Communication5 steps

The Teach-Me-Something-In-One-Minute Interview

A 60-second timed teaching exercise scored on completeness, complexity, and clarity

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 2

Hot Take39:30

Framing Copilot as an AI Pair Programmer

To reason about the product's ethical and behavioral questions, the team framed Copilot as an AI pair programmer — the second developer sitting beside you, talking through ideas and making corrections. That persona clarified what 'appropriate' behavior means: a pair programmer whispering politics, slander, or slang in your ear would wreck your focus, so those outputs don't belong in the experience.

  • A pair programmer is two developers, one at the keyboard, one talking it through
  • Framing Copilot as that persona guided its acceptable behavior
  • An AI 'whispering crazy stuff' would be distracting and break focus
  • The persona became the basis for developer-experience guidelines

“our our stance on it what we ended up coming to is actually the the framing of Copilot as an AI pair programmer.”

Ryan Salva · 39:30

“Well, if Copilot is your AI pair programmer and they're whispering crazy stuff into your ear and they're bringing politics into it or gender identity…”

Ryan Salva · 40:00
#ai-ethics#product-framing#copilot#ux
Hot Take52:00

Copilot Should Augment Developers, Never Replace Them

Ryan is emphatic that Copilot is not and never will be a replacement for a developer. The team explicitly does not want it auto-generating code without a reasoning human at the keyboard, and it should not replace static analysis, unit tests, or other quality safeguards. The goal is augmentation — freeing developers for creative work and flow, not writing 90% of the code unsupervised.

  • Copilot is not a replacement for a developer and never will be
  • No auto-generated code should ship without a reasoning human deciding
  • It should not replace static analysis, unit tests, or other safeguards
  • The explicit goal is to augment, not replace, human developers

“Like, we do not want Copilot auto-generating code where a thinking, reasoning, breathing human being is not on the other side of that keyboard making…”

Ryan Salva · 52:00

“We don't want it to replace anything. We want it to augment.”

Ryan Salva · 53:00
#ai-philosophy#augmentation#copilot#developer-experience

Explainer· 4

Explainer11:30

What GitHub Copilot Actually Is: Multi-Line Autocomplete

Ryan explains Copilot as a magnified version of the intellisense autocomplete developers have used for 20 years — but generating many lines at once. It's powered by an AI model called Codex, a derivative of GPT-3, and surfaces suggestions as gray italicized text inside the editor. Copilot infers intent from surrounding variables, class names, methods, and comments, then scaffolds code the developer can riff on.

  • Copilot is multi-line autocomplete, versus traditional single-token intellisense
  • Powered by Codex, a code-trained derivative of GPT-3
  • Suggestions appear inline as gray italicized text
  • It infers intent from surrounding variables, class/method names, and comments
  • Its value is keeping developers 'in the flow' and out of docs/Stack Overflow

“Copilot kind of infers what you intend to create and then hopefully does a pretty good job at nailing it by providing scaffolding code template…”

Ryan Salva · 12:30
#copilot#ai#developer-tools#codex
Explainer25:30

Why 200 Milliseconds Is Copilot's Latency Sweet Spot

Ryan explains that performance was a key optimization target: in a code editor you can't make a developer wait one to three seconds for a suggestion without breaking their flow. Through experiments measuring how many milliseconds feel non-interruptive, the team landed on roughly 200 milliseconds as the sweet spot, varying a bit by the developer's global latency.

  • Slow suggestions (1-3 seconds) break a developer's flow
  • The team ran experiments on how many milliseconds feel non-interruptive
  • ~200 milliseconds emerged as the sweet spot
  • Actual latency varies by the developer's location in the world

“So, depending upon where you're in the world, your latency can go up or down a little bit from there, but it seems like the…”

Ryan Salva · 26:00
#latency#performance#copilot#ux
Explainer28:00

The Three Horizons: Ambiguity, Not Calendar Dates

Ryan describes how GitHub thinks about first-, second-, and third-horizon projects. As a rough ballpark, horizon one is the next year, horizon two the next three years, and horizon three the next five. But he stresses it's really a measure of ambiguity and confidence level rather than fixed calendar dates.

  • Horizon 1 ≈ next year, Horizon 2 ≈ next three years, Horizon 3 ≈ next five years
  • These are ballparks, not concrete definitions
  • The real dimension is ambiguity and confidence, not calendar time
  • GitHub ring-fenced a team (GitHub Next) for Horizon 2 and 3 work

“for me, I usually ballpark it as first horizon is the next year, second horizon the next three years, third horizon next five years, but…”

Ryan Salva · 28:30
#strategy#moonshots#portfolio#r-and-d
Explainer45:00

40% of Python Code — and Why the Number Varies by Language

Ryan clarifies the widely cited stat: the 40% figure is specifically for Python developers. The share varies by language because some languages are better represented in public training data, and both the volume and diversity of that data correlate with suggestion quality. Across all languages, Copilot's contribution ranges from the upper 20s to the 40s in percent.

  • The 40% code-written stat applies specifically to Python developers
  • Suggestion quality tracks the volume and diversity of public training data
  • Better-represented languages get better suggestions
  • Across all languages the figure ranges from the upper 20s to the 40s

“And so, just to put a fine point on that stat, it is uh 40% is specifically for Python developers.”

Ryan Salva · 45:00

“We we see it range anywhere from the upper 20s to the 40s uh across all the different languages.”

Ryan Salva · 45:30
#copilot#metrics#training-data#python

Story· 4

Story15:30

Using Copilot to Teach Kids to Code by Building Real Tools

Ryan shares a story about an educator teaching high-schoolers to code by pairing student groups with small and medium businesses that need internal tools. The students build real software with Copilot 'whispering in their ear,' so they learn by solving actual problems rather than doing arbitrary exercises. They walk away with resume-worthy projects and early fluency in the AI tooling that will define the developer stack.

  • An educator pairs groups of ~6-8 students with real small/medium businesses
  • Students build the internal tool the business needs, using Copilot for guidance
  • Learning by building something useful beats arbitrary coding exercises
  • Students get a real project for college applications plus AI-tool fluency

“many of us, we learn to code best by not by like arbitrary exercises, but by actually building something that's going to be useful, right?…”

Ryan Salva · 15:30
#education#copilot#learning-to-code
Story20:00

When OpenAI Accidentally Looked Like a DDoS on GitHub

Ryan recounts how GitHub's infrastructure team noticed they were being 'absolutely hammered' with clone requests and feared a denial-of-service attack. They quickly traced it to OpenAI cloning all of GitHub's repositories to harvest training data. Rather than block it, GitHub stepped in to mitigate the load and offered to hand OpenAI the data in a more responsible, packaged way.

  • GitHub's infra team saw a spike in clone requests and suspected a DoS attack
  • The source was OpenAI cloning all repositories to harvest training data
  • GitHub mitigated the load without an uptime incident
  • GitHub then offered to package the data more responsibly for OpenAI

“And we figured out pretty quickly that it was actually OpenAI. They were cloning all of our repositories to harvest kind of the data out…”

Ryan Salva · 20:00

“And we noticed one day that we were getting hammered, I mean, absolutely hammered with a tremendous amount of clone requests.”

Ryan Salva · 20:00
#openai#github#copilot-origin#infrastructure
Story20:30

The Arctic Code Vault Snapshot That Seeded Copilot

A year before Copilot, GitHub had snapshotted its public code onto silver film stored in the Arctic Code Vault in northern Finland — meant to preserve open source for thousands of years, alongside a nearby seed vault. That same snapshot became the responsible way to hand public code to OpenAI. Feeding it to large language models revealed you could 'translate' English to Python or Python to C#, plus generate predicted code text.

  • GitHub preserved a public-code snapshot on silver film in an Arctic vault
  • The vault sits near a seed vault meant to preserve the world's flora
  • That same snapshot was repurposed as training data for OpenAI's models
  • LLMs on code enabled language-to-language translation and predicted text

“And so we had put this snapshot of public repositories on these like this like silver film that would be preserved for thousands of years…”

Ryan Salva · 21:30
#arctic-code-vault#open-source#copilot-origin#llm
Story42:00

From Crude Blocklists to Responsible-AI Sentiment Models

Early Copilot had no content filter at all; the team then introduced a simple blocklist of words, which proved 'fraught with peril' — suddenly they were editors of language, sometimes blocking words developers legitimately needed. They eventually partnered with Azure's Department of Responsible AI, whose sentiment-detection models handle context far better than crude blocklists, since the same word can be offensive or reasonable depending on context (e.g. medical software).

  • Very early Copilot shipped with no content filter
  • A simple word blocklist made the team into reluctant editors of language
  • Blocklists sometimes blocked words developers legitimately needed
  • Azure Responsible AI models detect 'sentiment' with better context awareness
  • A parent company's contribution solved a problem GitHub couldn't alone

“And so, we introduced a simple block list of words. And these blocklists are always fraught with peril.”

Ryan Salva · 42:00

“All of a sudden we become editors, and that's just a of like language, and that's kind of a scary place to be.”

Ryan Salva · 42:00
#ai-safety#content-moderation#responsible-ai#copilot

Q&A· 1

Q&A1:01:00

Ryan's Interview Question: Teach Me Something in One Minute

Ryan's icebreaker for early-to-mid-career PM candidates is to ask them to teach him something new in one minute, timed on his phone. He grades on three criteria: completeness (did they finish inside the minute), complexity (how substantive the topic), and clarity (did he actually understand and learn something). His most memorable answer taught him about 18th-century art and its connection to religious trends — an 'extremely strong yes hire.'

  • The question: teach me something new in one minute, timed live
  • Graded on completeness, complexity, and clarity
  • Complexity rewards substantive topics over trivial tricks
  • A candidate taught him about 18th-century art and religion — a strong yes hire

“So I ask them to teach me something new in 1 minute.”

Ryan Salva · 1:01:00

“So one is completeness. Do they actually finish the lesson inside of 1 minute?”

Ryan Salva · 1:01:00
#hiring#interviewing#product-management

Tool· 1

Tool57:00

Ryan's Book Picks: 'Make It So' and 'Brief Interviews with Hideous Men'

In the lightning round Ryan recommends two very different books. 'Make It So' is a UX book (a Star Trek reference) arguing that sci-fi user experiences tend to show up in real products 20-30 years later. For entertainment, he recommends David Foster Wallace's short-story collection 'Brief Interviews with Hideous Men,' likening each story to a movie villain's monologue explaining who they are.

  • 'Make It So' — a UX book on how sci-fi interfaces predict real products
  • Its thesis: sci-fi UX arrives in everyday tools 20-30 years later
  • 'Brief Interviews with Hideous Men' by David Foster Wallace — short stories
  • Each story reads like a villain's revealing monologue

“So one of them is a book on user experience called Make It So.”

Ryan Salva · 57:30

“There's a David Foster Wallace book called Brief Interviews with Hideous Men that I love.”

Ryan Salva · 57:30
#books#recommendations#ux#reading

Takeaway· 2

Takeaway35:30

Moving a Research Project Into a Real Product Team

Ryan shares lessons on graduating a moonshot from the research shop into an operational product team. The trigger for moving researchers back can't be a calendar date — it must be a replacement in seat who has actually picked up all the skills. And the product team taking over must own its roadmap, not have innovation outsourced to it.

  • Don't move researchers back on a calendar; move them when a replacement is fully up to speed
  • Ensure continuity of expertise, skills, and domain familiarity before the handoff
  • The product team must feel it controls its own roadmap and future
  • Engineering fundamentals feel unnatural to researchers and require culture change

“that can't be based on a calendar. It needs to be based on a replacement in seat who's actually doing the job and has picked…”

Ryan Salva · 36:00

“make sure that you've got continuity of expertise and skill sets and and domain familiarity before you move over.”

Ryan Salva · 36:00
#product-management#r-and-d#team-building#incubation
Takeaway55:00

How Ryan Splits Team Capacity: ~10% Bets, ~30% Ops, ~60% Iteration

As a portfolio manager of multiple product lines, Ryan shares his rough capacity split for large teams. Around 5-10% goes to bold, uncertain experimental bets; about 25-30% to operations keeping in-market products meeting expectations; and the remaining ~60% to incremental improvement of existing products. He notes the math changes entirely at a startup, where you're essentially all-in on one bet.

  • ~5-10% of capacity on bold, uncertain experimental research
  • ~25-30% on operations for in-market products
  • ~60% on incremental improvement of existing products
  • At a startup the split collapses to being all-in on one bet

“You can kind of think of those like those really uncertain bets as being 5% to 10% of the team's capacity. About 25, maybe 30%…”

Ryan Salva · 55:30

“the remainder of it, what is that about 60% or so, is really on incremental progress for our in-market products.”

Ryan Salva · 56:00
#resource-allocation#portfolio#product-strategy