LLenny's Podcast
← All episodes
Ryan J. Salva (VP of Product at GitHub, Copilot)04 September 2022

The role of AI in product development

5Frameworks
14Insights

Episode overview

GitHub VP of Product Ryan J. Salva explains how GitHub and OpenAI turned a snapshot of public code into GitHub Copilot, an AI pair programmer powered by Codex. He describes moving the project from GitHub Next into operational product teams, using customer enthusiasm and a careful transfer of expertise to scale it. The conversation also examines latency, infrastructure, licensing, security, offensive output, and the principle that AI should augment developers rather than replace their judgment.

Key ideas

  • Copilot uses contextual signals from a developer's code and comments to offer multi-line suggestions directly inside the editor.
  • GitHub's Arctic Code Vault snapshot gave OpenAI a responsibly packaged public-code dataset for experimentation with code-focused language models.
  • Promising research products should move into operational teams only after customer validation, with researchers remaining until replacement expertise is established.
  • The product team—not the research group—must own the roadmap and customer feedback loop after a project graduates from experimentation.
  • Copilot's team targeted roughly 200-millisecond responses to preserve developer flow while experimenting extensively with prompt crafting.
  • Responsible deployment requires human review, existing testing and static-analysis safeguards, community dialogue, and contextual filtering of offensive output.
  • Salva's rough portfolio allocation is 5–10% for bold research, 25–30% for operations, and about 60% for incremental product improvements.
  • Salva expects AI to spread across the development stack by removing repetitive work while leaving developers focused on design, outcomes, and creativity.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Productivity5 steps

Flow-Preserving AI Assistance Design

Design AI suggestions around the user's flow state: no panel switches, no waiting, ephemeral by default

Read →
Innovation5 steps

Persona-Framing for AI Product Behavior

Pick a human metaphor for your AI, then derive its behavioral guardrails from that role

Read →
Innovation7 steps

Research-to-Product Graduation

Move an incubated moonshot from the research lab to a product team without killing it or trapping the researchers

Read →
Strategy5 steps

The 60/30/10 Portfolio Capacity Split

Allocate team capacity across incremental wins, operations, and audacious bets before priorities compete

Read →
Communication5 steps

The Teach-Me-Something-In-One-Minute Interview

A 60-second timed teaching exercise scored on completeness, complexity, and clarity

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 8

  • Ryan J. SalvaMentionsRyan Salva is VP of product at GitHub, where amongst other projects he incubated and launched GitHub Copilot

    “Ryan Salva is VP of product at GitHub, where amongst other projects he incubated and launched GitHub Copilot”

    ▶ Watch at ~1:00approx.
  • Tim FerrissMentionsCited for his fear-setting framework and as an influence on Paul Millerd's five-episode podcast experiment.

    “I've been hearing about AG1 on basically every podcast that I listen to, like Tim Ferriss and Lex Fridman.”

    ▶ Watch at ~3:00approx.
  • Lex FridmanMentionsthis reminds me of a story I just listened to on Lex Friedman's podcast

    “I've been hearing about AG1 on basically every podcast that I listen to, like Tim Ferriss and Lex Fridman.”

    ▶ Watch at ~3:00approx.
  • Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter

    “Ryan, welcome to the podcast.”

    ▶ Watch at ~4:30approx.
  • David Foster WallaceCoinedmentioned

    “There's a David Foster Wallace book called Brief Interviews with Hideous Men that I love.”

    ▶ Watch at ~57:30approx.
  • Nate DiMeoCoinedmentioned

    “The Memory Palace with Nate DiMeo is an excellent storytelling podcast.”

    ▶ Watch at ~59:00approx.
  • Ted ChiangMentionsanything about that Chang or Ken Liu I highly recommend

    “Have you read Ted Chiang's books and short stories?”

    ▶ Watch at ~1:00:00approx.
  • Oege de MoorCoinedthe primary researcher who really kind of like is the true innovator for Copilot

    “Hugo is the primary researcher who really kind of like is the true innovator for Copilot. He deserves credit for the initial work”

    ▶ Watch at ~1:02:30approx.

Resources · 33

  • GitHub Arctic Code VaultCoinedother · GitHub

    “we had actually created a snapshot of GitHub's public code for what we call the Arctic Code Vault”

    ▶ Watch at ~0:00approx.
  • GitHub CopilotMentionssoftware · GitHub

    “he incubated and launched GitHub Copilot, which in my opinion is one of the most magical products that you'll come across.”

    ▶ Watch at ~1:00approx.
  • GitHubMentionssoftware · GitHub

    “Ryan Salva is VP of product at GitHub”

    ▶ Watch at ~1:00approx.
  • AG1Usesproduct · Athletic Greens

    “I finally gave it a shot earlier this year, and it has quickly become a core part of my morning routine”

    ▶ Watch at ~3:00approx.
  • AmplitudeRecommendssoftware · Amplitude

    “Try Amplitude for free. Just visit amplitude.com to get started.”

    ▶ Watch at ~3:00approx.
  • MicrosoftMentionscompany · Bill Gates and Paul Allen

    “About 10 years at Microsoft and now 3 years at GitHub.”

    ▶ Watch at ~6:30approx.
  • Azure DevOpsMentionssoftware · Microsoft

    “a kind of Microsoft devops solution called Azure devops.”

    ▶ Watch at ~8:00approx.
  • Microsoft OfficeMentionssoftware · Microsoft

    “the shared developer infrastructure for all Microsoft products like Windows and Office and Azure”

    ▶ Watch at ~8:00approx.
  • Microsoft WindowsMentionssoftware · Microsoft

    “the shared developer infrastructure for all Microsoft products like Windows and Office and Azure”

    ▶ Watch at ~8:00approx.
  • Microsoft AzureMentionsproduct · Microsoft

    “the shared developer infrastructure for all Microsoft products like Windows and Office and Azure”

    ▶ Watch at ~8:00approx.
  • GitHub CodespacesMentionssoftware · GitHub

    “a you know, cloud-hosted development environments like code spaces”

    ▶ Watch at ~9:00approx.
  • GitHub ActionsMentionssoftware · GitHub

    “it's also actions for CICD and advanced security.”

    ▶ Watch at ~10:00approx.
  • GitHub Advanced SecurityMentionssoftware · GitHub

    “it's also actions for CICD and advanced security.”

    ▶ Watch at ~10:00approx.
  • GitHub CopilotUsessoftware · GitHub

    “Essentially as you are typing, Copilot will provide suggestions usually in kind of this like italicized gray text”

    ▶ Watch at ~12:00approx.
  • CodexMentionssoftware · OpenAI

    “It is multi-line auto complete that is fundamentally powered by an AI model called Codex”

    ▶ Watch at ~12:00approx.
  • GPT-3Mentionssoftware · OpenAI

    “Codex which is a derivative of another kind of one that you might be familiar with GPT-3.”

    ▶ Watch at ~12:00approx.
  • Stack OverflowUseswebsite · Joel Spolsky and Jeff Atwood

    “preventing them from having to go check out documentation or watch a tutorial or go to Stack Overflow”

    ▶ Watch at ~13:30approx.
  • FinlandMentionsplace

    “Essentially, this is up in like way in the Northlands of Finland.”

    ▶ Watch at ~21:00approx.
  • Visual Studio CodeMentionssoftware · Microsoft

    “partner with our friends in Visual Studio Code.”

    ▶ Watch at ~23:00approx.
  • GitHub NextCoinedother · GitHub

    “the original team that was working on Copilot at GitHub was, you know, the team that we call GitHub Next.”

    ▶ Watch at ~27:30approx.
  • Modern TreasuryMentionscompany

    “Modern Treasury is a next-generation operating system for moving and tracking money.”

    ▶ Watch at ~28:30approx.
  • Hacker NewsMentionswebsite · Y Combinator

    “threads on Hacker News about people getting really, really excited about it.”

    ▶ Watch at ~34:00approx.
  • DALL-EMentionssoftware · OpenAI

    “We're getting a lot lot better at it especially in the last couple of years, you know, with things like Dolly and Copilot.”

    ▶ Watch at ~39:00approx.
  • Make It SoRecommendsbook

    “one of them is a book on user experience called Make It So.”

    ▶ Watch at ~57:00approx.
  • Star TrekMentionstv show

    “It's a reference back to Star Trek”

    ▶ Watch at ~57:00approx.
  • Brief Interviews with Hideous MenRecommendsbook · David Foster Wallace

    “There's a David Foster Wallace book called Brief Interviews with Hideous Men that I love.”

    ▶ Watch at ~57:30approx.
  • The Memory PalaceRecommendspodcast · Nate DiMeo

    “The Memory Palace with Nate DiMeo is an excellent storytelling podcast.”

    ▶ Watch at ~59:00approx.
  • ArrivalRecommendsfilm

    “I don't know how if this counts as recent, but it's it's one that I watched recently, which was Arrival.”

    ▶ Watch at ~1:00:00approx.
  • Vanderbilt UniversityMentionscompany

    “She was from Vanderbilt University.”

    ▶ Watch at ~1:02:00approx.
  • GitHub CopilotRecommendssoftware · GitHub

    “there is a 60-day free trial of co-pilot that is you know, there for everyone to pick up and use.”

    ▶ Watch at ~1:03:30approx.
  • TwitterUseswebsite · Twitter

    “I am Ryan J Salva everywhere. Twitter, GitHub, any you know, pick your pick your choice.”

    ▶ Watch at ~1:03:30approx.
  • LinkedInUseswebsite · LinkedIn

    “LinkedIn, Ryan J Salva.”

    ▶ Watch at ~1:03:30approx.
  • GitHub DiscussionsRecommendswebsite · GitHub

    “post either on Twitter or Hacker News or on discussions, GitHub discussions, your experience.”

    ▶ Watch at ~1:04:00approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 2

Hot Take39:30

Framing Copilot as an AI Pair Programmer

To reason about the product's ethical and behavioral questions, the team framed Copilot as an AI pair programmer — the second developer sitting beside you, talking through ideas and making corrections. That persona clarified what 'appropriate' behavior means: a pair programmer whispering politics, slander, or slang in your ear would wreck your focus, so those outputs don't belong in the experience.

  • A pair programmer is two developers, one at the keyboard, one talking it through
  • Framing Copilot as that persona guided its acceptable behavior
  • An AI 'whispering crazy stuff' would be distracting and break focus
  • The persona became the basis for developer-experience guidelines

“our our stance on it what we ended up coming to is actually the the framing of Copilot as an AI pair programmer.”

Ryan Salva · 39:30

“Well, if Copilot is your AI pair programmer and they're whispering crazy stuff into your ear and they're bringing politics into it or gender identity…”

Ryan Salva · 40:00
#ai-ethics#product-framing#copilot#ux
Hot Take52:00

Copilot Should Augment Developers, Never Replace Them

Ryan is emphatic that Copilot is not and never will be a replacement for a developer. The team explicitly does not want it auto-generating code without a reasoning human at the keyboard, and it should not replace static analysis, unit tests, or other quality safeguards. The goal is augmentation — freeing developers for creative work and flow, not writing 90% of the code unsupervised.

  • Copilot is not a replacement for a developer and never will be
  • No auto-generated code should ship without a reasoning human deciding
  • It should not replace static analysis, unit tests, or other safeguards
  • The explicit goal is to augment, not replace, human developers

“Like, we do not want Copilot auto-generating code where a thinking, reasoning, breathing human being is not on the other side of that keyboard making…”

Ryan Salva · 52:00

“We don't want it to replace anything. We want it to augment.”

Ryan Salva · 53:00
#ai-philosophy#augmentation#copilot#developer-experience

Explainer· 4

Explainer11:30

What GitHub Copilot Actually Is: Multi-Line Autocomplete

Ryan explains Copilot as a magnified version of the intellisense autocomplete developers have used for 20 years — but generating many lines at once. It's powered by an AI model called Codex, a derivative of GPT-3, and surfaces suggestions as gray italicized text inside the editor. Copilot infers intent from surrounding variables, class names, methods, and comments, then scaffolds code the developer can riff on.

  • Copilot is multi-line autocomplete, versus traditional single-token intellisense
  • Powered by Codex, a code-trained derivative of GPT-3
  • Suggestions appear inline as gray italicized text
  • It infers intent from surrounding variables, class/method names, and comments
  • Its value is keeping developers 'in the flow' and out of docs/Stack Overflow

“Copilot kind of infers what you intend to create and then hopefully does a pretty good job at nailing it by providing scaffolding code template…”

Ryan Salva · 12:30
#copilot#ai#developer-tools#codex
Explainer25:30

Why 200 Milliseconds Is Copilot's Latency Sweet Spot

Ryan explains that performance was a key optimization target: in a code editor you can't make a developer wait one to three seconds for a suggestion without breaking their flow. Through experiments measuring how many milliseconds feel non-interruptive, the team landed on roughly 200 milliseconds as the sweet spot, varying a bit by the developer's global latency.

  • Slow suggestions (1-3 seconds) break a developer's flow
  • The team ran experiments on how many milliseconds feel non-interruptive
  • ~200 milliseconds emerged as the sweet spot
  • Actual latency varies by the developer's location in the world

“So, depending upon where you're in the world, your latency can go up or down a little bit from there, but it seems like the…”

Ryan Salva · 26:00
#latency#performance#copilot#ux
Explainer28:00

The Three Horizons: Ambiguity, Not Calendar Dates

Ryan describes how GitHub thinks about first-, second-, and third-horizon projects. As a rough ballpark, horizon one is the next year, horizon two the next three years, and horizon three the next five. But he stresses it's really a measure of ambiguity and confidence level rather than fixed calendar dates.

  • Horizon 1 ≈ next year, Horizon 2 ≈ next three years, Horizon 3 ≈ next five years
  • These are ballparks, not concrete definitions
  • The real dimension is ambiguity and confidence, not calendar time
  • GitHub ring-fenced a team (GitHub Next) for Horizon 2 and 3 work

“for me, I usually ballpark it as first horizon is the next year, second horizon the next three years, third horizon next five years, but…”

Ryan Salva · 28:30
#strategy#moonshots#portfolio#r-and-d
Explainer45:00

40% of Python Code — and Why the Number Varies by Language

Ryan clarifies the widely cited stat: the 40% figure is specifically for Python developers. The share varies by language because some languages are better represented in public training data, and both the volume and diversity of that data correlate with suggestion quality. Across all languages, Copilot's contribution ranges from the upper 20s to the 40s in percent.

  • The 40% code-written stat applies specifically to Python developers
  • Suggestion quality tracks the volume and diversity of public training data
  • Better-represented languages get better suggestions
  • Across all languages the figure ranges from the upper 20s to the 40s

“And so, just to put a fine point on that stat, it is uh 40% is specifically for Python developers.”

Ryan Salva · 45:00

“We we see it range anywhere from the upper 20s to the 40s uh across all the different languages.”

Ryan Salva · 45:30
#copilot#metrics#training-data#python

Story· 4

Story15:30

Using Copilot to Teach Kids to Code by Building Real Tools

Ryan shares a story about an educator teaching high-schoolers to code by pairing student groups with small and medium businesses that need internal tools. The students build real software with Copilot 'whispering in their ear,' so they learn by solving actual problems rather than doing arbitrary exercises. They walk away with resume-worthy projects and early fluency in the AI tooling that will define the developer stack.

  • An educator pairs groups of ~6-8 students with real small/medium businesses
  • Students build the internal tool the business needs, using Copilot for guidance
  • Learning by building something useful beats arbitrary coding exercises
  • Students get a real project for college applications plus AI-tool fluency

“many of us, we learn to code best by not by like arbitrary exercises, but by actually building something that's going to be useful, right?…”

Ryan Salva · 15:30
#education#copilot#learning-to-code
Story20:00

When OpenAI Accidentally Looked Like a DDoS on GitHub

Ryan recounts how GitHub's infrastructure team noticed they were being 'absolutely hammered' with clone requests and feared a denial-of-service attack. They quickly traced it to OpenAI cloning all of GitHub's repositories to harvest training data. Rather than block it, GitHub stepped in to mitigate the load and offered to hand OpenAI the data in a more responsible, packaged way.

  • GitHub's infra team saw a spike in clone requests and suspected a DoS attack
  • The source was OpenAI cloning all repositories to harvest training data
  • GitHub mitigated the load without an uptime incident
  • GitHub then offered to package the data more responsibly for OpenAI

“And we figured out pretty quickly that it was actually OpenAI. They were cloning all of our repositories to harvest kind of the data out…”

Ryan Salva · 20:00

“And we noticed one day that we were getting hammered, I mean, absolutely hammered with a tremendous amount of clone requests.”

Ryan Salva · 20:00
#openai#github#copilot-origin#infrastructure
Story20:30

The Arctic Code Vault Snapshot That Seeded Copilot

A year before Copilot, GitHub had snapshotted its public code onto silver film stored in the Arctic Code Vault in northern Finland — meant to preserve open source for thousands of years, alongside a nearby seed vault. That same snapshot became the responsible way to hand public code to OpenAI. Feeding it to large language models revealed you could 'translate' English to Python or Python to C#, plus generate predicted code text.

  • GitHub preserved a public-code snapshot on silver film in an Arctic vault
  • The vault sits near a seed vault meant to preserve the world's flora
  • That same snapshot was repurposed as training data for OpenAI's models
  • LLMs on code enabled language-to-language translation and predicted text

“And so we had put this snapshot of public repositories on these like this like silver film that would be preserved for thousands of years…”

Ryan Salva · 21:30
#arctic-code-vault#open-source#copilot-origin#llm
Story42:00

From Crude Blocklists to Responsible-AI Sentiment Models

Early Copilot had no content filter at all; the team then introduced a simple blocklist of words, which proved 'fraught with peril' — suddenly they were editors of language, sometimes blocking words developers legitimately needed. They eventually partnered with Azure's Department of Responsible AI, whose sentiment-detection models handle context far better than crude blocklists, since the same word can be offensive or reasonable depending on context (e.g. medical software).

  • Very early Copilot shipped with no content filter
  • A simple word blocklist made the team into reluctant editors of language
  • Blocklists sometimes blocked words developers legitimately needed
  • Azure Responsible AI models detect 'sentiment' with better context awareness
  • A parent company's contribution solved a problem GitHub couldn't alone

“And so, we introduced a simple block list of words. And these blocklists are always fraught with peril.”

Ryan Salva · 42:00

“All of a sudden we become editors, and that's just a of like language, and that's kind of a scary place to be.”

Ryan Salva · 42:00
#ai-safety#content-moderation#responsible-ai#copilot

Q&A· 1

Q&A1:01:00

Ryan's Interview Question: Teach Me Something in One Minute

Ryan's icebreaker for early-to-mid-career PM candidates is to ask them to teach him something new in one minute, timed on his phone. He grades on three criteria: completeness (did they finish inside the minute), complexity (how substantive the topic), and clarity (did he actually understand and learn something). His most memorable answer taught him about 18th-century art and its connection to religious trends — an 'extremely strong yes hire.'

  • The question: teach me something new in one minute, timed live
  • Graded on completeness, complexity, and clarity
  • Complexity rewards substantive topics over trivial tricks
  • A candidate taught him about 18th-century art and religion — a strong yes hire

“So I ask them to teach me something new in 1 minute.”

Ryan Salva · 1:01:00

“So one is completeness. Do they actually finish the lesson inside of 1 minute?”

Ryan Salva · 1:01:00
#hiring#interviewing#product-management

Tool· 1

Tool57:00

Ryan's Book Picks: 'Make It So' and 'Brief Interviews with Hideous Men'

In the lightning round Ryan recommends two very different books. 'Make It So' is a UX book (a Star Trek reference) arguing that sci-fi user experiences tend to show up in real products 20-30 years later. For entertainment, he recommends David Foster Wallace's short-story collection 'Brief Interviews with Hideous Men,' likening each story to a movie villain's monologue explaining who they are.

  • 'Make It So' — a UX book on how sci-fi interfaces predict real products
  • Its thesis: sci-fi UX arrives in everyday tools 20-30 years later
  • 'Brief Interviews with Hideous Men' by David Foster Wallace — short stories
  • Each story reads like a villain's revealing monologue

“So one of them is a book on user experience called Make It So.”

Ryan Salva · 57:30

“There's a David Foster Wallace book called Brief Interviews with Hideous Men that I love.”

Ryan Salva · 57:30
#books#recommendations#ux#reading

Takeaway· 2

Takeaway35:30

Moving a Research Project Into a Real Product Team

Ryan shares lessons on graduating a moonshot from the research shop into an operational product team. The trigger for moving researchers back can't be a calendar date — it must be a replacement in seat who has actually picked up all the skills. And the product team taking over must own its roadmap, not have innovation outsourced to it.

  • Don't move researchers back on a calendar; move them when a replacement is fully up to speed
  • Ensure continuity of expertise, skills, and domain familiarity before the handoff
  • The product team must feel it controls its own roadmap and future
  • Engineering fundamentals feel unnatural to researchers and require culture change

“that can't be based on a calendar. It needs to be based on a replacement in seat who's actually doing the job and has picked…”

Ryan Salva · 36:00

“make sure that you've got continuity of expertise and skill sets and and domain familiarity before you move over.”

Ryan Salva · 36:00
#product-management#r-and-d#team-building#incubation
Takeaway55:00

How Ryan Splits Team Capacity: ~10% Bets, ~30% Ops, ~60% Iteration

As a portfolio manager of multiple product lines, Ryan shares his rough capacity split for large teams. Around 5-10% goes to bold, uncertain experimental bets; about 25-30% to operations keeping in-market products meeting expectations; and the remaining ~60% to incremental improvement of existing products. He notes the math changes entirely at a startup, where you're essentially all-in on one bet.

  • ~5-10% of capacity on bold, uncertain experimental research
  • ~25-30% on operations for in-market products
  • ~60% on incremental improvement of existing products
  • At a startup the split collapses to being all-in on one bet

“You can kind of think of those like those really uncertain bets as being 5% to 10% of the team's capacity. About 25, maybe 30%…”

Ryan Salva · 55:30

“the remainder of it, what is that about 60% or so, is really on incremental progress for our in-market products.”

Ryan Salva · 56:00
#resource-allocation#portfolio#product-strategy