LLenny's Podcast
← All episodes
Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter)10 April 2025

OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more

7Frameworks
15Insights

Frameworks in this episode

Innovation4 steps

Decompose-and-Ensemble

Break a broad problem into specific tasks, then solve each with a specialized model in an ensemble.

Read →
Innovation4 steps

Eval-Driven Product Development

Design the test for your AI product alongside the product, then hill-climb the model against it.

Read →
Strategy3 steps

Model Maximalism

Build at the edge of what models can barely do — the next model will make it sing.

Read →
Productivity2 steps

Poor Man's Fine-Tuning (Few-Shot + Role Priming)

Steer a model with in-prompt examples and a role identity instead of a full fine-tune.

Read →
Innovation3 steps

Reason About It Like a Human

To design or debug AI behavior, ask what an equivalent human would do in the same situation.

Read →
Leadership5 steps

Ship-Fast Operating Model

Lightweight bottoms-up planning, PM-light teams, and iterative deployment in a fast-moving domain.

Read →
Entrepreneurship3 steps

Vertical AI Opportunity Finder

Build where the winning data is locked behind a company or industry's walls — foundation models won't go there.

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 4

Hot Take00:00

Today's AI Is the Worst You'll Ever Use

Weil's core mental model for reasoning about AI: whatever model you touch today is the least capable version you will ever experience, because a materially better one arrives every couple of months. Internalizing this changes how you build — you should design for capabilities that are 'almost there' rather than today's limits.

  • Every two months computers can do something they've never done before
  • Unlike traditional software, the technology you build on is not fixed — it keeps improving underneath you
  • The right posture is 'model maximalism' — build for where the puck is going, not where it is

“The AI models that you're using today is the worst AI model you will ever use for the rest of your life.”

Kevin Weil · 00:00

“Every 2 months, computers can do something they've never been able to do before, and you need to completely think differently about what you're doing.”

Kevin Weil · 00:00
#ai#product#mindset#openai
Hot Take31:00

Model Maximalism: Build on the Edge of What's Possible

Weil describes OpenAI's 'model maximalism' philosophy — don't over-invest in scaffolding around a model's current weaknesses, because in two months a better model will erase them. His advice to developers: if your product barely works because it's at the edge of model capabilities, keep going, because it will soon 'sing.'

  • Avoid heavy scaffolding around model limitations that the next model will fix anyway
  • Products sitting right at the edge of capability are a sign you're doing something right
  • A couple months of model improvement can turn a barely-working product into a great one

“our general mindset is in two months there's going to be a better model and it's going to blow away whatever you know the current…”

Kevin Weil · 31:30

“keep going because you're doing something right because you give it another couple months and the models are going to be great and suddenly the…”

Kevin Weil · 32:00
#ai#product#philosophy#openai
Hot Take41:00

Why Chat Is Actually the Ideal AI Interface

Against the common view that chat is a placeholder for something better, Weil argues chat is an amazing, uniquely versatile interface because it mirrors how humans naturally communicate — unstructured, high-bandwidth language. It only works now because LLMs can finally handle the nuance of human speech, so chat is precisely fit to their power.

  • Chat is versatile because it's the low-restriction way humans already communicate
  • More rigid interfaces would cap the range of what you could express to a model
  • Chat never worked before because no model understood the complexity of human speech; more prescribed interfaces still make sense for high-volume, narrow tasks

“I actually think chat is an amazing interface because it's so versatile.”

Kevin Weil · 41:30

“So to me it's like an interface that's exactly fit to the power of these things.”

Kevin Weil · 42:30
#interface#chat#llm#product
Hot Take1:06:00

Why a Billion-Kid AI Tutor Should Already Exist

Weil says personalized tutoring may be one of the most important things AI can do, and he's genuinely surprised no one has built a two-billion-kid AI tutoring product yet. The models are good enough now, ChatGPT is free, Android devices are everywhere, and research consistently shows combining classroom learning with personalized tutoring yields multiple-standard-deviation gains.

  • Combining classrooms with personalized tutoring produces multiple standard deviation improvements in learning speed
  • Models are already good enough and ChatGPT is free and globally accessible
  • Weil is surprised no massive personalized-tutoring product exists yet and wants it to

“when you combine that with personalized tutoring, you get like multiple standard deviation improvements in learning”

Kevin Weil · 1:07:00

“the models are good enough. Like it still just kind of blows my mind that there isn't something amazing out there”

Kevin Weil · 1:07:30
#education#tutoring#ai#impact

Explainer· 4

Explainer17:30

Model Accuracy Dictates What Product You Can Build

Weil explains that LLMs break the classic software contract of defined inputs producing defined outputs — they take fuzzy inputs and give fuzzy, non-deterministic outputs. Because of that, the model's success rate on a use case fundamentally changes the product you should build around it.

  • Traditional computers give the same output every time; LLMs give 'spiritually the same answer' but not the same words
  • A use case the model gets right 60% of the time needs a totally different product than one it gets right 99.5% of the time
  • You have to get into the weeds on your specific use case and its evals to know what to build

“If the model gets it right 60% of the time, you build a very different product than if the model gets it right 95% of…”

Kevin Weil · 18:00
#ai#product#llm#evals
Explainer18:30

Writing Evals Is Becoming a Core Product Skill

Weil explains what an eval is — essentially a quiz or unit test that measures how good a model is at a specific task — and why writing them is quickly becoming a must-have skill for product managers. Evals aren't just static measurement; you design them alongside the product and 'hill climb' on them to make the model better at your hero use cases.

  • An eval is like a quiz or unit test that gauges a model's capability on a set of tasks
  • OpenAI designed deep-research evals in parallel with the product, then fine-tuned the model against them
  • Evals turn model quality into a continuous, measurable learning process rather than a static hope

“I I think the easiest way to think about it is almost like a a quiz for a model”

Kevin Weil · 19:00
#evals#product-management#ai#skills
Explainer55:30

What Vibe Coding Actually Means

Weil explains vibe coding (a term he credits to Andrej Karpathy): with tools like Cursor, Windsurf, and Copilot getting better, you increasingly 'let go of the wheel' — accepting the model's suggestions, pasting errors back in, and telling it to keep going. It's not yet for tight production code, but it's ideal for proofs of concept and demos.

  • Vibe coding = accepting model suggestions rapidly instead of hand-editing every step
  • When code doesn't compile you paste the error back and say 'go go go'
  • Not for production-tight code yet, but great for demos and proofs of concept — teams should be doing it more instead of Figma mockups

“as the models are getting better, and as people are getting more used to it, you can kind of just like uh let go of…”

Kevin Weil · 56:00

“And it's not that you would do that for production code that needed to be super uh tight today yet, but for so many things.”

Kevin Weil · 56:30
#vibe-coding#coding#tools#ai
Explainer1:00:00

Break Problems Down and Ensemble Specialized Models

Weil reveals OpenAI uses ensembles of models far more than people realize — solving 10 problems might mean 20 model calls across differently-sized and fine-tuned models with custom prompts, chosen by latency and cost needs. His advice: break a problem into specific tasks and use specialized models for each, rather than throwing a single generic prompt at one big model.

  • OpenAI solves problems with many specialized model calls, not one generic model
  • Model choice per sub-task is driven by latency and cost (e.g. 4o-mini for fast cheap checks)
  • Weil's analogy: a company is itself an ensemble of fine-tuned 'models' — people with different skills combined for better output

“you want to break the problem down into more specific tasks versus some broader set of highle tasks and then you can use models very…”

Kevin Weil · 1:00:30

“in general it's like specific models for specific purposes and then you you you ensemble them together to solve problems”

Kevin Weil · 1:02:30
#ai#architecture#fine-tuning#ensembles

Story· 3

Story06:00

When Internal Usage Explodes, You Know It'll Work

Weil describes a repeated career signal for a winning product: when the team can't stop using it internally. It happened with Instagram Stories and again with the new image model, where an internal gallery generated non-stop buzz. Especially for social products, if it isn't taking off inside the company, question what you're doing.

  • The Instagram Stories team felt it would work because they used it themselves over weekends
  • OpenAI's image model had 'non-stop buzz' in an internal generation gallery before launch
  • For social products, weak internal adoption is a warning sign

“More than anything else in my career, we could feel it was going to work because we were all using it internally”

Kevin Weil · 06:00
#product#launch#signal#instagram
Story1:18:00

Libra: The Biggest Disappointment of Weil's Career

Weil calls the Facebook cryptocurrency project Libra the biggest disappointment of his career, because it solved very real problems — regressive remittance fees where people pay ~20% and wait days to send money home. In hindsight he'd have introduced the change more gently rather than launching a new blockchain, a currency basket, and WhatsApp/Messenger integration all at once during Facebook's reputational low point.

  • Remittances are regressive — people pay outrageous fees (~20%) and wait days to send money home
  • Libra tried too much at once: new blockchain, basket of currencies, and messaging integration
  • Bad timing amid Facebook's reputation low; Weil owns the decisions but wishes it existed today

“Libra is probably the biggest disappointment of my career.”

Kevin Weil · 1:18:30

“Why can't you send money as immediately, as cheaply, as simply as you send a text message?”

Kevin Weil · 1:19:00
#libra#crypto#facebook#remittances
Story1:24:30

The Life Motto: Good Work, Consistently, Over a Long Time

Weil's guiding philosophy came from a Mark Zuckerberg answer on a Facebook earnings call — asked what one thing drove huge growth, Zuckerberg said sometimes it isn't any one thing, just good work consistently over a long period of time. Weil made it into a poster; people look for silver bullets, but excellence is showing up daily and letting the compounding gains add up.

  • People too often hunt for a silver bullet instead of consistent daily work
  • Small daily improvements are unnoticeable week to week but compound massively over years
  • Weil, an ultramarathon runner, frames excellence as grinding and getting a little better every day

“sometimes it's not any one thing. It's just good work consistently over a long period of time.”

Kevin Weil · 1:25:00

“a lot of like excellence is actually showing up day in and day out doing good work getting a little bit better every single day”

Kevin Weil · 1:25:30
#philosophy#motto#consistency#career

Tool· 1

Tool1:27:00

Two Prompting Tricks: Examples and Personas

Weil first wants to kill the idea that you must be a good prompt engineer — over time models should make that unnecessary. But today, two techniques work: include examples of question-and-good-answer pairs in your prompt ('poor man's fine-tuning'), and assign the model a persona ('you are the world's greatest brand marketer') to shift it into a useful mindset.

  • Including example question/good-answer pairs in the prompt acts as 'poor man's fine-tuning'
  • People don't provide in-prompt examples nearly often enough
  • Persona framing ('you are Einstein', 'the world's greatest marketer') shifts the model into a more positive mindset — a human analog to how we frame people

“you can do like, you know, effectively poor man's fine-tuning by including examples in your prompt”

Kevin Weil · 1:27:30

“You can also say things like I want you to be Einstein. Now answer this physics problem for me. Or you are the world's greatest…”

Kevin Weil · 1:28:30
#prompting#tips#llm#tools

Takeaway· 3

Takeaway24:30

Where Startups Can Win That OpenAI Won't Go

For founders worried OpenAI will crush them, Weil argues the opposite: most of the world's valuable data and processes live behind company and industry walls, and OpenAI can't and doesn't want to build for every vertical. Their focus on a great API (3M+ developers) is a bet that most AI value will be built by others.

  • There are far more smart people outside your company's walls than inside them
  • Most valuable data is industry- or company-specific and not in the training set
  • OpenAI intentionally doesn't want to build every vertical — immense opportunity exists for startups to own them

“there are way more smart people outside your walls than there are inside your walls.”

Kevin Weil · 25:00

“And there are immense opportunities in every industry and every vertical in the world to go build AI based products that improve upon the the…”

Kevin Weil · 26:00
#startups#moats#api#strategy
Takeaway36:00

The Counterintuitive Trick: Reason About AI Like a Human

Weil's most surprising lesson from building AI products: you can often figure out how an AI feature should work — or why an AI behavior occurs — by reasoning about what an equivalent human would do. It shaped the reasoning-model UI (little status updates instead of silence or a firehose of thoughts) and the use of model ensembles that mimic group brainstorming.

  • The reasoning model's 'thinking' UI was designed by asking how a human would behave while pausing to think
  • Ensembles of models attacking one problem then integrating outputs mirror human brainstorming
  • Treating the model like a person gives useful intuition for designing AI experiences

“there's just like all these situations where you can actually kind of reason about it like a group of humans or an individual human and…”

Kevin Weil · 39:00
#ai#product-design#reasoning#ux
Takeaway48:00

Stay PM-Light and Hire for High Agency

OpenAI runs with surprisingly few PMs (~25) on purpose — Weil believes too many PMs fill the world with decks instead of execution and tempt micromanagement. He'd rather have PMs stretched across too many product-focused engineers, and he hires for high agency and comfort with extreme ambiguity, which makes it a hard fit for junior PMs.

  • OpenAI has roughly 25 PMs and deliberately keeps the ratio PM-light
  • Too many PMs produces decks and micromanagement rather than shipping
  • Hiring bar is high agency, comfort with ambiguity, and leading through influence — hard for early-career PMs who want defined scope

“my personal belief is that you want to be pretty PM light as an organization just in general.”

Kevin Weil · 48:00

“high agency, very comfortable with ambiguity, ready to come in and help execute and move really quickly.”

Kevin Weil · 51:00
#product-management#hiring#org-design#openai