LLenny's Podcast
← All episodes
Aman Khan (Arize AI, ex-Spotify, Apple, Cruise)14 November 2024

Becoming an AI PM

6Frameworks
15Insights

Episode overview

Lenny Rachitsky interviews Arize AI product director Aman Khan about entering AI product management, distinguishing three varieties of AI PM, and using new tools to build convincing prototypes. Khan argues that strong AI PMs begin with customer problems rather than fashionable interfaces such as chatbots or agents, then learn through hands-on experimentation, hackathons, and product teardowns. The conversation also explores thriving as a long-term individual-contributor PM through energy, comfort with uncertainty, signal amplification, curiosity, and fun.

Key ideas

  • AI product management includes platform PMs serving AI engineers, PMs packaging model capabilities into products, and conventional PMs whose work is amplified by AI tools.
  • Aspiring AI PMs can build foundational literacy, follow their curiosity, and demonstrate ability and enthusiasm through a portfolio of working prototypes.
  • Tools such as Cursor, Replit, v0, Midjourney, and DALL-E let PMs communicate ideas with higher-resolution prototypes and visual stories.
  • AI should be selected to solve a real customer or business problem; copying chatbot or agent patterns without that grounding is a warning sign.
  • Hackathons create hands-on intuition, but teams should expect many attempted AI applications to fail and treat those failures as learning.
  • Successful AI experiences may preserve meaningful user control instead of automating everything, reflecting the IKEA effect.
  • Long-term IC PMs can increase their influence by bringing constructive energy, doing difficult work alongside teammates, and wandering into uncertain territory to find product direction.
  • AI can amplify sparse customer signal by synthesizing large collections of call transcripts, helping PMs identify recurring problems at scale.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 11

  • Aman KhanMentionsAman is director of product at arise Ai and over the course of his PM career he's focused on Building Products in the AI space

    Aman is director of product at arise Ai and over the course of his PM career he's focused on Building Products in the AI space

  • Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter

    amazing thank you so much for having me Lenny

  • Greg IsenbergMentionsI credit Greg Eisenberg he's really kind of like pioneered some of this format

    I credit Greg Eisenberg he's really kind of like pioneered some of this format

  • Mahika KapoorMentionsone of her superpowers and wellink to her episode is that she is designer engineer PM

    one of her superpowers and wellink to her episode is that she is designer engineer PM

  • Kevin YienMentionsKevin leads product for merchant experiences at stripe before that he built the restaurant business and the ecosystem teams at square

    I'll kind of Steal This one a little bit from our friend Kevin Yin

  • LeBron JamesMentionsyou're kind of like being LeBron James like you're telling you're coaching people you're kind of like drawing up the play

    you're kind of like being LeBron James like you're telling you're coaching people you're kind of like drawing up the play

  • Bill BrysonCoinedShort history of nearly everything by Bill brys

    I really love uh Short history of nearly everything by Bill brys

  • Charles DarwinMentionswe have to have some degree of and Darwin spotted this some degree of kind of random number generator in our makeup

    or like Darwin was it maybe perceived as like such an amazing scientist at the time

  • Bill BurnettCoinedthis book called designing your life which is written by uh these actually um you know one of the founders of Ido

    this book called designing your life which is written by uh these actually um you know one of the founders of Ido

  • Steve JobsCoinedSteve Jobs is just uh the bar he held for the company and for technical talent and for excellence was not uh wavering.

    the quote is from Steve Jobs um which is your time is limited so don't waste living someone else's life

  • John DoerrMentionsthere was you know this this note from John dur spoke at uh the college I went to

    there was you know this this note from John dur spoke at uh the college I went to

Resources · 49

  • SpotifyMentionssoftware · Spotify

    including at Spotify on the ml platform team and rolls at Cru zipline and apple

  • CruiseMentionscompany

    including at Spotify on the ml platform team and rolls at Cru zipline and apple

  • ZiplineMentionscompany

    including at Spotify on the ml platform team and rolls at Cru zipline and apple

  • AppleMentionscompany · Steve Jobs, Steve Wozniak, and Ronald Wayne

    including at Spotify on the ml platform team and rolls at Cru zipline and apple

  • PendoMentionssoftware · Pendo

    this episode is brought to you by pendo the only all-in-one product experience platform for any type of application

  • Pendo certification coursesRecommendscourse · Pendo

    check out pendo lineup of free certification courses led by top product experts and designed to help you grow

  • VantaMentionssoftware · Christina Cacioppo

    this episode is brought to you by vanta when it comes to ensuring your company has top-notch security practices

  • Arize AIUsescompany

    I work at a company called arise uh we're a observability and evaluation platform for artificial intelligence

  • OpenAI APIMentionssoftware · OpenAI

    with the rise of all of these tools that are being built on top of you know open AI apis or anthropic apis

  • Anthropic APIMentionssoftware · Anthropic

    with the rise of all of these tools that are being built on top of you know open AI apis or anthropic apis

  • ReplitUsessoftware · Replit

    I'm obsessed with cursor I'm obsessed with repet these are tools I use all the time to just really build a prototype

  • CursorUsessoftware · Anysphere

    I'm obsessed with cursor I'm obsessed with repet these are tools I use all the time to just really build a prototype

  • v0Usessoftware · Vercel

    versell now has uh a sort of starting point called vzer which allows you to type in a prompt

  • VercelRecommendscompany · Guillermo Rauch

    there's a really uh great tool a lot of folks might be familiar with versel if you're trying to create a landing page

  • MidjourneyRecommendssoftware · Midjourney, Inc.

    I think that becomes so much more approachable with tools like mid Journey or even Dolly

  • DALL-ERecommendssoftware · OpenAI

    I think that becomes so much more approachable with tools like mid Journey or even Dolly

  • OpenAI Realtime APIMentionssoftware · OpenAI

    a really cuttingedge one from the last couple weeks has been open AI releasing a realtime API

  • ChatGPTUsessoftware · OpenAI

    you can open up chat GPT and if you're a plus subscriber you can try this uh Cutting Edge sort of voice chat bot experience

  • Replit AgentUsessoftware · Replit

    what you can do is type in a prompt to their replit agent and it will literally build a website for you

  • FigmaMentionssoftware · Figma

    that partly has been why she was has been so successful at figma and influencing leadership to build these products

  • InstacartMentionscompany · Apoorva Mehta, Max Mullen, and Brandon Leonardo

    we have you know customers that are super well known household Brands um Spotify and instacart

  • ParagonMentionssoftware · Brandon Foo

    this episode is brought to you by Paragon the developer platform for building native customer facing Integrations with thirdparty apps

  • AI21 LabsMentionscompany

    engineering teams at copy ai ai 21 and over 100 other B2B SAS companies use Paragon

  • Copy.aiMentionscompany

    engineering teams at copy ai ai 21 and over 100 other B2B SAS companies use Paragon

  • SlackUsessoftware · Slack Technologies

    one of them was this slackbot that actually alerted a person that was on call for a problem

  • Betty Crocker cake mixMentionsproduct · Betty Crocker

    there's this story of you know Betty Crocker like actually you know launching this this cake mix

  • WaymoMentionscompany · Alphabet

    if you've been able to like take a wh or a self-driving car like the car is fully self-driving

  • The IKEA effectMentionsother

    that's kind of like the Ikea effect applied to AI like people feel more empowered when they can

  • Blue ApronMentionscompany

    it reminds me of like blue apron and all these food food making

  • DoorDashMentionscompany · DoorDash

    you'd probably build something Clos to like door Dash which is just you press a button and food shows up

  • NotebookLMUsessoftware · Google Labs

    we did a tear down of the notebook LM product just yesterday as part of our company

  • GongUsessoftware · Gong

    there's really powerful tools like we use gong uh for instance to like understand like what are maybe prospects

  • UltraspeakingUsescourse · Tristan de Montebello and Michael Gendler

    this course that I took all Ultra speaking the biggest piece of advice they have is speaking should be fun

  • A Short History of Nearly EverythingRecommendsbook · Bill Bryson

    I really love uh Short history of nearly everything by Bill brys

  • Designing Your LifeRecommendsbook · Bill Burnett and Dave Evans

    another one which is maybe like a little bit more in the like career lens is the this book called designing your life

  • IDEOMentionscompany

    this book called designing your life which is written by uh these actually um you know one of the founders of Ido

  • Stanford UniversityMentionsplace

    these these folks are uh Stanford professors in the design lab

  • Tour de France: UnchainedRecommendsdocumentary · Netflix

    I've been obsessed with the tour def France uh documentary on Netflix

  • NetflixUsescompany · Netflix

    I've been obsessed with the tour def France uh documentary on Netflix

  • Formula 1: Drive to SurviveMentionstv show · Netflix

    it's actually the same producers as drive to survive which I know people really got into as well around like F1

  • WebsimRecommendssoftware

    the product that really that I've been like using and having a ton of fun with is webson

  • RedditUseswebsite · Steve Huffman, Alexis Ohanian, and Aaron Swartz

    they literally just went on Reddit and they were just looking at the at the web Sim subreddit

  • Appeel notebookRecommendsproduct · Appeel

    it's called app heal uh a PP e Ling

  • I Read a Bunch of Self-Help Books So You Don't Have ToCoinedother · Aman Khan

    this essay that I wrote a while back with called you I read a bunch of self-help books you don't have to

  • GoogleMentionscompany · Google

    it's funny because my partner works at Google uh the future might look like

  • XMentionswebsite · X Corp.

    LinkedIn is probably the best way I'm I'm not Super Active on X

  • LinkedInUseswebsite · LinkedIn

    LinkedIn is probably the best way I'm I'm not Super Active on X

  • Lenny's PodcastCoinedpodcast · Lenny Rachitsky

    you can find all past episodes or learn more about the show at Lenny podcast.com

  • Apple PodcastsRecommendssoftware · Apple Inc.

    you can subscribe to the show on Apple podcast Spotify or your favorite podcast app

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster29:30

Stop Building Another ChatGPT on Your Own Data

After ChatGPT launched, Aman saw nearly every AI PM build the same thing: an internal chatbot on top of their knowledge base. He questions whether those products got real usage. The top 5% of AI PMs don't copy the familiar interface just because everyone's bought in; they ask whether it's actually the problem the business needs solved.

  • Post-ChatGPT, everyone gravitated to building an internal chatbot on their own data
  • Usage of those copycat chatbots is poorly understood and often low
  • Don't replicate a familiar interface just because it feels intuitive
  • The right AI interface often looks nothing like a chatbot (e.g. automating data analysis)

the first thing that everyone seemed to build was actually another chat PT but on top of their own data

Aman Khan · 30:30

don't do what everyone else is doing just because it feels intuitive to go and like replicate this interface that feels really great

Aman Khan · 31:30
#myth-buster#ai-products#chatbots#product-strategy

Hot Take· 2

Hot Take23:00

It's Easier to Break Into AI PM Now Than Before

Counterintuitively, Aman argues it's easier to break into AI product management today than a year ago. Before, you needed a deep machine learning background (training data, model types, infrastructure) to get a shot. Now the tools have lowered the bar, and YouTube videos show you how to build an app in under an hour.

  • Before, you needed deep ML knowledge on training data, model types and launching
  • Now an AI PM builds experiences around AI rather than the core models
  • Videos (he credits Greg Eisenberg) show building a product in ~45 minutes
  • He built a working newsletter signup page on his phone via Replit in five minutes

I actually think it's it's probably easier now to break into AI product management than it was before

Aman Khan · 23:00
#career#ai-pm#hot-take#accessibility
Hot Take25:30

Why PMs Are the Highest-Leverage Role in the Age of AI

Lenny raises the fear that AI tooling makes PMs obsolete, then flips it: the hardest part becomes knowing what to build, which is exactly the PM's job. Aman agrees strongly. AI tools max out a PM's influence, letting them communicate ideas to design, engineering, and leadership, making the AI PM the highest-leverage position at a company.

  • AI is great at building things; the hard part is knowing what to build
  • The AI PM is the representative of how to use AI tools at highest leverage
  • AI tools amplify a PM's core skill of influence across design, eng and leadership
  • PMs are the best-positioned function to thrive in an AI world

being an AI product manager feels like the highest leverage position you could be in at the company right now especially in the age of…

Aman Khan · 28:00
#hot-take#ai-pm#leverage#influence

Explainer· 2

Explainer06:30

The Three Flavors of AI Product Manager

Aman breaks the AI PM title into three distinct roles. AI platform PMs build tools for AI engineers (his own role at Arize). AI product PMs work where the core product is the model itself, like ChatGPT or NotebookLM. AI-powered PMs use AI across their existing workflow without building models from scratch. He predicts most PMs will soon be one flavor of AI PM.

  • AI platform PM: builds observability/eval tools for AI engineers
  • AI product PM: the core experience is enabled by the model under the hood (ChatGPT, NotebookLM)
  • AI-powered PM: uses AI to build better experiences without building the core model
  • AI will become as common as a database in SaaS apps

most PMS are probably going to be one flavor of an AI product manager either building tools for other PMS or other other companies that…

Aman Khan · 10:30
#ai-pm#product-management#career#roles
Explainer57:00

Waiting vs Wandering: The PM as the Wanderer

Aman frames a core tension: most companies and executives lean toward waiting to see how AI plays out, doing what has worked so far. But the PM's role is to be the Wanderer who pushes into the unknown to find signal and bring it back. His CPO reassured him that this squishy uncertainty is exactly what building zero to one feels like.

  • Companies default to waiting for the next model before building
  • The PM's role is to wander into the unknown and bring back signal
  • Engineering can feel settled with a roadmap while product feels lost
  • Zero-to-one feels uncertain, but you know it when you get there

to me I actually think the role of a PM is to be that Wanderer

Aman Khan · 59:00

that's what building zero to one really feels like it kind of feels like you're not sure what the right decision is

Aman Khan · 1:01:00
#explainer#ic-pm#zero-to-one#decision-making

Story· 3

Story40:00

The Betty Crocker Lesson: Don't Automate Everything Away

Aman tells the story of Betty Crocker's cake mix that only needed water, which tanked sales. When they changed the instructions so customers had to add an egg, sales skyrocketed. People want a little control over the experience. He calls this the IKEA effect applied to AI: leave knobs and levers so users still feel in control.

  • A one-ingredient (just add water) cake mix tanked Betty Crocker's sales
  • Requiring customers to add eggs made sales skyrocket
  • Even a fully self-driving car still lets you control the AC and music
  • The IKEA effect: people feel empowered when they impact the end experience

they realized that the customer actually wants to have a little bit bit of control over the experience

Aman Khan · 40:30

that's kind of like the Ikea effect applied to AI like people feel more empowered when they can when they actually have an impact on…

Aman Khan · 41:00
#story#ux#automation#ikea-effect
Story38:00

The On-Call Slackbot That Flopped at a Hackathon

At a recent hackathon, Aman's team built a Slackbot to read support-channel questions, classify them, and ping the right on-call person. They assumed it was a perfect, simple AI use case. It turned out to be very hard because so much context was missing, like who had handed off a problem to another team, illustrating that identifying the right problems is the real challenge.

  • The bot was meant to classify support questions and ping the right person
  • It failed because critical context (handoffs, project changes) was missing
  • 'Perfect use case for AI' assumptions often collapse on real context
  • Roughly 9 of 10 problems you try to solve with AI won't work well

one of them was this slackbot that actually alerted a person that was on call for a problem

Aman Khan · 38:30

it turns out uh that's actually really hard problem to do because there's so much context that is missing

Aman Khan · 39:00
#story#hackathons#failure#ai-limits
Story48:30

Energy Counts, and Sometimes You Become the Salesperson

Aman learned from his CEO that energy counts a lot when you're not sure which direction to go; showing up energized breaks down friction. Facing an existential call on whether to invest in LLMs, he became his own outbound salesperson, cold-messaging people with 'AI' in their LinkedIn titles. It both taught him how to message the product and showed the team he'd get into the details with them.

  • Bringing energy to a meeting removes friction when the path is unclear
  • Facing an LLM investment decision, he cold-messaged AI titles on LinkedIn himself
  • He maxed out his five LinkedIn messages a day and sat with the sales team
  • Getting into the weeds signals to the team you'll do the hard work with them

energy can count a lot when you're not sure which direction to go

Aman Khan · 49:30

I'm just going to be our like outbound salesperson and just start writing LinkedIn messages to people that have the title AI in their LinkedIn…

Aman Khan · 51:00
#story#ic-pm#energy#leadership

Tool· 1

Tool11:00

The AI Tools Aman Uses to Prototype as a PM

Aman shares the specific tools he uses to show up with functional prototypes instead of PRDs. Cursor and Replit for building working prototypes, Vercel's v0 for landing pages and UIs, and Midjourney or DALL-E for logos and visual user stories. The point is to arrive at a meeting with something clickable, even if it's wrong.

  • Cursor and Replit for building working prototypes with wired-up buttons
  • Vercel's v0 turns a prompt into a beautiful landing page or working UI
  • Midjourney and DALL-E for logos and visual user stories
  • Prototypes are not production-ready but are enough to tell a story

I'm obsessed with cursor I'm obsessed with repet these are tools I use all the time to just really build a prototype and try to…

Aman Khan · 11:00
#tools#prototyping#ai-pm#cursor#replit

Takeaway· 6

Takeaway20:30

Build a Portfolio to Stand Out as an AI PM Applicant

There's no college degree in AI product management, so Aman says you stand out by showing interest and building your own skill stack. A portfolio of products you've tried to build, even rough prototypes, answers the hiring manager's core questions before the interview: can this person do the job, are they excited, and do I like them.

  • There's no degree in product management, let alone AI product management
  • A portfolio of built products (even prototypes) shortcuts the hiring process
  • Hiring answers three things: can they do the job, are they excited, do I like them
  • Building things demonstrates how you think and what your interests are

the way that you stand out is through showing interest in the space and and building your own skill set and skilled stack around what…

Aman Khan · 20:30

you can actually go and have a you know almost like a portfolio of products that you've tried to build and maybe they're just prototypes…

Aman Khan · 20:30
#career#hiring#portfolio#ai-pm
Takeaway33:00

Describe the Problem, Not 'An AI Agent'

The AI agent hype is repeating the chatbot mistake. Aman warns that pitching 'we're building an AI agent' describes a solution, not a problem. Often it's better to let foundation model companies build the agentic layer, and make your job to weave that capability so seamlessly into your product that it doesn't even feel like AI.

  • Saying 'we're building an AI agent' describes a solution, not a problem
  • Ask whether it makes sense to build an agent or let foundation models build that layer
  • The PM's edge is applying the technology, not inventing the cutting-edge technique
  • Best AI integrations feel so seamless they don't feel like AI

you're not really describing the problem you're describing the solution an AI agent to do something

Aman Khan · 33:30

make that experience so amazingly seamless in your existing product that it doesn't even feel like AI in the first place

Aman Khan · 34:00
#ai-agents#product-strategy#takeaway#hype
Takeaway36:30

Every AI PM Needs a Metric: Count Your Shots

Aman notes that every PM is tasked with moving a business metric, but there isn't an obvious one for building AI prototypes since they don't immediately move revenue. His answer is to measure how many shots you're taking in the first place, tied to getting the whole organization hands-on through hackathons.

  • AI prototyping doesn't cleanly map to a revenue or business metric
  • The metric to track is how many shots (attempts) you're taking
  • Hackathons get the whole org hands-on and remove the fear that AI is unapproachable
  • Come to hackathons with problems to solve, not AI solutions to force-fit

every PM within a business is tasked with moving some business metric but there actually isn't one for building prototypes with AI

Aman Khan · 37:00

how many shots are you taking in the first place

Aman Khan · 37:00
#metrics#hackathons#ai-pm#takeaway
Takeaway43:00

Walk and Chew Gum: Deliver Value While Making Space to Fail

Borrowing a line from Kevin Yin, Aman says the AI PM job isn't to ship AI products, it's to solve customer problems. You're constantly pushed to move a business metric, but you must also carve out space to prototype, run hackathons, and tear down great AI products. None of those move a KPI directly, yet they're how you stay ahead.

  • The job is to solve customer problems, not to ship AI for its own sake
  • You must deliver customer value while creating space to iterate and fail
  • Arize does weekly livestreamed teardowns of cutting-edge AI products
  • Accept that the tech changes fast and some experiments will fail

your job is is not to go and ship AI products it's to go solve customers problems

Aman Khan · 43:00

you have to continuously deliver a customer value while also creating space to iterate and fail

Aman Khan · 44:30
#takeaway#ai-pm#prioritization#experimentation
Takeaway1:02:00

Amplify Signal Through the Noise With AI

A PM can't sit in a hundred customer meetings a week, so Aman scales himself with AI. He pulls call transcripts from Gong and feeds them into long-context LLMs to extract the most common problems and pain points, effectively being in many places at once. He treats this as a superpower for finding signal through the noise.

  • Uses Gong to capture what prospects and engineers are actually saying
  • Feeds transcripts into long-context LLMs to surface the most common problems
  • Can turn the output into a NotebookLM episode to consume it
  • Scaling yourself with AI lets you 'be in so many places at once'

we fed them into some of these large language models that have super long context Windows now and what you can do is actually pull…

Aman Khan · 1:02:30
#takeaway#customer-research#ai-tools#gong
Takeaway1:03:30

The Best Career Advice: Just Have Fun

Aman asked a serial-entrepreneur board member for advice and was surprised the answer was simply 'have fun.' He argues that staying curious, building for customers you care about, and having fun keeps you learning and iterating faster. Product feels heavy and serious, but reframing toward fun makes you go further.

  • A board member's core advice for an early-stage PM was just 'have fun'
  • Curiosity plus fun keeps you learning and iterating faster
  • Lenny keeps a 'have fun' Post-it in front of him while recording
  • Even scary, high-stakes work rarely is the end of the world, so reframe toward fun

I was so surprised by the feedback he had which was just have fun

Aman Khan · 1:03:30

if you're constantly learning and having fun you're going to iterate so much faster too

Aman Khan · 1:04:00
#takeaway#career#mindset#fun