LLenny's Podcast
← All episodes
Aman Khan (Arize AI, ex-Spotify, Apple, Cruise)14 November 2024

Becoming an AI PM

6Frameworks
15Insights

Frameworks in this episode

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster29:30

Stop Building Another ChatGPT on Your Own Data

After ChatGPT launched, Aman saw nearly every AI PM build the same thing: an internal chatbot on top of their knowledge base. He questions whether those products got real usage. The top 5% of AI PMs don't copy the familiar interface just because everyone's bought in; they ask whether it's actually the problem the business needs solved.

  • Post-ChatGPT, everyone gravitated to building an internal chatbot on their own data
  • Usage of those copycat chatbots is poorly understood and often low
  • Don't replicate a familiar interface just because it feels intuitive
  • The right AI interface often looks nothing like a chatbot (e.g. automating data analysis)

the first thing that everyone seemed to build was actually another chat PT but on top of their own data

Aman Khan · 30:30

don't do what everyone else is doing just because it feels intuitive to go and like replicate this interface that feels really great

Aman Khan · 31:30
#myth-buster#ai-products#chatbots#product-strategy

Hot Take· 2

Hot Take23:00

It's Easier to Break Into AI PM Now Than Before

Counterintuitively, Aman argues it's easier to break into AI product management today than a year ago. Before, you needed a deep machine learning background (training data, model types, infrastructure) to get a shot. Now the tools have lowered the bar, and YouTube videos show you how to build an app in under an hour.

  • Before, you needed deep ML knowledge on training data, model types and launching
  • Now an AI PM builds experiences around AI rather than the core models
  • Videos (he credits Greg Eisenberg) show building a product in ~45 minutes
  • He built a working newsletter signup page on his phone via Replit in five minutes

I actually think it's it's probably easier now to break into AI product management than it was before

Aman Khan · 23:00
#career#ai-pm#hot-take#accessibility
Hot Take25:30

Why PMs Are the Highest-Leverage Role in the Age of AI

Lenny raises the fear that AI tooling makes PMs obsolete, then flips it: the hardest part becomes knowing what to build, which is exactly the PM's job. Aman agrees strongly. AI tools max out a PM's influence, letting them communicate ideas to design, engineering, and leadership, making the AI PM the highest-leverage position at a company.

  • AI is great at building things; the hard part is knowing what to build
  • The AI PM is the representative of how to use AI tools at highest leverage
  • AI tools amplify a PM's core skill of influence across design, eng and leadership
  • PMs are the best-positioned function to thrive in an AI world

being an AI product manager feels like the highest leverage position you could be in at the company right now especially in the age of…

Aman Khan · 28:00
#hot-take#ai-pm#leverage#influence

Explainer· 2

Explainer06:30

The Three Flavors of AI Product Manager

Aman breaks the AI PM title into three distinct roles. AI platform PMs build tools for AI engineers (his own role at Arize). AI product PMs work where the core product is the model itself, like ChatGPT or NotebookLM. AI-powered PMs use AI across their existing workflow without building models from scratch. He predicts most PMs will soon be one flavor of AI PM.

  • AI platform PM: builds observability/eval tools for AI engineers
  • AI product PM: the core experience is enabled by the model under the hood (ChatGPT, NotebookLM)
  • AI-powered PM: uses AI to build better experiences without building the core model
  • AI will become as common as a database in SaaS apps

most PMS are probably going to be one flavor of an AI product manager either building tools for other PMS or other other companies that…

Aman Khan · 10:30
#ai-pm#product-management#career#roles
Explainer57:00

Waiting vs Wandering: The PM as the Wanderer

Aman frames a core tension: most companies and executives lean toward waiting to see how AI plays out, doing what has worked so far. But the PM's role is to be the Wanderer who pushes into the unknown to find signal and bring it back. His CPO reassured him that this squishy uncertainty is exactly what building zero to one feels like.

  • Companies default to waiting for the next model before building
  • The PM's role is to wander into the unknown and bring back signal
  • Engineering can feel settled with a roadmap while product feels lost
  • Zero-to-one feels uncertain, but you know it when you get there

to me I actually think the role of a PM is to be that Wanderer

Aman Khan · 59:00

that's what building zero to one really feels like it kind of feels like you're not sure what the right decision is

Aman Khan · 1:01:00
#explainer#ic-pm#zero-to-one#decision-making

Story· 3

Story40:00

The Betty Crocker Lesson: Don't Automate Everything Away

Aman tells the story of Betty Crocker's cake mix that only needed water, which tanked sales. When they changed the instructions so customers had to add an egg, sales skyrocketed. People want a little control over the experience. He calls this the IKEA effect applied to AI: leave knobs and levers so users still feel in control.

  • A one-ingredient (just add water) cake mix tanked Betty Crocker's sales
  • Requiring customers to add eggs made sales skyrocket
  • Even a fully self-driving car still lets you control the AC and music
  • The IKEA effect: people feel empowered when they impact the end experience

they realized that the customer actually wants to have a little bit bit of control over the experience

Aman Khan · 40:30

that's kind of like the Ikea effect applied to AI like people feel more empowered when they can when they actually have an impact on…

Aman Khan · 41:00
#story#ux#automation#ikea-effect
Story38:00

The On-Call Slackbot That Flopped at a Hackathon

At a recent hackathon, Aman's team built a Slackbot to read support-channel questions, classify them, and ping the right on-call person. They assumed it was a perfect, simple AI use case. It turned out to be very hard because so much context was missing, like who had handed off a problem to another team, illustrating that identifying the right problems is the real challenge.

  • The bot was meant to classify support questions and ping the right person
  • It failed because critical context (handoffs, project changes) was missing
  • 'Perfect use case for AI' assumptions often collapse on real context
  • Roughly 9 of 10 problems you try to solve with AI won't work well

one of them was this slackbot that actually alerted a person that was on call for a problem

Aman Khan · 38:30

it turns out uh that's actually really hard problem to do because there's so much context that is missing

Aman Khan · 39:00
#story#hackathons#failure#ai-limits
Story48:30

Energy Counts, and Sometimes You Become the Salesperson

Aman learned from his CEO that energy counts a lot when you're not sure which direction to go; showing up energized breaks down friction. Facing an existential call on whether to invest in LLMs, he became his own outbound salesperson, cold-messaging people with 'AI' in their LinkedIn titles. It both taught him how to message the product and showed the team he'd get into the details with them.

  • Bringing energy to a meeting removes friction when the path is unclear
  • Facing an LLM investment decision, he cold-messaged AI titles on LinkedIn himself
  • He maxed out his five LinkedIn messages a day and sat with the sales team
  • Getting into the weeds signals to the team you'll do the hard work with them

energy can count a lot when you're not sure which direction to go

Aman Khan · 49:30

I'm just going to be our like outbound salesperson and just start writing LinkedIn messages to people that have the title AI in their LinkedIn…

Aman Khan · 51:00
#story#ic-pm#energy#leadership

Tool· 1

Tool11:00

The AI Tools Aman Uses to Prototype as a PM

Aman shares the specific tools he uses to show up with functional prototypes instead of PRDs. Cursor and Replit for building working prototypes, Vercel's v0 for landing pages and UIs, and Midjourney or DALL-E for logos and visual user stories. The point is to arrive at a meeting with something clickable, even if it's wrong.

  • Cursor and Replit for building working prototypes with wired-up buttons
  • Vercel's v0 turns a prompt into a beautiful landing page or working UI
  • Midjourney and DALL-E for logos and visual user stories
  • Prototypes are not production-ready but are enough to tell a story

I'm obsessed with cursor I'm obsessed with repet these are tools I use all the time to just really build a prototype and try to…

Aman Khan · 11:00
#tools#prototyping#ai-pm#cursor#replit

Takeaway· 6

Takeaway20:30

Build a Portfolio to Stand Out as an AI PM Applicant

There's no college degree in AI product management, so Aman says you stand out by showing interest and building your own skill stack. A portfolio of products you've tried to build, even rough prototypes, answers the hiring manager's core questions before the interview: can this person do the job, are they excited, and do I like them.

  • There's no degree in product management, let alone AI product management
  • A portfolio of built products (even prototypes) shortcuts the hiring process
  • Hiring answers three things: can they do the job, are they excited, do I like them
  • Building things demonstrates how you think and what your interests are

the way that you stand out is through showing interest in the space and and building your own skill set and skilled stack around what…

Aman Khan · 20:30

you can actually go and have a you know almost like a portfolio of products that you've tried to build and maybe they're just prototypes…

Aman Khan · 20:30
#career#hiring#portfolio#ai-pm
Takeaway33:00

Describe the Problem, Not 'An AI Agent'

The AI agent hype is repeating the chatbot mistake. Aman warns that pitching 'we're building an AI agent' describes a solution, not a problem. Often it's better to let foundation model companies build the agentic layer, and make your job to weave that capability so seamlessly into your product that it doesn't even feel like AI.

  • Saying 'we're building an AI agent' describes a solution, not a problem
  • Ask whether it makes sense to build an agent or let foundation models build that layer
  • The PM's edge is applying the technology, not inventing the cutting-edge technique
  • Best AI integrations feel so seamless they don't feel like AI

you're not really describing the problem you're describing the solution an AI agent to do something

Aman Khan · 33:30

make that experience so amazingly seamless in your existing product that it doesn't even feel like AI in the first place

Aman Khan · 34:00
#ai-agents#product-strategy#takeaway#hype
Takeaway36:30

Every AI PM Needs a Metric: Count Your Shots

Aman notes that every PM is tasked with moving a business metric, but there isn't an obvious one for building AI prototypes since they don't immediately move revenue. His answer is to measure how many shots you're taking in the first place, tied to getting the whole organization hands-on through hackathons.

  • AI prototyping doesn't cleanly map to a revenue or business metric
  • The metric to track is how many shots (attempts) you're taking
  • Hackathons get the whole org hands-on and remove the fear that AI is unapproachable
  • Come to hackathons with problems to solve, not AI solutions to force-fit

every PM within a business is tasked with moving some business metric but there actually isn't one for building prototypes with AI

Aman Khan · 37:00

how many shots are you taking in the first place

Aman Khan · 37:00
#metrics#hackathons#ai-pm#takeaway
Takeaway43:00

Walk and Chew Gum: Deliver Value While Making Space to Fail

Borrowing a line from Kevin Yin, Aman says the AI PM job isn't to ship AI products, it's to solve customer problems. You're constantly pushed to move a business metric, but you must also carve out space to prototype, run hackathons, and tear down great AI products. None of those move a KPI directly, yet they're how you stay ahead.

  • The job is to solve customer problems, not to ship AI for its own sake
  • You must deliver customer value while creating space to iterate and fail
  • Arize does weekly livestreamed teardowns of cutting-edge AI products
  • Accept that the tech changes fast and some experiments will fail

your job is is not to go and ship AI products it's to go solve customers problems

Aman Khan · 43:00

you have to continuously deliver a customer value while also creating space to iterate and fail

Aman Khan · 44:30
#takeaway#ai-pm#prioritization#experimentation
Takeaway1:02:00

Amplify Signal Through the Noise With AI

A PM can't sit in a hundred customer meetings a week, so Aman scales himself with AI. He pulls call transcripts from Gong and feeds them into long-context LLMs to extract the most common problems and pain points, effectively being in many places at once. He treats this as a superpower for finding signal through the noise.

  • Uses Gong to capture what prospects and engineers are actually saying
  • Feeds transcripts into long-context LLMs to surface the most common problems
  • Can turn the output into a NotebookLM episode to consume it
  • Scaling yourself with AI lets you 'be in so many places at once'

we fed them into some of these large language models that have super long context Windows now and what you can do is actually pull…

Aman Khan · 1:02:30
#takeaway#customer-research#ai-tools#gong
Takeaway1:03:30

The Best Career Advice: Just Have Fun

Aman asked a serial-entrepreneur board member for advice and was surprised the answer was simply 'have fun.' He argues that staying curious, building for customers you care about, and having fun keeps you learning and iterating faster. Product feels heavy and serious, but reframing toward fun makes you go further.

  • A board member's core advice for an early-stage PM was just 'have fun'
  • Curiosity plus fun keeps you learning and iterating faster
  • Lenny keeps a 'have fun' Post-it in front of him while recording
  • Even scary, high-stakes work rarely is the end of the world, so reframe toward fun

I was so surprised by the feedback he had which was just have fun

Aman Khan · 1:03:30

if you're constantly learning and having fun you're going to iterate so much faster too

Aman Khan · 1:04:00
#takeaway#career#mindset#fun