LLenny's Podcast
← All episodes
13 December 2023

Failure

8Frameworks
12Insights

Frameworks in this episode

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Explainer· 1

Explainer33:00

Shri Bhachu: How to Fail Conclusively in Growth Experiments

Shri Bhachu (former head of growth at Ramp) explains that with growth experiments succeeding only ~30% of the time, failure only counts if you learn, and you can only learn if you designed the test to fail conclusively. Because B2B rarely offers large sample sizes, he recommends maximizing the treatment effect: throw every tactic at a hypothesis so a failure genuinely rules it out.

  • Only about 30% of growth experiments succeed, so a culture that isn't afraid to fail is essential.
  • Failure isn't lack of revenue; failure is not learning, which requires a conclusively designed test.
  • Inconclusive tests get re-run for years because each new executive retries the same untested idea.
  • Experiments succeed via a large sample size or a strong treatment effect; B2B lacks the large-n luxury.
  • Maximize the treatment effect (throw all tactics and resources at the hypothesis) so a failure lets you stop trying it, then optimize down later if it works.

for me failure is not that you didn't drive Revenue failure is not learning

Shri Bhachu · 33:30

just maximize the treatment effect and if with all of that it didn't work then you can say hey like we're not going to try…

Shri Bhachu · 35:00
#growth#experimentation#b2b#testing

Story· 8

Story03:30

Katie Dill: When the Airbnb Design Team Rebelled Against Her

Katie Dill (head of design at Stripe, formerly Airbnb and Lyft) recounts how, about a month into leading Airbnb's 10-person design org, half the team plus HR gathered to read her a list of everything she was doing wrong. The core lesson was that she hadn't earned their trust before pushing change. She shifted from 'coming in swinging' to listening first, and a couple months later the team had the best engagement scores in the company.

  • She joined 'ready to go' and started driving change before building trust with the existing team.
  • Five team members and an HR partner took turns reading criticisms of her aloud from prepared papers.
  • The unifying theme was that she hadn't earned their trust, so nobody believed in what she was building.
  • You can inflict change on people, but doing it with them requires trust.
  • After shifting to a listen-first approach, the team hit the best engagement scores in the company.

the theme that was across all of that is that I hadn't earned their trust

Katie Dill · 06:00

instead of coming in swinging you know come in listening so that you can really you know set out to make change

Katie Dill · 07:30
#leadership#design#trust#management
Story08:00

Paul Adams: Freezing on Stage at Cannes in Front of Thousands

Paul Adams (CPO at Intercom) describes representing Facebook on the keynote stage at Cannes, the world's biggest advertising festival, and having a panic attack three or four minutes in. He walked off stage still miked up, cursed, then recovered and finished the talk. His takeaway is that even living out the worst public-speaking nightmare turns out to not be a big deal.

  • It was the first time he'd ever rehearsed a talk word-for-word, media-trained not to say the wrong thing.
  • He froze, couldn't remember his lines, had a panic attack, and walked off stage.
  • His mic was still on, so the audience heard him curse, then he walked back out and finished well.
  • He was 'famous and infamous' in Cannes that night for the performance.
  • The lesson: adapt to curveballs; none of these things are as big a deal as they feel in the moment.

I just froze I I couldn't remember what I was supposed to say

Paul Adams · 09:00

none of these things are that big a deal at the end of the day you know you kind of move on live and learn

Paul Adams · 10:30
#public-speaking#resilience#failure
Story11:30

Paul Adams: Why Google's Social Products (Buzz, Plus) Failed

Paul Adams reflects on working on failed Google social projects like Google Buzz and Google+, arguing they came from a place of competitive fear rather than genuine user need. His own research mapping dozens of people's social networks revealed people needed better ways to talk to small groups of family and friends, an insight obvious in hindsight (WhatsApp, iMessage) but missed at the time.

  • Google Buzz and Google+ were motivated by competitive fear, not by understanding user problems.
  • His formative research with dozens of people over ~18 months surfaced the same pattern every time: people need better small-group communication.
  • Google+ secrecy was extreme: a separate building with a different key card for people not on the project.
  • Leaving mid-project to join Facebook got him quarantined and his laptop forensically analyzed as a suspected spy.
  • Projects run from a place of competitive fear don't lead to good products.

I think a lot of the motivation for those projects came from a place of fear

Paul Adams · 11:30

the project has been run from a place of fear competitive fear which I don't think leads to good things

Paul Adams · 15:30
#product#google#strategy#fear
Story19:30

Tom Conrad: How Too Much Money Sank Pets.com

Tom Conrad (CPO at Quibi, engineering leader at Pets.com) shares that three overfunded pet e-commerce sites each raised ~$50M and, treating it as a zero-sum game, spiraled into an unwinnable advertising arms race. He notes an excess of investment can become its own liability, and that timing mattered enormously (Chewy later succeeded when the same model was viable).

  • Three pet e-commerce sites each raised roughly $50M and outspent each other on national TV advertising.
  • An excess of investment can be its own albatross, pushing teams toward unwise decisions.
  • The 'shipping dog food is a stupid business' critique was wrong; the problem was 80% of the US was still on dialup.
  • Chewy later executed the same model brick-by-brick and became a multi-billion-dollar company.
  • Pets.com didn't go bankrupt: the team wound down early and returned the remaining cash to investors, which no public company had done before.

it became this kind of unwinable arms race

Tom Conrad · 20:30

we actually didn't go bankrupt we shut the company down and returned the the remaining balance to the investors

Tom Conrad · 22:30
#startups#funding#e-commerce#timing
Story23:00

Tom Conrad: Quibi's Math Was Broken Before COVID Hit

Tom Conrad argues Quibi failed not on execution but on a fundamentally broken equation: you can't build a bespoke content library big enough to drive subscriptions and retention for only a couple billion dollars. COVID compounded it, forcing the premium daily content meant to differentiate Quibi from YouTube to be shot in hosts' garages, exactly like YouTube.

  • Companies are like a math problem turning investment into returns; a broken equation can't be fixed by iteration or execution.
  • Quibi made 70 shows in 18 months, more than all major broadcast networks combined make in a year.
  • About a third to half of content spend went to daily content designed to be shot day-of in professional studios.
  • Launching two weeks into COVID forced that content into hosts' garages, making it look exactly like YouTube.
  • The real bet needed six to ten billion dollars, not two; that risk-reward was more than anyone could stomach.

no amount of like iteration and execution can like kind of get you out of the the failed outputs of the broken equation

Tom Conrad · 30:00

we launched two weeks into covid and we couldn't make any of that content except literally in the garages of the host homes

Tom Conrad · 31:30
#startups#product#media#unit-economics
Story40:30

Jiaona Zhang: Why Airbnb Plus Was Doomed by Unit Economics

Jiaona Zhang tells the untold story of Airbnb Plus, where the team went solution-first (inspecting and managing inventory) out of competitor fear, rather than solving the real problem: people not knowing what they were getting into. She argues the unit economics never would have worked, and cheaper, on-brand fixes like reviews, lockboxes, or cleaner partnerships targeted the actual problems better than inspectors.

  • Airbnb Plus was solution-first and competitor-afraid, reacting to managed marketplaces.
  • The real problem was that people want to know what they're getting into and homes needed better representation.
  • Airbnb's strategic weakness was operations; building inspection from the ground up wasn't in its wheelhouse.
  • 'Magical thinking' about unit economics assumes it all works out at scale instead of validating it early.
  • Cheaper, on-platform solutions (guest reviews, lockboxes, cleaner partnerships) beat sending physical inspectors.

we essentially were like let's go inspect our inventory let's actually try to manage our inventory more

Jiaona Zhang · 41:00

I personally don't believe the unit economics ever would have really worked out

Jiaona Zhang · 44:00
#product#airbnb#unit-economics#strategy
Story52:30

Gina Gotthilf: The Duolingo Badges Miss and Forgetting to Dogfood

Gina Gotthilf recounts delaying a Duolingo badges experiment for six months on ROI grounds, then running a lean version that failed because no one is proud of just signing up. The deeper embarrassment was that the growth team wasn't dogfooding; had they tested it themselves they'd have known the badge was lame. Once done right, badges impacted almost every company metric.

  • She blocked the badges experiment for ~six months to focus on lower-hanging fruit, a mistake on her end.
  • The lean MVP awarded a badge just for signing up, which produced no results because signup isn't an exciting, shareable moment.
  • The growth team hadn't been dogfooding, so they never caught how weak the badge was.
  • Dogfooding became a permanent team practice after this.
  • Done properly, badges lifted nearly all company metrics and unlocked a treasure trove of engagement asks.

we hadn't been dog fooding in the growth team

Gina Gotthilf · 54:30

why are we not testing our experiments

Gina Gotthilf · 55:00
#growth#duolingo#experimentation#dogfooding
Story58:00

Maggie Crowley: 'What's the Worst Product You Ever Shipped?'

Maggie Crowley (VP of product at Toast) shares her favorite interview question, what's the worst product you ever shipped, arguing a PM who can't name a truly bad one hasn't done enough or can't admit it. She then tells her own worst project: a side-by-side rewrite estimated at six months that took two and a half years and never reached parity, with the flat lesson to never do a rewrite.

  • She asks every PM candidate what's the worst product they ever shipped, and admitting it is as important as having done it.
  • Her worst project was a rewrite of an existing product estimated at six months that ran two and a half years without reaching feature parity.
  • Root causes: arrogance, skipping discovery, no one-pager, and insufficient technical and design research into requirements.
  • The project didn't change the business trajectory, but she didn't get fired.
  • Her blunt advice: don't do a side-by-side rewrite; once teams build in the new world and the launch goes negative, you're stuck clawing back to neutral.

a question I ask in every product interview is uh what's the worst product you ever shipped

Maggie Crowley · 58:00

don't rewrite if anyone ever tells you to do a rewrite don't do it a side bys side rewrite nope

Maggie Crowley · 1:00:00
#product#hiring#rewrites#engineering

Takeaway· 3

Takeaway16:30

Paul Adams: Big Bets, Shipping Fast, and Designing the Company

Paul Adams explains why he joined Intercom, drawn by co-founder Owen's pitch that at Facebook you design the product but at Intercom you design the company. He built a culture around big bets and shipping fast to learn, while acknowledging the constant tension between that and a design-founder team's high craft standards.

  • Owen's pitch: at Facebook you design the product, at Intercom you design the company.
  • He favors big bets (higher risk, higher reward) over incremental improvements.
  • Intercom's principle evolved from 'ship to learn' to 'ship fast, ship early, ship often'.
  • Shipping early and fast lets you learn fast, change fast, and improve fast.
  • There's a real tension between shipping fast and a craft-driven team's fear of being embarrassed by what they ship.

at Facebook you can you can design the product but at intercom you can design the company

Paul Adams · 16:30

if you ship early and fast and learn fast you can change fast and you can improve fast

Paul Adams · 17:30
#product#culture#shipping#big-bets
Takeaway39:00

Jiaona Zhang: The Mistake of Jumping to Solutions

Jiaona Zhang (JZ), a product leader who was at Airbnb and Webflow, says the hardest habit to untrain in new PMs is getting attached to a solution they can already picture building. Her fix is to force focus onto users and their real-world problems first, before deciding whether an opportunity even exists.

  • Every human tends to jump to solutions and get attached to something they can see in their head.
  • New PMs arrive wanting to build a specific X thing rather than starting from a problem.
  • The retraining is to not think at all about what you want to build and instead focus on users and their problems.
  • Understand the problem first, then assess whether there's a real opportunity.

get really attached to a solution a way of implementing something something that you you can see in your head that you want to build

Jiaona Zhang · 39:30

we are not going to think at all about the thing that you want to build but instead we're going to be focused on users

Jiaona Zhang · 39:30
#product#pm#user-research
Takeaway44:30

Gina Gotthilf: Everyone Has a B-Side to Their Career

Gina Gotthilf (early growth leader at Duolingo, now COO of Latitude) argues we're trained to broadcast our impressive 'A-sides' while burying the 'B-sides', creating an Instagram-able version of careers. Her own path was full of rejections, lost visas, layoffs, and being fired repeatedly before things took off, and she stresses that careers are long and there's far more time than people think.

  • We're encouraged to talk only about our A-sides, so podcasts and stages show the Instagram-able version.
  • Her B-side: dropped out of Reed College while depressed, applied to ~100 companies, lost her visa twice, and was fired and laid off multiple times.
  • Most people have B-moments every day, week, and period of life; not recognizing them as moments makes them feel permanent.
  • Careers are long and people underestimate how much time and opportunity is ahead of them.
  • Success is partly about telling a story and understanding what others perceive as valuable, not just doing things that matter.

we are very encouraged in our lives ESP especially professionally to talk about our as sides all the time

Gina Gotthilf · 44:30

a lot of what you hear in podcasts and on on stage ends up being the instagramable version of someone

Gina Gotthilf · 45:00
#career#resilience#storytelling#mindset