LLenny's Podcast
← All episodes
Laura Schaffer (VP of Growth at Amplitude)09 March 2023

Career frameworks, A/B testing mistakes, counterintuitive onboarding tips, selling to developers

7Frameworks
15Insights

Frameworks in this episode

Leadership5 steps

Customer-Insight Career Leverage

Grow your career by pulling ground-level customer insights up to executives instead of waiting to be promoted

Read →
Innovation4 steps

Fail-Fast Iterative Validation

Since ~80% of hypotheses fail, use cheap validation methods first and reserve A/B testing for pre-vetted ideas

Read →
Marketing3 steps

Good Friction vs Bad Friction

Adding friction that addresses the user's psyche can increase conversion, because not all friction is bad

Read →
Strategy4 steps

Lower-Confidence, Higher-Volume Experimentation

Drop the 95% confidence bar to run more experiments, backstopped by qualitative corroboration and a pre-set game plan

Read →
Sales4 steps

Selling to Developers via Self-Serve Proof

Developers must prove products themselves, so invest in self-serve — sales can't convert them at the build stage

Read →
Marketing5 steps

The Pill-in-the-Hot-Dog Onboarding

Bury the intimidating-but-necessary onboarding step inside familiar, exciting content instead of leading with it

Read →
Strategy4 steps

Year-Horizon Growth Portfolio

Protect growth teams from weekly-win pressure by committing to low/medium/high bets over a year-long horizon

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 2

Hot Take42:30

Drop the 95% Confidence Interval to Run More Experiments

Laura argues the 95% confidence standard comes from high-stakes fields like academia and pharma where false positives are dangerous. Teams converting or upselling users don't carry that burden, so accepting a lower confidence interval to double the number of experiments run in a year nets out positive. She stresses the game plan must be set before running, and lower confidence should be paired with harder qualitative corroboration.

  • 95% confidence belongs to fields where a false success is genuinely dangerous
  • Growth teams converting users can afford more risk to run more experiments
  • Lowering the confidence interval can double annual experiment volume
  • Decide the acceptance criteria before running, and corroborate with qualitative data

“the 95 confidence rate belongs in some companies in some Industries because the risk of of you know the impact of a false success is…”

Laura Schaffer · 43:30

“there are definitely times where I will advocate for and I will push for an owl myself used lower lower confidence intervals and 95 especially…”

Laura Schaffer · 44:00
#experimentation#statistics#confidence interval#growth
Hot Take58:30

Shifting to PLG: Start With the Customer, Don't Chop Up Your Sales Product

Laura says growth and sales are two sides of the same coin, one selling with the product and one selling person-to-person, and companies need both. The mistake sales-led companies make when adding PLG is treating it as copy-paste: taking the enterprise offering, cutting features, and slapping a price on it. Instead you must reset and start with the unique problems of the self-serve customer, then orient the product experience around them.

  • Growth = selling with your product; sales = selling one-to-one; companies need both
  • The PLG mistake is chopping the enterprise product for parts and pricing it
  • Self-serve users have different problems than sales-led prospects
  • Start strategy with the customer and their psyche, not the product

“the difference is that with growth you are selling with your product and with sales you're selling with person like one to one”

Laura Schaffer · 59:00

“do not think this is a copy paste but like chop it for parts thing don't start with your product when you're building out your…”

Laura Schaffer · 1:03:30
#product-led growth#plg vs slg#strategy#amplitude

Explainer· 2

Explainer1:05:30

Why Developers Are a Completely Different Audience to Sell To

Laura explains that people who succeed selling to other audiences often struggle with developers because they're so different. Roughly 2-to-1, developers skip your marketing website entirely and go straight to signup, so any context you assume they absorbed is lost. They're like Ikea buyers who tear open the bags and start building rather than reading the manual, and they have an intense aversion to talking to sales.

  • Almost 2-to-1, developers skip the marketing website and go straight to signup
  • Assumptions that they read pricing or product context don't apply
  • They behave like Ikea buyers: build first, seek instructions only when stuck
  • Strong aversion to sales - a Fang company ran a POC to production without ever engaging sales; another team used personal emails to avoid it

“one is developers like almost 2-1 do not look at your marketing website at all they go straight to your sign up flow”

Laura Schaffer · 1:06:30

“they're the the Ikea buyers who you know when like a package comes they're not opening up the instruction manual and reading it”

Laura Schaffer · 1:07:00
#developers#selling to developers#self-serve#audience
Explainer1:08:30

Developers Must Prove It Themselves Because the Stakes Are Theirs

Laura explains the root reason developers behave differently: they're responsible when things break. If a service goes down, doesn't integrate, or wakes the pager, it's on the developer, potentially costing them their job, team trust, and reputation. Because the stakes are so high, they can't take a sales rep's word and must build a proof of concept, which means companies requiring developers to build must invest heavily in self-serve.

  • Developers own the outcome when a bought tool fails - it's their responsibility, not the vendor's
  • That failure can cost them their job, reputation, and their team's trust
  • High stakes mean they can't trust content or a sales rep's word; they must prove it
  • Companies requiring developers to build must invest in self-serve to convert them

“if your service goes down that's their responsibility not just for themselves but their team”

Laura Schaffer · 1:09:00

“they can't afford to trust your content or someone's word they must do it they must prove it themselves”

Laura Schaffer · 1:10:00
#developers#self-serve#proof of concept#trust

Story· 4

Story12:30

How a Self-Made 'Voice of the Customer' Report Built Twilio's Growth Team

Joining Twilio in product marketing with no growth team, Laura heard customers struggling to get started, which directly contradicted the exec team's high conviction that Twilio was famously easy. Rather than pitch immediately, she just started writing up and sharing her customer insights as a digest, which grew into a quarterly Voice of the Customer session that even CEO Jeff Lawson attended. That built trust let her successfully pitch and create Twilio's growth engineering and product team.

  • Customers said Twilio was hard to get started with, contradicting the company's core 'developers love us, we're so easy' conviction
  • She unilaterally started a shared customer-insights digest that senior leaders asked to be added to
  • It grew into a quarterly Voice of the Customer session attended by CEO Jeff Lawson
  • The accumulated trust made her 2015 pitch for a growth team almost pre-approved

“I just started sharing during a voice the customer report I started sharing my insights started writing down and just sharing them and it became…”

Laura Schaffer · 14:00
#twilio#growth team#customer research#org building
Story20:30

Adding Questions to Twilio Signup Raised Conversion by 5%

Twilio had strong conviction that adding anything to the developer signup flow would hurt conversion, so signup asked only for username, email and password. Laura quietly ran a Friday-night test adding a few questions (coding language, use case, product, developer-or-not) with no personalization. Conversion went up about 5%, because the questions reassured nervous first-time users that they were in the right place.

  • Existing belief: any friction in developer signup would be disastrous for conversion
  • She added ~4 drop-down questions (language, use case, product, developer vs not)
  • Signups improved ~5% with no personalization, just the questions
  • The data doubled as insight for docs, growth and product teams on who was signing up

“it improved conversion by like five percent like just improve signups”

Laura Schaffer · 23:30
#experimentation#onboarding#conversion#twilio
Story29:30

Hiding the Scary Onboarding Step: The 'Pill in the Hot Dog' Test

A prescriptive step-by-step onboarding underperformed because step one, getting a Telecom phone number, was psychologically intimidating to developers unfamiliar with telecom. Laura's team ran an MVP that kicked users out of the portal into a docs page leading with code, then embedded the phone-number step midway rather than up front. Burying the scary step, like hiding a pill in a hot dog, converted better.

  • Telecom/phone numbers were outside developers' comfort zone and triggered anxiety
  • Making 'get a phone number' step one hurt conversion despite being logical
  • The test led with familiar code and embedded the scary step later, with no easy return button
  • Burying the unpleasant step converted better by respecting user psyche

“you can't just feed the pill to the animal it's never gonna happen but if you shove it inside of a hot dog which like…”

Laura Schaffer · 32:30

“if something like very logical isn't converting well sometimes it means that you're battling against the psyche of a user”

Laura Schaffer · 33:30
#onboarding#developer experience#experimentation#twilio
Story54:30

Twilio's 'Quick Deploy' Let Non-Developers Build, Driving Tens of Millions

Noticing a growing group of non-developers signing up who genuinely wanted to build with Twilio, Laura's team explored what they needed to succeed. Since they couldn't write code from scratch or stand up a server, the team built Quick Deploy on Code Exchange, effectively a create-your-own-demo experience that let anyone deploy an app without writing code. It gave non-engineering buyers momentum and sales a powerful hook, generating tens of millions in pipeline.

  • A growing non-developer segment wanted to build with Twilio but got stuck in developer onboarding
  • Sales insight: if a non-dev can get from zero to one, spend can scale from $1 to $100k
  • Quick Deploy on Code Exchange let people deploy an app without writing code
  • It functioned as a create-your-own-demo, giving non-engineering buyers an aha moment

“there's one thing that we did that generated like tens of millions of dollars in Pipelines really really powerful”

Laura Schaffer · 54:00

“it's called a quick Deploy on code exchange um anyone can go there and deploy an app without having to write code and kind of…”

Laura Schaffer · 57:00
#product-led growth#twilio#feature story#activation

Tool· 2

Tool1:13:30

Interview Question: Ask About a Ship They Didn't Cherry-Pick

Laura's favorite interview question is to ask about a release or ship the candidate didn't hand-pick, for example their most recent one. Everyone has a polished success story that reveals little, but a random ship probably wasn't fantastic, so the candidate talks more about how they approached it. That surfaces their frameworks and process rather than a rehearsed outcome.

  • Don't ask for a great ship - everyone has a cherry-picked success story
  • Ask about a non-cherry-picked ship, like the most recent one, to constrain the answer
  • A random ship likely wasn't great, so they explain their approach instead of the outcome
  • This reveals the candidate's frameworks and how they think

“I love asking about a ship or release that is not cherry-picked by the person you're talking to”

Laura Schaffer · 1:13:30
#hiring#interview question#frameworks#management
Tool1:14:30

SaaS Tools Laura Relies On for Growth

Laura shares the SaaS products she keeps coming back to. Hotjar and Qualaroo let you put quick qualitative-feedback prompts in front of customers, a critical supplement to quantitative data. She also names Slack, and Builder.io as a drag-and-drop headless CMS that lets non-engineers make changes without relying on developers, plus ChatGPT.

  • Hotjar and Qualaroo for quick qualitative feedback from customers
  • Qualitative feedback is a critical supplement to quantitative data
  • Builder.io is a drag-and-drop headless CMS letting non-engineers ship changes
  • Also Slack and ChatGPT, which she says teams need to figure out how to adopt

“one is hot jar um hot jar qualaru also work just anything that allows you to put some quick little thing in front of customers…”

Laura Schaffer · 1:14:30
#tools#hotjar#builder.io#qualitative feedback

Takeaway· 5

Takeaway06:30

Carve Your Own Career Path by Owning Customer Insights

Laura argues the common way to grow a career, working hard in your role and hoping your manager advocates for you, caps your trajectory at your manager's ability and your role's ceiling. Instead she takes that power back by staying close to customers, because executives lose access to customer insights as a company grows. Surfacing those insights makes you valuable to the most senior leaders and builds a personal brand independent of any one manager.

  • The default career-growth method limits you to your manager's ability to advocate and your role's explicit trajectory
  • As companies grow, executives get disconnected from customers and lose access to the best insights
  • Pulling in customer insights and bringing them to life is a superpower valuable at every level
  • Becoming known as the person who 'just knows our customers' builds brand and opens cross-team opportunities

“you're limited to what your manager's ability is to advocate for you to promote you”

Laura Schaffer · 06:30

“your superpower is in really pulling those insights in and bringing them to life staying close to the customer”

Laura Schaffer · 09:30
#career growth#customer insights#personal brand#influence
Takeaway25:00

Bad Friction Is Bad, Good Friction Is Good

The surprising signup result taught Laura that not all friction is bad. New users signing up for the first time are scared and expecting difficulty, so questions that align with what they're already worried about (does it support my language, can I do my use case) are comforting rather than obstructive. The core lesson is that the psyche of the user is as important as understanding your product or market.

  • First-time users arrive anxious, 'looking for the bogeyman' and expecting friction
  • Questions that address their organic worries feel reassuring, not obstructive
  • The assumption that 'all friction is bad' is wrong
  • User psyche matters as much as product and market understanding

“the psyche of the user is so so critical right”

Laura Schaffer · 25:00

“the learning here was bad friction is bad and good friction is good right there's no such thing as being simple as just all friction…”

Laura Schaffer · 25:30
#onboarding#user psychology#conversion#friction
Takeaway34:00

Iterate Instead of Redesigning: 80% of Bets Are Wrong

Laura cites research from companies like Netflix and Microsoft showing 80-90% of hypotheses fail. Because building for months in an attic makes it likely you ship the wrong 80%, being iterative lets you catch failure sooner and use it as a compass. She tells teams that if the first version isn't embarrassing, they've gone too far.

  • Research shows 80-90%+ of product hypotheses fail
  • Big multi-month redesigns maximize the chance of shipping the wrong thing
  • Iteration turns failure into a compass instead of a wall
  • 'If it's not embarrassing you've gone too far' - ship ugly, early versions

“failure doesn't have to be a wall it can be a compass right it can be the thing that leads you to the right thing”

Laura Schaffer · 35:00

“if it's not embarrassing you've gone too far right gotta be embarrassing”

Laura Schaffer · 35:30
#experimentation#iteration#product strategy#failure
Takeaway38:30

Fail Fast With Painted Doors and Mocks, Not Expensive A/B Tests

Laura says you can't raise the base hit rate of untested ideas, but you can validate them far faster and cheaper. A/B testing is one of the most expensive validation methods, needing design, engineering and time. Painted doors and mocks can invalidate many hypotheses before you build, so only vetted ideas reach a full A/B test.

  • You can't beat the ~80% fail rate, but you can validate ideas faster
  • A/B tests are expensive: design, engineering, PM time, plus run time
  • Painted doors test a concept before it exists; mocks test the experience
  • Reserve deep A/B testing for hypotheses already vetted by cheaper methods

“a b testing is one of the most expensive kinds of ways to validate an experiment right”

Laura Schaffer · 39:00

“you can do that with painted doors right which is where you test rate the concept and the idea before it exists versus the actual…”

Laura Schaffer · 40:00
#experimentation#validation#painted door#product strategy
Takeaway50:00

Don't Massage Data to Fit, and Give Growth Teams Long Timelines

A common failure mode is running an experiment and then bending the data to fit the hypothesis under pressure to show monthly progress. Laura says growth teams beholden to short-timeline wins drift into vanity metrics and data massaging. Real success comes from reporting over longer periods so teams have enough time to fail and learn the right thing.

  • Teams under pressure make the data fit the hypothesis, or run tests with no hypothesis at all
  • Set the validation plan in advance to avoid the trap
  • Short-timeline expectations breed vanity metrics and data massaging
  • Growth success requires longer reporting windows and time to fail

“that notion of kind of making the data wrap to fit like a concept”

Laura Schaffer · 50:00

“any growth team that's that's beholden to short timeline wins and Improvement is always going to be dangerous that's a that's an environment that's conducive…”

Laura Schaffer · 51:30
#experimentation#growth teams#metrics#leadership