Fail-Fast Iterative Validation
Since ~80% of hypotheses fail, use cheap validation methods first and reserve A/B testing for pre-vetted ideas
- Difficulty
- Moderate
- Time to result
- ~ongoing to results
- Steps
- 4
- Confidence
- 90%
Roughly 80-90% of product/growth hypotheses fail, so the goal isn't to be smarter than that rate — it's to fail faster and cheaper. Iterative shipping and cheap validation (painted doors, mocks, qualitative interviews) let failure act as a compass, and reserve expensive A/B testing for ideas already vetted along the way.
Origin
Laura Schaffer draws on published research (she cites a Microsoft experimentation-platform builder's article showing 80%+ of hypotheses fail across companies like Netflix and Microsoft) and her own experience that full redesigns are negative nearly 100% of the time.
Core principles
- 01~80-90% of untested hypotheses fail; success requires the problem, person, timing, solution, and presentation all to be right
- 02Big-bang redesigns maximize the chance you ship the wrong thing after a long build
- 03Iterative shipping catches failure sooner — failure is a compass, not a wall
- 04You can't beat the base hit-rate, but you can change how fast and cheaply you validate
- 05A/B testing is one of the most expensive validation methods; use cheaper ones to invalidate first
How to run it
- 1
Default to iterative, not big-bang
Prefer small experiments piece-by-piece over disappearing for six months to build and ship a full redesign.
Watch out Launching a whole redesign was, in Schaffer's experience, negative nearly 100% of the time.
- 2
Invalidate cheaply before you build
Use painted doors (test demand for a concept before it exists) and mocks put in front of users to kill weak hypotheses early.
Pro tip You can invalidate tons of hypotheses at the mock stage for a fraction of the cost of a real test.
- 3
Ship things ugly enough to be embarrassing
Get rough versions in front of customers fast so failure surfaces early and cheaply — 'if it's not embarrassing, you've gone too far.'
- 4
Reserve A/B testing for vetted ideas
Only ideas that survived cheaper validation should reach the expensive A/B testing environment, reducing your fail rate there.
In the wild
Schaffer cites a Microsoft experimentation-platform author who applied the scientific method to how often teams are wrong about their hypotheses and found 80%+ (some say 90%) fail across companies like Netflix and Microsoft. She reasons it makes sense: to succeed you must get the problem, the person, the timing, the solution, and its presentation all right at once.
→ Reframes failure as expected and cheap-to-harvest, shifting the team's strategy toward fast validation rather than trying to be right more often.
Common mistakes
Burying yourself to build a big redesign, then shipping it
The longer and bigger the build before customer contact, the more likely you ship the ~80% wrong thing after heavy investment; redesigns were negative nearly 100% of the time.
Sending unvetted ideas straight to A/B testing
A/B testing needs design, engineering, and run-time; using it as first-line validation for ideas you could have killed with a mock or painted door wastes the most expensive resource you have.
Is it for you?
Best for
Growth and product teams deciding how to sequence validation and whether to iterate or redesign
Not ideal for
High-risk domains (pharma, safety) where a false success is dangerous and rigor must dominate speed
From the transcript
“80 plus percent some companies saying 90 of things fail”
“failure doesn't have to be a wall it can be a compass”
“a b testing is one of the most expensive kinds of ways to validate an experiment”
“you can do that with painted doors”
“let's fail fast by using those tools”
From the episode
Career frameworks, A/B testing mistakes, counterintuitive onboarding tips, selling to developers
Laura Schaffer (VP of Growth at Amplitude)