Lower-Confidence, Higher-Volume Experimentation
Drop the 95% confidence bar to run more experiments, backstopped by qualitative corroboration and a pre-set game plan
- Difficulty
- Advanced
- Time to result
- ~ongoing to results
- Steps
- 4
- Confidence
- 85%
The 95% confidence standard belongs to domains where a false positive is dangerous (research, pharma). Growth teams converting and upselling users don't carry that burden, so accepting a lower confidence interval lets you run far more experiments; over a year the net is more wins. It requires deciding the plan before you run, and hardening lower-confidence results with qualitative data.
Origin
Laura Schaffer draws on her academic background running lab experiments for publication (where 95% confidence was essential because results influenced education and bias research) to argue growth teams are 'fortunate' to not carry that burden and can trade confidence for volume.
Core principles
- 0195% confidence exists for domains where false positives/negatives are genuinely dangerous
- 02Growth teams converting users don't carry that risk and can accept more false successes
- 03Lowering the confidence interval can double or triple the experiments you run in a year
- 04Netted over a year, more experiments at lower confidence yields more real wins
- 05Never let low confidence become an excuse to make the data fit the hypothesis
How to run it
- 1
Match the confidence bar to the stakes
Assess the real cost of a false success. For most growth work it's low, so a sub-95% interval is defensible; for high-stakes domains keep 95%.
- 2
Set the game plan before running
Decide the hypothesis, the confidence bar, and how you'll judge the result before you start — never after seeing the data.
Watch out A common failure mode is running first, then making the data fit the hypothesis (or having no hypothesis at all).
- 3
Harden lower-confidence results with qualitative data
When you accept more risk, corroborate with qualitative feedback — e.g. confirm the winning variant's users report it felt easier and the losing variant's users report getting stuck.
Pro tip Qualitative responses let you lean less on the quantitative bar to make the call.
- 4
Tell the data story to skeptics
Explain to data scientists that the ~80% base fail rate is itself hard data: running too few experiments (e.g. 10/year) may yield only ~2 wins, so raising volume nets more successes.
In the wild
Schaffer describes how, if accepting a lower confidence interval on the pill-in-the-hot-dog phone-number test, she would want qualitative feedback: users thrown into the existing flow reporting they felt 'out of my depth' on the phone number, and users in the docs variant reporting it felt easier and more reassuring.
→ Combining a lower confidence bar with corroborating qualitative signals makes it more likely you ship things that genuinely help customers while running more experiments overall.
Common mistakes
Deciding the confidence approach after seeing results
If the plan isn't set in advance, teams under pressure make the data fit a concept — sometimes with no real hypothesis at all — which invalidates the learning.
Running too few high-confidence experiments
Given the ~80% fail rate, running only ~10 experiments a year may net just ~2 wins; refusing to lower the bar to increase volume is itself a large hidden risk.
Is it for you?
Best for
Growth/experimentation teams and their data scientists deciding how strict a statistical bar to hold
Not ideal for
Pharma, safety, or research contexts where a false positive carries serious real-world harm
From the transcript
“the 95 confidence rate belongs in some companies in some Industries because the risk of of you know the impact of a false success is…”
“especially if that like doubles amount of experiments that you can run in a year”
“you must have this game plan set before you run something”
“I would very much want to see qualitative feedback to confirm that that hypothesis was true”
From the episode
Career frameworks, A/B testing mistakes, counterintuitive onboarding tips, selling to developers
Laura Schaffer (VP of Growth at Amplitude)