False Positive Risk and the Replication Protocol
A p<0.05 win can be 26% likely false when idea success rates are low — replicate borderline results
- Difficulty
- Expert
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 93%
Most people misread a p-value as the probability the treatment is better, which is wrong. Because a p-value is conditional on the null hypothesis, the real quantity you want — false positive risk — depends on your prior success rate and can be far above 5%. When idea success rates are low, statistically significant results must be replicated before you trust them.
Origin
Kohavi's application of Bayesian false-positive-risk reasoning to online experiments, with the Airbnb 26% figure drawn from his published work.
Core principles
- 01One-minus-p-value is NOT the probability the treatment beats control — that's the most common and incorrect interpretation
- 02A p-value assumes the null hypothesis is true and gives the probability of the data under it
- 03To get what you want you must invert with Bayes' rule, which requires a prior: the base rate that ideas succeed
- 04With a low success rate (Airbnb search ~8%), a p<0.05 win carries ~26% false-positive risk, not 5%
- 05Lowering the required p-value and replicating borderline results drives false-positive risk far down
How to run it
- 1
Estimate your prior success rate
Use historical data on how often your ideas actually win (e.g. two-thirds fail, or 92% fail in a mature domain) as the prior.
Pro tip Keeping a running record of your historical win rate is what makes this calculation possible.
- 2
Compute false positive risk, not raw p-value
Combine the prior with the p-value to get false-positive risk — the actual chance a significant result is spurious.
Pro tip At an 8% success rate, a p<0.05 result is ~26% likely to be a false positive.
Watch out Reporting only the p-value badly overstates how sure you should be.
- 3
Replicate borderline significant results
When p is between 0.01 and 0.05, re-run the experiment instead of shipping on the first result.
Watch out Don't broadcast a borderline win as institutional knowledge until it's replicated.
- 4
Combine experiments for a joint p-value
Merge the original and replication using Fisher's or Stouffer's method to get a joint probability, which is usually far lower.
Pro tip Requiring p<0.01 plus replication makes shared 'wins' genuinely trustworthy.
In the wild
At Airbnb search where only ~8% of ideas succeeded, a statistically significant result at p<0.05 carried a 26% chance of being a false positive, so the team required borderline results (0.01-0.05) to be replicated and combined.
→ Combining replicated experiments produced much lower joint p-values, making declared wins reliable enough to enter institutional knowledge.
Common mistakes
Reading 1-minus-p as probability treatment wins
A p=0.02 result does not mean 98% chance the treatment is better; the p-value is conditional on the null, so this interpretation vastly overstates confidence.
Shipping borderline significant results without replication
When base success rates are low, a single p<0.05 win can be ~26% spurious, so publishing it as a proven success pollutes institutional learning.
Is it for you?
Best for
Data scientists and PMs who report experiment outcomes and want calibrated confidence
Not ideal for
High-throughput low-stakes optimization where the cost of an occasional false positive is negligible
From the transcript
“many people assign one minus p-value as the probability that your treatment is better than control ... that is wrong”
“if you get a statistically significant result with the p-value less than 0.05 there's a 26 percent chance that this is a false positive result”
“if you're less than 0.05 but above 0.01 rerun replica when you replicate you can combine the two experiments and get a combined p-value”
From the episode
The ultimate guide to A/B testing
Ronny Kohavi (Airbnb, Microsoft, Amazon)