Ronny Kohavi (Airbnb, Microsoft, Amazon)27 July 2023

The ultimate guide to A/B testing

7Frameworks

15Insights

Episode overview

Ronny Kohavi explains why organizations should test nearly every software change while maintaining a portfolio that includes risky, potentially transformative bets. Drawing on experiments at Microsoft, Amazon, Bing, and Airbnb, he shows that most ideas fail and that reliable platforms, institutional memory, and a lifetime-value-oriented overall evaluation criterion are essential. He also covers sample-ratio mismatch, false-positive risk, replication, variance reduction, and practical thresholds for companies beginning experimentation.

Key ideas

Most experiments fail: Kohavi reports approximate failure rates of two-thirds at Microsoft, 85% in mature Bing domains, and 92% in Airbnb search relevance.
Teams should test every software change when feasible, while deliberately allocating part of their portfolio to high-risk, high-reward ideas.
An overall evaluation criterion should predict long-term customer lifetime value and balance business gains against user-experience guardrails.
Organizations should document and revisit surprising winners and losers so experimental learning becomes durable institutional memory.
Start building experimentation capability with tens of thousands of users; around 200,000 users makes testing smaller effects substantially more practical.
Trustworthy platforms should detect sample-ratio mismatches and withhold apparently successful results when assignment or data pipelines may be invalid.
A p-value is not the probability that treatment beats control; historical success rates affect false-positive risk, making replication valuable.
Self-service platforms, automated scorecards, metric capping, and CUPED-style variance reduction can lower experimentation cost and shorten time to reliable results.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Mindset4 steps

False Positive Risk and the Replication Protocol

A p<0.05 win can be 26% likely false when idea success rates are low — replicate borderline results

Read →

Innovation4 steps

Incremental OFAT Over Big-Bang Redesign

Decompose big redesigns into one-factor-at-a-time tested steps because ~80% of ideas fail

Read →

Leadership4 steps

Institutional Memory of Experiments

Capture wins, losses, and surprises so the org keeps learning instead of relaunching old ideas

Read →

Strategy4 steps

Sample Ratio Mismatch (SRM) Validity Check

Verify your traffic split matches its design before trusting any experiment result

Read →

Strategy4 steps

The 200K-User Experimentation Readiness Rule

Below ~200K users detect only big effects; use the wait to build platform and culture

Read →

Strategy4 steps

The Overall Evaluation Criterion (OEC)

Define one success metric with countervailing guardrails that predicts long-term lifetime value

Read →

Mindset4 steps

Twyman's Law: Investigate the Too-Good-to-Be-True Result

Any surprising or extreme experiment result is probably wrong until you've hunted for the flaw

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 7

Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter
“welcome to Lenny's podcast where I interview world-class product leaders and growth experts”
▶ Watch at ~1:00approx.
Ronny KohaviMentionsRani is seen by many as the world expert on a b testing and experimentation
“today my guest is Ronnie kohavi”
▶ Watch at ~1:00approx.
RicardoMentionsmentioned
“shout out to Ricardo or mutual friend who helped make this conversation happen”
▶ Watch at ~10:30approx.
Jakub LinowskiCoinedhe asks people to send them results of experiments and he derives he puts them into patterns
“the name is Jacob linovsky he asks people to send them results of experiments”
▶ Watch at ~16:00approx.
Brian CheskyMentionsBrian chesky at Airbnb there was a whole article where we unpacked this moment in their product development cycle
“it's becoming a lot more top-down Brian Vision oriented”
▶ Watch at ~45:30approx.
Ramesh JohariMentionsRamesh is a professor at Stanford University where he researches and teaches data science methods, especially for online marketplace design and operation.
“Ramesh Johari at Stanford pointed this out became a consultant of the company and then they fixed it”
▶ Watch at ~54:30approx.
Jeff BezosMentionsthe Jeff Bezos thing that like the success of today was set in motion three years ago
“I still remember that email from Jeff which is no more PowerPoint”
▶ Watch at ~1:19:00approx.

Resources · 36

AirbnbMentionscompany · Brian Chesky, Joe Gebbia, and Nathan Blecharczyk
“most recently usvp and Technical fellow of relevance at Airbnb where you led their search experience team”
▶ Watch at ~1:00approx.
Trustworthy Online Controlled ExperimentsCoinedbook · Ronny Kohavi and co-authors
“he's also the author of the go-to book on experimentation called trustworthy online controlled experiments”
▶ Watch at ~1:30approx.
AmazonMentionscompany · Jeff Bezos
“before that he was director of data mining and personalization at Amazon”
▶ Watch at ~1:30approx.
MicrosoftMentionscompany · Bill Gates and Paul Allen
“prior to that he was corporate vice president and Microsoft where he led Microsoft's experimentation platform team”
▶ Watch at ~1:30approx.
MixpanelMentionssoftware · Mixpanel
“this episode is brought to you by mixpanel”
▶ Watch at ~2:30approx.
RoundMentionscompany
“round is the private Network built by Tech leaders for Tech leaders”
▶ Watch at ~3:00approx.
BingUsesproduct · Microsoft
“somebody proposed to change the way that ads were displayed on Bing the search engine”
▶ Watch at ~5:00approx.
BingMentionsproduct · Microsoft
“the biggest Revenue impact of being in all its history”
▶ Watch at ~7:30approx.
MSNUseswebsite · Microsoft
“it was done in the UK for opening Hotmail and then we moved it to MSN”
▶ Watch at ~9:30approx.
HotmailUsessoftware · Microsoft
“it was done in the UK for opening Hotmail and then we moved it to MSN”
▶ Watch at ~9:30approx.
AirbnbUsescompany · Brian Chesky, Joe Gebbia, and Nathan Blecharczyk
“we ran some 250 experiments in my tenure there in search relevance”
▶ Watch at ~12:30approx.
Rules of ThumbCoinedpaper · Ronny Kohavi and colleagues
“one of them is a paper that we wrote called rules of thumb”
▶ Watch at ~15:30approx.
GoodUIRecommendswebsite · Jakub Linowski
“resource that's useful that I recommend to people and it's a site called goodui.org”
▶ Watch at ~16:00approx.
Windows IndexerUsessoftware · Microsoft
“we were running this experiment at Microsoft to improve the windows indexer”
▶ Watch at ~18:30approx.
TwitterUseswebsite · Twitter
“we hooked into the Tweeter fire hose feed and we hooked it to Facebook”
▶ Watch at ~23:00approx.
FacebookUsescompany · Meta
“we hooked into the Tweeter fire hose feed and we hooked it to Facebook”
▶ Watch at ~23:00approx.
NetflixMentionscompany · Netflix
“at Netflix they tried a social component that also failed”
▶ Watch at ~24:00approx.
MicrosoftUsescompany · Bill Gates and Paul Allen
“we got to that at Microsoft where after a while the cost of running experiments was so low”
▶ Watch at ~25:30approx.
EppoMentionssoftware · Eppo
“EPO is a Next Generation a b testing platform built by Airbnb alums for modern growth teams”
▶ Watch at ~41:30approx.
Cambridge University PressMentionscompany
“we were approached by Cambridge after a tutorial that we did to write a book”
▶ Watch at ~50:30approx.
OptimizelyMentionssoftware
“optimizely in its early days were very statistically naive”
▶ Watch at ~53:00approx.
LinkedInUseswebsite · LinkedIn
“there's a whole series of eight sessions that I posted on LinkedIn”
▶ Watch at ~1:06:30approx.
Mistakes Were Made (But Not by Me)Recommendsbook · Carol Tavris and Elliot Aronson
“it's called mistakes were made but not by me”
▶ Watch at ~1:15:00approx.
Hard Facts, Dangerous Half-Truths, and Total NonsenseRecommendsbook · Jeffrey Pfeffer and Robert I. Sutton
“there's a slightly older book that I love called hard facts dangerous half-truths and total nonsense”
▶ Watch at ~1:15:00approx.
ChernobylRecommendstv show · Craig Mazin
“I recently saw a short series called Chernobyl on the disaster it I thought it was amazingly well done”
▶ Watch at ~1:15:30approx.
Blink camerasRecommendsproduct · Blink
“a bling camera is this small camera you stick in two double a batteries”
▶ Watch at ~1:17:30approx.
Blink camerasUsesproduct · Blink
“I threw five cameras out and I saw where he came in”
▶ Watch at ~1:18:00approx.
PowerPointMentionssoftware · Microsoft
“I still remember that email from Jeff which is no more PowerPoint”
▶ Watch at ~1:19:00approx.
Google DocsUsessoftware · Google
“now it's all you know based on word or Google Docs where people comment”
▶ Watch at ~1:19:30approx.
MavenUseswebsite · Maven
“there's a class that I teach every quarter on Maven”
▶ Watch at ~1:22:00approx.
Ronny Kohavi’s live cohort-based experimentation courseCoinedcourse · Ronny Kohavi
“there's a class that I teach every quarter on Maven”
▶ Watch at ~1:22:00approx.
Trustworthy Online Controlled ExperimentsRecommendsbook · Ronny Kohavi and co-authors
“learn more by reading my book if you want again all proceeds go to charity”
▶ Watch at ~1:22:00approx.
Lenny's Podcast websiteMentionswebsite · Lenny Rachitsky
“you can find all past episodes or learn more about the show at lennyspodcast.com”
▶ Watch at ~1:22:30approx.
Apple PodcastsMentionssoftware · Apple Inc.
“you can subscribe to the show on Apple podcast Spotify or your favorite podcast app”
▶ Watch at ~1:22:30approx.
SpotifyMentionssoftware · Spotify
“you can subscribe to the show on Apple podcast Spotify or your favorite podcast app”
▶ Watch at ~1:22:30approx.
Ronny Kohavi’s live cohort-based experimentation courseRecommendscourse · Ronny Kohavi
“there's gonna be a code to get a discount on your course”
▶ Watch at ~1:22:30approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster1:02:00

What a P-Value Actually Means (and Why 0.05 Isn't 5%)

Kohavi corrects the most common p-value error: people read 1 minus the p-value as the probability the treatment beats control, which is wrong. The p-value assumes the null hypothesis is true; getting the number you actually want requires Bayes' rule and a prior. Given Airbnb search's 8% success rate, a result at p < 0.05 carries a 26% chance of being a false positive — not 5% — which is why he pushed teams to p < 0.01 plus replication.

People wrongly read 1 minus p-value as the probability treatment beats control
A p-value assumes the null hypothesis is true; it's a conditional probability
Recovering the desired probability needs Bayes' rule plus a prior success rate
At Airbnb search's 8% success rate, a p < 0.05 result is 26% likely to be a false positive
Fix: require p < 0.01 and replicate, combining experiments via Fisher's/Stouffer's method

“many people assign one minus p-value as the probability that your treatment is better than control”
Ronny Kohavi · 1:02:30

“if you get a statistically significant result with the p-value less than 0.05 there's a 26 percent chance that this is a false positive result…”
Ronny Kohavi · 1:04:30

#ab-testing#p-value#statistics#false-positive

Hot Take· 3

Hot Take20:30

You Can't Experiment Too Much — But Balance Your Portfolio

Kohavi is a strong advocate of 'test everything' — any code change or feature should sit inside an experiment, because even small bug fixes can have surprising impact. He argues you can't over-experiment, but you can over-focus on incremental changes. Like a stock portfolio, you need a mix of safe incremental bets and high-risk, high-reward swings that will usually fail.

Every code change or feature should be inside an experiment
Even small bug fixes can have surprising, unexpected impact
It is not possible to experiment too much
You should allocate some effort to high-risk, high-reward bets
Be ready to fail ~80% of the time on the big swings

“and so I don't think it's possible to experiment too much”
Ronny Kohavi · 21:00

“it's like in stock you need a portfolio you need some experiments that are incremental that move you in the direction”
Ronny Kohavi · 21:30

#ab-testing#experimentation-culture#risk#portfolio

Hot Take36:00

Big Redesigns Almost Always Fail

Both agree that full product redesigns rarely test positive, and teams then have to claw back what they broke. Kohavi's advice: if you must redesign, do it in steps and test along the way rather than shipping 17 changes at once, since combining many likely-to-fail changes makes the whole more likely to be negative. He calls the incremental approach OFAT — one factor at a time.

Full redesigns almost never produce a positive result
Doing 17 changes together is more likely to be negative
Redesign in steps and adjust as you learn
OFAT — one factor at a time — isolates what works
The sunk-cost fallacy pressures teams to ship redesigns even when they hurt users

“so the right way to do this is to say yes we want to do a redesign but let's do it in steps and test…”
Ronny Kohavi · 36:30

“do them in smaller increments learn from it's called o-fat one factor at a time do one factor learn from it and adjust”
Ronny Kohavi · 38:00

#ab-testing#redesign#incremental#ofat

Hot Take44:00

Never Ship a Flat Result

Kohavi argues that if you're not experimenting, roughly 70% of what you ship is hurting the business — some flat, some negative. A flat result should be a no-ship, because new code adds maintenance overhead with no offsetting value. The only exception is a legal requirement, and even then you should test several options and ship the one that hurts least.

Without experiments, ~70% of shipped work is flat to negative
A statistically-flat result is a no-ship: new code carries maintenance cost
'The team will be demotivated if we don't ship' is the wrong reason to ship
Legal requirements are the exception where you may ship on flat or negative
Even for legal, test options and ship the least harmful

“if you're not writing an experiment 70 of stuff you're shipping is hurting your business”
Ronny Kohavi · 44:00

“you don't ship on flat unless it's a sort of a legal requirement”
Ronny Kohavi · 44:30

#ab-testing#shipping#maintenance#decision-making

Explainer· 4

Explainer10:30

Big Wins Are Rare; Progress Comes Inch by Inch

Kohavi warns that the one-hour-of-work-for-a-massive-win 'gold nugget' is very rare. Most gains accumulate through many tiny improvements: Bing's relevance team targets a 2% annual lift built from many 0.1–0.2 gains, and at Airbnb search relevance ~250 experiments added up to a 6% revenue improvement — even though 92% of the ideas failed to move the target metric.

Bing relevance aims for ~2% improvement per year, assembled from tiny gains
Airbnb search relevance ran ~250 experiments in his tenure
Those small wins summed to a 6% revenue improvement
92% of those experiments failed to move the target metric
Only ~8% of ideas actually moved the key metrics

“but yes I think most results are inch by inch you improve small amounts lots of them”
Ronny Kohavi · 11:30

“of these experiments 92 percent failed to improve the metric that we were trying to move so only eight percent of our ideas actually were…”
Ronny Kohavi · 12:30

#ab-testing#incremental-gains#airbnb#bing

Explainer13:00

How Often Experiments Actually Fail: 66% to 92%

Kohavi shares real failure rates across his career: roughly two-thirds of ideas failed at Microsoft, ~85% at the more optimized Bing, and 92% at Airbnb — the highest he's observed. Other companies like Google Ads and Booking publish 80–90% failure rates. He notes these are rates of experiments, not ideas, since ~10% of experiments are aborted on day one for implementation bugs.

Microsoft: about two-thirds of ideas fail
Bing: ~85%, because a well-optimized domain is harder to improve
Airbnb: 92%, the highest he has observed
Google Ads, Booking and others publish 80–90% failure rates
~10% of experiments are aborted on the first day for implementation issues, not bad ideas

“so overall at Microsoft about 66 two-thirds of ideas fail right”
Ronny Kohavi · 13:30

“and then at Airbnb uh this 92 number is you know the highest failure rate that I've observed”
Ronny Kohavi · 14:00

#ab-testing#failure-rate#benchmarks

Explainer55:30

Sample Ratio Mismatch: The Most Common Way Experiments Go Wrong

Kohavi's most common validity red flag is a sample ratio mismatch: if a 50/50 design comes back off (say 50.2/49.8 on a million users), a formula can show that split should occur only once in half a million experiments — so something is wrong, often bots or a data pipeline. Microsoft found ~8% of experiments suffered from it, and Kohavi tells a funny story of red-lining every scorecard number so screenshots couldn't hide the flag.

A 50/50 design that returns off-ratio signals something is broken
A formula computes the probability the split happened by chance
Bots and data-pipeline issues are common causes
~8% of Microsoft experiments suffered a sample ratio mismatch
Product managers ignored banners, so the team red-lined every scorecard number for screenshots

“but I'll say maybe one of the things that is the most common occurs by far which is a sample ratio mismatch”
Ronny Kohavi · 55:30

“every number in the scorecard was highlighted with a red line so that if you took a screenshot other people could tell you that a…”
Ronny Kohavi · 1:00:00

#ab-testing#validity#sample-ratio-mismatch#bots

Explainer1:00:30

Twyman's Law: If It Looks Too Good, It's Probably Wrong

Kohavi explains Twyman's Law — any figure that looks interesting or different is usually wrong. When a metric that normally moves under 1% suddenly jumps 10%, hold the celebration and investigate, because nine out of ten times a flaw turns up. Genuine outliers exist (like the ad-title win) but survive only after repeated replication and double-checking.

Twyman's Law: any figure that looks interesting or different is usually wrong
A sudden 10% move where movement is normally under 1% is a warning, not a win
Nine out of ten times investigating reveals a flaw
Real outliers exist but must be replicated and double-checked
Fight the natural bias to want to see success

“the general statement is if any figure that looks interesting or different is usually wrong”
Ronny Kohavi · 1:00:30

“nine out of ten when we call out time is law it is the case that we find some flaw in the experiment”
Ronny Kohavi · 1:01:30

#ab-testing#twymans-law#skepticism#validity

Story· 5

Story05:00

The Two-Line Swap That Made Bing $100M

Kohavi's favorite public example: an engineer, tired of an idea languishing in the backlog, spent a couple of days moving an ad's second line up to the larger title line. The experiment tripped a revenue alarm because it lifted revenue ~12% without hurting user guardrail metrics — the biggest revenue win in Bing's history from a trivial change. It illustrates how badly teams predict experiment outcomes and how ROI-driven quick tests uncover them.

The idea sat in the backlog for months, rated below many others
Implementation took only a couple of days; an alarm fired for a suspected revenue bug
Revenue rose ~12% with no bug found after repeated replication
Worth ~$100M at the time, and it did not hurt the user experience guardrails
Teams are routinely humbled at predicting which experiments win

“this thing was worth a hundred million dollars at the time when Bing was a lot smaller and the key thing is it didn't hurt…”
Ronny Kohavi · 07:00

“but to me this was an example of a tiny change that was the best Revenue generating idea in Bing's history”
Ronny Kohavi · 08:30

#ab-testing#bing#surprising-results#revenue

Story08:30

Opening Links in a New Tab Keeps Winning

Both host and guest describe the same surprising winner appearing across companies: opening a clicked search result in a new tab. Kohavi first ran it around 2008 for Hotmail and MSN despite heavy designer pushback, and it was highly beneficial. At Airbnb the win had been discovered, then semi-forgotten, until it was reintroduced — a lesson in institutional memory.

Opening results in a new tab was one of the biggest search wins at Airbnb
Kohavi ran the same test ~2008 for Hotmail then MSN, with strong results
Designers pushed back because users didn't ask for it
Airbnb had implemented it, forgot the learning, then reintroduced it
Winners must be documented so the organization remembers them

“the search team just ran a small experiment of what if we were to open a new tab every time someone clicked on a search…”
Lenny Rachitsky · 09:00

“which is one of the things you learned about institutional memories when you have winners make sure to address them and remember them”
Ronny Kohavi · 10:00

#ab-testing#institutional-memory#search#airbnb

Story22:30

Bing's Big Social Bet That Quietly Died

A cautionary tale of a large, high-conviction bet: Bing tried to reshape search by integrating social, hooking into the Twitter firehose and Facebook. After roughly a hundred person-years and hundreds of experiments, all results were negative to flat, and the feature was aborted after about a year and a half. Kohavi notes similar social attempts failed at Netflix and Airbnb.

Bing hooked into the Twitter firehose and Facebook to integrate social
Roughly a hundred person-years were spent on the idea
Hundreds of experiments were all negative to flat
The feature existed ~18 months before being aborted
Netflix and Airbnb had similar failed social attempts

“we hooked into the Tweeter fire hose feed and we hooked it to Facebook and we spent a hundred percent years on this idea and…”
Ronny Kohavi · 23:00

“it's a million dollar question you know you continue and then maybe the breakthrough will come next month”
Ronny Kohavi · 23:30

#ab-testing#big-bets#social#bing

Story33:00

How Amazon's Email Team Gamed Its Own Metric

At Amazon, the recommendation-email team was credited for any purchase that followed an email — a metric with no countervailing cost, so they simply sent more emails and 'made more money.' Kohavi's team modeled the long-term cost of an unsubscribe (a few dollars of lost lifetime value), and found more than half the campaigns were actually negative. The insight led to per-campaign unsubscribe defaults.

The email team got credit for any purchase following an email
With no countervailing metric, more emails meant more claimed revenue — leading to spam
A data science study priced the lost lifetime value of an unsubscribe
Once modeled, more than half the campaigns were net negative
The fix: default to unsubscribing from that specific campaign, not all email

“the more emails you send the more money you're going to credit the theme and so that led to spam literally”
Ronny Kohavi · 34:00

“when we started to incorporate those formula more than half the campaigns that were being sent were negative”
Ronny Kohavi · 35:00

#ab-testing#email#lifetime-value#amazon

Story53:00

How Optimizely Lost Trust With Real-Time P-Values

Kohavi uses early Optimizely as a cautionary tale about trust. Their real-time approach let users stop an experiment the moment the p-value hit significance — a practice that inflates the false positive rate from a nominal 5% to around 30%. Customers saw 'wins' that never showed up in real revenue, a famous 'Optimizely almost got me fired' post surfaced, and the company eventually fixed the statistics.

Early Optimizely computed p-values in real time and let users stop at significance
Continuous monitoring inflates the type-one (false positive) error rate
A nominal 5% error rate could balloon to ~30%
Customers' reported wins didn't materialize in actual revenue
After public criticism, Optimizely corrected its methodology

“optimizely in its early days were very statistically naive they sort of said hey we're real time we can compute your P values in real…”
Ronny Kohavi · 53:30

“using real-time sort of p-value monitoring to optimize the offered you would probably have a 30 percent error rate”
Ronny Kohavi · 54:00

#ab-testing#trust#statistics#optimizely

Tool· 1

Tool15:30

goodui.org: A Library of Patterns That Often Win

Asked for a list of changes that tend to work, Kohavi points to two resources: Microsoft's 'Rules of Thumb' paper extracting patterns from thousands of experiments, and goodui.org by Jakub Linowski. goodui.org crowdsources experiment results into ~140 patterns, each showing how often it helped and by how much — even the 'open in a new window' win appears there.

Microsoft's 'Rules of Thumb' paper extracted patterns from thousands of experiments
goodui.org collects submitted results into named patterns
~140 patterns, each with a win rate and effect size
The 'open a new window' pattern appears in it
Kohavi suggests these patterns could feed an AI-generated roadmap

“but there's another more more accurate I would say uh resource that's useful that I recommend to people and it's a site called goodui.org”
Ronny Kohavi · 16:00

“he puts them into patterns there's probably like 140 patterns I think at this point”
Ronny Kohavi · 16:30

#ab-testing#tools#patterns#roadmap

Takeaway· 1

Takeaway26:30

The 200,000-User Rule for When to Start A/B Testing

Kohavi's practical heuristic for when experimentation starts paying off: below tens of thousands of users the statistics don't work for most metrics. To detect the 5–10% improvements startups should target on a typical retail conversion rate, you need around 200,000 users. Below that, start building the culture and platform so value compounds as you scale.

Below tens of thousands of users the math doesn't work for most metrics
Startups should chase 5–10% effects, not 1% ones
Detecting a ~5% retail conversion lift needs roughly 200,000 users
At 10s of thousands you can only detect large effects
Below the threshold, build culture, platform and integration for later

“unless you have at least tens of thousands of users the math the statistics just don't work out for most of the metrics that you're…”
Ronny Kohavi · 27:00

“so you ask for rule of thumb 200 000 users you're magical below that start building the culture start building the platform”
Ronny Kohavi · 27:30

#ab-testing#startups#statistics#rule-of-thumb

Key ideas

Frameworks in this episode

False Positive Risk and the Replication Protocol

Incremental OFAT Over Big-Bang Redesign

Institutional Memory of Experiments

Sample Ratio Mismatch (SRM) Validity Check

The 200K-User Experimentation Readiness Rule

The Overall Evaluation Criterion (OEC)

Twyman's Law: Investigate the Too-Good-to-Be-True Result

People & resources mentioned

People · 7

Resources · 36

Insights & moments

⚡Myth Buster· 1

What a P-Value Actually Means (and Why 0.05 Isn't 5%)

◆Hot Take· 3

You Can't Experiment Too Much — But Balance Your Portfolio

Big Redesigns Almost Always Fail

Never Ship a Flat Result

✶Explainer· 4

Big Wins Are Rare; Progress Comes Inch by Inch

How Often Experiments Actually Fail: 66% to 92%

Sample Ratio Mismatch: The Most Common Way Experiments Go Wrong

Twyman's Law: If It Looks Too Good, It's Probably Wrong

❝Story· 5

The Two-Line Swap That Made Bing $100M

Opening Links in a New Tab Keeps Winning

Bing's Big Social Bet That Quietly Died

How Amazon's Email Team Gamed Its Own Metric

How Optimizely Lost Trust With Real-Time P-Values

✦Tool· 1

goodui.org: A Library of Patterns That Often Win

▲Takeaway· 1

The 200,000-User Rule for When to Start A/B Testing

Myth Buster· 1

Hot Take· 3

Explainer· 4

Story· 5

Tool· 1

Takeaway· 1