Sample Ratio Mismatch (SRM) Validity Check
Verify your traffic split matches its design before trusting any experiment result
- Difficulty
- Moderate
- Time to result
- ~days to results
- Steps
- 4
- Confidence
- 95%
A sample ratio mismatch (SRM) is when the actual split of users between control and treatment deviates from the intended design (e.g. 50/50) by more than chance can explain. It is the single most common invalidator of experiments, and any experiment failing the check should not be trusted regardless of how good the results look.
Origin
Diagnostic formalized by Ronny Kohavi's team at Microsoft; a 2018 paper documented that ~8% of Microsoft experiments suffered from SRM, and later third-party platforms adopted the same test.
Core principles
- 01A designed 50/50 split that comes back materially off (e.g. 50.2/49.8 on a million users) signals something is broken
- 02A simple probability formula tells you how unlikely the observed split is under the intended design
- 03Bots are the most common cause, often hitting control and treatment in different proportions
- 04Data-pipeline issues and mid-funnel experiment starts with external campaign traffic are other frequent causes
- 05An experiment with SRM cannot be trusted — but you still need to see the metrics to diagnose why
How to run it
- 1
Record the designed split
Note the intended allocation (e.g. 50% control, 50% treatment) that your randomization or hash function is supposed to produce.
- 2
Compare actual counts against design
Plug the observed user counts per arm and the designed ratio into the SRM probability formula (Kohavi shares a spreadsheet) to get the probability the deviation happened by chance.
Pro tip If the formula says the split should occur once in half a million experiments and you haven't run that many, something is wrong.
Watch out 'It's close to 50%' is not a valid judgment — use the formula, not intuition.
- 3
Block and diagnose failed experiments
When SRM is detected, treat results as invalid; investigate bots, pipeline filters, and traffic sources. Keep the metrics visible so you can trace the cause.
Pro tip You still need the scorecard numbers to debug, so gate rather than fully hide them.
Watch out PMs will rationalize ('my sample was small so I can still present it') — that bias must be actively fought.
- 4
Make the warning impossible to ignore
Escalate the UI treatment until people cannot quietly present invalid results — from a banner, to a blanked scorecard with an 'OK to expose' button, to red-lining every number so screenshots reveal the SRM.
Pro tip Red-line every metric so a screenshot carries the SRM warning with it and can't be laundered into a slide deck.
Watch out Banners alone get ignored; people pressed the reveal button and presented anyway.
In the wild
When Kohavi's team first implemented the SRM check across Microsoft's ~20,000 experiments/year, a 2018 paper found around 8% of experiments suffered from SRM despite years of running experiments.
→ Thousands of previously-trusted experiments were revealed as invalid, and diagnosing them exposed systemic bot and pipeline problems.
A plain SRM banner was ignored; blanking the scorecard behind an 'OK to expose' button was clicked through anyway; only red-lining every number so screenshots carried the warning stopped PMs presenting invalid results.
→ Invalid results could no longer be quietly laundered into presentations, protecting organizational trust.
Common mistakes
Eyeballing the split instead of using the formula
A 50.2/49.8 split on a million users looks trivially close but can be a one-in-half-a-million event, so intuition passes experiments the math would reject.
Presenting SRM-flagged results anyway
Rationalizing a flagged experiment ('the sample was small') propagates invalid conclusions and erodes trust in the whole platform.
Is it for you?
Best for
Teams operating an experimentation platform who need an automated trust check on every experiment
Not ideal for
Very low-volume experiments where counts are too small for the ratio test to be meaningful
From the transcript
“one of the things that is the most common occurs by far which is a sample ratio mismatch”
“around eight percent of experiments that suffered from the sample ratio mismatch and it's a big number”
“every number in the scorecard was highlighted with a red line so that if you took a screenshot other people could tell you that a…”
From the episode
The ultimate guide to A/B testing
Ronny Kohavi (Airbnb, Microsoft, Amazon)