The One-Month Experimentation Rule
If you can't collect the sample size in a month, don't A/B test it; ship it and measure pre vs. post.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 91%
Testing everything is a paralyzing disease for growth teams: it slows velocity and learnings and makes teams afraid to change anything untested. Verna's discipline: reserve rigorous experimentation for high-traffic or high-stakes decisions where you can hit sample size fast, and for everything else trust intuition and use pre-versus-post measurement, which still lets you roll back.
Origin
Elena Verna, on over-indexing on statistical significance in the age of data.
Core principles
- 01Experimenting on everything paralyzes growth teams and is hard to escape once locked in
- 02Data is only good if you have enough of it, fast enough
- 03Statistics is directional, not gospel: a 95% significance still carries a 5% chance the effect isn't there
- 04Trust intuition more; knowing your user and market lets your brain connect dots data can't
How to run it
- 1
Apply the one-month sample-size rule
If you can't collect a sufficient sample size within a month, don't run the A/B test; low-volume real estate that takes 8 months to answer isn't worth testing.
Pro tip 'If we cannot collect the sample size in a month, we shouldn't test it. Period.'
- 2
Reserve rigorous testing for high-stakes or high-traffic cases
Run tight statistical experiments when validating a big strategic pivot, or on high-traffic real estate where even a 0.1% difference means millions.
- 3
Otherwise ship and measure pre vs. post
For everything else, just go: release, and do a 24-hour, 7-day, and 28-day read-out, even returning a year later to measure retention/expansion. Roll back if it doesn't work.
Pro tip Pre-versus-post is powerful, still lets you assess impact, and still lets you roll back, without demanding a scientific explanation for every change.
Watch out 'Go go go' does not mean release blindly and move on; still do the structured read-outs.
- 4
Weight your own intuition
Decide where you truly need precision (and can get it fast) versus where you should just go for it, accept some failures, and roll back.
Watch out Don't treat a statistically significant result as face-value certainty; it's a directional data point.
In the wild
Verna's illustration: low-volume real estate where reaching a statistically valid answer would take eight months. Testing it that long stalls progress for no proportionate gain.
→ Better to ship and use pre-versus-post measurement than to freeze the change for eight months chasing significance.
Common mistakes
Making experimentation the only way decisions get made
Requiring a scientific measurement for every change is a paralyzing disease that kills velocity and learnings and traps teams in a state where they fear any untested change.
Is it for you?
Best for
Growth teams over-indexed on A/B testing and statistical significance for every change
Not ideal for
Genuinely high-traffic surfaces or major strategic pivots where precise measurement is worth the wait
From the transcript
“If every single one of your initiatives that you're doing on growth is an experiment, that's a problem.”
“My rule of thumb, if we cannot collect the sample size in a month, we shouldn't test it. Period.”
“Pre versus post is pretty powerful.”
“people should trust their intuition a little bit more. Data is good, but data is only good if you have enough of it.”
From the episode
10 growth tactics that never work
Elena Verna (Amplitude, Miro, Dropbox, SurveyMonkey)