Long-Term Holdout Experimentation
Ship experiment winners fast, but keep the exposed cohort held and re-check GMV impact at 3, 6, and 12 months.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 95%
Most teams call an experiment after a few weeks based on a short-term lift and move on. Shopify instruments experiments so the assigned cohort stays trackable long after the winner ships to 100%, and the tooling automatically pings every experimenter with updated long-term results at 3, 6, 9, and 12 months. This reveals that a large share of short-term wins have no lasting impact, and occasionally that a flat result was actually a big long-term winner.
Origin
Archie Abrams describing Shopify's internal experimentation platform and two-layer holdout system.
Core principles
- 01A short-term metric lift is not the same as durable business impact
- 02Roughly 30-40% of experiments showing early lift show no incremental long-term GMV lift
- 03Neutral-then-positive happens; negative-then-positive essentially never does
- 04Automated long-term pings make teams accountable to what actually mattered
How to run it
- 1
Run a two-layer holdout system
Hold out ~5% of users across all quarterly changes as a blanket holdout, and for new-merchant-only changes run a 50-50 split for a few weeks.
Pro tip Long-term holdouts work cleanly on NEW users; existing-user holdouts are more complicated.
- 2
Ship the winner to 100% but keep the cohort tagged
After calling the experiment (~3 weeks), ship the winner broadly but continue tracking the originally-assigned cohort so you can compare them a year later.
Pro tip This lets you ship fast AND preserve the measurement.
- 3
Automate long-horizon result pings
Have the experimentation tool email everyone attached to the experiment updated results at 3, 6, 9, and 12 months so no one can hide from the true long-term outcome.
Pro tip Track cohort GMV curves over time rather than retention curves.
- 4
Revisit and re-learn
When the long-term curve differs from the short-term call, feed that back into which input metrics you trust and which bets to double down on.
Pro tip Look at the downstream metrics a year or two out on the experiments you thought were your biggest winners.
Watch out You'll be surprised how often the long-term metric differs from what you assumed.
In the wild
Aggressive alerts about failed credit-card payments produced major short-term lift, but 6-12 months out showed no long-term GMV lift because the people letting payments fail weren't truly committed entrepreneurs, a selection-bias effect.
→ Team learned it was pulling forward apparent wins and redirected effort to bigger opportunities.
Giving merchants pre-filled image/text/collage blocks had no lift on converting to a paying merchant, but six months later drove a large increase in merchants actually selling and producing GMV because they built better, higher-converting stores.
→ A neutral short-term result turned into a major long-term winner, validating shipping neutral experiments on good intuition.
Common mistakes
Calling an experiment done at week three
A third of early wins evaporate on a long horizon; stopping measurement there over-credits the team and misdirects future work.
Using retention curves instead of value curves
For a business where winners are power-law GMV producers, cohort GMV over time is the truer signal than cohort retention.
Is it for you?
Best for
Growth teams with enough traffic and tooling to hold out cohorts and a business willing to think in years
Not ideal for
Early-stage teams that cannot wait a year or lack the volume for stable long-horizon reads
From the transcript
“we so we've constantly will relook at an experiment a year later see that the way the gmv curve for the distribution was different than…”
“in all cases the group is held we watch them and that's where that ping comes back three six months 12 months to relook at…”
“it's probably in the it's it's in the 30 to 40% range”
“I've seen neutral be positive but I haven't seen negative”
From the episode
Breaking the rules of growth: Why Shopify bans KPIs, optimizes for churn, prioritizes intuition, and builds toward a 100-year vision
Archie Abrams (VP Product, Head of Growth at Shopify)