The YOLO Rule: When Not to Run an Experiment
Experiments have a cost — sometimes shipping 40 things fast beats testing 10 things properly
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 90%
Every growth team is told to experiment with rigour. Levesque's counterintuitive addition is that experiments carry real cost — build time, analysis time, learning time — and not everything deserves one. When a change is obviously a better product experience or you simply know it will work, ship it (YOLO it) and use cheaper safety nets. Speed can outweigh the cost of the experiment.
Origin
A recent change Luc Levesque implemented on his growth teams at Shopify, explicitly framed as the subtlety that growth teams almost never discuss because 'you need to experiment' is treated as gospel.
Core principles
- 01Experimentation rigour is common knowledge; the cost of that rigour is not.
- 02The cost of an experiment is not just the test — it's building it, waiting, analysing, and learning.
- 03Speed is a real return, and it can exceed the value of the certainty an experiment buys.
- 04You still need guardrails — you just need cheaper ones.
How to run it
- 1
Price the experiment before you run it
For each proposed test, count the full cost: building the experiment, waiting for the result, analysing the outcome, and extracting the learning. That's the price of the certainty you're buying.
- 2
Ask whether you already know
If the change is unambiguously a better product experience, or you simply know it's going to work, that's a candidate to skip the experiment and ship. The certainty an A/B test would buy you is certainty you already have.
Watch out This is a judgment call and it can be abused. It works because Levesque's teams are calibrated on growth; it is not a licence to skip measurement entirely.
- 3
YOLO in volume, not in singles
The model works at volume: ship 40 things fast, accept that only three land, and net out ahead of a rigorous programme that would have tested a fraction of them in the same window.
- 4
Keep cheap guardrails
Replace the experiment with lighter safety nets: look at pre-post metrics, and use holdouts. These give you enough protection against causing major damage without paying the full experiment tax.
Watch out Skipping the experiment does not mean skipping the check. Without pre-post and holdouts you're flying blind.
In the wild
Asked what relatively minor change had a big impact on his team's ability to execute, Levesque pointed to consciously loosening the experimentation mandate. Rather than experimenting on everything, the team now ships changes it is confident about, running 40 in the time a rigorous programme would test far fewer, and watching pre-post and holdouts to catch damage.
→ A meaningful lift in execution velocity — the speed advantage outweighed the cost and time of running experiments on everything.
Common mistakes
Treating 'we must experiment' as an unconditional rule
Rigour is the default advice and it's right most of the time — but applied to every change it becomes a throughput bottleneck nobody in growth ever names.
YOLOing without pre-post or holdouts
The whole model depends on cheap damage detection. Ship-and-forget with no measurement is not this framework, it's just guessing.
Is it for you?
Best for
Mature growth teams with strong intuition and an existing experimentation culture, where the experiment queue has become the constraint on velocity.
Not ideal for
Teams without calibrated judgment or any measurement infrastructure, and any change where being wrong is expensive or irreversible.
From the transcript
“build the experiment so there's there's a cost to an experiment and not everything needs to be experimented and that's not something that I generally…”
“so one thing that we're definitely focused more on lately is this idea of sometimes you just need to YOLO it because it's a better…”
“know cause major damage but the speed can outweigh the cost and time it takes to do experiments”
From the episode
Leveraging growth advisors, hiring well, mastering SEO, and honing your craft
Luc Levesque (Shopify, Meta, TripAdvisor)