Ship Neutral Experiments on Intuition (Aim-Heavy)
When a test comes back neutral, ship the version you'd have built from a blank slate, not the incumbent control.
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 3
- Confidence
- 90%
Most teams keep the control when an experiment is neutral. Shopify's 'aim-heavy' culture inverts this: if the result is neutral, neither option is objectively better, so ask what you would have shipped from a blank slate and pick the version your product intuition favors. Because neutral results sometimes turn positive on a long horizon (and negative ones essentially never turn positive), betting on intuition when the data is tied is low-risk and occasionally unlocks large delayed wins.
Origin
Archie Abrams describing Shopify's 'aim-heavy' decision culture, rooted in Toby Lutke's long-term orientation.
Core principles
- 01A neutral result means neither variant is measurably better, so data can't decide
- 02Default to the blank-slate choice, not the incumbent, when intuition is strong
- 03Neutral can become positive long-term; negative essentially never becomes positive
- 04Intuition is a legitimate tiebreaker when evidence is genuinely balanced
How to run it
- 1
Confirm the result is genuinely neutral
Verify the experiment shows no meaningful short-term difference between variants.
Watch out This applies to neutral results, not to variants that clearly lost.
- 2
Ask the blank-slate question
Ignore which option happens to be the current control and ask: if we were starting fresh, which would we ship?
Pro tip 'Let's start with what would we have shipped if we were blank slate.'
- 3
Ship the intuition pick and monitor long-term
Ship the version your product sense favors and keep it under long-term holdout monitoring in case the neutral result turns positive.
Pro tip Only trust this when a small number of strong-taste owners hold the quality bar.
Watch out Requires genuine product conviction; don't use it to rationalize arbitrary choices.
In the wild
The pre-configured online-store blocks tested neutral on paid conversion, so Shopify shipped them on intuition; six months later they drove a large lift in merchants selling and producing GMV.
→ A neutral experiment became a major long-term winner precisely because it was shipped on conviction.
Common mistakes
Reflexively keeping the control
Defaulting to the incumbent on a neutral result wastes the option value of your best intuition.
Using 'aim-heavy' without taste-holders
Shipping on intuition without a few strong-taste decision-makers leads to inconsistent, low-quality output.
Is it for you?
Best for
Teams with strong product intuition and long-term monitoring who face frequent neutral test results
Not ideal for
Teams lacking clear taste-holders or long-term measurement, where intuition can't be trusted or validated
From the transcript
“if the intuition is right that this probably is helping Merchants why do we start with that the original control was better if it's neutral…”
“it's why we tend to ship neutral it's like it could be positive and so let's like Let It Go if we have good intuition…”
From the episode
Breaking the rules of growth: Why Shopify bans KPIs, optimizes for churn, prioritizes intuition, and builds toward a 100-year vision
Archie Abrams (VP Product, Head of Growth at Shopify)