Experiment Only When You Need To
Treat A/B tests as risk mitigation, not as the default proof of impact
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 92%
Most growth teams A/B test reflexively because experiments are how individuals get credit for impact. Isford argues there are only two legitimate reasons to run an experiment — precise metric measurement, and risk mitigation on dramatic changes — and that only the second is usually worth its cost. The alternative is to push rigor upstream into the product development process (customer conversations, mocks, foundational analysis) until you have enough conviction to ship to everyone.
Origin
Developed by Lauryn Isford as Head of Growth at Airtable, in reaction to the experiment-everything culture she experienced doing consumer growth at scale at Facebook/Meta (user growth, internet.org, growth in India).
Core principles
- 01There are exactly two reasons to experiment: measurement precision, and risk mitigation.
- 02Measurement precision is usually only worth it for the performance review, not for the business.
- 03Experiments have a real cost: engineer, analyst and PM time that could go to roadmapping, foundational analysis, or shipping.
- 04You shipped it or you didn't — the rigor of your process was either there or it wasn't, regardless of whether an A/B test measured it.
- 05Escaping the experiment trap requires top-down culture change, because it changes how people are rewarded.
How to run it
- 1
Classify why you want to test
Before writing a test plan, name which of the two reasons applies: are you trying to measure the impact precisely, or are you mitigating the risk of a dramatic change going badly in production?
Pro tip If the honest answer is 'so I can say I moved activation 7%', that is a performance-review reason, not a business reason.
- 2
Price the precision
Ask whether knowing 6% vs 7% would actually change any decision you make. If the decision is the same either way, the precision is not worth the engineering, analytics and PM time the experiment consumes.
Watch out Experiment cost is invisible on the roadmap — it shows up as foundational work that never got done.
- 3
Move the rigor upstream
Instead of buying conviction with a test, buy it with process: more time with customers, sharper articulation of the problem being solved, mocks in front of real users, and data work to confirm demand.
Pro tip Target enough conviction that you'd be comfortable if every customer saw the change tomorrow.
- 4
Ship it and use attribution to learn afterward
For changes with strong qualitative and data-backed evidence, roll out to 100% and read impact from top-line metrics plus post-hoc attribution analysis on the base.
Pro tip A change big enough to be visible in top-line numbers rarely needed a test to detect it.
Watch out Reserve full rollout for changes where the downside is bounded; keep the A/B test for genuinely risky, dramatic changes.
- 5
Rebuild how the team gets credit
Create non-experiment ways to motivate, reward, retain and develop talent — impact on customers via qualitative feedback, deals closed, deals not lost, and other indicators of doing right by the customer.
Watch out If pointing to a number is the only path to recognition, especially in engineering, the team will bias toward experimenting on everything and you will pay for it.
In the wild
Airtable Forms had a feature-parity gap — a form submitter could not request a copy of their own submission. The team validated the need through customer research and data work, built the feature (which required the submitter to create an Airtable account), and shipped it turned on for everyone with no A/B test, accepting that it might shift signup mix and activation rate composition.
→ The change had a large enough effect that the impact on signup metrics was visible at top-line, and attribution analysis let the team learn from the base after rollout — no experiment required.
Common mistakes
Experimenting as a substitute for conviction
Teams reach for an A/B test because they are unsure whether customers want the thing. That uncertainty should be resolved in discovery — customer conversations, mocks, data — not paid for with weeks of engineering and analysis time.
Using 'no experiment' as cover for sloppy building
Skipping the test is only legitimate when the upstream rigor is genuinely higher. It cannot become an excuse for imprecision about what customers need.
Letting the IC try to opt out alone
A single PM declining to run an experiment reads as dodging accountability. The norm has to be set top-down by the growth leader, alongside alternative ways of recognizing impact.
Is it for you?
Best for
Growth leaders and PMs at companies where experimentation has become the default and roadmap time is being eaten by measurement rather than shipping
Not ideal for
Genuinely risky, dramatic changes where a bad outcome in production would be costly, or very high-volume consumer surfaces where tests are cheap and small deltas compound
From the transcript
“a growth team wants to experiment and one of them is to understand more precisely the metric impact of what they're building and what they're…”
“it also is expensive to have folks on the ground be it Engineers analysts product managers spending time understanding the results of an experiment”
“generally my advice is to experiment when you need to and to primarily see it as a risk mitigation tactic when you're making dramatic changes…”
From the episode
Mastering onboarding
Lauryn Isford (Head of Growth at Airtable)