Fail Conclusively (Maximize the Treatment Effect)
Design experiments that kill an idea for good, so no future exec can resurrect it.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 95%
Sri Batchu (former Head of Growth at Ramp) argues that with a ~30% experiment success rate, the real cost isn't failure — it's inconclusive failure. A half-hearted test that doesn't move the needle teaches you nothing, so the same idea gets re-tried every time a new executive arrives with the same intuition. Since B2B teams rarely have the sample size to get statistical power, the lever you actually control is the treatment: throw every plausible tactic at the hypothesis at once, so a null result is a real verdict.
Origin
Sri Batchu's growth practice, developed at Ramp and applied to account-based marketing — a tactic he'd repeatedly watched companies half-test, abandon, and re-test on every go-to-market leadership change.
Core principles
- 01Failure isn't not driving revenue; failure is not learning.
- 02Only two things make an experiment conclusive: a very large n, or a very large treatment effect.
- 03B2B has no large n. Facebook gets stat-sig in two hours; a B2B company can take two years. So you buy power with treatment size.
- 04Test the strategy, not the tactic — the hypothesis is 'this whole approach works', not 'this email works'.
- 05You can always cost-rationalize after it works; you can never learn from a test that was too weak to speak.
- 06A conclusive no is a permanent asset — it ends the loop of re-litigation.
How to run it
- 1
Write the hypothesis at strategy altitude
State what you actually believe, e.g. 'a coordinated multi-touch approach to these target accounts drives conversion' — not 'this subject line lifts opens'. The altitude of the hypothesis determines the altitude of the learning.
Pro tip If a null result on your test wouldn't change what the company does next quarter, you've written the hypothesis too small.
- 2
Check which lever you have: n or treatment
Estimate whether you can realistically reach statistical significance on volume. In B2B/enterprise you almost never can. If n is out of reach, you must buy conclusiveness with treatment strength instead.
Pro tip Do this math before designing the test, not while agonizing over an ambiguous readout afterwards.
Watch out Running an underpowered test and then arguing about the result is how ideas become zombies.
- 3
Throw everything at it
Stack every tactic and resource you believe could move that needle into the single treatment: trigger, content, personalization, design, channels, human touches. Deliberately overspend. The point is to give the hypothesis its best possible shot.
Pro tip Cost is not a constraint at this stage — 'you can always cost rationalize later if it works'.
Watch out Accept the trade-off consciously: a maximal treatment cannot tell you WHICH lever worked. That's a second-round question, not a first-round one.
- 4
Read the verdict as binary
If the maximal, most expensive version of the idea fails, declare it dead and document it: we tried everything we could to test this hypothesis, it didn't work in its best version, it is not worth more time. That document is what protects you from the next exec's intuition.
Pro tip Circulate the kill memo widely. Its value is entirely in being findable two years later.
Watch out Do not soften a conclusive no into 'we should try it again with better execution' — that reopens the loop you just closed.
- 5
If it works, strip it back
A win on the maximal treatment means the strategy is real but the cost is unknown. Now run a second version with half the tactics — or only the ones you suspect carried the effect — and optimize down toward an efficient version over time.
Pro tip Ablate one component at a time on the winning stack; that's where the efficiency gains hide.
In the wild
ABM is repeatedly half-tested in enterprise software: a team picks high-priority accounts, tries three tactics, sees the control group hold, and concludes it doesn't work — until a new go-to-market exec arrives and makes them do it all over again. At Ramp, Batchu's team instead maximized both the number of target accounts and the number and types of touches used to demonstrate value, so the result would settle the question either way.
→ A test designed to be conclusive one way or the other, ending the repeated re-litigation cycle rather than feeding it.
The lazy version is to tweak the copy; if it fails, tweak different copy, forever. The conclusive version asks: what are ALL the things that could be wrong here — the trigger, the content, the personalization, the design — and changes all of them in one test, testing the real hypothesis: 'this touchpoint is wrong'.
→ A null result then indicts the touchpoint itself rather than one copy variant, which is a decision you can actually act on.
Common mistakes
Testing halfway and calling it evidence
Three tactics, a flat control, and a shrug. The idea isn't dead — it's dormant. It comes back with the next executive who has the same intuition, and the company pays the same cost again.
Treating failure as the absence of revenue
If you define failure as 'didn't drive revenue', you build a culture afraid to take risks. Define it as 'didn't learn' and a null result from a well-designed test counts as a win.
Using maximal treatment on micro-tests
This framework earns its cost on cross-functional, larger-scale bets. Applying it to a low-stakes copy tweak is overkill — there, a cheap iterative test is fine and being wrong costs nothing.
Is it for you?
Best for
B2B and enterprise growth leaders who lack the traffic for statistical power and keep getting asked to re-test the same strategic ideas by each new executive.
Not ideal for
Consumer products with huge n where a clean, isolated A/B test can attribute the effect precisely — and for cheap micro-optimizations where iterative testing costs almost nothing.
From the transcript
“for me failure is not that you didn't drive Revenue failure is not learning like so it's really important that you learn when you fail”
“we celebrate failure as long as you're learning and and you can only learn if you've designed the right test and you failed conclusively”
“there's only two two ways to you know make a an experiment successful uh either you have a very large n uh or you have…”
“just maximize the treatment effect and if with all of that it didn't work then you can say hey like we're not going to try…”
From the episode
Failure