Maximize the Treatment Effect to Fail Conclusively
In low-sample B2B tests, throw every tactic at a hypothesis at once so a failure kills the idea for good instead of resurfacing for years.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 90%
An experiment can only succeed with a large sample size or a large treatment effect. B2B rarely offers large N, so Batchu maximizes the treatment: for a given hypothesis, deploy all plausible tactics and resources at once. If the maxed-out version fails, the idea is conclusively dead and won't be re-tried every time a new executive has the same intuition. If it works, you then strip tactics back to optimize cost.
Origin
Sri Batchu's experimentation philosophy applied across Ramp, Instacart, and Opendoor; he frames failure as 'not learning' rather than not driving revenue.
Core principles
- 01Celebrate failure only when it produces learning
- 02An experiment succeeds via large N or large treatment effect; pick the lever you can control
- 03B2B lacks large N, so maximize the treatment instead
- 04A conclusive failure prevents the same idea from being re-litigated for years
- 05Reserve this rigor for expensive, slow-to-plan tests; cheap fast tests (email, web copy) can just be re-run
How to run it
- 1
State the hypothesis at the strategy level
Frame what you are actually testing as a strategy (e.g. 'this way of pursuing customers converts'), not a single tactic (e.g. 'this one email subject line works').
- 2
Load in every plausible tactic and resource
Deploy all the tactics and resources you believe could move the needle simultaneously to maximize the treatment effect, since you can cost-rationalize later if it works.
Pro tip You can always add back cost-efficiency after you know the ceiling; you can rarely recover a hypothesis you killed with a half-hearted test.
Watch out You lose attribution: if the maxed-out test works you won't know which individual element drove it.
- 3
Read the result as a verdict on the hypothesis
If the best, most expensive version fails, conclude the hypothesis is wrong and stop spending time on it. If it works, run a follow-up with half the tactics to learn what actually mattered.
Watch out Only worth this effort for cross-functional, larger-scale, expensive tests; for cheap fast surfaces just iterate quickly instead.
In the wild
ABM is commonly tried halfway - a few touches, control group doesn't convert, team concludes it doesn't work, then a new go-to-market executive arrives and re-tries it. At Ramp, Batchu instead maximized both the number of target customers and the number and types of ways they touched them, designing the test to be conclusive.
→ A definitive read on whether ABM works for Ramp, avoiding the endless re-litigation cycle that plagues most companies.
Common mistakes
Running a half-hearted test of an expensive strategy
Testing only a few tactics leaves the hypothesis inconclusive, so the same idea gets re-tried every time a new leader has the same intuition, wasting years.
Treating no-revenue as the definition of failure
Failure should be defined as not learning; a test that produces a clear conclusion is a success even if it drove no revenue.
Is it for you?
Best for
Growth teams running expensive, slow, cross-functional B2B experiments where sample sizes are small
Not ideal for
Cheap, fast, high-volume tests (email tweaks, web copy) where iterating on isolated variables is better for attribution
From the transcript
“there's only two two ways to you know make a an experiment successful uh either you have a very large n uh or you have…”
“to counteract that I recommend people like just trying to maximize the treatment effect which is like if you have a hypothesis that you're testing…”
“for me failure is not that you didn't drive Revenue failure is not learning”
From the episode
Lessons from scaling Ramp
Sri Batchu (Ramp, Instacart, Opendoor)