Incrementality via Conversion Lift & Geo Tests
Prove paid actually caused growth by withholding ads from a control group and measuring the lift
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 88%
Attribution alone can't tell you whether a user would have converted anyway, so it is inherently biased. To measure true incrementality, run a conversion-lift or geo (GeoX) experiment: deliberately withhold your ad from a control group, accept the resulting spend hold-back and opportunity cost, and measure the causal lift. Pair this with a time-decay multi-touch attribution model for day-to-day credit assignment.
Origin
Davis's applied method, citing incrementality studies by Netflix and eBay (eBay's ~2012 study led it to cut most brand-keyword spend after finding users found them organically anyway).
Core principles
- 01Attribution does not answer whether a user would have converted without the ad — it is biased by construction
- 02Multi-touch, time-decay attribution is the fairest day-to-day model because users quickly forget where they first found you
- 03Platforms will partner on lift tests because their creative-heavy inventory is under-credited — but only above spend thresholds
- 04Incrementality results should feed an 'incrementality-adjusted factor' quantifying each channel's real efficiency, ideally by region
How to run it
- 1
Default to time-decay multi-touch attribution
For everyday credit, use a multi-touch model weighted by time decay: give some credit to the first touch but far more to the interactions near conversion, since users forget early brand exposure.
Watch out First-touch or last-touch alone misassigns credit; linear is acceptable but still ignores recency effects.
- 2
Run a conversion-lift or geo experiment for causality
To measure true incrementality, intentionally do not show your ad to a control group when you win an auction (Google serves the next bidder instead), or hold out entire geos, then compare converted outcomes.
Pro tip Ask your platform rep to set this up — Facebook, LinkedIn and Google all know their visual creative is under-credited and will partner on lift tests.
Watch out Below roughly $50k/month in a platform you won't have enough signal, so incrementality testing isn't worth the effort when starting out.
- 3
Account for spend hold-back and opportunity cost
Recognize the test's cost: the impression share and spend you deliberately withhold from the control group is real opportunity cost that must be budgeted.
- 4
Convert results into a per-channel adjustment factor
Feed the experiment results into an incrementality-adjusted factor that makes each channel's true efficiency precise, ideally broken out by region — but per-platform is fine if regional data isn't available.
In the wild
Davis cites eBay's ~2012 incrementality experiment, which found users were already going to find eBay organically, so brand-keyword spend was largely wasted — and competitors couldn't outrank eBay anyway because eBay's quality scores were so high.
→ eBay cut almost all of its brand-keyword spend after the study.
Common mistakes
Trusting attribution as proof of causation
Attribution can't say whether the conversion would have happened without the ad, so crediting paid for organic-inevitable conversions overstates ROI.
Running lift tests below the signal threshold
Under ~$50k/month per platform there isn't enough signal, so the test is far more work than the result is worth.
Is it for you?
Best for
Growth and finance teams at meaningful spend levels needing to prove paid's true causal contribution
Not ideal for
Early-stage advertisers under ~$50k/month per platform who lack the signal to run valid lift tests
From the transcript
“when you're running a conversion lift test you intentionally do not show your ad to some users when you win an auction and instead Google…”
“numerous companies like Netflix and eBay have done studies to get an understanding of the incrementality of paid advertising campaigns”
“if you're not spending more than would say 50k a month in the platform doing this is going to be a lot more work uh…”
From the episode
The ultimate guide to performance marketing
Timothy Davis (Shopify)