Predict vs. Decide: Think in Differentials
Prediction finds correlations; decisions need causal differences. Optimize the lift, not the level.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 93%
A machine-learning model that predicts an absolute (who will be hired, who has high LTV) answers the wrong question. Decisions require the causal differential: how much MORE will happen because of the action you take. Johari reframes model use so data teams evaluate options by their incremental effect, not their ability to recreate the past.
Origin
Ramesh Johari, from directing data science at odesk and teaching causal inference at Stanford.
Core principles
- 01Prediction is inherently about correlation; decision-making is about causation
- 02The safe-looking move (serve the highest-predicted segment) is often not the value-maximizing move
- 03Evaluate two algorithms by future outcomes they cause, not by how well they recreate historical choices
- 04Correlation-is-not-causation is not an abstract slogan; it is the daily gap between predicting and deciding
How to run it
- 1
Restate the prediction as a decision
Whatever the model predicts, ask what decision it will drive and whether that decision maximizes net business value.
Watch out Don't rank by absolute predicted value just because it's defensible in a monthly report; defensible is not the same as optimal.
- 2
Switch from absolute to differential
Ask not 'what is their value?' but 'how much does the value change BECAUSE of my intervention?' Target the incremental lift.
Pro tip For promotions, target customers whose LTV rises most because they received the promo, not customers with the highest baseline LTV.
- 3
Evaluate options by caused future outcomes
Compare two ranking or matching algorithms by whether one causes more bookings/matches going forward, not by which better reproduces past behavior.
Pro tip Ask 'does Lenny's ranking algorithm lead to more bookings than Ramesh's?' — a causal, forward question — instead of 'which better predicts the last two years of bookings?'
- 4
Close the loop with downstream quality signals
Judge match quality by what happens next (re-hires, ratings) rather than by the model's original propensity score.
In the wild
In an executive class, Johari asked a CMO who should get the best promotions; the instinctive answer was 'the highest-LTV customers.' Johari's point: that's the predictable, blameless choice, but the right question is whose spend increases because of the promotion.
→ Illustrates that optimizing the absolute (LTV level) diverges from optimizing the decision (incremental LTV from the action).
Common mistakes
Ranking by absolute predicted value
Sending top promotions to already-high-LTV customers is defensible but may add little incremental value versus targeting high-uplift customers.
Grading algorithms on recreating the past
An algorithm that best reproduces historical choices may not be the one that causes the most future bookings or matches.
Is it for you?
Best for
Data scientists, PMs, and marketing leaders deciding where to point ML models in a marketplace or growth org
Not ideal for
Pure forecasting tasks where an accurate absolute prediction is genuinely the deliverable
From the transcript
“I'm not interested in their absolute LTV I'm ABS interested in the difference in their LTV because I sent them this promotion”
“prediction isn't the same thing as making decisions”
“the way I'm really going to evaluate those is in my market does one of those lead to better matches or more matches than the…”
“when we ask people to make decisions we're asking him to think about causation”
From the episode
Marketplace lessons from Uber, Airbnb, Bumble, and more
Ramesh Johari (Stanford professor, startup advisor)