The Overall Evaluation Criterion (OEC)
Define one success metric with countervailing guardrails that predicts long-term lifetime value
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 95%
The OEC forces you to answer 'what are we optimizing for?' before running any experiment. The trap is optimizing a single easy metric like revenue, which you can game with tactics that hurt users. A good OEC pairs the goal metric with countervailing metrics and is chosen so it is causally predictive of the user's lifetime value.
Origin
Developed and popularized by Ronny Kohavi across his work at Amazon, Microsoft/Bing, and Airbnb, and codified in his book 'Trustworthy Online Controlled Experiments.'
Core principles
- 01A single naive metric (revenue) can always be improved by user-hostile tactics in the short term
- 02Every goal metric needs a countervailing (guardrail) metric that protects the user experience
- 03The OEC must be causally predictive of long-term lifetime value, not just short-term gain
- 04Framing as a constrained optimization ('increase revenue within a fixed pixel/real-estate budget') removes the perverse incentive
- 05If a team cannot agree on the direction of their chosen metric, they do not yet have a valid OEC
How to run it
- 1
State what you are truly optimizing for
Name the primary business goal, then reject the lazy single-metric version of it (e.g. 'more revenue', 'more time on site') because it can be gamed.
Pro tip Ask the team out loud whether a move in the metric is good or bad; if half the room disagrees, you have no OEC yet.
Watch out 'Time on site' is ambiguous — on a support site more time can mean users are failing to find help.
- 2
Add countervailing metrics
Attach guardrail metrics that capture the user-experience cost of pursuing the goal — churn, time-to-successful-result, percentage of successful sessions.
Watch out A goal metric with no counter-metric produces spam-like behavior (see the Amazon email example).
- 3
Tie the OEC to lifetime value
Choose or model the metric so it is causally predictive of long-term LTV; if the true signal (e.g. a listing's stay-rating three months later) is not yet available, build a training set to predict it.
Pro tip Model longer-term surrogates from historical data so you don't have to wait months for ground truth.
Watch out Optimizing pure conversion ignores whether the user is actually happy months later.
- 4
Reframe as constrained optimization where possible
Instead of 'make more money' (unbounded), fix a budget the team must stay within — e.g. a set number of vertical ad pixels — and reward making more within that budget.
Pro tip Constraints turn a badly-defined goal into a well-posed problem the team can safely optimize.
In the wild
Rather than letting any page plaster more ads to lift short-term revenue, Kohavi's team mapped how each additional ad increased churn and time-to-success, then gave teams a fixed vertical-pixel budget and rewarded earning more within it.
→ Revenue could grow without degrading the user experience, aligning short-term gains with long-term growth.
The email team was credited whenever a user purchased after an email, a metric with no counter-metric, so they ramped up email volume and 'claimed to make more money.' The team modeled the LTV lost per unsubscribe (a few dollars) and folded it into the OEC.
→ More than half the campaigns turned out to be net-negative, and the insight spawned a per-campaign unsubscribe default that preserved future LTV.
Common mistakes
Optimizing a single gameable metric
Choosing revenue or conversion alone lets teams win the metric with tactics (more ads, more emails) that quietly destroy user experience and long-term value.
Picking an ambiguous metric everyone interprets differently
If the team can't agree whether up is good or bad (like time-on-site for a support site), the OEC gives no real guidance and experiments become uninterpretable.
Is it for you?
Best for
Product and growth leaders standing up an experimentation program who need to decide what 'winning' means before running tests
Not ideal for
Domains where you lack enough traffic to measure the countervailing metrics reliably, or one-off irreversible decisions
From the transcript
“the question is what are you optimizing for and it's a much harder question that people think because it's very easy to say we're going…”
“to me the key here the key word is lifetime value which is you have to define the oec such that it is causally predictive…”
“the more emails you send the more money you're going to credit the theme and so that led to spam”
From the episode
The ultimate guide to A/B testing
Ronny Kohavi (Airbnb, Microsoft, Amazon)