Cohort-and-Control Data Reading
Never trust pre/post dashboards — read behavior by cohort and judge changes by variant-vs-control so the macro can't fool you.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 85%
Aggregate dashboards blend users across many years who behave completely differently, so raw pre/post comparisons produce bad decisions. Two corrections: read development at the cohort level, and evaluate any change as an experiment (variant vs control) rather than as a before/after. This isolates your product's effect from macro noise — a conversion drop caused by a Bitcoin crash looks like a product failure in pre/post, but the experiment shows the variant still beating control. Building this experimentation culture is also what makes product a 'science' and gives PMs a defensible discipline.
Origin
Mayur Kamat's practice, introduced as a 'foreign concept' at Binance and central to the experimentation cultures he builds; tooling via Statsig (founded by ex-Facebook experimentation lead Vijay).
Core principles
- 01Aggregate data mixes cohorts with different behavior — it misleads by default
- 02Judge changes by variant-vs-control, not before-vs-after
- 03Cohort-level reading yields better decisions than whole-population dashboards
- 04There's noise between when you start tracking and when you act — the world moves in between
- 05Experimentation turns product from opinion into science, giving PMs a real discipline
How to run it
- 1
Segment by cohort, not aggregate
Instead of one blended dashboard spanning users acquired over 6 months to 20 years, look at how specific cohorts develop, since different cohorts behave differently.
Watch out Even cohort data has lag noise — the world can change between when you start tracking and when you decide.
- 2
Frame every change as variant vs control
Evaluate a product change by comparing the treated variant against a control group, not the metric before versus after the launch.
Pro tip Watch p-values and time-to-statistical-significance in a tool like Statsig so you know when the signal is real.
Watch out Pre/post can show a decline even when your change is winning — you'll wrongly conclude the product got worse.
- 3
Attribute movement to product vs macro
When a metric moves, separate your product's effect from external forces (a crypto crash, seasonality) — the experiment does this automatically, pre/post cannot.
Watch out Some domains (EU pricing, legal, compliance) can't be experimented on; there you need deeper cohort reasoning up front.
- 4
Build the culture, incentives, and tools
On taking a new role, install the right culture, incentives, and experimentation tooling first, since that democratizes performance measurement for PMs.
Pro tip Once it lands, running an experiment and watching a metric move becomes a natural dopamine hit that self-reinforces the culture.
Watch out It takes a long time to move a team into this mode — expect it to be slow before it's fun.
In the wild
At Binance, product conversion fell even though the product was doing well, because Bitcoin had crashed and nobody wanted to open an exchange account. A pre/post reading would blame the product; measured as an experiment, the variant still beat control while overall conversion was down.
→ The experiment correctly separated product performance from a macro crash, preventing a wrong 'we broke the product' conclusion.
Common mistakes
Reading aggregate dashboards
A blended view of users spanning many years mixes cohorts that behave differently and leads to completely bad decisions.
Pre/post comparison
Comparing a metric before and after a change conflates your effect with the macro, so you take credit or blame that belongs to external forces.
Is it for you?
Best for
Product managers and growth teams with enough traffic to run controlled experiments and segment cohorts
Not ideal for
Non-testable, irreversible, or heavily-regulated decisions (EU pricing, legal, compliance) where you can't get a clean control
From the transcript
“a lot of companies really look at data without looking at cohorts you make completely bad decisions”
“if you look at a cohort level development of certain users, you generally end up making better decisions”
“If you just measure pre and post you would think that you have done something wrong in the product. If you measure it as an…”
“the moment you build experimentation you now made it scientific”
From the episode
Unconventional product lessons from Binance, N26, Google, more
Mayur Kamat (CPO at N26, ex-Binance Head of Product)