Measurements Are Not Insights (Instrumentation for Analytics)
Turn raw metrics into segmented, causal insights you actually act on.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 90%
Crystal Widjaja's diagnosis of why most analytics efforts fail: teams treat metric-gathering as entertainment and confuse observations with insights. The fix is instrumenting event properties so you can segment who does what behavior, form hypotheses, test them, and derive causal insight that changes what you do.
Origin
From Crystal Widjaja (former CPO at Gojek/Kumu), drawing on her Reforge post 'Why most analytics efforts fail'.
Core principles
- 01Real news is information that changes what you do; if you don't change behavior, you're just consuming entertainment.
- 02A measurement is an observation (a data point); an insight has context that lets you act.
- 03Insight requires answering the 'why' behind a behavior.
- 04Instrument properties into events so behavior can be segmented and hypotheses tested.
How to run it
- 1
Stop treating metrics as entertainment
Recognise that tracking data only to see if an OKR goes up or down, and finding numbers 'interesting' without acting, is entertainment, not analysis.
Pro tip Ask: does this change what I do in the real world? If not, it's not real news.
Watch out Novelty without action is the core failure mode.
- 2
Separate observations from insights
Note that a fact like 'power users do 4x more bookings' is an observation from your transactional database — true but not actionable because it lacks context.
Watch out An observation dressed up as a finding leads to no behavior change.
- 3
Instrument properties into events
Add properties to events so you can segment who is doing which behavior under which circumstances (e.g. power users on a high-GMV basket with free shipping).
Pro tip The insight is 'instrumenting properties into an event so that you can segment who is doing what behavior'.
- 4
Form and test hypotheses for causal insight
Make hypotheses on the segmented observation, test them, and derive a causal representation of whether the hypothesis was right — which then tells you how to change action (e.g. marketing spend).
Pro tip A segmented, causal insight like 'power users convert on free shipping only on high-GMV baskets' directly reallocates marketing spend.
In the wild
Seeing your girlfriend with an unknown man is an observed fact; the hypothesis is she's cheating, but the actual fact is it's her cousin, so the real insight is 'I am paranoid and need to change my behavior'.
→ Illustrates that an observation plus context and a 'why' produces an insight that changes your own behavior.
'Power users use a voucher' is an observation; the insight is that power users on a high-GMV basket are more likely to use a free-shipping discount while non-power users won't convert any better.
→ This reallocates marketing spend to power users on high-GMV baskets and stops wasting discounts on non-power users.
Common mistakes
Consuming metrics like Twitter for entertainment
Gathering data to feel informed without changing any decision produces novelty, not value; you must act on it for it to count as real news.
Treating an observation as an insight
A raw data point from your database has no context and doesn't tell you how to act, so shipping it as a 'finding' misleads teams into inaction.
Is it for you?
Best for
Growth and product teams whose dashboards are full of numbers but produce no decisions.
Not ideal for
Very early products with too little traffic to segment behavior meaningfully.
From the transcript
“do not treat metric Gathering as entertainment like it's not there for you to be like oh that's interesting how novel and then not act…”
“a measurement would be an observation it's a data point in your database”
“the Insight is instrumenting properties into an event so that you can segment who is doing what behavior and make some hypotheses on that observation…”
From the episode
Countdown of the top 10 episodes of the year