Data Is a Compass, Not a GPS
Data disproves the ridiculous — it rarely hands you the answer, so validate findings before you trust them
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 90%
Clowes, a self-described former 'data-driven' PM, reframes data's role: it is a compass, not a GPS. Data rarely gives you the answer; it tells you whether what you just said is ridiculous or whether there might be something there. Because being data-obsessed is easy to overdo, the discipline is to trust intuition first on wildly counterintuitive results, then validate any exciting finding with a rigorous before/after/upstream check before presenting it with authority.
Origin
Clowes' own evolution across running data teams at multiple companies; he created Reforge's 'Data for Product Managers' course.
Core principles
- 01Data is more like a compass than a GPS — it disproves, it rarely answers
- 02A wildly counterintuitive result is most likely just wrong (Occam's razor)
- 03Intuition is itself compressed data, so believe it first and then prove it out
- 04Presenting authoritative-but-holey analysis loses you more credibility than presenting none
- 05Rigor means checking what's upstream, downstream, and one level up before trusting a finding
How to run it
- 1
Sanity-check against intuition
When a result feels insanely wrong, believe your intuition first and go prove yourself right rather than accepting the number at face value. The most likely explanation for a wildly unintuitive result is an error.
Pro tip Occasionally the counterintuitive result is real gold — those are the moments that make it worth it — but earn that conclusion through diligence.
Watch out Presenting a 'dumb' analysis that gets shot down live costs you a whole bunch of credibility — better to show up with none than with one full of holes.
- 2
Check upstream and downstream
For any exciting finding, ask what happened immediately before it and does that look normal, and what happened after. A win with no sensible before/after is suspect.
Pro tip Go 'a click to the left and a click to the right' — before and after — to see if the effect is real.
Watch out An intervention that only applies to 2% of the inbound stream is meaningless or a random aberration, even if the lift looks huge.
- 3
Go one click up to the real goal
Step up a level to the outcome that actually matters. A retained cohort that churns next week, or retains but at lower ASP, may not advance the true goal of happy customers paying you money.
Pro tip If the finding survives before, after, and one-click-up, you have a compelling story people want to hear.
Watch out Chasing a metric that doesn't ladder to the real business goal is 'fighting false gold.'
In the wild
Clowes walks through an onboarding-to-second-week-retention 'win': go upstream and it only applies to 2% of the onboarding stream (meaningless); go downstream and they all churn in week three (pointless); step back and their average ASP is lower, missing the revenue goal. Only if the effect survives the click-left, click-right, and click-up checks is it real.
→ A rigorous check distinguishes a genuinely compelling result from a random aberration or false gold.
Common mistakes
Treating data as the answer
Using data as a GPS makes you 'always wrong or slow or both' — data mostly tells you if a claim is ridiculous, not what to do, so over-relying on it is both error-prone and slow.
Presenting analysis you can drive a truck through
Authoritative presentation of a hole-ridden analysis gets you shot down live and forfeits credibility; it would have been better not to show up with an analysis than to show up with a dumb one.
Is it for you?
Best for
Data-driven PMs and growth teams evaluating experiment results before acting on or presenting them
Not ideal for
Genuinely well-powered, clean quantitative decisions where the data legitimately is the answer and second-guessing adds only delay
From the transcript
“data is more like a compass than a GPS”
“something that is insanely not intuitive is that it's just wrong”
“if you go upstream and you find out that this intervention only applies to 2% of the inbound onboarding stream it's meaningless”
“it would be better not to show up with an analysis that isn't clear then it would be to show up with an analysis that's…”
“really be rigorous about it to to avoid F fighting false gold”
From the episode
Why great AI products are all about the data
Shaun Clowes (CPO Confluent, ex-Salesforce, Atlassian)