The Data-Driven Red Flag
When someone brags about being 'data-driven,' suspect weak product judgment, not strong analytics.
- Difficulty
- Easy
- Time to result
- ~ongoing to results
- Steps
- 3
- Confidence
- 88%
Crowley's contrarian mental model: loud enthusiasm for being 'data-driven,' especially from executives who make every decision via dashboard, is often a red flag that a team over-weights quantitative data and neglects qualitative understanding of the humans using the product. Dashboards tell you what is happening but never why, and without the why you can't generate good insights about what to do next.
Origin
A contrarian opinion offered by Maggie Crowley; she illustrates the counter-instinct with a story about Adam Medros (VP Product at Tripadvisor) cutting off a metrics-heavy justification with 'if it's obviously better, just do it.'
Core principles
- 01Dashboards show what; only talking to users shows why
- 02Quantitative data at the expense of qualitative is a judgment failure
- 03For most products, 10 user conversations beat any dashboard for insight
- 04You still must instrument your product; the error is forgetting the why
- 05Sometimes the obviously/logically better thing needs no experiment
How to run it
- 1
Treat loud 'data-driven' claims as a signal to probe
When a person or team emphasizes making all decisions with dashboards, ask whether they also do direct user research and understand why users behave as they do.
Watch out It isn't always a red flag; high-volume businesses where experiments are cheap and easy can legitimately run this way.
- 2
Pair quantitative with qualitative
Instrument the product to know if it works at scale, but talk to ~10 users to understand why things are happening before deciding what to build next.
Pro tip For most PMs, most jobs, and most products, 10 user conversations yield more and better insight than any dashboard.
Watch out Numbers won't tell you why; without the why you can't come up with good insights about what to do next.
- 3
Let obvious wins skip the analysis
When something is logically or obviously better, use good judgment and just do it instead of manufacturing metrics to justify it.
Pro tip Reserve this card for cases that are genuinely, obviously good; don't use it to dodge all rigor.
In the wild
In a product-review meeting, a PM was justifying a project with numbers when VP Product Adam Medros said, in effect, this is obviously better, stop with the numbers, use your good judgment and move on.
→ Crowley internalized it as permission to act on clearly-better ideas without exhaustive quantitative justification.
Common mistakes
Managing via dashboard
Over-emphasizing quantitative data means never really understanding the humans using the product, what they need, or why things happen, so insights and decisions suffer.
Is it for you?
Best for
PMs and leaders evaluating a team's culture, or individuals deciding how much weight to put on analytics versus user research.
Not ideal for
High-volume products where experiments are cheap and quantitative optimization genuinely works well.
From the transcript
“people who are really excited about being data driven to me that is often times a red flag for their product thinking”
“that says that the team is overemphasizing quantitative data at the expense of qualitative data and they're not using good judgment”
“most PMS most jobs most products you're going to be better off talking to 10 users and you'll get more and better insights out of…”
“it won't tell you why anything is happening if you don't understand why it's happening I don't think you can come up with good insights…”
“if it's obviously better if it's logically better use your good judgment to do that thing”
From the episode
Mastering product strategy and growing as a PM
Maggie Crowley (Toast, Drift, Tripadvisor)