Evaluative User Research (Hunt for Reasons They Won't Use It)
Test prototypes to find every reason people won't use it — and never let the researcher go alone
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 93%
Cagan attacks the most common misuse of user research: showing a prototype until enough users say they like it, then building it. Real research is mostly evaluative — you are hunting for the reasons users will not use your product, not collecting approval. He pairs this with a hard operational rule: if the product manager and designer cannot attend the test, cancel it, because a research report delivered second-hand is a report that gets ignored.
Origin
Marty Cagan / SVPG. The 'find all the reasons they won't use your product' formulation Cagan attributes to Elon Musk; the generative-vs-evaluative distinction comes from the user-research discipline itself. The related idea that research should refine your mental model of the user (rather than dictate what to build) he credits to a Collison brother at Stripe.
Core principles
- 01Most of the value in user research is evaluative — the reasons they won't use it.
- 02Collecting 'they liked it' signal fools nobody and no smart leader.
- 03Research builds and refines your mental model of the user; the model informs what you build.
- 04A report the team wasn't present for is a report that gets ignored.
How to run it
- 1
Set the goal as disconfirmation
Before the session, write down what you are trying to find out — specifically, the reasons this person will not use the product. If your success criterion is 'enough users like it', reset it.
Watch out Prototype tests that count likes produce false confidence and cannot survive scrutiny from smart leaders.
- 2
Gate the session on PM and designer attendance
If the product manager and designer cannot be there for the test, cancel the test. Their presence is what converts observations into team decisions.
Pro tip Get engineers into sessions too — Cagan and Lenny both name this as where the magic happens.
Watch out The failure mode is not researcher incompetence — researchers know their craft. It is that the resulting report is too often ignored.
- 3
Test broadly, not just with end users
Solution discovery means testing constantly — with users, with customers, with stakeholders, and with your own developers. Each surfaces a different class of reason the solution will fail.
Pro tip Stakeholder prototype tests are how you discharge the viability risk before you build.
- 4
Feed the findings into your model of the user
Treat each round of research as refining a durable mental model of the user and customer — not as a lookup service that answers 'what should we build'. Your model, continually corrected, is what informs what you build.
In the wild
Cagan describes a team he spoke to that morning: they test their prototypes and, when enough users say how much they like it, they build it. He told them flatly that this is not why we do user research, and that it will not fool any smart leader.
→ The team's research was generating approval, not the disconfirming evidence that would have told them what to fix — or whether to build at all.
Lenny recounts flying to Paris with Airbnb's head of design and their researcher for host focus groups. The researcher insisted the PM and designer come rather than receiving her insights second-hand.
→ The team saw the findings first-hand and acted on them, illustrating Cagan's rule that the PM and designer must be present or the test should be cancelled.
Common mistakes
Using research as an approval mechanism
Testing until enough people say they like the prototype is confirmation seeking. Cagan: 'we're not just trying to find out if they like it — in fact it's just the opposite.'
Outsourcing research to a report
When researchers run studies alone and hand back a document, the document is too often ignored. The insight has to enter the team through the people who will make the decisions.
Is it for you?
Best for
Product trios running prototype tests during solution discovery, and research teams whose findings keep getting shelved
Not ideal for
Early generative exploration in a brand-new domain, where you genuinely are looking for unmet needs rather than evaluating a candidate solution
From the transcript
“yeah what they do with user research is they they test their prototypes and is and and when they get enough of users saying how…”
“we're testing constantly we're not just trying to find out you know if they like it in fact it's just the opposite”
“it's weird when we're doing user research we're finding all the reasons they don't like it in fact that's an elon musk quote is when…”
“i hate it when the user researchers go off and do the research themselves and bring back the report”
“to user researchers at the companies that i coach if the product manager and the designer are not available to be there during your product…”
From the episode
The nature of product
Marty Cagan, Silicon Valley Product Group