Learn by Making, Test the Extremes
Stop debating consequential decisions — run the experiment that shows the upper and lower bounds now.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 90%
When a decision feels consequential and could be debated for months, don't. Instead express the idea by making something — an experiment, prototype, doc, or mock — and specifically test the extremes to see the best- and worst-case bounds immediately. Conviction comes from data, not endless discussion.
Origin
Shackleton learned this from his YouTube boss Phil Farhi during the launch of skippable ads, a controversial and unproven bet nobody wanted. Farhi stopped the debate and said to test the extremes and start the experiment tomorrow.
Core principles
- 01Consequential, unproven decisions can be debated for months or years to no end
- 02Testing the extremes reveals the upper and lower bounds fast
- 03Expressing ideas by making beats pontificating or circular discussion
- 04This transcends level — ICs through CEOs should learn by putting ideas out
How to run it
- 1
Catch yourself in an endless debate
Notice when a decision is consequential, unproven, and could be argued indefinitely with no resolution in sight.
Pro tip The tell is months of back-and-forth with no new information entering.
Watch out Prolonged debate feels like rigor but often just delays learning.
- 2
Design experiments at the extremes
Build variants that probe the upper and lower bounds — e.g. a tiny skip button vs a giant skip button covering the whole player — to see how good or bad it could get.
Pro tip Extremes surface the real signal faster than a timid middle test.
Watch out A cautious single variant won't show you the bounds you need to form conviction.
- 3
Ship it and read the data
Start the experiment immediately (start tomorrow), and let a few weeks of data build conviction rather than argument.
Pro tip Substitute any artifact — prototype, doc, mock — when a live experiment isn't possible; just make something.
Watch out It's almost never as bad as you fear; the real question is how much better it can be.
In the wild
As a new PM, Shackleton was handed the unwanted, unproven skippable-ads project. Instead of debating, Phil Farhi had them launch experiments testing the extremes — a tiny skip button versus a giant one covering the whole player.
→ Within a few weeks the data gave them conviction they were onto something; a lesson Shackleton has since seen proven hundreds of times.
Common mistakes
Debating a consequential decision for months
Unproven, high-stakes ideas invite endless circular discussion that produces no new information; making something and testing extremes resolves it in weeks.
Testing only a cautious middle option
A timid single variant won't reveal the upper and lower bounds, so you never see how good or bad the idea really is.
Is it for you?
Best for
Product teams and leaders stuck debating a controversial, unproven bet
Not ideal for
Irreversible one-way-door decisions where a live extreme test would cause real harm
From the transcript
“just test the extremes start the experiment tomorrow”
“stop talking about it and like go make something”
From the episode
What sets great teams apart
Lane Shackleton (CPO of Coda)