Prove-It-At-Every-Step Rollout
Never take the next expansion step until the data from the last one has already convinced you.
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 90%
A discipline for shipping a radical, low-trust idea: make the product earn the right to advance at each stage, gathering the data that convinces you before you ask anyone else to buy in. Each phase answers one existential question — do people want it, can people make it, will it work in the wild — and you only expand once that phase's data says yes.
Origin
Keith Coleman's approach to launching Birdwatch/Community Notes (2020–2022), used to navigate skepticism from Twitter's existing trust-and-safety apparatus and to survive four CEOs.
Core principles
- 01Prove it to yourself with data before you try to convince others.
- 02Each stage should retire the single biggest remaining risk.
- 03Hold such a high quality bar yourself that you rarely propose anything unwise.
- 04Working proof beats research papers, strategy docs, and debate.
How to run it
- 1
Test the concept with mockups
Show static mockups depicting the idea to people across the political spectrum to learn whether they'd even value it — including when it criticizes their own side.
- 2
Test feasibility with paid strangers
Run an internal pilot through an Amazon Mechanical Turk-style test to see whether ordinary people can produce output of the needed quality.
Pro tip You don't need every output to be good — you need proof that some people can produce gold.
- 3
Run a small public pilot
Release to a small real-world cohort (Community Notes started at ~1,000 contributors) to see what actually shows up under real incentives.
Watch out Prepare to set expectations — the team seriously mocked up a 'dumpster fire' GIF to warn users output might be rough (ultimately cut to keep the page focused).
- 4
Expand only on convincing data
At each expansion — more users, new countries, US-wide — carry the data that already convinced you it's the right move, and share it to bring stakeholders along.
Pro tip Because you held a high bar, you rarely propose anything unwise, which builds credibility with skeptics.
Watch out Don't skip stages to move faster; the data trail is what earns trust from trust-and-safety teams and successive leaders.
In the wild
In 2020 the team ran unmoderated user tests of a figma prototype. A participant leaked it to an NBC reporter; the ensuing chatter reached Elon Musk — then just a Twitter user — who replied 'definitely worth trying IMO' two years before he acquired the company.
→ Early, consistent high-profile support that later helped the product survive the ownership change.
With just 1,000 contributors, notes were a mixed bag but clearly contained informative, neutral notes on hard topics.
→ Proved the real task was sifting the gold from the rest — validating the model before any wider launch.
Common mistakes
Selling the vision before you have proof
Pitching a radical idea with docs and strategy alone invites 'no way this works'; leading with a working, self-proven product converts skeptics far more reliably.
Expanding on hope instead of data
Jumping to a bigger audience without the data that convinced you risks a public failure that permanently damages trust in a trust-dependent product.
Is it for you?
Best for
Founders and product leaders launching a counterintuitive, trust-sensitive product against internal skepticism.
Not ideal for
Commodity features where speed-to-market matters more than trust, and a staged proof burden just slows you down.
From the transcript
“we were very disciplined I guess guest you could say about having the product prove itself at at every given point”
“anytime that we were proposing doing something with the product like running some research test or running the pilot or expanding the pilot we always…”
“there was gold in there and from the very early days with just a thousand contributors it was obvious that that people could write notes”
From the episode
An inside look at X’s Community Notes
Keith Coleman (VP of Product) and Jay Baxter (ML Lead)