Principles-First Product Design
Set non-negotiable principles up front — voice of the people, no override button, full transparency — and never bend them.
- Difficulty
- Advanced
- Time to result
- ~ongoing to results
- Steps
- 4
- Confidence
- 90%
Define a small set of principles that seem almost crazy at first, then refuse to compromise them even under pressure. For Community Notes the core three were: the product is the voice of the people (not the company's), there is no button to override a note's status, and everything — code, data, ratings — is open and auditable. Coleman argues this rigidity is exactly why the product is trusted and works.
Origin
Keith Coleman and the Community Notes team, established on day one and revisited constantly. Radical transparency was validated externally when researchers like Vitalik Buterin independently downloaded the code and data to verify the algorithm does what it claims.
Core principles
- 01If a note is bad enough that you want an override button, the system is broken — fix the system, not the note.
- 02Trust comes from an open, transparent process, not from a company's assurances.
- 03Objective, minimal entry criteria (e.g. a verified phone) beat curated editorial gatekeeping.
- 04Principles cost real engineering effort — pay it, because breaking them destroys trust.
How to run it
- 1
Declare the product the voice of the people
Decide that outputs represent the public, not the company. Concretely: build no mechanism for the company to change a note's status once the crowd has rated it.
Watch out This unsettles stakeholders who want a kill switch — hold firm: if it doesn't work well enough to have no override, it doesn't work.
- 2
Open the gate to all of humanity
Don't appoint journalists or a curated elite. Let anyone in with basic objective criteria and random selection, so the system feels fair, open, and trustable.
Pro tip A verified phone number reduces bots without becoming an editorial filter.
Watch out Curating who participates reintroduces exactly the 'trust the elites' problem the product exists to escape.
- 3
Make it fully transparent and reproducible
Publish the scoring code to GitHub and release the ratings data daily so anyone can replicate, audit, and critique the whole service — and feed good observations back into the code.
Watch out Open-sourcing a real large-scale service is hard: you must architect it so it runs as a downloadable script, which forces unusual engineering choices you can't retrofit easily later.
- 4
Return to the principles on every change
Revisit the principles constantly; every code or scoring update ships openly so no change quietly violates them.
Watch out Bending even one principle (any black-box element) makes the whole product much harder to trust.
In the wild
When the team first proposed that a crowd-rated note would show and the company could not change its status, stakeholders were unsettled — 'wait, the company can't take it down?' The team held the line, framing any bad note as a system-design problem.
→ The no-override principle became core to why the product is trusted across the political spectrum.
Because the Python scoring code and data are public, outsiders can run the whole algorithm on one machine (about 500GB RAM, roughly a day). Vitalik Buterin published a blog post exploring whether the algorithm really does what it claims.
→ Enough trusted third parties have replicated it that most users know someone credible who has verified it.
Common mistakes
Keeping a company override 'just in case'
Any ability for the company to change a note's status reintroduces the black-box, trust-us dynamic that undermines the product's legitimacy.
Open-sourcing only for show
Publishing code that can't actually be run on the real data isn't meaningful transparency; the team took deliberate architectural hits so outsiders could truly replicate and audit it.
Is it for you?
Best for
Founders building trust-critical civic or moderation products where legitimacy depends on not being the sole arbiter.
Not ideal for
Proprietary products where competitive secrecy or a required human-in-the-loop kill switch outweighs open legitimacy.
From the transcript
“this thing is going to be the voice of the people it's going to represent the voice of people it's not going to represent the…”
“first of all there is no we don't have a button that will change the status of the note of a note”
“if there's a problem with a note that's so bad you want to do something about it it's a problem with the system”
“the code that decides what note show has to be out in the open all the data and ratings that that make it happen have…”
From the episode
An inside look at X’s Community Notes
Keith Coleman (VP of Product) and Jay Baxter (ML Lead)