Keith Coleman (VP of Product) and Jay Baxter (ML Lead)27 February 2025

An inside look at X’s Community Notes

6Frameworks

15Insights

Episode overview

Keith Coleman and Jay Baxter explain how X’s Community Notes lets contributors propose context and uses bridging-based agreement among people who typically disagree to decide which notes appear. They trace the product from its Birdwatch prototype and an internal algorithm competition through leadership changes at Twitter/X, emphasizing public participation, transparency, open code and data, and a deliberately high quality threshold. They also discuss measurable reductions in reposting after notes appear, the system’s response during the Israel–Hamas conflict, lean team practices, and possible LLM-assisted “Supernotes.”

Key ideas

Community Notes adds context to potentially misleading posts rather than acting as a conventional centralized fact-checking system.
Its scoring algorithm seeks agreement among contributors who have historically disagreed, instead of relying on a simple majority vote.
Every post is eligible for notes, while contributors earn writing privileges through ratings that align with broad, bridging-based consensus.
The team prioritizes note quality over coverage, showing only a small percentage of proposed notes and publishing the scoring code and rating data.
Displayed notes materially reduce likes and reposts through user behavior even though the system does not algorithmically demote noted posts.
A small, self-selected, fully focused team with a clear owner, senior sponsor, lightweight coordination, and milestone-driven goals built and scaled the product.
Pseudonymous participation increased willingness to contribute on controversial topics and cross partisan boundaries.
The team is exploring LLM assistance that generates and evaluates candidate notes while retaining human verification and community-based judgment.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Innovation4 steps

Bridging-Based Agreement

Surface content only when people who normally disagree agree — that's the signal of truth.

Read →

Productivity3 steps

Delete-First Lean Engineering

On a small team, deleting code beats writing it — audit for features whose maintenance cost exceeds their gain.

Read →

Strategy4 steps

Principles-First Product Design

Set non-negotiable principles up front — voice of the people, no override button, full transparency — and never bend them.

Read →

Strategy4 steps

Prove-It-At-Every-Step Rollout

Never take the next expansion step until the data from the last one has already convinced you.

Read →

Influence3 steps

Pseudonymity for Honest Contribution

Let contributors stay anonymous — people cross partisan lines far more readily when their name isn't attached.

Read →

Leadership5 steps

The Thermal Team

Run a startup inside the company: one owner, one senior sponsor, total focus, self-set pace.

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 17

Jay BaxterMentionsJ bter founding ml engineer and researcher for Community notes
“J bter founding ml engineer and researcher for Community notes”
▶ Watch at ~1:30approx.
Keith ColemanMentionsKeith Coleman product lead for Community notes
“Keith Coleman product lead for Community notes”
▶ Watch at ~1:30approx.
Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter
“today my guests are Keith Coleman product lead for Community notes and J bter founding ml engineer and researcher for Community notes”
▶ Watch at ~1:30approx.
Hillary ClintonMentionsmentioned
“it was the middle of the election between Donald Trump and Hillary Clinton”
▶ Watch at ~30:30approx.
Donald TrumpMentionsthere's about three pages in the book where I basically say being a bit Donald Trump has its virtues
“it was the middle of the election between Donald Trump and Hillary Clinton”
▶ Watch at ~30:30approx.
Kayvon BeykpourMentionsKavon was the Beloved and longest tenured head of product at Twitter and also GM of the consumer business at Twitter
“I went to my boss Kavon”
▶ Watch at ~33:00approx.
Jack DorseyMentionsThis is something that Jack, our CEO, talks about a lot.
“Jack had re relatively recently come back as CEO to turn the company around”
▶ Watch at ~34:00approx.
Elon MuskMentionsWhether you like him or not, Elon, this is similar to his um if you watch him, he has sets these really ambitious goals
“Elon came in acquired the company”
▶ Watch at ~35:00approx.
Blake SchollMentionsmentioned
“I think it's Blake scholes uh LinkedIn the other day who went from like director of coupons”
▶ Watch at ~39:00approx.
Kayvon BeykpourCoinedKavon was the Beloved and longest tenured head of product at Twitter and also GM of the consumer business at Twitter
“he had this idea like hey why don't we create this program called it thermal”
▶ Watch at ~41:00approx.
Elon MuskMentionsWhether you like him or not, Elon, this is similar to his um if you watch him, he has sets these really ambitious goals
“if we need need something or have a question about something I talked to Elon”
▶ Watch at ~42:30approx.
Keith ColemanCoinedKeith Coleman product lead for Community notes
“it started with just me and then we when we decided to build the thing it we we figured we needed about five”
▶ Watch at ~45:00approx.
Jay BaxterCoinedJ bter founding ml engineer and researcher for Community notes
“the actual first algorithm that I put into production was very focused on anti-manipulation”
▶ Watch at ~46:30approx.
Chris BailMentionsmentioned
“I think Keith had found some of Chris Bills work”
▶ Watch at ~49:00approx.
Vitalik ButerinUsesmentioned
“vitalic peteran had a blog post where he you know talks about uh his explorations”
▶ Watch at ~1:16:30approx.
Linda YaccarinoMentionsmentioned
“and then parag running Twitter and then Elon and then Linda taking over”
▶ Watch at ~1:32:30approx.
Parag AgrawalMentionsmentioned
“and then parag running Twitter and then Elon and then Linda taking over”
▶ Watch at ~1:32:30approx.

Resources · 25

Community NotesMentionsproduct · X
“Community notes is one of the most impactful and clever and also underappreciated products in the world right now”
▶ Watch at ~1:30approx.
GranolaRecommendssoftware · Granola
“you now get a year free of notion and superhuman and granola and linear and perplexity Pro”
▶ Watch at ~2:30approx.
NotionRecommendssoftware · Notion Labs
“you now get a year free of notion and superhuman and granola and linear and perplexity Pro”
▶ Watch at ~2:30approx.
SuperhumanRecommendssoftware · Superhuman
“you now get a year free of notion and superhuman and granola and linear and perplexity Pro”
▶ Watch at ~2:30approx.
Perplexity ProRecommendssoftware · Perplexity AI
“you now get a year free of notion and superhuman and granola and linear and perplexity Pro”
▶ Watch at ~2:30approx.
LinearRecommendssoftware · Linear
“you now get a year free of notion and superhuman and granola and linear and perplexity Pro”
▶ Watch at ~2:30approx.
Lenny's NewsletterCoinednewsletter · Lenny Rachitsky
“if you become a subscriber of my newsletter you now get a year free of notion and superhuman and granola and linear”
▶ Watch at ~2:30approx.
XUseswebsite · X Corp.
“someone on x can see a post if they think it's misleading they can propose a note”
▶ Watch at ~6:00approx.
MetaUsescompany
“meta recently as you all know uh decided to uh adopt this exact system for them”
▶ Watch at ~10:30approx.
UC BerkeleyMentionscompany
“I saw some stats recently from someone at UC Berkeley”
▶ Watch at ~13:30approx.
Matrix factorizationUsessoftware
“it uses this machine learning algorithm you know called Matrix factorization”
▶ Watch at ~17:30approx.
White HouseMentionscompany
“a note appeared on a White House tweet and the White House deleted the tweet and reissued an updated statement”
▶ Watch at ~25:00approx.
TwitterMentionswebsite · Twitter
“why I joined you was then Twitter um in 2016”
▶ Watch at ~30:00approx.
PeriscopeMentionssoftware · Periscope
“Kon was running Periscope with a bunch of video stuff”
▶ Watch at ~34:30approx.
BirdwatchMentionsproduct · Twitter
“it was originally called bird watch when um uh when we started the project”
▶ Watch at ~34:30approx.
FigmaUsessoftware · Figma
“the very first figma mockup I made depicting this thing was called Community notes”
▶ Watch at ~35:30approx.
LinkedInUseswebsite · LinkedIn
“I was just reading screenshot it I think it's Blake scholes uh LinkedIn the other day”
▶ Watch at ~39:00approx.
PageRankUsessoftware · Google
“the actual first algorithm that I put into production was very focused on anti-manipulation it was this kind of page rank variant”
▶ Watch at ~46:30approx.
PolisMentionssoftware
“other other projects like uh polus out there that that look for agreement among uh you know people who typically disagree”
▶ Watch at ~49:30approx.
Google DocsUsessoftware · Google
“there is a very long running dock um that has had to be uh chopped and purged because it was breaking Google Docs”
▶ Watch at ~50:30approx.
SpacesMentionsproduct · Twitter
“we had this product called spaces”
▶ Watch at ~59:30approx.
GitHubUsessoftware · GitHub
“whenever we update the code there's or update the scoring system there's an update in GitHub”
▶ Watch at ~1:14:30approx.
Israel–Hamas conflictMentionsevent
“the start of the Israel Hamas conflict uh in 2023 in October”
▶ Watch at ~1:18:30approx.
Arma 3Mentionssoftware · Bohemia Interactive
“there were people making fake like battle footage in the video game simulator Arma 3”
▶ Watch at ~1:19:30approx.
SupernotesMentionssoftware
“the supernotes of this idea that we can uh basically take existing notes input”
▶ Watch at ~1:39:00approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 2

Myth Buster08:00

You Don't Need Fact-Checker Labels to Bootstrap a Trust System

The prevailing assumption among ML engineers was that a system like this had to be closed-source and trained on ground-truth labels from professional fact-checkers, or manipulators would overrun it. Community Notes proved you can bootstrap the whole thing with no external labels using a bridging-based agreement algorithm.

Conventional ML wisdom in 2020 said the system had to be closed-source and label-dependent
Fear was constant manipulation without ground-truth fact-checker labels
The bridging-based agreement approach works without any external labels
Cross-disagreement agreement also provides strong anti-manipulation properties

“I think a room of ml Engineers would say oh you have to keep it closed Source you know people are going to be manipulating…”
08:00

#machine-learning#anti-manipulation#algorithm#open-source

Myth Buster1:26:30

The Counterintuitive Win: Anonymous Contributors Are More Honest

The team originally assumed notes should carry real names for trust — and were completely wrong. The pilot's biggest lesson was that pseudonymity works better: people were afraid to be harassed writing notes under their real name, and studies plus their own data show people cross partisan boundaries far more readily when anonymous. Their many quality mechanisms kept quality high regardless.

The first prototypes assumed real names would build trust; that assumption was wrong
Under real names, people feared harassment and self-censored on controversial topics
People are far more willing to agree with the other side when anonymous or pseudonymous
Existing quality mechanisms meant pseudonymity didn't lower note quality

“one people were hesitant to write a note on a controversial topic because they didn't want to get like attacked or harassed online and so…”
1:27:00

“it allows it allows freedom for honesty which is pretty great”
1:28:00

#anonymity#psychology#product-design#honesty

Hot Take· 3

Hot Take37:30

The Myth That Managing More People Means More Impact

Keith argues the belief that a bigger org or wider scope equals more impact is a myth that traps ambitious leaders. Had he stayed running a large consumer PM team he'd have produced more OKR pages; instead a tiny team built the industry standard for handling information quality at internet scale.

Bigger scope / more direct reports does not equal more impact
Staying on the management track would have produced documents, not a world-changing product
Community Notes became the industry standard for dealing with misinformation
The classic career path rewards headcount, but working on what you love matters more

“myth that that can get in people's way is the idea that the the more people you manage or something or the larger your scope…”
Keith Coleman · 38:30

#leadership#career#impact#management

Hot Take1:03:30

On a Small Team, Deleting Code Matters More Than Writing It

Jay argues that a forced-small team is an advantage because it makes you delete code. Engineers tend to add little incremental wins from short A/B tests without appreciating the long-term maintenance burden they create. Auditing systems and cutting features whose upkeep cost exceeds their gains keeps a small team fast.

Deleting code is often more important than writing it
Promotion incentives push engineers to add small, ship-a-win features
A one-month A/B test hides the eternal maintenance cost of what you add
A small team is forced to audit systems and delete low-value complexity

“deleting code is more important than writing it a lot of the time uh so I I think so often maybe due to promotion incentives…”
Jay Baxter · 1:04:00

#engineering#small-teams#maintenance#code

Hot Take1:11:00

The Craziest Principle: There Is No Button to Take a Note Down

Keith says the foundational, most unsettling principle is that Community Notes is the voice of the people, not the company. There is deliberately no button for the company to change a note's status: if a note is rated helpful, it shows. If a note is bad enough to want to override, that's treated as a signal to fix the system, not to reach in and pull the note.

The product represents the voice of the people, not the company's voice
There is intentionally no button to change a note's status
If a helpful-rated note shows, the company can't take it down
A note bad enough to want removed is a system problem to redesign, not to override

“probably the craziest one is just that this thing is going to be the voice of the people it's going to represent the voice of…”
Keith Coleman · 11:30

“it has to work that well if it doesn't work well enough to do that then it doesn't work”
Keith Coleman · 12:00

#principles#trust#product-design#governance

Explainer· 3

Explainer06:00

How Community Notes Uses "Surprising Agreement" to Stay Neutral

Community Notes doesn't take a majority-rules vote. It looks for agreement between people who have historically disagreed with each other, and only shows a note when those opposed groups converge on it. That surprising cross-partisan agreement is what makes a note read as neutral, accurate, and well written.

Anyone on X can flag a misleading post and propose a note; others rate it
A note only shows if people who normally disagree find it helpful
Majority-rules voting or instant publishing would produce biased or inaccurate notes
Cross-partisan agreement acts as a proxy for accuracy and neutrality

“if the note is found helpful by people who normally disagree with each other indicating that it's probably accurate it's probably really neutrally worded it's…”
06:30

“we actually look for agreement from people who have disagreed in the past uh and and what we see is when people actually have that…”
07:30

#community-notes#algorithm#misinformation#trust

Explainer19:30

Only ~8% of Proposed Notes Ever Show — Because a Bad Note Is the Worst Outcome

Community Notes deliberately shows only about 7-11% of proposed notes. People ask why they don't show more, but the team holds an intentionally conservative bar because a single bad note undermines the trust the whole product depends on. Writing notes that opposing groups rate unhelpful can cost you your ability to write.

Roughly 8% of proposed notes get shown (range ~7-11% over time)
The threshold is set conservatively on purpose to keep note quality high
Showing a bad note is viewed as the worst possible mistake because it erodes trust
Writers whose notes are found unhelpful by cross-partisan raters lose and must re-earn writing ability

“we probably show about 8% of notes that get proposed um I think that's it's been between it's say 7% and 10% or 11% something…”
20:00

“we view the worst possible mistake as showing a bad note because that's going to undermine trust and the trust is is is why people…”
20:30

#community-notes#quality#trust#moderation

Explainer1:13:30

Radical Transparency: Anyone Can Download the Code and Reproduce Every Note

Trust required total openness. The ranking code and all rating data are published so anyone can replicate the entire service and audit it. This forced awkward architectural choices — the matrix-factorization model is built to be re-run from a downloaded TSV — and it really is runnable: on one machine with ~500GB of RAM in about a day. Vitalik Buterin is among those who've verified it independently.

The code that decides which notes show, plus all ratings data, is public
Anyone can take the code and data and replicate the whole service
Being genuinely runnable forced unusual architectural decisions from scratch
Replicating it takes ~500GB RAM on one machine for about a day; Vitalik Buterin verified it in a blog post

“the code that decides what note show has to be out in the open all the data and ratings that that make it happen have…”
Keith Coleman · 1:14:00

“oh like 500 gigs it it'll take like a day if you don't uh do anything special to speed it up”
Jay Baxter · 1:16:30

#open-source#transparency#trust#audit

Story· 5

Story26:30

A Note Cuts Reposts 50-60% — and Makes Authors 80% More Likely to Delete

In an A/B test, showing a post with a note drove 30-40% drops in likes and reposts, versus the 1% effect size that's considered great for an algorithm change. External research groups found total reposts fall 50-60% once a note is applied, killing virality within a few generations. Authors also become 80% more likely to delete a noted post.

1% is a strong effect size; notes produced 30-40% drops in likes and reposts in an A/B test
Independent research groups found ~50-60% drops in total reposts after a note is applied
At 50-60% per generation, virality quickly collapses to zero
Authors are 80% more likely to delete their post after being noted

“1% is typically an awesome effect size for any sort of algorithm change we saw more like 30 to 40% engagement rate drops uh for…”
27:00

“authors become 80% more likely to decrease or sorry to delete their post after they get noted”
28:30

#community-notes#virality#engagement#research

Story25:00

A Contributor With 12 Followers Made the White House Delete a Tweet

When Community Notes launched US-wide in 2022, a note appeared on a White House tweet and the White House deleted it and reissued an updated statement. Keith uses it to show the leverage an ordinary contributor has: someone with a handful of followers changed a government's public talking points.

A note on a 2022 White House tweet led the White House to delete and re-issue the statement
The note's author was an ordinary contributor, not a public figure
Demonstrates that anyone correct and credible can shape public discourse
Intrinsic impact, not pay, is what motivates top contributors

“when we first launched us-wide this was like in 2022 a note appeared on a White House tweet and the White House deleted the tweet…”
25:00

“here you just put a put a note on the White House and they changed their public talking points based on what you did like…”
25:30

#community-notes#impact#story#contributors

Story30:00

"How About I Just Stop Doing My Job" — The Origin of Community Notes

Keith joined Twitter in 2016 during the Trump-Clinton election, struck by how important and unsolved the misinformation problem was. After years running a large PM team without seeing the change he wanted, he came back from paternity leave and asked his boss Kavon to let him drop his job entirely to try crazy ideas on misleading information.

Keith joined Twitter in 2016 via acquisition during the presidential election
Existing approaches (fact-checkers, internal trust & safety) weren't working or trusted
He was weighing starting a company but kept returning to the misinformation problem
He asked his boss to let him quit his PM role to prototype; it became Community Notes

“I went to my boss Kavon I was like Hey Kayon how about I just stop doing my job and I go work on this…”
Keith Coleman · 33:00

#origin-story#career#product#misinformation

Story35:00

How Elon's DM Changed the Name Back From Bird Watch to Community Notes

The project was originally called Community Notes in Keith's very first Figma mockup, then renamed Bird Watch. After acquiring Twitter, Elon DM'd Keith calling it "this community notes thing" — the exact original name — and they changed it back the next day. Jack Dorsey mocked the name as the most boring Facebook-style name possible.

The first Figma mockup was already called Community Notes before it became Bird Watch
Elon kept referring to it as "community notes" in a post-acquisition DM
The name was switched back the next day
Descriptive, intuitive names have served the product well despite Jack's ribbing

“he kept referring to it as this community notes thing and I was like you know it's interesting that you keep calling that calling it…”
Keith Coleman · 35:30

“Elon was like hey let's just call it that and so the next day we just changed the name”
Keith Coleman · 36:00

#naming#product#elon-musk#story

Story1:18:30

The Israel-Hamas Deluge: The Biggest Stress Test — and Why Notes Held

The October 2023 Israel-Hamas conflict was the largest flood of misleading information Jay had ever seen, including out-of-context photos and fake battle footage made in the Arma 3 game engine. Features shipped just weeks earlier — media matching and a speed-up shaving hours off note time — proved decisive. Median time from post to note was about five hours, versus the two-to-four days typical of fact-checking.

Oct 2023 conflict produced an overwhelming volume of misleading posts and media
Fake battle footage was created in the Arma 3 video game simulator
Recently shipped image/video note-matching and a ~3-hour speed-up landed just in time
Median post-to-note time was ~5 hours vs. the 2-4 days typical of traditional fact-checking

“there were people making fake like battle footage in the video game simulator Arma 3 so there like notes explaining the stuff looked really looked…”
1:19:30

“the median time from a post going live to a note showing up was five hours which is like crazy fast typical factchecking is like…”
Keith Coleman · 1:21:00

#community-notes#misinformation#stress-test#speed

Takeaway· 2

Takeaway13:30

The Scale: 95,000 Notes, 30 Billion Views, Nearly a Million Contributors

Keith shares the numbers that surprise people. In 2024 roughly 95,000 notes were seen about 30 billion times, more than double the prior year. A single note on an image or video is auto-matched to every post containing that media, so one note can cover thousands of posts. Around 950,000 contributors now participate worldwide.

2024: ~95,000 notes seen ~30 billion times, more than double 2023's ~37k notes / 14 billion views
By contrast a UC Berkeley figure cited ~10 traditional fact checks a day versus hundreds of notes a day
One note is matched to all posts sharing the same image or video, covering thousands of posts
~950,000 contributors globally, with more on a waitlist

“we had something like 95,000 notes that were seeing about 30 billion times that's more than double the prior year prior year was something like…”
14:30

“there's something like 950,000 contributors around the world that's you know nearing a million people making this happen”
15:00

#community-notes#scale#metrics#fact-checking

Takeaway56:30

Let People Opt In: Self-Selection Beats Assigning a Team

No one was assigned to Community Notes — every member reached out or applied and was interviewed for mutual fit, so everyone was fully bought into the mission. Keith notes Elon later applied the same principle at company scale with the "fork in the road" email that asked all of Twitter to click a button to opt into the hardcore Twitter 2.0.

No one was assigned to the project; contributors self-selected and were interviewed
Self-selection produces total buy-in to the goal, team, and way of working
Elon's "fork in the road" email asked employees to actively opt into Twitter 2.0
Opt-in works at the start of a crazy project and, surprisingly, at large scale too

“people are self- selecting to join we did not assign anyone to this project like people reached out to join or they applied to join”
Keith Coleman · 56:30

“he sent an email out that was like Hey Twitter 2.0 like Fork fork in the road right fork in the road fork in the…”
Keith Coleman · 57:30

#team-building#hiring#culture#motivation

Key ideas

Frameworks in this episode

Bridging-Based Agreement

Delete-First Lean Engineering

Principles-First Product Design

Prove-It-At-Every-Step Rollout

Pseudonymity for Honest Contribution

The Thermal Team

People & resources mentioned

People · 17

Resources · 25

Insights & moments

⚡Myth Buster· 2

You Don't Need Fact-Checker Labels to Bootstrap a Trust System

The Counterintuitive Win: Anonymous Contributors Are More Honest

◆Hot Take· 3

The Myth That Managing More People Means More Impact

On a Small Team, Deleting Code Matters More Than Writing It

The Craziest Principle: There Is No Button to Take a Note Down

✶Explainer· 3

How Community Notes Uses "Surprising Agreement" to Stay Neutral

Only ~8% of Proposed Notes Ever Show — Because a Bad Note Is the Worst Outcome

Radical Transparency: Anyone Can Download the Code and Reproduce Every Note

❝Story· 5

A Note Cuts Reposts 50-60% — and Makes Authors 80% More Likely to Delete

A Contributor With 12 Followers Made the White House Delete a Tweet

"How About I Just Stop Doing My Job" — The Origin of Community Notes

How Elon's DM Changed the Name Back From Bird Watch to Community Notes

The Israel-Hamas Deluge: The Biggest Stress Test — and Why Notes Held

▲Takeaway· 2

The Scale: 95,000 Notes, 30 Billion Views, Nearly a Million Contributors

Let People Opt In: Self-Selection Beats Assigning a Team

Myth Buster· 2

Hot Take· 3

Explainer· 3

Story· 5

Takeaway· 2