The Two-Sided Quality Signal
Great curation does two opposite jobs at once: kill the worst, surface the best.
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 88%
A model for measuring and enforcing quality in any large system, borrowed from how Google Search ranks web pages. Quality is not one dial — it's two distinct problems: a content-moderation job (remove the worst of the worst) and a discovery job (find the best of the best). Use it when building any evaluation, ranking, or curation system where 'throwing bodies at it' or a checklist would fail.
Origin
Chen explains why most people misunderstand data quality — they think you can throw bodies at it and pass a checklist. He uses the Google Search analogy to show that finding truly great work is a fundamentally different operation from filtering out garbage, and both must run at once.
Core principles
- 01Checklists ('is it 8 lines, does it contain the word moon') measure compliance, not quality.
- 02Removing spam and the worst content is a separate problem from discovering excellence.
- 03The best work has subtle, subjective qualities a rubric can't fully capture.
- 04Gather many weak signals and feed them into a model rather than trusting a single score.
How to run it
- 1
Define quality richly, not as checkboxes
Articulate what greatness actually means for the task — for a poem, does it surprise you, tug your heart, teach you something — rather than reducing it to a checklist.
Pro tip Set the bar at the best that could exist ('Nobel Prize winning poetry'), not the minimum viable output.
Watch out A checklist-passing output can be completely different from what you actually want.
- 2
Run the moderation job — remove the worst of the worst
Systematically strip out the spam, the low-quality, the broken — the equivalent of pages that don't load.
- 3
Run the discovery job — surface the best of the best
Separately, build signals that identify genuine excellence, which requires different criteria than filtering out the bad.
Pro tip The person who writes high-school-level poetry that checks every box is not the person you want.
- 4
Gather thousands of signals and let a model judge
Collect many signals about the work and the worker (background, expertise, actual performance, keystrokes, review scores) and feed them into an ML system, exactly as a search engine does.
Pro tip Track quality per domain — someone great at poetry may be weak at technical documentation.
In the wild
A shallow quality check asks: is it a poem, 8 lines, contains 'moon'? Chen's team instead asks whether the poem is unique, full of subtle imagery, whether it surprises you and teaches you something about moonlight — because they're looking for the equivalent of Nobel-Prize-winning poetry.
→ A far higher, subjective bar that requires real technology to measure rather than a checklist.
Search must both remove spam and dead pages AND discover the single best page for a query — two different operations feeding the same ranking system. Surge applies the identical structure to workers, tasks and projects.
→ A quality apparatus with thousands of signals per worker and per task.
Common mistakes
Throwing bodies at the problem
Adding more people to a quality task doesn't produce quality — it produces more checklist-passing mediocrity.
Confusing the two jobs
Filtering out the worst does nothing to surface the best; a clean, spam-free pool of average work is still average.
Is it for you?
Best for
Builders of evaluation, ranking, hiring, or content-curation systems who need to define and enforce quality at scale.
Not ideal for
Small one-off judgments where a single expert can simply eyeball the result.
From the transcript
“They think you can just throw bodies at a problem and get good data and that's completely wrong.”
“One is you want to remove all the worst of the worst web pages.”
“But then you also want to discover the best of the best.”
From the episode
The 100-person AI lab that became Anthropic and Google's secret weapon
Edwin Chen (Surge AI)