Scaling Expert Judgment Through Algorithms
Structure your experts' judgment into a signal, train on that — not only on engagement
- Difficulty
- Expert
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 92%
Most ranking systems can only optimise for engagement outcomes because the platform doesn't own the content and cannot judge its quality. Hardiman's counter-model: if you own the full stack — the content, the distribution and the product — you can capture your domain experts' judgment as structured data (e.g. an editorial importance score assigned by journalists), train algorithms on that signal, and thereby scale expert judgment to a mass audience while still driving toward reach, engagement and conversion.
Origin
Alex Hardiman contrasting the New York Times home screen with her experience running news ranking and feed at Facebook, where only engagement outcomes were trainable because Facebook did not own or control the content passing through the platform.
Core principles
- 01Owning the full stack — content, distribution, product — is what makes quality a trainable signal
- 02Start with expert judgment, then use algorithms to scale it, not to replace it
- 03Engagement-only training optimises for what people click, never for what is good
- 04The product job is to structure expertise into machine-readable signal
- 05PMs must become domain-minded and domain experts must become product-minded
How to run it
- 1
Check whether you own the full stack
Determine whether you control the content itself, not just the software and the distribution. If you don't, you have real limits on knowing whether what's passing through is high or low quality — and engagement is all you can train on.
Watch out This is exactly the constraint that produced Facebook's news-quality problems: 'we can build the software and the distribution, but we didn't control the content.'
- 2
Start every surface with expert judgment
On the home screen, always begin with expert editorial judgment about the most important and interesting stories. The algorithm layers on top of that baseline; it doesn't originate it.
- 3
Structure the expertise into a signal
Get your experts to emit their judgment as data. At the Times, journalists produce 'editorial importance scores' — a specific dataset capturing the judgment of 2,000+ journalists in a form an algorithm can train on.
Pro tip This is the load-bearing product work: designing the capture mechanism so that experts producing the signal is cheap and natural inside their existing workflow.
Watch out If emitting the signal feels like extra bureaucracy to the experts, the dataset will be sparse and the model will fall back to engagement by default.
- 4
Train on the quality signal, optimise for the business outcome
Train the ranking algorithms on the editorial signal, and still let them drive toward outcomes like reach, engagement and conversion. The quality signal constrains the search space; the business metrics rank within it.
- 5
Converge the two disciplines
Deliberately develop PMs who are editorially minded and editors who are product minded. The blend of art and science — valuing expert judgment alongside KPIs, customer research and insights — is the skill the system requires.
In the wild
The home screen begins with expert editorial judgment on the most important and interesting stories. On top of that, algorithms are trained on datasets like editorial importance scores that come directly from the journalists, then optimise toward reach, engagement and conversion.
→ Editorial judgment is scaled to a large group of readers rather than being applied by hand to a single front page — Hardiman: 'No one else is really doing something in that space.'
Running news ranking and feed at Facebook, Hardiman's team could only train content on engagement outcomes: 'We couldn't actually train it based on the quality of that piece of information itself,' because the platform never owned or classified the content it distributed.
→ A structural ceiling on quality that no amount of ranking work could lift — the argument for owning the content stack if quality is your differentiator.
Common mistakes
Trying to bolt a quality signal onto a platform that doesn't own its content
If you didn't build the system to classify what passes through it, you have no ground truth for quality — and you cannot fix that with a ranking model. Facebook's content model was binary (friends-and-family vs public), so 'public' spanned everything from a reputable news organisation to someone's brother declaring something true.
Replacing expert judgment with the algorithm instead of scaling it
The order matters: 'We always start with expert editorial judgment.' An algorithm trained on expert signal but allowed to originate the ranking drifts back toward the outcome metric it optimises, and the expertise stops being the anchor.
Is it for you?
Best for
Product and ML leaders at companies that own their content or supply (media, marketplaces with curated inventory, education, health) and want quality — not just engagement — as an optimisation target
Not ideal for
Pure UGC platforms with no ownership or classification of the content flowing through them — the ground-truth signal simply doesn't exist
From the transcript
“you work across the full stack of the product, meaning we own our journalism and our content, we own our distribution, and we”
“We always start with expert editorial judgment the most important and interesting stories. But on top of that we're training algorithms on specific data sets…”
“when I was at Facebook and we were focused on news ranking and feed, all we could do was train pieces of in- information based…”
“you have like 2,000 plus journalists and you're actually trying to structure their expertise into things that can actually translate into really great algorithmic”
“So product managers are becoming very editorially minded. And we're also getting editors to become more product minded.”
From the episode
An inside look at how the New York Times builds product
Alex Hardiman (CPO at The New York Times)