RLAIF Reward Design
Have an expert define success criteria and a rubric once, then let AI reinforce the capability — more scalable than labeling examples
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 3
- Confidence
- 85%
Foody contrasts three data paradigms and argues the market is converging on the third. Instead of supplied input-output pairs (SFT) or humans ranking generations (RLHF), the expert defines a success criterion and a way to measure it — a unit test where code can verify, or a rubric where it can't — and that verifier reinforces the model's own attempts. Because one rubric scores infinite attempts, it is far more data-efficient than hand-labeling examples.
Origin
Brendan Foody, CEO of Mercor, describing the industry shift toward reinforcement learning from AI feedback. He notes a co-founder of Anthropic told him the same, and cites Andrej Karpathy's point that 'RL environments' and 'evals' are the same underlying data type used for different purposes.
Core principles
- 01SFT is input-output pairs; RLHF ranks model generations; RLAIF has a human define success criteria and lets AI reinforce — the market is moving to the third
- 02A verifier is either code (a unit test) where success is programmatically checkable, or a rubric where it is not
- 03One rubric scores unlimited attempts, making RLAIF far more scalable and data-efficient than labeling examples
- 04An eval and an RL-environment verifier are the same artifact used two ways: benchmark vs reward signal
- 05Model quality is bounded by the quality of the experts who authored the verifier — 'models are only as good as their evals'
How to run it
- 1
Have an expert define the success criteria
Rather than collecting example outputs, an expert articulates what 'good' looks like as measurable criteria for the target capability.
Pro tip The expert's caliber caps the ceiling — a Fang engineer or Goldman analyst author beats an undergrad author.
- 2
Encode the criteria as a verifier
Where success is programmatically checkable, write a unit test in code. Where it isn't (legal redlines, creative work), write a scoring rubric.
Pro tip Prefer code-checkable verifiers when the domain allows — they scale without human scoring.
Watch out Watch for reward hacking; the reward must actually capture the capability, not a proxy the model can game.
- 3
Reinforce the model's own trajectories against the verifier
Let the model generate attempts; the verifier rewards good trajectories, upweighting the reasoning chains that produce the wanted capability.
Pro tip This upweights the right tokens from pre-training rather than injecting new knowledge — post-training, not more pre-training data.
In the wild
A lawyer defines the rubric for a correct contract redline; the model attempts redlines and is rewarded per criterion satisfied, the way a TA applies a professor's grading criteria.
→ Scalable reward signal that improves the model's legal capability without hand-writing thousands of redlines.
For an X-ray a model can't fully read, radiologists build the post-training dataset — a stasis point of correct diagnosis with rewards and penalties. The quality of those radiologists went directly into the quality of ChatGPT's recommendation.
→ The model focuses on the right tokens and reasons better in a generalized way on medical imaging.
Common mistakes
Defaulting to input-output labeling
Collecting SFT example pairs is less scalable and data-efficient than defining one verifier that scores infinite attempts.
Ignoring reward hacking
A poorly specified reward lets the model optimize the metric without acquiring the capability.
Is it for you?
Best for
AI teams and labs building post-training data pipelines for a specific capability
Not ideal for
Injecting net-new factual knowledge — that belongs in pre-training, not RL reward design
From the transcript
“the human defines some sort of success criteria, some way to measure that and examples in code. It could be a unit test, right? We…”
“one of my favorite quotes from customers is that models are only as good as their evals”
“it's really unlocking allowing the model to focus on all the right tokens from pre-training all the right things in model context”
From the episode
Why experts writing AI evals is creating the fastest-growing companies in history
Brendan Foody (CEO of Mercor)