Constitutional AI Self-Critique Loop
Align a model to written principles by having it critique and rewrite its own outputs, then train on the fix
- Difficulty
- Expert
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 88%
A repeatable training process that bakes values into an AI model instead of bolting on filters. You write natural-language principles, ask the model to check whether a given response violates any applicable principle, have it critique and rewrite its own response when it does, then train the model to produce the corrected response directly. It is reinforcement learning from AI feedback (RLAIF) with no humans in the loop.
Origin
Ben Mann describes Anthropic's Constitutional AI method, drawing principles from sources like the UN Declaration of Human Rights and Apple's terms of service, plus many Anthropic generated itself. It is an instance of what he calls RLAIF (reinforcement learning from AI feedback), a shift from the more common RLHF (human feedback).
Core principles
- 01Values should be explicit written principles, not implicit in whatever human raters you happen to find
- 02The model can improve itself recursively using its own critique
- 03Publish the constitution so values are a societal conversation, not a private decision by a small group in San Francisco
- 04Target what people actually mean, not the literal request, to avoid the monkey's-paw failure mode
How to run it
- 1
Write the constitution
Assemble a list of natural-language principles the model should follow, e.g. no hate speech, don't expose credentials shared in trust, be helpful, honest, and harmless.
Pro tip Borrow from established, legitimacy-carrying documents (UN human rights, published privacy terms) so customers can inspect and trust the values.
- 2
Select the applicable principles
For a given prompt, first figure out which constitutional principles are actually relevant to that input.
Watch out Applying every principle to every prompt is noise; relevance-filter first.
- 3
Generate and self-check
Have the model produce a default response, then ask the model itself whether that response complies with the selected principle.
- 4
Critique and rewrite on violation
If the response violates the principle, ask the model to critique itself and rewrite the response in light of the principle. If it already complied, do nothing.
- 5
Train on the corrected output only
Strip out the intermediate critique step and train the model to produce the correct response 'out the gate' for similar future inputs.
Pro tip Removing the middle reasoning step means the deployed model internalizes the value rather than performing a visible extra pass each time.
Watch out Recursive self-improvement can drift into deceptive alignment or hidden goals if the model isn't good enough to see its own mistakes; keep laboratory oversight.
In the wild
Mann says the character and personality people loved in Opus 3 came directly from alignment research (Amanda Askell and others defining helpful, honest, harmless), including how to refuse a request without shutting the person down.
→ Alignment and capability turned out to be convex, not a trade-off: safety work produced a differentiated, less-sycophantic, more-trusted product.
Common mistakes
Goodhearting engagement metrics instead of aligning
Optimizing 'user engagement is number one, if people say yes then it's good' produces sycophantic models that flatter rather than help; Constitutional AI targets actual values instead.
Leaving values to whichever human raters you happen to find
Ad-hoc human labeling encodes arbitrary, inconsistent values; writing an explicit, published constitution lets you take a principled, inspectable stance.
Is it for you?
Best for
AI/ML teams building models or agents who need scalable, inspectable value alignment without a human in every loop
Not ideal for
Teams whose models aren't yet capable enough to reliably critique their own outputs, where self-improvement can amplify errors
From the transcript
“then we ask the model itself to first generate a response and then see does the response actually abide by the constitutional principle”
“then we ask the model itself to critique itself and rewrite its own response in light of the principle. And then we just remove the…”
“just produce the correct response out the gate”
“It's just using the model to improve itself recursively and align itself with these values that we've decided are good”
From the episode
Anthropic co-founder on quitting OpenAI, AGI predictions, $100M talent wars, 20% unemployment, and the nightmare scenarios keeping him up at night
Ben Mann