Adaptive Evaluation Over Static Benchmarks
Measure AI robustness with attackers that learn, not with a frozen dataset of yesterday's attacks
- Difficulty
- Advanced
- Time to result
- ~ongoing to results
- Steps
- 3
- Confidence
- 85%
You cannot honestly claim a robustness percentage from a static list of malicious prompts — especially prompts built to attack an older model. Schulhoff argues the only credible measurement is adaptive evaluation: attackers (human or RL/search-based) that test, observe, learn, and improve against your specific defense. This is how frontier labs should report robustness and how any team should validate its own system.
Origin
Advanced by Sander Schulhoff based on years running adaptive red-teaming competitions and a joint paper with OpenAI, Google DeepMind, and Anthropic; he notes even labs' constitutional-classifier reporting still leans too heavily on static datasets.
Core principles
- 01Static benchmarks constructed against an earlier model are an unfair, uninformative test of a newer one.
- 02Adaptive attackers learn what works and iterate — that's what real adversaries do.
- 03Humans remain the strongest adaptive attackers, beating automated systems that need orders of magnitude more attempts.
- 04Adversarial robustness is genuinely hard to estimate; a single percentage from a static set is close to meaningless.
How to run it
- 1
Reject static-dataset robustness claims
Treat any robustness number derived from replaying a fixed set of known malicious prompts as unreliable, especially if those prompts were designed to attack a different or earlier model.
Pro tip Ask which model each benchmark prompt was originally crafted against; mismatches invalidate the comparison.
Watch out Static datasets are 'really quite useless' for a newer model they weren't built for.
- 2
Build or hire adaptive attackers
Evaluate with attackers that can learn over time — human red-teamers, and RL/search-based automated methods — turned loose against your actual defense to iterate on what works.
Pro tip Include humans; they broke 100% of tested defenses in 10-30 attempts, far fewer than automated systems needed.
- 3
Report robustness honestly and re-run over time
Report results from adaptive evaluation rather than a headline catch-rate, and re-run as models and attacks evolve, since progress on this problem has been minimal and defenses degrade against new techniques.
Pro tip If a defense holds against automated attackers but falls to a human in under an hour, say so — that's the real posture.
Watch out Reporting a static-benchmark percentage as your robustness invites overconfidence in a system humans can still break quickly.
In the wild
Anthropic's constitutional classifiers made it genuinely harder to extract CBRNE information from Claude, and static benchmarks reflect improvement. But under adaptive evaluation, humans can still succeed in under an hour and automated systems still get through.
→ Shows the gap between static-benchmark improvement and true adaptive robustness — the honest measure reveals the problem is still open.
Common mistakes
Reusing a static malicious-prompt dataset built for an older model
Applying prompts crafted against an earlier model to a new one is not a fair comparison and produces an inflated, meaningless robustness score.
Assuming automated red teaming alone is sufficient
People expected the process could be fully automated, but automated attackers need orders of magnitude more attempts and still only beat ~90% of cases; dropping human red-teamers understates your true vulnerability.
Is it for you?
Best for
Frontier labs and serious security teams that need to actually quantify a system's adversarial robustness
Not ideal for
Tier-1 chatbot deployments where no robustness claim is needed and the measurement effort isn't warranted
From the transcript
“the best measurement you can do is an adaptive evaluation. And what that means is you take your defense... and you build an attacker that…”
“humans break everything 100% of of the defenses in maybe like 10 to 30 attempts”
“they rely a lot on static evaluations where they say... this like data set of malicious prompts which were usually constructed to attack a particular…”
“they need to focus on adaptive evaluations rather than static data sets. Uh which are really uh quite quite useless”
From the episode
The coming AI security crisis (and what to do about it)
Sander Schulhoff