The Angry-God Containment Lens
Assume the AI is a malicious agent trying to hurt you, then engineer so it structurally can't
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 90%
Because you can patch a bug but you can't patch a brain, you cannot make a model refuse every attack. Schulhoff's lens borrows from AI 'control' research: treat the AI as an angry, malicious god that wants to hurt you, ask what is the worst it could do, then use classical cyber-security to contain the blast radius so that even a fully-tricked model can't cause damage. It shifts effort from stopping the AI from being fooled to making a fooled AI harmless.
Origin
Framed by Sander Schulhoff, drawing on the AI-safety subfield of 'control' (associated with the MATS / ML Alignment & Theory Scholars program he runs), which asks: given a malicious AI, can we still make it useful and ensure nothing bad happens?
Core principles
- 01You can patch a bug but you can't patch a brain — assume the model can always be tricked.
- 02The valuable security work lives at the intersection of AI security and classical cyber-security, not in either alone.
- 03Design for the worst output the model could produce, not the output you expect.
- 04Containment (sandboxing, permissioning) beats persuasion (guardrails, prompts) because it doesn't depend on the model behaving.
How to run it
- 1
Assume the model is fully compromised
Start from the premise that a user can make the AI produce any output or take any permitted action in any order. Do not assume OpenAI/Anthropic 'won't let it' do bad things — even state-of-the-art models can be prompted into anything.
Pro tip This reframe is the whole unlock — classical security people often skip it because AI 'seems infallible.'
Watch out The most common failure is not even asking 'what if someone tricks the AI?' — the question simply doesn't occur to traditional security reviewers.
- 2
Ask 'what's the worst that could happen?'
Trace the malicious path concretely. If the AI can output anything, what does that output do next? Where does it run? What can it reach? Follow the chain to the actual damage.
Pro tip Walk it literally: 'the AI outputs malicious code → the code runs → where? → on my application server → that's a problem.'
- 3
Contain with classical cyber-security
Once you know the worst case, use permissioning and isolation to neutralize it — dockerize/sandbox code execution, scope data access, restrict actions. The goal is that a fully-tricked model still can't cause harm.
Pro tip Put a research-literate AI security person alongside classical security staff; the intersection is where both the value and the vulnerabilities are.
Watch out Human-in-the-loop escalation contains risk but fights the market's demand for autonomous agents, so treat it as a stopgap, not the architecture.
In the wild
MathGPT sent user problems to GPT-3 to write code, then executed that code on the same server running the app. An attacker asked it to write code exfiltrating secrets and stole the OpenAI API key. An intersection-minded reviewer would trace: AI can output any code → it runs on my app server → dockerize it in an isolated container and read only the sanitized output.
→ Containerizing the code-run 'completely solved' the prompt injection — the model can still be tricked, but a tricked model can no longer reach anything valuable.
Common mistakes
Trusting the model because it 'seems smart and infallible'
Reviewers assume a capable AI will simply do what it's supposed to, so they never model an adversary tricking it — leaving execution environments and permissions wide open.
Trying to fix a brain like you'd patch a bug
Teams treat model vulnerabilities like software bugs they can close with a patch or a prompt; because you can't patch a brain, the vulnerability remains and the effort should have gone into containment.
Is it for you?
Best for
Security teams and architects reviewing an agentic or code-executing AI system before deployment
Not ideal for
Low-stakes chatbots with no actions and no untrusted data, where the containment work is unnecessary overhead
From the transcript
“you can patch a bug but you can't patch a brain”
“not only do you have a god in the box, but that god is angry. That god's malicious. that God wants to hurt you. And…”
“this AI could write any any possible output. Uh some user could trick it into outputting anything. What's the worst that could happen?”
“we can just dockerize that code run um put it in a a container so it's running on a different system and take a look…”
From the episode
The coming AI security crisis (and what to do about it)
Sander Schulhoff