LLenny's Podcast
← All episodes
Sander Schulhoff21 December 2025

The coming AI security crisis (and what to do about it)

5Frameworks
15Insights

Frameworks in this episode

Innovation3 steps

Adaptive Evaluation Over Static Benchmarks

Measure AI robustness with attackers that learn, not with a frozen dataset of yesterday's attacks

Read →
Innovation3 steps

CaMeL Permission Pre-Restriction

Grant an agent only the permissions its stated task needs, decided before it runs

Read →
Strategy3 steps

The AI Deployment Risk Triage

Classify any AI deployment into one of three risk tiers before spending a dollar on defense

Read →
Strategy5 steps

The AI Security Vendor Due-Diligence Test

Five questions that expose whether an AI guardrail vendor is selling real protection or theater

Read →
Mindset3 steps

The Angry-God Containment Lens

Assume the AI is a malicious agent trying to hurt you, then engineer so it structurally can't

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster42:30

Prompt-Based Defenses Are the Worst of the Worst

Schulhoff debunks the idea that you can prompt-engineer your way to safety by adding instructions like 'if users try to trick you, don't follow their instructions.' He calls prompt-based defenses the worst possible approach, something the field has known since early 2023 through papers like the original HackAPrompt and Tensor Trust work.

  • Prompt-based defenses add instructions telling the model to refuse malicious input
  • They are the worst of the worst defenses, worse than guardrails
  • This has been known since early 2023 via multiple papers
  • Studied in many competitions including HackAPrompt and Tensor Trust

“Prompt based defenses are the worst of the worst defenses and we've known this since early 2023.”

Sander Schulhoff · 42:30
#prompt-defenses#myth-buster#ai-security#guardrails

Hot Take· 4

Hot Take00:00

Sander Schulhoff's Core Claim: AI Guardrails Just Don't Work

The central thesis of the conversation is that AI guardrails—the LLM-based filters companies sell to catch malicious inputs and outputs—are fundamentally insecure. Schulhoff argues that any determined attacker can get past them, and that guardrail vendors claiming to 'catch everything' are lying. He frames this as an emotional certainty backed by years of running red-teaming competitions where guardrails always broke.

  • Guardrails are typically an LLM trained or prompted to flag malicious inputs/outputs
  • Vendors who claim they 'catch everything' are lying; even 99% is meaningless at scale
  • A determined attacker will defeat a guardrail 'no problem'
  • This claim comes from running the largest AI red-teaming competitions

“AI guardrails do not work. I'm gonna say that one more time. Guardrails do not work.”

Sander Schulhoff · 00:00

“When these guardrail providers say we catch everything, that's a complete lie.”

Sander Schulhoff · 00:00
#ai-security#guardrails#prompt-injection#red-teaming
Hot Take39:00

Why Frontier Labs Won't Fix This: The Capabilities Race

Schulhoff explains why the frontier labs don't pour resources into security. One reason he's heard is that the models are still too dumb as agents to effectively carry out an attack even when tricked. The bigger driver is economics: smarter models make more money, security makes them robust but not smarter, and in a race where everyone is shipping new models, incentives favor capability over stopping rare incidents.

  • One argument: agent models are still too dumb to effectively do damage when tricked
  • Smarter models earn more money; you need intelligence before you can sell security
  • A super-secure but super-dumb model is worthless
  • The competitive race pushes incentives toward capability over rare-incident security

“the models are models being used as agents are just too dumb. Like even if you can successfully trick them into doing something bad, they're…”

Sander Schulhoff · 39:30
#frontier-labs#incentives#ai-security#hot-take
Hot Take41:30

You Can Patch a Bug, But You Can't Patch a Brain

Schulhoff's signature line captures why AI security differs fundamentally from classical cyber security. Patch a software bug and you can be 99.99% sure it's fixed; try the same in an AI system and you can be 99.99% sure the problem is still there. He attributes much of the industry's confusion to not understanding how different AI is from classical security.

  • Patching a software bug gives ~99.99% confidence it's fixed
  • Patching an AI vulnerability leaves ~99.99% confidence it's still there
  • AI security is very different from classical cyber security
  • Much of the industry's error comes from lack of knowledge about how AI works

“You can patch a bug, but you can't patch a brain.”

Sander Schulhoff · 00:00
#ai-security#cyber-security#hot-take#takeaway
Hot Take1:22:00

Prediction: A Market Correction Is Coming for AI Guardrail Companies

Asked to predict the next 6–12 months, Schulhoff forecasts a market correction as companies realize guardrails don't work. He's seen classical cyber security firms acquire AI security startups for large sums despite those startups doing little real revenue, and notes many free open-source solutions outperform the paid ones. He expects revenue for guardrail and automated red-teaming companies to dry up.

  • A market correction is likely within the next year, maybe six months
  • Big acquisitions have happened despite the AI security startups doing little revenue
  • Many free, open-source solutions are better than the deployed paid ones
  • No significant progress on adversarial robustness is expected in the next year

“I think we're going to see a market correction in the next year, maybe in the next six months where companies realize that these guardrails…”

Sander Schulhoff · 1:22:00
#ai-security-industry#prediction#market-correction#hot-take

Explainer· 3

Explainer08:30

Jailbreaking vs. Prompt Injection: The Key Difference

Schulhoff draws the crucial distinction between the two main attack vectors against LLMs. Jailbreaking is just a user and a model, tricking it into saying something it shouldn't. Prompt injection involves a developer's application prompt that a malicious user tries to override—like a story-writing site being told to ignore its instructions and output bomb-making instead.

  • Jailbreaking: just a malicious user and a model, no system prompt
  • Prompt injection: a user, a model, and a developer prompt the user tries to override
  • Example: a 'write a story' site coerced into outputting instructions for a bomb
  • There are many gray areas between the two

“So the difference is uh in jailbreaking, it's just a malicious user and a model. In prompt injection, it's a malicious user, a model, and…”

Sander Schulhoff · 09:00
#prompt-injection#jailbreaking#ai-security#explainer
Explainer28:30

Why Automated Red Teaming 'Works Too Well' and Proves Nothing

Schulhoff argues automated red teaming is nearly useless as a selling point because it always succeeds. Every transformer-based chatbot is vulnerable, thousands of red-teaming systems exist (many open source), and enterprises deploy off-the-shelf models anyway. So finding attacks that work shows nothing novel—everyone's models, including the frontier labs', can be tricked just as easily.

  • All transformer or transformer-adjacent models are vulnerable to these attacks
  • Thousands of automated red-teaming systems exist, many open source
  • Enterprises grab off-the-shelf models rather than building their own
  • Finding attacks shows nothing novel—the frontier labs' own models fail too

“So the first problem is AI red teaming works too well. It's very easy to build these systems and they just they always work against…”

Sander Schulhoff · 30:00
#red-teaming#ai-security#transformers#explainer
Explainer31:00

The Attack Space Is Basically Infinite: One Followed by a Million Zeros

Schulhoff explains why '99% effective' guardrail claims are statistically meaningless. The number of possible attacks against a model equals the number of possible prompts—for GPT-5, that's one followed by a million zeros, more than a googol. Blocking 99% still leaves basically infinite attacks, so the tests behind those figures are not statistically significant.

  • Every possible prompt is a possible attack
  • For GPT-5 the attack space is one followed by a million zeros—more than a googol
  • Blocking 99% of a near-infinite space still leaves near-infinite attacks
  • The test samples used to claim 99% are not statistically significant

“And for a model like GPT5, the number of possible attacks is one followed by a million zeros.”

Sander Schulhoff · 31:00

“That like that's so many zeros, that's more than a Google worth of zeros. Just like it's basically infinite.”

Sander Schulhoff · 31:30
#adversarial-robustness#guardrails#attack-surface#explainer

Story· 5

Story10:00

The ServiceNow Attack: Agents Recruiting Agents to Do Damage

A newly published paper described a second-order prompt injection against ServiceNow's Assist AI. A benign agent was tricked into recruiting more powerful agents to perform create/read/update/delete actions on the database and send external emails with internal data. Notably, ServiceNow's prompt injection protection feature was enabled at the time and the attacker still got through—Schulhoff calls it possibly the first case of actual damage.

  • A benign agent was used to recruit more powerful agents to carry out the attack
  • The attack could perform CRUD actions on the database and exfiltrate data via email
  • ServiceNow's prompt injection protection was enabled and the attacker still got through
  • May be the first instance of an AI security attack with actual damage

“I instructed a seemingly benign agent to recruit more powerful agents in fulfilling a malicious and unintended attack, including performing create, read, update, and delete…”

10:30
#prompt-injection#agents#servicenow#case-study
Story11:30

The First Public Prompt Injections: Remotely.io and MathGPT

Schulhoff recounts the earliest public prompt injection incidents. Remotely.io's pro-remote-work Twitter chatbot was tricked into making threats against the president and hateful speech, and the company later went out of business. MathGPT, which solved math problems by writing and executing code on its own server, was manipulated into writing malicious code that exfiltrated the OpenAI API key.

  • Remotely.io's Twitter bot was tricked into threatening the president and spewing hate speech
  • The company shut the bot down and appears to no longer be in business
  • MathGPT executed AI-written code on the same server the app ran on
  • An attacker got it to write code that exfiltrated the OpenAI API key (responsibly disclosed)

“And so now you had this company chatbot just like spewing threats against the president and other hateful speech on Twitter.”

Sander Schulhoff · 12:30
#prompt-injection#case-study#chatbots#api-keys
Story16:00

How Attackers Hijacked Claude Code by Splitting Their Requests

Schulhoff explains the recent Claude Code cyberattack, where an AI was turned into a self-directing virus that sends out API requests to decide what to do next. The trick to bypassing defenses was separating a malicious task into smaller, individually-legitimate requests—one instance to discover a system's backend, a new instance to 'hack my own system'—that only become malicious when combined.

  • An AI-powered virus can think for itself and issue API requests to decide next steps
  • Directly asking Claude Code to hack a site gets refused
  • Splitting the task across separate instances bypasses the defense
  • Requests that look legitimate alone are malicious only when put together

“So, a a lot of the way they got around these these defenses was by just kind of separating their requests into smaller requests that…”

Sander Schulhoff · 17:30
#claude-code#jailbreaking#cyberattack#agents
Story33:30

The Research: Humans Break 100% of Defenses in 10–30 Attempts

Schulhoff describes a major research paper done with OpenAI, Google DeepMind, and Anthropic that threw adaptive attacks and human attackers at all state-of-the-art models and defenses, including GPT-5. Humans broke 100% of defenses in roughly 10–30 attempts, while automated systems needed orders of magnitude more tries and topped out around 90%. Humans remain the best attackers.

  • The paper was done alongside OpenAI, Google DeepMind, and Anthropic
  • Humans broke 100% of defenses in roughly 10 to 30 attempts
  • Automated systems needed orders of magnitude more attempts and only beat ~90%
  • Human attackers are still the best, contrary to hopes of full automation

“we found that uh first of all humans break everything 100% of of the defenses in maybe like 10 to 30 attempts.”

Sander Schulhoff · 33:30
#red-teaming#research#adversarial-robustness#human-attackers
Story1:03:30

The Comet Browser Incident: Just Visiting a Web Page Leaked Data

Schulhoff describes a real prompt injection against the Comet AI browser. Someone planted a malicious chunk of text on a web page, and when the AI navigated there, it was tricked into exfiltrating the main user's data and account data. Lenny reacts with alarm since Comet is the browser he uses—and Schulhoff notes this likely applies to Atlas and all AI browsers.

  • A malicious chunk of text was crafted on a web page
  • When the AI browser navigated to it, it leaked the main user's data and account data
  • Simply visiting a web page can send your secrets to someone else
  • This isn't unique to Comet—probably Atlas and all AI browsers are exposed

“somebody crafted a malicious uh chunk of text on a web page and when the AI navigated to that web page on the internet, it…”

Sander Schulhoff · 1:03:30

“So, just going to web page uh has it send secrets from my computer to someone else.”

Lenny · 1:04:00
#ai-browsers#prompt-injection#comet#data-exfiltration

Tool· 1

Tool1:05:00

CAMEL: Google's Permission-Based Defense for Agents

Schulhoff recommends CAMEL, a framework out of Google that restricts an agent's possible actions ahead of time based on what the user actually asked for. Unlike guardrails, which inspect the prompt, CAMEL works on the permission side—granting only the permissions a task needs (e.g. send-only, not read). Its limit: when a task genuinely combines read and write, the attack surface returns.

  • CAMEL restricts an agent's possible actions up front so it can't do anything malicious
  • For 'send a holiday email' it grants send permission only, not read
  • It fails when a task legitimately combines read and write permissions
  • It's a framework/concept you code into your tools, not a product—can be complex to implement

“camel is out of Google and basically what camel says is hey depending on what the user wants we might be able to restrict the…”

Sander Schulhoff · 1:05:00
#camel#agents#permissions#defense

Takeaway· 1

Takeaway1:25:30

Don't Write That Jailbreak Paper: Stop Doing Offensive Research

Schulhoff's parting advice to researchers is to stop doing offensive adversarial security research. Referencing a blog post titled 'don't write that jailbreak paper,' he argues we already know models can be broken in a thousand million ways—publishing new attacks is no longer a meaningful contribution to defense and mostly just hands attackers easier tools.

  • Don't do offensive adversarial security research on models
  • References the blog post 'don't write that jailbreak paper'
  • We already know models can be broken in countless ways
  • New jailbreaks no longer improve defense and mostly help attackers

“we know the models can be broken. We know they can be broken in a thousand million ways. We don't need to keep knowing that.”

Sander Schulhoff · 1:26:00
#research-ethics#jailbreaking#takeaway#ai-security