LLenny's Podcast
← All episodes
Sander Schulhoff21 December 2025

The coming AI security crisis (and what to do about it)

5Frameworks
15Insights

Episode overview

Sander Schulhoff argues that current AI guardrails and prompt-based defenses cannot reliably stop adaptive attackers, while automated red teaming mostly reconfirms that transformer-based systems are vulnerable. The immediate danger grows as AI shifts from read-only chatbots to agents, browsers, and robots with access to sensitive data and real-world actions. He recommends strict data and action permissions, sandboxing, monitoring, education, adaptive evaluations, and frameworks such as CaMeL, while stressing that some highly exposed agentic systems may be too risky to deploy.

Key ideas

  • The space of possible prompt attacks is effectively enormous, so static claims that a guardrail blocks 99% of attacks are not meaningful evidence of robustness.
  • Human adaptive attackers broke every tested defense in roughly 10 to 30 attempts, outperforming automated attack systems in the research Sander describes.
  • Automated red teaming works against nearly every deployed transformer-based system, but repeatedly proving vulnerability does not by itself improve defense.
  • Read-only chatbots with access only to the current user's data have limited harm potential; systems that can act or reach shared data require much stronger controls.
  • Any data available to an AI may be leaked and any permitted action may be induced, making least-privilege permissions and sandboxed execution central mitigations.
  • CaMeL can restrict an agent's permissions based on the user's requested task, but cannot fully protect workflows that legitimately require combined read and write access.
  • AI security requires practitioners who understand both model behavior and classical cybersecurity because, unlike conventional software bugs, model behavior cannot simply be patched away.
  • Frontier labs should prioritize adaptive human evaluations and earlier adversarial training, while organizations should avoid overconfidence created by guardrails.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Innovation3 steps

Adaptive Evaluation Over Static Benchmarks

Measure AI robustness with attackers that learn, not with a frozen dataset of yesterday's attacks

Read →
Innovation3 steps

CaMeL Permission Pre-Restriction

Grant an agent only the permissions its stated task needs, decided before it runs

Read →
Strategy3 steps

The AI Deployment Risk Triage

Classify any AI deployment into one of three risk tiers before spending a dollar on defense

Read →
Strategy5 steps

The AI Security Vendor Due-Diligence Test

Five questions that expose whether an AI guardrail vendor is selling real protection or theater

Read →
Mindset3 steps

The Angry-God Containment Lens

Assume the AI is a malicious agent trying to hurt you, then engineer so it structurally can't

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 4

  • Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter

    “Today my guest is Sander Schulhoff.”

    ▶ Watch at ~0:47approx.
  • Sander SchulhoffMentionsI'm an artificial intelligence researcher. I've been doing AI research for the last probably like seven years now

    “Sander is a leading researcher in the field of adversarial robustness”

    ▶ Watch at ~1:00approx.
  • Sander SchulhoffCoinedI'm an artificial intelligence researcher. I've been doing AI research for the last probably like seven years now

    “I ended up running the first ever generative AI red teaming competition.”

    ▶ Watch at ~6:30approx.
  • Alex KomoroskeMentionsAlex spent 13 years at Google where he worked on search double click he led Chrome's open web platform team for8 years

    “I asked Alex Kamaroski, who's also really big in this topic.”

    ▶ Watch at ~11:00approx.

Resources · 34

  • DatadogMentionscompany · Datadog

    “This episode is brought to you by Data Dog”

    ▶ Watch at ~2:48approx.
  • EppoMentionssoftware · Eppo

    “now home to EPO, the leading experimentation and feature flagging platform.”

    ▶ Watch at ~2:52approx.
  • AirbnbUsescompany · Brian Chesky, Joe Gebbia, and Nathan Blecharczyk

    “I saw this firsthand when I was at Airbnb, where our experimentation platform was critical for analyzing what worked”

    ▶ Watch at ~3:30approx.
  • MetronomeMentionscompany

    “This episode is brought to you by Metronome.”

    ▶ Watch at ~4:10approx.
  • Lenny's PodcastMentionspodcast · Lenny Rachitsky

    “as we saw in in the the last podcast with you”

    ▶ Watch at ~6:18approx.
  • Learn PromptingCoinedwebsite · Sander Schulhoff

    “I wrote the first guide on the internet on learn prompting.”

    ▶ Watch at ~6:30approx.
  • Generative AI Red Teaming ChallengeCoinedevent · Sander Schulhoff

    “I ended up running the first ever generative AI red teaming competition.”

    ▶ Watch at ~6:40approx.
  • Hugging FaceMentionscompany

    “We had OpenAI, Scale, Hugging Face, about 10 other AI companies sponsor it.”

    ▶ Watch at ~6:48approx.
  • Scale AIMentionscompany · Alex Wang and Lucy Guo

    “We had OpenAI, Scale, Hugging Face, about 10 other AI companies sponsor it.”

    ▶ Watch at ~6:48approx.
  • OpenAIMentionscompany

    “We had OpenAI, Scale, Hugging Face, about 10 other AI companies sponsor it.”

    ▶ Watch at ~6:48approx.
  • EMNLP 2023Mentionsevent

    “that paper went on to win best theme paper at EMNLP 2023”

    ▶ Watch at ~7:00approx.
  • ServiceNowMentionscompany

    “so service now, they have this agent that you can use on your site.”

    ▶ Watch at ~10:00approx.
  • ServiceNow Assist AIMentionssoftware · ServiceNow

    “It's called Service Now Assist AI.”

    ▶ Watch at ~10:06approx.
  • remotely.ioMentionscompany

    “was this Twitter chatbot by a company called remotely.io”

    ▶ Watch at ~12:00approx.
  • TwitterMentionswebsite · Twitter

    “they put together this chatbot to respond to people on Twitter”

    ▶ Watch at ~12:15approx.
  • MathGPTMentionswebsite

    “we had stuff like math GPT, which was a website that solve math problems for you.”

    ▶ Watch at ~12:50approx.
  • GPT-3Mentionssoftware · OpenAI

    “it would send it off to GPT3 at the time.”

    ▶ Watch at ~13:07approx.
  • ChatGPTMentionssoftware · OpenAI

    “the person behind that used chat GPT to plan out this bombing.”

    ▶ Watch at ~14:41approx.
  • Claude CodeMentionssoftware · Anthropic

    “this group was able to hijack Claude code into into performing a cyber attack basically.”

    ▶ Watch at ~16:46approx.
  • GPT-5Mentionssoftware · OpenAI

    “for a model like GPT5, the number of possible attacks is one followed by a million zeros.”

    ▶ Watch at ~31:15approx.
  • AnthropicMentionscompany

    “OpenAI, Google, Anthropic, they can't solve this problem.”

    ▶ Watch at ~36:48approx.
  • GoFundMe Giving FundsMentionsproduct · GoFundMe

    “This episode is brought to you by GoFundMe Giving Funds, the zero fee donor advised fund.”

    ▶ Watch at ~43:37approx.
  • GeminiMentionssoftware · Google

    “they could go to chat GPT or claude or Gemini and do the exact same thing.”

    ▶ Watch at ~46:04approx.
  • CometUsessoftware · Perplexity

    “with Comet, which is what I use.”

    ▶ Watch at ~1:03:56approx.
  • AtlasMentionssoftware · Atlassian

    “This is probably Atlas, probably all the AI browsers.”

    ▶ Watch at ~1:04:12approx.
  • CaMeLMentionsother · Google

    “camel is out of Google”

    ▶ Watch at ~1:05:08approx.
  • MavenUseswebsite · Maven

    “I will take this opportunity uh to to plug the Maven course we run”

    ▶ Watch at ~1:10:08approx.
  • Hack Prompt and Learn Prompting AI Security CourseRecommendscourse · Hack Prompt and Learn Prompting

    “if you want to check it out uh we got a domain at hackai.co”

    ▶ Watch at ~1:11:02approx.
  • Anthropic Constitutional ClassifiersMentionspaper · Anthropic

    “if you look at like enthropics constitutional classifiers, it's much more difficult to get like SEAB burn information out of claw models”

    ▶ Watch at ~1:12:51approx.
  • Repello AIRecommendscompany

    “Repello is one I saw that at first they seem to be doing just automated red teaming and guardrails”

    ▶ Watch at ~1:20:03approx.
  • Don't Write That Jailbreak PaperRecommendsother

    “there's a an article a blog post out there called like don't write that jailbreak paper.”

    ▶ Watch at ~1:26:04approx.
  • Apple PodcastsMentionssoftware · Apple Inc.

    “you can subscribe to the show on Apple Podcasts, Spotify or your favorite podcast app.”

    ▶ Watch at ~1:32:08approx.
  • SpotifyMentionssoftware · Spotify

    “you can subscribe to the show on Apple Podcasts, Spotify or your favorite podcast app.”

    ▶ Watch at ~1:32:08approx.
  • lennyspodcast.comMentionswebsite · Lenny Rachitsky

    “You can find all past episodes or learn more about the show at lennispodcast.com.”

    ▶ Watch at ~1:32:25approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster42:30

Prompt-Based Defenses Are the Worst of the Worst

Schulhoff debunks the idea that you can prompt-engineer your way to safety by adding instructions like 'if users try to trick you, don't follow their instructions.' He calls prompt-based defenses the worst possible approach, something the field has known since early 2023 through papers like the original HackAPrompt and Tensor Trust work.

  • Prompt-based defenses add instructions telling the model to refuse malicious input
  • They are the worst of the worst defenses, worse than guardrails
  • This has been known since early 2023 via multiple papers
  • Studied in many competitions including HackAPrompt and Tensor Trust

“Prompt based defenses are the worst of the worst defenses and we've known this since early 2023.”

Sander Schulhoff · 42:30
#prompt-defenses#myth-buster#ai-security#guardrails

Hot Take· 4

Hot Take00:00

Sander Schulhoff's Core Claim: AI Guardrails Just Don't Work

The central thesis of the conversation is that AI guardrails—the LLM-based filters companies sell to catch malicious inputs and outputs—are fundamentally insecure. Schulhoff argues that any determined attacker can get past them, and that guardrail vendors claiming to 'catch everything' are lying. He frames this as an emotional certainty backed by years of running red-teaming competitions where guardrails always broke.

  • Guardrails are typically an LLM trained or prompted to flag malicious inputs/outputs
  • Vendors who claim they 'catch everything' are lying; even 99% is meaningless at scale
  • A determined attacker will defeat a guardrail 'no problem'
  • This claim comes from running the largest AI red-teaming competitions

“AI guardrails do not work. I'm gonna say that one more time. Guardrails do not work.”

Sander Schulhoff · 00:00

“When these guardrail providers say we catch everything, that's a complete lie.”

Sander Schulhoff · 00:00
#ai-security#guardrails#prompt-injection#red-teaming
Hot Take39:00

Why Frontier Labs Won't Fix This: The Capabilities Race

Schulhoff explains why the frontier labs don't pour resources into security. One reason he's heard is that the models are still too dumb as agents to effectively carry out an attack even when tricked. The bigger driver is economics: smarter models make more money, security makes them robust but not smarter, and in a race where everyone is shipping new models, incentives favor capability over stopping rare incidents.

  • One argument: agent models are still too dumb to effectively do damage when tricked
  • Smarter models earn more money; you need intelligence before you can sell security
  • A super-secure but super-dumb model is worthless
  • The competitive race pushes incentives toward capability over rare-incident security

“the models are models being used as agents are just too dumb. Like even if you can successfully trick them into doing something bad, they're…”

Sander Schulhoff · 39:30
#frontier-labs#incentives#ai-security#hot-take
Hot Take41:30

You Can Patch a Bug, But You Can't Patch a Brain

Schulhoff's signature line captures why AI security differs fundamentally from classical cyber security. Patch a software bug and you can be 99.99% sure it's fixed; try the same in an AI system and you can be 99.99% sure the problem is still there. He attributes much of the industry's confusion to not understanding how different AI is from classical security.

  • Patching a software bug gives ~99.99% confidence it's fixed
  • Patching an AI vulnerability leaves ~99.99% confidence it's still there
  • AI security is very different from classical cyber security
  • Much of the industry's error comes from lack of knowledge about how AI works

“You can patch a bug, but you can't patch a brain.”

Sander Schulhoff · 00:00
#ai-security#cyber-security#hot-take#takeaway
Hot Take1:22:00

Prediction: A Market Correction Is Coming for AI Guardrail Companies

Asked to predict the next 6–12 months, Schulhoff forecasts a market correction as companies realize guardrails don't work. He's seen classical cyber security firms acquire AI security startups for large sums despite those startups doing little real revenue, and notes many free open-source solutions outperform the paid ones. He expects revenue for guardrail and automated red-teaming companies to dry up.

  • A market correction is likely within the next year, maybe six months
  • Big acquisitions have happened despite the AI security startups doing little revenue
  • Many free, open-source solutions are better than the deployed paid ones
  • No significant progress on adversarial robustness is expected in the next year

“I think we're going to see a market correction in the next year, maybe in the next six months where companies realize that these guardrails…”

Sander Schulhoff · 1:22:00
#ai-security-industry#prediction#market-correction#hot-take

Explainer· 3

Explainer08:30

Jailbreaking vs. Prompt Injection: The Key Difference

Schulhoff draws the crucial distinction between the two main attack vectors against LLMs. Jailbreaking is just a user and a model, tricking it into saying something it shouldn't. Prompt injection involves a developer's application prompt that a malicious user tries to override—like a story-writing site being told to ignore its instructions and output bomb-making instead.

  • Jailbreaking: just a malicious user and a model, no system prompt
  • Prompt injection: a user, a model, and a developer prompt the user tries to override
  • Example: a 'write a story' site coerced into outputting instructions for a bomb
  • There are many gray areas between the two

“So the difference is uh in jailbreaking, it's just a malicious user and a model. In prompt injection, it's a malicious user, a model, and…”

Sander Schulhoff · 09:00
#prompt-injection#jailbreaking#ai-security#explainer
Explainer28:30

Why Automated Red Teaming 'Works Too Well' and Proves Nothing

Schulhoff argues automated red teaming is nearly useless as a selling point because it always succeeds. Every transformer-based chatbot is vulnerable, thousands of red-teaming systems exist (many open source), and enterprises deploy off-the-shelf models anyway. So finding attacks that work shows nothing novel—everyone's models, including the frontier labs', can be tricked just as easily.

  • All transformer or transformer-adjacent models are vulnerable to these attacks
  • Thousands of automated red-teaming systems exist, many open source
  • Enterprises grab off-the-shelf models rather than building their own
  • Finding attacks shows nothing novel—the frontier labs' own models fail too

“So the first problem is AI red teaming works too well. It's very easy to build these systems and they just they always work against…”

Sander Schulhoff · 30:00
#red-teaming#ai-security#transformers#explainer
Explainer31:00

The Attack Space Is Basically Infinite: One Followed by a Million Zeros

Schulhoff explains why '99% effective' guardrail claims are statistically meaningless. The number of possible attacks against a model equals the number of possible prompts—for GPT-5, that's one followed by a million zeros, more than a googol. Blocking 99% still leaves basically infinite attacks, so the tests behind those figures are not statistically significant.

  • Every possible prompt is a possible attack
  • For GPT-5 the attack space is one followed by a million zeros—more than a googol
  • Blocking 99% of a near-infinite space still leaves near-infinite attacks
  • The test samples used to claim 99% are not statistically significant

“And for a model like GPT5, the number of possible attacks is one followed by a million zeros.”

Sander Schulhoff · 31:00

“That like that's so many zeros, that's more than a Google worth of zeros. Just like it's basically infinite.”

Sander Schulhoff · 31:30
#adversarial-robustness#guardrails#attack-surface#explainer

Story· 5

Story10:00

The ServiceNow Attack: Agents Recruiting Agents to Do Damage

A newly published paper described a second-order prompt injection against ServiceNow's Assist AI. A benign agent was tricked into recruiting more powerful agents to perform create/read/update/delete actions on the database and send external emails with internal data. Notably, ServiceNow's prompt injection protection feature was enabled at the time and the attacker still got through—Schulhoff calls it possibly the first case of actual damage.

  • A benign agent was used to recruit more powerful agents to carry out the attack
  • The attack could perform CRUD actions on the database and exfiltrate data via email
  • ServiceNow's prompt injection protection was enabled and the attacker still got through
  • May be the first instance of an AI security attack with actual damage

“I instructed a seemingly benign agent to recruit more powerful agents in fulfilling a malicious and unintended attack, including performing create, read, update, and delete…”

10:30
#prompt-injection#agents#servicenow#case-study
Story11:30

The First Public Prompt Injections: Remotely.io and MathGPT

Schulhoff recounts the earliest public prompt injection incidents. Remotely.io's pro-remote-work Twitter chatbot was tricked into making threats against the president and hateful speech, and the company later went out of business. MathGPT, which solved math problems by writing and executing code on its own server, was manipulated into writing malicious code that exfiltrated the OpenAI API key.

  • Remotely.io's Twitter bot was tricked into threatening the president and spewing hate speech
  • The company shut the bot down and appears to no longer be in business
  • MathGPT executed AI-written code on the same server the app ran on
  • An attacker got it to write code that exfiltrated the OpenAI API key (responsibly disclosed)

“And so now you had this company chatbot just like spewing threats against the president and other hateful speech on Twitter.”

Sander Schulhoff · 12:30
#prompt-injection#case-study#chatbots#api-keys
Story16:00

How Attackers Hijacked Claude Code by Splitting Their Requests

Schulhoff explains the recent Claude Code cyberattack, where an AI was turned into a self-directing virus that sends out API requests to decide what to do next. The trick to bypassing defenses was separating a malicious task into smaller, individually-legitimate requests—one instance to discover a system's backend, a new instance to 'hack my own system'—that only become malicious when combined.

  • An AI-powered virus can think for itself and issue API requests to decide next steps
  • Directly asking Claude Code to hack a site gets refused
  • Splitting the task across separate instances bypasses the defense
  • Requests that look legitimate alone are malicious only when put together

“So, a a lot of the way they got around these these defenses was by just kind of separating their requests into smaller requests that…”

Sander Schulhoff · 17:30
#claude-code#jailbreaking#cyberattack#agents
Story33:30

The Research: Humans Break 100% of Defenses in 10–30 Attempts

Schulhoff describes a major research paper done with OpenAI, Google DeepMind, and Anthropic that threw adaptive attacks and human attackers at all state-of-the-art models and defenses, including GPT-5. Humans broke 100% of defenses in roughly 10–30 attempts, while automated systems needed orders of magnitude more tries and topped out around 90%. Humans remain the best attackers.

  • The paper was done alongside OpenAI, Google DeepMind, and Anthropic
  • Humans broke 100% of defenses in roughly 10 to 30 attempts
  • Automated systems needed orders of magnitude more attempts and only beat ~90%
  • Human attackers are still the best, contrary to hopes of full automation

“we found that uh first of all humans break everything 100% of of the defenses in maybe like 10 to 30 attempts.”

Sander Schulhoff · 33:30
#red-teaming#research#adversarial-robustness#human-attackers
Story1:03:30

The Comet Browser Incident: Just Visiting a Web Page Leaked Data

Schulhoff describes a real prompt injection against the Comet AI browser. Someone planted a malicious chunk of text on a web page, and when the AI navigated there, it was tricked into exfiltrating the main user's data and account data. Lenny reacts with alarm since Comet is the browser he uses—and Schulhoff notes this likely applies to Atlas and all AI browsers.

  • A malicious chunk of text was crafted on a web page
  • When the AI browser navigated to it, it leaked the main user's data and account data
  • Simply visiting a web page can send your secrets to someone else
  • This isn't unique to Comet—probably Atlas and all AI browsers are exposed

“somebody crafted a malicious uh chunk of text on a web page and when the AI navigated to that web page on the internet, it…”

Sander Schulhoff · 1:03:30

“So, just going to web page uh has it send secrets from my computer to someone else.”

Lenny · 1:04:00
#ai-browsers#prompt-injection#comet#data-exfiltration

Tool· 1

Tool1:05:00

CAMEL: Google's Permission-Based Defense for Agents

Schulhoff recommends CAMEL, a framework out of Google that restricts an agent's possible actions ahead of time based on what the user actually asked for. Unlike guardrails, which inspect the prompt, CAMEL works on the permission side—granting only the permissions a task needs (e.g. send-only, not read). Its limit: when a task genuinely combines read and write, the attack surface returns.

  • CAMEL restricts an agent's possible actions up front so it can't do anything malicious
  • For 'send a holiday email' it grants send permission only, not read
  • It fails when a task legitimately combines read and write permissions
  • It's a framework/concept you code into your tools, not a product—can be complex to implement

“camel is out of Google and basically what camel says is hey depending on what the user wants we might be able to restrict the…”

Sander Schulhoff · 1:05:00
#camel#agents#permissions#defense

Takeaway· 1

Takeaway1:25:30

Don't Write That Jailbreak Paper: Stop Doing Offensive Research

Schulhoff's parting advice to researchers is to stop doing offensive adversarial security research. Referencing a blog post titled 'don't write that jailbreak paper,' he argues we already know models can be broken in a thousand million ways—publishing new attacks is no longer a meaningful contribution to defense and mostly just hands attackers easier tools.

  • Don't do offensive adversarial security research on models
  • References the blog post 'don't write that jailbreak paper'
  • We already know models can be broken in countless ways
  • New jailbreaks no longer improve defense and mostly help attackers

“we know the models can be broken. We know they can be broken in a thousand million ways. We don't need to keep knowing that.”

Sander Schulhoff · 1:26:00
#research-ethics#jailbreaking#takeaway#ai-security