CompoundTrust

Guardrail theatre: filter, or boundary

Twelve published defences against prompt injection were tested properly last October. Most had reported near-zero attack success. Under an attacker who knew how they worked, they broke — and the defences had not got worse. The evaluation had got honest.

Episode 11 · 8 min read · Filter vs boundary · Adaptive attack · The register entry

The demonstration that always works

Someone types ignore all previous instructions. The product turns red. The room relaxes.

Last April researchers took six prompt injection filters — five open source, one from a major cloud provider — and hid the same attack inside an emoji. Every filter passed it through. A hundred per cent evasion, injections and jailbreaks alike.

Nobody in that demonstration was lying. The control worked exactly as designed. It was designed for the attack you already know about.

Bruce Schneier named this more than twenty years ago: security theatre is a measure that produces the feeling of security rather than security itself. The tell has always been the same. The measure gets evaluated against the threat its designer imagined, not against the threat that turns up.

What twelve defences did under real attack

In October 2025 a team drawn from OpenAI, Anthropic, Google DeepMind and ETH Zürich published The Attacker Moves Second. Twelve published defences. Gradient descent, reinforcement learning, random search, and five hundred human red teamers.

The majority of those defences originally reported near-zero attack success.

The human competition — twenty thousand dollars in prizes — broke every single system it was given.

Three places this shows up in an ordinary enterprise

The classifier in front of the refund tool. A customer-service assistant reads inbound email and can issue credits below a threshold. An injection classifier sits in front of it, and it is genuinely good at the obvious attempt. The register records a preventive control. What it does not record is that the classifier reads text and the refund tool takes actions, and that an attacker gets unlimited attempts to find phrasing the first will pass and the second will honour. The control is a rate reducer sitting in front of an unbounded action.

The content filter on the internal copilot. Bought to stop the assistant repeating secrets into a chat window. It catches the plain request. It does not catch the same request encoded, or split across turns, or framed as a compliance-reporting task. Meanwhile it is the reason the deployment cleared review, because somebody wrote data loss prevention: in place on a slide.

The AI firewall bought to satisfy an auditor. This is the purest case. The control was procured because a control was required. Nobody has asked what it was evaluated against, because the register entry — not the behaviour — was the deliverable. The theatre here is not the product. It is the assurance process that accepted it.

In all three the product is doing what it claims. The failure is one column over in a spreadsheet.

The tell

A control you can phrase your way past is a filter. A control you cannot phrase your way past is a boundary.

Ask what happens when the attacker knows how the control works. Publishing the design should not weaken a boundary. It reliably weakens a filter, because an adaptive attacker reads the design and rewrites the attack around it. That is the whole meaning of the title: the attacker moves second. They always do.

A static benchmark cannot see any of this. It asks whether your filter blocks a fixed list of known strings. It measures yesterday.

What I would actually do

Re-classify before you re-procure. Move every content filter and every injection classifier out of the preventive column of the control register and into detective. This takes an afternoon, costs nothing, and produces a real finding: look at what is left carrying the risk. If the honest answer is nothing, you have discovered that a deployment was signed off on the strength of something that does not do the job it was credited with.

Change the question you ask a vendor. Not does it block this. Ask whether the product was evaluated against an adaptive attacker who knew how it worked. Near-zero attack success from a static benchmark is the number twelve broken defences were all reporting.

Buy boundaries with the next budget line. Egress restriction. Capability separation. Human approval on consequential actions. None of these read text, so none of them can be rephrased past. That is the entire reason they hold.

Keep the filter. It stops the lazy attempt, it reduces noise, it raises cost. All three are real. The error was never having it — the error is what you wrote down next to it.

Three questions for a vendor

  1. Was this evaluated against an adaptive attacker who had your design, or against a fixed benchmark?
  2. What is the attack success rate when the attacker gets unlimited attempts?
  3. What still holds if your control is bypassed — and is that in my register?

If the answer to the first comes back quickly and confidently, ask to see the methodology. The gap between "we tested it" and "we let someone who knew how it worked attack it repeatedly" is the whole subject of this note.

The part that is governance, not engineering

There is a deeper limit underneath the engineering. In May 2026 a senior scientist at NIST published a peer-reviewed argument that no finite set of rules can cover the ambiguity of natural language. That is one researcher's argument in a journal rather than a standards-body proof, and a good deal of the coverage overstated it. But the sentence published alongside it is hard to argue with: there will always be ways to prompt the model that make it disregard these rules.

If that is even roughly true, then no amount of filter tuning closes the path, and the control register is where the problem actually lives.

Sources

Keep the guardrail. Move it to the detective column. Then find out what is actually holding the risk, because right now something is, and it probably is not in your register.

← All field notes