Skip to content
Beta
All articles

Agent swarms as prompt injection attackers: why benchmark scores near zero mislead

A Google DeepMind report used teams of AI agents to search for prompt injections. Even the most robust model it tested fell for a third of them.

By The SpanDock team

5 min read

A bar chart of mean attack resistance across defender models falling from 42.1% to 23.1% as the attacking model gets stronger, from the report's Table 2

Prompt injection benchmarks have started to look solved. Some frontier models now resist almost every human-written attack in public arenas. A technical report from Google DeepMind, published on arXiv on 9 November 2026, argues that this is a measurement problem, not a solution. Its authors built an attacker out of a swarm of cooperating AI agents and found that no model they tested was fully robust, and that attacks got markedly better as the attacking model got stronger. For anyone relying on a model's built-in resistance to keep an agent safe, the findings are worth understanding.

The claim being tested

Prompt injection is when instructions planted in data an agent reads, such as a web page, an issue or a tool result, steer what the agent does. The OWASP entry on prompt injection puts the underlying difficulty bluntly: "it is unclear if there are fool-proof methods of prevention".

The report's starting point is that static attacks stopped working on frontier models by mid-2025, and that human red-teaming in the GraySwan arena is now close to saturation, with recent Gemini models and some Claude versions scoring under 1% there. Earlier work, notably Nasr et al.'s "The Attacker Moves Second", had already shown that adaptive attackers, ones that change strategy to counter a defence, bypassed 12 published defences with success rates above 90% for most, though most had reported near-zero attack success. The new report asks whether the same holds for today's models.

How the experiment worked

A dataset that separates obedience from security knowledge

The authors built 100 synthetic tool-use scenarios from 249 tools across 53 domains. In each, the agent works on a user's task in one domain, and the injection, placed in the last tool result, tries to make it call a tool from an unrelated domain. In their example, an agent booking a tutoring session is pushed into logging someone's vital signs.

The attacker's goal is deliberately harmless. The authors argue that this measures whether a model respects the instruction hierarchy, rather than whether it recognises an obviously malicious request. They also note that harmful actions can be composed from benign-looking steps. Scoring is strict: an attack counts only if the model calls the exact target tool with every argument key, exact numbers and near-identical strings.

The attacker: teams that share notes

The attacker, which the authors call an Agentic Optimizer, ran 8 teams of 2 agents. Each team got a starting strategy from a database of strategies that had worked before. Agents tried candidate injections, read team and global leaderboards, and posted notes to shared whiteboards. Each agent had at most 20 evaluation calls, so 320 in total per scenario.

The main metric is the attack resistance rate (ARR): the share of scenarios where the attacker found no working injection. ARR@150 counts only the first 150 queries to the defending model.

One condition matters for interpreting the numbers: all Gemini models were queried "without any safety classifiers and with disabled out-of-model prompt injection defenses". The results describe models on their own, not Google's production systems.

What they found

No model was fully robust. Against the strongest attacker, Gemini 4 Argon, the most robust defender, also Gemini 4 Argon, resisted 87% of scenarios within 150 queries and 66% at the full budget. That means the attacker found a working injection in roughly a third of scenarios.

Open-weight models resisted almost nothing. Kimi K3, GLM-5.3 and DeepSeek-V4.1-Flash each had an ARR of 1% at the full budget against the strongest attacker. The authors describe their resistance as "remarkably low" given their capability scores.

Stronger attackers win more. Averaged across defenders, ARR@150 fell from 42.1% with Gemini 3.6 Flash as the attacker to 23.1% with Gemini 4 Argon. The authors found that stronger attackers converged on promising strategies faster, framed the target call as a data dependency of the legitimate task, and reworded injections to avoid suspicion. They also report a link between attack strength and a model's score on an exploit-development benchmark.

Visible reasoning helps the attacker. When the defender's chain of thought was shown to the attacker, resistance dropped further.

The method beats the reproducible baseline. On 80 AgentDojo scenarios, the open-source Prompt Siren attack broke 2, while the swarm found 6 within 25 queries.

Analysis: what the numbers do and don't say

The headline is not that a particular model is unsafe. It is that a near-zero score on a human-written benchmark says little about an adaptive, automated attacker, and that attacker capability rises with model capability. Today's best model is tomorrow's attacker.

Some caution is due. The scenarios are synthetic and single-turn: the attack succeeds or fails in the one model turn after the poisoned result. Real agents have more context, more tools and sometimes more defences. The Gemini models ran with their external defences off, so production exposure is likely lower than these figures. And the target actions are benign by design, so these are measurements of obedience, not of harm.

Even so, the direction matches earlier work and our own reading of what low success rates hide. The authors' conclusion is the practical one: defences "should perhaps not rely on model-hardening alone, and should involve a full stack with defenses in depth".

What it means for engineering and security leaders

If your agent's safety case rests on "the model resists injection", this report says that claim decays as attackers improve. Assume some injections land, and design so that a hijacked tool call has limited reach. The OWASP mitigations point the same way: least privilege, functions handled in code rather than by the model, and human approval for high-risk actions.

The open-weight results deserve particular attention. Teams choosing open-weight models for cost or data control should not expect them to resist injection on their own, and should put stricter limits on what those agents can call.

Practical takeaways

  • Treat model robustness as one layer. Put tool permissions, approvals and output checks outside the model.
  • Scope tools per task. In the report's attacks, the target tool was available but unrelated to the task. If it had not been available, there was nothing to hijack.
  • Hide reasoning from untrusted channels where your stack exposes it.
  • Red-team adaptively. Static payload lists understate risk; budget for automated, iterative testing.
  • Be stricter with open-weight agents, especially on tools that write, send or delete.

Keeping unrelated tools out of reach is something SpanDock's MCP gateway does on Premium and Enterprise: each person gets only the connectors and tools they are allowed, read-only access hides write and destructive tools completely, and tools that need approval don't run until an administrator approves the call.

Sources

  1. Kurakin et al.: Scaling Adaptive Prompt Injection Attacks with Agent Swarms (arXiv, 9 November 2026)
  2. Nasr et al.: The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections (arXiv)
  3. OWASP GenAI Security Project: LLM01 Prompt Injection
Cookie categories

Details: Cookie Policy.