What a low prompt-injection success rate doesn't tell you about an agent

Vendors and internal teams increasingly summarise an agent's resistance to prompt injection with a single number: the attack success rate on a benchmark such as AgentDojo or InjecAgent. Three papers published this month suggest that number hides most of what a security reviewer needs to know. A low rate can mean the agent ignored the injection rather than recognised it; a nine-word change in the system prompt that reverses its policy on tool outputs can move the rate by tens of points; and the detectors teams put in front of agents can rank very differently once they see replayed agent tool outputs. This article summarises what the research found and how to read, or run, these evaluations more usefully.

Context: how the number is produced

Indirect prompt injection is the case where an agent reads untrusted content, such as a web page, an email or a tool result, that contains instructions meant to hijack its next actions. Benchmarks plant such instructions in tool outputs and count how often the agent carries out the attacker's action. That fraction is the attack success rate (ASR). Every other outcome counts as a successful defence.

Analysis

Failed attacks are mostly ignored, not refused

In Ignored, Not Refused, submitted on 16 October 2026, a team at BRAC University audited 33,000 trajectories from eleven open-weight agents of 2 to 14 billion parameters on InjecAgent and AgentDojo. At the baseline, pooled ASR was 8.09% on InjecAgent and 11.96% on AgentDojo, so most agents looked secure. A judge model then labelled what actually happened in the failed attacks: 76 to 80% were ignored, and only 17 to 18% refused.

The difference matters because, as the authors note, only refusal is observable: an operator can log a refusal, but an ignored injection leaves no trace. Noticing an injection is not the same as resisting it either. When the agents did recognise an injection, they still carried it out in 28.9% of those AgentDojo cases and 16.9% of the InjecAgent ones. For comparison, the authors ran one frontier model, Kimi K3, through the same judge: 96.6% and 99.0% of its failed attacks were refusals.

The paper's outcome matrix adds a second dimension. In 21.7% of AgentDojo episodes, the agent stayed safe but did not complete the user's task. A defence that stops the agent doing useful work will also score well on ASR.

The number moves with the deployment, not just the model

With the model weights unchanged, the authors varied only the system prompt. One policy told the agent that directives found in tool outputs are threats to report; another, of almost the same length, said they are legitimate tasks. InjecAgent ASR moved from 0.2% to 43%. In the other direction, a 182-word guardrail block telling agents to treat tool outputs strictly as data cut pooled InjecAgent ASR across nine agents from 9.69% to 0.95%, although two models moved the other way. Serving weights at lower precision lowered ASR slightly, but the authors attribute this to weaker agents, not better defences.

In short, an ASR describes a model inside one specific prompt, toolset and serving setup. It does not transfer to your deployment automatically.

Detector rankings don't transfer between benchmarks

Many teams add a small classifier that screens tool outputs before the agent reads them. In Passing the Test You Trained On, published on 2 October 2026, fifteen detectors, including Meta's Prompt Guard 2, were evaluated on tool outputs replayed from AgentDojo and tau-bench, and on the BIPIA benchmark. The abstract summarises the result: "the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector that catches 72% of AgentDojo injections catches 15% on tau-bench."

False positives on benign tool outputs ranged from none to over 90%, although the paper finds that false-positive rates on tool outputs do transfer between the two agent benchmarks, which makes them a more dependable thing to test first. A detector that flags ordinary tool results blocks legitimate work as well as attacks. Where training data was public, the form of the training inputs explained much of the ranking: a detector trained on agent-style inputs did best on both agent benchmarks.

Approval and sandbox settings move attack success, even with the same model

The third paper looks at coding agent harnesses rather than models. HarnessSecurity-Bench ran 2,500 trials across Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and GitHub Copilot with one fixed model, GLM-5.2. Turning on auto-approve raised attack success from 29.2% to 95.6% while also raising utility; in the baseline, a separate Qwen3.8-27B model reviewed native permission requests rather than a person. Network isolation reduced attack success from 57.1% to 0.8%, and read-only mode from 8.3% to 0.8%, a smaller drop from a low starting point, at a cost of 24.5 and 34.0 percentage points of utility. Command allowlists reduced attacks with a small utility loss, and denylists brought what the paper calls "modest utility improvements". Because the study used a single model, it cannot say how these effects compare with switching models; its authors note that different models can vary in tool use and instruction following. The authors also note that restricting a shared capability can block legitimate and malicious operations alike, and that allowed tools can still be used for harm.

What it means for engineering and security leaders

• Ask how attacks failed. A single ASR from a vendor or an internal red team is not enough to approve an agent. Ask for the split between ignored and refused, and the task completion rate alongside it.
• Treat prompts and settings as part of the security boundary. If a prompt change can move ASR from 0.2% to 43%, prompt edits deserve the same review as permission changes. This echoes Reflection's Beam safety report, where a short tool-output policy also moved results sharply.
• Don't buy a detector on its leaderboard score. Its false-positive rate on your own tool outputs may matter more than its headline detection rate.
• Approval and sandbox settings move attack success a lot, even with the same model. Auto-approve settings and sandboxing decide how much an injection can do; see our earlier pieces on sandboxing coding agents and binding approvals to effects.

Practical takeaways

1. When you evaluate an agent, record three things per run: whether the attack succeeded, whether a failed attack was refused or ignored, and whether the user's task was completed.
2. Run injection tests against your own system prompt and tools, and repeat them whenever either changes.
3. Test any detector on benign outputs from your own tools first, and set its threshold for a low false-positive rate before measuring detection.
4. Ask detector vendors what the model was trained on, and whether it overlaps with the benchmarks they quote.
5. Check which coding agents in your organisation run with auto-approve, and prefer allowlists or network limits where the utility cost is acceptable.
6. Log refusals and denials from agents where the tool supports it. A refusal is the one defensive outcome you can see.