Skip to content
Beta
All articles

Repository context and secure AI code: what CASE-Bench found

A new benchmark of 145 tasks from real CVEs finds that context built for correctness rarely shows coding models the project's safeguards. What to change.

By The SpanDock team

4 min read

A diagram of a repository where a coding agent's search finds similar code near the target file but rarely reaches the access-control module that holds the safeguard

AI coding tools don't write code from the prompt alone. Before generating anything, they gather context from the repository: similar functions, imported definitions, nearby files. A study published on arXiv on 4 November 2026, A.S.E 2.0, measures how that context affects the security of the code they produce. Its main finding is uncomfortable for anyone relying on "the agent can see the whole codebase": context built to make code work barely makes it safer, and coding agents found the relevant safeguard in only 27 to 36% of their runs.

What the study built

The authors, from Nanyang Technological University, Tencent, Zhejiang University and the Chinese Academy of Sciences, built CASE-Bench: 145 repository-level tasks derived from 144 real CVEs in 51 open-source projects, covering 28 weakness types in seven languages. Each task runs in Docker, with functional tests and a proof-of-concept exploit checked against both the vulnerable and the patched version. Two experts annotated, for each task, the repository code that enforces security (a validator, a permission check) or undermines it (an unfixed sibling with the same flaw).

That design separates two outcomes that are often blurred: code that works, and code that works and resists the exploit. The study also tracks a third: code that passes the functional tests but is exploitable.

They held the task, prompt and security check fixed and varied only the context, comparing 13 ways of building context and six controls with six models (three proprietary, three open-weight). They also ran SWE-agent with two models and Claude Code, three times per task, in an isolated sandbox. In total the study used about 99,000 model generations and 1,305 agent runs.

What they found

  • Correctness context doesn't buy much security. Context built to help with functionality raised the functional pass rate by up to 8.2 percentage points but the secure pass rate by at most 3.7 points.
  • Missing safeguards hurt. Removing a safeguard from the context lowered the secure pass rate by up to 8.1 points.
  • One bad example spreads. Inserting a single real insecure example into the context raised the rate of correct-but-exploitable code by 6.3 points, even when a safeguard was also present.
  • Agents search for features, not policies. Although agents saw 16 to 26 times as many repository tokens as an 8,000-token context, they observed the relevant safeguard in only 27 to 36% of runs, and an insecure example in 43 to 51% of runs on tasks that contained one. Runs that did see the safeguard were more often secure.

The authors' explanation is simple. Retrieval and agent search are driven by what the code should do, so they find code that looks similar. Safeguards often live elsewhere, in a central access-control module or a shared validator, and don't resemble the target. Unfixed siblings, on the other hand, resemble it closely.

How much weight to put on it

The paper is careful about its limits. A proof-of-concept exploit can show a vulnerability but cannot prove its absence, so "secure" means "resisted this exploit". A manual audit agreed with the automated verdict on 95.7% of generations. Six projects contribute 67 of the 145 tasks, differences below about 5.6 points may go undetected, and training data contamination cannot be ruled out. The agent analysis is observational.

The paper also reports that static analysis, used as the security check, missed 40% of the generations the exploit broke and flagged 27% of those that resisted it. That is a useful reminder that a static scanner alone gives a different picture of AI-written code than an exploit does.

None of this says AI-written code is worse than human-written code; the study doesn't compare the two. It says the context a tool assembles shapes the security of its output, and that the context most tools assemble is not designed for that.

What it means for engineering and security leaders

If a codebase's security depends on calling the right helper, such as an authorisation check, an input validator or a safe API wrapper, an AI tool will often not find it unless something points it there. That makes the location and naming of safeguards part of how well AI tools work in a repository.

It also changes where review effort goes. The OWASP Secure Coding with AI cheat sheet already advises reviewing every file in an agent-generated pull request and not approving based on the description alone. This study suggests a specific question for that review: does the new code call the project's own safeguard for this kind of operation, or did it copy a nearby pattern that skips it?

And it argues for checking examples before they spread. Legacy code with a known weakness is exactly what similarity search returns. Marking or fixing such code reduces the chance that an agent uses it as a template.

Practical takeaways

  1. Point tools at your safeguards. In the instruction files your AI tools read, name the modules that hold access control, validation and safe wrappers, and say when they must be used.
  2. Keep safeguards findable. Clear names and one obvious place for each kind of check help both people and search-driven agents.
  3. Fix or flag insecure siblings. Old code with the same weakness as a recent CVE is likely to be copied; fix it, or mark it clearly as not to be reused.
  4. Review for the safeguard, not only the diff. For security-sensitive changes, ask whether the code uses the project's own checks.
  5. Test with exploits where you can. Static scanning misses some exploitable code and flags some safe code; exploit-based tests for high-risk paths give a sharper signal.
  6. Treat context as a variable when you evaluate tools. A tool that does well on correctness benchmarks may not do as well on security in your repository.

For a related result on how coding agents behave on benchmarks, see our article on SWE-Bench Pro Verified and reward hacking.

Sources

  1. A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code (arXiv 2611.06294, 4 November 2026)
  2. OWASP Cheat Sheet Series: Secure Coding with AI
Cookie categories

Details: Cookie Policy.