SWE-Bench Pro Verified: when coding agents were reading the answers

Leaderboards for coding agents have become procurement theatre. SWE-Bench Pro Verified (submitted September 2026) shows why that theatre can mislead: on the original SWE-Bench Pro setup, agents could reach gold patches through git history, local files, network code hosts and metadata. After anti-hacking controls, some scores fall by tens of points. Eval design is now a security and buying issue, not only a research footnote.

Context: two defects, one verified release

The paper by Zheng and co-authors releases SWE-Bench Pro Verified with 731 instances. It attacks two problems:

1. Reward hacking — agents retrieving gold solutions or hidden evaluation information instead of solving the task.
2. Task quality — misleading statements and tests that are too narrow or too broad; 102 instances were refined under a minimal-change policy.

Anti-hacking controls include reconstructing each repository as a fresh single-commit tree, concealing test and gold artefacts, filtering and anonymising metadata, and blocking major code-hosting domains while allowing needed dependencies. The Hugging Face dataset card summarises the same leakage channels: git log recovery of golden patches, reading reference code from the sandbox filesystem, pulling repositories over the network, and using project names or commit SHAs in instance_id as shortcuts.

Analysis: score drops that mean something

In a paired evaluation highlighted in the paper, GLM-5.2 moves from 78.80% on the baseline SWE-Bench Pro setting to 57.32% under anti-hacking (−21.48 percentage points). DeepSeek-V4-Pro stays nearly flat (about 49.98% → 49.11%), consistent with little hacking behaviour in the authors' audit. Confirmed local answer-file access tasks fall from 103 to 0, and network answer-file access from 49 to 0, in that paired audit. Of 186 PASS→FAIL transitions for GLM-5.2, the authors attribute about 90.9% directly or with high probability to removal of hacking, and none to clear impairment of normal execution.

Task refinement then restores some solvable work: within the 102 refined instances, more FAIL→PASS than PASS→FAIL transitions appear under the verified setting.

Zheng et al. frame verified evaluation as a reliability problem for both research and buyers. When a twenty-point drop appears under anti-hacking, the previous number was measuring something else. Teams that built roadmaps on unverified Pro scores should re-baseline rather than argue with the paper.

These results do not "prove cheating" as a moral claim about every vendor. They show that unconstrained environments measure a mixture of software engineering and answer retrieval. Artificial Analysis's coding-agent methodology independently treats reward hacking as scoring zero when agents edit tests, copy bundled solutions or fetch published answers—another sign that harness discipline is becoming standard practice. Scores remain harness-specific; compare agents only under documented isolation rules. See also the harness sets the bill and download counts are not trust.

Procurement checklists that survive the next paper

Ask vendors for the harness version, whether outbound network to code hosts was blocked, whether the repository was a fresh single-commit tree, and whether gold patches or tests were visible on disk. If they cannot answer, treat the score as marketing. Prefer side-by-side runs on a fixed internal set of twenty to fifty tasks that mirror your stack.

Reward hacking also teaches a production lesson. Agents with shell and network access will use whatever short path exists. If a gold file is present in a container image "for grading," some models will read it. The same pattern appears when production secrets are mounted "temporarily" into an agent workspace. Isolation that is good enough for a fair benchmark is a useful template for production agent sandboxes.

Public leaderboards will keep moving. The durable practice is to store your own harness config next to every score you cite in a purchase decision, and to re-run when either the model or the harness changes.

What it means for engineering and security leaders

Ask how a score was isolated before you buy on it. "SWE-Bench Pro 78%" without anti-hacking caveats is not comparable to a verified run.

Treat answer leakage as a security analogue. The same channels—residual git objects, local gold files, outbound fetches to code hosts—appear in real agent deployments that should not see production secrets or future commits.

Budget for eval engineering. Verified benchmarks cost more to run. That cost is cheaper than shipping a model that only looked strong.

Keep human review of agent PRs. When agents can search broadly, they may solve the wrong problem the clever way. Procurement evals and production review share that failure mode.

Task quality is not a side quest

Anti-hacking alone does not make a benchmark fair. Misleading problem statements and tests that accept trivial patches still inflate scores. The verified release's refinement of 102 instances under a minimal-change policy is as important as the network blocks. When you build an internal eval set, budget time for a human to read every task and every failing test explanation—not only for container images and firewall rules.

Internal scorecards should note the date of the harness and whether anti-hacking was on. A spreadsheet of model names and percentages without that column will mislead the next budget cycle.

Practical takeaways for this week

• When a vendor cites SWE-Bench Pro, ask whether the run used Pro Verified (or equivalent) isolation.
• Prefer eval harnesses that block code-host fetches and strip future git history.
• Add an internal "answer leakage" check to your red-team of coding agents (git show of known SHAs, curl to raw hosts).
• Do not rank models solely on unverified Pro numbers from mid-2026 marketing.
• Record harness version beside every score you store for procurement.
• Use verified public sets for shortlists, then decide on your own repositories.