Testing coding agents with Singapore's agentic testing starter kit

Most teams test a coding agent the way they would test a demo: a handful of tasks, one run each, and a look at whether the result seems right. Singapore's Infocomm Media Development Authority (IMDA) has published a more disciplined alternative. Its Starter Kit for Testing Agentic Systems for Safety and Reliability, version 1.0, was announced on 14 October 2026 as part of a set of AI safety initiatives. It is general guidance for any agentic system, but two of its case studies are about agents that write code, and much of the method transfers directly.

This is different from the Cyber Security Agency's handbook for assessing AI coding assistants, which we covered on 17 October. That handbook is about attack paths and security controls. The starter kit is about testing for safety and reliability: does the agent do the right thing, and can you show it?

What the starter kit covers

The kit says its focus is testing for safety and reliability, against four common risks: erroneous or unauthorised actions, biased or unfair actions, data breaches, and disruption to connected systems. It says plainly that it does not focus on cost and latency, and that it does not provide an exhaustive treatment of cybersecurity, pointing to other guidance for that. It calls itself a living document and invites feedback.

Its method has four steps:

1. Identify the relevant risks for the specific system and use case.
2. Create realistic and robust tests: the test environment and harness, what to log, and the tasks, metrics and evaluators.
3. Run tests and analyse logs.
4. Continuously monitor and re-test after deployment.

None of this will surprise anyone who tests software. The value is in the agent-specific detail under each step.

Ideas that apply to coding agents

Run the same task more than once

The kit lists three variables to plan for when deciding how many runs to do: repeated runs, because agent behaviour varies; different model and harness combinations, where users can swap models; and context window size, because "compaction may cause agentic systems to lose previous instructions". One case study states the consequence well: a single successful run was not treated as evidence that the agent is reliable, and for tasks where every miss matters, the worst run can be more informative than the average.

For coding agents, that means running each test task several times (that case study used three to five runs per case), recording the spread, and treating a model or harness change as a reason to re-run.

Log the whole trajectory, then check it against what the agent says

The kit asks testers to capture the agent's full trajectory: reasoning where available, tool calls and their results, and interactions with users. Two consistency checks follow from that:

• Claim-action consistency: did the agent actually do what it says it did? The kit's example is an agent claiming to have updated a database without a successful call to the update tool.
• Plan-action consistency: did the agent follow the plan it proposed? This matters wherever a person approves a plan before the agent executes it.

It recommends structured logs that can be parsed and searched, and names OpenTelemetry as one framework that can inform how traces, logs and metrics are collected and correlated. We looked at what coding tools emit today in observing AI coding tools with OpenTelemetry.

Test in an environment that is realistic enough to fail

The kit warns that agents can behave differently when they detect a simulation, giving placeholder names such as "John Doe" as an example of a giveaway. It also suggests testing for instructions lost after compaction by introducing large amounts of information or running long-horizon tasks.

The code-fix case study

One case study describes a code-fix agent from CyberSierra that finds security issues and control gaps, writes fixes and opens pull requests. It cannot merge them; a person approves every one. The failure modes the team looked for are familiar to anyone who has reviewed agent output:

• the agent reporting a fix that its actual diff doesn't contain;
• imports of modules that don't exist, and placeholder logic left in code;
• role names that are plausible but absent from the real schema;
• middleware that is written correctly but never wired into the request path.

The team tested in a temporary cloned workspace with the real repository structure, dependencies and schemas, on the reasoning that simplified mocks would hide exactly these failures. Checks ran in two stages: deterministic scans for things like truncated code and non-existent imports, then independent reviewer agents with access to the wider repository. Each observed failure became a new check.

The kit reports that average pull request quality rose from 2.8 to 3.8 out of 5, with no hallucinated imports in the sampled batch, and that a comparison with GitHub Copilot across more than 20 dimensions found the bespoke pipeline better on control compliance and cross-file analysis, and Copilot better on speed and reviewer experience. These are the vendor's own results, reported in the case study, not an independent evaluation.

The multi-agent case study

A second case study, from KBTG, covers three agents working in sequence on requirements, design and code. The agents never talk to each other directly; they hand artefacts over through existing work-management, document and source repositories, and generated code still goes through automated and human review. That design makes handoffs visible, and the team tracked four signals: which version of an upstream artefact was used, what changed across a handoff, who or what authorised each transition, and how often work was sent back upstream.

One line from the insights is worth pinning to any multi-agent design review: "Handoff procedures specified only through prompts may not be reliably followed." The recommended answer is technical controls, such as scoped tool permissions, controlled repositories and approval gates, before more autonomy.

What it means for engineering and security leaders

• Write down the risks first. For coding agents, the useful list is concrete: wrong code merged, actions outside the task, secrets exposed, systems disrupted.
• Budget for repeated runs and re-run when the model, harness or compaction settings change.
• Check claims against diffs and tool calls, automatically where you can. An agent's summary is not evidence.
• Test in a realistic sandbox: a cloned workspace with real dependencies, not a mock.
• Make handoffs into artefacts with versions and approvals, rather than instructions in a prompt.
• Keep testing after launch, including whether human review still catches what it should.

The kit is short enough to read in an afternoon, and its case studies are more useful than most vendor benchmarks because they describe what went wrong.

If your coding tools already send OpenTelemetry, SpanDock can receive it and forward it to the backends you choose while keeping a copy in its local store, which gives you the structured traces the kit asks for. Sending to more than one destination is on the Standard plan and up; see destinations.