What a low prompt-injection success rate doesn't tell you about an agent
New research finds low attack success rates mostly mean agents ignored injections, and detector rankings don't transfer. How to read these evaluations.
5 min read
Blog tag
#Evaluation
Measuring model and agent quality: benchmarks, evals and their limits.
DeepSeek's 1017 update and the model name that moves under you
DeepSeek swapped the model behind deepseek-flash on 17 October, and legacy names followed. Why aliases change agent behaviour, and how to pin and re-test.
Reflection's Beam: what a 501B open-weight model's safety report tells agent teams
Beam arrived on 16 October under Apache 2.0 with a long safety report. Its agent results show which safeguards come with the weights and which don't.
Three Copilot models retire on 16 November: what to check before then
Claude Opus 4.8, Claude Opus 5 and GPT-5.6 Terra leave GitHub Copilot on 16 November 2026. What the replacements cost, and a routine for the next retirement.
Open-weight model risk, model layer first: the Microsoft and NVIDIA guidance
Microsoft and NVIDIA published a joint risk taxonomy for open-weight models. What it covers, what it leaves for later, and how coding teams can use it.
Muse Spark Open: what an 816B open-weight coding model asks of you
Meta released weights for Muse Spark Open, an 816B mixture-of-experts model for coding agents: the hardware it needs, its license and its own benchmarks.
Terminal-Bench 4.0 and honest comparisons of coding agents
Terminal-Bench 4.0 is a new task set, not a harder 2.1, and vendor and independent scores already differ. How to read the numbers before they reach a decision.
4 min read
Sol, Sonnet 5.5 or Haiku 5.5: a practical coding model map
Three coding-relevant models landed within ten days. Rather than picking one default, map task types to model tiers and check vendor benchmarks with care.
Claude Sonnet 5.5: faster mid-tier coding without a list-price rise
Sonnet 5.5 kept Sonnet 5's $2 and $10 token prices, then had its cache-read price halved. What that does to coding costs, and where Opus 5.5 still leads.
5 min read
Gemini 4 Argon is generally available: price, limits and the 1 December change
Gemini 4 Argon launched on 13 October 2026 at an introductory API price that doubles on 1 December. What it costs, what Google claims, and what to plan for.
6 min read
SWE-Bench Pro Verified: when coding agents were reading the answers
SWE-Bench Pro Verified closes gold-patch leakage channels and refines 102 tasks. Some model scores fall sharply once agents can no longer fetch the solution.
4 min read
GPT-6.1 Sol: near-Astra coding performance at one-fifth the token price
OpenAI's GPT-6.1 Sol lists at $2/$10 per million tokens against Astra's $10/$50. What that changes for default coding models, caching and harness cost.
Browse by tag
- #Security (54)
- #Agents (40)
- #AI coding tools (37)
- #LLM costs (23)
- #MCP (16)
- #Evaluation (14)
- #Observability (12)
- #Compliance (11)
- #Enterprise (9)
- #Platform engineering (9)
- #Data privacy (8)
- #DevOps (7)
- #Model routing (7)
- #Open-weight models (7)
- #Research (7)
- #Tutorials (6)
- #Code review (4)
- #Developer productivity (4)
- #Guardrails (4)
- #Networking (4)
- #OpenTelemetry (4)
- #Budgets (3)
- #Prompt engineering (3)
- #Self-hosting (3)
- #MCP gateway (2)
- #Release notes (2)
- #Skills (1)
- #Testing (1)