Observing AI coding tools with OpenTelemetry: what you can measure today

Most engineering organisations now pay for AI coding tools and can't say much about how they are used: which teams use them, on which models, at what cost, and with what result. That is changing because the tools themselves now emit OpenTelemetry, the same open standard most companies already use for application traces, metrics and logs. This article explains what the OpenTelemetry conventions for generative AI cover, what the coding tools actually send, and how a team can start measuring this week.

The context: GenAI semantic conventions

OpenTelemetry defines "semantic conventions": agreed names for attributes, so that a token count means the same thing whichever tool reports it. The OpenTelemetry project's own walkthrough, published in May, describes the GenAI conventions as standardising "the model being called, input and output token counts, and when opted in, the full content of prompts, completions, tool calls, and tool results."

A typical agent trace has a top-level invoke_agent span, with child chat spans for each model call and execute_tool spans for each tool call. Attributes include gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.response.finish_reasons. Two metrics, gen_ai.client.operation.duration and gen_ai.client.token.usage, cover latency and token consumption.

The conventions for agents are still marked Development in the agent spans specification, which now also defines spans for workflows, planning and tool execution. "Development" means names can still change. Dashboards built on these attributes today should expect some maintenance.

What the coding tools send

The OpenTelemetry walkthrough summarises the state of the main tools: VS Code Copilot "emits traces, metrics, and events for every agent interaction", OpenAI Codex "exports structured log events and OTel metrics", and Claude Code "exports metrics and log events via OTel, with trace support in beta."

The Claude Code monitoring documentation shows how much detail is available. It exports metrics such as claude_code.session.count, claude_code.token.usage, claude_code.cost.usage, claude_code.lines_of_code.count, claude_code.commit.count and claude_code.pull_request.count, and events for each prompt, API request, tool result and permission decision. Token counts are split into input, output, cache reads and cache writes. Its beta traces use some GenAI convention attributes, such as gen_ai.request.model, alongside tool-specific ones.

Two details matter for organisations:

• Central configuration. Administrators can set the telemetry variables through managed settings. When they do, the documentation says Claude Code removes conflicting developer-set endpoint variables at startup, so telemetry can't be redirected to a different collector.
• Repositories can't turn it on. Claude Code ignores the exporter variables in a repository's settings files, "so a repository can't use them to turn telemetry on, choose where it goes, or capture content."

Analysis: what you can and can't learn

What works well today

• Adoption and cost. Sessions, active time, tokens and cost per person, team and model are reliable and comparable over time. Custom resource attributes, such as a team or cost-centre ID, let you group them.
• Where time goes. Traces show whether a slow task was the model, a tool call or a wait for a person's approval. Claude Code records the time spent waiting on a permission decision as its own span.
• Tool and MCP usage. Tool events show which tools run, how often they fail, and which decisions people make at approval prompts.

What is harder

• Comparing tools. Each tool names its own metrics, and the shared GenAI conventions are still in development. Comparing Claude Code with Copilot means mapping two vocabularies.
• Outcomes. Telemetry shows lines changed, commits and pull requests. It doesn't show whether the change was good. Linking AI usage to review outcomes, incidents or delivery speed needs other data.
• Content. Prompts, tool arguments and responses are off by default in both the GenAI conventions and Claude Code, for good reason. Turning them on is a data-protection decision, not a configuration detail.

What is overstated

Telemetry won't by itself tell you whether AI tools make your team faster. Cost and adoption numbers are easy to collect and easy to over-read. Treat them as inputs to a conversation with teams, not as productivity scores for individuals.

What it means for engineering and security leaders

The standard is now good enough to build on. The main risks are organisational: collecting more than you need, especially content; letting each team point tools at its own collector; and building reports that rank people instead of informing decisions about tools, models and spending.

A sensible setup has one collector the organisation controls, configured centrally, metadata only by default, and a written decision about who can see what.

Practical takeaways for this week

• Turn on metrics and events for one AI coding tool on a pilot team, through managed settings rather than individual shells.
• Add resource attributes for team and cost centre so the data can be grouped without identifying more than you need.
• Leave prompt, response and tool-content capture off unless you have a specific, approved need.
• Start with three questions: who uses which tools, what does it cost per team per month, and how often do tool calls fail.
• Expect GenAI attribute names to change while the conventions are in development, and keep dashboards simple.

Where SpanDock fits

SpanDock is a self-hosted observability hub for this telemetry. Claude Code, Codex, Gemini CLI, GitHub Copilot and other tools send it OpenTelemetry; it shows every session, tool call, model call, token and dollar on a local dashboard, and forwards it to Langfuse, Grafana Cloud, New Relic or any OTLP endpoint you choose (destinations; more than one destination is on Standard and up). The getting started guide covers setting up the tools.