open-telemetry

AI agent observability with OpenTelemetry: what traces show, and what they cannot prove Most teams running AI agents in production already trace them. If that tracing is built on OpenTelemetry, there is now a shared vocabulary for it: the GenAI semantic conventions, which define spans for model calls, agent invocations and tool executions. This piece walks through what those conventions record about an agent, what they leave out on purpose, and why a trace that answers "what went wrong?" is a different thing from a record that answers "show me what your agent did" when the person asking is a customer or an auditor. Where the conventions live now The GenAI conventions started inside OpenTelemetry's main semantic-conventions repository. They have since moved to a dedicated repository, and the old pages now say only that the conventions "have moved". Every document in the set is marked Status: Development. The new repository has not cut a release yet, and its pending changelog includes breaking changes, such as removing a required attribute from the invoke_agent internal span. If you build dashboards or alerts on these attribute names, pin the version your instrumentation emits and expect to migrate. What an agent trace contains The agent spans document defines spans for creating an agent, invoking one, running a workflow, planning, and executing a tool. Tool execution has refinements for specific cases: a tool call that hands work to another agent, loading an agent skill, and a "command execution span" that "describes a tool call that executes one or more commands". Separate documents cover MCP and Agent2Agent calls. The command execution span is the one most relevant to accountability, because it covers the moment an agent touches a real system. Its required attributes are gen_ai.operation.name and gen_ai.tool.name. Conditionally required ones include the agent's name, the tool call ID, process.executable.name and process.exit.code. The span name should take the form execute_tool {gen_ai.tool.name} {process.executable.name} when the executable is known. An illustrative span for an agent that deletes something through a shell tool, with default settings, looks like this: execute_tool bash rm gen_ai.operation.name = execute_tool gen_ai.tool.name = bash gen_ai.agent.name = release-bot gen_ai.tool.call.id = call_7f3a... process.executable.name = rm process.exit.code = 0 That is useful. You can see which agent ran which binary, whether it succeeded, how long it took and where it sat in the agent's run. What you cannot see is what rm was told to delete. Content is off by default, on purpose The arguments and result of a tool call are carried by gen_ai.tool.call.arguments and gen_ai.tool.call.result. Both are Opt-In, and both carry the warning "This attribute may contain sensitive information." The model spans document sets the general rule: "Model instructions, user messages, and model outputs are considered sensitive and are often large in size", and "OpenTelemetry instrumentations SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in." That option, where it is an environment variable, must be called OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT. Its values are NO_CONTENT (the default), SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT, and the setting governs tool-call arguments and results as well as prompts and completions. The convention describes three patterns: record nothing (the default), record content on the spans, which it suggests for cases such as "pre-production environments", or upload content to external storage and put a reference on the span, which it recommends for production environments where volume or sensitive data is a concern. For that last pattern it is explicit that it "does not prescribe the URI scheme, object layout, serialization format, retention policy, or access mechanism." This is the right default for telemetry. Prompts and tool results routinely contain personal data, and an observability backend is usually readable by far more people than should see it. But it means a default agent trace records that a destructive action happened, without recording what the action was. Observability is allowed to forget Traces are also sampled. OpenTelemetry's sampling guide describes head sampling, where a decision "is made based on the trace ID and the desired percentage of traces to sample", giving its example as "5% of all traces". Its own list of reasons not to sample includes being "bound by circumstances such as regulation that prohibit dropping data", where unsampled data cannot be routed to cheaper storage instead. The span format itself, in the OTLP trace definition, has counters for attributes, events and links that were dropped. None of that is a flaw. Observability exists to answer aggregate and diagnostic questions cheaply, and losing a fraction of the data is an acceptable trade. One detail in the GenAI conventions is worth knowing here: if an external content upload hook is configured, instrumentations "SHOULD invoke it regardless of the span sampling decision". So the content can survive even when the span that refers to it was dropped. Check what your own pipeline does before relying on either. Nothing in a span is signed The OTLP Span message holds IDs, a name, start and end times, attributes, events, links and a status. It has no signature field, and the OTLP specification deals with delivering and acknowledging telemetry, not with proving its integrity. The start time is, in the proto's own words, "the time kept by the local machine where the span execution starts". The W3C Trace Context recommendation, which OpenTelemetry's default propagator settings use to link spans across services, is frank about abuse. Its security section warns that when a service with a public API naively continues incoming traces, an attacker could "forge trace-id collisions that make monitoring data unusable". Put together, a trace is a set of statements that the instrumented process made about itself, timed by its own clock, passed through a collector and stored in a backend the operator controls. Anything that can run code in that process, or write to that backend, can add, change or drop spans. For debugging, that is fine: you trust your own telemetry because you have nothing to gain from lying to yourself. When the question comes from outside, after an agent has deleted a customer's data or been steered by injected instructions, the trace is the operator vouching for the operator. > A quick test for your tracing set-up: pick one destructive tool call your agents made last week. Can you show the exact arguments it ran with? Do you know that its trace was sampled? Could anyone with backend access have edited the span since? If someone outside your organisation asked, what would you hand them, and how would they check it without trusting you? Traces and signed records side by side A signed execution record answers a different question. The Traceseal receipt specification, for example, defines a JSON document in which "an operator attests that a specific piece of code ran inside a specific sandbox and produced a specific result", signed with ed25519 and checkable by a third party "without access to the operator's machine, the original code, or any private keys". Compared point by point: - Purpose. A trace is for finding out what went wrong. A receipt is for showing someone else what ran. - Coverage. Traces are often sampled. A signed record only helps if one exists for the run in question, so our advice is not to sample them for consequential actions. - Content. Traces carry plaintext content only if you opt in. A receipt carries hashes of inputs and outputs, so the record can be shared while the content stays private; if a verifier later needs the content, the operator supplies it and the verifier recomputes the hash. - Integrity. A span is unsigned. A receipt is signed, so any change after signing is detectable. - Time. A span's time is the local clock. A receipt's time is also only the signer's claim. Fixing when a record existed needs something outside the operator, such as a trusted timestamp or a witnessed log covering it. To be exact about the limits: a receipt is not proof that the operator told the truth. The specification says so: it "proves that the operator's key signed the attestation. It does NOT prove the operator is honest." It also records what the operator claims the sandbox was, not that the sandbox enforced it, and hashes of short or guessable values can be reversed by guessing. What a receipt adds is that the claim is fixed, attributable and checkable by someone else. Who holds the signing key matters as much as the signature; if the agent can reach the key, it can sign its own account of events, which is why the sandbox and the signer should be separate. Using both There is no need to choose. Keep OpenTelemetry for what it is good at: latency, cost, error rates and debugging individual runs. For the actions that would matter in a dispute (deleting data, sending messages, moving money, changing production), produce a signed record outside the agent's process as well. Then link the two. The GenAI conventions do not define an attribute for an external evidence record, so add one under your own namespace, for example the receipt's hash on the execute_tool span. An investigator can then start from the trace, which is easy to search, and end at the receipt, which can be checked. For what to capture on each tool call in the first place, our piece on MCP tool-call logging goes into the fields, and the audit trail piece covers retention and who should hold the record. What to take from this - The OpenTelemetry GenAI conventions are a good, still-moving baseline. They now have agent, tool, command-execution, MCP and A2A spans. Pin your version. - Defaults record that a tool ran, not what it did. Arguments and results are opt-in, for sound privacy reasons. Decide deliberately where that content goes. - Traces are sampled, unsigned and timed by the local clock. That suits debugging and does not make them evidence. - Link traces to signed records for consequential actions. Searchable on one side, verifiable on the other. The Traceseal receipt format, verifier and log are open: the spec, the verifier on PyPI, and the public log. Originally published at traceseal.io.

Observability for AI Agents: Tracing, Evaluation and Cost Control in Production How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices. 1. Why Is Observability Different for AI Agents? Traditional backend observability is built around relatively predictable execution paths. A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred. AI agents introduce a different level of complexity. An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input. Consider a relatively simple task: A user asks an AI assistant to analyze a merge request and identify potential problems. Behind the scenes, the application might: 1. Receive the user's request. 2. Invoke a coordinator agent. 3. Call an LLM to determine the next action. 4. Retrieve the merge request from GitLab. 5. Read the changed files. 6. Delegate security analysis to another agent. 7. Execute additional tool calls. 8. Evaluate the collected results. 9. Generate a final report. The user sees one response. The application has actually executed a multi-step workflow involving several components. Now imagine the response takes 45 seconds instead of the expected 8 seconds. Where should we investigate? - Did the model take too long to respond? - Did a tool call time out? - Was the same operation executed repeatedly? - Did the agent generate unnecessary steps? - Did a subagent introduce additional latency? - Was too much context sent to the model? - Did a retry increase the overall execution time and cost? Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why. AI observability is about understanding the entire execution path, not just measuring the final API response. This is where OpenTelemetry becomes particularly useful. 2. What Is OpenTelemetry? <escape>**</escape><Entity value="OpenTelemetry (OTel)" category="software" disambig="Open-source observability framework"/> is an open-source observability framework for collecting, processing, and exporting telemetry data. It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms. Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported. OpenTelemetry is built around three primary signals: | Signal | What it tells us | AI agent example | |---|---|---| | Traces | How an operation was executed | Model call → tool → subagent | | Metrics | How the system performs over time | Token usage, latency, error rate | | Logs | What happened at a specific point | Tool timeout or agent exception | These signals complement each other. A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation. OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector. An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal. References: OpenTelemetry Documentation · OpenTelemetry JavaScript 3. Understanding Distributed Tracing Before discussing AI-specific instrumentation, we need to understand traces and spans. A trace represents the complete execution path of an operation. A span represents one individual unit of work within that execution. For example: Trace: POST /api/agent | +-- HTTP Request | +-- Coordinator Agent | +-- LLM Generation | +-- Tool: GitLab - Get Merge Request | +-- Tool: GitLab - Get Changed Files | +-- Security Agent | | | +-- LLM Generation | | | +-- Tool: Security Analysis | +-- Testing Agent | | | +-- LLM Generation | | | +-- Tool: Test Coverage | +-- Final Response Generation Each span has information such as: - Trace ID - Span ID - Parent span ID - Start and end timestamps - Duration - Attributes - Events - Status and exception information This creates a hierarchical representation of the complete execution. The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them. For AI agents, this distinction is important because one user request can generate many independent operations. Why is context propagation important? In distributed systems, a trace often crosses multiple services. Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service. Without context propagation, every service may generate its own independent trace. We would see several unrelated operations instead of one complete workflow. OpenTelemetry uses W3C Trace Context as its standard propagation format. The traceparent header carries trace and parent-span information between instrumented services. The same principle applies to agent-based architectures. When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution. Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction. The important point is that protocol-level interoperability and observability context propagation are separate concerns. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace. References: OpenTelemetry Traces · Context Propagation · W3C Trace Context 4. OpenTelemetry and GenAI Semantic Conventions One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers. For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes. This makes it difficult to compare different implementations using the same monitoring platform. OpenTelemetry addresses this through Semantic Conventions. Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently. The GenAI semantic conventions extend this approach to generative AI applications. Some important attributes include: | Attribute | Purpose | |---|---| | gen_ai.provider.name | Identifies the model provider | | gen_ai.request.model | Identifies the requested model | | gen_ai.response.model | Identifies the actual response model | | gen_ai.operation.name | Describes the operation | | gen_ai.usage.input_tokens | Input token usage | | gen_ai.usage.output_tokens | Output token usage | | gen_ai.conversation.id | Associates operations with a conversation | | gen_ai.agent.name | Identifies the agent | | gen_ai.tool.name | Identifies the executed tool | For example, an LLM invocation could be represented by: { "gen_ai.operation.name": "chat", "gen_ai.provider.name": "openai", "gen_ai.request.model": "example-model", "gen_ai.usage.input_tokens": 1200, "gen_ai.usage.output_tokens": 350 } This is a simplified span attribute example. The benefit is that observability platforms do not need to understand a different custom schema for every application. Agent-specific conventions The GenAI semantic conventions also define operations for agents and workflows. Some examples include: - invoke_agent - invoke_workflow - execute_tool - plan - retrieval This is particularly valuable for multi-agent architectures. We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents. For example: invoke_workflow merge-request-review | +-- invoke_agent coordinator | | | +-- chat model | +-- execute_tool fetch_merge_request | +-- invoke_agent security-agent | | | +-- chat model | +-- execute_tool analyze_vulnerabilities | +-- invoke_agent testing-agent | | | +-- chat model | +-- execute_tool analyze_test_coverage | +-- Final Aggregation This makes the execution model much easier to understand. An important maturity consideration: As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract. The current convention documents include: - Semantic Conventions for GenAI Spans - Semantic Conventions for Agent Spans - Semantic Conventions for GenAI Metrics 5. Integrating OpenTelemetry with Vercel AI SDK Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration. The AI SDK already supports OpenTelemetry-based instrumentation. It can generate spans for: - Text generation - Streaming generation - Individual model calls - Agent execution - Tool execution - Embedding operations - Reranking This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility. Step 1: Install the packages For an AI SDK v7 application: pnpm add @ai-sdk/otel @vercel/otel The OpenTelemetry integration is provided through @ai-sdk/otel. This is an important version-specific detail. In AI SDK v7, OpenTelemetry integration moved out of the core ai package. Older examples using the v6 experimental_telemetry configuration should not be copied directly into a v7 application. Step 2: Register telemetry in Next.js Create an instrumentation.ts file in the project root. import { registerOTel } from '@vercel/otel'; import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; export function register() { registerOTel({ serviceName: 'ai-agent-platform', }); registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); } There are two responsibilities here: - registerOTel initializes the OpenTelemetry infrastructure for the Next.js application. - registerTelemetry registers the AI SDK-specific integration that creates GenAI spans. I have disabled input and output capture in this example intentionally. AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision. We will return to this in the security section. For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup. Step 3: Instrument AI SDK calls Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration. We can also provide a functionId to identify the business operation or agent. import { generateText } from 'ai'; const result = await generateText({ model: myModel, instructions: ` You are a financial analysis assistant. Analyze the supplied information carefully. `, prompt: 'Analyze the latest portfolio performance.', telemetry: { functionId: 'portfolio-analysis-agent', }, }); The functionId helps distinguish operations in traces. For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as: invoke_agent portfolio-analysis-agent Step 4: Understand the generated spans The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions. For generateText and streamText, the integration records three main types of spans: 1. Agent invocation span The root span represents the complete generation operation, including its execution steps and tool calls. Example: invoke_agent portfolio-analysis-agent 2. Step and model-call spans Each model interaction is represented by step and provider-call spans. These allow us to understand how many model calls occurred during an operation and how long each one took. 3. Tool execution spans Each client-side tool execution receives its own span. For example: invoke_agent portfolio-analysis-agent | +-- step 0 | | | +-- chat example-model | +-- execute_tool get_portfolio | +-- step 1 | | | +-- chat example-model | +-- execute_tool get_market_data | +-- step 2 | +-- chat example-model The resulting trace provides a much more complete representation of an agent workflow than a single model request log. The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available. Step 5: Add additional application context Sometimes the standard GenAI attributes are not enough. For example, we may want to know which business workflow initiated an operation or which application component owns it. The AI SDK provides enrichSpan for adding custom attributes: import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, enrichSpan: ({ spanType, operationId }) => ({ 'app.component': 'portfolio-assistant', 'app.operation_id': operationId, 'app.span_type': spanType, }), }), ); The exact context available to enrichSpan depends on the span and the SDK's integration API. The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry. References: AI SDK Telemetry · Next.js OpenTelemetry · AI SDK v7 Migration Guide 6. Instrumenting Your Own Agent Runtime Built-in instrumentation is useful, but it does not automatically understand every application-level decision. For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval. These are application-level operations. We should instrument them explicitly. OpenTelemetry provides a tracing API for creating custom spans. Consider a custom agent orchestration function: import { trace, SpanStatusCode, } from '@opentelemetry/api'; const tracer = trace.getTracer('agent-orchestrator'); export async function runReviewWorkflow() { return tracer.startActiveSpan( 'workflow.merge-request-review', async (span) => { span.setAttribute( 'workflow.name', 'merge-request-review', ); span.setAttribute( 'workflow.version', '1.0.0', ); try { const result = await executeReview(); span.setStatus({ code: SpanStatusCode.OK, }); return result; } catch (cause) { const error = cause instanceof Error ? cause : new Error(String(cause)); span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR, }); throw error; } finally { span.end(); } }, ); } Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated. We now have two levels of observability: - Framework-level instrumentation describes model and tool execution. - Application-level instrumentation describes business workflows and orchestration decisions. This distinction is important. We should not expect a model instrumentation library to understand the complete business meaning of an application request. Instrumenting remote agent calls For A2A-based systems, distributed tracing becomes even more important. Imagine a coordinator running in one service and a security agent running in another service. The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated. Conceptually: Trace: merge-request-review | +-- Coordinator Agent | | | +-- Model Call | | | +-- A2A: Delegate to Security Agent | | | +-- Security Agent | | | +-- Model Call | | | +-- Tool Execution | +-- Final Aggregation The HTTP request must carry the appropriate trace context, typically through W3C traceparent and tracestate headers. For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context. This is how we avoid fragmented traces across independently deployed agents. 7. What Should We Measure? Collecting telemetry is not enough. We need to define which metrics are useful for understanding the health and quality of an agent platform. I would divide AI observability metrics into five categories. 7.1. Latency Metrics These describe how long operations take. | Metric | Why it matters | |---|---| | End-to-end workflow duration | Measures total user-facing execution time | | Model request duration | Identifies slow inference calls | | Time to first token | Measures perceived response latency | | Tool execution duration | Identifies slow external operations | | Agent invocation duration | Measures the execution time of one agent | | Queue wait duration | Identifies scheduling bottlenecks | For streaming applications, time to first token is particularly useful. A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds. 7.2. Reliability Metrics These help us understand whether workflows complete successfully. Examples: - Agent execution success rate - Tool execution failure rate - Model request failure rate - Timeout rate - Retry rate - Task cancellation rate - Human intervention rate - Incomplete workflow rate A high retry rate can be an early indicator of a problematic integration or an unstable external dependency. We should also distinguish between a successful model response and a successful business workflow. An LLM can return a perfectly valid response while the overall task fails to achieve its objective. 7.3. Token and Model Usage Token usage should be tracked at multiple levels. | Metric | Description | |---|---| | Input tokens | Tokens sent to the model | | Output tokens | Tokens generated by the model | | Cached input tokens | Tokens served from provider cache | | Reasoning tokens | Model-reported reasoning token usage | | Tokens per workflow | Total usage across an entire task | | Tokens per agent | Usage attributed to an individual agent | For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response. 7.4. Agent and Tool Behavior These metrics describe how efficiently agents perform their work. Examples: - Number of tool calls per task - Number of model calls per agent invocation - Number of agent handoffs - Repeated tool-call frequency - Tool-selection success - Average steps per workflow - Average number of clarification requests An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem. The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs. 7.5. Quality Metrics This is where AI observability starts to overlap with AI evaluation. Examples: - Answer correctness - Groundedness - Retrieval relevance - Tool-selection accuracy - Task completion - Policy adherence - Human acceptance rate These metrics cannot always be derived from execution telemetry alone. We need an evaluation strategy to measure the actual quality of the result. 8. Cost Control: Understanding the Real Cost of an Agent One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow. An agent-based application may consume resources across several layers: - LLM inference - Embedding generation - Reranking - External API calls - Database queries - Vector database operations - Container or sandbox execution - Remote agent invocations - Infrastructure and networking The total cost should reflect the entire execution path. A useful starting point is to define a model cost formula: [ C_{\text{model}} = \sum_{i=1}^{n} \left( T_{\text{input},i}P_{\text{input},i} + T_{\text{output},i}P_{\text{output},i} \right) ] Where: - (T) represents token usage. - (P) represents the applicable price per token. - (i) represents an individual model operation. The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities. We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets. The complete workflow cost can then be represented as: [ C_{\text{workflow}} = C_{\text{model}} + C_{\text{tools}} + C_{\text{compute}} + C_{\text{storage}} ] This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available. Why is cost per successful task more useful? Imagine two implementations: | Metric | Agent A | Agent B | |---|---:|---:| | Average cost per attempt | $0.04 | $0.07 | | Successful completion rate | 90% | 98% | | Cost per successful task | ~$0.044 | ~$0.071 | Agent A is cheaper per attempt, but Agent B completes more tasks successfully. Neither is automatically better. The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts. A useful metric is: [ \text{Cost per successful task} = \frac{\text{Total execution cost}} {\text{Number of successful tasks}} ] The numerator should include failed attempts that consumed resources, rather than counting only successful runs. Model selection and routing Observability data can also help us make better model-routing decisions. For example: - Use a smaller model for classification or simple extraction. - Use a more capable model for complex analysis. - Avoid unnecessary model calls when a deterministic operation is sufficient. - Reuse cached context when appropriate. - Set explicit iteration and tool-call limits. - Stop an agent when a task exceeds its budget. The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request. Token metrics in OpenTelemetry The current GenAI token metric conventions include proposed counters such as: - gen_ai.client.inference.usage.input_tokens - gen_ai.client.inference.usage.output_tokens - gen_ai.client.inference.usage.cache_read.input_tokens - gen_ai.client.inference.usage.reasoning.output_tokens These conventions are still in Development. There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics. A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations. For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices. This makes historical cost calculations more reliable when provider pricing changes. References: GenAI Token Metrics · GenAI Metrics 9. Observability Is Not Evaluation This distinction is worth emphasizing. Observability tells us what happened during execution. Evaluation tells us whether the result was good enough. A trace can show that an agent: - Called the correct model. - Used three tools. - Retrieved five documents. - Completed within four seconds. - Returned a response without an exception. But none of these facts proves that the answer was correct. Consider a RAG application. The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information. From an operational perspective, the request succeeded. From a quality perspective, it failed. We need both observability and evaluation. Offline Evaluation Offline evaluation measures an AI application against a predefined dataset. For example, we can create 100 representative questions and expected outcomes. Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again. This helps identify regressions before deploying changes. Useful evaluation dimensions include: - Answer correctness - Relevance - Groundedness - Tool selection - Expected task completion - Policy compliance Online Evaluation Online evaluation measures actual production interactions. We can select a sample of production traces and evaluate them using: - Rule-based checks - Code-based evaluators - LLM-as-a-Judge - Human feedback The important part is to turn real production failures into future test cases. For example: 1. A production trace shows that an agent selected the wrong tool. 2. We identify the relevant input and expected behavior. 3. We create a regression dataset entry. 4. We update the prompt or agent implementation. 5. We rerun offline evaluations. 6. We deploy and monitor the next set of production traces. This creates a continuous improvement loop. Production Traces | v Failure Analysis | v Evaluation Dataset | v Offline Evaluation | v Prompt / Model / Agent Changes | v Deployment | v Online Evaluation | +------> New Regression Cases Tools worth evaluating Two examples of AI-focused observability and evaluation platforms are: Langfuse Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements. Documentation: langfuse.com Arize Phoenix An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions. It provides tracing, evaluation, datasets, experiments, and debugging capabilities. Documentation: phoenix.arize.com These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy. Reference: Langfuse Evaluation Concepts · Arize Phoenix 10. The Role of OpenTelemetry Collector In a production architecture, applications should not necessarily export telemetry directly to every observability backend. This is where the OpenTelemetry Collector becomes useful. The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations. It provides a central place for: - Batching telemetry - Filtering unwanted data - Applying processors - Handling authentication - Reducing telemetry volume - Sampling traces - Routing signals to different backends A simplified architecture looks like this: AI Applications | +----------+----------+ | | | v v v Next.js .NET API Python Agent | | | +----------+----------+ | OTLP | v OpenTelemetry Collector | +--------+--------+ | | | v v v Traces Metrics Logs | | | v v v Jaeger Prometheus Log Backend | v Grafana The same telemetry can also be routed to AI-specific observability platforms where supported. Why centralize collection? In enterprise environments, we may have several applications and teams using different frameworks and languages. A Collector provides a common ingestion point and makes it possible to apply centralized policies. For example, a Collector can filter sensitive attributes before telemetry reaches the final backend. It also allows us to change exporters or observability providers without changing the instrumentation in every application. Agent and gateway deployment OpenTelemetry supports different Collector deployment patterns. A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors. This provides a separation between local telemetry collection and centralized processing. However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling. Reference: OpenTelemetry Collector · Gateway Deployment Pattern 11. Sampling: How Do We Control Telemetry Volume? Agent-based workflows can generate a significant amount of telemetry. Consider an agent that performs 15 model calls and 20 tool calls during a single task. If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive. Sampling helps control this volume. There are two common approaches. Head-Based Sampling The sampling decision is made when the trace starts. For example, we may decide to sample 10% of incoming requests. This is relatively simple and reduces the amount of telemetry generated or exported. However, we do not yet know whether a trace will eventually fail or become unusually slow. Tail-Based Sampling The sampling decision is made after considering the trace's collected spans. This makes it possible to retain: - All failed traces. - All traces exceeding a latency threshold. - Traces containing specific tool failures. - A percentage of successful traces. For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed. However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance. This generally requires trace-ID-aware routing when using multiple Collector instances. A practical approach is to combine sampling strategies: - Preserve error and high-latency traces. - Retain a representative sample of successful traces. - Maintain complete aggregate metrics where possible. - Apply stricter retention policies to verbose telemetry. Reference: OpenTelemetry Sampling · Tail Sampling Processor 12. Security and Privacy Considerations This is one of the most important aspects of AI observability in enterprise environments. Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names. AI telemetry can contain much more sensitive information. For example: - User prompts - System instructions - Generated responses - Tool arguments - Tool results - Retrieved documents - Conversation context - Customer or business identifiers Sending all this information to an observability backend can create additional privacy and data governance risks. Do not capture prompts and responses by default A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise. We can disable this behavior: registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); For many production applications, this is a better starting point. We can still collect useful operational telemetry: - Model identifiers - Token counts - Duration - Finish reasons - Tool execution times - Error information - Workflow identifiers without exporting the complete conversation. Apply data minimization Before exporting telemetry, ask whether each attribute is genuinely needed. Avoid sending: - Access tokens - Authorization headers - Passwords - Raw customer records - Full database query results - Unfiltered tool arguments For sensitive workflows, use approved metadata rather than full payloads. Separate operational telemetry from application data An observability backend should not automatically become another storage system for business data. The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely. Define access and retention policies Enterprise environments should define: - Who can access traces. - Which teams can view prompt and response content. - How long telemetry is retained. - How data is deleted. - Which environments are allowed to capture full content. - How telemetry exports are authenticated and encrypted. A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production. Reference: OpenTelemetry GenAI Content Capture 13. Limitations and Trade-offs OpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself. Instrumentation coverage varies Not every framework, provider, or agent runtime exposes the same information. Some integrations can automatically trace model calls and tool execution, while others require custom instrumentation. We should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility. GenAI semantic conventions are evolving The GenAI conventions are still under development. Attribute names, operation models, and implementation support may evolve. Teams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity. Observability does not guarantee answer quality A trace can explain what happened but does not prove that the model's result is correct. Evaluation datasets, deterministic checks, and human feedback remain necessary. Telemetry has its own cost Collecting every input, output, tool argument, and execution event can generate significant storage and processing overhead. We should measure the cost of observability itself. High-cardinality dimensions can become expensive Adding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series. Use traces for request-level correlation and keep metric dimensions bounded. Distributed tracing requires correct context propagation A2A communication, queues, background tasks, and remote tool execution may cross process boundaries. If context is lost, we can end up with disconnected traces. AI evaluation requires careful design LLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased. A single evaluation score should not become the only definition of quality. Combine deterministic checks, representative datasets, human review, and calibrated evaluators. 14. What Can an AI Chapter Lead Contribute? I think this is where observability moves from an implementation detail to an engineering capability. When multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms. Over time, this creates fragmented visibility. One team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model. As an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications. 1. Define a Common Telemetry Standard Establish a baseline for the telemetry every agent-based application should produce. For example: | Category | Required information | |---|---| | Application | Service name, environment, version | | Workflow | Workflow name, workflow version | | Agent | Agent name, execution status | | Model | Provider, requested model, response model | | Usage | Input/output tokens, cache usage | | Tools | Tool name, duration, execution status | | Execution | Trace ID, span ID, error information | | Business outcome | Completed, failed, canceled, escalated | The objective is to make observability consistent without forcing every team to use the same agent framework. 2. Build a Shared Instrumentation Package We can create an internal TypeScript package that wraps the common OpenTelemetry configuration. For example: @company/ai-observability | +-- OpenTelemetry initialization +-- AI SDK integration +-- Agent span utilities +-- Workflow instrumentation +-- Tool execution tracking +-- Cost attribution +-- Evaluation metadata +-- Privacy and redaction policies This package can provide standard helpers for application-level tracing and reduce duplicated implementation. For teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations. 3. Standardize Agent and Workflow Metrics We should define a common set of metrics that helps teams understand both technical performance and business outcomes. Examples: - Agent success rate - Workflow completion rate - Tool-call failure rate - Model latency - Cost per successful task - Average token usage - Human intervention rate - Evaluation quality scores A shared dashboard can make it easier to compare systems and identify performance regressions. 4. Establish an Internal Evaluation Framework Observability without evaluation gives us only part of the picture. We can create reusable evaluation datasets and testing patterns for common agent behaviors. For example: - Tool selection accuracy - Missing-source detection - Correct handling of incomplete input - Permission violations - Unsupported claims - Failure recovery - Task completion quality This is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified. 5. Establish Privacy and Data Governance Define what can and cannot be captured by default. The shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads. This should include clear ownership, access policies, retention, and audit requirements. 6. Run a Cross-Team Pilot A useful pilot would be a multi-agent workflow that involves several independently deployed services. For example, a merge request review involving coordinator, security, and testing agents. The pilot should demonstrate: 1. One trace across the complete request. 2. Parent-child relationships between agents. 3. Model and tool execution spans. 4. Token and cost attribution. 5. Error and retry visibility. 6. Evaluation results associated with the completed workflow. This would provide a reference architecture that other teams can reuse. 7. Make Observability Part of the Development Lifecycle Observability should not begin after production deployment. We can introduce telemetry and evaluation requirements into the development process itself. For example: - New agents must have execution tracing. - New tools must expose execution status and duration. - Prompt changes must pass relevant regression evaluations. - New workflows must define success and failure criteria. - Production deployments must have basic operational dashboards. This makes observability a standard engineering practice rather than a separate monitoring task. 15. A Practical Adoption Roadmap I would avoid introducing every observability component at once. A phased approach is more useful. | Phase | Focus | Expected result | |---|---|---| | 1 | OpenTelemetry initialization | Basic traces and metrics | | 2 | AI SDK instrumentation | Model and tool visibility | | 3 | Custom agent spans | Complete workflow tracing | | 4 | Cost attribution | Usage and cost per task | | 5 | Evaluation | Offline and online quality checks | | 6 | Security and governance | Privacy and access controls | | 7 | Shared dashboards | Cross-team operational visibility | The first milestone should be simple: For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result. Once this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce. Conclusion AI agents introduce a new level of complexity to application observability. A single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution. OpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components. Combined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production. But the most important distinction remains: Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it. For an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices. The goal should not be to collect more telemetry. It should be to make every production agent workflow understandable, measurable, and continuously improvable. --- References and Further Reading OpenTelemetry 1. OpenTelemetry Official Documentation — Core concepts, signals, instrumentation, and architecture. 2. OpenTelemetry JavaScript — TypeScript and Node.js instrumentation. 3. OpenTelemetry Tracing — Traces, spans, and distributed execution. 4. OpenTelemetry Context Propagation — Distributed trace context. 5. OpenTelemetry Collector — Receiving, processing, and exporting telemetry. 6. OpenTelemetry Gateway Deployment — Centralized telemetry collection. 7. OpenTelemetry Sampling — Head-based and tail-based sampling. GenAI Semantic Conventions 8. GenAI Semantic Conventions — Spans — Model operations, retrieval, and tool execution. 9. GenAI Agent Spans — Agent invocation, workflows, and tool spans. 10. GenAI Metrics — Agent duration, tool calls, and workflow metrics. 11. GenAI Token Metrics — Token consumption and usage measurements. Vercel AI SDK 12. AI SDK Telemetry — OpenTelemetry integration, span hierarchy, and custom attributes. 13. AI SDK v7 Migration Guide — Version-specific telemetry changes. 14. Next.js OpenTelemetry — Application-level instrumentation. AI Evaluation and Observability Platforms 15. Langfuse Evaluation — Online and offline evaluation. 16. Langfuse Core Concepts — Datasets, experiments, and evaluators. 17. Arize Phoenix — Open-source AI observability and evaluation. 18. OpenInference — OpenTelemetry-based instrumentation conventions for AI applications.

AIエージェントの実行証跡をどう設計するか ― 監査・説明責任・障害調査 AIエージェントが業務システムを操作したとき、後からその処理をどこまで追跡できる必要があるだろうか。 通常のWebアプリケーションでも、アクセスログや監査ログを使って操作履歴を確認できる。 では、AIエージェントでも同じログを収集すれば十分なのだろうか。 AIエージェントでは、ユーザーの依頼を受けてLLMがToolを選択し、取得した情報をもとに次の操作を判断する。 そのため、業務APIの呼び出し結果だけでは、実行に至った経緯を把握できない場合がある。 今回は社内の経費精算システムを例に、監査・説明責任・障害調査・セキュリティという複数の目的を満たす実行証跡をどう設計するか考える。 1. 実行証跡は何のために必要なのか まず、ログを収集する目的を整理する。 目的 確認したいこと 監査 誰が何を実行し、権限や手続きに問題がなかったか 説明責任 AIがどの情報をもとに何を行ったか説明できるか 障害調査 どこで失敗し、なぜ期待した結果にならなかったか セキュリティ 不正な操作や異常な動作が発生していないか 目的は異なるが、必要な情報には重複が多い。 例えば、Toolの実行履歴は、障害調査だけでなく監査やセキュリティ調査にも利用できる。 そこで、目的ごとに別々のログ出力処理を実装するのではなく、共通の実行証跡を収集し、複数の目的で活用する構成を考える。 また、証跡を共通的に収集することと、それぞれの目的の要件を満たすことは別だ。 例えば監査では、必要な記録を欠落させないことや、保存期間、改ざん防止などが求められる場合がある。 説明責任についても、ログだけで完結するわけではなく、責任者の明確化や問題発生時の対応も必要になる。 収集はできるだけ共通化し、保存・管理は目的に応じて設計する。 2. 通常のWebアプリと何が違うのか 経費精算システムで、先週の大阪出張の経費を提出する場面を考える。 通常のWebアプリでは、次のように操作する。 ユーザーが申請一覧画面を開く ↓ アプリが申請一覧APIを呼ぶ ↓ ユーザーが対象申請を選ぶ ↓ アプリが申請詳細APIを呼ぶ ↓ ユーザーが内容を確認して提出 ↓ アプリが提出APIを呼ぶ アクセスログで誰がいつAPIを呼び出したかを確認し、業務DBや変更履歴から申請内容を調査できる。 これらが適切に記録・関連付けられ、監査要件を満たしているなら、専用の監査ログを別途実装する必要はない。 一方、AIエージェントには、ユーザーが次のように依頼する。 先週の大阪出張の経費を提出しておいて。 この場合は、次のように処理が進む。 ユーザーが経費精算を依頼 ↓ LLMが申請一覧取得Toolを選択・実行 ↓ LLMが対象申請を選ぶ ↓ LLMが申請詳細取得Toolを選択・実行 ↓ LLMが提出Toolを選択 ↓ ユーザーが内容を確認して承認 ↓ 提出Toolから提出APIを呼ぶ どちらも申請一覧の取得、申請内容の取得、提出という業務処理は同じだ。 違うのは、通常のWebアプリではユーザーが対象や操作を選び、アプリが決められたAPIを呼ぶのに対し、AIエージェントではLLMが取得した情報をもとに次のToolや対象を選ぶことだ。 例えば、エージェントが間違った申請を提出してしまったとする。 業務サービスの記録からは、どの申請が提出されたか確認できる。しかし、それだけでは次の疑問に答えられない。 ユーザーは何を依頼したのか エージェントはどの申請情報を参照したのか どのToolと引数を選択したのか ユーザーはどの内容を承認したのか これらを調査するには、業務APIの呼び出し以前の実行経緯が必要になる。 ただし、LLMの内部的な推論過程を完全に記録・再現できるわけではない。後からLLMに判断理由を説明させても、実際の判断過程を正確に表している保証はない。 重要なのは、内部的な思考を再現することではなく、参照情報、LLMの出力、Toolの選択、承認、実行結果など、実際に観測できた事実を追跡することだと考える。 3. 何を記録する必要があるのか 経費精算エージェントを例に、記録対象を整理する。 分類 記録情報の例 共通的に収集 実行日時、実行主体、実行ID、Tool名、呼び出し結果 イベント発生時に記録 認可失敗、承認・否認（承認者・対象・日時・結果など）、重要な業務状態変更 リスク・監査要件に応じて追加 認可成功の詳細、適用したルール、対象データ、モデルやAgent定義のバージョン 個別に判断 依頼全文、LLM入出力、Tool引数・結果の全文、参照文書の内容 この分類は設計の出発点であり、すべてのシステムで必須となる項目を示したものではない。 参照した情報をどう残すか 例えば、エージェントが誤った申請を選択した場合。 原因を調査するには、LLMがどのような申請情報を受け取っていたか確認したい。 しかし、取得した申請一覧をすべてログに保存すると、情報量が増え、個人情報の管理も難しくなる。 そこで、参照データのIDやバージョンを記録し、必要な場合に実行時点のデータを確認する方法が考えられる。 ただし、データが更新・削除されるなら履歴管理が必要になる。また、取得したデータを加工してLLMに渡す場合は、その内容も追跡できるか検討する。 モデルやプロンプトの変更が誤処理に影響する可能性もある。 重要な処理では、モデル識別子、プロンプトやAgent定義のバージョン、Toolの実装バージョンなども記録候補になる。 実行されなかった操作も記録する もう一つ重要なのが、Tool実行前に拒否された操作だ。 例えば、エージェントが権限外の申請を提出しようとしたが、認可処理で拒否された場合。 業務APIが呼び出されていなければ、そのアクセスログには何も残らない。 しかし、セキュリティ調査では、エージェントが何を実行しようとし、どの制御によって拒否されたかが重要になる。 認可失敗は原則として記録する。また、承認・否認が行われた場合は、操作の重要度だけで判断せず、誰が何に対してどのような判断をしたかを残す。認可成功の詳細をどこまで記録するかは、業務リスクや監査要件に応じて決める。 OWASPのAI Agent Security Cheat Sheetでも、高リスク操作の認可結果、承認ID、適用したポリシー、実行結果などの記録が推奨されている。 4. 実行証跡をどう収集するか 必要な情報を整理できたら、次はどこで収集するかを考える。 すべてを個別実装する必要はない。既存のLoggerやOpenTelemetryを活用し、不足する情報だけを追加する。 共通Loggerで収集する 日時、ユーザーID、リクエストIDなどは、共通のログフォーマットで出力できる。 例えばJavaなら、SLF4JのMDCなどを使い、処理中のログにユーザーIDやリクエストIDを自動付与できる。 AIエージェントでは、それに加えて一連の実行を関連付けるIDを用意したい。 例えば、次のような関係になる。 経費精算の依頼（実行ID：run-001） │ ├─ 申請一覧を取得 │ ├─ 申請内容を取得 │ ├─ 提出操作の承認待ち │ └─ 承認後に提出APIを実行 各Toolの呼び出しにも個別の識別子を付け、一連の実行IDと関連付ける。 実行IDは処理コンテキストを通じて引き継ぎ、各コンポーネントのログやトレースに付与する。追跡のために、既存の業務DBの各テーブルへ実行IDを追加する必要はない。 承認待ちなどで処理が中断され、翌日再開される場合もある。 HTTPリクエストや分散トレースのIDは再開前後で変わる場合があるため、一連の業務依頼を追跡できるIDを保持するとよい。 なお、IDの名称や管理方法は利用するフレームワークによって異なる。既存の識別子を活用できるなら、新たに作る必要はない。 OpenTelemetryで収集する LLMやToolの実行については、OpenTelemetryのGenAI Semantic Conventionsを活用できる。 これは、生成AIの処理をトレースするために、記録する情報の名前や形式を共通化する仕様だ。 例えば、次のような情報を表現できる。 情報 OpenTelemetryの属性例 実行した処理 gen_ai.operation.name Tool名 gen_ai.tool.name Tool呼び出しID gen_ai.tool.call.id Toolの引数 gen_ai.tool.call.arguments Toolの結果 gen_ai.tool.call.result ここで注意したいのは、OpenTelemetryを導入すれば、すべての情報が自動的に記録されるわけではないことだ。 実際に取得できる内容は、利用するフレームワークや計測ライブラリの対応状況によって異なる。 また、Toolの引数やLLMの入出力には機密情報が含まれる可能性があるため、必要な場合に限定して取得する。 OpenTelemetryは共通の記録形式を提供するが、何を取得し、どこまで残すかはシステム側で設計する必要がある。 なお、GenAI関連の仕様は更新が続いているため、実装時には利用するバージョンと対応状況を確認する。 参考：OpenTelemetry GenAI Semantic Conventions どのコンポーネントで記録するか 既存の経費精算システムにAIエージェントを追加する場合、例えば次の構成になる。 ユーザー ↓ Agent基盤 │ Tool選択・承認 ↓ MCP Server │ Tool受付・実行 ↓ 業務API │ 業務処理 ↓ 業務DB 記録対象は次のように分けられる。 コンポーネント 主な記録対象 Agent基盤 LLM呼び出し、Tool選択、承認・拒否 MCP Server Toolの受付・実行結果 業務サービス 業務処理結果、対象データの変更 例えば、Toolの呼び出しが正常終了しても、非同期で実行される後続の業務処理が失敗することはある。 そのため、最終的な業務処理結果は業務サービス側で確認する。 既存のアクセスログやDBの変更履歴を置き換えるのではなく、Agent側の実行証跡と関連付けて活用する。 これにより、誤った経費申請が提出された場合も、次のように調査できる。 実行IDからユーザーの依頼を特定する Agent側の記録から参照情報とToolの選択を確認する 承認履歴からユーザーが何を承認したか確認する 業務サービスの記録から実際の提出結果を確認する これらを関連付けることで、最初の依頼から実際の業務操作までを追跡できる。 5. 実行証跡をどう保存・管理するか ここまで収集方法を整理したが、取得した情報をすべて同じ方法で保存すればよいわけではない。 例えば、分散トレースでは性能調査のために一部の記録を間引いたり、短期間で削除したりする場合がある。 一方、監査に必要な操作履歴を同じように間引くと、必要な証跡が残らない可能性がある。 そこで、記録項目は共通化しつつ、保存方法は目的に応じて分ける構成を考える。 これは論理的な構成であり、必ず別々のログ基盤を用意するという意味ではない。 同じ基盤を使いながら、記録の種類によって保存期間や保存方法を変えることもできる。 ただし、収集や転送の途中で失われた記録は、保存先を分けても復元できない。 そのため、監査に必要な重要イベントでは、収集から保存までの一連の処理で欠落を防ぐ設計が必要になる。 監査に必要な管理要件 監査に利用する記録では、主に次の点を検討する。 観点 設計内容 記録の欠落防止 必要な操作履歴を確実に残す 保存期間 業務・法令・契約要件に従って決める 改ざん対策 不正な変更・削除を防止・検知する アクセス制御 閲覧・管理できる人を制限する 機密保護 個人情報や認証情報を保護する 追跡可能性 関連するログやDB情報を対応付ける これらは従来のWebアプリでも必要な設計であり、AIエージェント固有の仕組みを作る必要はない。 ただし、重要な操作では、ログの保存に失敗した場合の扱いも決める必要がある。 例えば、経費申請の正式提出で、提出自体は成功したのに必要な監査記録が残らなかったとする。 この状態を許容できない業務なら、業務処理と監査記録の整合性を確保する必要がある。 業務DBと同じトランザクションで記録する方法や、記録を確実に転送・再送する仕組みなどが考えられる。 重要なのは、ログ出力処理を追加しただけで、監査に必要な記録が必ず残るとは限らないことだ。 参考：OWASP Logging Cheat Sheet 6. まとめ 今回は、AIエージェントの実行証跡をどう設計するか考えた。 最初の疑問は、 「AIエージェントでも、従来のWebアプリと同じログを収集すれば十分なのか？」 だった。 私の答えは、「従来のログ設計を基本にできるが、LLMによるTool選択や情報参照、承認・拒否など、業務API実行前の経緯も追跡する必要がある」だ。 今回の設計では、次のように整理した。 監査・説明責任・障害調査・セキュリティに必要な情報を、共通の実行証跡として収集する LoggerやOpenTelemetryを活用し、個別実装を最小限にする 業務固有の情報は既存システムで管理し、実行IDなどで関連付ける 監査に必要な保存期間、改ざん対策、記録の欠落防止などは別途満たす LLMの入出力やTool引数など、機密情報を含む内容の記録範囲は慎重に決める AIエージェントの実行証跡設計では、ログの種類を増やすのではなく、何を追跡できる必要があるかを整理し、既存の仕組みと組み合わせて証跡を確保する。 これが今回の設計の結論だ。

Claude Code Sends a Background Agent's Whole Transcript Every 30s for a 3-5 Word Status - Book: LLM Observability Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- You start three background agents on a refactor and go make coffee. While each one works, about every 30 seconds, Claude Code sends that agent's transcript back to the agent's own model with one request: "describe your most recent action in 3-5 words." The answer becomes the little label next to the agent in the agents panel. That is the finding in a gist from the flamerix-ai account that reached 584 points and 236 comments on Hacker News at the time of writing. In the author's OpenTelemetry data for one week, the agent_summary calls were 16.4% of estimated spend and 19.7% of API requests. The main conversation was 15.7%. A Claude Code engineer replied in the thread that the basic mechanism is right, and that the typical share is far lower. Both can be true. Which one is your bill depends on how you work. How the agent_summary timer works Per the gist, the feature has been on by default since Claude Code 2.1.232, in August 2026. Each actively working background agent gets a timer. When it fires, Claude Code forks a request that carries the agent's transcript to the agent's own model, usually Opus, and asks for the 3-5 word description. The gist puts the input-to-output ratio at about 9,000 tokens in for every token out. Three details matter: - The interval comes from the server. The gist says it is a per-model-family server flag, clamped to 30-240 seconds, and the default is already the minimum. No value you can set turns it off. - It uses the agent's model. A GitHub issue on 2.1.312 measured a Haiku subagent and saw the calls go to Haiku, so an Opus subagent gets summarized by Opus. - It skips the transcript. The original feature request found the calls run with skipTranscript: true, so cost tools that read session transcripts never see them. The Claude Code reply on Hacker News narrows the scope: it "only applies to background agents that are actively working." Idle agents don't trigger it. Workflow agents don't get summary timers either, according to a commenter on #86879. This is not the status line you configure yourself. The status line docs say that one "runs locally and does not consume API tokens." The agent_summary label lives in the agents panel, and it is a model call. !A stopwatch whose hand pulls a long paper scroll into a large machine, which prints a single tiny tag at the other end The trade in one picture: the whole transcript in, a few words out. There is no supported off switch. CLAUDE_CODE_AGENT_PROGRESS_SUMMARIES and the SDK's agentProgressSummaries can only turn summaries on, per the gist. CLAUDE_CODE_FORK_SUBAGENT=0 has mixed reports on #86879 and #105981: some people saw zero calls, others still saw summary calls. One commenter suggested the difference comes from a settings env block overriding the shell export. It also disables fork mode, so setting it loses you more than the summaries. What the Bun preload does, and why it breaks The gist ships an MIT-licensed no-agent-summary.js. It relies on Claude Code's native build running on Bun, and on Bun's BUN_OPTIONS variable, which "prepends command-line arguments to any Bun execution." So BUN_OPTIONS=--preload=<path> loads a script before Claude Code's own code runs. The gist says to use the = form, because a space breaks bun subcommands, and to run claude daemon stop --any so background processes pick up the new environment. The script wraps the global setTimeout. When a callback's source text contains one of two anchors, [AgentSummary] Timer fired or agent_summary, it returns an inert timer instead of scheduling the real one. It also checks that it is running inside Claude Code before doing anything. The technique, in its smallest form, looks like this: // Illustration of the technique. Not the gist's script. const realSetTimeout = globalThis.setTimeout; const ANCHOR = "[AgentSummary] Timer fired"; globalThis.setTimeout = (fn, ms, ...args) => { const src = typeof fn === "function" ? String(fn) : ""; if (src.includes(ANCHOR)) { const t = realSetTimeout(() => {}, 2 ** 31 - 1); t.unref?.(); return t; } return realSetTimeout(fn, ms, ...args); }; Don't run that sketch. It has none of the gist's gating or verification, and it shows exactly why this approach is fragile. It matches source text inside a minified binary. Rename one log line, or move the timer off the global setTimeout, and summaries come back with no error. The gist says this itself and asks you to verify after every update. A commenter on #86879 also warned against installing a preload by curl-ing the latest version, since whatever it downloads then runs inside every Claude Code process. Two more caveats from the gist's own pages. Its README says the summary never reaches the model or your conversation, but a commenter on Claude Code 2.1.314 found accepted summary text stored in task.progress.summary in later context, and called that claim too strong. And the license section says plainly: "Changing a client's runtime behaviour may conflict with the terms of the service you use it with; checking that is on you." I haven't run the hook myself. Measure your own agent_summary share You don't need the hook to find out whether this matters to you. Claude Code's monitoring docs log a claude_code.api_request event for each request, with cost_usd and a query_source attribute, the "subsystem that issued the request." The gist grouped those events by query_source, where summaries show up as agent_summary. Use the events, not the metrics. On the claude_code.cost.usage metric, query_source collapses to main, subagent or auxiliary, which hides the summaries inside a bucket. Turn on telemetry and the logs exporter: export CLAUDE_CODE_ENABLE_TELEMETRY=1 export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_PROTOCOL=grpc # your collector's OTLP gRPC address, usually port 4317 export OTEL_EXPORTER_OTLP_ENDPOINT="$COLLECTOR_URL" If you don't have a backend, an OpenTelemetry Collector with the file exporter writes newline-delimited OTLP JSON: receivers: otlp: protocols: grpc: exporters: file: path: ./claude-logs.jsonl service: pipelines: logs: receivers: [otlp] exporters: [file] Work normally for a few days, then group by source with jq and awk: jq -r '.resourceLogs[].scopeLogs[].logRecords[] | [.attributes[] | {(.key): (.value | to_entries[0].value)}] | add | select(.["event.name"] == "api_request") | [(.query_source // "unknown"), (.cost_usd // 0 | tonumber)] | @tsv' claude-logs.jsonl \ | awk -F'\t' ' { cost[$1] += $2; reqs[$1]++; tc += $2; tr++ } END { for (k in cost) printf "%-22s %5.1f%% cost %5.1f%% reqs\n", k, 100 * cost[k] / tc, 100 * reqs[k] / tr }' Each output row is one query_source with its share of estimated cost and of requests. Two things to keep in mind when you read it. cost_usd is Claude Code's own estimate at list price, not an invoice, as the cost docs say. And the share depends on how much of your day runs as background agents, which is the whole argument. !Paper receipts dropping from a conveyor into labelled bins, with one bin filling faster than expected Group the request events by source and the summaries get a bin of their own. Why the numbers disagree Here is every figure on the table, each from a different method and workload: | Source | Share | What it measured | |---|---|---| | The gist | 16.4% of spend, 19.7% of requests | The author's setup, one week, OpenTelemetry estimates | | Issue #86879 | 22.7% of requests, 18.0% of weighted tokens | One 4-hour session, 27 subagents, 2.1.233 | | Commenters on #86879 | From about 10% of spend over 30 days to 26% of cache reads in one session | Separate workloads and methods | | Claude Code engineer on HN | About 2% of cost, just under 4% of requests | Telemetry users, last week | | Same reply, terminal only | About 6% of cost | Where nearly all these calls happen | The high numbers come from people running mostly background agents. The engineer said as much: 16.4% "sounds like it was for a team running mostly background agents, and doesn't seem like a typical session." The low numbers are an average over everyone who sends telemetry, most of whom run few background agents. Neither one is your number. A replay of the 30-second timer over one person's transcripts, posted on the gist, landed at about 8%, partly because it assumed a warm cache. The cache is the part that moves the bill. The engineer said a fix for "summaries that miss the prompt cache" shipped in 2.1.311 and later, calling misses the most expensive case. The team is also working on "a much cheaper mode that sends short notes to a smaller model instead of the full transcript." The reply adds: "we agree that's a lot for a status line." So if you measured before 2.1.311, measure again on a current build before you decide anything. A status that doesn't need a model call I build Hermes IDE, so weigh this section with that in mind. First, what it doesn't do: by default, Hermes runs Claude Code in a real terminal, unchanged, so background agents that Claude Code starts there get summarized exactly as they would anywhere else. What Hermes does differently is its own per-session status. Every session shows one state: working, needs approval, asked you, done, error or idle, and Hermes says when a state is a guess. The README says Hermes "observes what agents report on purpose and what git shows," and the architecture guide describes that status as a state machine driven by the terminal output. Neither document describes a model call in that path. The cost: a state tells you less than a sentence about what the agent is doing right now. !Two real agent sessions side by side in Hermes IDE, Codex and Claude Code, each in its own worktree, with the sessions sidebar showing status, context use, cost and branch Two real agent sessions side by side: Codex on orders-types after adding a RefundStatus type, and Claude Code on orders-api after a follow-up turn adding a log line, each in its own hermes/ worktree, with the sessions sidebar (status, context use, cost, branch) on the left. Each session also shows its spend: the cost the agent reports, or an estimate from its transcript, clearly marked as estimated, with token totals. That tells you which session the money went to. It won't break out agent_summary, and since those calls skip the transcript, a transcript estimate can't see them. For that share, the OpenTelemetry query above is the tool. You can download Hermes. It is source-available under BSL 1.1 and runs on macOS, Windows and Linux. The current release is 2.3.4. It is not affiliated with Nous Research or its Hermes Agent. What to do with this today 1. Run the query above for a few working days on a current Claude Code build. 2. If agent_summary is a small slice, leave it alone. The label is useful. 3. If it is large, look at why first: how many background agents run at once, and on which model. A subagent on Haiku gets summarized by Haiku. 4. Add your numbers and version to #86879. A supported opt-out is what that issue asks for, and real measurements are what moves it. 5. If you reach for the preload anyway, read it first, check your terms, and verify it after every update. --- If this was useful The fix for this kind of surprise is the same every time: per-request telemetry, grouped by who issued the request, before you trust anyone's percentage, including the ones in this post. The LLM Observability Pocket Guide is the short read on picking the tracing and eval tools that give you that view, so a hidden 3-5 word call shows up as its own line instead of a mystery in the monthly total. 

Claude CodeのOpenTelemetryをBigQueryに流してAIコーディングのコストと失敗を可視化する はじめに AIコーディングエージェントを日常業務に組み込むと、ほどなくして「で、これ結局いくらかかってるんだ？」という疑問にぶつかります。 月末の請求額は分かる。でも、どのタスクが高かったのか、どのツール呼び出しが失敗を繰り返しているのか、キャッシュはちゃんと効いているのかは分からない。ダッシュボードの合計値だけ見て「まあ、こんなもんか」と流している方も多いのではないでしょうか。 これは要するにオブザーバビリティの問題です。そして幸いなことに、Claude Code は OpenTelemetry をネイティブでサポートしています。メトリクス・イベント・分散トレースを OTLP で吐き出せるので、あとは受け止めて分析するだけです。 この記事では、Claude Code のテレメトリを OpenTelemetry Collector で受け、GCP 経由で BigQuery に着地させて SQL で分析するまでを構築します。普段 GA4×BigQuery でデータ基盤を触っている人間にとっては、慣れた道具でそのまま料理できる構成です。 ! 本記事の設定ファイルは OpenTelemetry Collector v0.115.0（contrib ディストリビューション）で validate と疎通を確認済みです。Claude Code 側の環境変数・メトリクス名・属性名は公式ドキュメントの記載に準拠しています。 自己紹介 普段は EC 事業者向けに GA4×BigQuery のデータ基盤構築と分析を請け負っています。Looker Studio でのダッシュボード化、dbt での 3 層設計、最近は Claude Code を使った SQL 生成・レポート自動化まわりが主戦場です。 つまり「BigQuery にデータを置いて SQL で殴る」のが手癖なので、今回もその型に持ち込みます。 なぜ「AIエージェントの可視化」なのか 従来のオブザーバビリティは、リクエストを受けてレスポンスを返すサーバーが対象でした。AIエージェントはここに厄介な性質を持ち込みます。 1回の指示のコストが数十倍ブレる — 同じ「バグを直して」でも、5秒で終わることも、20回ツールを叩いて数ドル溶かすこともある 失敗が可視化されにくい — ツール実行が失敗してもエージェントがリトライして最終的に成功するので、内部の失敗率は表に出てこない コストの内訳が入れ子 — サブエージェント、スキル、MCP サーバー呼び出しが階層的に発生する 「合計いくら」ではなく「どのプロンプトが、どの経路で、いくら使ったか」が見えないと改善のしようがありません。これはまさにトレースとメトリクスが解く問題です。 全体構成 Claude Code (OTLP) │ ├─ metrics ─┐ ├─ logs ────┼─→ OpenTelemetry Collector └─ traces ──┘ │ ├─→ Cloud Monitoring （メトリクス：アラート・監視） ├─→ Cloud Trace （トレース：1プロンプトの内訳） └─→ Cloud Logging （イベント） │ └─→ ログシンク → BigQuery → SQL 分析 ポイントは イベント（ログ）を BigQuery に落とすことです。Claude Code の claude_code.api_request イベントにはトークン数とコストが属性として乗っているため、ここさえ BigQuery に入れば、あとは使い慣れた SQL で好きなだけ切り刻めます。 メトリクスは Cloud Monitoring に置いてアラート用に使い、BigQuery 側は分析用と役割分担させるのがすっきりします。 Step 1：Claude Code のテレメトリを有効化する 環境変数だけで有効になります。 export CLAUDE_CODE_ENABLE_TELEMETRY=1 エクスポート先 export OTEL_METRICS_EXPORTER=otlp export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_PROTOCOL=grpc export OTEL_EXPORTER_OTLP_ENDPOINT=localhost ツールの詳細（Bash のコマンド内容や失敗メッセージ）まで記録する export OTEL_LOG_TOOL_DETAILS=1 検証中は出力間隔を短くすると即座に確認できる export OTEL_METRIC_EXPORT_INTERVAL=10000 export OTEL_LOGS_EXPORT_INTERVAL=2000 取得できる主なメトリクスは以下です。 メトリクス 内容 claude_code.cost.usage セッションのコスト（USD） claude_code.token.usage 消費トークン数 claude_code.lines_of_code.count 変更行数 claude_code.commit.count 作成したコミット数 claude_code.pull_request.count 作成した PR 数 claude_code.session.count セッション数 claude_code.active_time.total アクティブ時間（秒） そして今回の主役、イベント側です。 イベント 内容 claude_code.api_request API リクエスト1回ごとのコスト・トークン・レイテンシ claude_code.tool_result ツール実行の成否・所要時間・エラー種別 claude_code.api_error API 呼び出しの失敗 claude_code.tool_decision ツール実行許可の判断 claude_code.user_prompt ユーザーのプロンプト送信 ! OTEL_LOG_TOOL_DETAILS=1 を有効にすると、Bash で実行したコマンド文字列やツールの引数まで記録されます。便利な反面、機密情報が混ざる可能性があります。共有のバックエンドに送る場合は、この設定を入れるかどうかを必ず検討してください。また既定でも user.email などが送信対象になり得ます。 Step 2：OpenTelemetry Collector を立てる まずローカルで「ちゃんと届いているか」を目視できる構成から始めるのが確実です。いきなり GCP に繋ぐと、届かなかったときに Claude Code 側の問題なのか Collector なのか GCP 権限なのか切り分けられなくなります。 local-test.yaml receivers: otlp: protocols: grpc: endpoint: 127.0.0.1:4317 http: endpoint: 127.0.0.1:4318 processors: batch: timeout: 1s exporters: debug: verbosity: detailed file: path: ./out.jsonl service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [debug, file] logs: receivers: [otlp] processors: [batch] exporters: [debug, file] traces: receivers: [otlp] processors: [batch] exporters: [debug, file] 起動します。 contrib 版が必要（googlecloud exporter は contrib にのみ含まれる） curl -sSLO github.com/open-telemetry/opentelemetry-collector… tar xzf otelcol-contrib_0.115.1_linux_amd64.tar.gz 設定ファイルの妥当性を先に確認できる ./otelcol-contrib validate --config local-test.yaml ./otelcol-contrib --config local-test.yaml この状態で別ターミナルから Claude Code を起動して何か作業させると、out.jsonl にテレメトリが流れ込みます。ここで claude_code.cost.usage や claude_code.tool_result が見えれば、Claude Code 側の設定は正しいと確定できます。 Step 3：GCP に送る 疎通が取れたら送り先を差し替えます。googlecloud exporter はトレース・メトリクス・ログの 3 シグナルすべてに対応しており、それぞれ Cloud Trace / Cloud Monitoring / Cloud Logging に着地します。 gcp.yaml receivers: otlp: protocols: grpc: endpoint: 127.0.0.1:4317 processors: memory_limiter: check_interval: 1s limit_percentage: 65 spike_limit_percentage: 20 resource: attributes: - key: service.name value: claude-code action: upsert batch: timeout: 10s exporters: googlecloud: project: your-project log: default_log_name: claude-code-otel service: pipelines: metrics: receivers: [otlp] processors: [memory_limiter, resource, batch] exporters: [googlecloud] logs: receivers: [otlp] processors: [memory_limiter, resource, batch] exporters: [googlecloud] traces: receivers: [otlp] processors: [memory_limiter, resource, batch] exporters: [googlecloud] 認証はサービスアカウント経由が確実です。 gcloud iam service-accounts create otel-collector --project=your-project for role in roles/monitoring.metricWriter roles/logging.logWriter roles/cloudtrace.agent; do gcloud projects add-iam-policy-binding your-project \ --member="serviceAccount:otel-collector@your-project.iam.gserviceaccount.com" \ --role="$role" done gcloud iam service-accounts keys create ./otel-sa.json \ --iam-account=otel-collector@your-project.iam.gserviceaccount.com export GOOGLE_APPLICATION_CREDENTIALS=./otel-sa.json ./otelcol-contrib --config gcp.yaml memory_limiter を入れておくのは実運用上わりと大事で、これが無いとバックエンド側が詰まったときに Collector がメモリを食い潰します。 Step 4：Cloud Logging から BigQuery へシンクする ここが BigQuery に持ち込む肝です。ログシンクを作ると、以後のログエントリが自動的に BigQuery のテーブルへ流れ込みます。 受け皿のデータセット bq --location=asia-northeast1 mk --dataset your-project:claude_code_otel シンク作成 gcloud logging sinks create claude-code-to-bq \ bigquery.googleapis.com/projects/your-project/datasets/claude_code_otel \ --log-filter='logName:"claude-code-otel"' \ --project=your-project 作成時に出力されるサービスアカウントに、データセットへの書き込み権限を付与します。 出力された writerIdentity を控えて付与する gcloud projects add-iam-policy-binding your-project \ --member="serviceAccount:<出力されたwriterIdentity>" \ --role="roles/bigquery.dataEditor" ! シンクは作成後に発生したログにのみ適用されます。過去分は遡って入りません。先にシンクを作ってから Claude Code を回し始めるのが正解です。 Step 5：BigQuery で分析する まずテーブル構造を確認する ログシンク経由のテーブルは、ログの中身に応じてスキーマが自動生成されます。属性が jsonPayload の下に入るのか labels に入るのかは、Collector のバージョンやログの形で変わります。最初に必ず実物を1行見てください。 SELECT * FROM your-project.claude_code_otel.claude_code_otel_* LIMIT 1; -- カラム名を一覧で確認する SELECT column_name, data_type FROM your-project.claude_code_otel.INFORMATION_SCHEMA.COLUMNS WHERE table_name LIKE 'claude_code_otel_%' ORDER BY ordinal_position; 以下の SQL は属性が jsonPayload.attributes 配下に入る前提で書いています。実際の構造に合わせてパスを読み替えてください。 ① 日別・モデル別のコスト推移 SELECT DATE(timestamp, 'Asia/Tokyo') AS dt, JSON_VALUE(jsonPayload.attributes, '$.model') AS model, COUNT(*) AS requests, ROUND(SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.cost_usd') AS FLOAT64)), 4) AS cost_usd, SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.input_tokens') AS INT64)) AS input_tokens, SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.output_tokens') AS INT64)) AS output_tokens FROM your-project.claude_code_otel.claude_code_otel_* WHERE JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'api_request' AND DATE(timestamp) >= DATE_SUB(CURRENT_DATE('Asia/Tokyo'), INTERVAL 30 DAY) GROUP BY dt, model ORDER BY dt DESC, cost_usd DESC; ② キャッシュがどれだけ効いているか プロンプトキャッシュはコストに直結します。cache_read_tokens の比率が低いなら、セッションの回し方に改善余地があります。 SELECT DATE(timestamp, 'Asia/Tokyo') AS dt, SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.input_tokens') AS INT64)) AS input_tokens, SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.cache_read_tokens') AS INT64)) AS cache_read, SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.cache_creation_tokens') AS INT64)) AS cache_creation, ROUND( SAFE_DIVIDE( SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.cache_read_tokens') AS INT64)), SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.input_tokens') AS INT64)) ) * 100, 1 ) AS cache_hit_pct FROM your-project.claude_code_otel.claude_code_otel_* WHERE JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'api_request' GROUP BY dt ORDER BY dt DESC; ③ どのツールが失敗しているか これが個人的に一番効きました。エージェントはツールが失敗しても勝手にリトライして最終的に成功させてしまうので、失敗は表に出てきません。しかし失敗はそのままトークンの浪費です。 SELECT JSON_VALUE(jsonPayload.attributes, '$.tool_name') AS tool_name, JSON_VALUE(jsonPayload.attributes, '$.error_type') AS error_type, COUNT(*) AS n, ROUND(AVG(CAST(JSON_VALUE(jsonPayload.attributes, '$.duration_ms') AS INT64))) AS avg_ms FROM your-project.claude_code_otel.claude_code_otel_* WHERE JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'tool_result' AND JSON_VALUE(jsonPayload.attributes, '$.success') = 'false' GROUP BY tool_name, error_type ORDER BY n DESC LIMIT 20; ツール別の成功率も並べて見ておきます。 SELECT JSON_VALUE(jsonPayload.attributes, '$.tool_name') AS tool_name, COUNT(*) AS calls, COUNTIF(JSON_VALUE(jsonPayload.attributes, '$.success') = 'false') AS failures, ROUND( COUNTIF(JSON_VALUE(jsonPayload.attributes, '$.success') = 'false') / COUNT(*) * 100, 1 ) AS failure_pct FROM your-project.claude_code_otel.claude_code_otel_* WHERE JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'tool_result' GROUP BY tool_name HAVING calls >= 10 ORDER BY failure_pct DESC; ④ 「高かったプロンプト」を特定する prompt.id は、1つのユーザープロンプトから派生したすべてのイベントに付与される UUID です。これを使うと 「あの指示、結局いくらかかったの？」に答えられます。 WITH per_prompt AS ( SELECT JSON_VALUE(jsonPayload.attributes, '$."prompt.id"') AS prompt_id, MIN(timestamp) AS started_at, SUM(CASE WHEN JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'api_request' THEN CAST(JSON_VALUE(jsonPayload.attributes, '$.cost_usd') AS FLOAT64) END) AS cost_usd, COUNTIF(JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'api_request') AS api_calls, COUNTIF(JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'tool_result') AS tool_calls, COUNTIF(JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'tool_result' AND JSON_VALUE(jsonPayload.attributes, '$.success') = 'false') AS tool_failures FROM your-project.claude_code_otel.claude_code_otel_* WHERE JSON_VALUE(jsonPayload.attributes, '$."prompt.id"') IS NOT NULL GROUP BY prompt_id ) SELECT * FROM per_prompt ORDER BY cost_usd DESC LIMIT 20; コストの高い順に並べて tool_failures が多いものを見ると、「エージェントが同じところで空回りしていた」プロンプトが浮かび上がります。ここがプロンプト改善や CLAUDE.md 整備の投資先になります。 ⑤ サブエージェント・スキル・MCP の内訳 api_request には agent.name / skill.name / mcp_server.name / query_source が乗るので、コストを構成要素に分解できます。 SELECT COALESCE(JSON_VALUE(jsonPayload.attributes, '$."agent.name"'), '(main)') AS agent, JSON_VALUE(jsonPayload.attributes, '$.query_source') AS query_source, COUNT(*) AS requests, ROUND(SUM(CAST(JSON_VALUE(jsonPayload.attributes, '$.cost_usd') AS FLOAT64)), 4) AS cost_usd FROM your-project.claude_code_otel.claude_code_otel_* WHERE JSON_VALUE(jsonPayload.attributes, '$."event.name"') = 'api_request' GROUP BY agent, query_source ORDER BY cost_usd DESC; サブエージェントは並列で走らせると便利ですが、その分コストも積み上がります。「便利だから」で多用していたものが実際いくらだったのかが、ここで数字になります。 分散トレース（beta）で1プロンプトの中身を見る SQL で「どのプロンプトが高いか」を特定したら、次はその中身をトレースで追います。 export CLAUDE_CODE_ENABLE_TELEMETRY=1 export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 export OTEL_TRACES_EXPORTER=otlp これでプロンプトから API 呼び出し・ツール実行までが span として繋がります。 claude_code.interaction ← ユーザープロンプト1回 ├── claude_code.llm_request ← モデル呼び出し ├── claude_code.tool ← ツール実行 ├── claude_code.llm_request └── claude_code.tool Cloud Trace のウォーターフォールで見ると、待ち時間がモデル推論なのかツール実行なのかが一目で分かります。「体感が遅い」の正体がテストの実行時間だった、というようなことが可視化されます。 ! beta 機能のため、仕様が変わる可能性があります。恒久的なダッシュボードを組むより、まずは調査用途で使うのが無難です。 ハマりどころ 1. BigQuery にデータが来ない シンクは作成後のログにしか適用されません。また --log-filter が default_log_name と一致していないと何も流れません。まず Cloud Logging のコンソールで該当ログが見えているかを確認し、そこから下流を辿るのが早いです。 2. Collector は contrib 版が必要 googlecloud exporter はコアの Collector には入っていません。otelcol ではなく otelcol-contrib を使います。設定は validate サブコマンドで起動前に検査できます。 3. スキーマが想定と違う ログシンクのテーブルは自動生成なので、属性の入り方が環境によって変わります。SQL を書く前に INFORMATION_SCHEMA.COLUMNS を見るのが確実です。日付シャードテーブルになるため、ワイルドカード _* での参照と _TABLE_SUFFIX での絞り込みを忘れるとスキャン量が膨らみます。 4. テレメトリ自体のコスト Cloud Logging も BigQuery も従量課金です。エクスポート間隔を短くしたまま放置しないこと、シンクのフィルタを絞ることを意識してください。 実際に運用してみて 【要差し替え：実測値】 このセクションには、実際に自分の環境で1〜2週間回して得られた数字を入れてください。具体的には以下があると説得力が出ます。 期間中の総コストと、その内訳（モデル別 / サブエージェント別） キャッシュヒット率の実測値 最も失敗率が高かったツールと、そのエラー種別 最もコストが高かったプロンプトの正体と、そこから何を改善したか 改善前後でのコスト・失敗率の変化 【ここまで要差し替え】 まとめ Claude Code は OpenTelemetry をネイティブサポートしており、環境変数だけでメトリクス・イベント・トレースを出せる googlecloud exporter で GCP に送り、Cloud Logging のシンク経由で BigQuery に着地させれば、使い慣れた SQL で分析できる prompt.id による相関が強力で、「1つの指示にいくらかかったか」を単位にできる ツールの失敗率は普段見えないが、そのままトークンの浪費になっている AIエージェントは「便利だから使う」フェーズから「コストと品質を管理して使う」フェーズに移りつつあります。計測できないものは改善できないので、まずは可視化から始めるのが結局のところ近道でした。 データ基盤をやっている人間にとっては、送り先が BigQuery になった時点でいつもの仕事です。既存の SQL とダッシュボードの資産をそのまま流用できるのが、この構成の一番おいしいところだと思います。 ! GA4・BigQuery・Looker Studio・AI自動化の構築や設定代行を承っています（中小EC・個人事業主向け／スポット相談1万円〜）。「自社の場合はどうすれば？」のご相談も歓迎です。 ココナラでの販売実績は156件、評価は4.9をいただいています。 ウェブの便利屋（ろじかる） ココナラからのご依頼はこちら → Claudeの活用法を一緒に設計します

After 9 months it merged T_T at this point it’s not just a PR. Anyway it’s in opentelemetry’s go-build-tools, the thing every otel go project builds on :) opentelemetry-go-build-tools#1552 t.co

GitHub ActionsとDockerのビルドを1本のトレースにしてGrafana Cloudで見る ! 2026年10月27日に、GitHubホストランナーのUbuntu 24.04で次のバージョンを使って確認した内容です。送信するシグナルはトレースだけです。 Docker Engine 28.0.4 Buildx v0.29.1 BuildKit v0.33.1 OpenTelemetry Collector contrib v0.164.0 GitHubレシーバーのトレース機能は、このバージョンではalphaです。後述する親スパンのID計算もレシーバーの実装に依存します。 はじめに 先日書いたDockerのビルドを遅くしたコミットをGrafana Tempoのトレースで見つけるでは、手元のビルドをトレースにして、遅くなったコミットやDockerfileの手順を調べました。次のステップはCIです。GitHub Actionsの画面で「Dockerのビルドが遅い」とわかっても、そのステップの中で何に時間がかかったかを調べるには、ビルドのログや記録を別に開く必要があります。ワークフロー全体からDockerfileの手順まで、同じトレースの中でたどれるようにしてみました。 TL;DR BuildKitからGrafana CloudのOTLPエンドポイントへトレースを送信できることを確認しました。CIのステップにつなぐには、OpenTelemetry CollectorのGitHubレシーバーがステップに付けるIDをビルドの前に計算し、TRACEPARENTでBuildxへ渡します。CI側のスパンを送るのは、完了した実行の情報をGitHub APIで取得し、Actionsの中で一時的に動かすCollectorです。その結果、ワークフローと2つのジョブが1本のトレースにまとまり、buildジョブのビルドステップの下にBuildKitのスパンがつながりました。 2つの送信経路 今回の構成では、ビルドのトレースとGitHub Actionsのトレースを別々に送ります。両方でトレースIDと親スパンIDを合わせると、Grafana Cloudで1本に表示できます。 ビルド側の送信元は、OpenTelemetryでトレースを送る機能を持つBuildKitです。Dockerfileの各手順のスパンに加えて、クライアントであるBuildxのスパンも、BuildKitへ転送してまとめて送ります。 一方、GitHub Actions側には、BuildKitのようにトレースを送る送信元がありません。ワークフローの実行の記録はWebhookのイベントやREST APIで取得できますが、トレースの形式にはなっていないためです。この記録をスパンに変換するのが、OpenTelemetry CollectorのGitHubレシーバーです。GitHubレシーバーは、GitHubから届くworkflow_runとworkflow_jobのWebhookのイベントを受け取り、ワークフロー、ジョブ、ステップのスパンを作ります。レシーバーはCollectorに組み込んで動かすコンポーネントなので、GitHub Actions側のスパンはCollectorを通し、エクスポーターでGrafana Cloudへ送ります。 今回は、CI側のイベントの取得から送信までをActionsの中で実行する構成を選びました1。ワークフローの完了後に別のワークフローを起動し、そのランナーで一時的にCollectorを動かします。補助スクリプトが完了した実行の情報をGitHub APIで取得し、Webhookと同じ形式のイベントに組み立てて、ランナー内のGitHubレシーバーへ渡す構成です。BuildKitからの送信だけを試す段階では、このCollectorは使いません。 図1: 矢印はデータの移動を表します。左側では計算したTRACEPARENTをBuildxへ渡します。右側は完了後の別ワークフローで、補助スクリプトがGitHub APIの情報をイベント形式へ変換し、OpenTelemetry Collectorへ渡します。Collector内のGitHubレシーバーでCIスパンを生成して送信します。 題材はBusyBox 1.37.0の小さなイメージです。ファイルをコピーしてRUNで別ファイルへ書き出し、ビルドしたイメージを実行して出力も確かめました。実証用のファイル一式を公開しています。 BuildKitからGrafana Cloudへ まず、Actionsの親スパンを付けず、BuildKitからの送信だけを確認しました。Grafana Cloudで取得した接続情報を、リポジトリのActionsシークレットに次の名前で登録します。 シークレット 設定する値 GRAFANA_CLOUD_OTLP_ENDPOINT /otlpまでを含むOTLPのベースURL GRAFANA_CLOUD_OTLP_HEADERS Grafana Cloudの接続画面から取得したAuthorization=...の認証ヘッダー Actionsのステップでは、これらをOTEL_EXPORTER_OTLP_ENDPOINTとOTEL_EXPORTER_OTLP_HEADERSという環境変数に渡しています。scripts/setup-builder.shの内容は次のとおりです。 #!/usr/bin/env bash set -euo pipefail : "${OTEL_EXPORTER_OTLP_ENDPOINT:?Missing OTLP endpoint}" : "${OTEL_EXPORTER_OTLP_HEADERS:?Missing OTLP authentication headers}" docker buildx create --name ci-otel --driver docker-container \ --driver-opt image=moby/buildkit:v0.33.1 \ --driver-opt env.OTEL_TRACES_EXPORTER=otlp \ --driver-opt env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT%/}/v1/traces" \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}" \ --bootstrap >/dev/null docker-containerドライバーではBuildKitが専用のコンテナで動くので、BuildKitのOTLP設定は--driver-opt env.…でそのコンテナの環境変数として渡します。プロトコルはhttp/protobufで、OTLP/HTTPを使います。トレース専用のエンドポイントは、ベースURLの末尾に/v1/tracesを足したURLです。 このbuilderを選んでdocker buildx build --builder ci-otel --load …を実行すると、BuildxとBuildKitの両方のスパンがGrafana Cloudへ届きました。ベースラインの実行から取得したのは、build .をルートにした87スパンのトレースです。イメージのビルドと実行テストも成功しました。 図2: Grafana Cloudで取得したベースラインのトレース。BuildxとBuildKitの87スパンを確認できます2。 Actionsのステップを親にする ここまでのトレースのルートはbuild .です。GitHub Actionsのビルドステップを親にするには、Buildxを実行するステップに環境変数TRACEPARENTを設定します。Buildxは環境変数からトレースコンテキストを読み込む処理でこの値を取り込み、build .を指定された親の子にします。TRACEPARENTに入れるのは、接続先のトレースIDと、親になるステップのスパンIDです。 ただし、ビルドの時点では親になるステップのスパンはまだありません。CI側のスパンは、ワークフローの完了後に作るからです。それでも親のIDを指定できるのは、GitHubレシーバーがトレースIDとスパンIDを乱数ではなく実行IDやジョブIDから決まった規則で計算するためです。ビルドの前に同じ規則で計算すれば、あとで届くステップのスパンと同じIDが得られます。 補助スクリプトscripts/ci-parent.pyは、今回の実行ID、再実行回数、ジョブIDから、レシーバーのID生成の実装と同じ規則でIDを計算します3。計算部分を抜粋すると次のとおりです。 trace_id = hashlib.sha256(f"{run}{attempt}t".encode()).hexdigest()[:32] step_id = hashlib.sha256(f"{job_id}-Build Docker image-s".encode()).hexdigest()[16:32] traceparent = f"00-{trace_id}-{step_id}-01" TRACEPARENTの値は、W3C Trace Contextのtraceparentヘッダーと同じ00-トレースID-親スパンID-01の形式です。スクリプトは、この値をGITHUB_OUTPUTへ書き出します。 ワークフローでは、このスクリプトの出力をビルドステップだけに渡します。build.ymlから該当部分を抜粋すると、次のようになります。 - name: Calculate build step parent id: build_context env: GH_TOKEN: ${{ github.token }} run: python3 scripts/ci-parent.py - name: Build Docker image env: TRACEPARENT: ${{ steps.build_context.outputs.traceparent }} OTEL_RESOURCE_ATTRIBUTES: vcs.ref.head.revision=${{ github.sha }},ci.run.id=${{ github.run_id }},ci.run.attempt=${{ github.run_attempt }},ci.repository=${{ github.repository }},experiment.mode=linked run: docker buildx build --builder ci-otel --load --tag buildkit-otel-spike:ci . ステップのスパンIDはジョブIDとステップ名から決まるので、ステップ名はスクリプトの計算に使ったBuild Docker imageと一致させます。また、ステップ名はジョブ内で一意にしておきます。同じジョブに同名のステップが2つあると、レシーバーが両方に同じスパンIDを付けてしまうからです。 OTEL_RESOURCE_ATTRIBUTESでは、検索に使う実行IDやコミットを付けています。この属性が付くのはBuildxのスパンだけで、BuildKitのスパンには付きません。 CI側のスパンの送信 ビルドステップが終わっても、ワークフロー全体はまだ実行中です。CI側のスパンは、workflow_runのcompletedで起動する送信用のワークフローが作ります。 送信用のワークフローは、完了通知に含まれる実行IDと再実行回数を環境変数で補助スクリプトへ渡します。 OBSERVED_RUN_ID: ${{ github.event.workflow_run.id }} OBSERVED_RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }} 再実行回数を指定するのは、取得した時点で最新の再実行ではなく、通知された回のジョブを読むためです。補助スクリプトは、その回のジョブとステップの情報をGitHub APIで取得し、Webhookと同じ形式のイベントに組み立ててランナー内のCollectorへPOSTします。変換の処理はscripts/export-workflow.pyで確認できます。 collector.yamlの内容は次のとおりです4。 receivers: github: webhook: endpoint: 0.0.0.0:19418 path: /events health_path: /health service_name: github-actions scrapers: scraper: github_org: ymotongpoo processors: batch: timeout: 1s exporters: otlp_http/cloud: endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT} headers: Authorization: ${env:GRAFANA_AUTHORIZATION} timeout: 10s sending_queue: enabled: false retry_on_failure: enabled: false service: telemetry: logs: encoding: json pipelines: traces: receivers: [github] processors: [batch] exporters: [otlp_http/cloud] エクスポーターのendpointはベースURLのままです。otlp_httpのエクスポーターが/v1/tracesを付けるので、BuildKitにトレース専用のURLを渡したのとは指定方法が違います。認証ヘッダーには、BuildKitと同じシークレットからAuthorization=の後ろの値を取り出して渡します。 このCollectorは送信のたびに起動して停止する実証用の設定で、送信キューと再試行を無効にしています5。 自分の環境で試すには、2つのシークレットを登録し、ビルド用と送信用のワークフローをデフォルトブランチに置きます。そのうえでBuildKit linked traceを手動で実行し、ビルドとExport completed workflow traceの両方の完了を待ちます。APIの呼び出しに使うのはcontents: readとactions: readの権限を付けたGITHUB_TOKENで、GitHub AppやPATは不要です。 1本につながった結果 ビルド側の実行とCI側のスパンの送信は、どちらも成功しました。ワークフロー全体とビルドの内部が1本のトレースにつながっています。 図3: 同じ実行のワークフロー、2つのジョブ、各ステップが1本のトレースになりました。Build Docker imageの下にbuild .が接続しており、ビルド内部は折りたたんでいます。 サービスごとの内訳は次のとおりでした。 service.name スパン数 記録している処理 github-actions 21 ワークフロー、ジョブ、待ち時間、ステップ buildx 33 ビルドクライアントの処理 buildkitd 54 ビルドデーモンとDockerfileの手順 合計 108 同一トレースIDのスパン ビルド用のbuildと、その完了後に別のランナーで実行するverify-sourceの2ジョブが、同じルートにつながっています。CIの待ち時間やビルド内部の細部を省くと、親子関係は次のようになります。 BuildKit linked trace ├── build │ └── Build Docker image │ └── build . │ └── … │ └── [4/4] RUN cat message.txt > result.txt └── verify-source └── Verify source fixture build .の親IDはBuild Docker imageのスパンIDと一致し、取得した108スパンはすべて同じワークフローのルートへつながっていました6。 GrafanaのExploreでTempoのデータソースを選び、実行した時間帯に絞って、次のTraceQLで検索します。ci.run.idは、OTEL_RESOURCE_ATTRIBUTESでBuildxのスパンに付けた属性です。 { resource.ci.run.id = "40437818776" && name = "build ." } 前回の記事では、ルートのスパンだけを選ぶ条件（span:parentID = ""）でビルドの所要時間を集計しました。今回のビルドはbuild .にActionsの親があるので、この条件では選べません。手元のビルドとCIのビルドをまとめて選ぶ条件は、{ resource.service.name = "buildx" && name = "build ." }です。ベースラインとCIの両方のトレースが、この条件の検索結果に入ることを確認しました。 CI側のスパンは、送信用のワークフローが完了してから届きます。トレースを開くのは、両方のワークフローが完了してからにしてください7。 今回確認した範囲 BuildKitからGrafana Cloudへの送信と、2ジョブのワークフローへのビルド内部の接続までを確認しました。ただし、IDの生成規則はOpenTelemetryの共通の仕様ではなく、GitHubレシーバー v0.164.0の実装です。Collectorのバージョンやレシーバーのfeature gateを変えるときは、取得したトレースでbuild .が親のステップにつながっているかを確かめ直してください。 CI側の時刻はGitHub APIが秒単位で記録した値です。そのため、親子の接続が正しくても、BuildKitのスパンが親の時間範囲からわずかにはみ出すことがあります。今回の実行では、build .の終了が親のBuild Docker imageの終了を約0.84秒超えていました。 今回は信頼できるリポジトリでの手動実行だけで、matrix、reusable workflow、forkからのPR、実際の再実行、実行間でのキャッシュ再利用は検証していません8。 おわりに GitHub Actionsのワークフローとステップから、Dockerfileのどの手順やレジストリへの通信に時間を使ったかまで、1本のトレースでたどれるようになりました。自分のCIへ取り込むときには、まずBuildKitの送信だけを確認し、必要に応じてCIの親スパンを追加する順で試せます。 脚注 GitHubレシーバーは、常時稼働するCollectorでGitHubからのWebhookを受け取る使い方を想定しています。組織のCIを継続して記録するなら、その構成のほうが向いています。常時稼働のCollectorなら、送信キューと再試行を有効にして、Grafana Cloudへの送信に失敗したスパンを再送することもできます。ただし、GitHubからWebhookを受けるために、インターネットから届くエンドポイントを用意して運用する必要があります。今回はその準備をせずに試すため、Actionsの中で完結させました。 ↩︎ 図の上部に赤く表示されているHEAD HTTP HEADのエラーは、Docker Registryのトークン認証で最初に返る401の認証要求です。今回の実行では、トークン取得後の再試行が200となり、ビルドも成功しています。 ↩︎ ジョブの数値IDは、GitHub APIで今回の実行のジョブの一覧を取得し、名前がGITHUB_JOBと一致するジョブから取り出します。ジョブの表示名を変えたりmatrixを使ったりする場合は、別の方法でジョブを特定する必要があります。 ↩︎ v0.164.0のGitHubレシーバーは、トレースだけを使う場合でも、scrapersを1つ以上設定することを求めます。パイプラインはトレースだけなので、github_orgを使ったメトリクスの収集は動きません。 ↩︎ Collectorはbatchプロセッサーを通して送るので、レシーバーがHTTP 200を返しても、Grafana Cloudへの送信が成功したとは限りません。そこで補助スクリプトは、停止後のCollectorがJSON形式で出力したログを読み、levelがエラーを示していれば失敗として扱います。 ↩︎ 検証時のスクリプトと検証記録を公開しています。取得した108スパンは、スパンIDがすべて一意で、ルートが1つ、親が見つからないスパンは0でした。この検査では、送信したすべてのスパンがGrafana Cloudに届いたかまでは判定できません。 ↩︎ Tempoの最初の検索結果では、ルートのサービス名が<root span not yet received>と表示されました。その後にトレースIDで取得すると、CI側を含む108スパンがそろっていました。 ↩︎ 送信用のワークフローは、元の実行のコミットにある補助スクリプトを実行します。外部のPRが起点になる構成へ広げるときは、GitHubが説明するworkflow_runの権限と注意点を確認してください。 ↩︎

He co-leads the OTel Injector and Packaging SIGs. The Injector switches on auto-instrumentation at process start for Java, Node.js, .NET and Ruby, with no code changes. open-telemetry/opentelemetry-injector

OpenTelemetry の Semantic conventions を眺めていたら よくみている db.statement や db.connection_string が deprecated になっていたことに気づいてしまった。changelog を見る感じだと1.25 や 1.26 で大きく手が入ったぽいな。opentelemetry.io github.com/open-telemetry/semantic-conventions/bl…

Sounds good. Track opentelemetry.io#12073 then ;-)