Observability for AI Agents: Tracing, Evaluation and Cost Control in Production
How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices.
1. Why Is Observability Different for AI Agents?
Traditional backend observability is built around relatively predictable execution paths.
A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred.
AI agents introduce a different level of complexity.
An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input.
Consider a relatively simple task:
A user asks an AI assistant to analyze a merge request and identify potential problems.
Behind the scenes, the application might:
1. Receive the user's request.
2. Invoke a coordinator agent.
3. Call an LLM to determine the next action.
4. Retrieve the merge request from GitLab.
5. Read the changed files.
6. Delegate security analysis to another agent.
7. Execute additional tool calls.
8. Evaluate the collected results.
9. Generate a final report.
The user sees one response. The application has actually executed a multi-step workflow involving several components.
Now imagine the response takes 45 seconds instead of the expected 8 seconds.
Where should we investigate?
- Did the model take too long to respond?
- Did a tool call time out?
- Was the same operation executed repeatedly?
- Did the agent generate unnecessary steps?
- Did a subagent introduce additional latency?
- Was too much context sent to the model?
- Did a retry increase the overall execution time and cost?
Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why.
AI observability is about understanding the entire execution path, not just measuring the final API response.
This is where OpenTelemetry becomes particularly useful.
2. What Is OpenTelemetry?
<escape>**</escape><Entity value="OpenTelemetry (OTel)" category="software" disambig="Open-source observability framework"/> is an open-source observability framework for collecting, processing, and exporting telemetry data.
It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms.
Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported.
OpenTelemetry is built around three primary signals:
| Signal | What it tells us | AI agent example |
|---|---|---|
| Traces | How an operation was executed | Model call → tool → subagent |
| Metrics | How the system performs over time | Token usage, latency, error rate |
| Logs | What happened at a specific point | Tool timeout or agent exception |
These signals complement each other.
A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation.
OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector.
An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal.
References: OpenTelemetry Documentation · OpenTelemetry JavaScript
3. Understanding Distributed Tracing
Before discussing AI-specific instrumentation, we need to understand traces and spans.
A trace represents the complete execution path of an operation.
A span represents one individual unit of work within that execution.
For example:
Trace: POST /api/agent
|
+-- HTTP Request
|
+-- Coordinator Agent
|
+-- LLM Generation
|
+-- Tool: GitLab - Get Merge Request
|
+-- Tool: GitLab - Get Changed Files
|
+-- Security Agent
| |
| +-- LLM Generation
| |
| +-- Tool: Security Analysis
|
+-- Testing Agent
| |
| +-- LLM Generation
| |
| +-- Tool: Test Coverage
|
+-- Final Response Generation
Each span has information such as:
- Trace ID
- Span ID
- Parent span ID
- Start and end timestamps
- Duration
- Attributes
- Events
- Status and exception information
This creates a hierarchical representation of the complete execution.
The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them.
For AI agents, this distinction is important because one user request can generate many independent operations.
Why is context propagation important?
In distributed systems, a trace often crosses multiple services.
Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service.
Without context propagation, every service may generate its own independent trace.
We would see several unrelated operations instead of one complete workflow.
OpenTelemetry uses W3C Trace Context as its standard propagation format. The traceparent header carries trace and parent-span information between instrumented services.
The same principle applies to agent-based architectures.
When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution.
Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction.
The important point is that protocol-level interoperability and observability context propagation are separate concerns. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace.
References: OpenTelemetry Traces · Context Propagation · W3C Trace Context
4. OpenTelemetry and GenAI Semantic Conventions
One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers.
For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes.
This makes it difficult to compare different implementations using the same monitoring platform.
OpenTelemetry addresses this through Semantic Conventions.
Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently.
The GenAI semantic conventions extend this approach to generative AI applications.
Some important attributes include:
| Attribute | Purpose |
|---|---|
| gen_ai.provider.name | Identifies the model provider |
| gen_ai.request.model | Identifies the requested model |
| gen_ai.response.model | Identifies the actual response model |
| gen_ai.operation.name | Describes the operation |
| gen_ai.usage.input_tokens | Input token usage |
| gen_ai.usage.output_tokens | Output token usage |
| gen_ai.conversation.id | Associates operations with a conversation |
| gen_ai.agent.name | Identifies the agent |
| gen_ai.tool.name | Identifies the executed tool |
For example, an LLM invocation could be represented by:
{
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "example-model",
"gen_ai.usage.input_tokens": 1200,
"gen_ai.usage.output_tokens": 350
}
This is a simplified span attribute example.
The benefit is that observability platforms do not need to understand a different custom schema for every application.
Agent-specific conventions
The GenAI semantic conventions also define operations for agents and workflows.
Some examples include:
- invoke_agent
- invoke_workflow
- execute_tool
- plan
- retrieval
This is particularly valuable for multi-agent architectures.
We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents.
For example:
invoke_workflow merge-request-review
|
+-- invoke_agent coordinator
| |
| +-- chat model
| +-- execute_tool fetch_merge_request
|
+-- invoke_agent security-agent
| |
| +-- chat model
| +-- execute_tool analyze_vulnerabilities
|
+-- invoke_agent testing-agent
| |
| +-- chat model
| +-- execute_tool analyze_test_coverage
|
+-- Final Aggregation
This makes the execution model much easier to understand.
An important maturity consideration: As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract.
The current convention documents include:
- Semantic Conventions for GenAI Spans
- Semantic Conventions for Agent Spans
- Semantic Conventions for GenAI Metrics
5. Integrating OpenTelemetry with Vercel AI SDK
Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration.
The AI SDK already supports OpenTelemetry-based instrumentation.
It can generate spans for:
- Text generation
- Streaming generation
- Individual model calls
- Agent execution
- Tool execution
- Embedding operations
- Reranking
This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility.
Step 1: Install the packages
For an AI SDK v7 application:
pnpm add @ai-sdk/otel @vercel/otel
The OpenTelemetry integration is provided through @ai-sdk/otel.
This is an important version-specific detail.
In AI SDK v7, OpenTelemetry integration moved out of the core ai package. Older examples using the v6 experimental_telemetry configuration should not be copied directly into a v7 application.
Step 2: Register telemetry in Next.js
Create an instrumentation.ts file in the project root.
import { registerOTel } from '@vercel/otel';
import { registerTelemetry } from 'ai';
import { OpenTelemetry } from '@ai-sdk/otel';
export function register() {
registerOTel({
serviceName: 'ai-agent-platform',
});
registerTelemetry(
new OpenTelemetry({
recordInputs: false,
recordOutputs: false,
}),
);
}
There are two responsibilities here:
- registerOTel initializes the OpenTelemetry infrastructure for the Next.js application.
- registerTelemetry registers the AI SDK-specific integration that creates GenAI spans.
I have disabled input and output capture in this example intentionally.
AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision.
We will return to this in the security section.
For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup.
Step 3: Instrument AI SDK calls
Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration.
We can also provide a functionId to identify the business operation or agent.
import { generateText } from 'ai';
const result = await generateText({
model: myModel,
instructions: `
You are a financial analysis assistant.
Analyze the supplied information carefully.
`,
prompt: 'Analyze the latest portfolio performance.',
telemetry: {
functionId: 'portfolio-analysis-agent',
},
});
The functionId helps distinguish operations in traces.
For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as:
invoke_agent portfolio-analysis-agent
Step 4: Understand the generated spans
The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions.
For generateText and streamText, the integration records three main types of spans:
1. Agent invocation span
The root span represents the complete generation operation, including its execution steps and tool calls.
Example:
invoke_agent portfolio-analysis-agent
2. Step and model-call spans
Each model interaction is represented by step and provider-call spans.
These allow us to understand how many model calls occurred during an operation and how long each one took.
3. Tool execution spans
Each client-side tool execution receives its own span.
For example:
invoke_agent portfolio-analysis-agent
|
+-- step 0
| |
| +-- chat example-model
| +-- execute_tool get_portfolio
|
+-- step 1
| |
| +-- chat example-model
| +-- execute_tool get_market_data
|
+-- step 2
|
+-- chat example-model
The resulting trace provides a much more complete representation of an agent workflow than a single model request log.
The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available.
Step 5: Add additional application context
Sometimes the standard GenAI attributes are not enough.
For example, we may want to know which business workflow initiated an operation or which application component owns it.
The AI SDK provides enrichSpan for adding custom attributes:
import { registerTelemetry } from 'ai';
import { OpenTelemetry } from '@ai-sdk/otel';
registerTelemetry(
new OpenTelemetry({
recordInputs: false,
recordOutputs: false,
enrichSpan: ({ spanType, operationId }) => ({
'app.component': 'portfolio-assistant',
'app.operation_id': operationId,
'app.span_type': spanType,
}),
}),
);
The exact context available to enrichSpan depends on the span and the SDK's integration API.
The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry.
References: AI SDK Telemetry · Next.js OpenTelemetry · AI SDK v7 Migration Guide
6. Instrumenting Your Own Agent Runtime
Built-in instrumentation is useful, but it does not automatically understand every application-level decision.
For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval.
These are application-level operations.
We should instrument them explicitly.
OpenTelemetry provides a tracing API for creating custom spans.
Consider a custom agent orchestration function:
import {
trace,
SpanStatusCode,
} from '@opentelemetry/api';
const tracer = trace.getTracer('agent-orchestrator');
export async function runReviewWorkflow() {
return tracer.startActiveSpan(
'workflow.merge-request-review',
async (span) => {
span.setAttribute(
'workflow.name',
'merge-request-review',
);
span.setAttribute(
'workflow.version',
'1.0.0',
);
try {
const result = await executeReview();
span.setStatus({
code: SpanStatusCode.OK,
});
return result;
} catch (cause) {
const error =
cause instanceof Error
? cause
: new Error(String(cause));
span.recordException(error);
span.setStatus({
code: SpanStatusCode.ERROR,
});
throw error;
} finally {
span.end();
}
},
);
}
Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated.
We now have two levels of observability:
- Framework-level instrumentation describes model and tool execution.
- Application-level instrumentation describes business workflows and orchestration decisions.
This distinction is important.
We should not expect a model instrumentation library to understand the complete business meaning of an application request.
Instrumenting remote agent calls
For A2A-based systems, distributed tracing becomes even more important.
Imagine a coordinator running in one service and a security agent running in another service.
The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated.
Conceptually:
Trace: merge-request-review
|
+-- Coordinator Agent
| |
| +-- Model Call
| |
| +-- A2A: Delegate to Security Agent
| |
| +-- Security Agent
| |
| +-- Model Call
| |
| +-- Tool Execution
|
+-- Final Aggregation
The HTTP request must carry the appropriate trace context, typically through W3C traceparent and tracestate headers.
For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context.
This is how we avoid fragmented traces across independently deployed agents.
7. What Should We Measure?
Collecting telemetry is not enough.
We need to define which metrics are useful for understanding the health and quality of an agent platform.
I would divide AI observability metrics into five categories.
7.1. Latency Metrics
These describe how long operations take.
| Metric | Why it matters |
|---|---|
| End-to-end workflow duration | Measures total user-facing execution time |
| Model request duration | Identifies slow inference calls |
| Time to first token | Measures perceived response latency |
| Tool execution duration | Identifies slow external operations |
| Agent invocation duration | Measures the execution time of one agent |
| Queue wait duration | Identifies scheduling bottlenecks |
For streaming applications, time to first token is particularly useful.
A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds.
7.2. Reliability Metrics
These help us understand whether workflows complete successfully.
Examples:
- Agent execution success rate
- Tool execution failure rate
- Model request failure rate
- Timeout rate
- Retry rate
- Task cancellation rate
- Human intervention rate
- Incomplete workflow rate
A high retry rate can be an early indicator of a problematic integration or an unstable external dependency.
We should also distinguish between a successful model response and a successful business workflow.
An LLM can return a perfectly valid response while the overall task fails to achieve its objective.
7.3. Token and Model Usage
Token usage should be tracked at multiple levels.
| Metric | Description |
|---|---|
| Input tokens | Tokens sent to the model |
| Output tokens | Tokens generated by the model |
| Cached input tokens | Tokens served from provider cache |
| Reasoning tokens | Model-reported reasoning token usage |
| Tokens per workflow | Total usage across an entire task |
| Tokens per agent | Usage attributed to an individual agent |
For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response.
7.4. Agent and Tool Behavior
These metrics describe how efficiently agents perform their work.
Examples:
- Number of tool calls per task
- Number of model calls per agent invocation
- Number of agent handoffs
- Repeated tool-call frequency
- Tool-selection success
- Average steps per workflow
- Average number of clarification requests
An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem.
The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs.
7.5. Quality Metrics
This is where AI observability starts to overlap with AI evaluation.
Examples:
- Answer correctness
- Groundedness
- Retrieval relevance
- Tool-selection accuracy
- Task completion
- Policy adherence
- Human acceptance rate
These metrics cannot always be derived from execution telemetry alone.
We need an evaluation strategy to measure the actual quality of the result.
8. Cost Control: Understanding the Real Cost of an Agent
One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow.
An agent-based application may consume resources across several layers:
- LLM inference
- Embedding generation
- Reranking
- External API calls
- Database queries
- Vector database operations
- Container or sandbox execution
- Remote agent invocations
- Infrastructure and networking
The total cost should reflect the entire execution path.
A useful starting point is to define a model cost formula:
[
C_{\text{model}} =
\sum_{i=1}^{n}
\left(
T_{\text{input},i}P_{\text{input},i}
+
T_{\text{output},i}P_{\text{output},i}
\right)
]
Where:
- (T) represents token usage.
- (P) represents the applicable price per token.
- (i) represents an individual model operation.
The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities.
We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets.
The complete workflow cost can then be represented as:
[
C_{\text{workflow}} =
C_{\text{model}}
+
C_{\text{tools}}
+
C_{\text{compute}}
+
C_{\text{storage}}
]
This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available.
Why is cost per successful task more useful?
Imagine two implementations:
| Metric | Agent A | Agent B |
|---|---:|---:|
| Average cost per attempt | $0.04 | $0.07 |
| Successful completion rate | 90% | 98% |
| Cost per successful task | ~$0.044 | ~$0.071 |
Agent A is cheaper per attempt, but Agent B completes more tasks successfully.
Neither is automatically better.
The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts.
A useful metric is:
[
\text{Cost per successful task} =
\frac{\text{Total execution cost}}
{\text{Number of successful tasks}}
]
The numerator should include failed attempts that consumed resources, rather than counting only successful runs.
Model selection and routing
Observability data can also help us make better model-routing decisions.
For example:
- Use a smaller model for classification or simple extraction.
- Use a more capable model for complex analysis.
- Avoid unnecessary model calls when a deterministic operation is sufficient.
- Reuse cached context when appropriate.
- Set explicit iteration and tool-call limits.
- Stop an agent when a task exceeds its budget.
The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request.
Token metrics in OpenTelemetry
The current GenAI token metric conventions include proposed counters such as:
- gen_ai.client.inference.usage.input_tokens
- gen_ai.client.inference.usage.output_tokens
- gen_ai.client.inference.usage.cache_read.input_tokens
- gen_ai.client.inference.usage.reasoning.output_tokens
These conventions are still in Development.
There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics.
A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations.
For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices.
This makes historical cost calculations more reliable when provider pricing changes.
References: GenAI Token Metrics · GenAI Metrics
9. Observability Is Not Evaluation
This distinction is worth emphasizing.
Observability tells us what happened during execution.
Evaluation tells us whether the result was good enough.
A trace can show that an agent:
- Called the correct model.
- Used three tools.
- Retrieved five documents.
- Completed within four seconds.
- Returned a response without an exception.
But none of these facts proves that the answer was correct.
Consider a RAG application.
The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information.
From an operational perspective, the request succeeded.
From a quality perspective, it failed.
We need both observability and evaluation.
Offline Evaluation
Offline evaluation measures an AI application against a predefined dataset.
For example, we can create 100 representative questions and expected outcomes.
Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again.
This helps identify regressions before deploying changes.
Useful evaluation dimensions include:
- Answer correctness
- Relevance
- Groundedness
- Tool selection
- Expected task completion
- Policy compliance
Online Evaluation
Online evaluation measures actual production interactions.
We can select a sample of production traces and evaluate them using:
- Rule-based checks
- Code-based evaluators
- LLM-as-a-Judge
- Human feedback
The important part is to turn real production failures into future test cases.
For example:
1. A production trace shows that an agent selected the wrong tool.
2. We identify the relevant input and expected behavior.
3. We create a regression dataset entry.
4. We update the prompt or agent implementation.
5. We rerun offline evaluations.
6. We deploy and monitor the next set of production traces.
This creates a continuous improvement loop.
Production Traces
|
v
Failure Analysis
|
v
Evaluation Dataset
|
v
Offline Evaluation
|
v
Prompt / Model / Agent Changes
|
v
Deployment
|
v
Online Evaluation
|
+------> New Regression Cases
Tools worth evaluating
Two examples of AI-focused observability and evaluation platforms are:
Langfuse
Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements.
Documentation: langfuse.com
Arize Phoenix
An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions.
It provides tracing, evaluation, datasets, experiments, and debugging capabilities.
Documentation: phoenix.arize.com
These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy.
Reference: Langfuse Evaluation Concepts · Arize Phoenix
10. The Role of OpenTelemetry Collector
In a production architecture, applications should not necessarily export telemetry directly to every observability backend.
This is where the OpenTelemetry Collector becomes useful.
The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations.
It provides a central place for:
- Batching telemetry
- Filtering unwanted data
- Applying processors
- Handling authentication
- Reducing telemetry volume
- Sampling traces
- Routing signals to different backends
A simplified architecture looks like this:
AI Applications
|
+----------+----------+
| | |
v v v
Next.js .NET API Python Agent
| | |
+----------+----------+
|
OTLP
|
v
OpenTelemetry Collector
|
+--------+--------+
| | |
v v v
Traces Metrics Logs
| | |
v v v
Jaeger Prometheus Log Backend
|
v
Grafana
The same telemetry can also be routed to AI-specific observability platforms where supported.
Why centralize collection?
In enterprise environments, we may have several applications and teams using different frameworks and languages.
A Collector provides a common ingestion point and makes it possible to apply centralized policies.
For example, a Collector can filter sensitive attributes before telemetry reaches the final backend.
It also allows us to change exporters or observability providers without changing the instrumentation in every application.
Agent and gateway deployment
OpenTelemetry supports different Collector deployment patterns.
A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors.
This provides a separation between local telemetry collection and centralized processing.
However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling.
Reference: OpenTelemetry Collector · Gateway Deployment Pattern
11. Sampling: How Do We Control Telemetry Volume?
Agent-based workflows can generate a significant amount of telemetry.
Consider an agent that performs 15 model calls and 20 tool calls during a single task.
If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive.
Sampling helps control this volume.
There are two common approaches.
Head-Based Sampling
The sampling decision is made when the trace starts.
For example, we may decide to sample 10% of incoming requests.
This is relatively simple and reduces the amount of telemetry generated or exported.
However, we do not yet know whether a trace will eventually fail or become unusually slow.
Tail-Based Sampling
The sampling decision is made after considering the trace's collected spans.
This makes it possible to retain:
- All failed traces.
- All traces exceeding a latency threshold.
- Traces containing specific tool failures.
- A percentage of successful traces.
For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed.
However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance.
This generally requires trace-ID-aware routing when using multiple Collector instances.
A practical approach is to combine sampling strategies:
- Preserve error and high-latency traces.
- Retain a representative sample of successful traces.
- Maintain complete aggregate metrics where possible.
- Apply stricter retention policies to verbose telemetry.
Reference: OpenTelemetry Sampling · Tail Sampling Processor
12. Security and Privacy Considerations
This is one of the most important aspects of AI observability in enterprise environments.
Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names.
AI telemetry can contain much more sensitive information.
For example:
- User prompts
- System instructions
- Generated responses
- Tool arguments
- Tool results
- Retrieved documents
- Conversation context
- Customer or business identifiers
Sending all this information to an observability backend can create additional privacy and data governance risks.
Do not capture prompts and responses by default
A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise.
We can disable this behavior:
registerTelemetry(
new OpenTelemetry({
recordInputs: false,
recordOutputs: false,
}),
);
For many production applications, this is a better starting point.
We can still collect useful operational telemetry:
- Model identifiers
- Token counts
- Duration
- Finish reasons
- Tool execution times
- Error information
- Workflow identifiers
without exporting the complete conversation.
Apply data minimization
Before exporting telemetry, ask whether each attribute is genuinely needed.
Avoid sending:
- Access tokens
- Authorization headers
- Passwords
- Raw customer records
- Full database query results
- Unfiltered tool arguments
For sensitive workflows, use approved metadata rather than full payloads.
Separate operational telemetry from application data
An observability backend should not automatically become another storage system for business data.
The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely.
Define access and retention policies
Enterprise environments should define:
- Who can access traces.
- Which teams can view prompt and response content.
- How long telemetry is retained.
- How data is deleted.
- Which environments are allowed to capture full content.
- How telemetry exports are authenticated and encrypted.
A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production.
Reference: OpenTelemetry GenAI Content Capture
13. Limitations and Trade-offs
OpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself.
Instrumentation coverage varies
Not every framework, provider, or agent runtime exposes the same information.
Some integrations can automatically trace model calls and tool execution, while others require custom instrumentation.
We should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility.
GenAI semantic conventions are evolving
The GenAI conventions are still under development.
Attribute names, operation models, and implementation support may evolve.
Teams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity.
Observability does not guarantee answer quality
A trace can explain what happened but does not prove that the model's result is correct.
Evaluation datasets, deterministic checks, and human feedback remain necessary.
Telemetry has its own cost
Collecting every input, output, tool argument, and execution event can generate significant storage and processing overhead.
We should measure the cost of observability itself.
High-cardinality dimensions can become expensive
Adding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series.
Use traces for request-level correlation and keep metric dimensions bounded.
Distributed tracing requires correct context propagation
A2A communication, queues, background tasks, and remote tool execution may cross process boundaries.
If context is lost, we can end up with disconnected traces.
AI evaluation requires careful design
LLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased.
A single evaluation score should not become the only definition of quality.
Combine deterministic checks, representative datasets, human review, and calibrated evaluators.
14. What Can an AI Chapter Lead Contribute?
I think this is where observability moves from an implementation detail to an engineering capability.
When multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms.
Over time, this creates fragmented visibility.
One team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model.
As an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications.
1. Define a Common Telemetry Standard
Establish a baseline for the telemetry every agent-based application should produce.
For example:
| Category | Required information |
|---|---|
| Application | Service name, environment, version |
| Workflow | Workflow name, workflow version |
| Agent | Agent name, execution status |
| Model | Provider, requested model, response model |
| Usage | Input/output tokens, cache usage |
| Tools | Tool name, duration, execution status |
| Execution | Trace ID, span ID, error information |
| Business outcome | Completed, failed, canceled, escalated |
The objective is to make observability consistent without forcing every team to use the same agent framework.
2. Build a Shared Instrumentation Package
We can create an internal TypeScript package that wraps the common OpenTelemetry configuration.
For example:
@company/ai-observability
|
+-- OpenTelemetry initialization
+-- AI SDK integration
+-- Agent span utilities
+-- Workflow instrumentation
+-- Tool execution tracking
+-- Cost attribution
+-- Evaluation metadata
+-- Privacy and redaction policies
This package can provide standard helpers for application-level tracing and reduce duplicated implementation.
For teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations.
3. Standardize Agent and Workflow Metrics
We should define a common set of metrics that helps teams understand both technical performance and business outcomes.
Examples:
- Agent success rate
- Workflow completion rate
- Tool-call failure rate
- Model latency
- Cost per successful task
- Average token usage
- Human intervention rate
- Evaluation quality scores
A shared dashboard can make it easier to compare systems and identify performance regressions.
4. Establish an Internal Evaluation Framework
Observability without evaluation gives us only part of the picture.
We can create reusable evaluation datasets and testing patterns for common agent behaviors.
For example:
- Tool selection accuracy
- Missing-source detection
- Correct handling of incomplete input
- Permission violations
- Unsupported claims
- Failure recovery
- Task completion quality
This is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified.
5. Establish Privacy and Data Governance
Define what can and cannot be captured by default.
The shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads.
This should include clear ownership, access policies, retention, and audit requirements.
6. Run a Cross-Team Pilot
A useful pilot would be a multi-agent workflow that involves several independently deployed services.
For example, a merge request review involving coordinator, security, and testing agents.
The pilot should demonstrate:
1. One trace across the complete request.
2. Parent-child relationships between agents.
3. Model and tool execution spans.
4. Token and cost attribution.
5. Error and retry visibility.
6. Evaluation results associated with the completed workflow.
This would provide a reference architecture that other teams can reuse.
7. Make Observability Part of the Development Lifecycle
Observability should not begin after production deployment.
We can introduce telemetry and evaluation requirements into the development process itself.
For example:
- New agents must have execution tracing.
- New tools must expose execution status and duration.
- Prompt changes must pass relevant regression evaluations.
- New workflows must define success and failure criteria.
- Production deployments must have basic operational dashboards.
This makes observability a standard engineering practice rather than a separate monitoring task.
15. A Practical Adoption Roadmap
I would avoid introducing every observability component at once.
A phased approach is more useful.
| Phase | Focus | Expected result |
|---|---|---|
| 1 | OpenTelemetry initialization | Basic traces and metrics |
| 2 | AI SDK instrumentation | Model and tool visibility |
| 3 | Custom agent spans | Complete workflow tracing |
| 4 | Cost attribution | Usage and cost per task |
| 5 | Evaluation | Offline and online quality checks |
| 6 | Security and governance | Privacy and access controls |
| 7 | Shared dashboards | Cross-team operational visibility |
The first milestone should be simple:
For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result.
Once this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce.
Conclusion
AI agents introduce a new level of complexity to application observability.
A single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution.
OpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components.
Combined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production.
But the most important distinction remains:
Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it.
For an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices.
The goal should not be to collect more telemetry.
It should be to make every production agent workflow understandable, measurable, and continuously improvable.
---
References and Further Reading
OpenTelemetry
1. OpenTelemetry Official Documentation — Core concepts, signals, instrumentation, and architecture.
2. OpenTelemetry JavaScript — TypeScript and Node.js instrumentation.
3. OpenTelemetry Tracing — Traces, spans, and distributed execution.
4. OpenTelemetry Context Propagation — Distributed trace context.
5. OpenTelemetry Collector — Receiving, processing, and exporting telemetry.
6. OpenTelemetry Gateway Deployment — Centralized telemetry collection.
7. OpenTelemetry Sampling — Head-based and tail-based sampling.
GenAI Semantic Conventions
8. GenAI Semantic Conventions — Spans — Model operations, retrieval, and tool execution.
9. GenAI Agent Spans — Agent invocation, workflows, and tool spans.
10. GenAI Metrics — Agent duration, tool calls, and workflow metrics.
11. GenAI Token Metrics — Token consumption and usage measurements.
Vercel AI SDK
12. AI SDK Telemetry — OpenTelemetry integration, span hierarchy, and custom attributes.
13. AI SDK v7 Migration Guide — Version-specific telemetry changes.
14. Next.js OpenTelemetry — Application-level instrumentation.
AI Evaluation and Observability Platforms
15. Langfuse Evaluation — Online and offline evaluation.
16. Langfuse Core Concepts — Datasets, experiments, and evaluators.
17. Arize Phoenix — Open-source AI observability and evaluation.
18. OpenInference — OpenTelemetry-based instrumentation conventions for AI applications.
Claude Code Sends a Background Agent's Whole Transcript Every 30s for a 3-5 Word Status
- Book: LLM Observability Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
You start three background agents on a refactor and go make coffee. While each one works, about every 30 seconds, Claude Code sends that agent's transcript back to the agent's own model with one request: "describe your most recent action in 3-5 words." The answer becomes the little label next to the agent in the agents panel.
That is the finding in a gist from the flamerix-ai account that reached 584 points and 236 comments on Hacker News at the time of writing. In the author's OpenTelemetry data for one week, the agent_summary calls were 16.4% of estimated spend and 19.7% of API requests. The main conversation was 15.7%. A Claude Code engineer replied in the thread that the basic mechanism is right, and that the typical share is far lower. Both can be true. Which one is your bill depends on how you work.
How the agent_summary timer works
Per the gist, the feature has been on by default since Claude Code 2.1.232, in August 2026. Each actively working background agent gets a timer. When it fires, Claude Code forks a request that carries the agent's transcript to the agent's own model, usually Opus, and asks for the 3-5 word description. The gist puts the input-to-output ratio at about 9,000 tokens in for every token out.
Three details matter:
- The interval comes from the server. The gist says it is a per-model-family server flag, clamped to 30-240 seconds, and the default is already the minimum. No value you can set turns it off.
- It uses the agent's model. A GitHub issue on 2.1.312 measured a Haiku subagent and saw the calls go to Haiku, so an Opus subagent gets summarized by Opus.
- It skips the transcript. The original feature request found the calls run with skipTranscript: true, so cost tools that read session transcripts never see them.
The Claude Code reply on Hacker News narrows the scope: it "only applies to background agents that are actively working." Idle agents don't trigger it. Workflow agents don't get summary timers either, according to a commenter on #86879.
This is not the status line you configure yourself. The status line docs say that one "runs locally and does not consume API tokens." The agent_summary label lives in the agents panel, and it is a model call.
!A stopwatch whose hand pulls a long paper scroll into a large machine, which prints a single tiny tag at the other end
The trade in one picture: the whole transcript in, a few words out.
There is no supported off switch. CLAUDE_CODE_AGENT_PROGRESS_SUMMARIES and the SDK's agentProgressSummaries can only turn summaries on, per the gist. CLAUDE_CODE_FORK_SUBAGENT=0 has mixed reports on #86879 and #105981: some people saw zero calls, others still saw summary calls. One commenter suggested the difference comes from a settings env block overriding the shell export. It also disables fork mode, so setting it loses you more than the summaries.
What the Bun preload does, and why it breaks
The gist ships an MIT-licensed no-agent-summary.js. It relies on Claude Code's native build running on Bun, and on Bun's BUN_OPTIONS variable, which "prepends command-line arguments to any Bun execution." So BUN_OPTIONS=--preload=<path> loads a script before Claude Code's own code runs. The gist says to use the = form, because a space breaks bun subcommands, and to run claude daemon stop --any so background processes pick up the new environment.
The script wraps the global setTimeout. When a callback's source text contains one of two anchors, [AgentSummary] Timer fired or agent_summary, it returns an inert timer instead of scheduling the real one. It also checks that it is running inside Claude Code before doing anything. The technique, in its smallest form, looks like this:
// Illustration of the technique. Not the gist's script.
const realSetTimeout = globalThis.setTimeout;
const ANCHOR = "[AgentSummary] Timer fired";
globalThis.setTimeout = (fn, ms, ...args) => {
const src = typeof fn === "function" ? String(fn) : "";
if (src.includes(ANCHOR)) {
const t = realSetTimeout(() => {}, 2 ** 31 - 1);
t.unref?.();
return t;
}
return realSetTimeout(fn, ms, ...args);
};
Don't run that sketch. It has none of the gist's gating or verification, and it shows exactly why this approach is fragile. It matches source text inside a minified binary. Rename one log line, or move the timer off the global setTimeout, and summaries come back with no error. The gist says this itself and asks you to verify after every update. A commenter on #86879 also warned against installing a preload by curl-ing the latest version, since whatever it downloads then runs inside every Claude Code process.
Two more caveats from the gist's own pages. Its README says the summary never reaches the model or your conversation, but a commenter on Claude Code 2.1.314 found accepted summary text stored in task.progress.summary in later context, and called that claim too strong. And the license section says plainly: "Changing a client's runtime behaviour may conflict with the terms of the service you use it with; checking that is on you." I haven't run the hook myself.
Measure your own agent_summary share
You don't need the hook to find out whether this matters to you. Claude Code's monitoring docs log a claude_code.api_request event for each request, with cost_usd and a query_source attribute, the "subsystem that issued the request." The gist grouped those events by query_source, where summaries show up as agent_summary.
Use the events, not the metrics. On the claude_code.cost.usage metric, query_source collapses to main, subagent or auxiliary, which hides the summaries inside a bucket. Turn on telemetry and the logs exporter:
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
# your collector's OTLP gRPC address, usually port 4317
export OTEL_EXPORTER_OTLP_ENDPOINT="$COLLECTOR_URL"
If you don't have a backend, an OpenTelemetry Collector with the file exporter writes newline-delimited OTLP JSON:
receivers:
otlp:
protocols:
grpc:
exporters:
file:
path: ./claude-logs.jsonl
service:
pipelines:
logs:
receivers: [otlp]
exporters: [file]
Work normally for a few days, then group by source with jq and awk:
jq -r '.resourceLogs[].scopeLogs[].logRecords[]
| [.attributes[] | {(.key): (.value | to_entries[0].value)}]
| add
| select(.["event.name"] == "api_request")
| [(.query_source // "unknown"), (.cost_usd // 0 | tonumber)]
| @tsv' claude-logs.jsonl \
| awk -F'\t' '
{ cost[$1] += $2; reqs[$1]++; tc += $2; tr++ }
END { for (k in cost)
printf "%-22s %5.1f%% cost %5.1f%% reqs\n",
k, 100 * cost[k] / tc, 100 * reqs[k] / tr }'
Each output row is one query_source with its share of estimated cost and of requests. Two things to keep in mind when you read it. cost_usd is Claude Code's own estimate at list price, not an invoice, as the cost docs say. And the share depends on how much of your day runs as background agents, which is the whole argument.
!Paper receipts dropping from a conveyor into labelled bins, with one bin filling faster than expected
Group the request events by source and the summaries get a bin of their own.
Why the numbers disagree
Here is every figure on the table, each from a different method and workload:
| Source | Share | What it measured |
|---|---|---|
| The gist | 16.4% of spend, 19.7% of requests | The author's setup, one week, OpenTelemetry estimates |
| Issue #86879 | 22.7% of requests, 18.0% of weighted tokens | One 4-hour session, 27 subagents, 2.1.233 |
| Commenters on #86879 | From about 10% of spend over 30 days to 26% of cache reads in one session | Separate workloads and methods |
| Claude Code engineer on HN | About 2% of cost, just under 4% of requests | Telemetry users, last week |
| Same reply, terminal only | About 6% of cost | Where nearly all these calls happen |
The high numbers come from people running mostly background agents. The engineer said as much: 16.4% "sounds like it was for a team running mostly background agents, and doesn't seem like a typical session." The low numbers are an average over everyone who sends telemetry, most of whom run few background agents. Neither one is your number. A replay of the 30-second timer over one person's transcripts, posted on the gist, landed at about 8%, partly because it assumed a warm cache.
The cache is the part that moves the bill. The engineer said a fix for "summaries that miss the prompt cache" shipped in 2.1.311 and later, calling misses the most expensive case. The team is also working on "a much cheaper mode that sends short notes to a smaller model instead of the full transcript." The reply adds: "we agree that's a lot for a status line." So if you measured before 2.1.311, measure again on a current build before you decide anything.
A status that doesn't need a model call
I build Hermes IDE, so weigh this section with that in mind. First, what it doesn't do: by default, Hermes runs Claude Code in a real terminal, unchanged, so background agents that Claude Code starts there get summarized exactly as they would anywhere else.
What Hermes does differently is its own per-session status. Every session shows one state: working, needs approval, asked you, done, error or idle, and Hermes says when a state is a guess. The README says Hermes "observes what agents report on purpose and what git shows," and the architecture guide describes that status as a state machine driven by the terminal output. Neither document describes a model call in that path. The cost: a state tells you less than a sentence about what the agent is doing right now.
!Two real agent sessions side by side in Hermes IDE, Codex and Claude Code, each in its own worktree, with the sessions sidebar showing status, context use, cost and branch
Two real agent sessions side by side: Codex on orders-types after adding a RefundStatus type, and Claude Code on orders-api after a follow-up turn adding a log line, each in its own hermes/ worktree, with the sessions sidebar (status, context use, cost, branch) on the left.
Each session also shows its spend: the cost the agent reports, or an estimate from its transcript, clearly marked as estimated, with token totals. That tells you which session the money went to. It won't break out agent_summary, and since those calls skip the transcript, a transcript estimate can't see them. For that share, the OpenTelemetry query above is the tool.
You can download Hermes. It is source-available under BSL 1.1 and runs on macOS, Windows and Linux. The current release is 2.3.4. It is not affiliated with Nous Research or its Hermes Agent.
What to do with this today
1. Run the query above for a few working days on a current Claude Code build.
2. If agent_summary is a small slice, leave it alone. The label is useful.
3. If it is large, look at why first: how many background agents run at once, and on which model. A subagent on Haiku gets summarized by Haiku.
4. Add your numbers and version to #86879. A supported opt-out is what that issue asks for, and real measurements are what moves it.
5. If you reach for the preload anyway, read it first, check your terms, and verify it after every update.
---
If this was useful
The fix for this kind of surprise is the same every time: per-request telemetry, grouped by who issued the request, before you trust anyone's percentage, including the ones in this post. The LLM Observability Pocket Guide is the short read on picking the tracing and eval tools that give you that view, so a hidden 3-5 word call shows up as its own line instead of a mystery in the monthly total.
Tail sampling needs a trace-complete path, not just a clever policy
Tail sampling needs a trace-complete path, not just a clever policy
Head sampling decides early, when a collector has seen only the start of a trace. Tail sampling waits for spans to arrive and then evaluates policies against the completed trace. That extra context can keep errors and slow transactions while dropping routine traffic. The benefit depends on a routing assumption: every span for one trace must reach the same decision point.
The OpenTelemetry tail sampling processor groups spans by trace ID and keeps them for a decision window. If a load balancer sends spans from the same trace to different collector instances, each instance sees a fragment. One collector may see an error span and keep it. Another may see only routine spans and discard its part. The resulting backend view can be incomplete even when the policy was written correctly.
Use a trace-aware load-balancing path in front of tail sampling, or route all spans through one collector pool that shares the required state. Review the exporter and service architecture as part of the sampling policy. Autoscaling can change the routing layout. A deployment that worked with two replicas may stop making coherent decisions when a new replica joins or an old one is removed.
Policy order matters. A latency policy, an error policy and a probabilistic fallback can overlap. The collector needs enough memory and a suitable decision wait to hold spans for the slowest policy that applies. Memory pressure can drop spans before a decision, and an overloaded collector may turn a security-monitoring pipeline into another outage. Monitor incoming spans, pending traces, policy matches and dropped data, not only the final exported ratio.
Tenant boundaries need separate attention. Telemetry can contain account identifiers, URLs, query strings, email addresses, tokens embedded in errors, and internal hostnames. Tail sampling reduces volume; it does not make arbitrary telemetry safe to centralize. Restrict which attributes are allowed, redact sensitive fields before sampling, and separate tenants when policy or access rules require it. Sampling policies that use a service.criticality attribute should derive that attribute from a controlled resource pipeline, not from a client-supplied field.
The security value of tail sampling is strongest when the policy protects evidence of abnormal behavior. Keep traces with authentication failures, authorization denials, unexpected upstream errors and extreme latency, but test those policies with synthetic traces. A policy that samples every span containing the word error can be flooded by an application that writes user-controlled text into attributes. A policy that ignores errors from a critical service can hide the incident the monitoring team most needs.
Record the sampling decision and the reason without storing a second copy of sensitive payloads. Use bounded attribute allowlists and alerts for policy drift. Review the collector configuration when service topology changes, when a new region is added, or when the load balancer changes. Tail sampling is a data-selection control, so its correctness depends on the path that delivers the data.
References
- OpenTelemetry tail sampling processor documentation
- OpenTelemetry sampling concepts
October: Kubernetes memory semantics and Collector edge fixes
In October I had 10 PRs merged across four projects. The common thread was practical observability work: clearer Kubernetes memory conventions, Helm chart behavior that is easier to reason about, less flaky Collector tests, and maintainership for components that need active review.
Most of my week goes into one question: why does something that looks correctly configured not work in a real cluster? These PRs came from the same place.
> TL;DR
> - Fixed flaky Kubernetes inventory cache-sync tests.
> - Removed a Java 11+ startup failure in a DataPower monitoring extension.
> - Let Operator chart users override only the Target Allocator repository.
> - Added Kubernetes memory utilization conventions.
> - Clarified Kubernetes working set descriptions.
> - Added Pod-level memory limit and request conventions.
> - Joined Flink Metrics Receiver code ownership.
> - Fixed broad Flink operator metric matching.
> - Documented Operator chart certificate drift with Argo CD.
> - Restored Azure Data Explorer Exporter active ownership and beta status.
Kubernetes memory conventions became more precise
Pod memory limits and requests now have desired and current forms
The Kubernetes semantic conventions had Pod-level CPU limit and request metrics, but not the same shape for memory. I added k8s.pod.memory.limit.{desired,current} and k8s.pod.memory.request.{desired,current} so the model can represent values from spec.resources and status.resources.
This shows up in OpenTelemetry semantic conventions, which define standards for consistent telemetry. It closes issue #4133, and there is no release listed for this PR yet.
Once implemented downstream, dashboards and alerts can distinguish desired Pod memory settings from current Pod memory status.
open-telemetry/semantic-conventions PR 4158
Working set descriptions now say what Linux includes
The working set descriptions for k8s.pod.memory.working_set, k8s.node.memory.working_set, and container.memory.working_set were clarified. On Linux, working set excludes inactive file-backed memory, while active file-backed memory remains included.
The generated semantic convention documentation now also says that kubelet uses Pod working set when ranking Pods for eviction, and node working set when determining memory availability for the memory.available eviction signal. There is no release listed for this PR yet.
The next effect is clearer documentation for comparing usage, working set, and eviction behavior.
open-telemetry/semantic-conventions PR 4184
Memory utilization now uses working set as the numerator
I added four Kubernetes memory utilization metrics: Pod and container limit utilization, and Pod and container request utilization. These use memory working set as the numerator, not total memory usage.
The PR explains the reason: total usage includes reclaimable page cache, which can make a workload look close to its limit even when part of that memory is reclaimable. I validated the behavior against a kubelet Summary API using kind with Kubernetes v1.37.0. There is no release listed for this PR yet.
Once implemented downstream, memory utilization can be shown against limits and requests with the numerator semantics documented in the convention and migration guide.
open-telemetry/semantic-conventions PR 4219
Operator Helm chart configuration and docs got clearer
Private registry users can override only the Target Allocator repository
The OpenTelemetry Operator Helm chart did not let users override only manager.targetAllocatorImage.repository while keeping the chart’s default tag. I added explicit defaults for the Target Allocator repository and tag, plus validation that repository and tag are either both set or both empty.
This is for the OpenTelemetry Helm Charts project. It closes issue #1866, where 2 people asked for it, and the chart version was bumped to 0.125.0. There is no release listed for this PR yet.
The practical effect is simpler values for private registry or air-gapped installs that redirect the Target Allocator image. The upgrade guide also documents the configuration precedence change.
open-telemetry/opentelemetry-helm-charts PR 2456
Argo CD users get documentation for auto-generated webhook certs
When admissionWebhooks.autoGenerateCert is used, the Operator chart can generate a new CA on each render. With Argo CD, or other tools that use helm template, Helm cannot use lookup during rendering to reuse the existing Secret, so the webhook caBundle, certificate Secret, and running operator may become inconsistent.
I documented this in the README’s TLS certificate section, including the possible x509: certificate signed by unknown authority failure. I reproduced the behavior with Argo CD v3.5.5, chart 0.126.0, and Kubernetes v1.37.0. This shipped in opentelemetry-operator-0.127.1.
The change is documentation: Argo CD users have a clearer reason to use cert-manager or supply certificate material independently of render-time generation.
open-telemetry/opentelemetry-helm-charts PR 2473
Collector contrib tests, receivers, and ownership moved forward
Kubernetes inventory cache-sync tests are deterministic now
Two internal/k8sinventory/informer tests were flaky because they depended on timing around fake client cache sync. I changed the tests to block the initial Pod List call explicitly, so the failure paths are deterministic without changing production behavior.
This is internal test code in the OpenTelemetry Collector contrib repository. It closes issue #50433, which had 24 comments about failed tests on main. There is no release listed for this PR yet.
The effect is less noise from racey tests when contributors work on Kubernetes inventory code.
open-telemetry/opentelemetry-collector-contrib PR 51657
The Flink Metrics Receiver has another code owner
The Flink Metrics Receiver needed additional maintainers. I added myself as a code owner after reviewing receiver changes, running unit tests, and validating changes against a local Apache Flink 2.3.0 deployment in Docker.
This shows up in CODEOWNERS and generated README metadata in the OpenTelemetry Collector contrib repository. There is no release listed for this PR yet, and it does not change runtime behavior.
The practical change is maintenance coverage for reviews, triage, bug fixes, and receiver upkeep.
open-telemetry/opentelemetry-collector-contrib PR 52019
Flink operator metrics now match exact suffixes
The Flink Metrics Receiver used strings.Contains for operator-level metric names. That could also match longer names like numRecordsInPerSecond when looking for numRecordsIn.
I changed the four affected operator metric cases to exact suffix matching: numRecordsIn, numRecordsOut, numLateRecordsDropped, and currentOutputWatermark. Existing metric names, attributes, and configuration are preserved. There is no release listed for this PR yet.
The result is more precise matching for those receiver metrics.
open-telemetry/opentelemetry-collector-contrib PR 52406
Azure Data Explorer Exporter has active ownership again
The Azure Data Explorer Exporter had been marked unmaintained after its previous active code owner moved to emeritus. I added myself as an active code owner and restored the previous beta stability designation for traces, metrics, and logs.
This is visible in generated README, CODEOWNERS, component labels, status metadata, and the changelog. It closes issue #52282, and there is no release listed for this PR yet.
The practical effect is that the component again has an active maintainer path for review and triage.
open-telemetry/opentelemetry-collector-contrib PR 52500
A Machine Agent monitoring extension starts again on Java 11+
The DataPower extension no longer depends on JDK SAAJ
On Machine Agents running a Java 11+ JRE, the DataPower monitoring extension could fail to instantiate because the JDK no longer ships the SAAJ implementation removed by JEP 320. The extension only used SAAJ to build a fixed SOAP request envelope.
I removed the javax.xml.soap dependency and now build the envelope directly as a string, with XML attribute escaping for domain and status-class values. The PR bumps the extension version to 2.0.2, and there is no release link listed for this PR yet.
The practical change is that Machine Agent users on Java 8 through 21 have a startup path that does not depend on a removed JDK implementation.
Appdynamics/datapower-monitoring-extension PR 13
Still in review
- open-telemetry/opentelemetry-operator: Generate a list of collector components with specific operator support
- Appdynamics/process-monitoring-extension: Update dependencies to remediate reported vulnerabilities
- open-telemetry/opentelemetry-collector-contrib: update tested Flink versions and correct README configuration
- open-telemetry/opentelemetry-collector-contrib: Record JVM GC metrics for both JobManager and TaskManager
- open-telemetry/opentelemetry-collector-contrib: Add opt-in PVC-level volume metrics
- open-telemetry/opentelemetry-collector-contrib: Add Warnings section to probabilistic sampler README
- CiscoDevNet/appdynamics-charts: ENV variables fix in InfraViz
Sources
- Fix flaky Kubernetes inventory cache-sync tests
- Fix DataPower monitoring extension startup on Java 11+
- Allow overriding only the Target Allocator image repository
- Add Pod and container memory limit and request utilization metrics
- Clarify memory working set metric descriptions
- Add Pod-level memory limit and request metrics
- Add a Flink Metrics Receiver code owner
- Match Flink operator metrics on exact suffix
- Explain autoGenerateCert behavior with Argo CD
- opentelemetry-operator-0.127.1 release
- Restore Azure Data Explorer Exporter active ownership and beta stability
---
_Drafted with AI assistance from the PRs themselves and reviewed by me._
GitHub ActionsとDockerのビルドを1本のトレースにしてGrafana Cloudで見る
!
2026年10月27日に、GitHubホストランナーのUbuntu 24.04で次のバージョンを使って確認した内容です。送信するシグナルはトレースだけです。
Docker Engine 28.0.4
Buildx v0.29.1
BuildKit v0.33.1
OpenTelemetry Collector contrib v0.164.0
GitHubレシーバーのトレース機能は、このバージョンではalphaです。後述する親スパンのID計算もレシーバーの実装に依存します。
はじめに
先日書いたDockerのビルドを遅くしたコミットをGrafana Tempoのトレースで見つけるでは、手元のビルドをトレースにして、遅くなったコミットやDockerfileの手順を調べました。次のステップはCIです。GitHub Actionsの画面で「Dockerのビルドが遅い」とわかっても、そのステップの中で何に時間がかかったかを調べるには、ビルドのログや記録を別に開く必要があります。ワークフロー全体からDockerfileの手順まで、同じトレースの中でたどれるようにしてみました。
TL;DR
BuildKitからGrafana CloudのOTLPエンドポイントへトレースを送信できることを確認しました。CIのステップにつなぐには、OpenTelemetry CollectorのGitHubレシーバーがステップに付けるIDをビルドの前に計算し、TRACEPARENTでBuildxへ渡します。CI側のスパンを送るのは、完了した実行の情報をGitHub APIで取得し、Actionsの中で一時的に動かすCollectorです。その結果、ワークフローと2つのジョブが1本のトレースにまとまり、buildジョブのビルドステップの下にBuildKitのスパンがつながりました。
2つの送信経路
今回の構成では、ビルドのトレースとGitHub Actionsのトレースを別々に送ります。両方でトレースIDと親スパンIDを合わせると、Grafana Cloudで1本に表示できます。
ビルド側の送信元は、OpenTelemetryでトレースを送る機能を持つBuildKitです。Dockerfileの各手順のスパンに加えて、クライアントであるBuildxのスパンも、BuildKitへ転送してまとめて送ります。
一方、GitHub Actions側には、BuildKitのようにトレースを送る送信元がありません。ワークフローの実行の記録はWebhookのイベントやREST APIで取得できますが、トレースの形式にはなっていないためです。この記録をスパンに変換するのが、OpenTelemetry CollectorのGitHubレシーバーです。GitHubレシーバーは、GitHubから届くworkflow_runとworkflow_jobのWebhookのイベントを受け取り、ワークフロー、ジョブ、ステップのスパンを作ります。レシーバーはCollectorに組み込んで動かすコンポーネントなので、GitHub Actions側のスパンはCollectorを通し、エクスポーターでGrafana Cloudへ送ります。
今回は、CI側のイベントの取得から送信までをActionsの中で実行する構成を選びました1。ワークフローの完了後に別のワークフローを起動し、そのランナーで一時的にCollectorを動かします。補助スクリプトが完了した実行の情報をGitHub APIで取得し、Webhookと同じ形式のイベントに組み立てて、ランナー内のGitHubレシーバーへ渡す構成です。BuildKitからの送信だけを試す段階では、このCollectorは使いません。
図1: 矢印はデータの移動を表します。左側では計算したTRACEPARENTをBuildxへ渡します。右側は完了後の別ワークフローで、補助スクリプトがGitHub APIの情報をイベント形式へ変換し、OpenTelemetry Collectorへ渡します。Collector内のGitHubレシーバーでCIスパンを生成して送信します。
題材はBusyBox 1.37.0の小さなイメージです。ファイルをコピーしてRUNで別ファイルへ書き出し、ビルドしたイメージを実行して出力も確かめました。実証用のファイル一式を公開しています。
BuildKitからGrafana Cloudへ
まず、Actionsの親スパンを付けず、BuildKitからの送信だけを確認しました。Grafana Cloudで取得した接続情報を、リポジトリのActionsシークレットに次の名前で登録します。
シークレット
設定する値
GRAFANA_CLOUD_OTLP_ENDPOINT
/otlpまでを含むOTLPのベースURL
GRAFANA_CLOUD_OTLP_HEADERS
Grafana Cloudの接続画面から取得したAuthorization=...の認証ヘッダー
Actionsのステップでは、これらをOTEL_EXPORTER_OTLP_ENDPOINTとOTEL_EXPORTER_OTLP_HEADERSという環境変数に渡しています。scripts/setup-builder.shの内容は次のとおりです。
#!/usr/bin/env bash
set -euo pipefail
: "${OTEL_EXPORTER_OTLP_ENDPOINT:?Missing OTLP endpoint}"
: "${OTEL_EXPORTER_OTLP_HEADERS:?Missing OTLP authentication headers}"
docker buildx create --name ci-otel --driver docker-container \
--driver-opt image=moby/buildkit:v0.33.1 \
--driver-opt env.OTEL_TRACES_EXPORTER=otlp \
--driver-opt env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
--driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT%/}/v1/traces" \
--driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}" \
--bootstrap >/dev/null
docker-containerドライバーではBuildKitが専用のコンテナで動くので、BuildKitのOTLP設定は--driver-opt env.…でそのコンテナの環境変数として渡します。プロトコルはhttp/protobufで、OTLP/HTTPを使います。トレース専用のエンドポイントは、ベースURLの末尾に/v1/tracesを足したURLです。
このbuilderを選んでdocker buildx build --builder ci-otel --load …を実行すると、BuildxとBuildKitの両方のスパンがGrafana Cloudへ届きました。ベースラインの実行から取得したのは、build .をルートにした87スパンのトレースです。イメージのビルドと実行テストも成功しました。
図2: Grafana Cloudで取得したベースラインのトレース。BuildxとBuildKitの87スパンを確認できます2。
Actionsのステップを親にする
ここまでのトレースのルートはbuild .です。GitHub Actionsのビルドステップを親にするには、Buildxを実行するステップに環境変数TRACEPARENTを設定します。Buildxは環境変数からトレースコンテキストを読み込む処理でこの値を取り込み、build .を指定された親の子にします。TRACEPARENTに入れるのは、接続先のトレースIDと、親になるステップのスパンIDです。
ただし、ビルドの時点では親になるステップのスパンはまだありません。CI側のスパンは、ワークフローの完了後に作るからです。それでも親のIDを指定できるのは、GitHubレシーバーがトレースIDとスパンIDを乱数ではなく実行IDやジョブIDから決まった規則で計算するためです。ビルドの前に同じ規則で計算すれば、あとで届くステップのスパンと同じIDが得られます。
補助スクリプトscripts/ci-parent.pyは、今回の実行ID、再実行回数、ジョブIDから、レシーバーのID生成の実装と同じ規則でIDを計算します3。計算部分を抜粋すると次のとおりです。
trace_id = hashlib.sha256(f"{run}{attempt}t".encode()).hexdigest()[:32]
step_id = hashlib.sha256(f"{job_id}-Build Docker image-s".encode()).hexdigest()[16:32]
traceparent = f"00-{trace_id}-{step_id}-01"
TRACEPARENTの値は、W3C Trace Contextのtraceparentヘッダーと同じ00-トレースID-親スパンID-01の形式です。スクリプトは、この値をGITHUB_OUTPUTへ書き出します。
ワークフローでは、このスクリプトの出力をビルドステップだけに渡します。build.ymlから該当部分を抜粋すると、次のようになります。
- name: Calculate build step parent
id: build_context
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/ci-parent.py
- name: Build Docker image
env:
TRACEPARENT: ${{ steps.build_context.outputs.traceparent }}
OTEL_RESOURCE_ATTRIBUTES: vcs.ref.head.revision=${{ github.sha }},ci.run.id=${{ github.run_id }},ci.run.attempt=${{ github.run_attempt }},ci.repository=${{ github.repository }},experiment.mode=linked
run: docker buildx build --builder ci-otel --load --tag buildkit-otel-spike:ci .
ステップのスパンIDはジョブIDとステップ名から決まるので、ステップ名はスクリプトの計算に使ったBuild Docker imageと一致させます。また、ステップ名はジョブ内で一意にしておきます。同じジョブに同名のステップが2つあると、レシーバーが両方に同じスパンIDを付けてしまうからです。
OTEL_RESOURCE_ATTRIBUTESでは、検索に使う実行IDやコミットを付けています。この属性が付くのはBuildxのスパンだけで、BuildKitのスパンには付きません。
CI側のスパンの送信
ビルドステップが終わっても、ワークフロー全体はまだ実行中です。CI側のスパンは、workflow_runのcompletedで起動する送信用のワークフローが作ります。
送信用のワークフローは、完了通知に含まれる実行IDと再実行回数を環境変数で補助スクリプトへ渡します。
OBSERVED_RUN_ID: ${{ github.event.workflow_run.id }}
OBSERVED_RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }}
再実行回数を指定するのは、取得した時点で最新の再実行ではなく、通知された回のジョブを読むためです。補助スクリプトは、その回のジョブとステップの情報をGitHub APIで取得し、Webhookと同じ形式のイベントに組み立ててランナー内のCollectorへPOSTします。変換の処理はscripts/export-workflow.pyで確認できます。
collector.yamlの内容は次のとおりです4。
receivers:
github:
webhook:
endpoint: 0.0.0.0:19418
path: /events
health_path: /health
service_name: github-actions
scrapers:
scraper:
github_org: ymotongpoo
processors:
batch:
timeout: 1s
exporters:
otlp_http/cloud:
endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
headers:
Authorization: ${env:GRAFANA_AUTHORIZATION}
timeout: 10s
sending_queue:
enabled: false
retry_on_failure:
enabled: false
service:
telemetry:
logs:
encoding: json
pipelines:
traces:
receivers: [github]
processors: [batch]
exporters: [otlp_http/cloud]
エクスポーターのendpointはベースURLのままです。otlp_httpのエクスポーターが/v1/tracesを付けるので、BuildKitにトレース専用のURLを渡したのとは指定方法が違います。認証ヘッダーには、BuildKitと同じシークレットからAuthorization=の後ろの値を取り出して渡します。
このCollectorは送信のたびに起動して停止する実証用の設定で、送信キューと再試行を無効にしています5。
自分の環境で試すには、2つのシークレットを登録し、ビルド用と送信用のワークフローをデフォルトブランチに置きます。そのうえでBuildKit linked traceを手動で実行し、ビルドとExport completed workflow traceの両方の完了を待ちます。APIの呼び出しに使うのはcontents: readとactions: readの権限を付けたGITHUB_TOKENで、GitHub AppやPATは不要です。
1本につながった結果
ビルド側の実行とCI側のスパンの送信は、どちらも成功しました。ワークフロー全体とビルドの内部が1本のトレースにつながっています。
図3: 同じ実行のワークフロー、2つのジョブ、各ステップが1本のトレースになりました。Build Docker imageの下にbuild .が接続しており、ビルド内部は折りたたんでいます。
サービスごとの内訳は次のとおりでした。
service.name
スパン数
記録している処理
github-actions
21
ワークフロー、ジョブ、待ち時間、ステップ
buildx
33
ビルドクライアントの処理
buildkitd
54
ビルドデーモンとDockerfileの手順
合計
108
同一トレースIDのスパン
ビルド用のbuildと、その完了後に別のランナーで実行するverify-sourceの2ジョブが、同じルートにつながっています。CIの待ち時間やビルド内部の細部を省くと、親子関係は次のようになります。
BuildKit linked trace
├── build
│ └── Build Docker image
│ └── build .
│ └── …
│ └── [4/4] RUN cat message.txt > result.txt
└── verify-source
└── Verify source fixture
build .の親IDはBuild Docker imageのスパンIDと一致し、取得した108スパンはすべて同じワークフローのルートへつながっていました6。
GrafanaのExploreでTempoのデータソースを選び、実行した時間帯に絞って、次のTraceQLで検索します。ci.run.idは、OTEL_RESOURCE_ATTRIBUTESでBuildxのスパンに付けた属性です。
{ resource.ci.run.id = "40437818776" && name = "build ." }
前回の記事では、ルートのスパンだけを選ぶ条件（span:parentID = ""）でビルドの所要時間を集計しました。今回のビルドはbuild .にActionsの親があるので、この条件では選べません。手元のビルドとCIのビルドをまとめて選ぶ条件は、{ resource.service.name = "buildx" && name = "build ." }です。ベースラインとCIの両方のトレースが、この条件の検索結果に入ることを確認しました。
CI側のスパンは、送信用のワークフローが完了してから届きます。トレースを開くのは、両方のワークフローが完了してからにしてください7。
今回確認した範囲
BuildKitからGrafana Cloudへの送信と、2ジョブのワークフローへのビルド内部の接続までを確認しました。ただし、IDの生成規則はOpenTelemetryの共通の仕様ではなく、GitHubレシーバー v0.164.0の実装です。Collectorのバージョンやレシーバーのfeature gateを変えるときは、取得したトレースでbuild .が親のステップにつながっているかを確かめ直してください。
CI側の時刻はGitHub APIが秒単位で記録した値です。そのため、親子の接続が正しくても、BuildKitのスパンが親の時間範囲からわずかにはみ出すことがあります。今回の実行では、build .の終了が親のBuild Docker imageの終了を約0.84秒超えていました。
今回は信頼できるリポジトリでの手動実行だけで、matrix、reusable workflow、forkからのPR、実際の再実行、実行間でのキャッシュ再利用は検証していません8。
おわりに
GitHub Actionsのワークフローとステップから、Dockerfileのどの手順やレジストリへの通信に時間を使ったかまで、1本のトレースでたどれるようになりました。自分のCIへ取り込むときには、まずBuildKitの送信だけを確認し、必要に応じてCIの親スパンを追加する順で試せます。
脚注
GitHubレシーバーは、常時稼働するCollectorでGitHubからのWebhookを受け取る使い方を想定しています。組織のCIを継続して記録するなら、その構成のほうが向いています。常時稼働のCollectorなら、送信キューと再試行を有効にして、Grafana Cloudへの送信に失敗したスパンを再送することもできます。ただし、GitHubからWebhookを受けるために、インターネットから届くエンドポイントを用意して運用する必要があります。今回はその準備をせずに試すため、Actionsの中で完結させました。 ↩︎
図の上部に赤く表示されているHEAD HTTP HEADのエラーは、Docker Registryのトークン認証で最初に返る401の認証要求です。今回の実行では、トークン取得後の再試行が200となり、ビルドも成功しています。 ↩︎
ジョブの数値IDは、GitHub APIで今回の実行のジョブの一覧を取得し、名前がGITHUB_JOBと一致するジョブから取り出します。ジョブの表示名を変えたりmatrixを使ったりする場合は、別の方法でジョブを特定する必要があります。 ↩︎
v0.164.0のGitHubレシーバーは、トレースだけを使う場合でも、scrapersを1つ以上設定することを求めます。パイプラインはトレースだけなので、github_orgを使ったメトリクスの収集は動きません。 ↩︎
Collectorはbatchプロセッサーを通して送るので、レシーバーがHTTP 200を返しても、Grafana Cloudへの送信が成功したとは限りません。そこで補助スクリプトは、停止後のCollectorがJSON形式で出力したログを読み、levelがエラーを示していれば失敗として扱います。 ↩︎
検証時のスクリプトと検証記録を公開しています。取得した108スパンは、スパンIDがすべて一意で、ルートが1つ、親が見つからないスパンは0でした。この検査では、送信したすべてのスパンがGrafana Cloudに届いたかまでは判定できません。 ↩︎
Tempoの最初の検索結果では、ルートのサービス名が<root span not yet received>と表示されました。その後にトレースIDで取得すると、CI側を含む108スパンがそろっていました。 ↩︎
送信用のワークフローは、元の実行のコミットにある補助スクリプトを実行します。外部のPRが起点になる構成へ広げるときは、GitHubが説明するworkflow_runの権限と注意点を確認してください。 ↩︎
X-Ray SDK / Daemon → OpenTelemetryの主な違い
2026年2月25日、X-Ray SDK / Daemonがメンテナンスモードに入り、AWSはOpenTelemetry（OTel）への移行を勧めています。
docs.aws.amazon.com
自分が担当しているプロジェクトでもX-Ray SDKを使っているので、移行の準備を進めています。
この記事では、X-Ray SDK / DaemonからOpenTelemetryに移行すると書き方や設定がどう変わるのかを項目ごとにまとめます。
主な違い
1. 送信経路
X-Ray Daemon
xray -o -n ap-northeast-1
OpenTelemetry
collector.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch: {}
exporters:
awsxray: {}
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [awsxray]
X-Ray SDKではDaemonがX-Rayに送っていましたが、OTelではコレクターが送ります。コレクターはアプリからトレースを受け取って送り先へ中継するプロセスで、送り先もコレクターの設定で決まります。
コレクターには、AWSが配布しているADOT CollectorなどのOTelのコレクターかCloudWatchエージェントを使い、Daemonを動かしていた場所に置き換えます。送り先を変えたり、X-Ray以外の監視サービス（Datadog、Grafana Cloudなど）にも送ったりする場合も、exportersを変えるだけで、アプリのコードには手を入れずに済みます。
opentelemetry.io
2. 初期化
X-Ray SDK
func setupTracing() error {
return xray.Configure(xray.Config{
DaemonAddr: "127.0.0.1:2000",
})
}
OpenTelemetry
// xrayは、OTel用のX-Ray互換パッケージ（go.opentelemetry.io/contrib/propagators/aws/xray）
func setupTracing(ctx context.Context) (*sdktrace.TracerProvider, error) {
exp, err := otlptracegrpc.New(ctx) // localhost:4317のコレクターへ
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithIDGenerator(xray.NewIDGenerator()), // X-Ray形式のトレースID
sdktrace.WithResource(resource.NewSchemaless(
semconv.ServiceName("my-service"), // トレースマップに出る名前
)),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, // traceparent
xray.Propagator{}, // X-Amzn-Trace-Id
))
return tp, nil
}
X-Ray SDKではxray.Configureだけで済んでいましたが、OTelでは各コンポーネントを指定してTracerProviderを構成します。TracerProviderは、スパン（X-Rayのセグメントにあたる単位）を作る元になるものです。
コードは増えますが、エクスポーター（送り先）・ID生成・リソース・プロパゲーターがそれぞれ独立したコンポーネントになっていて、必要なものだけ差し替えられます。
opentelemetry.io
3. 終了処理
X-Ray SDK
// 特に何もしない
OpenTelemetry
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
tp, err := setupTracing(ctx)
if err != nil {
log.Fatalf("setup tracing: %v", err)
}
defer func() {
if err := tp.Shutdown(context.Background()); err != nil {
log.Printf("shutdown tracer provider: %v", err)
}
}()
終了処理は、プロセスを止める前に、まだ送っていないトレースを送り切るための処理です。
X-Ray SDKはセグメントが閉じるたびにDaemonへ送るので、終了処理はいりませんでした。一方OTelは、スパンをいったんメモリにためて、まとめて送ります。そのため、プロセス終了時にtp.Shutdownを呼んで残りを送り切る（フラッシュする）必要があり、呼ばずに終わるとためていた分が消えます。
4. 計装ライブラリ
X-Ray SDK
func instrument(mux http.Handler, cfg *aws.Config) (http.Handler, *http.Client) {
handler := xray.Handler(xray.NewFixedSegmentNamer("my-service"), mux)
client := xray.Client(&http.Client{})
awsv2.AWSV2Instrumentor(&cfg.APIOptions)
return handler, client
}
OpenTelemetry
func instrument(mux http.Handler, cfg *aws.Config) (http.Handler, *http.Client) {
handler := otelhttp.NewHandler(mux, "my-service")
client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
otelaws.AppendMiddlewares(&cfg.APIOptions)
return handler, client
}
X-Ray SDKではX-Ray用のラッパーを使っていましたが、OTelではOTelの計装ライブラリを使います。計装ライブラリは、HTTPやAWS SDKの呼び出しを自動でトレースするライブラリです。
OTelの計装ライブラリで計測した呼び出しも、X-Rayではこれまでと同じくサブセグメント（トレースの中で、呼び出し1つ分を表す単位）として表示されます。そのため、DBや外部APIの呼び出しごとにかかった時間を、移行後も確認できます。
5. 伝播ヘッダー
X-Ray SDK
X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1
OpenTelemetry（デフォルト）
traceparent: 00-5759e988bd862e3fe1be46a994272793-53995c3f42cd8ad8-01
伝播ヘッダーは、別のサービスを呼び出すときにトレースIDなどを渡すためのHTTPヘッダーで、これによって複数のサービスをまたいだ処理が1つのトレースにつながります。
X-Ray SDKではX-Amzn-Trace-Idでトレースを引き継いでいましたが、OTelのデフォルトはW3C Trace Contextという標準のtraceparentヘッダーになります。
ALBなどAWSのサービスはX-Amzn-Trace-Idを使うので、初期化で両方のプロパゲーター（伝播ヘッダーを読み書きするコンポーネント）を登録しないとトレースが途切れます。
w3.org
6. アノテーション（注釈）
X-Ray SDK
xray.AddAnnotation(ctx, "tenant_id", tenantID)
OpenTelemetry
span := trace.SpanFromContext(ctx)
span.SetAttributes(
attribute.String("tenant_id", tenantID),
// ここに挙げたキーだけがX-Rayのアノテーションになる
attribute.StringSlice("aws.xray.annotations", []string{"tenant_id"}),
)
アノテーションは、トレースに付けるキーと値の情報で、X-Rayコンソールでトレースを絞り込む条件に使えます。
X-Ray SDKではxray.AddAnnotationでアノテーションを付けていましたが、OTelではスパンの属性として付けます。
ただし、属性はそのままだとX-Rayでは検索に使えない情報（メタデータ）になります。annotation.tenant_id = "..."のような検索を使っている場合は、aws.xray.annotationsにキーを入れておく必要があります。コード側を変えたくなければ、コレクターのawsxrayエクスポーターのindexed_attributesでキーを指定する方法もあります。
docs.aws.amazon.com
github.com/open-telemetry/opentelemetry-collector…
7. サンプリング
X-Ray SDK
// X-Rayコンソールのサンプリングルールが自動で使われる
xray.Configure(xray.Config{
DaemonAddr: "127.0.0.1:2000",
})
OpenTelemetry
// xraysamplerは、github.com/aws-observability/aws-otel-go/samplers/aws/xray
remote, err := xraysampler.NewRemoteSampler(ctx, "my-service", "ecs") // 127.0.0.1:2000からルールを取得
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.ParentBased(remote)),
// ...
)
collector.yaml
extensions:
awsproxy: {}
service:
extensions: [awsproxy]
X-Ray SDKはX-Rayコンソールのサンプリングルールを使っていましたが、OTelのデフォルトはParentBased(AlwaysOn)です。呼び出し元が決めていなければ全件送るので、X-Rayのデフォルトルール（毎秒1件と、超えた分の5%）より送る量が増えます。
これまでのルールを使い続けるには、X-Rayからルールを取得して判定するリモートサンプラーを設定します。ルールの取得はDaemonが中継していたので、コレクターではawsproxy拡張を設定します。
8. Lambdaでの送信
X-Ray SDK
func main() {
// Lambdaが作るセグメントの下に、サブセグメントが自動で付く
lambda.Start(handler)
}
OpenTelemetry
// xrayudpは、github.com/aws-observability/aws-otel-go/exporters/xrayudp
// xrayconfigは、go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda/xrayconfig
func main() {
ctx := context.Background()
exp, err := xrayudp.NewSpanExporter(ctx) // Lambda組み込みのX-Rayエンドポイントへ
if err != nil {
log.Fatalf("create exporter: %v", err)
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithSpanProcessor(sdktrace.NewSimpleSpanProcessor(exp)),
sdktrace.WithIDGenerator(xray.NewIDGenerator()),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(xray.Propagator{})
// WithRecommendedOptionsにはotellambda.WithFlusher(tp)が含まれる
lambda.Start(otellambda.InstrumentHandler(handler, xrayconfig.WithRecommendedOptions(tp)...))
}
Lambdaでは、ECSのようにアプリの隣のコンテナ（サイドカー）でコレクターを動かせないので、送信経路が変わります。xrayudpエクスポーターを使うと、X-Ray SDKと同じくLambda組み込みのX-Rayエンドポイントに直接送れます。
また、Lambdaは呼び出しが終わると、次の呼び出しまで実行環境を一時停止します。プロセスは終了しないので、終了時に呼ぶShutdownだけでは、スパンが送られないまま残ります。otellambda.WithFlusherを指定すると、呼び出しごとにForceFlushが呼ばれ、たまっていたスパンが送られます。
docs.aws.amazon.com
9. ログに出すトレースID
X-Ray SDK
// 1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx の形式
slog.InfoContext(ctx, "order created", "trace_id", xray.TraceID(ctx))
OpenTelemetry
// W3C形式の32桁をX-Ray形式に直す（先頭8桁はUnix時刻）
func xrayTraceID(ctx context.Context) string {
id := trace.SpanContextFromContext(ctx).TraceID().String()
return "1-" + id[:8] + "-" + id[8:]
}
slog.InfoContext(ctx, "order created", "trace_id", xrayTraceID(ctx))
X-Ray SDKではトレースIDがX-Ray形式で取れていましたが、OTelのトレースIDはW3C形式の32桁の16進数です。
X-Rayのコンソールでは1-58406520-a006649127e371903a2de979のような形式で表示・検索するので、ログのIDをそのまま貼り付けても見つかりません。W3C形式の先頭8桁のあとにハイフンを入れ、頭に1-を付ければ変換できます。
ログに出したトレースIDをX-Rayで検索できれば、遅かったリクエストのログからトレースを開いたり、その逆をたどったりできます。
docs.aws.amazon.com
まとめ
X-Ray SDK / Daemonは2026年2月25日からメンテナンスモード（終了日は未定）で、X-Rayのサービスとコンソールはそのまま使える
移行後もX-Rayで見られ、送り先はあとからコレクターの設定で追加・変更できる
OTelにすると
X-Rayに送りながら、ほかの監視サービスも設定だけで試せる
計装がAWS専用でなくなり、メトリクスやログも同じサービス名でそろえられる
トレースマップ・サブセグメント・ログのトレースIDによる突き合わせは、移行後もこれまでどおり使える
主な違いは以下のとおり
観点
X-Ray SDK / Daemon
OpenTelemetry
送信経路
Daemon
コレクター
初期化
xray.Configureだけ
各コンポーネントを指定してTracerProviderを構成
終了処理
不要
tp.Shutdownで未送信分をフラッシュ
計装ライブラリ
X-Ray用ラッパー
OTelの計装ライブラリ
伝播ヘッダー
X-Amzn-Trace-Id
標準のtraceparent（併用可）
アノテーション
xray.AddAnnotation
属性＋aws.xray.annotationsにキーを指定
サンプリング
X-Rayのサンプリングルール
デフォルトは全件、ルールを使うならリモートサンプラー
Lambdaでの送信
SDKが組み込みのエンドポイントへ送る
xrayudpエクスポーターで送り、呼び出しごとにフラッシュ
my second contribution to @opentelemetry 🥳
opentelemetry-collector-contrib#51735
11 views
If you are using a Datadog, Langfuse, Langchain, or other OTEL target, you do nothing. Just query it. Build alerts, dashboards, and datasets; whatever you normally do with your OTEL collector. A forward OTEL collector deployment is still useful for filtering but the Datadog Agent, for example, already has buffering and filtering built into it.
If you want to deploy your own collector and write your own handler (e.g. put it into a database), you can generate the stubs for the OTEL payloads and convert to domain records.
1. Get the protos here: open-telemetry/opentelemetry-proto
2. Generate the type/class stubs and endpoints for your platform.
3. I strongly recommend pre-filtering with the open source `opentelemetry-collector` before your custom endpoint handles the inbound telemetry payload. The collector is optimized for performing the filtering and I put it upstream of my custom ingest endpoint on a very small VM just to pre-filter and perform auth checks[0][1] on the inbound channel.
4. Do what you need to with the inbound payloads. Examples below:
- Ingest endpoint (Codex/Claude -> OTEL collector -> API): github.com/zeeq-ai/zeeq-app/blob/main/src/backend…
- Codex telemetry adapter: github.com/zeeq-ai/zeeq-app/blob/main/src/backend…
- CC telemetry adapter: github.com/zeeq-ai/zeeq-app/blob/main/src/backend…
I run my backend on Postgres with partitioned tables. You can forward to Elastic or other OLAP-type database targets for querying. You can see examples of what I do with the session telemetry here: zeeq.ai
[0] github.com/zeeq-ai/zeeq-app/blob/main/build/otel-…
[1] github.com/open-telemetry/opentelemetry-collector…
Claude Code Already Emits OpenTelemetry: Who Spends What, on Which Model, in Grafana
- Book: LLM Observability Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
Your team's Claude Code spend lands as one number on an invoice. Then someone asks the obvious follow-ups. Which team drove last month's jump? Was it the expensive model or the cheap one? Are subagents quietly burning half the budget? Nobody can answer, because the data that would answer it was never switched on.
Claude Code ships an OpenTelemetry exporter. Export is off until you set one environment variable, CLAUDE_CODE_ENABLE_TELEMETRY=1, per the monitoring docs; a few more say where the data goes. Once it is on, every API request bumps a cost counter and a token counter, each tagged with the model, the user, the session and the part of Claude Code that made the call.
This walkthrough runs the whole pipeline on your laptop: Claude Code into an OpenTelemetry Collector, scraped by Prometheus, drawn by Grafana, all in one Docker Compose file. Then you build one panel per question from the opening.
What Claude Code exports (and what it keeps back)
Two signals matter for a spend dashboard. Metrics go out through the metrics exporter; events go out through the logs exporter. Per the docs, the metrics are:
- claude_code.cost.usage (USD), incremented after each API request, with model, query_source (main, subagent or auxiliary), effort, and attribution for the active skill, plugin or subagent.
- claude_code.token.usage (tokens), split by type: input, output, cacheRead, cacheCreation.
- claude_code.session.count, claude_code.lines_of_code.count, claude_code.commit.count, claude_code.pull_request.count, claude_code.active_time.total.
- claude_code.code_edit_tool.decision, which counts accept and reject decisions on Edit, Write and NotebookEdit, with the file's language.
The events include user_prompt, api_request (with cost_usd, duration_ms and token counts per call), api_error, tool_result and tool_decision.
Every metric and event carries session.id and user.id, a random installation identifier. When you sign in with a Claude account, you also get organization.id, user.account_uuid and user.email; the docs say the email goes only to the endpoint you configure, never to Anthropic.
The defaults lean private. Prompt text is not collected, only its length, unless you set OTEL_LOG_USER_PROMPTS=1. Bash commands and MCP tool arguments stay out unless you set OTEL_LOG_TOOL_DETAILS=1. The docs also state that raw file contents and code snippets are not part of metrics or events. Leave all of those flags off for a cost dashboard. You don't need them.
The stack: four small files
Make a directory and put four files in it. The Collector receives OTLP from Claude Code and re-exposes metrics in Prometheus format on port 8889. Prometheus scrapes that. Grafana reads Prometheus.
!A laptop terminal sending a stream of small coins and token tiles through a funnel into a database cylinder, which feeds three dashboard gauges
Claude Code pushes OTLP to the Collector; Prometheus scrapes it; Grafana draws it.
docker-compose.yml, with image tags pinned to the current releases of the Collector, Prometheus and Grafana:
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:0.162.0
command: ["--config=/etc/otelcol/config.yaml"]
volumes:
- ./otel-collector.yaml:/etc/otelcol/config.yaml:ro
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "8889:8889" # Prometheus scrape endpoint
prometheus:
image: prom/prometheus:v3.15.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
ports:
- "9090:9090"
grafana:
image: grafana/grafana:13.2.4
environment:
GF_SECURITY_ADMIN_PASSWORD: change-me
volumes:
- ./grafana:/etc/grafana/provisioning/datasources:ro
ports:
- "3000:3000"
otel-collector.yaml. Metrics go to the Prometheus exporter. Events go to the debug exporter so you can watch them in the container logs without running a log store yet:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch: {}
exporters:
prometheus:
endpoint: 0.0.0.0:8889
debug:
verbosity: detailed
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug]
prometheus.yml:
global:
scrape_interval: 15s
scrape_configs:
- job_name: otel-collector
static_configs:
- targets: ["otel-collector:8889"]
grafana/prometheus.yaml, a provisioned data source so you skip the click-through:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
Then docker compose up -d. Grafana is on port 3000 (user admin, the password from the Compose file).
Point Claude Code at it
In the shell where you start Claude Code:
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative
export OTEL_METRIC_EXPORT_INTERVAL=10000
export OTEL_RESOURCE_ATTRIBUTES="team.id=platform"
claude
Three of those lines deserve a word.
The temporality line. Claude Code's default is delta, and the docs say to switch to cumulative if your backend expects it. Prometheus counters are cumulative, and the Collector's Prometheus exporter README doesn't document what it does with delta sums, so asking for cumulative at the source takes the question off the table.
The export interval. The default is 60,000 ms. The docs suggest a shorter interval while you debug; 10 seconds means your first numbers show up fast. Put it back to the default once things work.
The resource attribute. OTEL_RESOURCE_ATTRIBUTES keys land on every metric data point as labels by default, so team.id=platform gives you a team dimension for free. Values can't contain spaces.
Run a prompt or two, then check what the Collector exposes:
curl -s localhost:8889/metrics | grep '^claude_code'
Read the metric names off that output rather than guessing. The exporter swaps dots for underscores and appends the unit and _total to counters, so the cost counter shows up as claude_code_cost_usage_USD_total and tokens as claude_code_token_usage_tokens_total. Those are the exact names the ROI measurement guide in Anthropic's GitHub organization queries. Labels follow the same rule: user.email becomes user_email, team.id becomes team_id. If nothing shows up, the docs recommend claude --debug-file <path> and a search for [3P telemetry] errors.
To roll this out to a team instead of one laptop, put the same variables in the env block of Claude Code's managed settings file. The docs note that a repository's .claude/settings.json cannot turn telemetry on or redirect it, so a stray repo config can't siphon your data somewhere else.
The three panels
Create a dashboard. For each panel, pick the Bar gauge visualization, set the query type to Instant, and use Grafana's $__range variable, which equals the dashboard's time window. Change the time picker to 30 days and every panel follows.
Who spends. Estimated USD per person over the selected window:
sum by (user_email) (
increase(claude_code_cost_usage_USD_total[$__range])
)
If your developers authenticate with API keys instead of OAuth, there is no email to group by (more on that below). Swap user_email for team_id and the panel becomes spend per team.
Which model. Same counter, different label:
sum by (model) (
increase(claude_code_cost_usage_USD_total[$__range])
)
This is the panel that starts conversations. When one model dominates, the next question is whether the work being done needs it.
What kind of work. query_source splits the main conversation from subagents and auxiliary requests:
sum by (query_source) (
increase(claude_code_cost_usage_USD_total[$__range])
)
For a finer cut, group by agent_name or skill_name instead. The docs redact user-defined agent names to custom and third-party plugin names to third-party unless OTEL_LOG_TOOL_DETAILS=1 is set, so expect those buckets. On the cost and token counters the flag needs Claude Code v2.1.273 or later; earlier versions stay redacted even with it set.
One more query worth a Stat panel: the share of input-side tokens served from the prompt cache. The docs point out that the input type excludes cache reads and writes, so the denominator has to add all three:
sum(rate(
claude_code_token_usage_tokens_total{type="cacheRead"}[1h]
))
/
sum(rate(
claude_code_token_usage_tokens_total{
type=~"input|cacheRead|cacheCreation"
}[1h]
))
If that ratio falls off a cliff, something changed what gets cached, and the token panel tells you when it started.
Read the numbers with these caveats
It's an estimate. The docs say it directly: cost metrics are approximations, and official billing data comes from your API provider (the Claude Console, Amazon Bedrock or Google Cloud). If you're on a seat-based plan, treat the USD figure as relative consumption, not as the invoice.
Upgrade before you trust a gateway setup. Before Claude Code v2.1.214, streams that reported usage across multiple frames through a proxy behind ANTHROPIC_BASE_URL inflated the cost and token counters by roughly one extra request per extra frame, per the docs.
API keys and cloud providers carry no identity. With a direct API key, Bedrock, Google Cloud or Microsoft Foundry, only user.id (a random installation identifier) and session.id are populated. Attach identity yourself through OTEL_RESOURCE_ATTRIBUTES, set per user in managed settings or a launch wrapper.
Session IDs are a cardinality cost. session.id is on every series by default, which helps drill-downs and bloats long-term storage. OTEL_METRICS_INCLUDE_SESSION_ID=false drops it.
Short sessions can under-count. Each session starts new series, and increase() works from the samples Prometheus actually scraped, so increments made before a series' first scrape can go missing (explained on the prometheus-users list). A short export interval narrows the gap. Prometheus also has a created-timestamp-zero-ingestion feature flag that may help here.
Desktop sessions use a different service name. Local sessions started from the Code tab in the Claude Desktop app report service.name as claude-code-desktop, not claude-code. Add both if you filter on it.
What this data can't tell you
The panels count spend, tokens and accepted edits. They don't tell you whether the work was any good. An accepted edit is a decision="accept" data point whether the change survives review or gets reverted the next morning. Lines added says nothing about lines that should have been deleted. With the privacy defaults on, you also can't see what was asked. That is the right default, and it is also why "why was this session expensive" has no answer in the metrics.
!A large magnifying glass held over a long receipt tape, sharp and readable, while a page of code beside it fades into fog
Telemetry makes the receipt sharp. The quality of the work stays out of frame.
Claude Code has beta tracing behind CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1, with claude_code.interaction spans that contain LLM request and tool spans. That gives you structure and latency. It still doesn't give you a verdict on output quality.
All of this measures the tool your developers use. Your product's LLM calls are a separate system. If you ship LLM features to customers, those calls need their own tracing, usually on the OpenTelemetry GenAI semantic conventions, and their own evals. Watch one mismatch if you put both on one dashboard: the docs note Claude Code doesn't set gen_ai.usage.* attributes, and its input_tokens excludes cache tokens, while the conventions count them. Add input, cache read and cache creation to compare like with like.
---
If this was useful
The Claude Code dashboard is the easy half, because the tool already emits the data. The harder half is the LLM calls inside your own product, where you choose the tracing backend, the eval approach and the trade-offs between them. That choice is what the LLM Observability Pocket Guide is about: picking tracing and eval tools that fit your team instead of the loudest vendor.
1 reaction
Also good to see how teams are already using this in their production, and this stability even pushes it further to easier adoption for others. Huge congrats to all the maintainers and contributors for making this happen 🙌 opentelemetry-collector-contrib#51236
13 views
Mentions
Observability for AI Agents: Tracing, Evaluation and Cost Control in Production How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices. 1. Why Is Observability Different for AI Agents? Traditional backend observability is built around relatively predictable execution paths. A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred. AI agents introduce a different level of complexity. An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input. Consider a relatively simple task: A user asks an AI assistant to analyze a merge request and identify potential problems. Behind the scenes, the application might: 1. Receive the user's request. 2. Invoke a coordinator agent. 3. Call an LLM to determine the next action. 4. Retrieve the merge request from GitLab. 5. Read the changed files. 6. Delegate security analysis to another agent. 7. Execute additional tool calls. 8. Evaluate the collected results. 9. Generate a final report. The user sees one response. The application has actually executed a multi-step workflow involving several components. Now imagine the response takes 45 seconds instead of the expected 8 seconds. Where should we investigate? - Did the model take too long to respond? - Did a tool call time out? - Was the same operation executed repeatedly? - Did the agent generate unnecessary steps? - Did a subagent introduce additional latency? - Was too much context sent to the model? - Did a retry increase the overall execution time and cost? Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why. AI observability is about understanding the entire execution path, not just measuring the final API response. This is where OpenTelemetry becomes particularly useful. 2. What Is OpenTelemetry? <escape>**</escape><Entity value="OpenTelemetry (OTel)" category="software" disambig="Open-source observability framework"/> is an open-source observability framework for collecting, processing, and exporting telemetry data. It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms. Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported. OpenTelemetry is built around three primary signals: | Signal | What it tells us | AI agent example | |---|---|---| | Traces | How an operation was executed | Model call → tool → subagent | | Metrics | How the system performs over time | Token usage, latency, error rate | | Logs | What happened at a specific point | Tool timeout or agent exception | These signals complement each other. A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation. OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector. An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal. References: OpenTelemetry Documentation · OpenTelemetry JavaScript 3. Understanding Distributed Tracing Before discussing AI-specific instrumentation, we need to understand traces and spans. A trace represents the complete execution path of an operation. A span represents one individual unit of work within that execution. For example: Trace: POST /api/agent | +-- HTTP Request | +-- Coordinator Agent | +-- LLM Generation | +-- Tool: GitLab - Get Merge Request | +-- Tool: GitLab - Get Changed Files | +-- Security Agent | | | +-- LLM Generation | | | +-- Tool: Security Analysis | +-- Testing Agent | | | +-- LLM Generation | | | +-- Tool: Test Coverage | +-- Final Response Generation Each span has information such as: - Trace ID - Span ID - Parent span ID - Start and end timestamps - Duration - Attributes - Events - Status and exception information This creates a hierarchical representation of the complete execution. The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them. For AI agents, this distinction is important because one user request can generate many independent operations. Why is context propagation important? In distributed systems, a trace often crosses multiple services. Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service. Without context propagation, every service may generate its own independent trace. We would see several unrelated operations instead of one complete workflow. OpenTelemetry uses W3C Trace Context as its standard propagation format. The traceparent header carries trace and parent-span information between instrumented services. The same principle applies to agent-based architectures. When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution. Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction. The important point is that protocol-level interoperability and observability context propagation are separate concerns. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace. References: OpenTelemetry Traces · Context Propagation · W3C Trace Context 4. OpenTelemetry and GenAI Semantic Conventions One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers. For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes. This makes it difficult to compare different implementations using the same monitoring platform. OpenTelemetry addresses this through Semantic Conventions. Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently. The GenAI semantic conventions extend this approach to generative AI applications. Some important attributes include: | Attribute | Purpose | |---|---| | gen_ai.provider.name | Identifies the model provider | | gen_ai.request.model | Identifies the requested model | | gen_ai.response.model | Identifies the actual response model | | gen_ai.operation.name | Describes the operation | | gen_ai.usage.input_tokens | Input token usage | | gen_ai.usage.output_tokens | Output token usage | | gen_ai.conversation.id | Associates operations with a conversation | | gen_ai.agent.name | Identifies the agent | | gen_ai.tool.name | Identifies the executed tool | For example, an LLM invocation could be represented by: { "gen_ai.operation.name": "chat", "gen_ai.provider.name": "openai", "gen_ai.request.model": "example-model", "gen_ai.usage.input_tokens": 1200, "gen_ai.usage.output_tokens": 350 } This is a simplified span attribute example. The benefit is that observability platforms do not need to understand a different custom schema for every application. Agent-specific conventions The GenAI semantic conventions also define operations for agents and workflows. Some examples include: - invoke_agent - invoke_workflow - execute_tool - plan - retrieval This is particularly valuable for multi-agent architectures. We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents. For example: invoke_workflow merge-request-review | +-- invoke_agent coordinator | | | +-- chat model | +-- execute_tool fetch_merge_request | +-- invoke_agent security-agent | | | +-- chat model | +-- execute_tool analyze_vulnerabilities | +-- invoke_agent testing-agent | | | +-- chat model | +-- execute_tool analyze_test_coverage | +-- Final Aggregation This makes the execution model much easier to understand. An important maturity consideration: As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract. The current convention documents include: - Semantic Conventions for GenAI Spans - Semantic Conventions for Agent Spans - Semantic Conventions for GenAI Metrics 5. Integrating OpenTelemetry with Vercel AI SDK Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration. The AI SDK already supports OpenTelemetry-based instrumentation. It can generate spans for: - Text generation - Streaming generation - Individual model calls - Agent execution - Tool execution - Embedding operations - Reranking This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility. Step 1: Install the packages For an AI SDK v7 application: pnpm add @ai-sdk/otel @vercel/otel The OpenTelemetry integration is provided through @ai-sdk/otel. This is an important version-specific detail. In AI SDK v7, OpenTelemetry integration moved out of the core ai package. Older examples using the v6 experimental_telemetry configuration should not be copied directly into a v7 application. Step 2: Register telemetry in Next.js Create an instrumentation.ts file in the project root. import { registerOTel } from '@vercel/otel'; import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; export function register() { registerOTel({ serviceName: 'ai-agent-platform', }); registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); } There are two responsibilities here: - registerOTel initializes the OpenTelemetry infrastructure for the Next.js application. - registerTelemetry registers the AI SDK-specific integration that creates GenAI spans. I have disabled input and output capture in this example intentionally. AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision. We will return to this in the security section. For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup. Step 3: Instrument AI SDK calls Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration. We can also provide a functionId to identify the business operation or agent. import { generateText } from 'ai'; const result = await generateText({ model: myModel, instructions: ` You are a financial analysis assistant. Analyze the supplied information carefully. `, prompt: 'Analyze the latest portfolio performance.', telemetry: { functionId: 'portfolio-analysis-agent', }, }); The functionId helps distinguish operations in traces. For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as: invoke_agent portfolio-analysis-agent Step 4: Understand the generated spans The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions. For generateText and streamText, the integration records three main types of spans: 1. Agent invocation span The root span represents the complete generation operation, including its execution steps and tool calls. Example: invoke_agent portfolio-analysis-agent 2. Step and model-call spans Each model interaction is represented by step and provider-call spans. These allow us to understand how many model calls occurred during an operation and how long each one took. 3. Tool execution spans Each client-side tool execution receives its own span. For example: invoke_agent portfolio-analysis-agent | +-- step 0 | | | +-- chat example-model | +-- execute_tool get_portfolio | +-- step 1 | | | +-- chat example-model | +-- execute_tool get_market_data | +-- step 2 | +-- chat example-model The resulting trace provides a much more complete representation of an agent workflow than a single model request log. The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available. Step 5: Add additional application context Sometimes the standard GenAI attributes are not enough. For example, we may want to know which business workflow initiated an operation or which application component owns it. The AI SDK provides enrichSpan for adding custom attributes: import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, enrichSpan: ({ spanType, operationId }) => ({ 'app.component': 'portfolio-assistant', 'app.operation_id': operationId, 'app.span_type': spanType, }), }), ); The exact context available to enrichSpan depends on the span and the SDK's integration API. The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry. References: AI SDK Telemetry · Next.js OpenTelemetry · AI SDK v7 Migration Guide 6. Instrumenting Your Own Agent Runtime Built-in instrumentation is useful, but it does not automatically understand every application-level decision. For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval. These are application-level operations. We should instrument them explicitly. OpenTelemetry provides a tracing API for creating custom spans. Consider a custom agent orchestration function: import { trace, SpanStatusCode, } from '@opentelemetry/api'; const tracer = trace.getTracer('agent-orchestrator'); export async function runReviewWorkflow() { return tracer.startActiveSpan( 'workflow.merge-request-review', async (span) => { span.setAttribute( 'workflow.name', 'merge-request-review', ); span.setAttribute( 'workflow.version', '1.0.0', ); try { const result = await executeReview(); span.setStatus({ code: SpanStatusCode.OK, }); return result; } catch (cause) { const error = cause instanceof Error ? cause : new Error(String(cause)); span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR, }); throw error; } finally { span.end(); } }, ); } Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated. We now have two levels of observability: - Framework-level instrumentation describes model and tool execution. - Application-level instrumentation describes business workflows and orchestration decisions. This distinction is important. We should not expect a model instrumentation library to understand the complete business meaning of an application request. Instrumenting remote agent calls For A2A-based systems, distributed tracing becomes even more important. Imagine a coordinator running in one service and a security agent running in another service. The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated. Conceptually: Trace: merge-request-review | +-- Coordinator Agent | | | +-- Model Call | | | +-- A2A: Delegate to Security Agent | | | +-- Security Agent | | | +-- Model Call | | | +-- Tool Execution | +-- Final Aggregation The HTTP request must carry the appropriate trace context, typically through W3C traceparent and tracestate headers. For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context. This is how we avoid fragmented traces across independently deployed agents. 7. What Should We Measure? Collecting telemetry is not enough. We need to define which metrics are useful for understanding the health and quality of an agent platform. I would divide AI observability metrics into five categories. 7.1. Latency Metrics These describe how long operations take. | Metric | Why it matters | |---|---| | End-to-end workflow duration | Measures total user-facing execution time | | Model request duration | Identifies slow inference calls | | Time to first token | Measures perceived response latency | | Tool execution duration | Identifies slow external operations | | Agent invocation duration | Measures the execution time of one agent | | Queue wait duration | Identifies scheduling bottlenecks | For streaming applications, time to first token is particularly useful. A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds. 7.2. Reliability Metrics These help us understand whether workflows complete successfully. Examples: - Agent execution success rate - Tool execution failure rate - Model request failure rate - Timeout rate - Retry rate - Task cancellation rate - Human intervention rate - Incomplete workflow rate A high retry rate can be an early indicator of a problematic integration or an unstable external dependency. We should also distinguish between a successful model response and a successful business workflow. An LLM can return a perfectly valid response while the overall task fails to achieve its objective. 7.3. Token and Model Usage Token usage should be tracked at multiple levels. | Metric | Description | |---|---| | Input tokens | Tokens sent to the model | | Output tokens | Tokens generated by the model | | Cached input tokens | Tokens served from provider cache | | Reasoning tokens | Model-reported reasoning token usage | | Tokens per workflow | Total usage across an entire task | | Tokens per agent | Usage attributed to an individual agent | For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response. 7.4. Agent and Tool Behavior These metrics describe how efficiently agents perform their work. Examples: - Number of tool calls per task - Number of model calls per agent invocation - Number of agent handoffs - Repeated tool-call frequency - Tool-selection success - Average steps per workflow - Average number of clarification requests An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem. The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs. 7.5. Quality Metrics This is where AI observability starts to overlap with AI evaluation. Examples: - Answer correctness - Groundedness - Retrieval relevance - Tool-selection accuracy - Task completion - Policy adherence - Human acceptance rate These metrics cannot always be derived from execution telemetry alone. We need an evaluation strategy to measure the actual quality of the result. 8. Cost Control: Understanding the Real Cost of an Agent One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow. An agent-based application may consume resources across several layers: - LLM inference - Embedding generation - Reranking - External API calls - Database queries - Vector database operations - Container or sandbox execution - Remote agent invocations - Infrastructure and networking The total cost should reflect the entire execution path. A useful starting point is to define a model cost formula: [ C_{\text{model}} = \sum_{i=1}^{n} \left( T_{\text{input},i}P_{\text{input},i} + T_{\text{output},i}P_{\text{output},i} \right) ] Where: - (T) represents token usage. - (P) represents the applicable price per token. - (i) represents an individual model operation. The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities. We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets. The complete workflow cost can then be represented as: [ C_{\text{workflow}} = C_{\text{model}} + C_{\text{tools}} + C_{\text{compute}} + C_{\text{storage}} ] This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available. Why is cost per successful task more useful? Imagine two implementations: | Metric | Agent A | Agent B | |---|---:|---:| | Average cost per attempt | $0.04 | $0.07 | | Successful completion rate | 90% | 98% | | Cost per successful task | ~$0.044 | ~$0.071 | Agent A is cheaper per attempt, but Agent B completes more tasks successfully. Neither is automatically better. The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts. A useful metric is: [ \text{Cost per successful task} = \frac{\text{Total execution cost}} {\text{Number of successful tasks}} ] The numerator should include failed attempts that consumed resources, rather than counting only successful runs. Model selection and routing Observability data can also help us make better model-routing decisions. For example: - Use a smaller model for classification or simple extraction. - Use a more capable model for complex analysis. - Avoid unnecessary model calls when a deterministic operation is sufficient. - Reuse cached context when appropriate. - Set explicit iteration and tool-call limits. - Stop an agent when a task exceeds its budget. The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request. Token metrics in OpenTelemetry The current GenAI token metric conventions include proposed counters such as: - gen_ai.client.inference.usage.input_tokens - gen_ai.client.inference.usage.output_tokens - gen_ai.client.inference.usage.cache_read.input_tokens - gen_ai.client.inference.usage.reasoning.output_tokens These conventions are still in Development. There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics. A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations. For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices. This makes historical cost calculations more reliable when provider pricing changes. References: GenAI Token Metrics · GenAI Metrics 9. Observability Is Not Evaluation This distinction is worth emphasizing. Observability tells us what happened during execution. Evaluation tells us whether the result was good enough. A trace can show that an agent: - Called the correct model. - Used three tools. - Retrieved five documents. - Completed within four seconds. - Returned a response without an exception. But none of these facts proves that the answer was correct. Consider a RAG application. The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information. From an operational perspective, the request succeeded. From a quality perspective, it failed. We need both observability and evaluation. Offline Evaluation Offline evaluation measures an AI application against a predefined dataset. For example, we can create 100 representative questions and expected outcomes. Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again. This helps identify regressions before deploying changes. Useful evaluation dimensions include: - Answer correctness - Relevance - Groundedness - Tool selection - Expected task completion - Policy compliance Online Evaluation Online evaluation measures actual production interactions. We can select a sample of production traces and evaluate them using: - Rule-based checks - Code-based evaluators - LLM-as-a-Judge - Human feedback The important part is to turn real production failures into future test cases. For example: 1. A production trace shows that an agent selected the wrong tool. 2. We identify the relevant input and expected behavior. 3. We create a regression dataset entry. 4. We update the prompt or agent implementation. 5. We rerun offline evaluations. 6. We deploy and monitor the next set of production traces. This creates a continuous improvement loop. Production Traces | v Failure Analysis | v Evaluation Dataset | v Offline Evaluation | v Prompt / Model / Agent Changes | v Deployment | v Online Evaluation | +------> New Regression Cases Tools worth evaluating Two examples of AI-focused observability and evaluation platforms are: Langfuse Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements. Documentation: langfuse.com Arize Phoenix An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions. It provides tracing, evaluation, datasets, experiments, and debugging capabilities. Documentation: phoenix.arize.com These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy. Reference: Langfuse Evaluation Concepts · Arize Phoenix 10. The Role of OpenTelemetry Collector In a production architecture, applications should not necessarily export telemetry directly to every observability backend. This is where the OpenTelemetry Collector becomes useful. The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations. It provides a central place for: - Batching telemetry - Filtering unwanted data - Applying processors - Handling authentication - Reducing telemetry volume - Sampling traces - Routing signals to different backends A simplified architecture looks like this: AI Applications | +----------+----------+ | | | v v v Next.js .NET API Python Agent | | | +----------+----------+ | OTLP | v OpenTelemetry Collector | +--------+--------+ | | | v v v Traces Metrics Logs | | | v v v Jaeger Prometheus Log Backend | v Grafana The same telemetry can also be routed to AI-specific observability platforms where supported. Why centralize collection? In enterprise environments, we may have several applications and teams using different frameworks and languages. A Collector provides a common ingestion point and makes it possible to apply centralized policies. For example, a Collector can filter sensitive attributes before telemetry reaches the final backend. It also allows us to change exporters or observability providers without changing the instrumentation in every application. Agent and gateway deployment OpenTelemetry supports different Collector deployment patterns. A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors. This provides a separation between local telemetry collection and centralized processing. However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling. Reference: OpenTelemetry Collector · Gateway Deployment Pattern 11. Sampling: How Do We Control Telemetry Volume? Agent-based workflows can generate a significant amount of telemetry. Consider an agent that performs 15 model calls and 20 tool calls during a single task. If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive. Sampling helps control this volume. There are two common approaches. Head-Based Sampling The sampling decision is made when the trace starts. For example, we may decide to sample 10% of incoming requests. This is relatively simple and reduces the amount of telemetry generated or exported. However, we do not yet know whether a trace will eventually fail or become unusually slow. Tail-Based Sampling The sampling decision is made after considering the trace's collected spans. This makes it possible to retain: - All failed traces. - All traces exceeding a latency threshold. - Traces containing specific tool failures. - A percentage of successful traces. For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed. However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance. This generally requires trace-ID-aware routing when using multiple Collector instances. A practical approach is to combine sampling strategies: - Preserve error and high-latency traces. - Retain a representative sample of successful traces. - Maintain complete aggregate metrics where possible. - Apply stricter retention policies to verbose telemetry. Reference: OpenTelemetry Sampling · Tail Sampling Processor 12. Security and Privacy Considerations This is one of the most important aspects of AI observability in enterprise environments. Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names. AI telemetry can contain much more sensitive information. For example: - User prompts - System instructions - Generated responses - Tool arguments - Tool results - Retrieved documents - Conversation context - Customer or business identifiers Sending all this information to an observability backend can create additional privacy and data governance risks. Do not capture prompts and responses by default A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise. We can disable this behavior: registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); For many production applications, this is a better starting point. We can still collect useful operational telemetry: - Model identifiers - Token counts - Duration - Finish reasons - Tool execution times - Error information - Workflow identifiers without exporting the complete conversation. Apply data minimization Before exporting telemetry, ask whether each attribute is genuinely needed. Avoid sending: - Access tokens - Authorization headers - Passwords - Raw customer records - Full database query results - Unfiltered tool arguments For sensitive workflows, use approved metadata rather than full payloads. Separate operational telemetry from application data An observability backend should not automatically become another storage system for business data. The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely. Define access and retention policies Enterprise environments should define: - Who can access traces. - Which teams can view prompt and response content. - How long telemetry is retained. - How data is deleted. - Which environments are allowed to capture full content. - How telemetry exports are authenticated and encrypted. A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production. Reference: OpenTelemetry GenAI Content Capture 13. Limitations and Trade-offs OpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself. Instrumentation coverage varies Not every framework, provider, or agent runtime exposes the same information. Some integrations can automatically trace model calls and tool execution, while others require custom instrumentation. We should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility. GenAI semantic conventions are evolving The GenAI conventions are still under development. Attribute names, operation models, and implementation support may evolve. Teams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity. Observability does not guarantee answer quality A trace can explain what happened but does not prove that the model's result is correct. Evaluation datasets, deterministic checks, and human feedback remain necessary. Telemetry has its own cost Collecting every input, output, tool argument, and execution event can generate significant storage and processing overhead. We should measure the cost of observability itself. High-cardinality dimensions can become expensive Adding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series. Use traces for request-level correlation and keep metric dimensions bounded. Distributed tracing requires correct context propagation A2A communication, queues, background tasks, and remote tool execution may cross process boundaries. If context is lost, we can end up with disconnected traces. AI evaluation requires careful design LLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased. A single evaluation score should not become the only definition of quality. Combine deterministic checks, representative datasets, human review, and calibrated evaluators. 14. What Can an AI Chapter Lead Contribute? I think this is where observability moves from an implementation detail to an engineering capability. When multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms. Over time, this creates fragmented visibility. One team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model. As an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications. 1. Define a Common Telemetry Standard Establish a baseline for the telemetry every agent-based application should produce. For example: | Category | Required information | |---|---| | Application | Service name, environment, version | | Workflow | Workflow name, workflow version | | Agent | Agent name, execution status | | Model | Provider, requested model, response model | | Usage | Input/output tokens, cache usage | | Tools | Tool name, duration, execution status | | Execution | Trace ID, span ID, error information | | Business outcome | Completed, failed, canceled, escalated | The objective is to make observability consistent without forcing every team to use the same agent framework. 2. Build a Shared Instrumentation Package We can create an internal TypeScript package that wraps the common OpenTelemetry configuration. For example: @company/ai-observability | +-- OpenTelemetry initialization +-- AI SDK integration +-- Agent span utilities +-- Workflow instrumentation +-- Tool execution tracking +-- Cost attribution +-- Evaluation metadata +-- Privacy and redaction policies This package can provide standard helpers for application-level tracing and reduce duplicated implementation. For teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations. 3. Standardize Agent and Workflow Metrics We should define a common set of metrics that helps teams understand both technical performance and business outcomes. Examples: - Agent success rate - Workflow completion rate - Tool-call failure rate - Model latency - Cost per successful task - Average token usage - Human intervention rate - Evaluation quality scores A shared dashboard can make it easier to compare systems and identify performance regressions. 4. Establish an Internal Evaluation Framework Observability without evaluation gives us only part of the picture. We can create reusable evaluation datasets and testing patterns for common agent behaviors. For example: - Tool selection accuracy - Missing-source detection - Correct handling of incomplete input - Permission violations - Unsupported claims - Failure recovery - Task completion quality This is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified. 5. Establish Privacy and Data Governance Define what can and cannot be captured by default. The shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads. This should include clear ownership, access policies, retention, and audit requirements. 6. Run a Cross-Team Pilot A useful pilot would be a multi-agent workflow that involves several independently deployed services. For example, a merge request review involving coordinator, security, and testing agents. The pilot should demonstrate: 1. One trace across the complete request. 2. Parent-child relationships between agents. 3. Model and tool execution spans. 4. Token and cost attribution. 5. Error and retry visibility. 6. Evaluation results associated with the completed workflow. This would provide a reference architecture that other teams can reuse. 7. Make Observability Part of the Development Lifecycle Observability should not begin after production deployment. We can introduce telemetry and evaluation requirements into the development process itself. For example: - New agents must have execution tracing. - New tools must expose execution status and duration. - Prompt changes must pass relevant regression evaluations. - New workflows must define success and failure criteria. - Production deployments must have basic operational dashboards. This makes observability a standard engineering practice rather than a separate monitoring task. 15. A Practical Adoption Roadmap I would avoid introducing every observability component at once. A phased approach is more useful. | Phase | Focus | Expected result | |---|---|---| | 1 | OpenTelemetry initialization | Basic traces and metrics | | 2 | AI SDK instrumentation | Model and tool visibility | | 3 | Custom agent spans | Complete workflow tracing | | 4 | Cost attribution | Usage and cost per task | | 5 | Evaluation | Offline and online quality checks | | 6 | Security and governance | Privacy and access controls | | 7 | Shared dashboards | Cross-team operational visibility | The first milestone should be simple: For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result. Once this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce. Conclusion AI agents introduce a new level of complexity to application observability. A single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution. OpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components. Combined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production. But the most important distinction remains: Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it. For an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices. The goal should not be to collect more telemetry. It should be to make every production agent workflow understandable, measurable, and continuously improvable. --- References and Further Reading OpenTelemetry 1. OpenTelemetry Official Documentation — Core concepts, signals, instrumentation, and architecture. 2. OpenTelemetry JavaScript — TypeScript and Node.js instrumentation. 3. OpenTelemetry Tracing — Traces, spans, and distributed execution. 4. OpenTelemetry Context Propagation — Distributed trace context. 5. OpenTelemetry Collector — Receiving, processing, and exporting telemetry. 6. OpenTelemetry Gateway Deployment — Centralized telemetry collection. 7. OpenTelemetry Sampling — Head-based and tail-based sampling. GenAI Semantic Conventions 8. GenAI Semantic Conventions — Spans — Model operations, retrieval, and tool execution. 9. GenAI Agent Spans — Agent invocation, workflows, and tool spans. 10. GenAI Metrics — Agent duration, tool calls, and workflow metrics. 11. GenAI Token Metrics — Token consumption and usage measurements. Vercel AI SDK 12. AI SDK Telemetry — OpenTelemetry integration, span hierarchy, and custom attributes. 13. AI SDK v7 Migration Guide — Version-specific telemetry changes. 14. Next.js OpenTelemetry — Application-level instrumentation. AI Evaluation and Observability Platforms 15. Langfuse Evaluation — Online and offline evaluation. 16. Langfuse Core Concepts — Datasets, experiments, and evaluators. 17. Arize Phoenix — Open-source AI observability and evaluation. 18. OpenInference — OpenTelemetry-based instrumentation conventions for AI applications.
Claude Code Sends a Background Agent's Whole Transcript Every 30s for a 3-5 Word Status - Book: LLM Observability Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- You start three background agents on a refactor and go make coffee. While each one works, about every 30 seconds, Claude Code sends that agent's transcript back to the agent's own model with one request: "describe your most recent action in 3-5 words." The answer becomes the little label next to the agent in the agents panel. That is the finding in a gist from the flamerix-ai account that reached 584 points and 236 comments on Hacker News at the time of writing. In the author's OpenTelemetry data for one week, the agent_summary calls were 16.4% of estimated spend and 19.7% of API requests. The main conversation was 15.7%. A Claude Code engineer replied in the thread that the basic mechanism is right, and that the typical share is far lower. Both can be true. Which one is your bill depends on how you work. How the agent_summary timer works Per the gist, the feature has been on by default since Claude Code 2.1.232, in August 2026. Each actively working background agent gets a timer. When it fires, Claude Code forks a request that carries the agent's transcript to the agent's own model, usually Opus, and asks for the 3-5 word description. The gist puts the input-to-output ratio at about 9,000 tokens in for every token out. Three details matter: - The interval comes from the server. The gist says it is a per-model-family server flag, clamped to 30-240 seconds, and the default is already the minimum. No value you can set turns it off. - It uses the agent's model. A GitHub issue on 2.1.312 measured a Haiku subagent and saw the calls go to Haiku, so an Opus subagent gets summarized by Opus. - It skips the transcript. The original feature request found the calls run with skipTranscript: true, so cost tools that read session transcripts never see them. The Claude Code reply on Hacker News narrows the scope: it "only applies to background agents that are actively working." Idle agents don't trigger it. Workflow agents don't get summary timers either, according to a commenter on #86879. This is not the status line you configure yourself. The status line docs say that one "runs locally and does not consume API tokens." The agent_summary label lives in the agents panel, and it is a model call. !A stopwatch whose hand pulls a long paper scroll into a large machine, which prints a single tiny tag at the other end The trade in one picture: the whole transcript in, a few words out. There is no supported off switch. CLAUDE_CODE_AGENT_PROGRESS_SUMMARIES and the SDK's agentProgressSummaries can only turn summaries on, per the gist. CLAUDE_CODE_FORK_SUBAGENT=0 has mixed reports on #86879 and #105981: some people saw zero calls, others still saw summary calls. One commenter suggested the difference comes from a settings env block overriding the shell export. It also disables fork mode, so setting it loses you more than the summaries. What the Bun preload does, and why it breaks The gist ships an MIT-licensed no-agent-summary.js. It relies on Claude Code's native build running on Bun, and on Bun's BUN_OPTIONS variable, which "prepends command-line arguments to any Bun execution." So BUN_OPTIONS=--preload=<path> loads a script before Claude Code's own code runs. The gist says to use the = form, because a space breaks bun subcommands, and to run claude daemon stop --any so background processes pick up the new environment. The script wraps the global setTimeout. When a callback's source text contains one of two anchors, [AgentSummary] Timer fired or agent_summary, it returns an inert timer instead of scheduling the real one. It also checks that it is running inside Claude Code before doing anything. The technique, in its smallest form, looks like this: // Illustration of the technique. Not the gist's script. const realSetTimeout = globalThis.setTimeout; const ANCHOR = "[AgentSummary] Timer fired"; globalThis.setTimeout = (fn, ms, ...args) => { const src = typeof fn === "function" ? String(fn) : ""; if (src.includes(ANCHOR)) { const t = realSetTimeout(() => {}, 2 ** 31 - 1); t.unref?.(); return t; } return realSetTimeout(fn, ms, ...args); }; Don't run that sketch. It has none of the gist's gating or verification, and it shows exactly why this approach is fragile. It matches source text inside a minified binary. Rename one log line, or move the timer off the global setTimeout, and summaries come back with no error. The gist says this itself and asks you to verify after every update. A commenter on #86879 also warned against installing a preload by curl-ing the latest version, since whatever it downloads then runs inside every Claude Code process. Two more caveats from the gist's own pages. Its README says the summary never reaches the model or your conversation, but a commenter on Claude Code 2.1.314 found accepted summary text stored in task.progress.summary in later context, and called that claim too strong. And the license section says plainly: "Changing a client's runtime behaviour may conflict with the terms of the service you use it with; checking that is on you." I haven't run the hook myself. Measure your own agent_summary share You don't need the hook to find out whether this matters to you. Claude Code's monitoring docs log a claude_code.api_request event for each request, with cost_usd and a query_source attribute, the "subsystem that issued the request." The gist grouped those events by query_source, where summaries show up as agent_summary. Use the events, not the metrics. On the claude_code.cost.usage metric, query_source collapses to main, subagent or auxiliary, which hides the summaries inside a bucket. Turn on telemetry and the logs exporter: export CLAUDE_CODE_ENABLE_TELEMETRY=1 export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_PROTOCOL=grpc # your collector's OTLP gRPC address, usually port 4317 export OTEL_EXPORTER_OTLP_ENDPOINT="$COLLECTOR_URL" If you don't have a backend, an OpenTelemetry Collector with the file exporter writes newline-delimited OTLP JSON: receivers: otlp: protocols: grpc: exporters: file: path: ./claude-logs.jsonl service: pipelines: logs: receivers: [otlp] exporters: [file] Work normally for a few days, then group by source with jq and awk: jq -r '.resourceLogs[].scopeLogs[].logRecords[] | [.attributes[] | {(.key): (.value | to_entries[0].value)}] | add | select(.["event.name"] == "api_request") | [(.query_source // "unknown"), (.cost_usd // 0 | tonumber)] | @tsv' claude-logs.jsonl \ | awk -F'\t' ' { cost[$1] += $2; reqs[$1]++; tc += $2; tr++ } END { for (k in cost) printf "%-22s %5.1f%% cost %5.1f%% reqs\n", k, 100 * cost[k] / tc, 100 * reqs[k] / tr }' Each output row is one query_source with its share of estimated cost and of requests. Two things to keep in mind when you read it. cost_usd is Claude Code's own estimate at list price, not an invoice, as the cost docs say. And the share depends on how much of your day runs as background agents, which is the whole argument. !Paper receipts dropping from a conveyor into labelled bins, with one bin filling faster than expected Group the request events by source and the summaries get a bin of their own. Why the numbers disagree Here is every figure on the table, each from a different method and workload: | Source | Share | What it measured | |---|---|---| | The gist | 16.4% of spend, 19.7% of requests | The author's setup, one week, OpenTelemetry estimates | | Issue #86879 | 22.7% of requests, 18.0% of weighted tokens | One 4-hour session, 27 subagents, 2.1.233 | | Commenters on #86879 | From about 10% of spend over 30 days to 26% of cache reads in one session | Separate workloads and methods | | Claude Code engineer on HN | About 2% of cost, just under 4% of requests | Telemetry users, last week | | Same reply, terminal only | About 6% of cost | Where nearly all these calls happen | The high numbers come from people running mostly background agents. The engineer said as much: 16.4% "sounds like it was for a team running mostly background agents, and doesn't seem like a typical session." The low numbers are an average over everyone who sends telemetry, most of whom run few background agents. Neither one is your number. A replay of the 30-second timer over one person's transcripts, posted on the gist, landed at about 8%, partly because it assumed a warm cache. The cache is the part that moves the bill. The engineer said a fix for "summaries that miss the prompt cache" shipped in 2.1.311 and later, calling misses the most expensive case. The team is also working on "a much cheaper mode that sends short notes to a smaller model instead of the full transcript." The reply adds: "we agree that's a lot for a status line." So if you measured before 2.1.311, measure again on a current build before you decide anything. A status that doesn't need a model call I build Hermes IDE, so weigh this section with that in mind. First, what it doesn't do: by default, Hermes runs Claude Code in a real terminal, unchanged, so background agents that Claude Code starts there get summarized exactly as they would anywhere else. What Hermes does differently is its own per-session status. Every session shows one state: working, needs approval, asked you, done, error or idle, and Hermes says when a state is a guess. The README says Hermes "observes what agents report on purpose and what git shows," and the architecture guide describes that status as a state machine driven by the terminal output. Neither document describes a model call in that path. The cost: a state tells you less than a sentence about what the agent is doing right now. !Two real agent sessions side by side in Hermes IDE, Codex and Claude Code, each in its own worktree, with the sessions sidebar showing status, context use, cost and branch Two real agent sessions side by side: Codex on orders-types after adding a RefundStatus type, and Claude Code on orders-api after a follow-up turn adding a log line, each in its own hermes/ worktree, with the sessions sidebar (status, context use, cost, branch) on the left. Each session also shows its spend: the cost the agent reports, or an estimate from its transcript, clearly marked as estimated, with token totals. That tells you which session the money went to. It won't break out agent_summary, and since those calls skip the transcript, a transcript estimate can't see them. For that share, the OpenTelemetry query above is the tool. You can download Hermes. It is source-available under BSL 1.1 and runs on macOS, Windows and Linux. The current release is 2.3.4. It is not affiliated with Nous Research or its Hermes Agent. What to do with this today 1. Run the query above for a few working days on a current Claude Code build. 2. If agent_summary is a small slice, leave it alone. The label is useful. 3. If it is large, look at why first: how many background agents run at once, and on which model. A subagent on Haiku gets summarized by Haiku. 4. Add your numbers and version to #86879. A supported opt-out is what that issue asks for, and real measurements are what moves it. 5. If you reach for the preload anyway, read it first, check your terms, and verify it after every update. --- If this was useful The fix for this kind of surprise is the same every time: per-request telemetry, grouped by who issued the request, before you trust anyone's percentage, including the ones in this post. The LLM Observability Pocket Guide is the short read on picking the tracing and eval tools that give you that view, so a hidden 3-5 word call shows up as its own line instead of a mystery in the monthly total. 
Tail sampling needs a trace-complete path, not just a clever policy Tail sampling needs a trace-complete path, not just a clever policy Head sampling decides early, when a collector has seen only the start of a trace. Tail sampling waits for spans to arrive and then evaluates policies against the completed trace. That extra context can keep errors and slow transactions while dropping routine traffic. The benefit depends on a routing assumption: every span for one trace must reach the same decision point. The OpenTelemetry tail sampling processor groups spans by trace ID and keeps them for a decision window. If a load balancer sends spans from the same trace to different collector instances, each instance sees a fragment. One collector may see an error span and keep it. Another may see only routine spans and discard its part. The resulting backend view can be incomplete even when the policy was written correctly. Use a trace-aware load-balancing path in front of tail sampling, or route all spans through one collector pool that shares the required state. Review the exporter and service architecture as part of the sampling policy. Autoscaling can change the routing layout. A deployment that worked with two replicas may stop making coherent decisions when a new replica joins or an old one is removed. Policy order matters. A latency policy, an error policy and a probabilistic fallback can overlap. The collector needs enough memory and a suitable decision wait to hold spans for the slowest policy that applies. Memory pressure can drop spans before a decision, and an overloaded collector may turn a security-monitoring pipeline into another outage. Monitor incoming spans, pending traces, policy matches and dropped data, not only the final exported ratio. Tenant boundaries need separate attention. Telemetry can contain account identifiers, URLs, query strings, email addresses, tokens embedded in errors, and internal hostnames. Tail sampling reduces volume; it does not make arbitrary telemetry safe to centralize. Restrict which attributes are allowed, redact sensitive fields before sampling, and separate tenants when policy or access rules require it. Sampling policies that use a service.criticality attribute should derive that attribute from a controlled resource pipeline, not from a client-supplied field. The security value of tail sampling is strongest when the policy protects evidence of abnormal behavior. Keep traces with authentication failures, authorization denials, unexpected upstream errors and extreme latency, but test those policies with synthetic traces. A policy that samples every span containing the word error can be flooded by an application that writes user-controlled text into attributes. A policy that ignores errors from a critical service can hide the incident the monitoring team most needs. Record the sampling decision and the reason without storing a second copy of sensitive payloads. Use bounded attribute allowlists and alerts for policy drift. Review the collector configuration when service topology changes, when a new region is added, or when the load balancer changes. Tail sampling is a data-selection control, so its correctness depends on the path that delivers the data. References - OpenTelemetry tail sampling processor documentation - OpenTelemetry sampling concepts
October: Kubernetes memory semantics and Collector edge fixes In October I had 10 PRs merged across four projects. The common thread was practical observability work: clearer Kubernetes memory conventions, Helm chart behavior that is easier to reason about, less flaky Collector tests, and maintainership for components that need active review. Most of my week goes into one question: why does something that looks correctly configured not work in a real cluster? These PRs came from the same place. > TL;DR > - Fixed flaky Kubernetes inventory cache-sync tests. > - Removed a Java 11+ startup failure in a DataPower monitoring extension. > - Let Operator chart users override only the Target Allocator repository. > - Added Kubernetes memory utilization conventions. > - Clarified Kubernetes working set descriptions. > - Added Pod-level memory limit and request conventions. > - Joined Flink Metrics Receiver code ownership. > - Fixed broad Flink operator metric matching. > - Documented Operator chart certificate drift with Argo CD. > - Restored Azure Data Explorer Exporter active ownership and beta status. Kubernetes memory conventions became more precise Pod memory limits and requests now have desired and current forms The Kubernetes semantic conventions had Pod-level CPU limit and request metrics, but not the same shape for memory. I added k8s.pod.memory.limit.{desired,current} and k8s.pod.memory.request.{desired,current} so the model can represent values from spec.resources and status.resources. This shows up in OpenTelemetry semantic conventions, which define standards for consistent telemetry. It closes issue #4133, and there is no release listed for this PR yet. Once implemented downstream, dashboards and alerts can distinguish desired Pod memory settings from current Pod memory status. open-telemetry/semantic-conventions PR 4158 Working set descriptions now say what Linux includes The working set descriptions for k8s.pod.memory.working_set, k8s.node.memory.working_set, and container.memory.working_set were clarified. On Linux, working set excludes inactive file-backed memory, while active file-backed memory remains included. The generated semantic convention documentation now also says that kubelet uses Pod working set when ranking Pods for eviction, and node working set when determining memory availability for the memory.available eviction signal. There is no release listed for this PR yet. The next effect is clearer documentation for comparing usage, working set, and eviction behavior. open-telemetry/semantic-conventions PR 4184 Memory utilization now uses working set as the numerator I added four Kubernetes memory utilization metrics: Pod and container limit utilization, and Pod and container request utilization. These use memory working set as the numerator, not total memory usage. The PR explains the reason: total usage includes reclaimable page cache, which can make a workload look close to its limit even when part of that memory is reclaimable. I validated the behavior against a kubelet Summary API using kind with Kubernetes v1.37.0. There is no release listed for this PR yet. Once implemented downstream, memory utilization can be shown against limits and requests with the numerator semantics documented in the convention and migration guide. open-telemetry/semantic-conventions PR 4219 Operator Helm chart configuration and docs got clearer Private registry users can override only the Target Allocator repository The OpenTelemetry Operator Helm chart did not let users override only manager.targetAllocatorImage.repository while keeping the chart’s default tag. I added explicit defaults for the Target Allocator repository and tag, plus validation that repository and tag are either both set or both empty. This is for the OpenTelemetry Helm Charts project. It closes issue #1866, where 2 people asked for it, and the chart version was bumped to 0.125.0. There is no release listed for this PR yet. The practical effect is simpler values for private registry or air-gapped installs that redirect the Target Allocator image. The upgrade guide also documents the configuration precedence change. open-telemetry/opentelemetry-helm-charts PR 2456 Argo CD users get documentation for auto-generated webhook certs When admissionWebhooks.autoGenerateCert is used, the Operator chart can generate a new CA on each render. With Argo CD, or other tools that use helm template, Helm cannot use lookup during rendering to reuse the existing Secret, so the webhook caBundle, certificate Secret, and running operator may become inconsistent. I documented this in the README’s TLS certificate section, including the possible x509: certificate signed by unknown authority failure. I reproduced the behavior with Argo CD v3.5.5, chart 0.126.0, and Kubernetes v1.37.0. This shipped in opentelemetry-operator-0.127.1. The change is documentation: Argo CD users have a clearer reason to use cert-manager or supply certificate material independently of render-time generation. open-telemetry/opentelemetry-helm-charts PR 2473 Collector contrib tests, receivers, and ownership moved forward Kubernetes inventory cache-sync tests are deterministic now Two internal/k8sinventory/informer tests were flaky because they depended on timing around fake client cache sync. I changed the tests to block the initial Pod List call explicitly, so the failure paths are deterministic without changing production behavior. This is internal test code in the OpenTelemetry Collector contrib repository. It closes issue #50433, which had 24 comments about failed tests on main. There is no release listed for this PR yet. The effect is less noise from racey tests when contributors work on Kubernetes inventory code. open-telemetry/opentelemetry-collector-contrib PR 51657 The Flink Metrics Receiver has another code owner The Flink Metrics Receiver needed additional maintainers. I added myself as a code owner after reviewing receiver changes, running unit tests, and validating changes against a local Apache Flink 2.3.0 deployment in Docker. This shows up in CODEOWNERS and generated README metadata in the OpenTelemetry Collector contrib repository. There is no release listed for this PR yet, and it does not change runtime behavior. The practical change is maintenance coverage for reviews, triage, bug fixes, and receiver upkeep. open-telemetry/opentelemetry-collector-contrib PR 52019 Flink operator metrics now match exact suffixes The Flink Metrics Receiver used strings.Contains for operator-level metric names. That could also match longer names like numRecordsInPerSecond when looking for numRecordsIn. I changed the four affected operator metric cases to exact suffix matching: numRecordsIn, numRecordsOut, numLateRecordsDropped, and currentOutputWatermark. Existing metric names, attributes, and configuration are preserved. There is no release listed for this PR yet. The result is more precise matching for those receiver metrics. open-telemetry/opentelemetry-collector-contrib PR 52406 Azure Data Explorer Exporter has active ownership again The Azure Data Explorer Exporter had been marked unmaintained after its previous active code owner moved to emeritus. I added myself as an active code owner and restored the previous beta stability designation for traces, metrics, and logs. This is visible in generated README, CODEOWNERS, component labels, status metadata, and the changelog. It closes issue #52282, and there is no release listed for this PR yet. The practical effect is that the component again has an active maintainer path for review and triage. open-telemetry/opentelemetry-collector-contrib PR 52500 A Machine Agent monitoring extension starts again on Java 11+ The DataPower extension no longer depends on JDK SAAJ On Machine Agents running a Java 11+ JRE, the DataPower monitoring extension could fail to instantiate because the JDK no longer ships the SAAJ implementation removed by JEP 320. The extension only used SAAJ to build a fixed SOAP request envelope. I removed the javax.xml.soap dependency and now build the envelope directly as a string, with XML attribute escaping for domain and status-class values. The PR bumps the extension version to 2.0.2, and there is no release link listed for this PR yet. The practical change is that Machine Agent users on Java 8 through 21 have a startup path that does not depend on a removed JDK implementation. Appdynamics/datapower-monitoring-extension PR 13 Still in review - open-telemetry/opentelemetry-operator: Generate a list of collector components with specific operator support - Appdynamics/process-monitoring-extension: Update dependencies to remediate reported vulnerabilities - open-telemetry/opentelemetry-collector-contrib: update tested Flink versions and correct README configuration - open-telemetry/opentelemetry-collector-contrib: Record JVM GC metrics for both JobManager and TaskManager - open-telemetry/opentelemetry-collector-contrib: Add opt-in PVC-level volume metrics - open-telemetry/opentelemetry-collector-contrib: Add Warnings section to probabilistic sampler README - CiscoDevNet/appdynamics-charts: ENV variables fix in InfraViz Sources - Fix flaky Kubernetes inventory cache-sync tests - Fix DataPower monitoring extension startup on Java 11+ - Allow overriding only the Target Allocator image repository - Add Pod and container memory limit and request utilization metrics - Clarify memory working set metric descriptions - Add Pod-level memory limit and request metrics - Add a Flink Metrics Receiver code owner - Match Flink operator metrics on exact suffix - Explain autoGenerateCert behavior with Argo CD - opentelemetry-operator-0.127.1 release - Restore Azure Data Explorer Exporter active ownership and beta stability --- _Drafted with AI assistance from the PRs themselves and reviewed by me._
GitHub ActionsとDockerのビルドを1本のトレースにしてGrafana Cloudで見る ! 2026年10月27日に、GitHubホストランナーのUbuntu 24.04で次のバージョンを使って確認した内容です。送信するシグナルはトレースだけです。 Docker Engine 28.0.4 Buildx v0.29.1 BuildKit v0.33.1 OpenTelemetry Collector contrib v0.164.0 GitHubレシーバーのトレース機能は、このバージョンではalphaです。後述する親スパンのID計算もレシーバーの実装に依存します。 はじめに 先日書いたDockerのビルドを遅くしたコミットをGrafana Tempoのトレースで見つけるでは、手元のビルドをトレースにして、遅くなったコミットやDockerfileの手順を調べました。次のステップはCIです。GitHub Actionsの画面で「Dockerのビルドが遅い」とわかっても、そのステップの中で何に時間がかかったかを調べるには、ビルドのログや記録を別に開く必要があります。ワークフロー全体からDockerfileの手順まで、同じトレースの中でたどれるようにしてみました。 TL;DR BuildKitからGrafana CloudのOTLPエンドポイントへトレースを送信できることを確認しました。CIのステップにつなぐには、OpenTelemetry CollectorのGitHubレシーバーがステップに付けるIDをビルドの前に計算し、TRACEPARENTでBuildxへ渡します。CI側のスパンを送るのは、完了した実行の情報をGitHub APIで取得し、Actionsの中で一時的に動かすCollectorです。その結果、ワークフローと2つのジョブが1本のトレースにまとまり、buildジョブのビルドステップの下にBuildKitのスパンがつながりました。 2つの送信経路 今回の構成では、ビルドのトレースとGitHub Actionsのトレースを別々に送ります。両方でトレースIDと親スパンIDを合わせると、Grafana Cloudで1本に表示できます。 ビルド側の送信元は、OpenTelemetryでトレースを送る機能を持つBuildKitです。Dockerfileの各手順のスパンに加えて、クライアントであるBuildxのスパンも、BuildKitへ転送してまとめて送ります。 一方、GitHub Actions側には、BuildKitのようにトレースを送る送信元がありません。ワークフローの実行の記録はWebhookのイベントやREST APIで取得できますが、トレースの形式にはなっていないためです。この記録をスパンに変換するのが、OpenTelemetry CollectorのGitHubレシーバーです。GitHubレシーバーは、GitHubから届くworkflow_runとworkflow_jobのWebhookのイベントを受け取り、ワークフロー、ジョブ、ステップのスパンを作ります。レシーバーはCollectorに組み込んで動かすコンポーネントなので、GitHub Actions側のスパンはCollectorを通し、エクスポーターでGrafana Cloudへ送ります。 今回は、CI側のイベントの取得から送信までをActionsの中で実行する構成を選びました1。ワークフローの完了後に別のワークフローを起動し、そのランナーで一時的にCollectorを動かします。補助スクリプトが完了した実行の情報をGitHub APIで取得し、Webhookと同じ形式のイベントに組み立てて、ランナー内のGitHubレシーバーへ渡す構成です。BuildKitからの送信だけを試す段階では、このCollectorは使いません。 図1: 矢印はデータの移動を表します。左側では計算したTRACEPARENTをBuildxへ渡します。右側は完了後の別ワークフローで、補助スクリプトがGitHub APIの情報をイベント形式へ変換し、OpenTelemetry Collectorへ渡します。Collector内のGitHubレシーバーでCIスパンを生成して送信します。 題材はBusyBox 1.37.0の小さなイメージです。ファイルをコピーしてRUNで別ファイルへ書き出し、ビルドしたイメージを実行して出力も確かめました。実証用のファイル一式を公開しています。 BuildKitからGrafana Cloudへ まず、Actionsの親スパンを付けず、BuildKitからの送信だけを確認しました。Grafana Cloudで取得した接続情報を、リポジトリのActionsシークレットに次の名前で登録します。 シークレット 設定する値 GRAFANA_CLOUD_OTLP_ENDPOINT /otlpまでを含むOTLPのベースURL GRAFANA_CLOUD_OTLP_HEADERS Grafana Cloudの接続画面から取得したAuthorization=...の認証ヘッダー Actionsのステップでは、これらをOTEL_EXPORTER_OTLP_ENDPOINTとOTEL_EXPORTER_OTLP_HEADERSという環境変数に渡しています。scripts/setup-builder.shの内容は次のとおりです。 #!/usr/bin/env bash set -euo pipefail : "${OTEL_EXPORTER_OTLP_ENDPOINT:?Missing OTLP endpoint}" : "${OTEL_EXPORTER_OTLP_HEADERS:?Missing OTLP authentication headers}" docker buildx create --name ci-otel --driver docker-container \ --driver-opt image=moby/buildkit:v0.33.1 \ --driver-opt env.OTEL_TRACES_EXPORTER=otlp \ --driver-opt env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT%/}/v1/traces" \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}" \ --bootstrap >/dev/null docker-containerドライバーではBuildKitが専用のコンテナで動くので、BuildKitのOTLP設定は--driver-opt env.…でそのコンテナの環境変数として渡します。プロトコルはhttp/protobufで、OTLP/HTTPを使います。トレース専用のエンドポイントは、ベースURLの末尾に/v1/tracesを足したURLです。 このbuilderを選んでdocker buildx build --builder ci-otel --load …を実行すると、BuildxとBuildKitの両方のスパンがGrafana Cloudへ届きました。ベースラインの実行から取得したのは、build .をルートにした87スパンのトレースです。イメージのビルドと実行テストも成功しました。 図2: Grafana Cloudで取得したベースラインのトレース。BuildxとBuildKitの87スパンを確認できます2。 Actionsのステップを親にする ここまでのトレースのルートはbuild .です。GitHub Actionsのビルドステップを親にするには、Buildxを実行するステップに環境変数TRACEPARENTを設定します。Buildxは環境変数からトレースコンテキストを読み込む処理でこの値を取り込み、build .を指定された親の子にします。TRACEPARENTに入れるのは、接続先のトレースIDと、親になるステップのスパンIDです。 ただし、ビルドの時点では親になるステップのスパンはまだありません。CI側のスパンは、ワークフローの完了後に作るからです。それでも親のIDを指定できるのは、GitHubレシーバーがトレースIDとスパンIDを乱数ではなく実行IDやジョブIDから決まった規則で計算するためです。ビルドの前に同じ規則で計算すれば、あとで届くステップのスパンと同じIDが得られます。 補助スクリプトscripts/ci-parent.pyは、今回の実行ID、再実行回数、ジョブIDから、レシーバーのID生成の実装と同じ規則でIDを計算します3。計算部分を抜粋すると次のとおりです。 trace_id = hashlib.sha256(f"{run}{attempt}t".encode()).hexdigest()[:32] step_id = hashlib.sha256(f"{job_id}-Build Docker image-s".encode()).hexdigest()[16:32] traceparent = f"00-{trace_id}-{step_id}-01" TRACEPARENTの値は、W3C Trace Contextのtraceparentヘッダーと同じ00-トレースID-親スパンID-01の形式です。スクリプトは、この値をGITHUB_OUTPUTへ書き出します。 ワークフローでは、このスクリプトの出力をビルドステップだけに渡します。build.ymlから該当部分を抜粋すると、次のようになります。 - name: Calculate build step parent id: build_context env: GH_TOKEN: ${{ github.token }} run: python3 scripts/ci-parent.py - name: Build Docker image env: TRACEPARENT: ${{ steps.build_context.outputs.traceparent }} OTEL_RESOURCE_ATTRIBUTES: vcs.ref.head.revision=${{ github.sha }},ci.run.id=${{ github.run_id }},ci.run.attempt=${{ github.run_attempt }},ci.repository=${{ github.repository }},experiment.mode=linked run: docker buildx build --builder ci-otel --load --tag buildkit-otel-spike:ci . ステップのスパンIDはジョブIDとステップ名から決まるので、ステップ名はスクリプトの計算に使ったBuild Docker imageと一致させます。また、ステップ名はジョブ内で一意にしておきます。同じジョブに同名のステップが2つあると、レシーバーが両方に同じスパンIDを付けてしまうからです。 OTEL_RESOURCE_ATTRIBUTESでは、検索に使う実行IDやコミットを付けています。この属性が付くのはBuildxのスパンだけで、BuildKitのスパンには付きません。 CI側のスパンの送信 ビルドステップが終わっても、ワークフロー全体はまだ実行中です。CI側のスパンは、workflow_runのcompletedで起動する送信用のワークフローが作ります。 送信用のワークフローは、完了通知に含まれる実行IDと再実行回数を環境変数で補助スクリプトへ渡します。 OBSERVED_RUN_ID: ${{ github.event.workflow_run.id }} OBSERVED_RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }} 再実行回数を指定するのは、取得した時点で最新の再実行ではなく、通知された回のジョブを読むためです。補助スクリプトは、その回のジョブとステップの情報をGitHub APIで取得し、Webhookと同じ形式のイベントに組み立ててランナー内のCollectorへPOSTします。変換の処理はscripts/export-workflow.pyで確認できます。 collector.yamlの内容は次のとおりです4。 receivers: github: webhook: endpoint: 0.0.0.0:19418 path: /events health_path: /health service_name: github-actions scrapers: scraper: github_org: ymotongpoo processors: batch: timeout: 1s exporters: otlp_http/cloud: endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT} headers: Authorization: ${env:GRAFANA_AUTHORIZATION} timeout: 10s sending_queue: enabled: false retry_on_failure: enabled: false service: telemetry: logs: encoding: json pipelines: traces: receivers: [github] processors: [batch] exporters: [otlp_http/cloud] エクスポーターのendpointはベースURLのままです。otlp_httpのエクスポーターが/v1/tracesを付けるので、BuildKitにトレース専用のURLを渡したのとは指定方法が違います。認証ヘッダーには、BuildKitと同じシークレットからAuthorization=の後ろの値を取り出して渡します。 このCollectorは送信のたびに起動して停止する実証用の設定で、送信キューと再試行を無効にしています5。 自分の環境で試すには、2つのシークレットを登録し、ビルド用と送信用のワークフローをデフォルトブランチに置きます。そのうえでBuildKit linked traceを手動で実行し、ビルドとExport completed workflow traceの両方の完了を待ちます。APIの呼び出しに使うのはcontents: readとactions: readの権限を付けたGITHUB_TOKENで、GitHub AppやPATは不要です。 1本につながった結果 ビルド側の実行とCI側のスパンの送信は、どちらも成功しました。ワークフロー全体とビルドの内部が1本のトレースにつながっています。 図3: 同じ実行のワークフロー、2つのジョブ、各ステップが1本のトレースになりました。Build Docker imageの下にbuild .が接続しており、ビルド内部は折りたたんでいます。 サービスごとの内訳は次のとおりでした。 service.name スパン数 記録している処理 github-actions 21 ワークフロー、ジョブ、待ち時間、ステップ buildx 33 ビルドクライアントの処理 buildkitd 54 ビルドデーモンとDockerfileの手順 合計 108 同一トレースIDのスパン ビルド用のbuildと、その完了後に別のランナーで実行するverify-sourceの2ジョブが、同じルートにつながっています。CIの待ち時間やビルド内部の細部を省くと、親子関係は次のようになります。 BuildKit linked trace ├── build │ └── Build Docker image │ └── build . │ └── … │ └── [4/4] RUN cat message.txt > result.txt └── verify-source └── Verify source fixture build .の親IDはBuild Docker imageのスパンIDと一致し、取得した108スパンはすべて同じワークフローのルートへつながっていました6。 GrafanaのExploreでTempoのデータソースを選び、実行した時間帯に絞って、次のTraceQLで検索します。ci.run.idは、OTEL_RESOURCE_ATTRIBUTESでBuildxのスパンに付けた属性です。 { resource.ci.run.id = "40437818776" && name = "build ." } 前回の記事では、ルートのスパンだけを選ぶ条件（span:parentID = ""）でビルドの所要時間を集計しました。今回のビルドはbuild .にActionsの親があるので、この条件では選べません。手元のビルドとCIのビルドをまとめて選ぶ条件は、{ resource.service.name = "buildx" && name = "build ." }です。ベースラインとCIの両方のトレースが、この条件の検索結果に入ることを確認しました。 CI側のスパンは、送信用のワークフローが完了してから届きます。トレースを開くのは、両方のワークフローが完了してからにしてください7。 今回確認した範囲 BuildKitからGrafana Cloudへの送信と、2ジョブのワークフローへのビルド内部の接続までを確認しました。ただし、IDの生成規則はOpenTelemetryの共通の仕様ではなく、GitHubレシーバー v0.164.0の実装です。Collectorのバージョンやレシーバーのfeature gateを変えるときは、取得したトレースでbuild .が親のステップにつながっているかを確かめ直してください。 CI側の時刻はGitHub APIが秒単位で記録した値です。そのため、親子の接続が正しくても、BuildKitのスパンが親の時間範囲からわずかにはみ出すことがあります。今回の実行では、build .の終了が親のBuild Docker imageの終了を約0.84秒超えていました。 今回は信頼できるリポジトリでの手動実行だけで、matrix、reusable workflow、forkからのPR、実際の再実行、実行間でのキャッシュ再利用は検証していません8。 おわりに GitHub Actionsのワークフローとステップから、Dockerfileのどの手順やレジストリへの通信に時間を使ったかまで、1本のトレースでたどれるようになりました。自分のCIへ取り込むときには、まずBuildKitの送信だけを確認し、必要に応じてCIの親スパンを追加する順で試せます。 脚注 GitHubレシーバーは、常時稼働するCollectorでGitHubからのWebhookを受け取る使い方を想定しています。組織のCIを継続して記録するなら、その構成のほうが向いています。常時稼働のCollectorなら、送信キューと再試行を有効にして、Grafana Cloudへの送信に失敗したスパンを再送することもできます。ただし、GitHubからWebhookを受けるために、インターネットから届くエンドポイントを用意して運用する必要があります。今回はその準備をせずに試すため、Actionsの中で完結させました。 ↩︎ 図の上部に赤く表示されているHEAD HTTP HEADのエラーは、Docker Registryのトークン認証で最初に返る401の認証要求です。今回の実行では、トークン取得後の再試行が200となり、ビルドも成功しています。 ↩︎ ジョブの数値IDは、GitHub APIで今回の実行のジョブの一覧を取得し、名前がGITHUB_JOBと一致するジョブから取り出します。ジョブの表示名を変えたりmatrixを使ったりする場合は、別の方法でジョブを特定する必要があります。 ↩︎ v0.164.0のGitHubレシーバーは、トレースだけを使う場合でも、scrapersを1つ以上設定することを求めます。パイプラインはトレースだけなので、github_orgを使ったメトリクスの収集は動きません。 ↩︎ Collectorはbatchプロセッサーを通して送るので、レシーバーがHTTP 200を返しても、Grafana Cloudへの送信が成功したとは限りません。そこで補助スクリプトは、停止後のCollectorがJSON形式で出力したログを読み、levelがエラーを示していれば失敗として扱います。 ↩︎ 検証時のスクリプトと検証記録を公開しています。取得した108スパンは、スパンIDがすべて一意で、ルートが1つ、親が見つからないスパンは0でした。この検査では、送信したすべてのスパンがGrafana Cloudに届いたかまでは判定できません。 ↩︎ Tempoの最初の検索結果では、ルートのサービス名が<root span not yet received>と表示されました。その後にトレースIDで取得すると、CI側を含む108スパンがそろっていました。 ↩︎ 送信用のワークフローは、元の実行のコミットにある補助スクリプトを実行します。外部のPRが起点になる構成へ広げるときは、GitHubが説明するworkflow_runの権限と注意点を確認してください。 ↩︎
X-Ray SDK / Daemon → OpenTelemetryの主な違い 2026年2月25日、X-Ray SDK / Daemonがメンテナンスモードに入り、AWSはOpenTelemetry（OTel）への移行を勧めています。 docs.aws.amazon.com 自分が担当しているプロジェクトでもX-Ray SDKを使っているので、移行の準備を進めています。 この記事では、X-Ray SDK / DaemonからOpenTelemetryに移行すると書き方や設定がどう変わるのかを項目ごとにまとめます。 主な違い 1. 送信経路 X-Ray Daemon xray -o -n ap-northeast-1 OpenTelemetry collector.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: {} exporters: awsxray: {} service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [awsxray] X-Ray SDKではDaemonがX-Rayに送っていましたが、OTelではコレクターが送ります。コレクターはアプリからトレースを受け取って送り先へ中継するプロセスで、送り先もコレクターの設定で決まります。 コレクターには、AWSが配布しているADOT CollectorなどのOTelのコレクターかCloudWatchエージェントを使い、Daemonを動かしていた場所に置き換えます。送り先を変えたり、X-Ray以外の監視サービス（Datadog、Grafana Cloudなど）にも送ったりする場合も、exportersを変えるだけで、アプリのコードには手を入れずに済みます。 opentelemetry.io 2. 初期化 X-Ray SDK func setupTracing() error { return xray.Configure(xray.Config{ DaemonAddr: "127.0.0.1:2000", }) } OpenTelemetry // xrayは、OTel用のX-Ray互換パッケージ（go.opentelemetry.io/contrib/propagators/aws/xray） func setupTracing(ctx context.Context) (*sdktrace.TracerProvider, error) { exp, err := otlptracegrpc.New(ctx) // localhost:4317のコレクターへ if err != nil { return nil, err } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exp), sdktrace.WithIDGenerator(xray.NewIDGenerator()), // X-Ray形式のトレースID sdktrace.WithResource(resource.NewSchemaless( semconv.ServiceName("my-service"), // トレースマップに出る名前 )), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator( propagation.TraceContext{}, // traceparent xray.Propagator{}, // X-Amzn-Trace-Id )) return tp, nil } X-Ray SDKではxray.Configureだけで済んでいましたが、OTelでは各コンポーネントを指定してTracerProviderを構成します。TracerProviderは、スパン（X-Rayのセグメントにあたる単位）を作る元になるものです。 コードは増えますが、エクスポーター（送り先）・ID生成・リソース・プロパゲーターがそれぞれ独立したコンポーネントになっていて、必要なものだけ差し替えられます。 opentelemetry.io 3. 終了処理 X-Ray SDK // 特に何もしない OpenTelemetry ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() tp, err := setupTracing(ctx) if err != nil { log.Fatalf("setup tracing: %v", err) } defer func() { if err := tp.Shutdown(context.Background()); err != nil { log.Printf("shutdown tracer provider: %v", err) } }() 終了処理は、プロセスを止める前に、まだ送っていないトレースを送り切るための処理です。 X-Ray SDKはセグメントが閉じるたびにDaemonへ送るので、終了処理はいりませんでした。一方OTelは、スパンをいったんメモリにためて、まとめて送ります。そのため、プロセス終了時にtp.Shutdownを呼んで残りを送り切る（フラッシュする）必要があり、呼ばずに終わるとためていた分が消えます。 4. 計装ライブラリ X-Ray SDK func instrument(mux http.Handler, cfg *aws.Config) (http.Handler, *http.Client) { handler := xray.Handler(xray.NewFixedSegmentNamer("my-service"), mux) client := xray.Client(&http.Client{}) awsv2.AWSV2Instrumentor(&cfg.APIOptions) return handler, client } OpenTelemetry func instrument(mux http.Handler, cfg *aws.Config) (http.Handler, *http.Client) { handler := otelhttp.NewHandler(mux, "my-service") client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)} otelaws.AppendMiddlewares(&cfg.APIOptions) return handler, client } X-Ray SDKではX-Ray用のラッパーを使っていましたが、OTelではOTelの計装ライブラリを使います。計装ライブラリは、HTTPやAWS SDKの呼び出しを自動でトレースするライブラリです。 OTelの計装ライブラリで計測した呼び出しも、X-Rayではこれまでと同じくサブセグメント（トレースの中で、呼び出し1つ分を表す単位）として表示されます。そのため、DBや外部APIの呼び出しごとにかかった時間を、移行後も確認できます。 5. 伝播ヘッダー X-Ray SDK X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1 OpenTelemetry（デフォルト） traceparent: 00-5759e988bd862e3fe1be46a994272793-53995c3f42cd8ad8-01 伝播ヘッダーは、別のサービスを呼び出すときにトレースIDなどを渡すためのHTTPヘッダーで、これによって複数のサービスをまたいだ処理が1つのトレースにつながります。 X-Ray SDKではX-Amzn-Trace-Idでトレースを引き継いでいましたが、OTelのデフォルトはW3C Trace Contextという標準のtraceparentヘッダーになります。 ALBなどAWSのサービスはX-Amzn-Trace-Idを使うので、初期化で両方のプロパゲーター（伝播ヘッダーを読み書きするコンポーネント）を登録しないとトレースが途切れます。 w3.org 6. アノテーション（注釈） X-Ray SDK xray.AddAnnotation(ctx, "tenant_id", tenantID) OpenTelemetry span := trace.SpanFromContext(ctx) span.SetAttributes( attribute.String("tenant_id", tenantID), // ここに挙げたキーだけがX-Rayのアノテーションになる attribute.StringSlice("aws.xray.annotations", []string{"tenant_id"}), ) アノテーションは、トレースに付けるキーと値の情報で、X-Rayコンソールでトレースを絞り込む条件に使えます。 X-Ray SDKではxray.AddAnnotationでアノテーションを付けていましたが、OTelではスパンの属性として付けます。 ただし、属性はそのままだとX-Rayでは検索に使えない情報（メタデータ）になります。annotation.tenant_id = "..."のような検索を使っている場合は、aws.xray.annotationsにキーを入れておく必要があります。コード側を変えたくなければ、コレクターのawsxrayエクスポーターのindexed_attributesでキーを指定する方法もあります。 docs.aws.amazon.com github.com/open-telemetry/opentelemetry-collector… 7. サンプリング X-Ray SDK // X-Rayコンソールのサンプリングルールが自動で使われる xray.Configure(xray.Config{ DaemonAddr: "127.0.0.1:2000", }) OpenTelemetry // xraysamplerは、github.com/aws-observability/aws-otel-go/samplers/aws/xray remote, err := xraysampler.NewRemoteSampler(ctx, "my-service", "ecs") // 127.0.0.1:2000からルールを取得 if err != nil { return nil, err } tp := sdktrace.NewTracerProvider( sdktrace.WithSampler(sdktrace.ParentBased(remote)), // ... ) collector.yaml extensions: awsproxy: {} service: extensions: [awsproxy] X-Ray SDKはX-Rayコンソールのサンプリングルールを使っていましたが、OTelのデフォルトはParentBased(AlwaysOn)です。呼び出し元が決めていなければ全件送るので、X-Rayのデフォルトルール（毎秒1件と、超えた分の5%）より送る量が増えます。 これまでのルールを使い続けるには、X-Rayからルールを取得して判定するリモートサンプラーを設定します。ルールの取得はDaemonが中継していたので、コレクターではawsproxy拡張を設定します。 8. Lambdaでの送信 X-Ray SDK func main() { // Lambdaが作るセグメントの下に、サブセグメントが自動で付く lambda.Start(handler) } OpenTelemetry // xrayudpは、github.com/aws-observability/aws-otel-go/exporters/xrayudp // xrayconfigは、go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda/xrayconfig func main() { ctx := context.Background() exp, err := xrayudp.NewSpanExporter(ctx) // Lambda組み込みのX-Rayエンドポイントへ if err != nil { log.Fatalf("create exporter: %v", err) } tp := sdktrace.NewTracerProvider( sdktrace.WithSpanProcessor(sdktrace.NewSimpleSpanProcessor(exp)), sdktrace.WithIDGenerator(xray.NewIDGenerator()), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(xray.Propagator{}) // WithRecommendedOptionsにはotellambda.WithFlusher(tp)が含まれる lambda.Start(otellambda.InstrumentHandler(handler, xrayconfig.WithRecommendedOptions(tp)...)) } Lambdaでは、ECSのようにアプリの隣のコンテナ（サイドカー）でコレクターを動かせないので、送信経路が変わります。xrayudpエクスポーターを使うと、X-Ray SDKと同じくLambda組み込みのX-Rayエンドポイントに直接送れます。 また、Lambdaは呼び出しが終わると、次の呼び出しまで実行環境を一時停止します。プロセスは終了しないので、終了時に呼ぶShutdownだけでは、スパンが送られないまま残ります。otellambda.WithFlusherを指定すると、呼び出しごとにForceFlushが呼ばれ、たまっていたスパンが送られます。 docs.aws.amazon.com 9. ログに出すトレースID X-Ray SDK // 1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx の形式 slog.InfoContext(ctx, "order created", "trace_id", xray.TraceID(ctx)) OpenTelemetry // W3C形式の32桁をX-Ray形式に直す（先頭8桁はUnix時刻） func xrayTraceID(ctx context.Context) string { id := trace.SpanContextFromContext(ctx).TraceID().String() return "1-" + id[:8] + "-" + id[8:] } slog.InfoContext(ctx, "order created", "trace_id", xrayTraceID(ctx)) X-Ray SDKではトレースIDがX-Ray形式で取れていましたが、OTelのトレースIDはW3C形式の32桁の16進数です。 X-Rayのコンソールでは1-58406520-a006649127e371903a2de979のような形式で表示・検索するので、ログのIDをそのまま貼り付けても見つかりません。W3C形式の先頭8桁のあとにハイフンを入れ、頭に1-を付ければ変換できます。 ログに出したトレースIDをX-Rayで検索できれば、遅かったリクエストのログからトレースを開いたり、その逆をたどったりできます。 docs.aws.amazon.com まとめ X-Ray SDK / Daemonは2026年2月25日からメンテナンスモード（終了日は未定）で、X-Rayのサービスとコンソールはそのまま使える 移行後もX-Rayで見られ、送り先はあとからコレクターの設定で追加・変更できる OTelにすると X-Rayに送りながら、ほかの監視サービスも設定だけで試せる 計装がAWS専用でなくなり、メトリクスやログも同じサービス名でそろえられる トレースマップ・サブセグメント・ログのトレースIDによる突き合わせは、移行後もこれまでどおり使える 主な違いは以下のとおり 観点 X-Ray SDK / Daemon OpenTelemetry 送信経路 Daemon コレクター 初期化 xray.Configureだけ 各コンポーネントを指定してTracerProviderを構成 終了処理 不要 tp.Shutdownで未送信分をフラッシュ 計装ライブラリ X-Ray用ラッパー OTelの計装ライブラリ 伝播ヘッダー X-Amzn-Trace-Id 標準のtraceparent（併用可） アノテーション xray.AddAnnotation 属性＋aws.xray.annotationsにキーを指定 サンプリング X-Rayのサンプリングルール デフォルトは全件、ルールを使うならリモートサンプラー Lambdaでの送信 SDKが組み込みのエンドポイントへ送る xrayudpエクスポーターで送り、呼び出しごとにフラッシュ
my second contribution to @opentelemetry 🥳 opentelemetry-collector-contrib#51735
If you are using a Datadog, Langfuse, Langchain, or other OTEL target, you do nothing. Just query it. Build alerts, dashboards, and datasets; whatever you normally do with your OTEL collector. A forward OTEL collector deployment is still useful for filtering but the Datadog Agent, for example, already has buffering and filtering built into it. If you want to deploy your own collector and write your own handler (e.g. put it into a database), you can generate the stubs for the OTEL payloads and convert to domain records. 1. Get the protos here: open-telemetry/opentelemetry-proto 2. Generate the type/class stubs and endpoints for your platform. 3. I strongly recommend pre-filtering with the open source `opentelemetry-collector` before your custom endpoint handles the inbound telemetry payload. The collector is optimized for performing the filtering and I put it upstream of my custom ingest endpoint on a very small VM just to pre-filter and perform auth checks[0][1] on the inbound channel. 4. Do what you need to with the inbound payloads. Examples below: - Ingest endpoint (Codex/Claude -> OTEL collector -> API): github.com/zeeq-ai/zeeq-app/blob/main/src/backend… - Codex telemetry adapter: github.com/zeeq-ai/zeeq-app/blob/main/src/backend… - CC telemetry adapter: github.com/zeeq-ai/zeeq-app/blob/main/src/backend… I run my backend on Postgres with partitioned tables. You can forward to Elastic or other OLAP-type database targets for querying. You can see examples of what I do with the session telemetry here: zeeq.ai [0] github.com/zeeq-ai/zeeq-app/blob/main/build/otel-… [1] github.com/open-telemetry/opentelemetry-collector…
Claude Code Already Emits OpenTelemetry: Who Spends What, on Which Model, in Grafana - Book: LLM Observability Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Your team's Claude Code spend lands as one number on an invoice. Then someone asks the obvious follow-ups. Which team drove last month's jump? Was it the expensive model or the cheap one? Are subagents quietly burning half the budget? Nobody can answer, because the data that would answer it was never switched on. Claude Code ships an OpenTelemetry exporter. Export is off until you set one environment variable, CLAUDE_CODE_ENABLE_TELEMETRY=1, per the monitoring docs; a few more say where the data goes. Once it is on, every API request bumps a cost counter and a token counter, each tagged with the model, the user, the session and the part of Claude Code that made the call. This walkthrough runs the whole pipeline on your laptop: Claude Code into an OpenTelemetry Collector, scraped by Prometheus, drawn by Grafana, all in one Docker Compose file. Then you build one panel per question from the opening. What Claude Code exports (and what it keeps back) Two signals matter for a spend dashboard. Metrics go out through the metrics exporter; events go out through the logs exporter. Per the docs, the metrics are: - claude_code.cost.usage (USD), incremented after each API request, with model, query_source (main, subagent or auxiliary), effort, and attribution for the active skill, plugin or subagent. - claude_code.token.usage (tokens), split by type: input, output, cacheRead, cacheCreation. - claude_code.session.count, claude_code.lines_of_code.count, claude_code.commit.count, claude_code.pull_request.count, claude_code.active_time.total. - claude_code.code_edit_tool.decision, which counts accept and reject decisions on Edit, Write and NotebookEdit, with the file's language. The events include user_prompt, api_request (with cost_usd, duration_ms and token counts per call), api_error, tool_result and tool_decision. Every metric and event carries session.id and user.id, a random installation identifier. When you sign in with a Claude account, you also get organization.id, user.account_uuid and user.email; the docs say the email goes only to the endpoint you configure, never to Anthropic. The defaults lean private. Prompt text is not collected, only its length, unless you set OTEL_LOG_USER_PROMPTS=1. Bash commands and MCP tool arguments stay out unless you set OTEL_LOG_TOOL_DETAILS=1. The docs also state that raw file contents and code snippets are not part of metrics or events. Leave all of those flags off for a cost dashboard. You don't need them. The stack: four small files Make a directory and put four files in it. The Collector receives OTLP from Claude Code and re-exposes metrics in Prometheus format on port 8889. Prometheus scrapes that. Grafana reads Prometheus. !A laptop terminal sending a stream of small coins and token tiles through a funnel into a database cylinder, which feeds three dashboard gauges Claude Code pushes OTLP to the Collector; Prometheus scrapes it; Grafana draws it. docker-compose.yml, with image tags pinned to the current releases of the Collector, Prometheus and Grafana: services: otel-collector: image: otel/opentelemetry-collector-contrib:0.162.0 command: ["--config=/etc/otelcol/config.yaml"] volumes: - ./otel-collector.yaml:/etc/otelcol/config.yaml:ro ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP - "8889:8889" # Prometheus scrape endpoint prometheus: image: prom/prometheus:v3.15.0 volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro ports: - "9090:9090" grafana: image: grafana/grafana:13.2.4 environment: GF_SECURITY_ADMIN_PASSWORD: change-me volumes: - ./grafana:/etc/grafana/provisioning/datasources:ro ports: - "3000:3000" otel-collector.yaml. Metrics go to the Prometheus exporter. Events go to the debug exporter so you can watch them in the container logs without running a log store yet: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: {} exporters: prometheus: endpoint: 0.0.0.0:8889 debug: verbosity: detailed service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] logs: receivers: [otlp] processors: [batch] exporters: [debug] prometheus.yml: global: scrape_interval: 15s scrape_configs: - job_name: otel-collector static_configs: - targets: ["otel-collector:8889"] grafana/prometheus.yaml, a provisioned data source so you skip the click-through: apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus:9090 isDefault: true Then docker compose up -d. Grafana is on port 3000 (user admin, the password from the Compose file). Point Claude Code at it In the shell where you start Claude Code: export CLAUDE_CODE_ENABLE_TELEMETRY=1 export OTEL_METRICS_EXPORTER=otlp export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_PROTOCOL=grpc export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative export OTEL_METRIC_EXPORT_INTERVAL=10000 export OTEL_RESOURCE_ATTRIBUTES="team.id=platform" claude Three of those lines deserve a word. The temporality line. Claude Code's default is delta, and the docs say to switch to cumulative if your backend expects it. Prometheus counters are cumulative, and the Collector's Prometheus exporter README doesn't document what it does with delta sums, so asking for cumulative at the source takes the question off the table. The export interval. The default is 60,000 ms. The docs suggest a shorter interval while you debug; 10 seconds means your first numbers show up fast. Put it back to the default once things work. The resource attribute. OTEL_RESOURCE_ATTRIBUTES keys land on every metric data point as labels by default, so team.id=platform gives you a team dimension for free. Values can't contain spaces. Run a prompt or two, then check what the Collector exposes: curl -s localhost:8889/metrics | grep '^claude_code' Read the metric names off that output rather than guessing. The exporter swaps dots for underscores and appends the unit and _total to counters, so the cost counter shows up as claude_code_cost_usage_USD_total and tokens as claude_code_token_usage_tokens_total. Those are the exact names the ROI measurement guide in Anthropic's GitHub organization queries. Labels follow the same rule: user.email becomes user_email, team.id becomes team_id. If nothing shows up, the docs recommend claude --debug-file <path> and a search for [3P telemetry] errors. To roll this out to a team instead of one laptop, put the same variables in the env block of Claude Code's managed settings file. The docs note that a repository's .claude/settings.json cannot turn telemetry on or redirect it, so a stray repo config can't siphon your data somewhere else. The three panels Create a dashboard. For each panel, pick the Bar gauge visualization, set the query type to Instant, and use Grafana's $__range variable, which equals the dashboard's time window. Change the time picker to 30 days and every panel follows. Who spends. Estimated USD per person over the selected window: sum by (user_email) ( increase(claude_code_cost_usage_USD_total[$__range]) ) If your developers authenticate with API keys instead of OAuth, there is no email to group by (more on that below). Swap user_email for team_id and the panel becomes spend per team. Which model. Same counter, different label: sum by (model) ( increase(claude_code_cost_usage_USD_total[$__range]) ) This is the panel that starts conversations. When one model dominates, the next question is whether the work being done needs it. What kind of work. query_source splits the main conversation from subagents and auxiliary requests: sum by (query_source) ( increase(claude_code_cost_usage_USD_total[$__range]) ) For a finer cut, group by agent_name or skill_name instead. The docs redact user-defined agent names to custom and third-party plugin names to third-party unless OTEL_LOG_TOOL_DETAILS=1 is set, so expect those buckets. On the cost and token counters the flag needs Claude Code v2.1.273 or later; earlier versions stay redacted even with it set. One more query worth a Stat panel: the share of input-side tokens served from the prompt cache. The docs point out that the input type excludes cache reads and writes, so the denominator has to add all three: sum(rate( claude_code_token_usage_tokens_total{type="cacheRead"}[1h] )) / sum(rate( claude_code_token_usage_tokens_total{ type=~"input|cacheRead|cacheCreation" }[1h] )) If that ratio falls off a cliff, something changed what gets cached, and the token panel tells you when it started. Read the numbers with these caveats It's an estimate. The docs say it directly: cost metrics are approximations, and official billing data comes from your API provider (the Claude Console, Amazon Bedrock or Google Cloud). If you're on a seat-based plan, treat the USD figure as relative consumption, not as the invoice. Upgrade before you trust a gateway setup. Before Claude Code v2.1.214, streams that reported usage across multiple frames through a proxy behind ANTHROPIC_BASE_URL inflated the cost and token counters by roughly one extra request per extra frame, per the docs. API keys and cloud providers carry no identity. With a direct API key, Bedrock, Google Cloud or Microsoft Foundry, only user.id (a random installation identifier) and session.id are populated. Attach identity yourself through OTEL_RESOURCE_ATTRIBUTES, set per user in managed settings or a launch wrapper. Session IDs are a cardinality cost. session.id is on every series by default, which helps drill-downs and bloats long-term storage. OTEL_METRICS_INCLUDE_SESSION_ID=false drops it. Short sessions can under-count. Each session starts new series, and increase() works from the samples Prometheus actually scraped, so increments made before a series' first scrape can go missing (explained on the prometheus-users list). A short export interval narrows the gap. Prometheus also has a created-timestamp-zero-ingestion feature flag that may help here. Desktop sessions use a different service name. Local sessions started from the Code tab in the Claude Desktop app report service.name as claude-code-desktop, not claude-code. Add both if you filter on it. What this data can't tell you The panels count spend, tokens and accepted edits. They don't tell you whether the work was any good. An accepted edit is a decision="accept" data point whether the change survives review or gets reverted the next morning. Lines added says nothing about lines that should have been deleted. With the privacy defaults on, you also can't see what was asked. That is the right default, and it is also why "why was this session expensive" has no answer in the metrics. !A large magnifying glass held over a long receipt tape, sharp and readable, while a page of code beside it fades into fog Telemetry makes the receipt sharp. The quality of the work stays out of frame. Claude Code has beta tracing behind CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1, with claude_code.interaction spans that contain LLM request and tool spans. That gives you structure and latency. It still doesn't give you a verdict on output quality. All of this measures the tool your developers use. Your product's LLM calls are a separate system. If you ship LLM features to customers, those calls need their own tracing, usually on the OpenTelemetry GenAI semantic conventions, and their own evals. Watch one mismatch if you put both on one dashboard: the docs note Claude Code doesn't set gen_ai.usage.* attributes, and its input_tokens excludes cache tokens, while the conventions count them. Add input, cache read and cache creation to compare like with like. --- If this was useful The Claude Code dashboard is the easy half, because the tool already emits the data. The harder half is the LLM calls inside your own product, where you choose the tracing backend, the eval approach and the trade-offs between them. That choice is what the LLM Observability Pocket Guide is about: picking tracing and eval tools that fit your team instead of the loudest vendor. 
Also good to see how teams are already using this in their production, and this stability even pushes it further to easier adoption for others. Huge congrats to all the maintainers and contributors for making this happen 🙌 opentelemetry-collector-contrib#51236