Arize-ai

3）Arize Phoenix：仓库定位是 AI Observability & Evaluation，并提供文档、示例与多语言 README。它更适合作为观察和评估层，帮助团队把 Agent/LLM 调用转成可分析的记录，再结合自己的数据集和评分策略。原始仓库：Arize-ai/phoenix 点评：项目名叫 Phoenix 不代表部署后自动治愈；GitHub API 的许可证 SPDX 字段未给出，落地前要直接核对仓库许可证和目标部署方式。

Observability for AI Agents: Tracing, Evaluation and Cost Control in Production How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices. 1. Why Is Observability Different for AI Agents? Traditional backend observability is built around relatively predictable execution paths. A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred. AI agents introduce a different level of complexity. An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input. Consider a relatively simple task: A user asks an AI assistant to analyze a merge request and identify potential problems. Behind the scenes, the application might: 1. Receive the user's request. 2. Invoke a coordinator agent. 3. Call an LLM to determine the next action. 4. Retrieve the merge request from GitLab. 5. Read the changed files. 6. Delegate security analysis to another agent. 7. Execute additional tool calls. 8. Evaluate the collected results. 9. Generate a final report. The user sees one response. The application has actually executed a multi-step workflow involving several components. Now imagine the response takes 45 seconds instead of the expected 8 seconds. Where should we investigate? - Did the model take too long to respond? - Did a tool call time out? - Was the same operation executed repeatedly? - Did the agent generate unnecessary steps? - Did a subagent introduce additional latency? - Was too much context sent to the model? - Did a retry increase the overall execution time and cost? Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why. AI observability is about understanding the entire execution path, not just measuring the final API response. This is where OpenTelemetry becomes particularly useful. 2. What Is OpenTelemetry? <escape>**</escape><Entity value="OpenTelemetry (OTel)" category="software" disambig="Open-source observability framework"/> is an open-source observability framework for collecting, processing, and exporting telemetry data. It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms. Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported. OpenTelemetry is built around three primary signals: | Signal | What it tells us | AI agent example | |---|---|---| | Traces | How an operation was executed | Model call → tool → subagent | | Metrics | How the system performs over time | Token usage, latency, error rate | | Logs | What happened at a specific point | Tool timeout or agent exception | These signals complement each other. A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation. OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector. An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal. References: OpenTelemetry Documentation · OpenTelemetry JavaScript 3. Understanding Distributed Tracing Before discussing AI-specific instrumentation, we need to understand traces and spans. A trace represents the complete execution path of an operation. A span represents one individual unit of work within that execution. For example: Trace: POST /api/agent | +-- HTTP Request | +-- Coordinator Agent | +-- LLM Generation | +-- Tool: GitLab - Get Merge Request | +-- Tool: GitLab - Get Changed Files | +-- Security Agent | | | +-- LLM Generation | | | +-- Tool: Security Analysis | +-- Testing Agent | | | +-- LLM Generation | | | +-- Tool: Test Coverage | +-- Final Response Generation Each span has information such as: - Trace ID - Span ID - Parent span ID - Start and end timestamps - Duration - Attributes - Events - Status and exception information This creates a hierarchical representation of the complete execution. The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them. For AI agents, this distinction is important because one user request can generate many independent operations. Why is context propagation important? In distributed systems, a trace often crosses multiple services. Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service. Without context propagation, every service may generate its own independent trace. We would see several unrelated operations instead of one complete workflow. OpenTelemetry uses W3C Trace Context as its standard propagation format. The traceparent header carries trace and parent-span information between instrumented services. The same principle applies to agent-based architectures. When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution. Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction. The important point is that protocol-level interoperability and observability context propagation are separate concerns. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace. References: OpenTelemetry Traces · Context Propagation · W3C Trace Context 4. OpenTelemetry and GenAI Semantic Conventions One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers. For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes. This makes it difficult to compare different implementations using the same monitoring platform. OpenTelemetry addresses this through Semantic Conventions. Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently. The GenAI semantic conventions extend this approach to generative AI applications. Some important attributes include: | Attribute | Purpose | |---|---| | gen_ai.provider.name | Identifies the model provider | | gen_ai.request.model | Identifies the requested model | | gen_ai.response.model | Identifies the actual response model | | gen_ai.operation.name | Describes the operation | | gen_ai.usage.input_tokens | Input token usage | | gen_ai.usage.output_tokens | Output token usage | | gen_ai.conversation.id | Associates operations with a conversation | | gen_ai.agent.name | Identifies the agent | | gen_ai.tool.name | Identifies the executed tool | For example, an LLM invocation could be represented by: { "gen_ai.operation.name": "chat", "gen_ai.provider.name": "openai", "gen_ai.request.model": "example-model", "gen_ai.usage.input_tokens": 1200, "gen_ai.usage.output_tokens": 350 } This is a simplified span attribute example. The benefit is that observability platforms do not need to understand a different custom schema for every application. Agent-specific conventions The GenAI semantic conventions also define operations for agents and workflows. Some examples include: - invoke_agent - invoke_workflow - execute_tool - plan - retrieval This is particularly valuable for multi-agent architectures. We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents. For example: invoke_workflow merge-request-review | +-- invoke_agent coordinator | | | +-- chat model | +-- execute_tool fetch_merge_request | +-- invoke_agent security-agent | | | +-- chat model | +-- execute_tool analyze_vulnerabilities | +-- invoke_agent testing-agent | | | +-- chat model | +-- execute_tool analyze_test_coverage | +-- Final Aggregation This makes the execution model much easier to understand. An important maturity consideration: As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract. The current convention documents include: - Semantic Conventions for GenAI Spans - Semantic Conventions for Agent Spans - Semantic Conventions for GenAI Metrics 5. Integrating OpenTelemetry with Vercel AI SDK Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration. The AI SDK already supports OpenTelemetry-based instrumentation. It can generate spans for: - Text generation - Streaming generation - Individual model calls - Agent execution - Tool execution - Embedding operations - Reranking This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility. Step 1: Install the packages For an AI SDK v7 application: pnpm add @ai-sdk/otel @vercel/otel The OpenTelemetry integration is provided through @ai-sdk/otel. This is an important version-specific detail. In AI SDK v7, OpenTelemetry integration moved out of the core ai package. Older examples using the v6 experimental_telemetry configuration should not be copied directly into a v7 application. Step 2: Register telemetry in Next.js Create an instrumentation.ts file in the project root. import { registerOTel } from '@vercel/otel'; import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; export function register() { registerOTel({ serviceName: 'ai-agent-platform', }); registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); } There are two responsibilities here: - registerOTel initializes the OpenTelemetry infrastructure for the Next.js application. - registerTelemetry registers the AI SDK-specific integration that creates GenAI spans. I have disabled input and output capture in this example intentionally. AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision. We will return to this in the security section. For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup. Step 3: Instrument AI SDK calls Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration. We can also provide a functionId to identify the business operation or agent. import { generateText } from 'ai'; const result = await generateText({ model: myModel, instructions: ` You are a financial analysis assistant. Analyze the supplied information carefully. `, prompt: 'Analyze the latest portfolio performance.', telemetry: { functionId: 'portfolio-analysis-agent', }, }); The functionId helps distinguish operations in traces. For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as: invoke_agent portfolio-analysis-agent Step 4: Understand the generated spans The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions. For generateText and streamText, the integration records three main types of spans: 1. Agent invocation span The root span represents the complete generation operation, including its execution steps and tool calls. Example: invoke_agent portfolio-analysis-agent 2. Step and model-call spans Each model interaction is represented by step and provider-call spans. These allow us to understand how many model calls occurred during an operation and how long each one took. 3. Tool execution spans Each client-side tool execution receives its own span. For example: invoke_agent portfolio-analysis-agent | +-- step 0 | | | +-- chat example-model | +-- execute_tool get_portfolio | +-- step 1 | | | +-- chat example-model | +-- execute_tool get_market_data | +-- step 2 | +-- chat example-model The resulting trace provides a much more complete representation of an agent workflow than a single model request log. The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available. Step 5: Add additional application context Sometimes the standard GenAI attributes are not enough. For example, we may want to know which business workflow initiated an operation or which application component owns it. The AI SDK provides enrichSpan for adding custom attributes: import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, enrichSpan: ({ spanType, operationId }) => ({ 'app.component': 'portfolio-assistant', 'app.operation_id': operationId, 'app.span_type': spanType, }), }), ); The exact context available to enrichSpan depends on the span and the SDK's integration API. The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry. References: AI SDK Telemetry · Next.js OpenTelemetry · AI SDK v7 Migration Guide 6. Instrumenting Your Own Agent Runtime Built-in instrumentation is useful, but it does not automatically understand every application-level decision. For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval. These are application-level operations. We should instrument them explicitly. OpenTelemetry provides a tracing API for creating custom spans. Consider a custom agent orchestration function: import { trace, SpanStatusCode, } from '@opentelemetry/api'; const tracer = trace.getTracer('agent-orchestrator'); export async function runReviewWorkflow() { return tracer.startActiveSpan( 'workflow.merge-request-review', async (span) => { span.setAttribute( 'workflow.name', 'merge-request-review', ); span.setAttribute( 'workflow.version', '1.0.0', ); try { const result = await executeReview(); span.setStatus({ code: SpanStatusCode.OK, }); return result; } catch (cause) { const error = cause instanceof Error ? cause : new Error(String(cause)); span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR, }); throw error; } finally { span.end(); } }, ); } Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated. We now have two levels of observability: - Framework-level instrumentation describes model and tool execution. - Application-level instrumentation describes business workflows and orchestration decisions. This distinction is important. We should not expect a model instrumentation library to understand the complete business meaning of an application request. Instrumenting remote agent calls For A2A-based systems, distributed tracing becomes even more important. Imagine a coordinator running in one service and a security agent running in another service. The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated. Conceptually: Trace: merge-request-review | +-- Coordinator Agent | | | +-- Model Call | | | +-- A2A: Delegate to Security Agent | | | +-- Security Agent | | | +-- Model Call | | | +-- Tool Execution | +-- Final Aggregation The HTTP request must carry the appropriate trace context, typically through W3C traceparent and tracestate headers. For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context. This is how we avoid fragmented traces across independently deployed agents. 7. What Should We Measure? Collecting telemetry is not enough. We need to define which metrics are useful for understanding the health and quality of an agent platform. I would divide AI observability metrics into five categories. 7.1. Latency Metrics These describe how long operations take. | Metric | Why it matters | |---|---| | End-to-end workflow duration | Measures total user-facing execution time | | Model request duration | Identifies slow inference calls | | Time to first token | Measures perceived response latency | | Tool execution duration | Identifies slow external operations | | Agent invocation duration | Measures the execution time of one agent | | Queue wait duration | Identifies scheduling bottlenecks | For streaming applications, time to first token is particularly useful. A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds. 7.2. Reliability Metrics These help us understand whether workflows complete successfully. Examples: - Agent execution success rate - Tool execution failure rate - Model request failure rate - Timeout rate - Retry rate - Task cancellation rate - Human intervention rate - Incomplete workflow rate A high retry rate can be an early indicator of a problematic integration or an unstable external dependency. We should also distinguish between a successful model response and a successful business workflow. An LLM can return a perfectly valid response while the overall task fails to achieve its objective. 7.3. Token and Model Usage Token usage should be tracked at multiple levels. | Metric | Description | |---|---| | Input tokens | Tokens sent to the model | | Output tokens | Tokens generated by the model | | Cached input tokens | Tokens served from provider cache | | Reasoning tokens | Model-reported reasoning token usage | | Tokens per workflow | Total usage across an entire task | | Tokens per agent | Usage attributed to an individual agent | For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response. 7.4. Agent and Tool Behavior These metrics describe how efficiently agents perform their work. Examples: - Number of tool calls per task - Number of model calls per agent invocation - Number of agent handoffs - Repeated tool-call frequency - Tool-selection success - Average steps per workflow - Average number of clarification requests An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem. The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs. 7.5. Quality Metrics This is where AI observability starts to overlap with AI evaluation. Examples: - Answer correctness - Groundedness - Retrieval relevance - Tool-selection accuracy - Task completion - Policy adherence - Human acceptance rate These metrics cannot always be derived from execution telemetry alone. We need an evaluation strategy to measure the actual quality of the result. 8. Cost Control: Understanding the Real Cost of an Agent One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow. An agent-based application may consume resources across several layers: - LLM inference - Embedding generation - Reranking - External API calls - Database queries - Vector database operations - Container or sandbox execution - Remote agent invocations - Infrastructure and networking The total cost should reflect the entire execution path. A useful starting point is to define a model cost formula: [ C_{\text{model}} = \sum_{i=1}^{n} \left( T_{\text{input},i}P_{\text{input},i} + T_{\text{output},i}P_{\text{output},i} \right) ] Where: - (T) represents token usage. - (P) represents the applicable price per token. - (i) represents an individual model operation. The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities. We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets. The complete workflow cost can then be represented as: [ C_{\text{workflow}} = C_{\text{model}} + C_{\text{tools}} + C_{\text{compute}} + C_{\text{storage}} ] This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available. Why is cost per successful task more useful? Imagine two implementations: | Metric | Agent A | Agent B | |---|---:|---:| | Average cost per attempt | $0.04 | $0.07 | | Successful completion rate | 90% | 98% | | Cost per successful task | ~$0.044 | ~$0.071 | Agent A is cheaper per attempt, but Agent B completes more tasks successfully. Neither is automatically better. The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts. A useful metric is: [ \text{Cost per successful task} = \frac{\text{Total execution cost}} {\text{Number of successful tasks}} ] The numerator should include failed attempts that consumed resources, rather than counting only successful runs. Model selection and routing Observability data can also help us make better model-routing decisions. For example: - Use a smaller model for classification or simple extraction. - Use a more capable model for complex analysis. - Avoid unnecessary model calls when a deterministic operation is sufficient. - Reuse cached context when appropriate. - Set explicit iteration and tool-call limits. - Stop an agent when a task exceeds its budget. The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request. Token metrics in OpenTelemetry The current GenAI token metric conventions include proposed counters such as: - gen_ai.client.inference.usage.input_tokens - gen_ai.client.inference.usage.output_tokens - gen_ai.client.inference.usage.cache_read.input_tokens - gen_ai.client.inference.usage.reasoning.output_tokens These conventions are still in Development. There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics. A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations. For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices. This makes historical cost calculations more reliable when provider pricing changes. References: GenAI Token Metrics · GenAI Metrics 9. Observability Is Not Evaluation This distinction is worth emphasizing. Observability tells us what happened during execution. Evaluation tells us whether the result was good enough. A trace can show that an agent: - Called the correct model. - Used three tools. - Retrieved five documents. - Completed within four seconds. - Returned a response without an exception. But none of these facts proves that the answer was correct. Consider a RAG application. The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information. From an operational perspective, the request succeeded. From a quality perspective, it failed. We need both observability and evaluation. Offline Evaluation Offline evaluation measures an AI application against a predefined dataset. For example, we can create 100 representative questions and expected outcomes. Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again. This helps identify regressions before deploying changes. Useful evaluation dimensions include: - Answer correctness - Relevance - Groundedness - Tool selection - Expected task completion - Policy compliance Online Evaluation Online evaluation measures actual production interactions. We can select a sample of production traces and evaluate them using: - Rule-based checks - Code-based evaluators - LLM-as-a-Judge - Human feedback The important part is to turn real production failures into future test cases. For example: 1. A production trace shows that an agent selected the wrong tool. 2. We identify the relevant input and expected behavior. 3. We create a regression dataset entry. 4. We update the prompt or agent implementation. 5. We rerun offline evaluations. 6. We deploy and monitor the next set of production traces. This creates a continuous improvement loop. Production Traces | v Failure Analysis | v Evaluation Dataset | v Offline Evaluation | v Prompt / Model / Agent Changes | v Deployment | v Online Evaluation | +------> New Regression Cases Tools worth evaluating Two examples of AI-focused observability and evaluation platforms are: Langfuse Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements. Documentation: langfuse.com Arize Phoenix An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions. It provides tracing, evaluation, datasets, experiments, and debugging capabilities. Documentation: phoenix.arize.com These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy. Reference: Langfuse Evaluation Concepts · Arize Phoenix 10. The Role of OpenTelemetry Collector In a production architecture, applications should not necessarily export telemetry directly to every observability backend. This is where the OpenTelemetry Collector becomes useful. The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations. It provides a central place for: - Batching telemetry - Filtering unwanted data - Applying processors - Handling authentication - Reducing telemetry volume - Sampling traces - Routing signals to different backends A simplified architecture looks like this: AI Applications | +----------+----------+ | | | v v v Next.js .NET API Python Agent | | | +----------+----------+ | OTLP | v OpenTelemetry Collector | +--------+--------+ | | | v v v Traces Metrics Logs | | | v v v Jaeger Prometheus Log Backend | v Grafana The same telemetry can also be routed to AI-specific observability platforms where supported. Why centralize collection? In enterprise environments, we may have several applications and teams using different frameworks and languages. A Collector provides a common ingestion point and makes it possible to apply centralized policies. For example, a Collector can filter sensitive attributes before telemetry reaches the final backend. It also allows us to change exporters or observability providers without changing the instrumentation in every application. Agent and gateway deployment OpenTelemetry supports different Collector deployment patterns. A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors. This provides a separation between local telemetry collection and centralized processing. However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling. Reference: OpenTelemetry Collector · Gateway Deployment Pattern 11. Sampling: How Do We Control Telemetry Volume? Agent-based workflows can generate a significant amount of telemetry. Consider an agent that performs 15 model calls and 20 tool calls during a single task. If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive. Sampling helps control this volume. There are two common approaches. Head-Based Sampling The sampling decision is made when the trace starts. For example, we may decide to sample 10% of incoming requests. This is relatively simple and reduces the amount of telemetry generated or exported. However, we do not yet know whether a trace will eventually fail or become unusually slow. Tail-Based Sampling The sampling decision is made after considering the trace's collected spans. This makes it possible to retain: - All failed traces. - All traces exceeding a latency threshold. - Traces containing specific tool failures. - A percentage of successful traces. For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed. However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance. This generally requires trace-ID-aware routing when using multiple Collector instances. A practical approach is to combine sampling strategies: - Preserve error and high-latency traces. - Retain a representative sample of successful traces. - Maintain complete aggregate metrics where possible. - Apply stricter retention policies to verbose telemetry. Reference: OpenTelemetry Sampling · Tail Sampling Processor 12. Security and Privacy Considerations This is one of the most important aspects of AI observability in enterprise environments. Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names. AI telemetry can contain much more sensitive information. For example: - User prompts - System instructions - Generated responses - Tool arguments - Tool results - Retrieved documents - Conversation context - Customer or business identifiers Sending all this information to an observability backend can create additional privacy and data governance risks. Do not capture prompts and responses by default A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise. We can disable this behavior: registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); For many production applications, this is a better starting point. We can still collect useful operational telemetry: - Model identifiers - Token counts - Duration - Finish reasons - Tool execution times - Error information - Workflow identifiers without exporting the complete conversation. Apply data minimization Before exporting telemetry, ask whether each attribute is genuinely needed. Avoid sending: - Access tokens - Authorization headers - Passwords - Raw customer records - Full database query results - Unfiltered tool arguments For sensitive workflows, use approved metadata rather than full payloads. Separate operational telemetry from application data An observability backend should not automatically become another storage system for business data. The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely. Define access and retention policies Enterprise environments should define: - Who can access traces. - Which teams can view prompt and response content. - How long telemetry is retained. - How data is deleted. - Which environments are allowed to capture full content. - How telemetry exports are authenticated and encrypted. A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production. Reference: OpenTelemetry GenAI Content Capture 13. Limitations and Trade-offs OpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself. Instrumentation coverage varies Not every framework, provider, or agent runtime exposes the same information. Some integrations can automatically trace model calls and tool execution, while others require custom instrumentation. We should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility. GenAI semantic conventions are evolving The GenAI conventions are still under development. Attribute names, operation models, and implementation support may evolve. Teams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity. Observability does not guarantee answer quality A trace can explain what happened but does not prove that the model's result is correct. Evaluation datasets, deterministic checks, and human feedback remain necessary. Telemetry has its own cost Collecting every input, output, tool argument, and execution event can generate significant storage and processing overhead. We should measure the cost of observability itself. High-cardinality dimensions can become expensive Adding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series. Use traces for request-level correlation and keep metric dimensions bounded. Distributed tracing requires correct context propagation A2A communication, queues, background tasks, and remote tool execution may cross process boundaries. If context is lost, we can end up with disconnected traces. AI evaluation requires careful design LLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased. A single evaluation score should not become the only definition of quality. Combine deterministic checks, representative datasets, human review, and calibrated evaluators. 14. What Can an AI Chapter Lead Contribute? I think this is where observability moves from an implementation detail to an engineering capability. When multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms. Over time, this creates fragmented visibility. One team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model. As an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications. 1. Define a Common Telemetry Standard Establish a baseline for the telemetry every agent-based application should produce. For example: | Category | Required information | |---|---| | Application | Service name, environment, version | | Workflow | Workflow name, workflow version | | Agent | Agent name, execution status | | Model | Provider, requested model, response model | | Usage | Input/output tokens, cache usage | | Tools | Tool name, duration, execution status | | Execution | Trace ID, span ID, error information | | Business outcome | Completed, failed, canceled, escalated | The objective is to make observability consistent without forcing every team to use the same agent framework. 2. Build a Shared Instrumentation Package We can create an internal TypeScript package that wraps the common OpenTelemetry configuration. For example: @company/ai-observability | +-- OpenTelemetry initialization +-- AI SDK integration +-- Agent span utilities +-- Workflow instrumentation +-- Tool execution tracking +-- Cost attribution +-- Evaluation metadata +-- Privacy and redaction policies This package can provide standard helpers for application-level tracing and reduce duplicated implementation. For teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations. 3. Standardize Agent and Workflow Metrics We should define a common set of metrics that helps teams understand both technical performance and business outcomes. Examples: - Agent success rate - Workflow completion rate - Tool-call failure rate - Model latency - Cost per successful task - Average token usage - Human intervention rate - Evaluation quality scores A shared dashboard can make it easier to compare systems and identify performance regressions. 4. Establish an Internal Evaluation Framework Observability without evaluation gives us only part of the picture. We can create reusable evaluation datasets and testing patterns for common agent behaviors. For example: - Tool selection accuracy - Missing-source detection - Correct handling of incomplete input - Permission violations - Unsupported claims - Failure recovery - Task completion quality This is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified. 5. Establish Privacy and Data Governance Define what can and cannot be captured by default. The shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads. This should include clear ownership, access policies, retention, and audit requirements. 6. Run a Cross-Team Pilot A useful pilot would be a multi-agent workflow that involves several independently deployed services. For example, a merge request review involving coordinator, security, and testing agents. The pilot should demonstrate: 1. One trace across the complete request. 2. Parent-child relationships between agents. 3. Model and tool execution spans. 4. Token and cost attribution. 5. Error and retry visibility. 6. Evaluation results associated with the completed workflow. This would provide a reference architecture that other teams can reuse. 7. Make Observability Part of the Development Lifecycle Observability should not begin after production deployment. We can introduce telemetry and evaluation requirements into the development process itself. For example: - New agents must have execution tracing. - New tools must expose execution status and duration. - Prompt changes must pass relevant regression evaluations. - New workflows must define success and failure criteria. - Production deployments must have basic operational dashboards. This makes observability a standard engineering practice rather than a separate monitoring task. 15. A Practical Adoption Roadmap I would avoid introducing every observability component at once. A phased approach is more useful. | Phase | Focus | Expected result | |---|---|---| | 1 | OpenTelemetry initialization | Basic traces and metrics | | 2 | AI SDK instrumentation | Model and tool visibility | | 3 | Custom agent spans | Complete workflow tracing | | 4 | Cost attribution | Usage and cost per task | | 5 | Evaluation | Offline and online quality checks | | 6 | Security and governance | Privacy and access controls | | 7 | Shared dashboards | Cross-team operational visibility | The first milestone should be simple: For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result. Once this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce. Conclusion AI agents introduce a new level of complexity to application observability. A single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution. OpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components. Combined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production. But the most important distinction remains: Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it. For an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices. The goal should not be to collect more telemetry. It should be to make every production agent workflow understandable, measurable, and continuously improvable. --- References and Further Reading OpenTelemetry 1. OpenTelemetry Official Documentation — Core concepts, signals, instrumentation, and architecture. 2. OpenTelemetry JavaScript — TypeScript and Node.js instrumentation. 3. OpenTelemetry Tracing — Traces, spans, and distributed execution. 4. OpenTelemetry Context Propagation — Distributed trace context. 5. OpenTelemetry Collector — Receiving, processing, and exporting telemetry. 6. OpenTelemetry Gateway Deployment — Centralized telemetry collection. 7. OpenTelemetry Sampling — Head-based and tail-based sampling. GenAI Semantic Conventions 8. GenAI Semantic Conventions — Spans — Model operations, retrieval, and tool execution. 9. GenAI Agent Spans — Agent invocation, workflows, and tool spans. 10. GenAI Metrics — Agent duration, tool calls, and workflow metrics. 11. GenAI Token Metrics — Token consumption and usage measurements. Vercel AI SDK 12. AI SDK Telemetry — OpenTelemetry integration, span hierarchy, and custom attributes. 13. AI SDK v7 Migration Guide — Version-specific telemetry changes. 14. Next.js OpenTelemetry — Application-level instrumentation. AI Evaluation and Observability Platforms 15. Langfuse Evaluation — Online and offline evaluation. 16. Langfuse Core Concepts — Datasets, experiments, and evaluators. 17. Arize Phoenix — Open-source AI observability and evaluation. 18. OpenInference — OpenTelemetry-based instrumentation conventions for AI applications.

> Stopping a bad plan is cheaper than tracing a bad run. Most of this list watches an AI app after it's live. coduo-afk has a plan reviewer that refuses a ticket graph that can't be built, before any branch exists. sahildave/coduo-afk

AI Agent Regression Testing: Production Traces to CI AI Agent Regression Testing: How to Turn Production Traces into CI Gates TL;DR: Trace-based regression testing turns a failing production agent trace into a CI test case. The trace's recorded tool and model outputs become fixtures, so the test replays offline and deterministically on every pull request. Tracely is an MIT-licensed open-source project that packages this loop: OTLP ingest, graders, failure clustering, promotion of a trace to a case, and a GitHub Action gate. Langfuse, LangSmith and Promptfoo cover parts of the same loop. None of them removes the main engineering cost, which is routing every tool and model call through a seam you can record and replay. Key Takeaways - An agent regression test is a test built from a real failure. A trace that went wrong in production becomes a case your CI must pass forever after. - Hermetic replay is what makes the test usable in CI. Recorded tool and LLM responses stand in for live calls, so the suite needs no API keys, costs nothing in tokens, and gives the same result every time. - A fail-to-pass contract is the check that the test is trustworthy. The case must fail on the old code and pass on the fix, or the test proves nothing. - Tracely (about 1.2k GitHub stars, MIT, Python/FastAPI with ClickHouse) bundles ingest, grading, clustering, promotion and a tracely-gate GitHub Action in one self-hosted stack. - Langfuse and LangSmith let you add traces to datasets and run experiments over them. Promptfoo runs assertion suites in CI. All three re-call the model live unless you add your own fixture layer. - Replay only works if the agent sends tool and model calls through a recordable seam. Plan that refactor before you pick a tool. Why do AI agents need regression tests built from traces? Agent failures rarely show up in the test inputs a team writes on day one. They come from real conversations: a user phrases a refund request the planner did not expect, a tool returns an empty list instead of an error, or a model upgrade changes how often the agent calls a search tool. By the time someone sees the failure, it lives in a trace (a nested record of every prompt, model response, tool call and result in one run). Teams usually handle this in one of two ways, and both leak: 1. Hand-written eval sets. Someone reads the bad trace, writes a new test input and an expected output, and adds it to a YAML file or dataset. The new input is a paraphrase of what happened, not a copy. The tool responses that caused the failure are usually lost, and the test then calls live tools that now return different data. 2. Dashboards with no gate. The observability platform records the failure, an online evaluator scores it low, and the score sits on a chart. Nothing stops the next pull request from bringing the same failure back. Trace-based regression testing closes that gap. The trace is the test: its exact inputs, the exact tool outputs the agent saw, and the model responses in recorded order. A grader decides pass or fail. The suite runs on every pull request and blocks merges that bring a known failure back. This is the same discipline as "write a failing test before you fix the bug" in ordinary software. The difference is that agents are non-deterministic and depend heavily on external services, so the test needs recorded fixtures before it can be repeated. What does a trace-to-test pipeline look like? The loop has five stages. Every implementation, whether bought or built, needs all five. | Stage | What happens | What breaks if it is missing | |---|---|---| | Observe | Every production run emits a trace (OpenTelemetry gen_ai.* or OpenInference attributes) indexed by agent, conversation, turn and step | You cannot reproduce a failure you never recorded | | Grade | Structural checks (schema, tool-call order, no model needed) and LLM judges score runs online | Failures are found by users, not by the system | | Triage | Failing runs are clustered by structural signature and semantic similarity | 31 broken runs become 31 tickets instead of one issue with a count | | Promote | A representative failing trace becomes a hermetic case with its fixtures recorded | The "test" re-calls live services and drifts | | Gate | CI replays the suite against the change and fails the build on regressions | The fix ships, and the next refactor quietly reverts it | The Promote stage is where most homegrown setups stop. Copying a trace's input into a dataset is easy. Copying the world the agent saw (tool results, retrieval hits, intermediate model outputs) and replaying it in order is the actual work. How does Tracely implement trace-native CI/CD? Tracely is an open-source project by Julien Wuthrich that describes itself as "trace-native CI/CD for AI agents." It is MIT-licensed, has about 1.2k stars at the time of writing, and is still early (a few dozen commits). Its stack is a Python backend (FastAPI, Celery, uv) and a Next.js 15 UI. It stores traces in ClickHouse, the case registry in Postgres with pgvector, the queue in Redis and blobs in MinIO/S3. A local demo starts with docker compose --profile demo up -d --build --wait. Ingest: OTLP first, SDK second Any OpenTelemetry exporter can post to POST {endpoint}/v1/traces with a Bearer ingest key. Tracely reads both gen_ai.* semantic-convention attributes and OpenInference attributes, so existing instrumentation keeps working. The Python SDK (tracelyai-sdk) adds auto-instrumentation for OpenAI, Anthropic, Google, Mistral and LangChain: import tracely_sdk as tracely tracely.init( endpoint="http://localhost:8000", api_key="tracely_dev_key", service_name="support-agent", env="prod", # prod | staging | ci | dev — the gating axis instrument="auto", ) with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"): client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Where is order ORD-4471?"}], ) The env field matters more than it looks. It separates production traces (the source of failures) from CI traces (the ones the gate grades), so the same pipeline can serve both. Grade and cluster Evaluators come in two kinds. Structural checks need no model. LLM judges run at conversation, run or span level and can output a score, boolean, number, text or JSON-schema result. Failing runs are then clustered "structurally and semantically" with a structural signature plus embedding similarity. In the project's words, "31 broken runs become one issue with a count." Promote with a fail-to-pass contract This is the most important idea in the project. Promoting a failing trace bundles "recorded input, tool and LLM outputs" as fixtures. The case is trusted only if it "must fail on the old code and pass on the fix." A regression test that passes on the buggy code is a test that cannot fail, which is the most common way evaluation suites lie to their owners. The contract makes that check part of the workflow instead of leaving it to discipline. Multi-turn behaviour uses scenarios instead of single cases: either a scripted conversation, or an adversarial goal that a red-team model improvises against. Gate in GitHub Actions Replay runs offline against the recorded fixtures, with no API keys and no model spend. tracely gate exits non-zero, posts a commit status and upserts a PR comment. The bundled action has two modes. Tracely can call your agent through an entrypoint (tracely replay planner --entrypoint my_pkg.agent:run, or --cmd "node run.js" for non-Python agents), or it can grade the CI traces your own test job emitted: - uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master with: mode: gate agent: planner api: https://tracely.your-co.dev key: ${{ secrets.TRACELY_KEY }} Pin that @master reference to a commit SHA before you rely on it. A gate that can change underneath you is not a gate. The constraint you cannot skip The README states it plainly: hermetic replay "requires your agent to route tool/model calls through the SDK's call_tool / call_llm seam." Auto-instrumentation is enough to record a trace. It is not enough to replay one, because replay means intercepting the call and returning the recorded value instead of making it. If your agent calls requests.get from inside a tool, or a framework makes model calls you cannot reach, those calls go live during replay. Other caveats from the project itself: judges and failure analysis degrade without an LLM key, the default AUTH_MODE=dev with tracely_dev_key is open access and only suitable for local use, and the Celery worker does not hot-reload. How do Langfuse, LangSmith and Promptfoo handle the same loop? None of the established tools is called "trace-native CI," but each covers some of the stages. The question is which stages you get for free and which you build. | Stage | Tracely | Langfuse | LangSmith | Promptfoo | |---|---|---|---|---| | Production tracing | Yes (OTLP + SDK) | Yes, core product | Yes, core product | No (test runs only) | | Online grading | Structural + LLM judges | Online evaluators on sampled traffic | Online evaluators | No | | Failure clustering | Built in | Manual filtering and annotation | Manual filtering and annotation | No | | Trace → test | Promote to hermetic case | Add trace to dataset | Add run to dataset | Copy input into YAML test case | | Recorded tool fixtures | Yes, bundled on promote | Inputs/expected outputs only | Inputs/reference outputs only | No, you write mocks | | CI gate | tracely-gate action, PR comment | Experiments via SDK in your CI job | Evaluations via SDK in your CI job | promptfoo eval + GitHub Action | | Offline / no model spend | Yes, by design | No, experiments call the model | No, evaluations call the model | No, unless you mock providers | | Maturity | Early, small community | Mature, MIT core | Mature, commercial | Mature, MIT | A few clarifications on that table. Langfuse and LangSmith datasets store inputs and expected outputs, not the world. Adding a trace to a Langfuse dataset or a LangSmith dataset captures what went in and, optionally, what should come out. When you run an experiment, your application runs again, live. That is the right design for comparing a new prompt or model against a fixed set of inputs: you want the new model's real behaviour. It is the wrong design for a regression test of agent logic, because the tools return today's data instead of the data that triggered the bug. For the trade-offs between these two platforms, see our [LangSmith vs Langfuse vs Phoenix comparison](/articles/LangSmith-vs-Langfuse-vs-Phoenix-LLM-Agent-Observability). Promptfoo is the strongest CI gate and the weakest trace source. Its GitHub Action and assertion language are mature, and it runs red-team suites no trace tool matches. But it never sees production. Every case is something a human typed, so the "trace to test" step is copy-paste. Our [Langfuse vs Promptfoo comparison](/articles/langfuse-vs-promptfoo) covers why many teams pair the two. Tracely's real differentiator is the fixture bundle plus the fail-to-pass contract. The tracing and judging overlap heavily with Langfuse. Indeed, the project describes parts of its design as mirroring Langfuse's. What is new is treating promotion as a test-engineering operation with a validity check, not as a "save to dataset" button. How do you build hermetic replay without adopting a new platform? If you already run Langfuse or LangSmith and do not want another stack, you can build the missing stages yourself. The core is a single seam that every tool and model call goes through. This is a generic sketch of the pattern, not any vendor's API: import json, hashlib from contextvars import ContextVar _mode = ContextVar("mode", default="live") # live | record | replay _tape = ContextVar("tape", default=None) # list of recorded calls def _key(kind, name, args): blob = json.dumps([kind, name, args], sort_keys=True, default=str) return hashlib.sha256(blob.encode()).hexdigest()[:16] def seam(kind, name, fn, **args): """Every tool call and LLM call goes through here.""" mode, tape = _mode.get(), _tape.get() if mode == "replay": rec = tape.pop(0) # strict order, not lookup if rec["key"] != _key(kind, name, args): raise AssertionError(f"divergence at {kind}:{name}") return rec["output"] out = fn(**args) if mode == "record": tape.append({"key": _key(kind, name, args), "kind": kind, "name": name, "output": out}) return out Three design choices in that sketch decide whether the tests are useful. 1. Replay in strict order, and fail on divergence. If the fixed agent makes a different call than the recorded one, a lookup-by-key replay silently returns nothing or the wrong record. Raising on divergence turns "the agent took a different path" into an explicit result. Sometimes that is the fix working, and you re-record. Sometimes it is a new bug. 2. Decide which calls are fixtures and which are under test. In a test of planner logic, the model calls are the system under test, so replaying them proves only that the code around them is deterministic. A common split: replay tools and retrieval always, and replay the model only for the steps before the step that failed, then call the model live (or a pinned cheap model) for the step being tested. Tracely's offline mode replays everything. That is cheaper and deterministic, but it catches regressions in orchestration code, not in model behaviour. 3. Grade the outcome, not the transcript. Exact-match on model text breaks on every model upgrade. Grade structural facts ("called refund with the original order ID", "did not call cancel_subscription") and use an LLM judge only where structure cannot express the requirement. Then write the fail-to-pass check as an actual test step: run the case against the parent commit and require a failure, run it against the fix and require a pass. Without that step you will end up with cases that were promoted from a trace but never reproduced the bug, and they pass forever. When is trace-based regression testing the wrong tool? It is not a replacement for every kind of evaluation. - Model or prompt comparisons need live calls by definition. Use dataset experiments (Langfuse, LangSmith) or a Promptfoo matrix, and accept the token cost. - Security and adversarial testing need inputs no user has sent yet. A trace suite only knows past failures. Red-team generation (Promptfoo plugins, or Tracely's adversarial scenarios) fills that gap. - Agents whose tools have side effects you cannot record. Examples are a browser agent, a coding agent running shell commands, or anything that depends on wall-clock time. These need a sandbox or a simulated environment, not a tape. Replaying a recorded DOM is possible but brittle. - Very low traffic. If you have a handful of production runs a week, hand-written cases are faster than running an ingest stack. The approach pays off when you have steady production traffic, recurring failure modes, and orchestration code that changes often. That is the profile of most customer-facing tool-calling agents. How should a team adopt trace-to-test in practice? A sequence that keeps the investment proportional: 1. Standardise tracing on OpenTelemetry gen_ai.* attributes (see the OpenTelemetry GenAI semantic conventions). Every tool in this article can ingest them, so the choice of backend stays reversible. 2. Introduce the call seam in the agent code. Do this first, because it is the step every option needs and the one that takes longest. 3. Promote by hand for a month. Take the five worst production failures, record them, write structural graders, and run them in CI. You will learn which calls leak past the seam. 4. Then decide on a platform. If clustering and promotion are now the bottleneck, a tool like Tracely earns its operational cost (ClickHouse, Postgres, Redis and object storage). If your observability platform's datasets plus your own fixture layer are enough, stay there. 5. Treat the gate as production infrastructure. Pin action versions, alert on gate failures you did not expect, and review the suite whenever you upgrade a model. A failure across the whole suite after a model change is a signal, not noise. FAQ What is AI agent regression testing? AI agent regression testing checks that a change to an agent's code, prompts or model does not bring back a failure that was already fixed. The strongest form builds each test from a real failing trace and replays it on every pull request. What is hermetic replay for LLM agents? Hermetic replay runs an agent test with recorded tool and model responses instead of live calls. The test needs no network access, no API keys and no token spend, and it gives the same result on every run, which makes it usable as a CI gate. What is a fail-to-pass contract? A fail-to-pass contract is a rule that a regression case must fail on the code that had the bug and pass on the code that fixes it. It proves the test actually reproduces the failure, instead of being a test that cannot fail. Is Tracely a replacement for Langfuse or LangSmith? Not yet for most teams. Tracely overlaps with their tracing and judging but is an early project with a small community. Its distinctive parts are fixture-bundled promotion, failure clustering and an offline CI gate, which you can also build on top of an existing platform. Can Promptfoo test agents built from production traces? Promptfoo can gate CI on agent outputs and run red-team suites, but it does not ingest production traces. You copy the trace's input into a test case by hand and write your own mocks for tool responses. Do I need OpenTelemetry to do trace-based testing? No, but it keeps your options open. Tracely, Langfuse, LangSmith and Phoenix all accept OpenTelemetry traces, so instrumenting once with gen_ai.* attributes lets you change backends without re-instrumenting the agent. Sources - Tracely: trace-native CI/CD for AI agents (GitHub) - Langfuse datasets documentation - LangSmith evaluation documentation - Promptfoo GitHub Action integration - OpenTelemetry semantic conventions for generative AI - OpenInference instrumentation conventions (Arize) --- Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co.

Phoenix stored prompts could lose strict mode when converted for OpenAI. PR #17253 preserves true, false, and omitted. Read back the provider-ready request before any tool trusts the output. AgentOps Eval Note. phoenix#17253

Arize Prompt LearningでLLMプロンプトは自己改善できるのか？感情分類と金融ニュース要約で検証してみた はじめに LLMアプリケーションを開発していると、最初に作ったプロンプトだけで期待どおりの出力を得ることは難しく、次のような修正を何度も繰り返すことになります。 - 指示をより具体的にする - 出力フォーマットを固定する - 失敗例をFew-shot Exampleとして追加する - Hallucinationを防ぐルールを追加する - 評価結果を確認して、再びプロンプトを書き直す この作業を人間が毎回行うのではなく、モデルの失敗と自然言語フィードバックを利用して自動化できないかと考えました。 そこで本記事では、Arize AIのprompt-learning SDKを使い、次の2つのタスクでSelf-Improving Promptを実装・検証します。 1. 感情分類プロンプトの改善 2. 金融ニュース要約プロンプトの改善 結論を先に述べると、Prompt Learningは失敗を具体的な指示へ変換する仕組みとして有効でした。一方で、プロンプトを更新すれば必ず性能が上がるわけではありません。特にLLM Evaluatorを使用するタスクでは、評価基準の設計が最適化結果を大きく左右しました。 Prompt Learningとは Prompt Learningはモデルの重みを再学習するのではなく、評価結果を基にプロンプト自体を書き換えるアプローチです。 今回構築したパイプラインは、主に次の3つの役割で構成されています。 Agent 現在のプロンプトを使って分類や要約を実行する ↓ Evaluator Agentの出力を評価し、自然言語フィードバックを生成する ↓ Meta Model 現在のプロンプトとフィードバックから新しいプロンプトを作る 従来の正解・不正解や単純な数値スコアだけでなく、次のような具体的な自然言語フィードバックを利用できる点が特徴です。 The expected emotion was "anger", but the model predicted "sadness". Clarify how to distinguish anger from sadness using the text's central emotional state rather than isolated words. このフィードバックをMeta Modelへ渡すことで、失敗を防ぐための指示やFew-shot Exampleを元のプロンプトへ追加します。 Arize Prompt Learning SDKについて 今回使用したprompt-learningは、Arize AIが公開しているプロンプト最適化SDKです。 SDKがモデルをFine-tuningするわけではありません。現在のプロンプト、モデル出力、自然言語フィードバックをメタプロンプトとして構成し、GPT-4oに改善版プロンプトを生成させます。 インストール 今回使用したバージョンはGitHubリポジトリからインストールしました。 prompt-learning-enhanced @ git+https://github.com/Arize-ai/prompt-learning.git 検証時点では依存パッケージとの互換性を保つため、次の環境を使用しています。 Python 3.12 arize-phoenix-evals==2.7.0 インストール例は次のとおりです。 python3.12 -m venv .venv source .venv/bin/activate pip install -r requirements.txt SDKの基本的な使い方 from optimizer_sdk.prompt_learning_optimizer import ( PromptLearningOptimizer, ) optimizer = PromptLearningOptimizer( prompt=current_prompt, model_choice="gpt-4o", openai_api_key=os.environ["OPENAI_API_KEY"], ) optimized_prompt = optimizer.optimize( dataset=evaluated_data, output_column="output", feedback_columns=["feedback"], ) optimize()へ渡す主要な情報は次の3つです。 | 項目 | 内容 | |---|---| | dataset | 入力、モデル出力、フィードバックを含むDataFrame | | output_column | 現在のプロンプトで生成された出力列 | | feedback_columns | 問題点と改善方法を記述した自然言語フィードバック列 | SDK内部では、おおよそ次の処理が行われます。 現在のプロンプト + 入力データ + モデル出力 + 自然言語フィードバック ↓ Arize Prompt Learning SDK ↓ メタプロンプトを構築 ↓ GPT-4oへプロンプト改善を依頼 ↓ 新しいプロンプトを返す 今回のSDKは、最適化時にmetaprompt.txtも自動生成しました。このファイルには元のプロンプト、各入力、モデル出力、評価フィードバック、Meta Modelへの最終指示が含まれます。 ただし、SDKがSelf-Improving Pipelineのすべてを管理するわけではありません。 | 処理 | 担当 | |---|---| | 分類・要約の生成 | GPT-4o-mini | | フィードバック生成 | PythonロジックまたはEvaluator LLM | | メタプロンプト構築 | Arize Prompt Learning SDK | | 改善プロンプト生成 | SDK経由で呼び出すGPT-4o | | Iteration管理・再評価 | 独自のPythonコード | したがって、Arize Prompt Learningは「自然言語フィードバックを使ったプロンプト書き換えを自動化するSDK」と整理できます。 Self-Improving Pipelineの共通設計 2つの実験では、次のループを共通して使用しました。 1. Hugging Faceからデータを取得 2. 現在のプロンプトでAgentを実行 3. Agentの出力を評価 4. 自然言語フィードバックを生成 5. PromptLearningOptimizer.optimize()を実行 6. 更新されたプロンプトを次のIterationへ適用 なお、各Iterationの結果保存はタスクごとに異なります。金融ニュース要約では、回次別のプロンプト、評価結果CSV、集計スコアを保存します。感情分類では、回次別の結果は保存せず、最終プロンプトのみを保存します。 重要なのは、Optimizerへ数値スコアだけを渡すのではなく、何が問題で、次のプロンプトへどのようなルールを追加すべきかを文章で渡すことです。 実験1：感情分類プロンプトの自己改善 概要 最初の実験では、Hugging Faceのdair-ai/emotionを使用しました。 データセットの整数ラベルを次の文字列へ変換します。 LABELS = ( "sadness", "joy", "love", "anger", "fear", "surprise", ) 今回は動作をすばやく確認するため、先頭の5件で簡易実験を行いました。 初期プロンプト Classify the emotion expressed in the text. Choose exactly one label from: sadness, joy, love, anger, fear, surprise. Return only the lowercase label and no explanation. Text: {text} Label: 推論処理 現在のプロンプトへtextを挿入し、GPT-4o-miniを呼び出します。 response = client.chat.completions.create( model="gpt-4o-mini", temperature=0.0, messages=[ { "role": "user", "content": prompt_template.replace("{text}", text), } ], ) モデルがラベル以外の説明を付けた場合に備え、出力から許可されたラベルを正規化して取得する処理も追加しました。 自然言語フィードバック 感情分類では正解ラベルが存在するため、Evaluator LLMを使わずPythonで正解と予測を比較できます。 if expected == predicted: feedback = ( f"Correct. The response selected '{expected}', " "matching the expected emotion." ) else: feedback = ( f"Incorrect. The expected emotion was '{expected}', " f"but the model predicted '{predicted}'. " f"Clarify how to distinguish '{expected}' from '{predicted}' " "using the text's central emotional state rather than isolated words." ) 1回目の結果 | 入力の概要 | 正解 | 予測 | 結果 | |---|---|---|---| | humiliated | sadness | sadness | Correct | | hopelessからhopefulへ変化 | sadness | joy | Incorrect | | greedy, wrong | anger | sadness | Incorrect | | 暖炉を懐かしむ文章 | love | sadness | Incorrect | | grouchy | anger | anger | Correct | 正解は2件、誤分類は3件で、正解率は40%でした。 主な問題はjoy、anger、loveをsadnessと混同していたことです。モデルは文章全体の感情より、hopeful、wrong、nostalgicといった一部の単語に強く反応していました。 Arizeが生成したメタプロンプト Arize SDKが生成したmetaprompt.txtには、次の情報がまとめられていました。 - 元の分類プロンプト - 5件の入力テキスト - GPT-4o-miniの予測 - 正解・不正解に関する自然言語フィードバック - GPT-4oへ新しいプロンプトを生成させる指示 たとえば、sadnessをjoyと誤分類したケースは次のように渡されます。 LLM Output using baseline prompt: joy feedback: Incorrect. The expected emotion was 'sadness', but the model predicted 'joy'. Clarify how to distinguish 'sadness' from 'joy' using the text's central emotional state rather than isolated words. 改善後のプロンプト Prompt Learningによって、次の重要なルールが追加されました。 Consider the overall emotional tone of the text rather than focusing on isolated words. また、誤分類した文章がFew-shot Exampleとしてプロンプトへ追加されました。 Example: Text: "i can go from feeling so hopeless to so damned hopeful ..." Expected Label: sadness Example: Text: "im grabbing a minute to post i feel greedy wrong" Expected Label: anger Example: Text: "i am ever feeling nostalgic about the fireplace ..." Expected Label: love この結果から、Arize Prompt Learningが自然言語フィードバックを次の2種類の改善へ変換したことが分かります。 1. 一部の単語ではなく、文章全体の感情を判断するという一般ルール 2. 実際に失敗した入力を利用したFew-shot Example 注意点 この簡易実験では、改善後のプロンプトを別の未使用データで評価していません。そのため、Few-shot Exampleとして追加された文章を再び正しく分類できたとしても、それが一般化性能の向上なのか、単なる記憶なのかは判断できません。 本格的に評価する場合は、最適化に使うTrainデータと、最終性能を確認するTestデータを分離する必要があります。 実験2：金融ニュース要約プロンプトの自己改善 概要 2つ目の実験では、金融ニュース記事をGPT-4o-miniで要約し、GPT-4o Evaluatorの自然言語フィードバックからプロンプトを改善しました。 データセットにはHugging Faceのkdave/Indian_Financial_Newsを使用しました。 Content → article Summary → reference_summary このデータセットのSummaryはT5-baseによって生成されたものです。そのため、人間が作成した絶対的な正解ではなく、Evaluatorが比較するための参考要約として扱いました。事実確認では、常に元の記事を最優先します。 評価には先頭の20件を使用しました。 初期プロンプト Summarize the following financial news article: Article: {article} Summary: Evaluatorの設計 要約では完全一致による評価ができないため、GPT-4oをEvaluatorとして使用しました。 評価軸は次の3つです。 1. 重要な財務数値を含んでいるか 2. 不要な説明がなく、簡潔か 3. 元記事に忠実で、Hallucinationがないか EvaluatorにはJSON形式で回答させます。 { "financial_figures_score": 4, "conciseness_score": 5, "faithfulness_score": 5, "feedback": "具体的な評価と改善ルール" } 各スコアは1から5で、3つの平均を100点満点へ変換しました。 score_columns = [ "financial_figures_score", "conciseness_score", "faithfulness_score", ] evaluated["overall_score"] = ( evaluated[score_columns].mean(axis=1) * 20.0 ) 自然言語フィードバックには、単なる評価だけでなく、次のプロンプトへ反映できる具体的なルールを含めるよう指示しています。 Explicitly identify omissions, unnecessary details, and hallucinations. Give concrete prompt rules that would prevent the observed problems. Iterationごとの結果 3回の評価結果は次のとおりです。 | Iteration | 総合点 | 財務数値 | 簡潔性 | 忠実性 | |---|---:|---:|---:|---:| | 1 | 83.67 | 3.85 | 3.95 | 4.75 | | 2 | 83.67 | 4.00 | 3.95 | 4.60 | | 3 | 83.33 | 3.95 | 3.90 | 4.65 | Iteration 1と2の総合点は同じで、Iteration 3ではわずかに低下しました。 Iteration 1から2への変更 最初のフィードバックから、Arize Prompt Learningは次の要素をプロンプトへ追加しました。 - 重要な財務数値を強調する - 経済的要因と主要な影響を含める - 不要な詳細を避ける - 元記事の文脈を維持する - 根拠のない主張や推測を追加しない 生成されたプロンプトの一部は次のとおりです。 Summarize the following financial news article by highlighting key financial figures, significant economic factors, and major impacts or implications. Focus on providing concise and clear information that accurately represents the source content and avoids unnecessary details. 財務数値スコアは3.85から4.00へ改善しました。一方、忠実性は4.75から4.60へ低下したため、総合点は83.67のままでした。 Iteration 2から3への変更 次のIterationでは、金融市場や経済政策の重要な変化を特に重視する指示が追加されました。 Pay special attention to capturing the most significant shifts in financial markets or economic policies. しかし、Iteration 3の総合点は83.33へ低下しました。 なぜ改善しなかったのか 今回の結果から、大きく2つの問題が確認できました。 1. 財務数値を強調しすぎた Iteration 2では、財務数値を含める能力は改善しました。しかし、一部の記事では数値や周辺情報を過剰に列挙し、要約の中心が不明確になりました。 たとえば20番目の記事は、次のように変化しました。 Iteration 1: 66.67 Iteration 2: 46.67 Iteration 3: 73.33 Iteration 2では市場指数だけでなく、複数の企業ニュースや経済要因まで追加し、Evaluatorから「中心的ではない情報が含まれている」と指摘されました。 2. 数値が存在しない記事も低く評価された 一部の記事には売上、利益、成長率などの財務数値が存在しません。それにもかかわらずEvaluatorは、次のような理由で財務数値スコアを低くしました。 financial figures are not present in the source article, so the financial_figures_score is low これは要約モデルの失敗ではなく、評価基準の設計上の問題です。 さらに、プロンプトが財務数値を強調したことで、数値がない記事でも「Key financial figures」という構造を作り、元記事にない解釈や意味づけを追加するケースが発生しました。 19番目の記事は次のように低下しました。 Iteration 1: 66.67 Iteration 2: 66.67 Iteration 3: 40.00 Evaluatorは、具体的な財務数値がないにもかかわらず数値の存在を暗示し、記事から直接確認できない影響まで追加した点を問題として挙げました。 最終プロンプトの選択 実装では、後のプロンプトが必ず良いとは仮定せず、評価済みのプロンプトから平均点が最も高いものを保存します。 if mean_score > best_score: best_score = mean_score best_prompt = current_prompt 今回はIteration 1と2が同点でした。コードは同点の場合に先に登場したプロンプトを維持するため、最終ファイルには初期プロンプトが保存されました。 Summarize the following financial news article: Article: {article} Summary: つまり、今回の金融ニュース要約実験では、最適化後のプロンプトが初期プロンプトを明確に上回ったとは言えません。 2つの実験結果を比較する | 観点 | 感情分類 | 金融ニュース要約 | |---|---|---| | タスク | 単一ラベル分類 | 自由形式の要約生成 | | 評価方法 | 正解ラベルとの完全一致 | GPT-4o Evaluator | | フィードバック生成 | Pythonロジック | Evaluator LLM | | 改善内容 | 全体の感情を見るルールとFew-shot例 | 財務数値・簡潔性・忠実性のルール | | 主な問題 | 同一データの記憶、ラベルの主観性 | Evaluatorの偏り、評価軸のトレードオフ | | 結果 | 具体的な改善プロンプトを生成 | 総合スコアは改善しなかった | 感情分類では、正解ラベルと予測ラベルの差が明確です。そのため「何を間違えたか」を比較的安定してフィードバックへ変換できました。 一方、要約は複数の正しい回答が存在します。Evaluator自身が評価基準を誤って適用すると、その誤ったフィードバックがPrompt Learningを通じて次のプロンプトへ反映されます。 不適切な評価基準 ↓ 偏った自然言語フィードバック ↓ 偏ったメタプロンプト ↓ 過度に最適化された新しいプロンプト この違いから、自由形式の生成タスクではOptimizerより先にEvaluatorを十分に検証する必要があると分かりました。 Prompt Learningが有効だった点 失敗を読みやすいルールへ変換できる 従来のスコアだけでは、なぜ失敗したのか、何を変更すべきかが分かりません。自然言語フィードバックを使うことで、次のような具体的な変更を生成できました。 - 単語単位ではなく文章全体の感情を確認する - 重要な数値を含める - 根拠のない内容を追加しない - 失敗例をFew-shot Exampleとして追加する プロンプト変更の過程を追跡できる 金融ニュース要約では、各Iterationの結果を次の形式で保存しました。 financial_results/ ├── iteration_1_prompt.txt ├── iteration_1_results.csv ├── iteration_2_prompt.txt ├── iteration_2_results.csv ├── iteration_3_prompt.txt ├── iteration_3_results.csv ├── iteration_history.csv ├── iteration_history.json ├── optimization_1_metaprompt.txt └── optimization_2_metaprompt.txt これにより、次の内容を後から確認できます。 - どのフィードバックが生成されたか - プロンプトへどのルールが追加されたか - 各評価軸がどのように変化したか - 特定の記事でなぜスコアが低下したか 性能が低下した場合にロールバックできる 最新のプロンプトを無条件に採用するのではなく、評価済みプロンプトの中から最も高いスコアを持つものを選択できます。 今回の金融ニュース実験では、実際にIteration 3の性能が低下したため、このロールバック設計が重要でした。 Prompt Learningの限界 プロンプト更新は性能向上を保証しない Prompt Learningはフィードバックに沿ってプロンプトを書き換えますが、そのフィードバックが正しいことや、新しいプロンプトが未知データでも良い結果を出すことは保証しません。 Evaluatorの品質が最適化結果を決める 特に自由形式の生成タスクでは、Evaluatorが事実上の目的関数になります。 Evaluatorが「財務数値が存在しない記事」を低く評価すると、Optimizerはすべての記事で数値を強調する方向へ進みます。結果として、不要な数値の列挙やHallucinationを誘発する可能性があります。 評価指標間にはトレードオフがある 金融ニュース実験では、財務数値スコアが上がる一方で忠実性が下がりました。 財務数値を増やす → 情報量は増える → 不要な詳細も増える可能性がある → 簡潔性や忠実性が下がる可能性がある 単一の総合点だけでなく、評価軸ごとの変化を確認することが重要です。 同一データでの最適化と評価は過学習を招く 最適化に使った入力をそのまま再評価すると、特定例をFew-shotとして記憶しただけでもスコアが上がります。 実用的な検証では、最低でも次のように分離する必要があります。 Train フィードバック生成とプロンプト最適化に使用 Validation Iterationごとのプロンプト選択に使用 Test 最後に一度だけ一般化性能を確認 まとめ 本記事では、Arize Prompt LearningとOpenAI APIを利用し、感情分類と金融ニュース要約の2つのSelf-Improving Pipelineを構築しました。 感情分類では、自然言語フィードバックから「単語ではなく文章全体の感情を見る」というルールと、実際の失敗例を使ったFew-shot Exampleが自動的に追加されました。 金融ニュース要約では、財務数値を重視するプロンプトへ改善されましたが、忠実性とのトレードオフが発生し、最終的な総合スコアは向上しませんでした。 この結果から得た最も重要な知見は、次のとおりです。 > Prompt Learningはプロンプトを必ず改善する魔法の仕組みではない。しかし、LLMの失敗を自然言語で分析し、その分析を次のプロンプトへ反映するループを自動化できる点は非常に有用である。 そして、Self-Improving Pipelineの品質を決めるのはOptimizerだけではありません。 良いデータ + 良いEvaluator + 具体的な自然言語フィードバック + 独立した検証データ = 信頼できるPrompt Improvement 特に自由形式の生成タスクでは、プロンプトを最適化する前にEvaluatorが人間の判断と整合しているかを検証することが重要です。 Self-Improving Pipelineはどのような課題に活用できるか 今回扱った感情分類やニュース要約以外にも、出力の良し悪しを一定の基準で評価でき、フィードバックを次の試行へ反映できる課題であれば、Self-Improving Pipelineを活用できると思います。 1. カスタマーサポートの回答改善 問い合わせ、生成された回答、オペレーターによる修正内容を学習ループへ渡すことで、次のような観点からプロンプトを改善できます。 - 質問に対して直接回答しているか - 社内ポリシーや製品仕様と一致しているか - 不要に長い説明や不適切な表現がないか - 人間のオペレーターへ引き継ぐべきケースを正しく判断しているか たとえば「返金条件の説明が不足している」というフィードバックが繰り返し発生した場合、その失敗パターンをルールやFew-shot Exampleとして回答プロンプトへ反映できます。 2. RAGシステムの回答品質改善 社内文書やFAQを検索して回答するRAGでは、回答の正確性だけでなく、取得した根拠を正しく利用しているかも評価できます。 ユーザーの質問 ↓ 検索された文書 ↓ LLMの回答 ↓ 根拠との整合性・引用・回答漏れを評価 ↓ 回答プロンプトを改善 この仕組みにより、「検索文書にない内容を推測しない」「回答の根拠を明示する」「情報がない場合は分からないと答える」といったルールを、実際の失敗例から具体化できます。ただし、検索結果そのものが不適切な場合は、プロンプトだけでは解決できないため、Retrieverの評価と分けて考える必要があります。 3. 文書からの情報抽出 請求書、契約書、履歴書、問い合わせメールなどから構造化データを抽出するタスクにも適しています。 - 必須項目の抽出漏れ - 日付・金額・単位の誤認識 - JSON Schema違反 - 原文にない値の補完 これらはルールベースで評価しやすく、失敗理由も明確です。そのため、自由形式の文章生成よりも、改善前後の効果を客観的に測定しやすいユースケースです。 4. コンテンツ生成とブランドトーンの統一 商品説明、メール、SNS投稿、レポートなどでは、事実性に加えて文体やブランドガイドラインへの適合性も評価できます。編集者の修正履歴を自然言語フィードバックへ変換すれば、「専門用語を減らす」「断定しすぎない」「対象読者に合わせて説明する」といった組織固有のルールをプロンプトへ蓄積できます。 ただし、単一のEvaluatorへすべての判断を任せると、文章が画一的になったり、長さと情報量のような評価基準が競合したりします。事実性、読みやすさ、トーンなどを個別に評価し、優先順位を決める設計が必要です。 5. エージェントのツール選択と業務フロー改善 複数のツールを使用するLLM Agentでは、最終回答だけでなく、途中の行動も評価対象にできます。 - 適切なツールを選択したか - 必要な引数を正しく渡したか - 不要なAPI呼び出しをしていないか - 実行前にユーザー確認が必要な操作を判別したか - 失敗時に安全な代替手段を選んだか 成功・失敗の実行履歴から、ツール選択条件や停止条件をプロンプトへ追加することで、Agentの再現性と安全性を高められます。 参考リンク - Arize AI Prompt Learning GitHub Repository - Hugging Face: dair-ai/emotion - Hugging Face: kdave/Indian_Financial_News - OpenAI API Documentation

@kanikairl For the evals/observability side, have a look at Phoenix: Arize-ai/phoenix I'd start with its tracing quickstart, then add a small eval dataset

Atomic Agents 2.11 / 2.12: native Arize Phoenix tracing, plus the October MCP fixes Two releases today. 2.11.0: Phoenix tracing. Every agent run and every tool call now shows up as a proper OpenInference span (AGENT / TOOL) with the LLM calls nested underneath. Works with Arize Phoenix or any OpenTelemetry backend. Nothing changes in your agent code: pip install "atomic-agents[observability]" Then register(auto_instrument=True) from phoenix.otel picks it up through the entry point. - Observability guide - Runnable example 2.12.0: MCP fixes. - Paginated discovery of tools, resources and prompts. - Server error results (isError) raise a RuntimeError instead of getting parsed as output. - Schema refs to non-object definitions resolve. - The YouTube transcript tool works with youtube-transcript-api 1.x again. Thanks to pei711 for the error-result groundwork in #295. Release notes: v2.11.0 and v2.12.0 Also opened a PR to get Atomic Agents listed in the official Phoenix docs. Should land soon.

Updates on Trueup, building with @neatlogs Arize's head of open source reviewed the fix Trueup sent them: "the right goal." The bug: an AI agent run crashes, the record says OK. We applied his changes. 300 tests pass. openinference#4024 #neatHack @EntireHQ @arithecfo t.co

Tracing AI Agent Execution: Observability and Evidence in Distributed Systems --- title: Tracing AI Agent Execution: Observability and Evidence in Distributed Systems published: true description: A technical edition on traces, spans, evaluation, and governed evidence for distributed AI agents. tags: ai, observability, opentelemetry, multiagent canonical_url: aridiosilva.com --- > Technical edition of the same scholarly work. The content remains under author review; DOI and external URLs are pending. Contents - [Abstract](#abstract) - [1. Introduction](#introduction) - [2. Foundations: trace, span, and session](#foundations) - [3. Observability questions and their signals](#observability-questions) - [4. Method, sources, and limitations](#method) - [5. Open standard: OpenTelemetry GenAI](#opentelemetry-genai) - [6. Vendor platforms](#vendor-platforms) - [7. Independent platforms](#independent-platforms) - [8. Comparative analysis](#comparative-analysis) - [9. Discussion: gaps, risks, and engineering practices](#discussion) - [10. Conceptual implications for SGAEIA in distributed edge environments](#sgaeia-implications) - [11. Conclusion](#conclusion) - [Glossary](#glossary) - [Bibliography / References](#references) - [About the Author](#about-the-author) - [Research & Project Resources](#resources) - [Figures](#figures) - [License](#license) - [Series continuity](#series-continuity) - [Suggested citation](#suggested-citation) Observability, evaluation, and evidence for governing autonomous agents at the edge SGAEIA Research Series — Article 18 Aridio Silva Independent Researcher Creator of SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture ORCID: 0009-0008-2411-6995 Copyright: © 2026 Aridio Silva | License: CC BY 4.0 <a name="abstract"></a> Abstract As of October 2026, the major AI-agent platforms can record the individual steps of an execution, yet the ecosystem still lacks stable and uniform semantics. This paper combines three contributions: a conceptual framework for agent-observability questions; a comparative review of 13 platforms, with package versions checked on 10 October 2026; and a non-normative interpretation of the implications for SGAEIA in distributed multi-agent and edge environments. The results show that a trace reconstructs the path of execution but does not demonstrate the correctness of the outcome; that essential capabilities remain in beta, preview, or Development status; and that divergent content-capture defaults create direct privacy, governance, and evidence-continuity risks. The central conclusion is that tracing, evaluation, and governed evidence must operate as complementary functions: observing execution, judging outcomes, and preserving a verifiable basis for audit and assurance. Keywords: AI; AI agents; observability; distributed tracing; OpenTelemetry; spans; agent evaluation; multi-agent systems; edge AI; SGAEIA. <a name="introduction"></a> 1. Introduction When an AI agent is activated, operators need to know what it actually did: whether execution ended normally, where errors or deviations occurred, how long and how much it cost, which tools were invoked, and which agents received control. Agents choose tools and delegation paths at runtime, so their behavior is non-deterministic and cannot be reconstructed reliably from flat application logs alone. Engineering practice therefore adapts distributed tracing from microservice systems to the agent lifecycle. This paper addresses four research questions: 1. RQ1. How do traces and spans apply to the execution of one or more AI agents? 2. RQ2. Which signals answer questions about completion, error, deviation, duration, cost, and executed steps? 3. RQ3. Which platforms offer these capabilities, at what level of maturity, and with which strengths and limitations? 4. RQ4. How do tracing, evaluation, and evidence preservation relate to governance of distributed edge agents from the conceptual perspective of SGAEIA? The scope covers agent runtimes from Anthropic, OpenAI, Google, Microsoft, and AWS; seven independent observability platforms; and the OpenTelemetry standard. The language model alone does not create this operational record. Tracing is produced by the execution layer: the SDK, framework, orchestration environment, or hosting platform surrounding the model. <a name="foundations"></a> 2. Foundations: trace, span, and session A span is the basic unit of traced work. Each relevant operation can produce its own span, normally including start and end times, parent relationship, status, and operation-specific attributes. Common examples include a model call, a tool invocation, and a handoff or nested agent invocation. A trace groups causally related spans into the end-to-end journey of an operation, while a session or thread may group multiple traces that belong to the same conversation or longer-running task. In the OpenAI Agents SDK, spans expose fields such as started_at, ended_at, parent_id, and step-specific data [R4]. AWS AgentCore similarly records status and events [R15]. The vocabulary is not fully uniform: one platform may define a trace as a complete operation, another as one request-response cycle, and a third may use a session to provide the longer boundary. 2.1 Three nuances that change trace interpretation First, a trace is not always the entire task. OpenAI can connect several executions through a group_id [R4], whereas AgentCore treats a trace as a request-response cycle and groups traces into a session [R15]. A reviewer must therefore identify the platform's execution boundary before comparing duration, cost, or completion. Second, a handoff is not always represented by a dedicated span. OpenAI defines a handoff span [R4], while OpenTelemetry defines create_agent, invoke_agent, and execute_tool, allowing a subagent to appear as a nested invoke_agent operation [R18, R19]. Microsoft has also documented semantics for communication between agents [R11, R44]. Third, agent traces may include more than model, tool, and handoff spans. Guardrail checks, turns, permission decisions, and time spent awaiting human approval can be observable operations. The Claude Agent SDK, for example, can record a child span for a tool waiting on user authorization [R1]. 2.2 Tree structure Spans form a tree rather than a flat list. Figure 1 represents the transition from a session to traces and nested spans, including model calls, tools, and subagents. The hierarchy supports latency analysis, causal reconstruction, and failure localization without reducing the execution to an undifferentiated stream of messages. !Figure 1 — Session, trace, and span hierarchy Figure 1 — From a session to verifiable spans. A session can group multiple traces; each trace organizes agent, model, tool, and handoff spans into a causal structure. © 2026 Aridio Silva | Project SGAEIA | CC BY 4.0. <a name="observability-questions"></a> 3. Observability questions and their signals The main analytical distinction is between technical completion and substantive success. A run can terminate without an exception while still producing an incorrect answer, violating an instruction, or acting outside the intended scope. A trace reports operational facts; a separate evaluator, test, schema validator, or human review must judge the result. | Question | Signal to inspect | Interpretation limit | |---|---|---| | Did the task complete? | Final run status plus independent outcome validation | No exception proves only that execution did not crash | | Were there errors or deviations? | Error spans, failed tools, exceeded limits, denied permissions, guardrail events | Deviation requires comparison with expected behavior | | When did it finish and how long did it take? | Run and span duration; tokens and derived cost | Boundaries vary by platform | | What happened at each step? | Hierarchical trace with tool, arguments, and result | Content capture may be disabled or redacted | | Who requested and authorized the action? | End-user attributes and permission-decision events | Identity does not itself establish authority | | Was there a loop or waste? | Turn count, tool calls, and inference calls | Thresholds depend on task and policy | 3.1 Completion and outcome quality The Claude Agent SDK result can report termination subtype, error state, number of turns, duration, and cost, including outcomes such as success, maximum-turn termination, and execution error [R36]. These fields describe how execution stopped; they do not establish that the requested objective was achieved. Semantic failures can therefore remain invisible to ordinary error status [R37]. OpenAI's trace grading illustrates the complementary approach of evaluating traces and outcomes at scale [R5]. 3.2 Errors, deviations, and auditability Errors appear in span status, API error events, and failed tool results. Deviations require broader evidence: permission events, guardrail decisions, expected-path comparison, and independent observation of resulting state. Claude Code permission decisions can be exported as structured events associated with the end user, providing input to an audit or SIEM workflow [R2]. <a name="method"></a> 4. Method, sources, and limitations The review was conducted on 10 October 2026 using three source categories: package-registry data from PyPI and npm [R40]; official vendor and platform documentation; and secondary technical sources only where official documentation did not cover the relevant point. Secondary and competitor-produced comparisons are identified and treated cautiously. Package versions refer to the stable release visible in the registry on the consultation date. In the comparison matrix, ✔ indicates a capability confirmed in the consulted documentation, ◐ indicates partial or conditional support, and — means that the capability was not found in the reviewed material. Absence from the matrix is not proof that a feature does not exist. SaaS platforms do not have a single platform version, so the version column reports the associated SDK when applicable. This is a structured comparative review, not a systematic review or experimental benchmark. The work does not measure instrumentation overhead, reproduce vendor evaluations, or establish production effectiveness. Platform behavior, prices, retention policies, and maturity labels can change after the stated date; adoption decisions must revalidate the relevant documentation. <a name="opentelemetry-genai"></a> 5. Open standard: OpenTelemetry GenAI OpenTelemetry GenAI semantic conventions are the leading candidate for a common language across runtimes and backends, but they remain under active development [R18–R20]. The defined operations include create_agent, invoke_agent, and execute_tool; attributes cover agent and tool identity; and metrics include agent-run duration, tool-call count, and inference-call count [R18, R19]. Development status means that names and behavior can still change between versions. The practical value is portability. Datadog accepts traces compatible with OpenTelemetry GenAI 1.37 or later [R28], MLflow can ingest and export the conventions [R33], and Google ADK implements them natively [R7]. Portability is nevertheless incomplete because vendor SDKs may emit proprietary span types, require a distribution such as ADOT, or depend on third-party bridges. <a name="vendor-platforms"></a> 6. Vendor platforms 6.1 Anthropic: Claude Agent SDK, Claude Code, and Managed Agents The Claude Agent SDK runs Claude Code as a child process and can export spans for interactions, model calls, tools, tokens, cost, permission decisions, and nested subagents [R1, R2]. Its main strength is OTLP export without a mandatory backend and content-disabled-by-default behavior for read/write payloads. Limitations include beta span names, silent export failures by default, and extra beta configuration for hook spans; Managed Agents also uses a beta API header [R1, R3]. 6.2 OpenAI Agents SDK OpenAI tracing is enabled by default and includes run, task, turn, agent, generation, tool, guardrail, handoff, and audio spans [R4]. The platform connects traces through group_id and integrates trace grading and agent evaluations [R5, R6]. Its strengths are low-friction instrumentation and rich agent-specific semantics; limitations include incompatibility with Zero Data Retention tracing, default capture of inputs and outputs, batch export behavior, and dependence on third-party tooling for an OpenTelemetry bridge [R4, R26]. 6.3 Google ADK and Vertex AI Agent Engine Google ADK implements OpenTelemetry GenAI semantics, with an agent execution as the root span and child spans for model and tool operations [R7, R8]. Agent Engine sends traces to Cloud Trace and exposes response times and executed operations [R9]. The design benefits from native OpenTelemetry support and managed visualization, while Cloud Logging size limits and the lack of a documented integrated trace-evaluation workflow remain material constraints [R9, R10]. 6.4 Microsoft Foundry and Agent Framework Microsoft provides generally available tracing for prompt and hosted agents and preview support for workflows and external agents, storing telemetry in Application Insights [R11–R14]. Its multi-agent semantics and Azure Monitor integration are notable strengths. Costs, retention, access-role requirements, and differing maturity labels across documentation require careful operational review; the evaluation-related comparison includes an explicitly identified third-party source [R39]. 6.5 AWS Bedrock AgentCore AgentCore organizes observability around sessions, traces, and spans, including tool inputs, outputs, timing, and errors [R15]. Unified per-agent log groups can combine spans, prompts, and logs under IAM and customer-managed encryption controls [R17]. Agent spans require ADOT instrumentation, CloudWatch Transaction Search must be configured, and the reviewed documentation did not establish integrated outcome-quality evaluation [R15, R16]. <a name="independent-platforms"></a> 7. Independent platforms 7.1 LangSmith LangSmith treats runs as spans and groups them into projects and threads. It supports OpenTelemetry ingestion, distributed tracing, sampling, sensitive-data controls, token cost, dashboards, and online evaluation [R21–R23]. Relevant limitations include per-trace run limits, Python-only integration for the OpenAI Agents SDK, and enterprise conditions around self-hosting reported by a third party [R41]. 7.2 Langfuse Langfuse offers an MIT-licensed core, self-hosting, nested observations, sessions, agent graphs, human annotation queues, and OTLP-oriented ingestion [R24, R25, R45]. Open source removes license fees but not the operational cost of running and securing the platform [R38]. 7.3 Arize Phoenix and AX Phoenix uses OpenTelemetry and OpenInference semantics for LLM, Tool, Agent, and Retriever spans, and includes evaluations, datasets, experiments, and prompt management [R26, R27]. The Elastic License 2.0 permits broad internal use but limits offering the software as a managed service, while some TypeScript evaluation components remain early-stage [R26, R46]. 7.4 Datadog Agent Observability Datadog accepts OpenTelemetry GenAI traces and uses span links to build an execution graph for multi-agent pipelines [R28, R29]. It monitors latency, error rate, cost, and decisions with automatic instrumentation for selected frameworks. Cross-trace links may be stored without being drawn, and the offering remains SaaS-only in the reviewed scope. 7.5 Braintrust Braintrust combines production logs with offline and online evaluation and integrates with Claude, OpenAI, and Google agent SDKs [R30–R32]. Its self-hosting model is hybrid and its documentation describes telemetry sent to the vendor control plane, which is relevant to data-governance analysis. A third-party vendor profile used only as supplementary context is identified separately [R42]. 7.6 MLflow Tracing MLflow provides an open-source tracing layer compatible with OpenTelemetry, including ingestion and export of GenAI conventions and more than 30 automatic integrations [R33, R34]. Autologging must be enabled explicitly for each integration, which makes configuration discipline part of the assurance boundary. 7.7 Pydantic Logfire Logfire builds on OpenTelemetry with Python and JavaScript SDKs and integrates with Pydantic AI [R35, R43, R47]. Evaluation and prompt-management claims were not fully confirmed on an opened official page in the underlying review, so evaluation remains marked as conditional. Self-hosting is limited to enterprise arrangements [R43]. <a name="comparative-analysis"></a> 8. Comparative analysis 8.1 Versions and maturity on 10 October 2026 | Platform | Version reported | Maturity note | |---|---|---| | Claude Agent SDK / Claude Code | py 0.2.165; npm 0.3.296; CLI 2.1.296 | Tracing beta; 0.x SDKs | | Claude Managed Agents | API | Beta header | | OpenAI Agents SDK | py 0.23.1; JS 0.20.0 | Tracing enabled by default; 0.x SDKs | | Google ADK | py 2.11.0; JS 2.2.1 | 2.x; native telemetry | | Microsoft Agent Framework | py 1.21.0; azure-ai-projects 2.8.0 | GA and preview scopes differ | | AWS Bedrock AgentCore | py 1.24.1 | Rapidly evolving managed service | | LangSmith | py 0.14.7; JS 0.10.11 | Commercial SaaS; 0.x SDKs | | Langfuse | py 4.17.0; JS tracing 5.13.1 | OTLP-first v4 platform | | Arize Phoenix | 20.20.0 | Frequent releases; ELv2 | | Datadog | ddtrace 4.15.6 | SaaS | | Braintrust | py 0.45.0; JS 3.37.2 | SaaS and hybrid | | MLflow Tracing | 3.17.0 | Open source | | Pydantic Logfire | 5.1.1 | Commercial | | OpenTelemetry Python SDK | 1.45.1 | GenAI conventions in Development | 8.2 Capability matrix | Platform | LLM/tool steps | Multi-agent | OTel | Sessions/threads | Trace evaluation | Hosting | |---|---:|---:|---:|---:|---:|---| | Claude Agent SDK | ✔ beta | ✔ subagents | ✔ OTLP | ✔ | — | Backend of choice | | Claude Managed Agents | ✔ console | — | — | ✔ | — | Managed | | OpenAI Agents SDK | ✔ | ✔ handoffs | ◐ third party | ✔ | ✔ | OpenAI dashboard | | Google ADK / Agent Engine | ✔ | ◐ | ✔ native | — | — | Cloud Trace | | Microsoft Foundry / Agent Framework | ✔ | ✔ | ✔ | — | ◐ | Application Insights | | AWS AgentCore | ✔ with ADOT | ✔ | ✔ ADOT | ✔ | — | CloudWatch | | LangSmith | ✔ | ✔ | ✔ | ✔ | ✔ | SaaS / enterprise | | Langfuse | ✔ | ✔ graph | ✔ | ✔ | ✔ | SaaS / MIT self-host | | Phoenix / AX | ✔ | ✔ | ✔ | — | ✔ | ELv2 / SaaS | | Datadog | ✔ | ✔ graph | ✔ | — | ◐ | SaaS | | Braintrust | ✔ | ✔ | ✔ | — | ✔ | SaaS / hybrid | | MLflow | ✔ | ✔ | ✔ | — | — | OSS / managed | | Logfire | ✔ | ✔ | ✔ | — | ◐ | SaaS / enterprise | The symbols reflect only the reviewed documentation. A dash means “not found in the reviewed sources,” not “the feature does not exist.” <a name="discussion"></a> 9. Discussion: gaps, risks, and engineering practices The ecosystem is converging toward OpenTelemetry but still exposes incompatible dialects. Claude emits claude_code.* spans, OpenAI uses a native format with third-party bridges, Google and Microsoft follow GenAI conventions more directly, and AWS depends on ADOT. Multi-vendor deployments therefore need an OpenTelemetry collector or another normalization layer, together with explicit version management for semantic changes. Maturity is uneven. Several high-value capabilities remain beta, preview, or Development, and central SDKs still use pre-1.0 numbering. Long-lived deployments should expect schema migration and avoid binding assurance claims to unstable attribute names. Privacy risk is architectural rather than incidental. Claude does not capture read/write content by default [R1], while OpenAI tracing captures inputs and outputs by default [R4]. Prompts, tool arguments, retrieved records, and outputs can contain personal data, secrets, or regulated information, so collection, redaction, access control, retention, and deletion rules must be decided before production. Telemetry can fail silently. When export failure does not interrupt agent execution [R1], an absent trace can mean “nothing happened” or “evidence was lost.” Monitoring the telemetry pipeline is therefore part of observability itself. A defensible deployment should track span volume, exporter health, queue loss, clock quality, and gaps between agent activity and recorded evidence. Finally, a trace is not an evaluation. The platforms that connect trace data to grading or evaluation cover both operational reconstruction and outcome analysis; others require an independent evaluator. Practical adoption should separate “execution ended without error” from “the result was correct and authorized,” define capture policy explicitly, monitor telemetry export, and group related traces through a stable session boundary. <a name="sgaeia-implications"></a> 10. Conceptual implications for SGAEIA in distributed edge environments 10.1 Observability is not governance From the SGAEIA perspective, tracing is an observation and reconstruction capability, not authorization to act and not automatic proof of compliance. A trace can show that an agent invoked a tool, transferred control, or produced an output, but legitimacy depends on authority, context, and policy that cannot be inferred from telemetry alone. This preserves the distinction that technical capability, authenticated identity, and stated intent do not equal operational authority. 10.2 Evidence must survive distribution Across edge-cloud environments, execution can cross devices, services, administrative domains, and periods of intermittent connectivity. Tracing is useful only if context propagates coherently, identifiers and time information remain sufficiently reliable, retention is proportionate to risk, and evidence loss can be distinguished from lack of activity. The conceptual requirement is not to capture everything; it is to preserve enough governed evidence to reconstruct decisions and effects without turning the observability plane into another disclosure channel. !Figure 2 — Tracing, evaluation, and evidence in SGAEIA Figure 2 — Tracing, evaluation, and evidence in SGAEIA. The trace records the path; evaluation examines outcome quality and conformity; governed evidence supports audit and assurance under explicit assumptions. This is a non-normative conceptual interpretation. © 2026 Aridio Silva | Project SGAEIA | CC BY 4.0. 10.3 Outcome correctness requires an independent signal The most important finding for autonomous systems is that lack of an exception does not show that the objective was achieved. A run can finish cleanly while producing incorrect content, choosing an unsuitable path, or acting beyond legitimate scope. Assurance must therefore separate at least two axes: execution integrity and outcome quality or conformity, preventing one green status from hiding a semantic failure. !IFigure 3 — A trace is not correctness Figure 3 — A trace is not correctness. Technical execution status and outcome quality are independent dimensions; only the combination of tracing and evaluation distinguishes operational completion from substantive success. © 2026 Aridio Silva | Project SGAEIA | CC BY 4.0. 10.4 Research and validation requirements This interpretation suggests research questions for the governed evolution of SGAEIA, but it does not automatically promote controls or components into normative architecture. Future work should evaluate context continuity across agents and domains, resistance of the evidence chain to suppression and tampering, offline and degraded operation, the effects of sampling and redaction on auditability, and the relation between local events and distributed trajectories. Any requirement, invariant, or mechanism derived from these findings must pass the Research-to-Architecture process and be validated before supporting an assurance statement. <a name="conclusion"></a> 11. Conclusion Detailed tracing of agent execution is feasible across every platform reviewed, using spans for model calls, tools, handoffs, and related operations, arranged hierarchically within traces and sometimes grouped into sessions. The field is nevertheless in transition: Claude Agent SDK tracing and Managed Agents remain beta, Foundry workflows and external-agent tracing include preview scopes, OpenTelemetry GenAI conventions remain in Development, and several major SDKs retain 0.x versions. Adopters should expect migration cost and semantic change. The research questions can be answered as follows. For RQ1, distributed-tracing concepts apply directly to agent systems, with material variation in naming and boundaries. For RQ2, trace signals answer questions about completion, errors, duration, and executed steps, but outcome correctness requires independent evaluation. For RQ3, the ecosystem is broad but uneven in portability, privacy defaults, maturity, evaluation, and hosting. For RQ4, SGAEIA should treat tracing as a governed source of evidence that complements rather than replaces authority, evaluation, validation, and assurance. Future work includes empirical measurement of instrumentation overhead, direct product verification of capabilities marked as absent from the reviewed documentation, longitudinal monitoring of OpenTelemetry GenAI stabilization, and experiments on evidence continuity across intermittent edge environments. <a name="glossary"></a> Glossary - Agent: a system that uses a model to decide at runtime which tools or agents to invoke. - Evaluation: code-, model-, or human-based assessment of outcome quality. - Handoff: transfer of control from one agent to another. - OpenTelemetry (OTel): an open standard for producing and exporting traces, metrics, and logs. - OTLP: the OpenTelemetry protocol for telemetry transport. - Session/thread: a grouping of traces belonging to one conversation or extended task. - Span: a timed, attributed unit of traced work with a parent relationship and status. - Trace: the end-to-end record of an operation, composed of spans. - Trace grading: assigning scores to agent traces to identify failures and regressions. - ZDR: Zero Data Retention; OpenAI Agents SDK tracing is unavailable under ZDR. <a name="references"></a> Bibliography / References All sources were consulted on 10 October 2026. Validation markers from the source study are preserved: [A] page opened in full; [B] page recovered through search and matched to the claim; [C] registry API data. Third-party sources are identified explicitly. Vendors and standards - R1. Anthropic. Observability with OpenTelemetry (Agent SDK). [A] - R2. Anthropic. Monitoring (Claude Code). [A] - R3. Anthropic. Managed Agents: observability. [B] - R4. OpenAI. Tracing (Agents SDK). [A] - R5. OpenAI. Trace grading. [B] - R6. OpenAI. Evaluate agent workflows. [B] - R7. Google. Agent activity traces (ADK). [B] - R8. Google. Observability for agents (ADK). [B] - R9. Google Cloud. Trace an agent (Agent Engine). [B] - R10. Google Cloud. Observability for AI agent developers. [B] - R11. Microsoft. Trace agent concept (Foundry). [B] - R12. Microsoft. Set up tracing in Foundry. [B] - R13. Microsoft. Quickstart: tracing a hosted agent. [B] - R14. Microsoft. Enable observability for agents (Agent Framework). [B] - R15. AWS. Observability for agentic resources in AgentCore. [A] - R16. AWS. Add observability to AgentCore resources. [B] - R17. AWS. Amazon Bedrock AgentCore now delivers unified observability with traces and logs in a single log group (23 July 2026). [B] - R18. OpenTelemetry. OpenTelemetry GenAI Semantic Conventions. Official repository to which the former agent-spans documentation now points. [A] - R19. Dash0 (third party). OpenTelemetry GenAI semantic conventions explained. [B] - R20. DEV Community (third party). OpenTelemetry's GenAI semantic conventions are not stable yet. [B] - R44. Microsoft. Tracing in Microsoft Foundry (earlier page mentioning Outshift). [B] Independent platforms - R21. LangChain. Observability concepts (LangSmith). [B] - R22. LangChain. Trace with OpenTelemetry (LangSmith). [B] - R23. LangChain. Observability how-to guides (LangSmith). [B] - R24. Langfuse. Engineering clarifications. [B] - R25. Langfuse. Data model. [B] - R26. Arize. Phoenix repository. [B] - R27. Atlan (third party). What is Arize. [B] - R28. Datadog. OpenTelemetry instrumentation. [B] - R29. Datadog. Agent monitoring. [B] - R30. Braintrust. Self-hosting. [B] - R31. Braintrust. Release notes. [B] - R32. Braintrust. Glossary. [B] - R33. MLflow. Tracing for LLM and agent observability. [B] - R34. Databricks. Automatic tracing and integrations. [B] - R35. Pydantic. Logfire: AI observability. [B] - R45. Langfuse. AI SDK C++ integration. [B] - R47. Pydantic. Pydantic AI: Logfire. [B] Third-party sources and version data - R36. Inference.net (third party). Claude Agent SDK tracing and evaluation in production. [B] - R37. ClickHouse (third party). What is AI agent observability. [B] - R38. BenchLM (third party). Best LLM observability tools. [B] - R39. Jannik Reinhard (third party). Microsoft Foundry observability. [B] - R40. PyPI and npm registries, accessed through API on 10 October 2026 for version and publication dates. [C] - R41. Latitude (third party). LLM observability tools compared. [B] - R42. RFP.wiki (third party). Braintrust. [B] - R43. Pydantic. Logfire FAQ. [B] - R46. Inference.net (third party). Arize Phoenix alternatives for agent observability. [B] <a name="about-the-author"></a> About the Author Aridio Silva is an independent researcher based in Brazil working on the architecture, security, governance, and trustworthiness of autonomous and distributed artificial intelligence systems. His research focuses on Agentic AI, Multi-Agent Systems, Edge AI, AI Security, Zero Trust, Security-by-Design, AI Governance, Spec-Driven Development, and continuous security assurance. He is the creator and lead researcher of SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture, a research initiative investigating architectural foundations for secure, governed, auditable, and trustworthy autonomous AI systems operating across distributed edge-cloud environments. <a name="resources"></a> Research & Project Resources 1. ORCID: orcid.org 2. Google Scholar: scholar.google.com 3. Zenodo — SGAEIA Community: zenodo.org 4. OpenAIRE: explore.openaire.eu 5. Medium: medium.com 6. DEV Community: dev.to 7. GitHub: github.com/aridiosilva 8. LinkedIn: linkedin.com 9. Homepage: aridiosilva.com 10. SGAEIA Homepage: aridiosilva.com 11. SGAEIA LinkedIn: linkedin.com <a name="figures"></a> Figures The cover is not numbered. Figures 1–3 are public conceptual illustrations and do not disclose private SGAEIA protocols, algorithms, state machines, policy structures, thresholds, or enforcement mechanisms. Final assets must include visible credit and embedded creator, copyright, project, title, description, and license metadata. <a name="license"></a> License Except where otherwise noted, the text and original conceptual illustrations in this article are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). © 2026 Aridio Silva. You may share and adapt this work for any purpose, provided appropriate attribution is given. The SGAEIA software and research artifact remain subject to their own Apache License 2.0. Autonomous AI. Governed by Design. Trusted by Evidence. <a name="series-continuity"></a> Series continuity This work is SGAEIA Research Series — Article 18. The canonical homepage record, Zenodo deposit, DOI, Medium URL, and DEV Community URL remain pending until external publication and author verification. <a name="suggested-citation"></a> Suggested citation Silva, Aridio. (2026). Tracing AI Agent Execution: Foundations, State of the Art, and Implications for Distributed Multi-Agent Systems. SGAEIA Research Series, Article 18. CC BY 4.0. Canonical URL and DOI pending.