ai

AI SDK 7.0.160 Started Enforcing dataPartSchemas. Your Loading States Might Now Throw If you pass dataPartSchemas to useChat and bump ai to 7.0.160 or later, a chat that worked yesterday can end up in status: 'error' with a TypeValidationError. You didn't change anything. Your schemas just started running. What changed useChat and Chat accept dataPartSchemas, keyed by the bare part name (weather for a data-weather part). The stream processor looked the schema up by the full chunk type (data-weather), so the lookup always missed and streamed data parts were never validated. That's issue #24241, fixed in PR #24298 and shipped in ai 7.0.160: "Invalid data parts now raise a TypeValidationError." So a patch release turned on a check that was configured but never ran. What I ran I didn't want to guess at the edge cases, so I drove the Chat class from @ai-sdk/react with a fake transport that replays fixed UI message chunks. No model, no network. useChat runs on that same class (its chat option even takes an existing Chat instance), so the stream handling is the same code path. I ran the same script against ai 7.0.159 with @ai-sdk/react 4.0.163, ai 7.0.162 with @ai-sdk/react 4.0.166, and ai 6.0.322 with @ai-sdk/react 3.0.325, all with zod 4.6.7 on Node 20.19.2. | Stream | 7.0.159 | 7.0.162 | | --- | --- | --- | | Valid data-weather part | ready | ready | | city: 42, then some text | ready, bad data kept in parts, text arrives | error, no assistant message at all | | Loading update without weather, then the final value | ready | error, earlier text kept, final value lost | | Transient part that fails the schema | ready, onData called | error, onData not called | | data-notice part with no schema | ready | ready | On 6.0.322 every row matched the 7.0.159 column: bad data went through without an error. The one that'll bite: progressive updates The streaming data docs use a progressive pattern: write { city, status: 'loading' } first, then write to the same id again with the real value. The docs type weather as optional (weather?: string), which fits that pattern. But if your zod schema was written for the finished shape, the loading chunk fails now. Here's the excerpt from my test script (start and end are the start/finish chunks, text() writes one text part, and run() sends a message through Chat with these chunks): const strictSchemas = { weather: z.object({ city: z.string(), weather: z.string(), status: z.enum(['loading', 'success']), }), }; // Case 3: a loading update, then the final value under the same id await run('3 loading update without weather, then final', [...start, ...text('t0', 'Checking...'), { type: 'data-weather', id: 'w1', data: { city: 'Berlin', status: 'loading' } }, { type: 'data-weather', id: 'w1', data: { city: 'Berlin', weather: 'sunny', status: 'success' } }, ...end]); On 7.0.162 this ended with Type validation failed for message.parts[2].data (data-weather, id: "w1"). The "Checking..." text that streamed before it stayed in the message, the final value never arrived, and chat.status was error. Every chunk is checked, not just the last one per id. Transient parts too, even though they never land in message.parts. The fix Make the schema describe every state the server actually sends. A discriminated union keeps the final shape strict: const unionSchemas = { weather: z.discriminatedUnion('status', [ z.object({ status: z.literal('loading'), city: z.string() }), z.object({ status: z.literal('success'), city: z.string(), weather: z.string() }), ]), }; With this schema the same stream finished with status: ready on both versions, and the part ended up as { city: 'Berlin', weather: 'sunny', status: 'success' }. Before you upgrade, I'd do three things: 1. Search the client for dataPartSchemas. 2. For each data-* type in there, check every writer.write() on the server against its schema, including loading, progress and transient writes. 3. Make sure the UI shows error from useChat. In my runs a failed part stopped the whole response, so a silent failure here looks like a chat that just stops. Data parts with no schema aren't checked at all, so a schema-less type won't start failing. I'd rather have the error than let city: 42 slip into message.parts without a word, which is what 7.0.159 did. Have you hit this one after updating, or were your schemas already matching what the server sends? References - ai 7.0.160 release notes - Issue #24241: useChat/Chat never validates streamed data parts against dataPartSchemas - PR #24298: validate streamed data parts against dataPartSchemas - AI SDK docs: Streaming custom data - AI SDK docs: useChat reference

Observability for AI Agents: Tracing, Evaluation and Cost Control in Production How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices. 1. Why Is Observability Different for AI Agents? Traditional backend observability is built around relatively predictable execution paths. A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred. AI agents introduce a different level of complexity. An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input. Consider a relatively simple task: A user asks an AI assistant to analyze a merge request and identify potential problems. Behind the scenes, the application might: 1. Receive the user's request. 2. Invoke a coordinator agent. 3. Call an LLM to determine the next action. 4. Retrieve the merge request from GitLab. 5. Read the changed files. 6. Delegate security analysis to another agent. 7. Execute additional tool calls. 8. Evaluate the collected results. 9. Generate a final report. The user sees one response. The application has actually executed a multi-step workflow involving several components. Now imagine the response takes 45 seconds instead of the expected 8 seconds. Where should we investigate? - Did the model take too long to respond? - Did a tool call time out? - Was the same operation executed repeatedly? - Did the agent generate unnecessary steps? - Did a subagent introduce additional latency? - Was too much context sent to the model? - Did a retry increase the overall execution time and cost? Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why. AI observability is about understanding the entire execution path, not just measuring the final API response. This is where OpenTelemetry becomes particularly useful. 2. What Is OpenTelemetry? <escape>**</escape><Entity value="OpenTelemetry (OTel)" category="software" disambig="Open-source observability framework"/> is an open-source observability framework for collecting, processing, and exporting telemetry data. It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms. Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported. OpenTelemetry is built around three primary signals: | Signal | What it tells us | AI agent example | |---|---|---| | Traces | How an operation was executed | Model call → tool → subagent | | Metrics | How the system performs over time | Token usage, latency, error rate | | Logs | What happened at a specific point | Tool timeout or agent exception | These signals complement each other. A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation. OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector. An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal. References: OpenTelemetry Documentation · OpenTelemetry JavaScript 3. Understanding Distributed Tracing Before discussing AI-specific instrumentation, we need to understand traces and spans. A trace represents the complete execution path of an operation. A span represents one individual unit of work within that execution. For example: Trace: POST /api/agent | +-- HTTP Request | +-- Coordinator Agent | +-- LLM Generation | +-- Tool: GitLab - Get Merge Request | +-- Tool: GitLab - Get Changed Files | +-- Security Agent | | | +-- LLM Generation | | | +-- Tool: Security Analysis | +-- Testing Agent | | | +-- LLM Generation | | | +-- Tool: Test Coverage | +-- Final Response Generation Each span has information such as: - Trace ID - Span ID - Parent span ID - Start and end timestamps - Duration - Attributes - Events - Status and exception information This creates a hierarchical representation of the complete execution. The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them. For AI agents, this distinction is important because one user request can generate many independent operations. Why is context propagation important? In distributed systems, a trace often crosses multiple services. Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service. Without context propagation, every service may generate its own independent trace. We would see several unrelated operations instead of one complete workflow. OpenTelemetry uses W3C Trace Context as its standard propagation format. The traceparent header carries trace and parent-span information between instrumented services. The same principle applies to agent-based architectures. When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution. Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction. The important point is that protocol-level interoperability and observability context propagation are separate concerns. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace. References: OpenTelemetry Traces · Context Propagation · W3C Trace Context 4. OpenTelemetry and GenAI Semantic Conventions One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers. For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes. This makes it difficult to compare different implementations using the same monitoring platform. OpenTelemetry addresses this through Semantic Conventions. Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently. The GenAI semantic conventions extend this approach to generative AI applications. Some important attributes include: | Attribute | Purpose | |---|---| | gen_ai.provider.name | Identifies the model provider | | gen_ai.request.model | Identifies the requested model | | gen_ai.response.model | Identifies the actual response model | | gen_ai.operation.name | Describes the operation | | gen_ai.usage.input_tokens | Input token usage | | gen_ai.usage.output_tokens | Output token usage | | gen_ai.conversation.id | Associates operations with a conversation | | gen_ai.agent.name | Identifies the agent | | gen_ai.tool.name | Identifies the executed tool | For example, an LLM invocation could be represented by: { "gen_ai.operation.name": "chat", "gen_ai.provider.name": "openai", "gen_ai.request.model": "example-model", "gen_ai.usage.input_tokens": 1200, "gen_ai.usage.output_tokens": 350 } This is a simplified span attribute example. The benefit is that observability platforms do not need to understand a different custom schema for every application. Agent-specific conventions The GenAI semantic conventions also define operations for agents and workflows. Some examples include: - invoke_agent - invoke_workflow - execute_tool - plan - retrieval This is particularly valuable for multi-agent architectures. We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents. For example: invoke_workflow merge-request-review | +-- invoke_agent coordinator | | | +-- chat model | +-- execute_tool fetch_merge_request | +-- invoke_agent security-agent | | | +-- chat model | +-- execute_tool analyze_vulnerabilities | +-- invoke_agent testing-agent | | | +-- chat model | +-- execute_tool analyze_test_coverage | +-- Final Aggregation This makes the execution model much easier to understand. An important maturity consideration: As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract. The current convention documents include: - Semantic Conventions for GenAI Spans - Semantic Conventions for Agent Spans - Semantic Conventions for GenAI Metrics 5. Integrating OpenTelemetry with Vercel AI SDK Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration. The AI SDK already supports OpenTelemetry-based instrumentation. It can generate spans for: - Text generation - Streaming generation - Individual model calls - Agent execution - Tool execution - Embedding operations - Reranking This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility. Step 1: Install the packages For an AI SDK v7 application: pnpm add @ai-sdk/otel @vercel/otel The OpenTelemetry integration is provided through @ai-sdk/otel. This is an important version-specific detail. In AI SDK v7, OpenTelemetry integration moved out of the core ai package. Older examples using the v6 experimental_telemetry configuration should not be copied directly into a v7 application. Step 2: Register telemetry in Next.js Create an instrumentation.ts file in the project root. import { registerOTel } from '@vercel/otel'; import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; export function register() { registerOTel({ serviceName: 'ai-agent-platform', }); registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); } There are two responsibilities here: - registerOTel initializes the OpenTelemetry infrastructure for the Next.js application. - registerTelemetry registers the AI SDK-specific integration that creates GenAI spans. I have disabled input and output capture in this example intentionally. AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision. We will return to this in the security section. For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup. Step 3: Instrument AI SDK calls Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration. We can also provide a functionId to identify the business operation or agent. import { generateText } from 'ai'; const result = await generateText({ model: myModel, instructions: ` You are a financial analysis assistant. Analyze the supplied information carefully. `, prompt: 'Analyze the latest portfolio performance.', telemetry: { functionId: 'portfolio-analysis-agent', }, }); The functionId helps distinguish operations in traces. For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as: invoke_agent portfolio-analysis-agent Step 4: Understand the generated spans The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions. For generateText and streamText, the integration records three main types of spans: 1. Agent invocation span The root span represents the complete generation operation, including its execution steps and tool calls. Example: invoke_agent portfolio-analysis-agent 2. Step and model-call spans Each model interaction is represented by step and provider-call spans. These allow us to understand how many model calls occurred during an operation and how long each one took. 3. Tool execution spans Each client-side tool execution receives its own span. For example: invoke_agent portfolio-analysis-agent | +-- step 0 | | | +-- chat example-model | +-- execute_tool get_portfolio | +-- step 1 | | | +-- chat example-model | +-- execute_tool get_market_data | +-- step 2 | +-- chat example-model The resulting trace provides a much more complete representation of an agent workflow than a single model request log. The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available. Step 5: Add additional application context Sometimes the standard GenAI attributes are not enough. For example, we may want to know which business workflow initiated an operation or which application component owns it. The AI SDK provides enrichSpan for adding custom attributes: import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, enrichSpan: ({ spanType, operationId }) => ({ 'app.component': 'portfolio-assistant', 'app.operation_id': operationId, 'app.span_type': spanType, }), }), ); The exact context available to enrichSpan depends on the span and the SDK's integration API. The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry. References: AI SDK Telemetry · Next.js OpenTelemetry · AI SDK v7 Migration Guide 6. Instrumenting Your Own Agent Runtime Built-in instrumentation is useful, but it does not automatically understand every application-level decision. For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval. These are application-level operations. We should instrument them explicitly. OpenTelemetry provides a tracing API for creating custom spans. Consider a custom agent orchestration function: import { trace, SpanStatusCode, } from '@opentelemetry/api'; const tracer = trace.getTracer('agent-orchestrator'); export async function runReviewWorkflow() { return tracer.startActiveSpan( 'workflow.merge-request-review', async (span) => { span.setAttribute( 'workflow.name', 'merge-request-review', ); span.setAttribute( 'workflow.version', '1.0.0', ); try { const result = await executeReview(); span.setStatus({ code: SpanStatusCode.OK, }); return result; } catch (cause) { const error = cause instanceof Error ? cause : new Error(String(cause)); span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR, }); throw error; } finally { span.end(); } }, ); } Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated. We now have two levels of observability: - Framework-level instrumentation describes model and tool execution. - Application-level instrumentation describes business workflows and orchestration decisions. This distinction is important. We should not expect a model instrumentation library to understand the complete business meaning of an application request. Instrumenting remote agent calls For A2A-based systems, distributed tracing becomes even more important. Imagine a coordinator running in one service and a security agent running in another service. The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated. Conceptually: Trace: merge-request-review | +-- Coordinator Agent | | | +-- Model Call | | | +-- A2A: Delegate to Security Agent | | | +-- Security Agent | | | +-- Model Call | | | +-- Tool Execution | +-- Final Aggregation The HTTP request must carry the appropriate trace context, typically through W3C traceparent and tracestate headers. For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context. This is how we avoid fragmented traces across independently deployed agents. 7. What Should We Measure? Collecting telemetry is not enough. We need to define which metrics are useful for understanding the health and quality of an agent platform. I would divide AI observability metrics into five categories. 7.1. Latency Metrics These describe how long operations take. | Metric | Why it matters | |---|---| | End-to-end workflow duration | Measures total user-facing execution time | | Model request duration | Identifies slow inference calls | | Time to first token | Measures perceived response latency | | Tool execution duration | Identifies slow external operations | | Agent invocation duration | Measures the execution time of one agent | | Queue wait duration | Identifies scheduling bottlenecks | For streaming applications, time to first token is particularly useful. A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds. 7.2. Reliability Metrics These help us understand whether workflows complete successfully. Examples: - Agent execution success rate - Tool execution failure rate - Model request failure rate - Timeout rate - Retry rate - Task cancellation rate - Human intervention rate - Incomplete workflow rate A high retry rate can be an early indicator of a problematic integration or an unstable external dependency. We should also distinguish between a successful model response and a successful business workflow. An LLM can return a perfectly valid response while the overall task fails to achieve its objective. 7.3. Token and Model Usage Token usage should be tracked at multiple levels. | Metric | Description | |---|---| | Input tokens | Tokens sent to the model | | Output tokens | Tokens generated by the model | | Cached input tokens | Tokens served from provider cache | | Reasoning tokens | Model-reported reasoning token usage | | Tokens per workflow | Total usage across an entire task | | Tokens per agent | Usage attributed to an individual agent | For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response. 7.4. Agent and Tool Behavior These metrics describe how efficiently agents perform their work. Examples: - Number of tool calls per task - Number of model calls per agent invocation - Number of agent handoffs - Repeated tool-call frequency - Tool-selection success - Average steps per workflow - Average number of clarification requests An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem. The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs. 7.5. Quality Metrics This is where AI observability starts to overlap with AI evaluation. Examples: - Answer correctness - Groundedness - Retrieval relevance - Tool-selection accuracy - Task completion - Policy adherence - Human acceptance rate These metrics cannot always be derived from execution telemetry alone. We need an evaluation strategy to measure the actual quality of the result. 8. Cost Control: Understanding the Real Cost of an Agent One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow. An agent-based application may consume resources across several layers: - LLM inference - Embedding generation - Reranking - External API calls - Database queries - Vector database operations - Container or sandbox execution - Remote agent invocations - Infrastructure and networking The total cost should reflect the entire execution path. A useful starting point is to define a model cost formula: [ C_{\text{model}} = \sum_{i=1}^{n} \left( T_{\text{input},i}P_{\text{input},i} + T_{\text{output},i}P_{\text{output},i} \right) ] Where: - (T) represents token usage. - (P) represents the applicable price per token. - (i) represents an individual model operation. The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities. We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets. The complete workflow cost can then be represented as: [ C_{\text{workflow}} = C_{\text{model}} + C_{\text{tools}} + C_{\text{compute}} + C_{\text{storage}} ] This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available. Why is cost per successful task more useful? Imagine two implementations: | Metric | Agent A | Agent B | |---|---:|---:| | Average cost per attempt | $0.04 | $0.07 | | Successful completion rate | 90% | 98% | | Cost per successful task | ~$0.044 | ~$0.071 | Agent A is cheaper per attempt, but Agent B completes more tasks successfully. Neither is automatically better. The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts. A useful metric is: [ \text{Cost per successful task} = \frac{\text{Total execution cost}} {\text{Number of successful tasks}} ] The numerator should include failed attempts that consumed resources, rather than counting only successful runs. Model selection and routing Observability data can also help us make better model-routing decisions. For example: - Use a smaller model for classification or simple extraction. - Use a more capable model for complex analysis. - Avoid unnecessary model calls when a deterministic operation is sufficient. - Reuse cached context when appropriate. - Set explicit iteration and tool-call limits. - Stop an agent when a task exceeds its budget. The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request. Token metrics in OpenTelemetry The current GenAI token metric conventions include proposed counters such as: - gen_ai.client.inference.usage.input_tokens - gen_ai.client.inference.usage.output_tokens - gen_ai.client.inference.usage.cache_read.input_tokens - gen_ai.client.inference.usage.reasoning.output_tokens These conventions are still in Development. There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics. A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations. For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices. This makes historical cost calculations more reliable when provider pricing changes. References: GenAI Token Metrics · GenAI Metrics 9. Observability Is Not Evaluation This distinction is worth emphasizing. Observability tells us what happened during execution. Evaluation tells us whether the result was good enough. A trace can show that an agent: - Called the correct model. - Used three tools. - Retrieved five documents. - Completed within four seconds. - Returned a response without an exception. But none of these facts proves that the answer was correct. Consider a RAG application. The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information. From an operational perspective, the request succeeded. From a quality perspective, it failed. We need both observability and evaluation. Offline Evaluation Offline evaluation measures an AI application against a predefined dataset. For example, we can create 100 representative questions and expected outcomes. Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again. This helps identify regressions before deploying changes. Useful evaluation dimensions include: - Answer correctness - Relevance - Groundedness - Tool selection - Expected task completion - Policy compliance Online Evaluation Online evaluation measures actual production interactions. We can select a sample of production traces and evaluate them using: - Rule-based checks - Code-based evaluators - LLM-as-a-Judge - Human feedback The important part is to turn real production failures into future test cases. For example: 1. A production trace shows that an agent selected the wrong tool. 2. We identify the relevant input and expected behavior. 3. We create a regression dataset entry. 4. We update the prompt or agent implementation. 5. We rerun offline evaluations. 6. We deploy and monitor the next set of production traces. This creates a continuous improvement loop. Production Traces | v Failure Analysis | v Evaluation Dataset | v Offline Evaluation | v Prompt / Model / Agent Changes | v Deployment | v Online Evaluation | +------> New Regression Cases Tools worth evaluating Two examples of AI-focused observability and evaluation platforms are: Langfuse Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements. Documentation: langfuse.com Arize Phoenix An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions. It provides tracing, evaluation, datasets, experiments, and debugging capabilities. Documentation: phoenix.arize.com These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy. Reference: Langfuse Evaluation Concepts · Arize Phoenix 10. The Role of OpenTelemetry Collector In a production architecture, applications should not necessarily export telemetry directly to every observability backend. This is where the OpenTelemetry Collector becomes useful. The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations. It provides a central place for: - Batching telemetry - Filtering unwanted data - Applying processors - Handling authentication - Reducing telemetry volume - Sampling traces - Routing signals to different backends A simplified architecture looks like this: AI Applications | +----------+----------+ | | | v v v Next.js .NET API Python Agent | | | +----------+----------+ | OTLP | v OpenTelemetry Collector | +--------+--------+ | | | v v v Traces Metrics Logs | | | v v v Jaeger Prometheus Log Backend | v Grafana The same telemetry can also be routed to AI-specific observability platforms where supported. Why centralize collection? In enterprise environments, we may have several applications and teams using different frameworks and languages. A Collector provides a common ingestion point and makes it possible to apply centralized policies. For example, a Collector can filter sensitive attributes before telemetry reaches the final backend. It also allows us to change exporters or observability providers without changing the instrumentation in every application. Agent and gateway deployment OpenTelemetry supports different Collector deployment patterns. A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors. This provides a separation between local telemetry collection and centralized processing. However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling. Reference: OpenTelemetry Collector · Gateway Deployment Pattern 11. Sampling: How Do We Control Telemetry Volume? Agent-based workflows can generate a significant amount of telemetry. Consider an agent that performs 15 model calls and 20 tool calls during a single task. If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive. Sampling helps control this volume. There are two common approaches. Head-Based Sampling The sampling decision is made when the trace starts. For example, we may decide to sample 10% of incoming requests. This is relatively simple and reduces the amount of telemetry generated or exported. However, we do not yet know whether a trace will eventually fail or become unusually slow. Tail-Based Sampling The sampling decision is made after considering the trace's collected spans. This makes it possible to retain: - All failed traces. - All traces exceeding a latency threshold. - Traces containing specific tool failures. - A percentage of successful traces. For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed. However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance. This generally requires trace-ID-aware routing when using multiple Collector instances. A practical approach is to combine sampling strategies: - Preserve error and high-latency traces. - Retain a representative sample of successful traces. - Maintain complete aggregate metrics where possible. - Apply stricter retention policies to verbose telemetry. Reference: OpenTelemetry Sampling · Tail Sampling Processor 12. Security and Privacy Considerations This is one of the most important aspects of AI observability in enterprise environments. Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names. AI telemetry can contain much more sensitive information. For example: - User prompts - System instructions - Generated responses - Tool arguments - Tool results - Retrieved documents - Conversation context - Customer or business identifiers Sending all this information to an observability backend can create additional privacy and data governance risks. Do not capture prompts and responses by default A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise. We can disable this behavior: registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); For many production applications, this is a better starting point. We can still collect useful operational telemetry: - Model identifiers - Token counts - Duration - Finish reasons - Tool execution times - Error information - Workflow identifiers without exporting the complete conversation. Apply data minimization Before exporting telemetry, ask whether each attribute is genuinely needed. Avoid sending: - Access tokens - Authorization headers - Passwords - Raw customer records - Full database query results - Unfiltered tool arguments For sensitive workflows, use approved metadata rather than full payloads. Separate operational telemetry from application data An observability backend should not automatically become another storage system for business data. The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely. Define access and retention policies Enterprise environments should define: - Who can access traces. - Which teams can view prompt and response content. - How long telemetry is retained. - How data is deleted. - Which environments are allowed to capture full content. - How telemetry exports are authenticated and encrypted. A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production. Reference: OpenTelemetry GenAI Content Capture 13. Limitations and Trade-offs OpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself. Instrumentation coverage varies Not every framework, provider, or agent runtime exposes the same information. Some integrations can automatically trace model calls and tool execution, while others require custom instrumentation. We should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility. GenAI semantic conventions are evolving The GenAI conventions are still under development. Attribute names, operation models, and implementation support may evolve. Teams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity. Observability does not guarantee answer quality A trace can explain what happened but does not prove that the model's result is correct. Evaluation datasets, deterministic checks, and human feedback remain necessary. Telemetry has its own cost Collecting every input, output, tool argument, and execution event can generate significant storage and processing overhead. We should measure the cost of observability itself. High-cardinality dimensions can become expensive Adding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series. Use traces for request-level correlation and keep metric dimensions bounded. Distributed tracing requires correct context propagation A2A communication, queues, background tasks, and remote tool execution may cross process boundaries. If context is lost, we can end up with disconnected traces. AI evaluation requires careful design LLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased. A single evaluation score should not become the only definition of quality. Combine deterministic checks, representative datasets, human review, and calibrated evaluators. 14. What Can an AI Chapter Lead Contribute? I think this is where observability moves from an implementation detail to an engineering capability. When multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms. Over time, this creates fragmented visibility. One team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model. As an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications. 1. Define a Common Telemetry Standard Establish a baseline for the telemetry every agent-based application should produce. For example: | Category | Required information | |---|---| | Application | Service name, environment, version | | Workflow | Workflow name, workflow version | | Agent | Agent name, execution status | | Model | Provider, requested model, response model | | Usage | Input/output tokens, cache usage | | Tools | Tool name, duration, execution status | | Execution | Trace ID, span ID, error information | | Business outcome | Completed, failed, canceled, escalated | The objective is to make observability consistent without forcing every team to use the same agent framework. 2. Build a Shared Instrumentation Package We can create an internal TypeScript package that wraps the common OpenTelemetry configuration. For example: @company/ai-observability | +-- OpenTelemetry initialization +-- AI SDK integration +-- Agent span utilities +-- Workflow instrumentation +-- Tool execution tracking +-- Cost attribution +-- Evaluation metadata +-- Privacy and redaction policies This package can provide standard helpers for application-level tracing and reduce duplicated implementation. For teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations. 3. Standardize Agent and Workflow Metrics We should define a common set of metrics that helps teams understand both technical performance and business outcomes. Examples: - Agent success rate - Workflow completion rate - Tool-call failure rate - Model latency - Cost per successful task - Average token usage - Human intervention rate - Evaluation quality scores A shared dashboard can make it easier to compare systems and identify performance regressions. 4. Establish an Internal Evaluation Framework Observability without evaluation gives us only part of the picture. We can create reusable evaluation datasets and testing patterns for common agent behaviors. For example: - Tool selection accuracy - Missing-source detection - Correct handling of incomplete input - Permission violations - Unsupported claims - Failure recovery - Task completion quality This is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified. 5. Establish Privacy and Data Governance Define what can and cannot be captured by default. The shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads. This should include clear ownership, access policies, retention, and audit requirements. 6. Run a Cross-Team Pilot A useful pilot would be a multi-agent workflow that involves several independently deployed services. For example, a merge request review involving coordinator, security, and testing agents. The pilot should demonstrate: 1. One trace across the complete request. 2. Parent-child relationships between agents. 3. Model and tool execution spans. 4. Token and cost attribution. 5. Error and retry visibility. 6. Evaluation results associated with the completed workflow. This would provide a reference architecture that other teams can reuse. 7. Make Observability Part of the Development Lifecycle Observability should not begin after production deployment. We can introduce telemetry and evaluation requirements into the development process itself. For example: - New agents must have execution tracing. - New tools must expose execution status and duration. - Prompt changes must pass relevant regression evaluations. - New workflows must define success and failure criteria. - Production deployments must have basic operational dashboards. This makes observability a standard engineering practice rather than a separate monitoring task. 15. A Practical Adoption Roadmap I would avoid introducing every observability component at once. A phased approach is more useful. | Phase | Focus | Expected result | |---|---|---| | 1 | OpenTelemetry initialization | Basic traces and metrics | | 2 | AI SDK instrumentation | Model and tool visibility | | 3 | Custom agent spans | Complete workflow tracing | | 4 | Cost attribution | Usage and cost per task | | 5 | Evaluation | Offline and online quality checks | | 6 | Security and governance | Privacy and access controls | | 7 | Shared dashboards | Cross-team operational visibility | The first milestone should be simple: For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result. Once this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce. Conclusion AI agents introduce a new level of complexity to application observability. A single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution. OpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components. Combined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production. But the most important distinction remains: Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it. For an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices. The goal should not be to collect more telemetry. It should be to make every production agent workflow understandable, measurable, and continuously improvable. --- References and Further Reading OpenTelemetry 1. OpenTelemetry Official Documentation — Core concepts, signals, instrumentation, and architecture. 2. OpenTelemetry JavaScript — TypeScript and Node.js instrumentation. 3. OpenTelemetry Tracing — Traces, spans, and distributed execution. 4. OpenTelemetry Context Propagation — Distributed trace context. 5. OpenTelemetry Collector — Receiving, processing, and exporting telemetry. 6. OpenTelemetry Gateway Deployment — Centralized telemetry collection. 7. OpenTelemetry Sampling — Head-based and tail-based sampling. GenAI Semantic Conventions 8. GenAI Semantic Conventions — Spans — Model operations, retrieval, and tool execution. 9. GenAI Agent Spans — Agent invocation, workflows, and tool spans. 10. GenAI Metrics — Agent duration, tool calls, and workflow metrics. 11. GenAI Token Metrics — Token consumption and usage measurements. Vercel AI SDK 12. AI SDK Telemetry — OpenTelemetry integration, span hierarchy, and custom attributes. 13. AI SDK v7 Migration Guide — Version-specific telemetry changes. 14. Next.js OpenTelemetry — Application-level instrumentation. AI Evaluation and Observability Platforms 15. Langfuse Evaluation — Online and offline evaluation. 16. Langfuse Core Concepts — Datasets, experiments, and evaluators. 17. Arize Phoenix — Open-source AI observability and evaluation. 18. OpenInference — OpenTelemetry-based instrumentation conventions for AI applications.

写 AI 应用最烦的，是每个厂商 SDK 各写一套。 Vercel AI SDK 7.x 一个统一接口接多家厂商，TypeScript 写多模型产品，换厂商不用重写。 适合做多模型产品的 TypeScript 开发者，先从一个厂商接入试起。 接多家模型，自己封装还是用 SDK？ #AI开发 #开源 @grok vercel/ai

Vercel's AI SDK now gives sandbox tools a shared API for listing files, checking file metadata and deleting files. Developers can use those methods instead of parsing shell output separately for each supported environment. The API is experimental. github.com/vercel/ai/releases/tag/ai%407.0.158

Top line: xai@5.0.33 maps unknown reasoning levels down to the nearest supported level instead of jumping up. Direct consequence: 'xhigh' on Anthropic or Bedrock without xhigh now becomes 'high', not 'max'. github.com/vercel/ai/releases/tag/%40ai-sdk/xai%4…

Structured outputs for every Claude model on Amazon Bedrock: schema-valid JSON, plus an EKS alert-triage example Originally published in Spanish on CloudAcademy.ar. If you've put an LLM in the middle of an automated pipeline, you know the routine: ask for JSON in the prompt, add an example, beg for no extra text, and still get the occasional trailing comma, renamed field or "Sure, here's the JSON:" that breaks json.loads. So you add validation, retries and a forgiving parser, and the glue code ends up bigger than the business logic. On October 20, 2026, AWS announced structured outputs for all Anthropic Claude models on Amazon Bedrock, on both the bedrock-runtime and bedrock-mantle endpoints. You define a JSON Schema and the response follows it, because the constraint is applied during token generation rather than checked afterwards. TL;DR - Two mechanisms: a JSON Schema output format (shapes the model's response) and strict tool use ("strict": true, shapes tool-call arguments). You can use both in the same request. - It's constrained decoding. The first request with a new schema compiles a grammar (up to ~90 seconds), which is cached for 24 hours per account. - Every object needs additionalProperties: false. Max 12 optional properties and 16 union-typed/anyOf properties. No minimum/maximum, no minLength/maxLength, no recursive schemas. - Incompatible with citations on Anthropic models (400 error). - No separate charge announced: you pay model tokens, and output usually gets shorter. - Still check stop_reason: hitting max_tokens truncates the JSON. Why this matters now Structured outputs on Bedrock isn't new. It went GA in February 2026 across several providers. The gap was coverage: the Claude 5 family wasn't supported for months. You could see it in the ecosystem. A July issue in Vercel's AI SDK reported Bedrock rejecting strict for claude-sonnet-5 with a 400, so the SDK limited native structured outputs to Claude 4.x. The day of the announcement, a new issue asked to enable it for Claude 5. Lesson: Bedrock supporting it doesn't mean your SDK sends it. This launch closes that gap: every Claude model, both endpoints, plus batch inference, cross-Region inference and typed streaming events. How it works !Structured outputs pipeline 1. Bedrock validates the schema against a JSON Schema Draft 2020-12 subset (unsupported feature: immediate 400). 2. A new schema compiles into a grammar, which can take up to ~90 seconds. AWS recommends a client read timeout of at least 120 seconds. 3. The grammar is cached for 24 hours per account. Changing only name/description keeps the cache; structural changes don't. 4. The model can only emit tokens that keep the output valid. !Output format vs strict tool use Where the schema goes: output_config.format for InvokeModel and the Messages API (a JSON object), and outputConfig.textFormat for Converse (a JSON string inside structure.jsonSchema.schema). Hands-on: alert triage on EKS with Claude Haiku 5.5 The example: Alertmanager sends a webhook to a pod on EKS, the pod asks Claude Haiku 5.5 to classify the alert group against a schema, and the JSON decides who gets paged. !Triage flow Haiku 5.5 is what AWS suggests for fast, repetitive work like classification. I'm using the global inference profile, global.anthropic.claude-haiku-5-5, which is also the only way to reach Haiku 5.5 from sa-east-1. Permissions via EKS Pod Identity (trust pods.eks.amazonaws.com, allow bedrock:InvokeModel on the inference profile and foundation model ARNs), then: aws eks create-pod-identity-association --cluster-name production \ --namespace observability --service-account alert-triage \ --role-arn arn:aws:iam::123456789012:role/alert-triage-bedrock The schema keeps everything in required to stay under the 12-optional limit, and uses ["string", "null"] where a value may be absent: TRIAGE_SCHEMA = { "type": "object", "properties": { "severity": {"type": "string", "enum": ["info", "warning", "critical"]}, "owner_team": {"type": "string", "enum": ["platform", "payments", "data", "frontend", "unknown"]}, "page_oncall": {"type": "boolean"}, "probable_cause": {"type": "string"}, "suggested_action": {"type": "string"}, "runbook_url": {"type": ["string", "null"], "format": "uri"}, "related_alerts": {"type": "array", "items": {"type": "string"}} }, "required": ["severity", "owner_team", "page_oncall", "probable_cause", "suggested_action", "runbook_url", "related_alerts"], "additionalProperties": False } The call itself: bedrock = boto3.client("bedrock-runtime", region_name="sa-east-1", config=Config(read_timeout=120, retries={"max_attempts": 3, "mode": "adaptive"})) resp = bedrock.invoke_model( modelId="global.anthropic.claude-haiku-5-5", body=json.dumps({ "anthropic_version": "bedrock-2023-05-31", "max_tokens": 1024, "system": SYSTEM, "messages": [{"role": "user", "content": "Alertmanager group:\n" + json.dumps(alerts)}], "output_config": {"format": {"type": "json_schema", "schema": TRIAGE_SCHEMA}}, }), ) body = json.loads(resp["body"].read()) if body["stop_reason"] != "end_turn": raise RuntimeError(f"incomplete response: {body['stop_reason']}") result = json.loads(next(b["text"] for b in body["content"] if b["type"] == "text")) There's no jsonschema validation and no retry-on-malformed-JSON loop, because you don't need them. The one operational trick: warm up at pod start. Make one dummy call before the readiness probe goes green (I use periodSeconds: 5 and failureThreshold: 30), so grammar compilation happens during the rollout and not on the first real 3 a.m. alert. Run the receiver in parallel with your existing routing (continue: true) for a few weeks and compare decisions. For agents that act, use strict tools. The enum on namespace and a pattern on the deployment name constrain what the agent can ask for. Ranges like 1 <= replicas <= 20 can't go in the schema, so they stay in code, and execution stays behind human approval. Gotchas - Short read timeouts kill the first request during grammar compilation. - Low max_tokens gives you truncated JSON even with structured outputs on. - Going past 12 optional properties returns Schema is too complex after up to ~90 seconds. - A missing additionalProperties: false in a nested object is an instant 400. - Citations plus structured outputs on Claude is a 400. - Every structural schema change pays compilation again. Version schemas like API contracts. - Check what your SDK actually sends: some strip strict for models they don't know about. Pricing No separate charge was announced; you pay the model's tokens. Anthropic's list prices per million tokens: Haiku 5.5 $0.10 input / $0.50 output (prompts up to 100K tokens), Sonnet 5.5 $2 / $10, Opus 5.5 $4 / $20. Check Bedrock's pricing table for your Region and profile. My estimate for 100,000 alert groups a month at ~1,500 input and 150 output tokens on Haiku: about $22.50/month. Batch inference is 50% cheaper for non-urgent work and supports structured outputs. Prompt caching also helps, and Sonnet 5.5 cache reads on Bedrock cost half as much since October 7, 2026. Region and data residency From São Paulo, Haiku 5.5 is only available through global cross-Region inference. That's fine for infrastructure alerts. For regulated data, check what a global profile implies first, and keep secrets and customer data out of alert annotations. Some geographic profiles don't support structured outputs for every model, so test with the exact profile your policy requires. Is it worth it? If you're using Claude on Bedrock for anything that ends in json.loads: yes. Add output_config, make the schema follow the rules, and delete your validation layer. For agents whose tools touch real systems, strict: true should be the default from today. My take: the feature has existed since February. What's new is that it no longer depends on the model. While some Claude models supported it and others didn't, every model switch risked breaking parsing, and libraries had to maintain allowlists. With full coverage, moving from Haiku to Sonnet goes back to being a quality and cost decision, not a compatibility one. For platform teams, the pattern generalizes: a small model, a strict schema, enums that map to real decisions, and a human approving anything with impact. It's the most boring and safest way to put AI into an ops workflow, and I mean that as a compliment. Resources - What's New: structured outputs for Claude models - AWS ML Blog: Structured outputs for Anthropic Claude models on Amazon Bedrock - AWS ML Blog (Feb 2026): Structured outputs on Amazon Bedrock - Docs: Claude structured outputs - Docs: Get validated JSON results from models - Anthropic model Region compatibility - Vercel AI SDK issue #23605

@cramforce @Sagarvd01 @mxstbr ai#24061 🙏

想把聊天 AI 接进自己的网页？🤖 Vercel AI SDK 提供 TypeScript 模型调用和流式聊天工具，也能做 Agent。适合 Next.js 项目。💬 vercel/ai #AISDK #浇朋友 #蓝朋友 #互关 #fo

Your Zod .refine() Never Reaches the Model: What Survives the Trip to JSON Schema - Book: AI That Answers - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Pass a Zod 4 schema to a structured-output helper and every .refine() in it is dropped before the model sees it, with no warning. The OpenAI Node SDK and the Vercel AI SDK both call z.toJSONSchema() on Zod 4 schemas, and that JSON is what the model is constrained by. You can see why in Zod's JSON Schema converter at v4.6.7, the release from 12 October 2026. One line in json-schema-processors.ts decides which of your validation rules get written into the schema: for (const ch of list) contributors[ch._zod.def.check]?.(agg, ch._zod.def);. The contributors map has 13 keys: greater_than, less_than, multiple_of, min_length, string_format and friends. Every .refine() you write is stored as a check named custom (api.ts), and custom is not one of the keys. The optional call finds nothing, and the rule is skipped. Your refinement still runs later, when the response is parsed. The model just never heard about it. A realistic schema with four kinds of rule Here is a refund extraction schema of the kind you would hand to a model that reads support emails. It mixes rules JSON Schema can express with rules it can't. import { z } from "zod"; export const Refund = z .object({ orderId: z.string().regex(/^ORD-\d{6}$/), amount: z.number().positive(), currency: z.enum(["EUR", "USD"]), orderDate: z.iso.date(), refundDate: z.iso.date(), note: z.string().refine((s) => !/https?:\/\//.test(s), { message: "note must not contain links", }), sku: z.string().transform((s) => s.trim().toUpperCase()), }) .refine((r) => r.refundDate >= r.orderDate, { message: "refundDate is before orderDate", path: ["refundDate"], }); Four fields use rules JSON Schema has a keyword for: regex, positive, z.enum and the ISO date format. The rest are the problem: a .refine() on note, a cross-field .refine() on the whole object, and a .transform() on sku. The JSON the model actually gets Below is the shape z.toJSONSchema(Refund, { io: "input" }) produces, which is the mode the Vercel AI SDK uses. I haven't run this exact schema myself. Each property follows the converter's test snapshots in to-json-schema.test.ts and its processor source for the same kinds of Zod call, and I trimmed the long ISO-date regex and the $schema line. Print it on your own version with console.log(JSON.stringify(z.toJSONSchema(Refund, { io: "input" }), null, 2)). { "type": "object", "properties": { "orderId": { "type": "string", "pattern": "^ORD-\\d{6}$" }, "amount": { "type": "number", "exclusiveMinimum": 0 }, "currency": { "type": "string", "enum": ["EUR", "USD"] }, "orderDate": { "type": "string", "format": "date", "pattern": "<long ISO date regex, trimmed>" }, "refundDate": { "type": "string", "format": "date", "pattern": "<long ISO date regex, trimmed>" }, "note": { "type": "string" }, "sku": { "type": "string" } }, "required": [ "orderId", "amount", "currency", "orderDate", "refundDate", "note", "sku" ] } Line it up against the Zod source: | Field | Zod rule | What the model sees | |---|---|---| | orderId | .regex(...) | pattern | | amount | .positive() | exclusiveMinimum: 0 | | currency | z.enum | enum | | orderDate, refundDate | z.iso.date() | format: "date" plus a pattern | | note | .refine(no links) | nothing, just "type": "string" | | sku | .transform(...) | input mode: a plain string; output mode: an error | | the object | .refine(refund after order) | nothing | The test file backs up the note row. It unions z.string().refine((s) => s.length > 2) with z.string().refine((s) => s.length < 9), and the snapshot collapses both branches to a bare "type": "string". They can only dedupe if neither refinement made it into the output. The object-level refinement goes the same way, because .superRefine() and object .refine() also land as custom checks. One more detail from the Zod JSON Schema docs: in input mode, additionalProperties is not set on objects. The AI SDK adds it back with its own addAdditionalPropertiesToJsonSchema pass. In the default output mode, Zod writes additionalProperties: false itself. !A flat illustration of a customs checkpoint where some parcels pass through a gate and others are held back on a table Representable checks cross into JSON Schema. Refinements stay behind. Transforms either throw or tell the model the wrong type Transforms behave differently from refinements, and the behaviour depends on the io option. A .transform() on a string is a pipe: string in, transform out. The pipe processor picks a side with ctx.io === "input" ? (inIsTransform ? def.out : def.in) : def.out (source). In input mode, it takes the string side, so the model sees { "type": "string" } for sku and your uppercasing happens after. That is harmless for this field. It is less harmless for something like z.string().transform(Number), where the model is told only "string", so nothing stops it from writing "twelve" or "12 EUR", and Number() hands your code NaN without a validation error. In output mode, the default, the converter reaches the transform itself. The transform processor calls handleUnrepresentable with "Transforms cannot be represented in JSON Schema", and the docs say Zod throws on unrepresentable types unless you pass unrepresentable: "any". The OpenAI helper calls z4.toJSONSchema(schema, { target: 'draft-7', ... }) without changing io or unrepresentable, so a .transform() inside a schema you pass to zodTextFormat should fail when you build the format, before any request goes out. That is the better failure of the two: loud and early. The provider can strip more Getting a keyword into JSON Schema doesn't guarantee the model is held to it. Each API supports a subset. Anthropic's structured outputs docs list numerical constraints such as minimum, maximum and multipleOf, and string constraints (minLength, maxLength), as unsupported. Their SDK helpers remove those, append each one to the field's description as text, and then validate the response against your original schema. .positive() emits exclusiveMinimum, which the docs don't name but which is a numerical constraint of the same kind. So with zodOutputFormat(), your amount > 0 rule is likely to reach Claude as text in a description rather than as a decoding constraint. Check the transformed schema on your SDK version to confirm. OpenAI's supported subset includes pattern, format, numeric bounds and anyOf, and lists allOf, not and if/then/else as unsupported. It also requires every field to be in required, with optional fields written as a union with null. Unions are their own trap: Zod's union processor emits oneOf for discriminated unions (the source comment says so), and the OpenAI helper's override callback rewrites that oneOf into anyOf before sending. If you call z.toJSONSchema() yourself and send the result to an API directly, you inherit none of those fixes. The fix: describe the rule, validate after, retry once There is no way to make a refinement representable, so the work moves to two places: telling the model the rule in words, and catching the response when it breaks the rule anyway. Start with .describe(): every rule the model has to follow goes there. Zod copies metadata into the generated JSON Schema ("All metadata fields get copied into the resulting JSON Schema", docs), so the description travels with the field. Put .describe() last in the chain. The metadata docs show .meta() returning undefined on a schema created by .refine() from a described one, because metadata belongs to one schema instance and every method returns a new one. Then move transforms out of the schema you send. Keep the wire schema plain and reshape the data after it parses. export const Refund = z .object({ orderId: z .string() .regex(/^ORD-\d{6}$/) .describe("Order id, e.g. ORD-123456"), amount: z.number().positive().describe("Greater than 0"), currency: z.enum(["EUR", "USD"]), orderDate: z.iso.date(), refundDate: z.iso .date() .describe("ISO date, on or after orderDate"), note: z .string() .refine((s) => !/https?:\/\//.test(s), { message: "note must not contain links", }) .describe("Plain text. No URLs."), sku: z.string().describe("SKU exactly as written"), }) .refine((r) => r.refundDate >= r.orderDate, { message: "refundDate is before orderDate", path: ["refundDate"], }); The refinements are still there. They are now the second line of defence instead of the only one. !A flat illustration of a loop where a document passes a checkpoint, gets a correction note attached, and goes around once more Validate the response, send the Zod error back, allow one more pass. Last, validate the response yourself and send the Zod error back for one retry. The OpenAI SDK's responses.parse does call your schema's parse (parseZodObject), but it throws on failure. Calling responses.create and running safeParse yourself gives you the error object to feed back. The model ID below is one of the current ones on OpenAI's models page; swap in yours. import OpenAI from "openai"; import { zodTextFormat } from "openai/helpers/zod"; import { z } from "zod"; const openai = new OpenAI(); type Msg = { role: "user" | "assistant"; content: string }; export async function extract<T extends z.ZodType>( schema: T, name: string, prompt: string, ): Promise<z.infer<T>> { const input: Msg[] = [{ role: "user", content: prompt }]; for (let attempt = 0; attempt < 2; attempt++) { const res = await openai.responses.create({ model: "gpt-6-luna", input, text: { format: zodTextFormat(schema, name) }, }); const raw = res.output_text; const result = schema.safeParse(JSON.parse(raw)); if (result.success) return result.data; input.push( { role: "assistant", content: raw }, { role: "user", content: "That JSON failed validation:\n" + z.prettifyError(result.error) + "\nReturn the corrected JSON only.", }, ); } throw new Error(`${name}: failed validation twice`); } Then reshape after parsing: const refund = await extract(Refund, "refund", emailText); const sku = refund.sku.trim().toUpperCase(); z.prettifyError() turns each issue into a ✖ line with the message, followed by an → at <path> line when the issue has a path. That is close to the sentence a human reviewer would write. The path you set in .refine() is what puts the field name in that message, so set it on every object-level rule. Stop at one retry: if the model fails the same rule twice with the error in front of it, a third attempt is mostly cost. Throw, log the raw output, and look at whether the rule belongs in the prompt or in your code. JSON.parse can also throw on a cut-off or refused response; in production, check the response status before you parse. --- If this was useful The gap between "the schema I wrote" and "the schema the model saw" is one of the first things that bites when you move from a demo to a feature people rely on. AI That Answers is the book in the series that covers this ground in TypeScript: prompts, structured output, validating what comes back, and keeping token cost in view while you do it.  The whole AI in TypeScript series, from your first LLM call to agents in production: xgabriel.com

JSON.parse throws on every token your LLM streams. Here's a cleaner fix. If you've ever streamed structured output from an LLM, you've met this bug: for await (const part of stream) { buffer += part.choices[0]?.delta?.content ?? ""; const data = JSON.parse(buffer); // 💥 SyntaxError on literally every chunk render(data); } A model emits JSON one token at a time. Until the very last token arrives, the buffer is syntactically broken — {"city":"Par isn't valid JSON — so JSON.parse throws on every intermediate chunk. You want to show the UI filling in as it streams; the parser wants a complete document. Let's look at the usual workarounds, why they fall short, and a small library I built to do it properly. The usual fixes (and where they hurt) 1. "Just parse at the end." Accumulate everything, parse once when the stream finishes. Correct — but you've thrown away the entire point of streaming. The user stares at a spinner while a 2 KB object generates token by token. 2. Split on }. Tempting for a streamed array, but it breaks the instant an element contains a nested object or a } inside a string. You end up re-implementing a JSON tokenizer badly. 3. Repair-then-parse (jsonrepair and friends). Close the open braces/quotes, then JSON.parse. This mostly works, but: - It re-parses the whole buffer on every chunk — that's O(n²) over a stream. On a long response it gets genuinely slow. - Repair guesses. It can briefly surface values that aren't actually in the final object. 4. The incumbent, partial-json. It's the de-facto choice (~35M downloads/month) — but it hasn't shipped a release since May 2024 and is still on 0.1.x. And the better ideas (progressive snapshots, path subscriptions, schema typing, provider adapters) are scattered across a handful of tiny single-purpose packages. These aren't hypothetical rough edges. Two real bugs from the Vercel AI SDK tracker show exactly what goes wrong when a parseability check meets a half-streamed buffer: - #12052 — a parseability check passes on a truncated tool-call argument, emitting a tool call whose input is still a half-string (Unterminated string in JSON at position 3337). - #6687 — an empty arguments string "" is treated as unparseable, so a zero-arg tool call never fires. The common thread: deciding "is this done?" by asking "does it parse yet?" is fragile. You want a parser that always returns the best valid value so far and never throws — so you can call it on every chunk without a try/catch. trickle-json trickle-json is a zero-dependency, incremental partial-JSON parser built for LLM streams. The core idea: import { parsePartial } from "trickle-json"; parsePartial('{"user":{"name":"Ja') // → { user: { name: "Ja" } } parsePartial("[1,2,3") // → [1, 2, 3] parsePartial("") // → undefined (never throws) It returns the most-complete valid value available right now. A truncated string comes back as the partial string; a member whose value hasn't arrived yet is dropped rather than guessed; empty input is undefined, not an exception. Streaming, with progressive snapshots import { StreamingJsonParser } from "trickle-json"; const parser = new StreamingJsonParser(); parser.on("snapshot", (value) => render(value)); // best-effort value after each chunk for await (const part of stream) { const delta = part.choices[0]?.delta?.content ?? ""; if (delta) parser.write(delta); } const final = parser.end(); React to one field, or to a list filling in // fires as a single field grows parser.on("path", "choices[0].message.content", (text) => appendToken(text)); // wildcards + per-element "append" events for streaming lists parser.on("path", "items[*].id", (id, segments) => console.log(segments, id)); parser.on("append", "items", (item, index) => addRow(index, item)); Provider adapters (SSE + tool calls) Parse a raw text/event-stream body directly, including tool/function-call arguments as they stream: import { openAISSEToChunks, streamOpenAIToolCalls } from "trickle-json/openai"; for await (const calls of streamOpenAIToolCalls(openAISSEToChunks(res.body))) { // calls[0].name, calls[0].arguments — arguments fill in as they arrive } There's a matching trickle-json/anthropic (text + input_json_delta tool-use), and a trickle-json/zod entry point for typed partial snapshots. The part I'm proud of: it's O(n), not O(n²) The naive approach re-parses the whole accumulated buffer on every chunk. trickle-json uses a resumable, character-at-a-time parser that keeps its state across chunks and mutates a live value tree in place — so feeding N characters total costs O(N), no matter how they're split. Streaming a growing document in small chunks (reproduce with npm run bench:stream): | records | trickle-json (incremental) | re-parse each chunk | jsonriver | |---:|---:|---:|---:| | 250 | 1.0 ms | 182 ms | 2.3 ms | | 1,000 | 2.9 ms | 2,927 ms | 5.1 ms | | 2,000 | 5.7 ms | 12,347 ms | 9.0 ms | At 2,000 records the incremental engine is ~2,150× faster than re-parsing the buffer on every chunk — the quadratic cost simply disappears. Correct, and tested like it Partial parsing is only useful if it's right. The streaming engine is verified equivalent to the one-shot parser on every prefix of a broad corpus plus 200 randomly generated documents, with a never-throw guarantee checked against an adversarial/garbage corpus. Against the incumbents on a prefix-by-prefix sweep (npm run bench): | parser | throws on truncation | no invented data | empty input | |---|:---:|:---:|:---:| | trickle-json | 0% | 99.3% | undefined | | partial-json | 0.3% | 99.4% | throws | | best-effort-json-parser | 0% | 69.2% | "" | trickle-json is the only one that never throws on truncation and returns undefined (not an exception) on empty input — so it's safe to call on every chunk. (One honest caveat shared by every snapshot parser: a number truncated mid-exponent, like 1.5e-1 as a prefix of 1.5e-10, transiently reads as 0.15. It resolves as soon as the next digit arrives.) Try it npm install trickle-json - npm: npmjs.com - GitHub: H1manshu01/trickle-json - Zero runtime dependencies, ~2.6 kB min+gzip, ESM + CJS, full types, published with provenance. If you're building streaming AI UIs, give it a spin — and if it breaks on some input, open an issue with the string; the never-throw guarantee is the whole point, so I want to know. ⭐ appreciated if it saves you a try/catch.