Claude Reads Screenshots at 2,576px Now. The Same PNG Can Cost 3x the Tokens
- Book: AI That Answers
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
In April, Simon Willison fed one 3456×2234, 3.7 MB PNG screenshot into his Claude token counter. On Claude Opus 4.6 it came to 1,578 tokens. On Opus 4.7 the same file came to 4,744. That is a 3.01x jump for one image, at the same $5 per million input tokens on both models.
Same file, different model. Opus 4.7 accepts images up to 2,576 pixels on the long edge, about 3.75 megapixels, where earlier Claude models shrank anything bigger to fit 1,568. Anthropic's vision docs now put every model from Claude 4.7 on in that high-resolution tier, with no opt-in. If your app sends screenshots, the per-image cost you estimated on an older model can be a third of what the same PNG costs on a current one.
Small images didn't move. In the same post, a 682×318 image cost 314 tokens on 4.7 and 310 on 4.6. The extra cost lives only in big images, and the rule that decides it fits in one line of arithmetic.
Claude bills images in 28-pixel tiles
Claude doesn't price pixels. The vision docs say it cuts the image into 28×28-pixel patches and charges one visual token per patch: ceil(width / 28) × ceil(height / 28). A 1000×1000 image is 36 × 36 = 1,296 tokens.
Then there is a ceiling. Each model has a long-edge limit and a visual-token budget, and an image over either one is shrunk, aspect ratio kept, until it fits:
| Tier | Models | Max long edge | Max visual tokens |
|---|---|---|---|
| High-resolution | Claude 4.7 and later | 2,576 px | 4,784 |
| Standard | All other models | 1,568 px | 1,568 |
The token budget is the limit that bites. Anthropic's coordinates guide says that for nearly all photos and screenshots the token limit sets the final size, not the edge: a 1920×1080 screenshot on a standard-tier model becomes 1456×819, not 1568×882. On a high-resolution model the same screenshot isn't resized at all and costs 2,691 tokens instead of 1,560.
Run Simon's screenshot through those rules with the guide's reference resize function and his numbers line up. On the standard tier, 3456×2234 shrinks to 1372×887: 49 × 32 = 1,568 patches, exactly the budget. On the high-resolution tier it shrinks to 2383×1540: 86 × 55 = 4,730 patches. That ratio is 3.02x against his measured 3.01x. This is my arithmetic, not a measurement; the 10 to 14 tokens his counts carry on top are most likely the rest of the request, which a token counter includes.
!A laptop screenshot overlaid with a fine grid of square tiles, one tile lifted out under a magnifying glass
Claude bills per square tile, so a longer edge means more tiles on the bill.
Where the /750 shortcut drifts
The older shortcut, still quoted in write-ups like fast.io, is width × height / 750, which that write-up calls Anthropic's established rule of thumb. Since 28 × 28 is 784, the two agree within a few percent on mid-sized images: a 1000×1000 image is 1,333 by the shortcut and 1,296 by patches. It drifts in three places.
It ignores the ceiling. 3456×2234 / 750 is 10,294 tokens, a number neither tier will ever bill, because the image is shrunk first. That estimate is about 2x too high for a high-resolution model and over 6x too high for a standard one.
It ignores rounding. Patches round up on each axis. 1120×700 is exactly 40 × 25 = 1,000 tokens. Add one pixel to each side, 1121×701, and it becomes 41 × 26 = 1,066. The shortcut moves by about 2.
It doesn't know your model's tier. Same image, same formula, up to 3x apart depending on the model ID in the request.
What doesn't matter at all is file size. Nothing in the formula looks at bytes, and the coordinates guide notes that the token counting endpoint estimates an image's cost from its dimensions without fully processing it. A 3.7 MB PNG and a 300 KB JPEG at the same dimensions cost the same tokens. Compression only changes latency and request size, and the vision docs warn that lossy JPEG or WebP can add artifacts that make text hard to read, which is the opposite of what you want from a screenshot. File size still matters for limits: each image is capped at 10 MB base64-encoded on the Claude API and 5 MB on Amazon Bedrock.
Count it yourself in TypeScript
The token counting endpoint is free, rate-limited per usage tier (5,000 requests a minute on the Start tier), and works with every active model. It only counts images sent as base64, not by URL or file ID. That is enough to print your own table.
The script below uses the Anthropic TypeScript SDK and sharp. It resizes one screenshot to several long edges, counts each on two models, subtracts a text-only baseline so only the image is left, and prints tokens and dollars per thousand images. I haven't run this script myself, so treat the numbers it prints as the source of truth, not the ones in this post.
// count-image.mts
// npm i @anthropic-ai/sdk sharp
// ANTHROPIC_API_KEY=... npx tsx count-image.mts shot.png
import Anthropic from "@anthropic-ai/sdk";
import sharp from "sharp";
import { readFile } from "node:fs/promises";
const client = new Anthropic();
const MODELS = [
{ id: "claude-opus-4-6", usdPerMTok: 5 },
{ id: "claude-opus-4-7", usdPerMTok: 5 },
];
// long-edge targets in px; 0 = send the original size
const EDGES = [0, 2576, 1568, 1092, 768];
const PROMPT = "What is on this screen?";
const patches = (w: number, h: number) =>
Math.ceil(w / 28) * Math.ceil(h / 28);
async function count(model: string, png?: Buffer) {
const text = { type: "text" as const, text: PROMPT };
const content = png
? [
{
type: "image" as const,
source: {
type: "base64" as const,
media_type: "image/png" as const,
data: png.toString("base64"),
},
},
text,
]
: [text];
const res = await client.messages.countTokens({
model,
messages: [{ role: "user", content }],
});
return res.input_tokens;
}
Resize with sharp to each target long edge, never upscaling:
async function atEdge(src: Buffer, edge: number) {
const img = sharp(src);
if (edge > 0) {
img.resize({
width: edge,
height: edge,
fit: "inside",
withoutEnlargement: true,
});
}
return img.png().toBuffer();
}
The main loop counts every size on both models and prints the table:
const src = await readFile(process.argv[2]);
const rows: Record<string, unknown>[] = [];
for (const m of MODELS) {
const baseline = await count(m.id);
for (const edge of EDGES) {
const png = await atEdge(src, edge);
const meta = await sharp(png).metadata();
const w = meta.width ?? 0;
const h = meta.height ?? 0;
const tokens = (await count(m.id, png)) - baseline;
rows.push({
model: m.id,
sent: `${w}x${h}`,
kb: Math.round(png.length / 1024),
predicted: patches(w, h),
counted: tokens,
usdPer1k: ((tokens * m.usdPerMTok) / 1000).toFixed(2),
});
}
}
console.table(rows);
The kb column will swing a lot between rows and the counted column won't follow it, because bytes aren't in the formula. And predicted is the patch count of the pixels you sent: when counted lands on or near it, Claude saw your image at the size you sent. When counted is well below it, the server shrank the image to its tier's budget, and you paid for pixels you could have dropped yourself. For a 3456×2234 shot on Opus 4.7, the original and the 2,576 rows should both land near 4,730, the size the server shrinks this shot to. A 2576×1665 image is still 92 × 60 = 5,520 patches, over the 4,784 budget.
Pick the size before you ship
Here is what one screenshot's image tokens cost per thousand requests, computed from the docs' token table and each model's list input price. It leaves out your prompt text and the output tokens.
| Model | Tier | Input $/MTok | 1920×1080 tokens | $ per 1k | 3840×2160 tokens | $ per 1k |
|---|---|---|---|---|---|---|
| Haiku 4.5 | standard | $1 | 1,560 | $1.56 | 1,560 | $1.56 |
| Sonnet 4.6 | standard | $3 | 1,560 | $4.68 | 1,560 | $4.68 |
| Opus 4.6 | standard | $5 | 1,560 | $7.80 | 1,560 | $7.80 |
| Opus 4.7 | high | $5 | 2,691 | $13.46 | 4,784 | $23.92 |
| Haiku 5.5 (≤100K prompt) | high | $0.10 | 2,691 | $0.27 | 4,784 | $0.48 |
| Sonnet 5.5 | high | $2 | 2,691 | $5.38 | 4,784 | $9.57 |
| Opus 5.5 | high | $4 | 2,691 | $10.76 | 4,784 | $19.14 |
The Sonnet rows are the ones to stare at. Sonnet 5.5's input price is a third lower than Sonnet 4.6's, and a 4K screenshot costs about twice as much on it. Each token got cheaper, but the same image now takes about three times as many of them.
!A balance scale with two identical screenshots, one side weighed down by a much taller stack of coins
Same screenshot, same file: the model's resolution tier decides how heavy the bill is.
If you don't need the extra detail, shrink before you send. Anthropic's docs say it directly: downsample when you don't need the fidelity, and resize yourself to cut latency. Copy resizedSize from the coordinates guide (it ships a TypeScript version) and pin screenshots to the old standard budget on any model:
// resizedSize() is Anthropic's reference function,
// copied from the vision-coordinates guide.
const STANDARD = { edge: 1568, tokens: 1568 };
async function pinToStandard(src: Buffer) {
const meta = await sharp(src).metadata();
const [w, h] = resizedSize(
meta.width ?? 0,
meta.height ?? 0,
STANDARD.edge,
STANDARD.tokens,
);
return sharp(src)
.resize({ width: w, height: h, fit: "fill" })
.png()
.toBuffer();
}
Don't do it blind, though. The same docs say the high-resolution tier exists for computer use, screenshot understanding and dense documents, and small text is the first thing a downscale eats. Run your real screenshots through both sizes, check the answers, and only then pick the budget.
Screenshot apps hit a few more rules. A request with more than 20 images triggers a stricter per-image size limit, and the vision docs say to keep both sides at 3,000 px or under to stay clear on every platform. Screenshots returned to the computer use and browser use tools are rejected with a validation error when they exceed the model's limits instead of being downscaled. And if a silent server-side resize would break your pipeline, set "transformations": { "oversized_image": "error" } on the image block: the API, and the token counting endpoint, then reject the image instead of shrinking it, per the coordinates guide.
The habit that saves you is boring. Before a feature that takes screenshots ships, count one real screenshot on the exact model ID you'll deploy, write the per-image number next to the feature, and recount when you switch models.
---
If this was useful
Per-request token cost is one of the first things a working LLM app has to get right, and it's a big part of what AI That Answers covers: your first LLM app in TypeScript, from prompts and structured output to knowing what each call costs before your users find out for you. If you want the screenshot feature above to ship with a budget instead of a surprise, that book is where I'd start.
It's book one of AI in TypeScript, five books from your first LLM call to agents in production: the whole series is here.
1 reaction · 2 comments
Evaluating AI agents: small evals in TypeScript
Evaluating AI agents quickly leads to an uncomfortable question: how do you actually notice that your agent got worse after a change? A new system prompt, a model switch or a reworded tool description is enough, and suddenly the assistant reserves books that are sitting on the shelf. Without tests, you only find out when someone complains.
For regular code the answer is simple: unit tests. With agents that only half works, because the same model answers the same question differently from run to run. That is exactly what evals are for: repeatable checks that give an agent real tasks and grade the result. In this article I build a small eval suite in TypeScript with you, from an empty folder to a run in CI. I deliberately skip frameworks so you can see what happens under the hood.
> In short
>
> Evals give an AI agent real tasks in a fresh test environment and then grade the outcome, not the path that led there. To get started, 20 to 50 tasks taken from real failures, several trials per task and the pass^k metric for everything that has to work reliably are enough. The biggest pitfall is a broken grader, which is why you should read the transcripts regularly.
What you will learn in this article
- What task, trial, grader and transcript mean in agent evals
- How to build a small eval harness in TypeScript without a framework
- When a code grader is enough and when a model judge makes sense
- Why pass@k and pass^k tell such different stories
- How regression evals in CI catch problems before a release
Why evaluating AI agents matters
An agent has three parts: the model, a loop that connects the model to its tools, and the tools themselves. You turn all three knobs in everyday work. You swap the model because a newer one is faster or cheaper. You refine the system prompt. On top of that you polish tool descriptions, as I showed in How to write tool descriptions for AI agents. Each of these changes can improve one thing and break another.
Without evals, all you have is the gut-feeling test: try three or four prompts by hand, the answer looks fine, done. Anthropic's engineering team sums up the value of evals in one sentence: “Evals make problems and behavioral changes visible before they affect users.” That is the whole point. You want to see the regression before your users feel it.
There is also a quirk of language models. A single run tells you little because answers vary. If a task succeeds once, you do not know whether it succeeds nine times out of ten or three. Evaluating AI agents therefore means running every task several times and computing metrics from the results. More on that later.
The key terms in agent evals
Before we write code, a shared vocabulary helps. I stick to the terms from Anthropic's article “Demystifying evals for AI agents” because they map well onto your own code. The table also shows where you will find each term in the example.
| Term | Meaning | In the example |
| --- | --- | --- |
| Task | A single check: the input plus the criterion for passing it | an entry in tasks.ts |
| Trial | One single run of a task | one call to runTrial() |
| Grader | Logic that scores one aspect of the result | functions in graders.ts |
| Transcript | The full record of a trial with all messages and tool calls | one line in results/*.jsonl |
| Outcome | The state of the environment after the trial | the reservations in the library |
| Eval harness | The infrastructure that runs, grades and summarizes tasks | runner.ts |
| Agent harness | The loop that turns the model into an agent | agent.ts |
| Eval suite | A set of related tasks | all tasks of the lending assistant |
Keep two kinds of tasks apart from day one. Capability evals ask what the agent already does well. They may start with low pass rates, because they show you where you want to go. Regression evals, on the other hand, ask whether the agent can still do everything it could do before. They should sit close to 100 percent. Once a capability task reaches consistently high scores, it moves into the regression suite.
A lending assistant as the example
The example is a made-up lending assistant for a city library. It has three tools: search_books searches the catalog, check_availability shows free copies, and reserve_book creates a reservation. Two business rules come on top. A reservation is only made when no copy is available. And if the member number is missing, the agent asks for it instead of guessing.
What I like about this example: whether the agent did its job is visible in the state of the library. Is the right reservation there after the trial? Is there no wrong one? That is exactly the kind of outcome code can check reliably. This way you can evaluate AI agents without needing a human or a second model for every answer. The following diagram shows how a trial runs.
> 🎬 How one trial runs in the eval harness – this animated diagram only runs on my site: watch it on sascha-ms.de
Prerequisites
You need Node.js 22 or later and npm. For real runs you need either an API key for the Claude API or an AWS account with access to a Claude model in Amazon Bedrock. The tests for the harness itself run without a key and at no cost. Basic TypeScript is all you need; experience with agent frameworks is not required.
Building the eval suite step by step
1. Create the project
Create a new folder and install the Anthropic SDK, the Bedrock SDK and, as development tools, TypeScript and tsx. With tsx you run TypeScript files directly without compiling them first.
mkdir ai-agent-evals && cd ai-agent-evals
npm init -y
npm install @anthropic-ai/sdk@1.3.0 @anthropic-ai/bedrock-sdk@1.1.0
npm install -D typescript@7.0.2 tsx@4.23.15 @types/node@22
Then replace package.json with this version. The important parts are "type": "module" and the three scripts.
{
"name": "ai-agent-evals",
"version": "1.0.0",
"private": true,
"type": "module",
"scripts": {
"typecheck": "tsc --noEmit",
"test": "node --import tsx --test test/*.test.ts",
"evals": "tsx src/run-evals.ts"
},
"dependencies": {
"@anthropic-ai/bedrock-sdk": "1.1.0",
"@anthropic-ai/sdk": "1.3.0"
},
"devDependencies": {
"@types/node": "^22.20.0",
"tsx": "4.23.15",
"typescript": "7.0.2"
}
}
// tsconfig.json
{
"compilerOptions": {
"target": "ES2023",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"noEmit": true,
"skipLibCheck": true,
"allowImportingTsExtensions": true,
"types": ["node"]
},
"include": ["src", "test"]
}
2. A test environment with a fresh state
The library lives purely in memory. That is on purpose: every trial gets a new, clean state from createLibrary(). If trials shared a database, a reservation from trial 1 could skew the result of trial 2. Errors like that are especially nasty because they look random.
// src/library.ts
// A small, made-up city library as the test environment for the agent.
// Every trial gets a fresh state from createLibrary().
export interface Book {
id: string;
title: string;
author: string;
copies: number;
}
export interface Reservation {
bookId: string;
memberId: string;
}
export interface LibraryState {
books: Book[];
loans: Record<string, number>; // copies on loan per book ID
reservations: Reservation[];
}
export function createLibrary(): LibraryState {
return {
books: [
{ id: 'b-101', title: 'The Lighthouse on the Dyke', author: 'Jana Thiel', copies: 2 },
{ id: 'b-102', title: 'Cooking with Kids', author: 'Ole Brandt', copies: 1 },
{ id: 'b-103', title: 'TypeScript for Beginners', author: 'Mira Okafor', copies: 3 },
{ id: 'b-104', title: 'The Garden in Winter', author: 'Paul Seidel', copies: 1 },
],
loans: { 'b-101': 2, 'b-102': 0, 'b-103': 1, 'b-104': 1 },
reservations: [],
};
}
export function availableCopies(state: LibraryState, bookId: string): number {
const book = state.books.find((b) => b.id === bookId);
if (!book) return 0;
return book.copies - (state.loans[bookId] ?? 0);
}
3. The agent's tools
The tools validate their own input and answer problems with an understandable error message. That way the agent can correct itself instead of getting stuck. Note that reserve_book refuses to reserve an available book. The eval still checks whether the agent even tries, because a tool in a real system is not always that strict.
// src/tools.ts
// Three tools the agent may use. Their descriptions are part of what
// your evals test as well.
import { availableCopies, type LibraryState } from './library.ts';
export interface ToolDefinition {
name: string;
description: string;
input_schema: {
type: 'object';
properties: Record<string, unknown>;
required: string[];
};
}
export interface ToolResult {
content: string;
isError: boolean;
}
export const tools: ToolDefinition[] = [
{
name: 'search_books',
description:
'Searches the city library catalog by title or author. ' +
'Returns book IDs (e.g. "b-101") that the other tools need.',
input_schema: {
type: 'object',
properties: { query: { type: 'string', description: 'Search term, e.g. part of the title' } },
required: ['query'],
},
},
{
name: 'check_availability',
description: 'Shows how many copies of a book can be borrowed right now. Book ID from search_books.',
input_schema: {
type: 'object',
properties: { bookId: { type: 'string' } },
required: ['bookId'],
},
},
{
name: 'reserve_book',
description:
'Reserves a book for a member. Only use it when no copy is available ' +
'and the user has given their member number (format "M-<number>"). Never guess.',
input_schema: {
type: 'object',
properties: {
bookId: { type: 'string' },
memberId: { type: 'string', description: 'Member number, e.g. "M-17"' },
},
required: ['bookId', 'memberId'],
},
},
];
export function executeTool(state: LibraryState, name: string, input: unknown): ToolResult {
const args = (input ?? {}) as Record<string, string>;
switch (name) {
case 'search_books': {
const query = String(args.query ?? '').toLowerCase();
const hits = state.books.filter(
(b) => b.title.toLowerCase().includes(query) || b.author.toLowerCase().includes(query),
);
return { content: JSON.stringify(hits.map(({ id, title, author }) => ({ id, title, author }))), isError: false };
}
case 'check_availability': {
if (!state.books.some((b) => b.id === args.bookId)) {
return { content: `Unknown book ID "${args.bookId}". Use search_books first.`, isError: true };
}
return { content: JSON.stringify({ bookId: args.bookId, available: availableCopies(state, args.bookId) }), isError: false };
}
case 'reserve_book': {
if (!state.books.some((b) => b.id === args.bookId)) {
return { content: `Unknown book ID "${args.bookId}".`, isError: true };
}
if (!/^M-\d+$/.test(args.memberId ?? '')) {
return { content: 'Invalid member number. Please ask the user.', isError: true };
}
if (availableCopies(state, args.bookId) > 0) {
return { content: 'The book is available, no reservation needed.', isError: true };
}
const exists = state.reservations.some((r) => r.bookId === args.bookId && r.memberId === args.memberId);
if (exists) {
return { content: 'This member already has a reservation.', isError: false };
}
state.reservations.push({ bookId: args.bookId, memberId: args.memberId });
return { content: 'Reservation created.', isError: false };
}
default:
return { content: `Unknown tool "${name}".`, isError: true };
}
}
4. The agent loop
The loop is deliberately simple. It sends the history to the model, runs the requested tools, appends the results and repeats until the model stops calling tools or the maximum number of turns is reached. Along the way it collects everything for the transcript: messages, tool calls, turns and tokens.
The trick for testability lies in the ModelClient type. The loop only knows a function that takes a request and returns a response. Whether the Claude API, Amazon Bedrock or a test script sits behind it does not matter to the loop.
// src/agent.ts
// A lean agent loop: ask the model, run tools, return the results
// until the model is done. Everything ends up in the transcript.
import type { LibraryState } from './library.ts';
import { executeTool, tools, type ToolDefinition } from './tools.ts';
export type ContentBlock =
| { type: 'text'; text: string }
| { type: 'tool_use'; id: string; name: string; input: unknown }
| { type: 'tool_result'; tool_use_id: string; content: string; is_error: boolean };
export interface Message {
role: 'user' | 'assistant';
content: string | ContentBlock[];
}
export interface ModelRequest {
system: string;
messages: Message[];
tools: ToolDefinition[];
max_tokens: number;
}
export interface ModelResponse {
content: ContentBlock[];
stop_reason: string | null;
usage: { input_tokens: number; output_tokens: number };
}
// Any function with this signature can be the model: the Anthropic API,
// Amazon Bedrock or, in tests, a script with fixed answers.
export type ModelClient = (request: ModelRequest) => Promise<ModelResponse>;
export interface ToolCall {
name: string;
input: unknown;
isError: boolean;
}
export interface Transcript {
messages: Message[];
toolCalls: ToolCall[];
finalText: string;
turns: number;
inputTokens: number;
outputTokens: number;
stopReason: string | null;
}
export const SYSTEM_PROMPT =
'You are the lending assistant of the city library. Help members find, ' +
'check and reserve books. Use the tools instead of guessing. If information ' +
'such as the member number is missing, ask for it. Answer briefly and kindly.';
export async function runAgent(
client: ModelClient,
state: LibraryState,
prompt: string,
maxTurns = 8,
): Promise<Transcript> {
const messages: Message[] = [{ role: 'user', content: prompt }];
const transcript: Transcript = {
messages,
toolCalls: [],
finalText: '',
turns: 0,
inputTokens: 0,
outputTokens: 0,
stopReason: null,
};
while (transcript.turns < maxTurns) {
const response = await client({ system: SYSTEM_PROMPT, messages, tools, max_tokens: 1024 });
transcript.turns += 1;
transcript.inputTokens += response.usage.input_tokens;
transcript.outputTokens += response.usage.output_tokens;
transcript.stopReason = response.stop_reason;
messages.push({ role: 'assistant', content: response.content });
const text = response.content
.filter((b): b is Extract<ContentBlock, { type: 'text' }> => b.type === 'text')
.map((b) => b.text)
.join('\n')
.trim();
if (text) transcript.finalText = text;
if (response.stop_reason !== 'tool_use') break;
const results: ContentBlock[] = [];
for (const block of response.content) {
if (block.type !== 'tool_use') continue;
const result = executeTool(state, block.name, block.input);
transcript.toolCalls.push({ name: block.name, input: block.input, isError: result.isError });
results.push({ type: 'tool_result', tool_use_id: block.id, content: result.content, is_error: result.isError });
}
messages.push({ role: 'user', content: results });
}
return transcript;
}
The adapters for real models are short because both SDKs offer the same messages.create() method. The Bedrock client uses your regular AWS credentials and the region from AWS_REGION.
// src/model-clients.ts
// Adapters for real models. The agent loop only knows ModelClient.
import Anthropic from '@anthropic-ai/sdk';
import { AnthropicBedrock } from '@anthropic-ai/bedrock-sdk';
import type { ContentBlock, ModelClient } from './agent.ts';
// Both SDK clients offer messages.create() with the same shape.
interface MessagesApi {
messages: {
create(params: Anthropic.MessageCreateParamsNonStreaming): PromiseLike<Anthropic.Message>;
};
}
export function messagesClient(sdk: MessagesApi, model: string): ModelClient {
return async (request) => {
const response = await sdk.messages.create({
model,
max_tokens: request.max_tokens,
system: request.system,
tools: request.tools,
messages: request.messages as Anthropic.MessageParam[],
});
return {
// We only pass on text and tool blocks; the loop ignores everything else.
content: response.content.filter((b) => b.type === 'text' || b.type === 'tool_use') as ContentBlock[],
stop_reason: response.stop_reason,
usage: {
input_tokens: response.usage.input_tokens ?? 0,
output_tokens: response.usage.output_tokens ?? 0,
},
};
};
}
// Anthropic API: reads ANTHROPIC_API_KEY from the environment.
export function anthropicClient(model: string): ModelClient {
return messagesClient(new Anthropic(), model);
}
// Amazon Bedrock: uses your regular AWS credentials and the region from AWS_REGION.
export function bedrockClient(model: string): ModelClient {
return messagesClient(new AnthropicBedrock(), model);
}
5. Tasks and graders
Now comes the part that takes the most thinking. A grader receives the transcript and the final state of the library and returns “pass” or “fail” with a short reason. Most graders here are code graders. They are fast, free and always return the same verdict for the same input.
For soft criteria such as tone and clarity there is llmJudge. Here a model grades the answer against one criterion. The “unknown” option matters: a judge that is allowed to admit uncertainty invents verdicts less often. I count “unknown” as a fail, so it automatically ends up in manual review.
// src/graders.ts
// Graders check the outcome of a trial. Code graders are fast and
// reproducible; the model grader is meant for soft criteria such as tone.
import type { ModelClient, Transcript } from './agent.ts';
import type { LibraryState } from './library.ts';
export interface GradeResult {
grader: string;
pass: boolean;
detail: string;
}
export type Grader = (run: Transcript, state: LibraryState) => GradeResult | Promise<GradeResult>;
// Check the outcome: does the reservation really exist in the library state?
export const reservationExists =
(bookId: string, memberId: string): Grader =>
(_run, state) => {
const pass = state.reservations.some((r) => r.bookId === bookId && r.memberId === memberId);
return { grader: `reservationExists(${bookId}, ${memberId})`, pass, detail: JSON.stringify(state.reservations) };
};
export const reservationCount =
(expected: number): Grader =>
(_run, state) => ({
grader: `reservationCount(${expected})`,
pass: state.reservations.length === expected,
detail: `found: ${state.reservations.length}`,
});
// Counter-check: a tool must not be called at all in this task.
export const toolNotCalled =
(name: string): Grader =>
(run) => {
const calls = run.toolCalls.filter((c) => c.name === name).length;
return { grader: `toolNotCalled(${name})`, pass: calls === 0, detail: `calls: ${calls}` };
};
export const toolCalled =
(name: string): Grader =>
(run) => {
const calls = run.toolCalls.filter((c) => c.name === name && !c.isError).length;
return { grader: `toolCalled(${name})`, pass: calls > 0, detail: `successful calls: ${calls}` };
};
// Transcript metric: the agent should not run in circles forever.
export const maxTurns =
(limit: number): Grader =>
(run) => ({ grader: `maxTurns(${limit})`, pass: run.turns <= limit, detail: `turns: ${run.turns}` });
export const asksQuestion = (): Grader => (run) => ({
grader: 'asksQuestion',
pass: run.finalText.includes('?'),
detail: run.finalText.slice(0, 120),
});
// Model as judge: grades a soft criterion and may answer "unknown".
// An "unknown" counts as a fail and belongs in manual review.
export const llmJudge =
(judge: ModelClient, criterion: string): Grader =>
async (run) => {
const response = await judge({
system:
'You grade the answer of a library assistant. Reply with JSON only: ' +
'{"verdict":"pass"|"fail"|"unknown","reason":"<one sentence>"}. Use "unknown" if you are unsure.',
messages: [{ role: 'user', content: `Criterion: ${criterion}\n\nAssistant answer:\n${run.finalText}` }],
tools: [],
max_tokens: 200,
});
const text = response.content.map((b) => (b.type === 'text' ? b.text : '')).join('');
try {
const parsed = JSON.parse(text.slice(text.indexOf('{'), text.lastIndexOf('}') + 1)) as {
verdict?: string;
reason?: string;
};
return { grader: `llmJudge(${criterion})`, pass: parsed.verdict === 'pass', detail: `${parsed.verdict}: ${parsed.reason ?? ''}` };
} catch {
return { grader: `llmJudge(${criterion})`, pass: false, detail: `unreadable: ${text.slice(0, 80)}` };
}
};
The tasks cover both directions. Some cases require the agent to reserve, others forbid it outright. If you only test positive cases, you end up with an agent that acts too eagerly. The task “two-books-at-once” is a real capability task: the agent has to figure out on its own how to search the whole catalog.
// src/tasks.ts
// The tasks of the eval suite. Each task has a clear request, a
// starting state and graders that check the outcome.
import type { LibraryState } from './library.ts';
import { asksQuestion, maxTurns, reservationCount, reservationExists, toolCalled, toolNotCalled, type Grader } from './graders.ts';
export interface Task {
id: string;
// capability: what should the agent be able to do? regression: what must never break again?
kind: 'capability' | 'regression';
prompt: string;
setup?: (state: LibraryState) => void;
graders: Grader[];
}
export const tasks: Task[] = [
{
id: 'reserve-when-unavailable',
kind: 'regression',
prompt: 'Hi, I am member M-17. Please reserve “The Lighthouse on the Dyke” for me.',
graders: [reservationExists('b-101', 'M-17'), reservationCount(1), maxTurns(5)],
},
{
id: 'no-reservation-when-available',
kind: 'regression',
prompt: 'Member M-17 here. Please reserve “Cooking with Kids” for me.',
graders: [reservationCount(0), toolNotCalled('reserve_book')],
},
{
id: 'member-number-missing',
kind: 'regression',
prompt: 'Can you reserve “The Garden in Winter” for me?',
graders: [reservationCount(0), toolNotCalled('reserve_book'), asksQuestion()],
},
{
id: 'unknown-book',
kind: 'capability',
prompt: 'I am M-8 and would like to reserve “Advanced Sailing”.',
graders: [reservationCount(0), toolNotCalled('reserve_book')],
},
{
id: 'availability-by-author',
kind: 'capability',
prompt: 'Do you have anything by Ole Brandt? Can I borrow it today?',
graders: [toolCalled('check_availability'), reservationCount(0)],
},
{
id: 'two-books-at-once',
kind: 'capability',
prompt: 'I am M-23. Please reserve everything that is fully on loan right now.',
graders: [reservationExists('b-101', 'M-23'), reservationExists('b-104', 'M-23'), reservationCount(2)],
},
];
6. Metrics and harness
The harness runs every task several times, each time in a fresh library, with limited concurrency. That keeps you below the API rate limits. If a trial aborts with an error, for example a timeout, the harness flags it as an infrastructure error. This matters so you do not mistake a network error for an agent error.
Besides “pass” there is partial credit: the share of graders that passed. An agent that creates the reservation correctly but needs seven turns for it is better than one that gets nothing done. I explain the pass@k and pass^k metrics in detail in the next section.
// src/metrics.ts
// pass@k: how likely is at least one success in k trials?
// pass^k: how likely are all k trials successful?
// Both are estimated from n trials with c successes.
function binomial(n: number, k: number): number {
if (k < 0 || k > n) return 0;
let result = 1;
for (let i = 1; i <= k; i++) result = (result * (n - k + i)) / i;
return result;
}
export function passAtK(n: number, c: number, k: number): number {
if (k > n) throw new Error(`k (${k}) must not be greater than n (${n})`);
if (n - c < k) return 1;
return 1 - binomial(n - c, k) / binomial(n, k);
}
export function passHatK(n: number, c: number, k: number): number {
if (k > n) throw new Error(`k (${k}) must not be greater than n (${n})`);
return binomial(c, k) / binomial(n, k);
}
// src/runner.ts
// The eval harness: runs every task several times in a fresh library,
// grades the trials and summarizes them.
import { runAgent, type ModelClient, type Transcript } from './agent.ts';
import type { Grader, GradeResult } from './graders.ts';
import { createLibrary } from './library.ts';
import { passAtK, passHatK } from './metrics.ts';
import type { Task } from './tasks.ts';
export interface TrialResult {
taskId: string;
trial: number;
pass: boolean;
score: number; // share of passed graders, for partial credit
grades: GradeResult[];
transcript?: Transcript;
error?: string;
}
export interface TaskSummary {
taskId: string;
kind: Task['kind'];
trials: number;
successes: number;
avgScore: number;
passAtK: number;
passHatK: number;
avgTurns: number;
avgTokens: number;
}
export interface EvalOptions {
trials: number;
k: number;
concurrency: number;
extraGraders?: Grader[];
}
async function runTrial(client: ModelClient, task: Task, trial: number, extra: Grader[]): Promise<TrialResult> {
const state = createLibrary(); // every trial starts clean
task.setup?.(state);
try {
const transcript = await runAgent(client, state, task.prompt);
const grades: GradeResult[] = [];
for (const grader of [...task.graders, ...extra]) grades.push(await grader(transcript, state));
const passed = grades.filter((g) => g.pass).length;
return { taskId: task.id, trial, pass: passed === grades.length, score: passed / grades.length, grades, transcript };
} catch (error) {
// An infrastructure error is not the agent's fault, so it is flagged separately.
return { taskId: task.id, trial, pass: false, score: 0, grades: [], error: String(error) };
}
}
async function pool<T>(jobs: Array<() => Promise<T>>, concurrency: number): Promise<T[]> {
const results: T[] = new Array(jobs.length);
let next = 0;
const workers = Array.from({ length: Math.min(concurrency, jobs.length) }, async () => {
while (next < jobs.length) {
const index = next++;
results[index] = await jobs[index]();
}
});
await Promise.all(workers);
return results;
}
export async function runEvals(client: ModelClient, tasks: Task[], options: EvalOptions) {
const jobs = tasks.flatMap((task) =>
Array.from({ length: options.trials }, (_, trial) => () => runTrial(client, task, trial + 1, options.extraGraders ?? [])),
);
const results = await pool(jobs, options.concurrency);
const summaries: TaskSummary[] = tasks.map((task) => {
const own = results.filter((r) => r.taskId === task.id);
const successes = own.filter((r) => r.pass).length;
const withTranscript = own.filter((r) => r.transcript);
const avg = (values: number[]) => (values.length ? values.reduce((a, b) => a + b, 0) / values.length : 0);
return {
taskId: task.id,
kind: task.kind,
trials: own.length,
successes,
avgScore: avg(own.map((r) => r.score)),
passAtK: passAtK(own.length, successes, options.k),
passHatK: passHatK(own.length, successes, options.k),
avgTurns: avg(withTranscript.map((r) => r.transcript!.turns)),
avgTokens: avg(withTranscript.map((r) => r.transcript!.inputTokens + r.transcript!.outputTokens)),
};
});
return { results, summaries };
}
The command-line script ties everything together. It stores every trial as a line in a JSONL file, prints a table and sets an exit code. Exit code 1 means a regression task is below 90 percent. Exit code 2 means the run is invalid because trials failed on infrastructure.
// src/run-evals.ts
// Usage: EVAL_MODEL=<model-id> npm run evals
// Optional: EVAL_PROVIDER=bedrock, EVAL_TRIALS=5, EVAL_K=3, EVAL_JUDGE_MODEL=<model-id>
import { mkdir, writeFile } from 'node:fs/promises';
import { llmJudge } from './graders.ts';
import { anthropicClient, bedrockClient } from './model-clients.ts';
import { runEvals } from './runner.ts';
import { tasks } from './tasks.ts';
const model = process.env.EVAL_MODEL;
if (!model) {
console.error('Please set EVAL_MODEL, e.g. EVAL_MODEL=claude-sonnet-5-5');
process.exit(2);
}
const makeClient = process.env.EVAL_PROVIDER === 'bedrock' ? bedrockClient : anthropicClient;
const client = makeClient(model);
const trials = Number(process.env.EVAL_TRIALS ?? 5);
const k = Number(process.env.EVAL_K ?? 3);
const judgeModel = process.env.EVAL_JUDGE_MODEL;
const extraGraders = judgeModel
? [llmJudge(makeClient(judgeModel), 'The answer is friendly, factually correct and tells the member the next step.')]
: [];
const { results, summaries } = await runEvals(client, tasks, { trials, k, concurrency: 3, extraGraders });
// Save all transcripts: reading them is the most important part of the analysis.
await mkdir('results', { recursive: true });
const file = `results/${new Date().toISOString().replace(/[:.]/g, '-')}.jsonl`;
await writeFile(file, results.map((r) => JSON.stringify(r)).join('\n') + '\n');
console.table(
summaries.map((s) => ({
Task: s.taskId,
Kind: s.kind,
Successes: `${s.successes}/${s.trials}`,
Partial: s.avgScore.toFixed(2),
[`pass@${k}`]: s.passAtK.toFixed(2),
[`pass^${k}`]: s.passHatK.toFixed(2),
Turns: s.avgTurns.toFixed(1),
Tokens: Math.round(s.avgTokens),
})),
);
console.log(`Transcripts: ${file}`);
const infraErrors = results.filter((r) => r.error).length;
if (infraErrors > 0) {
// A run with API errors says nothing about the agent: better invalid than falsely red.
console.error(`${infraErrors} trials with infrastructure errors, run invalid. See ${file}`);
process.exit(2);
}
// Regression tasks must (almost) always pass, otherwise the run fails.
const broken = summaries.filter((s) => s.kind === 'regression' && s.successes / s.trials < 0.9);
if (broken.length > 0) {
console.error(`Regression: ${broken.map((s) => s.taskId).join(', ')}`);
process.exit(1);
}
7. Testing the harness itself
An eval harness is code and can have bugs. A broken grader is even particularly dangerous, because it gives you false confidence. That is why I test the harness with a canned model: the scriptedClient returns predefined answers in order. With it you check that a good solution passes and a bad one fails. The tests for the metrics secure the calculations.
// test/helpers.ts
// A model for tests: returns predefined answers in order.
import type { ContentBlock, ModelClient, ModelRequest, ModelResponse } from '../src/agent.ts';
export function scriptedClient(steps: ContentBlock[][]): ModelClient & { requests: ModelRequest[] } {
let index = 0;
const requests: ModelRequest[] = [];
const client = async (request: ModelRequest): Promise<ModelResponse> => {
requests.push(structuredClone(request));
const content = steps[Math.min(index++, steps.length - 1)];
const usesTool = content.some((b) => b.type === 'tool_use');
return { content, stop_reason: usesTool ? 'tool_use' : 'end_turn', usage: { input_tokens: 100, output_tokens: 20 } };
};
return Object.assign(client, { requests });
}
export const toolUse = (id: string, name: string, input: unknown): ContentBlock => ({ type: 'tool_use', id, name, input });
export const text = (value: string): ContentBlock => ({ type: 'text', text: value });
// test/graders.test.ts
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { runAgent } from '../src/agent.ts';
import { llmJudge } from '../src/graders.ts';
import { createLibrary } from '../src/library.ts';
import { tasks } from '../src/tasks.ts';
import { scriptedClient, text, toolUse } from './helpers.ts';
const task = (id: string) => tasks.find((t) => t.id === id)!;
async function grade(id: string, steps: Parameters<typeof scriptedClient>[0]) {
const state = createLibrary();
task(id).setup?.(state);
const run = await runAgent(scriptedClient(steps), state, task(id).prompt);
return Promise.all(task(id).graders.map((g) => g(run, state)));
}
test('a good solution passes “reserve-when-unavailable”', async () => {
const grades = await grade('reserve-when-unavailable', [
[toolUse('t1', 'search_books', { query: 'Lighthouse' })],
[toolUse('t2', 'reserve_book', { bookId: 'b-101', memberId: 'M-17' })],
[text('Reserved.')],
]);
assert.ok(grades.every((g) => g.pass), JSON.stringify(grades));
});
test('an agent that reserves an available book fails', async () => {
const grades = await grade('no-reservation-when-available', [
[toolUse('t1', 'reserve_book', { bookId: 'b-102', memberId: 'M-17' })],
[text('Sorry, that did not work.')],
]);
const failed = grades.filter((g) => !g.pass).map((g) => g.grader);
assert.deepEqual(failed, ['toolNotCalled(reserve_book)']);
});
test('missing member number: asking back passes', async () => {
const grades = await grade('member-number-missing', [[text('Sure! What is your member number?')]]);
assert.ok(grades.every((g) => g.pass));
});
test('model judge: "unknown" does not count as a pass', async () => {
const judge = scriptedClient([[text('{"verdict":"unknown","reason":"The answer is ambiguous"}')]]);
const run = await runAgent(scriptedClient([[text('Hm.')]]), createLibrary(), 'Hello');
const result = await llmJudge(judge, 'friendly')(run, createLibrary());
assert.equal(result.pass, false);
assert.match(result.detail, /^unknown/);
});
// test/metrics.test.ts
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { passAtK, passHatK } from '../src/metrics.ts';
test('at k = 1, pass@k and pass^k equal the success rate', () => {
assert.equal(passAtK(10, 7, 1), 0.7);
assert.equal(passHatK(10, 7, 1), 0.7);
});
test('pass@k rises and pass^k falls as k grows', () => {
assert.ok(passAtK(10, 7, 3) > passAtK(10, 7, 1));
assert.ok(passHatK(10, 7, 3) < passHatK(10, 7, 1));
});
test('edge cases', () => {
assert.equal(passAtK(5, 0, 3), 0);
assert.equal(passHatK(5, 5, 3), 1);
assert.equal(passHatK(5, 2, 3), 0);
assert.throws(() => passAtK(2, 1, 3));
});
Now check types and tests. Both run without an API key.
npm run typecheck
npm test
Expected result: tsc reports nothing, and the test runner ends with # pass 7 and # fail 0.
8. The first real run
Set the API key and the model and start the evals. EVAL_TRIALS sets how often each task runs, EVAL_K sets k for the metrics.
export ANTHROPIC_API_KEY="your-key"
EVAL_MODEL=claude-sonnet-5-5 EVAL_TRIALS=5 EVAL_K=3 npm run evals
With Amazon Bedrock the call looks like this. As the model, enter the model ID or inference profile that is enabled in your region.
export AWS_REGION=eu-central-1
EVAL_PROVIDER=bedrock EVAL_MODEL="<your-bedrock-model-id>" npm run evals
You get a table with one row per task: successes, partial credit, pass@3, pass^3, average turns and tokens. Below it is the path to the JSONL file with all transcripts. I am not giving you numbers here, because they depend on the model, the day and the tool descriptions. I do have one expectation, though: the regression tasks should score very high with a current model, while a low value for “two-books-at-once” would not surprise me. That is exactly what a capability task is for.
If the key is missing, every trial fails as an infrastructure error and the script exits with code 2. That is intentional: it keeps you from reading a broken run as a bad agent.
Which metric counts, pass@k or pass^k?
Both metrics are based on k trials but answer opposite questions. pass@k is the probability that at least one of k trials succeeds. It rises with k and fits tools where one good solution is enough, for example when a developer picks from several suggestions. pass^k asks how likely it is that every single one of the k trials works. This value drops as k grows and fits everything customers use directly. A lending assistant that reserves nonsense every third time is useless in practice, even if it “mostly” works.
Anthropic works through an example: at 75 percent success per trial, the chance that three out of three trials succeed drops to about 42 percent (0.75 to the power of 3). The chart shows how quickly the two curves drift apart.
!Line chart: pass@k rises from 75% to almost 100%, pass^k falls from 75% to 24% for k from 1 to 5
At 75% success per trial, pass@k quickly approaches 100%, while pass^k drops to about 42% at k = 3.
In metrics.ts I estimate both values from n trials with c successes. For pass@k I use the estimator from OpenAI's Codex paper: one minus the probability of drawing k trials only from the failures. For pass^k, which comes from the agent benchmark τ-bench, I draw k trials only from the successes. A practical rule follows: n must be at least as large as k, and the larger n, the more stable the estimate. With five trials per task and k = 3 you get a useful first impression.
Reading transcripts instead of just percentages
The table tells you that something is wrong. Why it is wrong is only in the transcript. Open the JSONL file and read at least the failed trials. You will find two kinds of errors, and you need to tell them apart.
The first kind are real agent errors. The agent calls reserve_book with a guessed member number or misses that a copy is available. A better tool description, a clearer system prompt or a different model helps here.
The second kind are errors in the eval itself. An example from my own code: asksQuestion passes as soon as there is a question mark anywhere. If the agent answers “Did you see our new opening hours?” and makes a wrong reservation at the same time, only the other two graders of that task catch it. That is why I always combine soft checks with a hard check of the outcome. If a task fails in almost every trial, look at the task and the grader first, then at the agent. Often the task is simply ambiguous.
Evaluating AI agents in CI
It gets really valuable once you are evaluating AI agents automatically, without having to remember it. I would hook the evals in at two points: on every pull request that changes prompts, tools or agent code, and once per working day against the current model. The daily run catches drift, meaning behavior changes that happen without a commit of your own, for example when a model alias points to a new version.
The following GitHub Actions workflow first runs the free tests and then the evals. The transcripts are attached to the run as an artifact, so you can read them even when it fails.
# .github/workflows/evals.yml
name: Agent-Evals
on:
pull_request:
paths: ['src/**']
schedule:
- cron: '17 3 * * 1-5' # early on weekdays so model drift shows up
workflow_dispatch:
jobs:
evals:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 24
cache: npm
- run: npm ci
- run: npm test
- run: npm run evals
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
EVAL_MODEL: claude-sonnet-5-5
EVAL_TRIALS: '5'
EVAL_K: '3'
- uses: actions/upload-artifact@v7
if: always()
with:
name: eval-transcripts
path: results/
Create the repository secret ANTHROPIC_API_KEY for this. Pull requests from forks do not get secrets on GitHub by default. The eval step fails there with exit code 2, and that is how it should be: outside code should not run with your API key.
Common mistakes when evaluating agents
Only one trial per task
One trial is an anecdote. Only with several trials can you see whether a result is stable. Five trials are a good start; use more for important regression tasks.
Checking the path instead of the outcome
If you prescribe that the agent must first search, then check and then reserve, you punish valid shortcuts. Check the final state. I make one exception for forbidden side effects: requiring that reserve_book is not called at all for an available book is a business rule, not a prescribed path.
Testing only positive cases
If you only test whether the agent reserves, sooner or later it reserves everything. Every “do X” task needs at least one “do not do X here” task. This balance keeps your agent from overreacting or underreacting.
Shared state between trials
If data from the previous trial is still lying around, you are measuring chance. Every trial starts clean, whether in memory like here or with a freshly provisioned test database.
Evals that no longer show anything
If a suite sits at 100 percent permanently, it no longer measures progress. For regressions that is the goal. For capability evals it means you need harder tasks.
Cost and security
Every run costs tokens. The math is simple: tasks times trials times turns per trial. With six tasks, five trials and three to four turns, you quickly end up with around a hundred model calls. Keep an eye on the “Tokens” column in the table, and feel free to use a smaller, cheaper model as the judge if it grades reliably. Check that first against a few cases you graded yourself.
On security: evals run against a test environment, never against real systems or real customer data. Write tools in particular can be called hundreds of times in an eval. The API key belongs in secrets, never in the code. And because rate limits can hit, the harness limits concurrency to three simultaneous trials.
My take
To me, evals are to agents what unit tests are to regular code: the precondition for changing things with confidence. If you can evaluate AI agents, you can afford to experiment. If you run an agent without evals, at some point you no longer dare to switch the model or touch the prompt. That is the opposite of what we want to achieve with AI.
At the same time, I do not think much of starting with a big framework and hundreds of tasks. Twenty tasks from real failures beat two hundred invented ones. And I would build code graders first. The model judge is handy, but it costs money, varies itself and has to be calibrated against human judgment. I use it where code really cannot go further.
Then there is the human side. The best tasks rarely come from the development team alone. They come from support, from the business side, from the people who know what goes wrong in daily use. When you introduce evals, invite these people to contribute tasks. That improves the agent and builds trust in what it does.
FAQ
What is the difference between evals and unit tests?
Unit tests check deterministic code and always return the same result for the same input. Evals check a system with a language model whose answers vary. That is why evals run several times and return rates instead of a simple green or red.
How many tasks do I need to start?
If you are evaluating AI agents for the first time, Anthropic recommends starting with roughly 20 to 50 simple tasks from real failures. You do not need hundreds of tasks at the beginning. What matters more is that every task is unambiguous and that two experts would independently reach the same verdict.
Do I need an eval framework?
Not to get started. A harness like the one in this article has a few hundred lines and shows you exactly what happens. A framework pays off once you need many suites, dashboards or comparisons between models.
Can I use a language model as a judge?
Yes, for soft criteria such as tone or completeness. Give the judge one clear criterion, allow the answer “unknown” and regularly compare its verdicts with human ratings. For hard facts such as “does the reservation exist”, code is more reliable.
How much does an eval run cost?
That depends on the model and the number of tasks, trials and turns. Multiply tasks by trials and average turns to get the number of model calls. The harness shows you the tokens per task after the first run.
If you want to not only test your agent but also run it on AWS, AgentCore CLI 1.0: Deploy an AI Agent with TypeScript shows the way into the AgentCore Runtime.
Your next step
Take the last three cases where your agent got it wrong and turn them into tasks with a grader that checks the outcome. Run them five times and read the transcripts. That is the smallest sensible start for evaluating AI agents. I bet at least one of the three cases leads you to a better tool description. You will find more about AI in everyday development on the blog.
Now I am curious: how do you check today whether your agent still does what it should after a change? Let me know in the comments.
Sources
- Anthropic Engineering: Demystifying evals for AI agents (Jan 9, 2026)
- Anthropic Engineering: Writing effective tools for agents (Sep 11, 2025)
- Claude documentation: Define tools
- Anthropic TypeScript SDK on GitHub
- npm: @anthropic-ai/bedrock-sdk
- Chen et al. (2021): Evaluating Large Language Models Trained on Code (pass@k estimator)
- Yao et al. (2024): τ-bench, A Benchmark for Tool-Agent-User Interaction (pass^k)
- Node.js documentation: Test runner
- GitHub Docs: Using secrets in GitHub Actions
This post was created with AI assistance.
---
👉 Next step: New here? My start page collects the best articles, tools and learning paths in one place.
Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de
1 reaction
How to never pay a generative API twice: a durable task table, not a retrying job queue
Short answer: the moment you might have been charged is not the provider's response, it is the instant before the request leaves your process. Write that instant to a database row before the HTTP call, and a worker that dies mid-request becomes recoverable without guessing: nothing on the row means nobody paid and the task can go back to the queue; a saved request id means poll and never resubmit; the boundary written with no id means a third terminal state that no automation is allowed to retry.
TL;DR
- Fix: own task table with a lease, and a submit_started_at column written before the paid call. A queue retries by design; pg-boss says so in its own docs: "the job can run twice and handlers should be idempotent."
- Alternative: a provider-side idempotency key, if your provider has one. Stripe does. The OpenAI and Anthropic clients ship maxRetries: 2 and send no idempotency header at all.
- Trap: treating an expired deadline as a failure. A poll that ran out of patience is "I don't know", and the difference costs money every time you get it wrong.
Why a job queue is the wrong unit of recovery for a paid call
Because retrying is the entire point of a job queue, and for a paid POST a retry is a second charge. I started by running generation steps as ordinary jobs on the queue the rest of the system already used, pg-boss, because that is where every other background task lived. That lasted until I read the queue's own documentation on what happens when a worker looks dead:
> A worker that only looked dead (a network partition, a stalled event loop) keeps running its handler after its job is failed this way, so the job can run twice and handlers should be idempotent.
That is the pg-boss queue reference, and it is being honest rather than apologetic. Every durable queue has the same sentence somewhere, because a queue cannot tell a dead worker from a slow one. The contract it offers is at-least-once, and it tells you to make the handler idempotent. For sending an email, fine. For a handler whose first line spends money at an external provider, "make it idempotent" is the hard part being handed back to you.
The defaults make it concrete. retryLimit is 2, so a failing job runs three times. expireInSeconds defaults to 15 minutes, after which an active job is retried or failed regardless of what the worker is doing. Image generation lands in tens of seconds and video generation in minutes, so a long clip can sail past the expiration while it is perfectly healthy, get failed by the monitor, and get retried into a second charge. You can raise the expiration to 24 hours, the documented maximum, but then a genuinely dead worker sits undetected for a day.
There is a second problem that is not about money. A job that spends twenty minutes polling a remote id occupies a worker slot for twenty minutes. Concurrency on the queue is shared with everything else in the system, and a handful of slow generations will starve the short jobs that the queue exists for.
The unit of recovery had to stop being a job and become a row.
What RFC 9110 actually says about retrying a POST
It says do not, unless you can prove one of two things, and the second one is the design brief for the whole task table. RFC 9110, section 9.2.2:
> A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.
Two escape hatches. The first one, knowing the semantics are idempotent, is what a provider gives you with an idempotency key. Most generative APIs do not give you one, which I will get to. The second, "some means to detect that the original request was never applied," is not a property of the protocol at all. It is a property of what you wrote down locally before you made the call.
The RFC also has a line about clients that skip both:
> Some clients take a riskier approach and attempt to guess when an automatic retry is possible. For example, a client might automatically retry a POST request if the underlying transport connection closed before any part of a response is received, particularly if an idle persistent connection was used.
That paragraph describes the official clients of the largest paid APIs in the world, which is the subject of the next section.
Do the generative APIs give you an idempotency key?
Mostly no, and the clients retry your POSTs anyway. I went looking for this expecting a boring answer and got a much sharper one by reading the SDK sources instead of the marketing pages.
| API | Idempotency key on a POST | Official client retries a POST by default |
|---|---|---|
| Stripe | Idempotency-Key, first result stored and replayed | not examined |
| OpenAI | none sent by openai-node, see below | yes, maxRetries: 2 on 408, 409, 429, ≥500, timeouts |
| Anthropic | no occurrence of the word in anthropic-sdk-typescript | yes, maxRetries: 2 on 408, 409, 429, ≥500, timeouts |
| Replicate | no occurrence in the published OpenAPI document | not examined |
Stripe is the benchmark for how this is supposed to work, and its wording is worth quoting because it names the property that matters:
> Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.
That is the Stripe API reference on idempotent requests. Note the retention rule in the same page: keys can be pruned once they are at least 24 hours old, and a key reused after pruning generates a new request. Even the gold standard has an expiry after which your safety net is gone, which is a good reminder that the client side still has to keep its own record.
The OpenAI side is where it gets strange. The OpenAI Node client carries the full apparatus for idempotency keys. It declares the field, it has a key generator whose default value announces where the machinery came from, and it builds the header:
protected idempotencyHeader?: string;
protected defaultIdempotencyKey(): string {
return `stainless-node-retry-${uuid4()}`;
}
// ...later, while assembling headers:
let idempotencyHeaders: HeadersLike = {};
if (this.idempotencyHeader && method !== 'get') {
if (!options.idempotencyKey) options.idempotencyKey = this.defaultIdempotencyKey();
idempotencyHeaders[this.idempotencyHeader] = options.idempotencyKey;
}
idempotencyHeader is declared and never assigned anywhere in the file. The condition is therefore always false and the header is never sent. RequestOptions even exposes idempotencyKey with the comment "A unique key for this request to enable idempotency", and setting it accomplishes nothing, because there is no header name to put it in. I spent a while convinced I had missed an assignment in a subclass. There is none.
Meanwhile the retry predicate in the same file does not look at the method at all:
private async shouldRetry(response: Response): Promise<boolean> {
const shouldRetryHeader = response.headers.get('x-should-retry');
if (shouldRetryHeader === 'true') return true;
if (shouldRetryHeader === 'false') return false;
if (response.status === 408) return true; // request timeouts
if (response.status === 409) return true; // lock timeouts
if (response.status === 429) return true; // rate limits
if (response.status >= 500) return true; // internal errors
return false;
}
DEFAULT_TIMEOUT in that client is 600000, ten minutes, and a timeout goes down the same retry path. The Anthropic TypeScript client has a byte-for-byte equivalent predicate, the same maxRetries: 2 and the same ten-minute default, and zero occurrences of the string idempot in the entire file.
The default behaviour of both clients, then: send a paid POST, and if the server returns 500 or the socket goes quiet for ten minutes, send it again, up to two more times, with no way for the server to recognise it as the same request. Whether the first attempt was billed is between you and the provider's meter. The client has no way to ask, and neither do you.
This is not a scandal. It is the reasonable default for an API where most calls are cheap and a dropped connection is more likely than a silently successful one. It stops being reasonable the moment a single call costs real money and the user can see the balance.
The boundary is before the request, not after the response
The instant that matters is the last moment before the send, not the first moment after the answer. This is the one idea the whole design rests on, and it took me an embarrassing amount of staring at a state machine to see it.
I had been drawing the boundary in the obvious place: the task is safe until the provider answers, and becomes committed when it answers. That is wrong, and the reason is that the answer is exactly the thing you do not have when the worker dies. A crash between "socket write completed" and "response parsed" leaves no evidence at all on the client side, and it is the most likely crash window there is, because that window is where all the waiting happens.
Flip it. The task is safe until you begin the request, and becomes unknown the instant you do. That boundary is knowable in advance, so it can be written down in advance:
async function runImageTask(task: Task, ports: Ports): Promise<void> {
if (task.status === "SUBMITTED" && task.remoteRequestId) {
return pollUntilDone(task, ports); // already paid for, never resubmit
}
await ports.repo.markSubmitStarted(task.id); // the boundary
const { requestId } = await ports.provider.submit(task.input);
await ports.repo.recordSubmitted(task.id, requestId); // now it is addressable
return pollUntilDone(task, ports);
}
Three writes, and every crash window between them maps to exactly one recovery action. markSubmitStarted is a separate round trip to the database before a call that is about to take thirty seconds, so the cost of the extra write is not worth discussing.
The one rule this imposes on the rest of the code is that nothing may sit between markSubmitStarted and the actual send. No input validation, no prompt assembly, no lazy credential fetch. Anything that can fail there produces a task parked in the unknown state for no reason, and parking a task costs a human's attention later. Build the request fully, then mark, then send.
Three recovery branches after an expired lease
A supervisor picking up a task whose lease has expired has exactly three cases, and the row tells it which one without any guessing:
| What is on the row | What happened | Recovery | Money |
|---|---|---|---|
| status = RUNNING, no submit_started_at | Died before the request went out | Back to PENDING | Nothing spent, safe to redo |
| status = SUBMITTED, remote_request_id set | Died after the provider acknowledged | Poll the saved id, never submit | Already spent, result is retrievable |
| status = RUNNING, submit_started_at set, no id | Died inside the request | OUTCOME_UNKNOWN, stop | Might be spent, nobody can tell |
In code it is short enough to fit in a screenshot, which is the point:
type Recovery =
| { action: "requeue" }
| { action: "poll" }
| { action: "park"; status: "OUTCOME_UNKNOWN" };
function recoveryFor(task: Task): Recovery {
if (task.status === "SUBMITTED" && task.remoteRequestId) return { action: "poll" };
if (task.submitStartedAt === null) return { action: "requeue" };
return { action: "park", status: "OUTCOME_UNKNOWN" };
}
Order matters in that function and it is not the order I wrote first. My first version checked submitStartedAt before the request id, which sends an already-submitted task back through the submit path whenever both columns are set, which is always for a healthy submitted task. The request id is the strongest evidence on the row, so it gets tested first.
Why OUTCOME_UNKNOWN has to be terminal
Because every automatic way out of it is a bet placed with someone else's money. This was the design decision I argued with myself about longest, since a state that requires a human is an operational cost forever.
The two tempting exits both fail. Retrying assumes the request never landed, and if it did land you have paid twice and the user sees one result. Failing assumes the request never landed too, just with a nicer label: the task disappears from the pipeline, the charge does not disappear from the invoice, and the reconciliation happens weeks later when someone asks why the numbers do not match. Both are the same guess wearing different clothes.
The state is therefore terminal, the UI says the outcome could not be determined and suggests checking the provider's dashboard, and a human presses the button. In exchange, the system never spends money on its own initiative. For a paid API that is the correct side of the error to be on, and the volume is low enough that it stays correct: the branch only fires when a process dies inside the send window, which is a few seconds out of a generation that takes minutes.
The same logic applies to deadlines, and this is where I got it wrong first. Put a cap on polling, because a remote id that never resolves cannot hold a slot forever. My original code marked the task FAILED when the cap was reached. That is the same lie as before: the provider might still be working, or might have finished and billed while my poll loop gave up. An exhausted deadline is not a failure, it is an unknown outcome that happens to be slow. It goes to the same terminal state, with a different reason string.
Claiming work with FOR UPDATE SKIP LOCKED
The claim is a single statement, and PostgreSQL documents this exact use case. From the SELECT reference:
> With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped. Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table.
"Inconsistent view of the data" is the point, not a caveat. Two supervisors claiming work must not see the same row, and neither needs an accurate count of what is pending.
update generation_task t
set lease_owner = $1,
lease_expires_at = now() + $2::interval,
status = case when t.status = 'PENDING' then 'RUNNING' else t.status end
where t.id in (
select id
from generation_task
where kind = $3
and (status = 'PENDING'
or (status in ('RUNNING', 'SUBMITTED') and lease_expires_at < now()))
order by created_at
limit $4
for update skip locked
)
returning t.*;
The case matters, and this is the part I got wrong. An earlier version of this statement set status = 'RUNNING' unconditionally, which erases the one signal that tells the executor to poll instead of submit. I rebuilt both versions on a scratch PostgreSQL 17.11 database to be sure I was not misremembering, and the buggy claim does exactly what you would fear:
status | remote_request_id
---------+-------------------
RUNNING | req-abc
The remote_request_id survives, the status does not. A recovered task that had already been paid for comes back looking like a fresh one. The guard at the top of runImageTask still catches it, because that guard checks the id as well, but at that point the query had quietly become the only thing standing between a restart and a double charge, and I would rather two things stood there.
On the same scratch database, SKIP LOCKED behaves as advertised: with three eligible rows and one of them held by a concurrent SELECT ... FOR UPDATE in another session, the claim returned 2 rows immediately instead of blocking on the third. That is the whole reason the clause exists, and it is worth confirming with your own eyes once, because the alternative failure mode, a supervisor tick that silently waits on a lock, looks identical to a supervisor tick that has nothing to do.
Concurrency is per kind, not global, because the kinds have nothing in common. Take four concurrent image slots, three for clips, two for speech and exactly one for the final local render: the remote ones are limited by the provider's rate limits and the local one is limited by CPU. Those particular numbers are arbitrary and yours will differ. The structure is not arbitrary: one global limit for all four would either starve the cheap fast work or oversubscribe the machine.
Heartbeat, because not every step polls
A lease needs its own renewal, separate from whatever the work is doing. I learned this from a step that had no polling at all.
Remote generation renews the lease for free as a side effect: the poll loop runs every few seconds, and touching the row at the same time costs nothing. Local work has no such loop. The final render is one long call into ffmpeg that returns when it returns, and speech synthesis for a short passage is a single request that either answers or does not. Both of them held a lease that nobody was extending, so the supervisor watched the lease expire on a task that was working perfectly, decided the worker was dead, and started recovery on a row that was still being written to.
The fix is a timer independent of the work: renew at one third of the lease duration, so two consecutive misses still leave a margin before expiry. The ratio is what matters, not the absolute value. pg-boss reaches the same conclusion from the other direction and documents it as a separate mechanism from expiration, with a floor of 10 seconds on heartbeatSeconds and a monitor that only notices staleness on its own interval, 60 seconds by default. Detection time is the sum of the two, which is worth internalising before you tune the lease down to something clever.
Pinning the order with a test
The order of those three writes is load-bearing and invisible, so it gets asserted directly rather than implied by behaviour. The executor takes its ports as an interface, the test passes recording fakes, and the assertion is on the call log:
expect(calls).toEqual([
"markSubmitStarted",
"submitImage",
"recordSubmitted",
"poll",
"complete",
]);
Then the recovery case, which is the one that actually costs money if it regresses:
// a task restored in SUBMITTED state with a saved request id
expect(calls).not.toContain("submitImage");
expect(calls).toEqual(["poll", "complete"]);
A test that asserts on a sequence of call names is usually a smell, because it pins implementation detail rather than behaviour. Here the sequence is the behaviour. Any refactor that moves markSubmitStarted after submitImage still passes every functional test, produces identical output on the happy path, and reintroduces the exact bug the design exists to prevent. The only way to catch it is to say out loud that the write comes first.
The price in the docs is not the price on the invoice
Store two numbers per task, estimated and actual, because the first one will be wrong. I built the cost display off a model catalogue populated from provider documentation, showed users an estimate before they pressed the button, and felt good about it. The first real run billed noticeably more than the catalogue said. Take a step documented at four cents and invoiced at six: that is half again as much, multiplied by every step in a job. An estimate that far off is worse than no estimate, because people plan around it.
The catalogue was not lying, it was quoting a headline number that did not include what the actual call resolved to. The correction is structural rather than a better catalogue: cost_estimated comes from the provider's quote endpoint at submit time, cost_actual comes from the billing figure that arrives with the result, and the user-facing total is the sum of whichever is available per task. When the two disagree the actual value wins and the catalogue entry gets fixed. Documentation prices are for capacity planning. Billing figures are for arithmetic.
Bottom line
If a background task spends money at an external provider, the queue's retry is no longer a reliability feature and the row, not the job, has to be the unit of recovery. Write the "I am about to spend money" boundary to the database before the request goes out, save the provider's request id the moment it exists, and accept a third terminal state for the narrow window in between, where neither retrying nor failing is honest. The RFC has been describing this for years under "some means to detect that the original request was never applied", and the means is a column.
Primary sources worth reading in full: RFC 9110 §9.2.2 on idempotent methods, the PostgreSQL SELECT reference on SKIP LOCKED, the pg-boss queue documentation on heartbeats versus expiration, Stripe on idempotent requests, and the retry predicates in openai-node and anthropic-sdk-typescript, which are worth twenty minutes of anyone's time who is sending paid POSTs through them.
---
Originally published at bitpage.me — BitPage, a technical blog on backend, databases, infrastructure and incident post-mortems.
State of AI Instruction Quality: Coding Agent Comparison
Claude Code, Codex, Copilot, Cursor and Gemini are the five coding agents in the Q3 report. We compared them across the 10,000 most-starred repos that ship an instruction file, and put the companies that make them through the same check, using their own repos.
Claude Code is everywhere. It's set up in 69 of the 100 most-starred repos and in 6 of the top 10. Copilot is in 26 of the top 100, Gemini in 11, Cursor in 7 and Codex in 3.
The file Claude Code reads is losing anyway. One CLAUDE.md in five is a line or two telling the agent to go read AGENTS.md.
Which coding agents the top repos use
| Agent | Top 10 | Top 100 | Top 1,000 | All 10,000 |
|---|--:|--:|--:|--:|
| Claude Code | 6 | 69 | 646 | 6,030 |
| AGENTS.md (shared) | 6 | 72 | 678 | 6,125 |
| Copilot | 2 | 26 | 225 | 1,756 |
| Gemini | 2 | 11 | 51 | 430 |
| Cursor | 1 | 7 | 96 | 734 |
| Codex | 0 | 3 | 32 | 162 |
!Share of repos that set up each agent, in the 100 most-starred repos against the 9,784 of the 10,000 with a setup: Claude Code 69% against 61.6%, AGENTS.md 72% against 62.6%, Copilot 26% against 17.9%, Gemini 11% against 4.4%, Cursor 7% against 7.5%, Codex 3% against 1.7%.
The most-starred repo in the set, freeCodeCamp, uses Copilot. React uses Claude Code. Gemini punches above its size at the top: 430 of the 10,000 set it up, and 11 of the top 100 do.
Many repos don't pick one. 4,213 of the 9,784 with a setup (43.1%) use two or more coding agents, and the most common pairing is Claude Code plus AGENTS.md, in 2,399 repos.
AGENTS.md is winning
998 of the 5,464 CLAUDE.md files in the 10,000 (18.3%) are 25 words or fewer and point at AGENTS.md. Going the other way, 39 of 6,128 AGENTS.md files point at CLAUDE.md. That's 26 to 1. 11.3% of GEMINI.md files point to AGENTS.md too, and 4.0% of Copilot's instructions files do.
The CLAUDE.md files people do write are the least instruction-like main files of any agent. About a quarter of what's in them tells the agent what to do. The rest describes the project.
!Median share of items in each agent's main file that are directives or constraints: CLAUDE.md 26.1%, GEMINI.md 35.2%, AGENTS.md 36.4%, Codex's AGENTS.md 38.1%, copilot-instructions.md 38.9%, .cursorrules 46.0%. Pointer files left out.
Claude Code setups hedge less. The "should" and "try to" kind runs at a median 3.3 per 100 instructions, against 4.7 for the other agents.
The instructions moved into skills
While the debate is still ongoing between AGENTS.md and other provider-specific files, most of the instructions moved to a different location. Skills hold 42.3% of every directive and constraint in the set. All the main files together hold 31.3%.
!Share of all directives and constraints in the 10,000: skills 42.3% (13,190 SKILL.md files plus their reference files), main files 31.3% (13,375 CLAUDE.md, AGENTS.md and other main files), everything else 26.4% (rules, sub-agents, commands).
Peculiar oddity, but only a quarter of repos (2,348) have any skills, and 249 repos hold half of all skill instructions between them. There are almost as many SKILL.md files as main files: 13,190 against 13,375.
They spread like packages, without a package manager. 977 SKILL.md files are byte-for-byte copies of one in another repo. The most-copied, frontend-design, sits unchanged in 14 repos. A fix upstream reaches none of the copies.
Two names, one file, and copies drifting apart
3,562 repos carry main files for two or more agents. A third of them (1,246) keep the same text under two names, mostly a CLAUDE.md identical to the AGENTS.md beside it. Another third use a pointer.
!How the 3,562 repos with main files for two or more agents relate the second file to the first: the same text under two names 1,246 repos (35.0%), a pointer to the other file 1,158 (32.5%), different files 937 (26.3%), copies drifting apart 221 (6.2%).
221 repos keep two copies that have already drifted. The typical pair shares 32 lines, and each side has two or three lines the other lacks.
In 92 of 430 Gemini setups (21.4%), GEMINI.md and AGENTS.md hold the same text, and Gemini reads both. For the other agents, the same shape turns up in 1.3% of setups. Gemini setups are also the heaviest at startup: 8.1% hand the agent more than 5,000 words before the first prompt, against 2.3% across the field.
Cursor goes the other way. 66.3% of Cursor setups carry rule files, and a third (33.5%) have no main file at all. The other agents' setups almost always have one.
The makers, by their own repos
| Company | Agent | Repos in the 10,000 | Median score | Share of repos with custom sub-agents |
|---|---|--:|--:|--:|
| Anthropic | Claude Code | 11 | 7.0 | 27% |
| OpenAI | Codex | 13 | 6.8 | 0% |
| Google | Gemini | 28 | 7.2 | 4% |
| GitHub | Copilot | 11 | 7.9 | 45% |
| Cursor | Cursor | 2 | 8.6 | 0% |
| All repos | | 10,000 | 7.5 | 5.7% |
The scores cluster around the field's. What each company builds is where they split.
!Which agents each company sets up in its own repos: Anthropic, 11 repos: Claude Code 11, AGENTS.md 1, Copilot 0, Gemini 0, Cursor 0. OpenAI, 13 repos: Claude Code 2, AGENTS.md 12, Copilot 1, Gemini 0, Cursor 0. Google with its Gemini orgs, 28 repos: Claude Code 5, AGENTS.md 15, Copilot 1, Gemini 18, Cursor 1. GitHub, 11 repos: Claude Code 1, AGENTS.md 5, Copilot 10, Gemini 0, Cursor 0. Cursor, 2 repos: Claude Code 0, AGENTS.md 1, Copilot 0, Gemini 0, Cursor 2.
GitHub builds its own agents. 5 of its 11 repos ship custom sub-agents, eight times the field's rate of 5.7%. One of them is spreading: an agentic-workflows sub-agent for GitHub's own workflow tool sits in 44 repos across the 10,000, the second most common sub-agent role after code-reviewer.
The agents' own repos shout too. A third of the 10,000 write rules in capitals. Anthropic's prompting guide tells users of its newest models to dial back that kind of language: "Where you might have said 'CRITICAL: You MUST use this tool when...', you can use more normal prompting like 'Use this tool when...'." Claude Code's own repo ships a triage command that says "IMPORTANT: Don't post any comments or messages to the issue." Codex's own AGENTS.md says "Features that change the agent logic MUST add an integration test", and a Gemini CLI command tells the agent "you MUST first consult the getting started documentation".
OpenAI's repos leave the agent guessing. Of every writing habit we compared across the makers, only one difference held up statistically. OpenAI's median repo leaves 48.7 of every 100 instructions without a file, command or tool to act on, against 30.8 for the field. One of its SDKs opens a section with:
> Solve the narrow problem with the simplest implementation. Avoid unrelated refactors, reformatting, dependency or lockfile churn, generated-file edits, speculative abstractions, compatibility shims, and capabilities not requested.
A senior engineer would agree with every word. An agent gets two judgment calls, "the narrow problem" and "the simplest implementation", and nothing to check its work against.
Cursor's two repos score highest, 8.6, with the fewest instructions left vague. One still carries the hedged kind, a skill that says "subtask_prompt should read like a standalone request".
Who checks the makers
These companies write the agents, the documentation and the prompting guides. Their engineers know their codebases far better than any tool does, and that's how the gaps get in. To the person who wrote it, "the narrow problem" has an obvious answer. The agent reading the file has the file, and nothing else.
The agent can't flag the gap either. It does what the line says, and when the line names nothing, it guesses. A company reviewing its own files reads them with the same knowledge that left the gap there.
ails check reads every instruction file with the same deterministic rules, whoever wrote it. The 10,000 most-starred repos and the companies that build the agents get the same treatment, and the findings above are what came back.
!Lines 26 to 41 of the AGENTS.md in OpenAI's openai-node repo, the section "Keep changes small and coherent", with ails check's Specificity Gap finding beside seven of its lines: 28, 31, 32, 34, 36, 37 and 38. Line 28 reads "Solve the narrow problem with the simplest implementation."
Check one yourself
Every repo in this post is public. Clone one and run the same check:
npx @reporails/cli check
It lists each finding by file, so you can read the instructions and decide whether we counted fairly.
This is the last post from the Q3 report. The data behind all of them is in the 2026-Q3 folder of reporails/reports.
---
From The State of AI Instruction Quality: Q3 2026: 10,000 of the most-starred GitHub repos that ship an instruction file, September 2026 snapshot; 9,784 have a setup we could map. "Top 100" and "top 1,000" rank the 10,000 by stars at the snapshot. A setup is one repo and one agent; repos often have several, so agent counts add up to more than the repos. Codex counts repos with a Codex marker in .codex/ (a config.toml, rules/, agents/ or hooks.json) or an AGENTS.override.md; a plain AGENTS.md setup sits in the shared row. A pointer is a main file of 25 words or fewer that names or imports another agent's main file. Items are what the instrument maps from a file; directives and constraints are the items that tell the agent what to do. "Skills" include the reference files beside SKILL.md; setups above the batch's 90-file cap are outside the run, so the skills share is a floor. "The same text" cannot tell a symlink from a copy; only the drifted pairs are certainly two files. "Gemini reads both" is the instrument's model of Gemini, which concatenates AGENTS.md and GEMINI.md. Companies are the GitHub organizations that ship each agent: anthropics, openai, google with google-gemini and gemini-cli-extensions, github and cursor; Cursor's two repos carry no test. Group differences are two-sided Mann-Whitney or Fisher's exact tests, read against a Bonferroni threshold for all 209 comparisons in the run; the top-100 shares are counts. Scores are each company's median repo score on ails check's 0 to 10 scale; for Anthropic, OpenAI, Google and GitHub the 95% bootstrap interval includes the field's 7.5. Vague instructions name no file, command or tool (CORE:C:0042), per 100 directives and constraints, in repos with at least 5.
Repos measured. anthropics: anthropic-sdk-typescript, buffa, claude-agent-sdk-python, claude-code, claude-code-action, claude-code-base-action, claude-for-legal, commerce-agents, defending-code-reference-harness, financial-services, launch-your-agent. openai: apps-sdk-ui, codex, codex-security, harmony, openai-agents-js, openai-cli, openai-cookbook, openai-dotnet, openai-go, openai-java, openai-node, openai-python, realtime-voice-component. google: adk-go, adk-java, adk-python, benchmark, clusterfuzz, comprehensive-rust, docsy, dotprompt, filament, go-github, gvisor, heir, mantis, nomulus, oss-fuzz, osv-scanner, osv.dev, perfetto, play-services-plugins, sam, site-kit-wp, syzkaller, xls. google-gemini: gemini-cli. gemini-cli-extensions: code-review, nanobanana, security, workspace. github: branch-deploy, codeql, codeql-action, copilot-cli-for-beginners, copilot-sdk, docs, gh-aw, gh-skyline, gh-stack, github-mcp-server, spec-kit. cursor: community-plugins, cookbook.
---
I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the instruction files your coding agent loads, from CLAUDE.md to Copilot and Cursor rules, measures which instructions couple to behavior, and points to the lines worth tightening.
11 reactions · 2 comments
Streamed LLM Responses Hide Their Token Count. Where Usage Lives in 3 TypeScript APIs
- Book: AI That Answers
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
You ship the streaming chat box. Text appears word by word, the UI feels fast, and the cost log you wired up last sprint shows a flat line of zeros. Nothing is broken on the network. The tokens were billed. Your code just looked for them in the place a non-streamed response keeps them, and a streamed response keeps them somewhere else.
Each API hides them in a different spot. OpenAI's Chat Completions sends usage only in one extra chunk at the end, and only if you ask for it. The Responses API attaches it to the response.completed event. Anthropic's Messages API splits it across two events, input on message_start and a running output count on message_delta.
This tutorial writes one small function per API. Each one streams text to a callback and returns the same typed object: input tokens, cached input tokens, output tokens, cost in dollars, and a flag that says whether the numbers are final. Every field name below was checked against the current source of the openai and @anthropic-ai/sdk packages.
Setup and the shared type
You need Node 22 or later (the openai README lists 22 as the minimum), both SDKs, and tsx to run TypeScript without a build step:
npm install openai @anthropic-ai/sdk
npm install -D tsx
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
All three functions return this shape. Put it in usage.ts:
export type Price = {
inputPerMTok: number;
cachedInputPerMTok: number;
outputPerMTok: number;
};
export type StreamUsage = {
inputTokens: number; // all input, cached included
cachedInputTokens: number; // subset of inputTokens
outputTokens: number;
costUsd: number;
final: boolean; // came from the closing usage event
};
export type OnText = (text: string) => void;
export function priceUsage(
u: Omit<StreamUsage, "costUsd">,
p: Price,
): StreamUsage {
const fresh = u.inputTokens - u.cachedInputTokens;
const usd =
fresh * p.inputPerMTok +
u.cachedInputTokens * p.cachedInputPerMTok +
u.outputTokens * p.outputPerMTok;
return { ...u, costUsd: usd / 1_000_000 };
}
The final flag is the part people skip. A streamed call can end three ways: the provider sends its closing usage event, the user closes the tab, or the network drops. Only the first gives you a billing-grade number. The flag lets your logger tell "this call cost $0" apart from "this call ended before anyone told us what it cost".
OpenAI Chat Completions: the opt-in last chunk
Chat Completions does not send usage on a stream unless you set stream_options: { include_usage: true }. The SDK's own doc comment spells out what happens then: one more chunk arrives before data: [DONE], its usage field covers the whole request, its choices array is always empty, and every other chunk carries usage: null (source).
That empty choices array matters. Code that does chunk.choices[0].delta.content without optional chaining throws on the very chunk you wanted.
// openai-chat.ts
import OpenAI from "openai";
import { priceUsage, type OnText } from "./usage";
const MODEL = "gpt-5.4-mini";
const PRICE = {
inputPerMTok: 0.75,
cachedInputPerMTok: 0.075,
outputPerMTok: 4.5,
};
export async function streamChat(
prompt: string,
onText: OnText,
signal?: AbortSignal,
) {
const client = new OpenAI();
let input = 0, cached = 0, output = 0;
let final = false;
try {
const stream = await client.chat.completions.create(
{
model: MODEL,
messages: [{ role: "user", content: prompt }],
stream: true,
stream_options: { include_usage: true },
},
{ signal },
);
for await (const chunk of stream) {
const text = chunk.choices[0]?.delta?.content;
if (text) onText(text);
if (chunk.usage) {
const u = chunk.usage;
input = u.prompt_tokens;
cached = u.prompt_tokens_details?.cached_tokens ?? 0;
output = u.completion_tokens;
final = true;
}
}
} catch (err) {
if (!signal?.aborted) throw err;
}
return priceUsage(
{
inputTokens: input,
cachedInputTokens: cached,
outputTokens: output,
final,
},
PRICE,
);
}
The prices are the standard rates for gpt-5.4-mini on OpenAI's pricing page as of 2026-10-17: $0.75 per million input tokens, $0.075 cached, $4.50 output. Copy yours from the same page; they change.
OpenAI counts cached tokens inside prompt_tokens; its prompt caching guide treats ordinary input as input minus cached minus cache writes, which is why priceUsage subtracts the cached part before applying the full rate.
!Three pipes carry streamed text toward a user, each with a small token meter mounted at a different point: one at the far end, one on a closing valve, one split between the start and the middle
Same idea, three places: where each API puts the token count on a stream.
The Responses API: read response.completed
The newer Responses API streams typed events instead of chunks. Text arrives as response.output_text.delta, and the run ends with response.completed, which carries the full Response object (OpenAI streaming guide). Usage sits on event.response.usage, with input_tokens, output_tokens and a required input_tokens_details.cached_tokens in the SDK types (source).
No opt-in flag here. The catch is a second terminal event: response.incomplete fires when a response finishes early, for example after hitting max_output_tokens (streaming events reference), and it carries the same response object. If you only listen for completed, every truncated answer logs zero.
// openai-responses.ts
import OpenAI from "openai";
import { priceUsage, type OnText } from "./usage";
const MODEL = "gpt-5.4-mini";
const PRICE = {
inputPerMTok: 0.75,
cachedInputPerMTok: 0.075,
outputPerMTok: 4.5,
};
export async function streamResponses(
prompt: string,
onText: OnText,
signal?: AbortSignal,
) {
const client = new OpenAI();
let input = 0, cached = 0, output = 0;
let final = false;
try {
const stream = await client.responses.create(
{ model: MODEL, input: prompt, stream: true },
{ signal },
);
for await (const event of stream) {
if (event.type === "response.output_text.delta") {
onText(event.delta);
} else if (
event.type === "response.completed" ||
event.type === "response.incomplete"
) {
const u = event.response.usage;
if (u) {
input = u.input_tokens;
cached = u.input_tokens_details.cached_tokens;
output = u.output_tokens;
final = true;
}
}
}
} catch (err) {
if (!signal?.aborted) throw err;
}
return priceUsage(
{
inputTokens: input,
cachedInputTokens: cached,
outputTokens: output,
final,
},
PRICE,
);
}
usage is optional on the Response type, hence the if (u) guard. TypeScript narrows event inside each branch, so event.delta and event.response type-check without casts.
Anthropic: input first, output last
Anthropic's stream opens with message_start, whose message.usage holds the input count. Output arrives later on message_delta, and the streaming docs put it in a warning box: the token counts in message_delta usage are cumulative. Overwrite, never add.
The docs' own example shows why the first event is a trap. Its message_start reports "input_tokens": 25, "output_tokens": 1, and the closing message_delta reports "output_tokens": 15. Log at the start and you record one output token for a fifteen-token reply.
In the SDK's MessageDeltaUsage type, input_tokens and both cache fields are number | null and described as cumulative (source). With server tools they move: the docs' web search example starts at 2,679 input tokens on message_start and ends at 10,682 on message_delta. Take the delta's value when it is not null.
// anthropic.ts
import Anthropic from "@anthropic-ai/sdk";
import { priceUsage, type OnText } from "./usage";
const MODEL = "claude-sonnet-5-5";
const PRICE = {
inputPerMTok: 2,
cachedInputPerMTok: 0.1,
outputPerMTok: 10,
};
export async function streamClaude(
prompt: string,
onText: OnText,
signal?: AbortSignal,
) {
const client = new Anthropic();
let input = 0, read = 0, write = 0, output = 0;
let final = false;
try {
const stream = await client.messages.create(
{
model: MODEL,
max_tokens: 1024,
messages: [{ role: "user", content: prompt }],
stream: true,
},
{ signal },
);
for await (const event of stream) {
if (event.type === "message_start") {
const u = event.message.usage;
input = u.input_tokens;
read = u.cache_read_input_tokens ?? 0;
write = u.cache_creation_input_tokens ?? 0;
output = u.output_tokens;
} else if (event.type === "content_block_delta") {
if (event.delta.type === "text_delta") {
onText(event.delta.text);
}
} else if (event.type === "message_delta") {
const u = event.usage;
output = u.output_tokens;
input = u.input_tokens ?? input;
read = u.cache_read_input_tokens ?? read;
write = u.cache_creation_input_tokens ?? write;
} else if (event.type === "message_stop") {
final = true;
}
}
} catch (err) {
if (!signal?.aborted) throw err;
}
return priceUsage(
{
inputTokens: input + read + write,
cachedInputTokens: read,
outputTokens: output,
final,
},
PRICE,
);
}
Note the sum at the bottom. Anthropic's input_tokens is not the total: per the prompt caching docs, total input is cache_read_input_tokens + cache_creation_input_tokens + input_tokens. OpenAI includes cached tokens in its input count and Anthropic excludes them, so one normalised type saves you from comparing two different numbers on the same dashboard.
The prices are Claude Sonnet 5.5's rates from Anthropic's pricing page as of 2026-10-17: $2 per million input, $0.10 for cache hits, $10 output. Cache writes cost more than base input there ($2.50 per million for the 5-minute tier), and OpenAI now reports cache_write_tokens too. priceUsage bills writes at the base rate, so if you use explicit caching, add a fourth rate.
If you prefer the client.messages.stream() helper, await stream.finalMessage() gives you the accumulated message with its usage. The SDK helpers doc says that promise throws if no message exists. It doesn't say what happens on abort, so don't count on finalMessage() for billing after a cancel.
The abort case: when the last event never comes
Users close tabs. Your server cancels a request when the client disconnects. Both SDKs accept an AbortSignal in the request options (openai README, Anthropic RequestOptions), and aborting makes the for await loop throw.
The OpenAI doc comment is blunt about what you lose: "If the stream is interrupted or cancelled, you may not receive the final usage chunk…" On Anthropic you keep the input count from message_start and the last cumulative output count you saw. The real bill is at least that.
That is why each function catches only when signal.aborted is true and returns final: false instead of throwing. A real network error still throws. Here is a runner, demo.ts, that can cut a stream short on purpose:
// demo.ts
import { streamChat } from "./openai-chat";
import { streamResponses } from "./openai-responses";
import { streamClaude } from "./anthropic";
const runners = {
chat: streamChat,
responses: streamResponses,
claude: streamClaude,
};
async function main() {
type Name = keyof typeof runners;
const name = (process.argv[2] ?? "chat") as Name;
const abortAfterMs = Number(process.argv[3] ?? 0);
const ac = new AbortController();
if (abortAfterMs > 0) setTimeout(() => ac.abort(), abortAfterMs);
const usage = await runners[name](
"Explain backpressure in two short paragraphs.",
(t) => process.stdout.write(t),
ac.signal,
);
console.log("\n", usage);
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
Run npx tsx demo.ts claude for a full call, then npx tsx demo.ts claude 400 to abort after 400 ms. The second run should print final: false. I haven't published output here because your token counts depend on the model version and the prompt; run it and read your own.
!A stream of text being cut by scissors partway along, with the token meter at the far end left blank
An aborted stream never reaches the event that carries the bill.
What to log, and what not to
Log every call, including the ones with final: false. Dropping them hides real spend; recording them as zero lies about it. Store the flag next to the numbers and let your cost dashboard sum only final rows, with a separate count of non-final calls so a spike in aborts is visible. And normalise input to "total, cached included" before any row reaches that sum, because the two vendors disagree on what input_tokens means.
---
If this was useful
Token cost is the first thing that surprises people once an LLM feature leaves the prototype stage, and streaming is where it hides. AI That Answers is the first book in my AI in TypeScript series: your first LLM app in TypeScript, prompts, structured output, and keeping track of what each call costs, written for developers who already ship Node code.
The rest of the series takes the same app through RAG, tool calling, agents and production: all five books here.
1 reaction · 2 comments
Mentions
Claude Reads Screenshots at 2,576px Now. The Same PNG Can Cost 3x the Tokens - Book: AI That Answers - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- In April, Simon Willison fed one 3456×2234, 3.7 MB PNG screenshot into his Claude token counter. On Claude Opus 4.6 it came to 1,578 tokens. On Opus 4.7 the same file came to 4,744. That is a 3.01x jump for one image, at the same $5 per million input tokens on both models. Same file, different model. Opus 4.7 accepts images up to 2,576 pixels on the long edge, about 3.75 megapixels, where earlier Claude models shrank anything bigger to fit 1,568. Anthropic's vision docs now put every model from Claude 4.7 on in that high-resolution tier, with no opt-in. If your app sends screenshots, the per-image cost you estimated on an older model can be a third of what the same PNG costs on a current one. Small images didn't move. In the same post, a 682×318 image cost 314 tokens on 4.7 and 310 on 4.6. The extra cost lives only in big images, and the rule that decides it fits in one line of arithmetic. Claude bills images in 28-pixel tiles Claude doesn't price pixels. The vision docs say it cuts the image into 28×28-pixel patches and charges one visual token per patch: ceil(width / 28) × ceil(height / 28). A 1000×1000 image is 36 × 36 = 1,296 tokens. Then there is a ceiling. Each model has a long-edge limit and a visual-token budget, and an image over either one is shrunk, aspect ratio kept, until it fits: | Tier | Models | Max long edge | Max visual tokens | |---|---|---|---| | High-resolution | Claude 4.7 and later | 2,576 px | 4,784 | | Standard | All other models | 1,568 px | 1,568 | The token budget is the limit that bites. Anthropic's coordinates guide says that for nearly all photos and screenshots the token limit sets the final size, not the edge: a 1920×1080 screenshot on a standard-tier model becomes 1456×819, not 1568×882. On a high-resolution model the same screenshot isn't resized at all and costs 2,691 tokens instead of 1,560. Run Simon's screenshot through those rules with the guide's reference resize function and his numbers line up. On the standard tier, 3456×2234 shrinks to 1372×887: 49 × 32 = 1,568 patches, exactly the budget. On the high-resolution tier it shrinks to 2383×1540: 86 × 55 = 4,730 patches. That ratio is 3.02x against his measured 3.01x. This is my arithmetic, not a measurement; the 10 to 14 tokens his counts carry on top are most likely the rest of the request, which a token counter includes. !A laptop screenshot overlaid with a fine grid of square tiles, one tile lifted out under a magnifying glass Claude bills per square tile, so a longer edge means more tiles on the bill. Where the /750 shortcut drifts The older shortcut, still quoted in write-ups like fast.io, is width × height / 750, which that write-up calls Anthropic's established rule of thumb. Since 28 × 28 is 784, the two agree within a few percent on mid-sized images: a 1000×1000 image is 1,333 by the shortcut and 1,296 by patches. It drifts in three places. It ignores the ceiling. 3456×2234 / 750 is 10,294 tokens, a number neither tier will ever bill, because the image is shrunk first. That estimate is about 2x too high for a high-resolution model and over 6x too high for a standard one. It ignores rounding. Patches round up on each axis. 1120×700 is exactly 40 × 25 = 1,000 tokens. Add one pixel to each side, 1121×701, and it becomes 41 × 26 = 1,066. The shortcut moves by about 2. It doesn't know your model's tier. Same image, same formula, up to 3x apart depending on the model ID in the request. What doesn't matter at all is file size. Nothing in the formula looks at bytes, and the coordinates guide notes that the token counting endpoint estimates an image's cost from its dimensions without fully processing it. A 3.7 MB PNG and a 300 KB JPEG at the same dimensions cost the same tokens. Compression only changes latency and request size, and the vision docs warn that lossy JPEG or WebP can add artifacts that make text hard to read, which is the opposite of what you want from a screenshot. File size still matters for limits: each image is capped at 10 MB base64-encoded on the Claude API and 5 MB on Amazon Bedrock. Count it yourself in TypeScript The token counting endpoint is free, rate-limited per usage tier (5,000 requests a minute on the Start tier), and works with every active model. It only counts images sent as base64, not by URL or file ID. That is enough to print your own table. The script below uses the Anthropic TypeScript SDK and sharp. It resizes one screenshot to several long edges, counts each on two models, subtracts a text-only baseline so only the image is left, and prints tokens and dollars per thousand images. I haven't run this script myself, so treat the numbers it prints as the source of truth, not the ones in this post. // count-image.mts // npm i @anthropic-ai/sdk sharp // ANTHROPIC_API_KEY=... npx tsx count-image.mts shot.png import Anthropic from "@anthropic-ai/sdk"; import sharp from "sharp"; import { readFile } from "node:fs/promises"; const client = new Anthropic(); const MODELS = [ { id: "claude-opus-4-6", usdPerMTok: 5 }, { id: "claude-opus-4-7", usdPerMTok: 5 }, ]; // long-edge targets in px; 0 = send the original size const EDGES = [0, 2576, 1568, 1092, 768]; const PROMPT = "What is on this screen?"; const patches = (w: number, h: number) => Math.ceil(w / 28) * Math.ceil(h / 28); async function count(model: string, png?: Buffer) { const text = { type: "text" as const, text: PROMPT }; const content = png ? [ { type: "image" as const, source: { type: "base64" as const, media_type: "image/png" as const, data: png.toString("base64"), }, }, text, ] : [text]; const res = await client.messages.countTokens({ model, messages: [{ role: "user", content }], }); return res.input_tokens; } Resize with sharp to each target long edge, never upscaling: async function atEdge(src: Buffer, edge: number) { const img = sharp(src); if (edge > 0) { img.resize({ width: edge, height: edge, fit: "inside", withoutEnlargement: true, }); } return img.png().toBuffer(); } The main loop counts every size on both models and prints the table: const src = await readFile(process.argv[2]); const rows: Record<string, unknown>[] = []; for (const m of MODELS) { const baseline = await count(m.id); for (const edge of EDGES) { const png = await atEdge(src, edge); const meta = await sharp(png).metadata(); const w = meta.width ?? 0; const h = meta.height ?? 0; const tokens = (await count(m.id, png)) - baseline; rows.push({ model: m.id, sent: `${w}x${h}`, kb: Math.round(png.length / 1024), predicted: patches(w, h), counted: tokens, usdPer1k: ((tokens * m.usdPerMTok) / 1000).toFixed(2), }); } } console.table(rows); The kb column will swing a lot between rows and the counted column won't follow it, because bytes aren't in the formula. And predicted is the patch count of the pixels you sent: when counted lands on or near it, Claude saw your image at the size you sent. When counted is well below it, the server shrank the image to its tier's budget, and you paid for pixels you could have dropped yourself. For a 3456×2234 shot on Opus 4.7, the original and the 2,576 rows should both land near 4,730, the size the server shrinks this shot to. A 2576×1665 image is still 92 × 60 = 5,520 patches, over the 4,784 budget. Pick the size before you ship Here is what one screenshot's image tokens cost per thousand requests, computed from the docs' token table and each model's list input price. It leaves out your prompt text and the output tokens. | Model | Tier | Input $/MTok | 1920×1080 tokens | $ per 1k | 3840×2160 tokens | $ per 1k | |---|---|---|---|---|---|---| | Haiku 4.5 | standard | $1 | 1,560 | $1.56 | 1,560 | $1.56 | | Sonnet 4.6 | standard | $3 | 1,560 | $4.68 | 1,560 | $4.68 | | Opus 4.6 | standard | $5 | 1,560 | $7.80 | 1,560 | $7.80 | | Opus 4.7 | high | $5 | 2,691 | $13.46 | 4,784 | $23.92 | | Haiku 5.5 (≤100K prompt) | high | $0.10 | 2,691 | $0.27 | 4,784 | $0.48 | | Sonnet 5.5 | high | $2 | 2,691 | $5.38 | 4,784 | $9.57 | | Opus 5.5 | high | $4 | 2,691 | $10.76 | 4,784 | $19.14 | The Sonnet rows are the ones to stare at. Sonnet 5.5's input price is a third lower than Sonnet 4.6's, and a 4K screenshot costs about twice as much on it. Each token got cheaper, but the same image now takes about three times as many of them. !A balance scale with two identical screenshots, one side weighed down by a much taller stack of coins Same screenshot, same file: the model's resolution tier decides how heavy the bill is. If you don't need the extra detail, shrink before you send. Anthropic's docs say it directly: downsample when you don't need the fidelity, and resize yourself to cut latency. Copy resizedSize from the coordinates guide (it ships a TypeScript version) and pin screenshots to the old standard budget on any model: // resizedSize() is Anthropic's reference function, // copied from the vision-coordinates guide. const STANDARD = { edge: 1568, tokens: 1568 }; async function pinToStandard(src: Buffer) { const meta = await sharp(src).metadata(); const [w, h] = resizedSize( meta.width ?? 0, meta.height ?? 0, STANDARD.edge, STANDARD.tokens, ); return sharp(src) .resize({ width: w, height: h, fit: "fill" }) .png() .toBuffer(); } Don't do it blind, though. The same docs say the high-resolution tier exists for computer use, screenshot understanding and dense documents, and small text is the first thing a downscale eats. Run your real screenshots through both sizes, check the answers, and only then pick the budget. Screenshot apps hit a few more rules. A request with more than 20 images triggers a stricter per-image size limit, and the vision docs say to keep both sides at 3,000 px or under to stay clear on every platform. Screenshots returned to the computer use and browser use tools are rejected with a validation error when they exceed the model's limits instead of being downscaled. And if a silent server-side resize would break your pipeline, set "transformations": { "oversized_image": "error" } on the image block: the API, and the token counting endpoint, then reject the image instead of shrinking it, per the coordinates guide. The habit that saves you is boring. Before a feature that takes screenshots ships, count one real screenshot on the exact model ID you'll deploy, write the per-image number next to the feature, and recount when you switch models. --- If this was useful Per-request token cost is one of the first things a working LLM app has to get right, and it's a big part of what AI That Answers covers: your first LLM app in TypeScript, from prompts and structured output to knowing what each call costs before your users find out for you. If you want the screenshot feature above to ship with a budget instead of a surprise, that book is where I'd start.  It's book one of AI in TypeScript, five books from your first LLM call to agents in production: the whole series is here.
Evaluating AI agents: small evals in TypeScript Evaluating AI agents quickly leads to an uncomfortable question: how do you actually notice that your agent got worse after a change? A new system prompt, a model switch or a reworded tool description is enough, and suddenly the assistant reserves books that are sitting on the shelf. Without tests, you only find out when someone complains. For regular code the answer is simple: unit tests. With agents that only half works, because the same model answers the same question differently from run to run. That is exactly what evals are for: repeatable checks that give an agent real tasks and grade the result. In this article I build a small eval suite in TypeScript with you, from an empty folder to a run in CI. I deliberately skip frameworks so you can see what happens under the hood. > In short > > Evals give an AI agent real tasks in a fresh test environment and then grade the outcome, not the path that led there. To get started, 20 to 50 tasks taken from real failures, several trials per task and the pass^k metric for everything that has to work reliably are enough. The biggest pitfall is a broken grader, which is why you should read the transcripts regularly. What you will learn in this article - What task, trial, grader and transcript mean in agent evals - How to build a small eval harness in TypeScript without a framework - When a code grader is enough and when a model judge makes sense - Why pass@k and pass^k tell such different stories - How regression evals in CI catch problems before a release Why evaluating AI agents matters An agent has three parts: the model, a loop that connects the model to its tools, and the tools themselves. You turn all three knobs in everyday work. You swap the model because a newer one is faster or cheaper. You refine the system prompt. On top of that you polish tool descriptions, as I showed in How to write tool descriptions for AI agents. Each of these changes can improve one thing and break another. Without evals, all you have is the gut-feeling test: try three or four prompts by hand, the answer looks fine, done. Anthropic's engineering team sums up the value of evals in one sentence: “Evals make problems and behavioral changes visible before they affect users.” That is the whole point. You want to see the regression before your users feel it. There is also a quirk of language models. A single run tells you little because answers vary. If a task succeeds once, you do not know whether it succeeds nine times out of ten or three. Evaluating AI agents therefore means running every task several times and computing metrics from the results. More on that later. The key terms in agent evals Before we write code, a shared vocabulary helps. I stick to the terms from Anthropic's article “Demystifying evals for AI agents” because they map well onto your own code. The table also shows where you will find each term in the example. | Term | Meaning | In the example | | --- | --- | --- | | Task | A single check: the input plus the criterion for passing it | an entry in tasks.ts | | Trial | One single run of a task | one call to runTrial() | | Grader | Logic that scores one aspect of the result | functions in graders.ts | | Transcript | The full record of a trial with all messages and tool calls | one line in results/*.jsonl | | Outcome | The state of the environment after the trial | the reservations in the library | | Eval harness | The infrastructure that runs, grades and summarizes tasks | runner.ts | | Agent harness | The loop that turns the model into an agent | agent.ts | | Eval suite | A set of related tasks | all tasks of the lending assistant | Keep two kinds of tasks apart from day one. Capability evals ask what the agent already does well. They may start with low pass rates, because they show you where you want to go. Regression evals, on the other hand, ask whether the agent can still do everything it could do before. They should sit close to 100 percent. Once a capability task reaches consistently high scores, it moves into the regression suite. A lending assistant as the example The example is a made-up lending assistant for a city library. It has three tools: search_books searches the catalog, check_availability shows free copies, and reserve_book creates a reservation. Two business rules come on top. A reservation is only made when no copy is available. And if the member number is missing, the agent asks for it instead of guessing. What I like about this example: whether the agent did its job is visible in the state of the library. Is the right reservation there after the trial? Is there no wrong one? That is exactly the kind of outcome code can check reliably. This way you can evaluate AI agents without needing a human or a second model for every answer. The following diagram shows how a trial runs. > 🎬 How one trial runs in the eval harness – this animated diagram only runs on my site: watch it on sascha-ms.de Prerequisites You need Node.js 22 or later and npm. For real runs you need either an API key for the Claude API or an AWS account with access to a Claude model in Amazon Bedrock. The tests for the harness itself run without a key and at no cost. Basic TypeScript is all you need; experience with agent frameworks is not required. Building the eval suite step by step 1. Create the project Create a new folder and install the Anthropic SDK, the Bedrock SDK and, as development tools, TypeScript and tsx. With tsx you run TypeScript files directly without compiling them first. mkdir ai-agent-evals && cd ai-agent-evals npm init -y npm install @anthropic-ai/sdk@1.3.0 @anthropic-ai/bedrock-sdk@1.1.0 npm install -D typescript@7.0.2 tsx@4.23.15 @types/node@22 Then replace package.json with this version. The important parts are "type": "module" and the three scripts. { "name": "ai-agent-evals", "version": "1.0.0", "private": true, "type": "module", "scripts": { "typecheck": "tsc --noEmit", "test": "node --import tsx --test test/*.test.ts", "evals": "tsx src/run-evals.ts" }, "dependencies": { "@anthropic-ai/bedrock-sdk": "1.1.0", "@anthropic-ai/sdk": "1.3.0" }, "devDependencies": { "@types/node": "^22.20.0", "tsx": "4.23.15", "typescript": "7.0.2" } } // tsconfig.json { "compilerOptions": { "target": "ES2023", "module": "NodeNext", "moduleResolution": "NodeNext", "strict": true, "noEmit": true, "skipLibCheck": true, "allowImportingTsExtensions": true, "types": ["node"] }, "include": ["src", "test"] } 2. A test environment with a fresh state The library lives purely in memory. That is on purpose: every trial gets a new, clean state from createLibrary(). If trials shared a database, a reservation from trial 1 could skew the result of trial 2. Errors like that are especially nasty because they look random. // src/library.ts // A small, made-up city library as the test environment for the agent. // Every trial gets a fresh state from createLibrary(). export interface Book { id: string; title: string; author: string; copies: number; } export interface Reservation { bookId: string; memberId: string; } export interface LibraryState { books: Book[]; loans: Record<string, number>; // copies on loan per book ID reservations: Reservation[]; } export function createLibrary(): LibraryState { return { books: [ { id: 'b-101', title: 'The Lighthouse on the Dyke', author: 'Jana Thiel', copies: 2 }, { id: 'b-102', title: 'Cooking with Kids', author: 'Ole Brandt', copies: 1 }, { id: 'b-103', title: 'TypeScript for Beginners', author: 'Mira Okafor', copies: 3 }, { id: 'b-104', title: 'The Garden in Winter', author: 'Paul Seidel', copies: 1 }, ], loans: { 'b-101': 2, 'b-102': 0, 'b-103': 1, 'b-104': 1 }, reservations: [], }; } export function availableCopies(state: LibraryState, bookId: string): number { const book = state.books.find((b) => b.id === bookId); if (!book) return 0; return book.copies - (state.loans[bookId] ?? 0); } 3. The agent's tools The tools validate their own input and answer problems with an understandable error message. That way the agent can correct itself instead of getting stuck. Note that reserve_book refuses to reserve an available book. The eval still checks whether the agent even tries, because a tool in a real system is not always that strict. // src/tools.ts // Three tools the agent may use. Their descriptions are part of what // your evals test as well. import { availableCopies, type LibraryState } from './library.ts'; export interface ToolDefinition { name: string; description: string; input_schema: { type: 'object'; properties: Record<string, unknown>; required: string[]; }; } export interface ToolResult { content: string; isError: boolean; } export const tools: ToolDefinition[] = [ { name: 'search_books', description: 'Searches the city library catalog by title or author. ' + 'Returns book IDs (e.g. "b-101") that the other tools need.', input_schema: { type: 'object', properties: { query: { type: 'string', description: 'Search term, e.g. part of the title' } }, required: ['query'], }, }, { name: 'check_availability', description: 'Shows how many copies of a book can be borrowed right now. Book ID from search_books.', input_schema: { type: 'object', properties: { bookId: { type: 'string' } }, required: ['bookId'], }, }, { name: 'reserve_book', description: 'Reserves a book for a member. Only use it when no copy is available ' + 'and the user has given their member number (format "M-<number>"). Never guess.', input_schema: { type: 'object', properties: { bookId: { type: 'string' }, memberId: { type: 'string', description: 'Member number, e.g. "M-17"' }, }, required: ['bookId', 'memberId'], }, }, ]; export function executeTool(state: LibraryState, name: string, input: unknown): ToolResult { const args = (input ?? {}) as Record<string, string>; switch (name) { case 'search_books': { const query = String(args.query ?? '').toLowerCase(); const hits = state.books.filter( (b) => b.title.toLowerCase().includes(query) || b.author.toLowerCase().includes(query), ); return { content: JSON.stringify(hits.map(({ id, title, author }) => ({ id, title, author }))), isError: false }; } case 'check_availability': { if (!state.books.some((b) => b.id === args.bookId)) { return { content: `Unknown book ID "${args.bookId}". Use search_books first.`, isError: true }; } return { content: JSON.stringify({ bookId: args.bookId, available: availableCopies(state, args.bookId) }), isError: false }; } case 'reserve_book': { if (!state.books.some((b) => b.id === args.bookId)) { return { content: `Unknown book ID "${args.bookId}".`, isError: true }; } if (!/^M-\d+$/.test(args.memberId ?? '')) { return { content: 'Invalid member number. Please ask the user.', isError: true }; } if (availableCopies(state, args.bookId) > 0) { return { content: 'The book is available, no reservation needed.', isError: true }; } const exists = state.reservations.some((r) => r.bookId === args.bookId && r.memberId === args.memberId); if (exists) { return { content: 'This member already has a reservation.', isError: false }; } state.reservations.push({ bookId: args.bookId, memberId: args.memberId }); return { content: 'Reservation created.', isError: false }; } default: return { content: `Unknown tool "${name}".`, isError: true }; } } 4. The agent loop The loop is deliberately simple. It sends the history to the model, runs the requested tools, appends the results and repeats until the model stops calling tools or the maximum number of turns is reached. Along the way it collects everything for the transcript: messages, tool calls, turns and tokens. The trick for testability lies in the ModelClient type. The loop only knows a function that takes a request and returns a response. Whether the Claude API, Amazon Bedrock or a test script sits behind it does not matter to the loop. // src/agent.ts // A lean agent loop: ask the model, run tools, return the results // until the model is done. Everything ends up in the transcript. import type { LibraryState } from './library.ts'; import { executeTool, tools, type ToolDefinition } from './tools.ts'; export type ContentBlock = | { type: 'text'; text: string } | { type: 'tool_use'; id: string; name: string; input: unknown } | { type: 'tool_result'; tool_use_id: string; content: string; is_error: boolean }; export interface Message { role: 'user' | 'assistant'; content: string | ContentBlock[]; } export interface ModelRequest { system: string; messages: Message[]; tools: ToolDefinition[]; max_tokens: number; } export interface ModelResponse { content: ContentBlock[]; stop_reason: string | null; usage: { input_tokens: number; output_tokens: number }; } // Any function with this signature can be the model: the Anthropic API, // Amazon Bedrock or, in tests, a script with fixed answers. export type ModelClient = (request: ModelRequest) => Promise<ModelResponse>; export interface ToolCall { name: string; input: unknown; isError: boolean; } export interface Transcript { messages: Message[]; toolCalls: ToolCall[]; finalText: string; turns: number; inputTokens: number; outputTokens: number; stopReason: string | null; } export const SYSTEM_PROMPT = 'You are the lending assistant of the city library. Help members find, ' + 'check and reserve books. Use the tools instead of guessing. If information ' + 'such as the member number is missing, ask for it. Answer briefly and kindly.'; export async function runAgent( client: ModelClient, state: LibraryState, prompt: string, maxTurns = 8, ): Promise<Transcript> { const messages: Message[] = [{ role: 'user', content: prompt }]; const transcript: Transcript = { messages, toolCalls: [], finalText: '', turns: 0, inputTokens: 0, outputTokens: 0, stopReason: null, }; while (transcript.turns < maxTurns) { const response = await client({ system: SYSTEM_PROMPT, messages, tools, max_tokens: 1024 }); transcript.turns += 1; transcript.inputTokens += response.usage.input_tokens; transcript.outputTokens += response.usage.output_tokens; transcript.stopReason = response.stop_reason; messages.push({ role: 'assistant', content: response.content }); const text = response.content .filter((b): b is Extract<ContentBlock, { type: 'text' }> => b.type === 'text') .map((b) => b.text) .join('\n') .trim(); if (text) transcript.finalText = text; if (response.stop_reason !== 'tool_use') break; const results: ContentBlock[] = []; for (const block of response.content) { if (block.type !== 'tool_use') continue; const result = executeTool(state, block.name, block.input); transcript.toolCalls.push({ name: block.name, input: block.input, isError: result.isError }); results.push({ type: 'tool_result', tool_use_id: block.id, content: result.content, is_error: result.isError }); } messages.push({ role: 'user', content: results }); } return transcript; } The adapters for real models are short because both SDKs offer the same messages.create() method. The Bedrock client uses your regular AWS credentials and the region from AWS_REGION. // src/model-clients.ts // Adapters for real models. The agent loop only knows ModelClient. import Anthropic from '@anthropic-ai/sdk'; import { AnthropicBedrock } from '@anthropic-ai/bedrock-sdk'; import type { ContentBlock, ModelClient } from './agent.ts'; // Both SDK clients offer messages.create() with the same shape. interface MessagesApi { messages: { create(params: Anthropic.MessageCreateParamsNonStreaming): PromiseLike<Anthropic.Message>; }; } export function messagesClient(sdk: MessagesApi, model: string): ModelClient { return async (request) => { const response = await sdk.messages.create({ model, max_tokens: request.max_tokens, system: request.system, tools: request.tools, messages: request.messages as Anthropic.MessageParam[], }); return { // We only pass on text and tool blocks; the loop ignores everything else. content: response.content.filter((b) => b.type === 'text' || b.type === 'tool_use') as ContentBlock[], stop_reason: response.stop_reason, usage: { input_tokens: response.usage.input_tokens ?? 0, output_tokens: response.usage.output_tokens ?? 0, }, }; }; } // Anthropic API: reads ANTHROPIC_API_KEY from the environment. export function anthropicClient(model: string): ModelClient { return messagesClient(new Anthropic(), model); } // Amazon Bedrock: uses your regular AWS credentials and the region from AWS_REGION. export function bedrockClient(model: string): ModelClient { return messagesClient(new AnthropicBedrock(), model); } 5. Tasks and graders Now comes the part that takes the most thinking. A grader receives the transcript and the final state of the library and returns “pass” or “fail” with a short reason. Most graders here are code graders. They are fast, free and always return the same verdict for the same input. For soft criteria such as tone and clarity there is llmJudge. Here a model grades the answer against one criterion. The “unknown” option matters: a judge that is allowed to admit uncertainty invents verdicts less often. I count “unknown” as a fail, so it automatically ends up in manual review. // src/graders.ts // Graders check the outcome of a trial. Code graders are fast and // reproducible; the model grader is meant for soft criteria such as tone. import type { ModelClient, Transcript } from './agent.ts'; import type { LibraryState } from './library.ts'; export interface GradeResult { grader: string; pass: boolean; detail: string; } export type Grader = (run: Transcript, state: LibraryState) => GradeResult | Promise<GradeResult>; // Check the outcome: does the reservation really exist in the library state? export const reservationExists = (bookId: string, memberId: string): Grader => (_run, state) => { const pass = state.reservations.some((r) => r.bookId === bookId && r.memberId === memberId); return { grader: `reservationExists(${bookId}, ${memberId})`, pass, detail: JSON.stringify(state.reservations) }; }; export const reservationCount = (expected: number): Grader => (_run, state) => ({ grader: `reservationCount(${expected})`, pass: state.reservations.length === expected, detail: `found: ${state.reservations.length}`, }); // Counter-check: a tool must not be called at all in this task. export const toolNotCalled = (name: string): Grader => (run) => { const calls = run.toolCalls.filter((c) => c.name === name).length; return { grader: `toolNotCalled(${name})`, pass: calls === 0, detail: `calls: ${calls}` }; }; export const toolCalled = (name: string): Grader => (run) => { const calls = run.toolCalls.filter((c) => c.name === name && !c.isError).length; return { grader: `toolCalled(${name})`, pass: calls > 0, detail: `successful calls: ${calls}` }; }; // Transcript metric: the agent should not run in circles forever. export const maxTurns = (limit: number): Grader => (run) => ({ grader: `maxTurns(${limit})`, pass: run.turns <= limit, detail: `turns: ${run.turns}` }); export const asksQuestion = (): Grader => (run) => ({ grader: 'asksQuestion', pass: run.finalText.includes('?'), detail: run.finalText.slice(0, 120), }); // Model as judge: grades a soft criterion and may answer "unknown". // An "unknown" counts as a fail and belongs in manual review. export const llmJudge = (judge: ModelClient, criterion: string): Grader => async (run) => { const response = await judge({ system: 'You grade the answer of a library assistant. Reply with JSON only: ' + '{"verdict":"pass"|"fail"|"unknown","reason":"<one sentence>"}. Use "unknown" if you are unsure.', messages: [{ role: 'user', content: `Criterion: ${criterion}\n\nAssistant answer:\n${run.finalText}` }], tools: [], max_tokens: 200, }); const text = response.content.map((b) => (b.type === 'text' ? b.text : '')).join(''); try { const parsed = JSON.parse(text.slice(text.indexOf('{'), text.lastIndexOf('}') + 1)) as { verdict?: string; reason?: string; }; return { grader: `llmJudge(${criterion})`, pass: parsed.verdict === 'pass', detail: `${parsed.verdict}: ${parsed.reason ?? ''}` }; } catch { return { grader: `llmJudge(${criterion})`, pass: false, detail: `unreadable: ${text.slice(0, 80)}` }; } }; The tasks cover both directions. Some cases require the agent to reserve, others forbid it outright. If you only test positive cases, you end up with an agent that acts too eagerly. The task “two-books-at-once” is a real capability task: the agent has to figure out on its own how to search the whole catalog. // src/tasks.ts // The tasks of the eval suite. Each task has a clear request, a // starting state and graders that check the outcome. import type { LibraryState } from './library.ts'; import { asksQuestion, maxTurns, reservationCount, reservationExists, toolCalled, toolNotCalled, type Grader } from './graders.ts'; export interface Task { id: string; // capability: what should the agent be able to do? regression: what must never break again? kind: 'capability' | 'regression'; prompt: string; setup?: (state: LibraryState) => void; graders: Grader[]; } export const tasks: Task[] = [ { id: 'reserve-when-unavailable', kind: 'regression', prompt: 'Hi, I am member M-17. Please reserve “The Lighthouse on the Dyke” for me.', graders: [reservationExists('b-101', 'M-17'), reservationCount(1), maxTurns(5)], }, { id: 'no-reservation-when-available', kind: 'regression', prompt: 'Member M-17 here. Please reserve “Cooking with Kids” for me.', graders: [reservationCount(0), toolNotCalled('reserve_book')], }, { id: 'member-number-missing', kind: 'regression', prompt: 'Can you reserve “The Garden in Winter” for me?', graders: [reservationCount(0), toolNotCalled('reserve_book'), asksQuestion()], }, { id: 'unknown-book', kind: 'capability', prompt: 'I am M-8 and would like to reserve “Advanced Sailing”.', graders: [reservationCount(0), toolNotCalled('reserve_book')], }, { id: 'availability-by-author', kind: 'capability', prompt: 'Do you have anything by Ole Brandt? Can I borrow it today?', graders: [toolCalled('check_availability'), reservationCount(0)], }, { id: 'two-books-at-once', kind: 'capability', prompt: 'I am M-23. Please reserve everything that is fully on loan right now.', graders: [reservationExists('b-101', 'M-23'), reservationExists('b-104', 'M-23'), reservationCount(2)], }, ]; 6. Metrics and harness The harness runs every task several times, each time in a fresh library, with limited concurrency. That keeps you below the API rate limits. If a trial aborts with an error, for example a timeout, the harness flags it as an infrastructure error. This matters so you do not mistake a network error for an agent error. Besides “pass” there is partial credit: the share of graders that passed. An agent that creates the reservation correctly but needs seven turns for it is better than one that gets nothing done. I explain the pass@k and pass^k metrics in detail in the next section. // src/metrics.ts // pass@k: how likely is at least one success in k trials? // pass^k: how likely are all k trials successful? // Both are estimated from n trials with c successes. function binomial(n: number, k: number): number { if (k < 0 || k > n) return 0; let result = 1; for (let i = 1; i <= k; i++) result = (result * (n - k + i)) / i; return result; } export function passAtK(n: number, c: number, k: number): number { if (k > n) throw new Error(`k (${k}) must not be greater than n (${n})`); if (n - c < k) return 1; return 1 - binomial(n - c, k) / binomial(n, k); } export function passHatK(n: number, c: number, k: number): number { if (k > n) throw new Error(`k (${k}) must not be greater than n (${n})`); return binomial(c, k) / binomial(n, k); } // src/runner.ts // The eval harness: runs every task several times in a fresh library, // grades the trials and summarizes them. import { runAgent, type ModelClient, type Transcript } from './agent.ts'; import type { Grader, GradeResult } from './graders.ts'; import { createLibrary } from './library.ts'; import { passAtK, passHatK } from './metrics.ts'; import type { Task } from './tasks.ts'; export interface TrialResult { taskId: string; trial: number; pass: boolean; score: number; // share of passed graders, for partial credit grades: GradeResult[]; transcript?: Transcript; error?: string; } export interface TaskSummary { taskId: string; kind: Task['kind']; trials: number; successes: number; avgScore: number; passAtK: number; passHatK: number; avgTurns: number; avgTokens: number; } export interface EvalOptions { trials: number; k: number; concurrency: number; extraGraders?: Grader[]; } async function runTrial(client: ModelClient, task: Task, trial: number, extra: Grader[]): Promise<TrialResult> { const state = createLibrary(); // every trial starts clean task.setup?.(state); try { const transcript = await runAgent(client, state, task.prompt); const grades: GradeResult[] = []; for (const grader of [...task.graders, ...extra]) grades.push(await grader(transcript, state)); const passed = grades.filter((g) => g.pass).length; return { taskId: task.id, trial, pass: passed === grades.length, score: passed / grades.length, grades, transcript }; } catch (error) { // An infrastructure error is not the agent's fault, so it is flagged separately. return { taskId: task.id, trial, pass: false, score: 0, grades: [], error: String(error) }; } } async function pool<T>(jobs: Array<() => Promise<T>>, concurrency: number): Promise<T[]> { const results: T[] = new Array(jobs.length); let next = 0; const workers = Array.from({ length: Math.min(concurrency, jobs.length) }, async () => { while (next < jobs.length) { const index = next++; results[index] = await jobs[index](); } }); await Promise.all(workers); return results; } export async function runEvals(client: ModelClient, tasks: Task[], options: EvalOptions) { const jobs = tasks.flatMap((task) => Array.from({ length: options.trials }, (_, trial) => () => runTrial(client, task, trial + 1, options.extraGraders ?? [])), ); const results = await pool(jobs, options.concurrency); const summaries: TaskSummary[] = tasks.map((task) => { const own = results.filter((r) => r.taskId === task.id); const successes = own.filter((r) => r.pass).length; const withTranscript = own.filter((r) => r.transcript); const avg = (values: number[]) => (values.length ? values.reduce((a, b) => a + b, 0) / values.length : 0); return { taskId: task.id, kind: task.kind, trials: own.length, successes, avgScore: avg(own.map((r) => r.score)), passAtK: passAtK(own.length, successes, options.k), passHatK: passHatK(own.length, successes, options.k), avgTurns: avg(withTranscript.map((r) => r.transcript!.turns)), avgTokens: avg(withTranscript.map((r) => r.transcript!.inputTokens + r.transcript!.outputTokens)), }; }); return { results, summaries }; } The command-line script ties everything together. It stores every trial as a line in a JSONL file, prints a table and sets an exit code. Exit code 1 means a regression task is below 90 percent. Exit code 2 means the run is invalid because trials failed on infrastructure. // src/run-evals.ts // Usage: EVAL_MODEL=<model-id> npm run evals // Optional: EVAL_PROVIDER=bedrock, EVAL_TRIALS=5, EVAL_K=3, EVAL_JUDGE_MODEL=<model-id> import { mkdir, writeFile } from 'node:fs/promises'; import { llmJudge } from './graders.ts'; import { anthropicClient, bedrockClient } from './model-clients.ts'; import { runEvals } from './runner.ts'; import { tasks } from './tasks.ts'; const model = process.env.EVAL_MODEL; if (!model) { console.error('Please set EVAL_MODEL, e.g. EVAL_MODEL=claude-sonnet-5-5'); process.exit(2); } const makeClient = process.env.EVAL_PROVIDER === 'bedrock' ? bedrockClient : anthropicClient; const client = makeClient(model); const trials = Number(process.env.EVAL_TRIALS ?? 5); const k = Number(process.env.EVAL_K ?? 3); const judgeModel = process.env.EVAL_JUDGE_MODEL; const extraGraders = judgeModel ? [llmJudge(makeClient(judgeModel), 'The answer is friendly, factually correct and tells the member the next step.')] : []; const { results, summaries } = await runEvals(client, tasks, { trials, k, concurrency: 3, extraGraders }); // Save all transcripts: reading them is the most important part of the analysis. await mkdir('results', { recursive: true }); const file = `results/${new Date().toISOString().replace(/[:.]/g, '-')}.jsonl`; await writeFile(file, results.map((r) => JSON.stringify(r)).join('\n') + '\n'); console.table( summaries.map((s) => ({ Task: s.taskId, Kind: s.kind, Successes: `${s.successes}/${s.trials}`, Partial: s.avgScore.toFixed(2), [`pass@${k}`]: s.passAtK.toFixed(2), [`pass^${k}`]: s.passHatK.toFixed(2), Turns: s.avgTurns.toFixed(1), Tokens: Math.round(s.avgTokens), })), ); console.log(`Transcripts: ${file}`); const infraErrors = results.filter((r) => r.error).length; if (infraErrors > 0) { // A run with API errors says nothing about the agent: better invalid than falsely red. console.error(`${infraErrors} trials with infrastructure errors, run invalid. See ${file}`); process.exit(2); } // Regression tasks must (almost) always pass, otherwise the run fails. const broken = summaries.filter((s) => s.kind === 'regression' && s.successes / s.trials < 0.9); if (broken.length > 0) { console.error(`Regression: ${broken.map((s) => s.taskId).join(', ')}`); process.exit(1); } 7. Testing the harness itself An eval harness is code and can have bugs. A broken grader is even particularly dangerous, because it gives you false confidence. That is why I test the harness with a canned model: the scriptedClient returns predefined answers in order. With it you check that a good solution passes and a bad one fails. The tests for the metrics secure the calculations. // test/helpers.ts // A model for tests: returns predefined answers in order. import type { ContentBlock, ModelClient, ModelRequest, ModelResponse } from '../src/agent.ts'; export function scriptedClient(steps: ContentBlock[][]): ModelClient & { requests: ModelRequest[] } { let index = 0; const requests: ModelRequest[] = []; const client = async (request: ModelRequest): Promise<ModelResponse> => { requests.push(structuredClone(request)); const content = steps[Math.min(index++, steps.length - 1)]; const usesTool = content.some((b) => b.type === 'tool_use'); return { content, stop_reason: usesTool ? 'tool_use' : 'end_turn', usage: { input_tokens: 100, output_tokens: 20 } }; }; return Object.assign(client, { requests }); } export const toolUse = (id: string, name: string, input: unknown): ContentBlock => ({ type: 'tool_use', id, name, input }); export const text = (value: string): ContentBlock => ({ type: 'text', text: value }); // test/graders.test.ts import assert from 'node:assert/strict'; import { test } from 'node:test'; import { runAgent } from '../src/agent.ts'; import { llmJudge } from '../src/graders.ts'; import { createLibrary } from '../src/library.ts'; import { tasks } from '../src/tasks.ts'; import { scriptedClient, text, toolUse } from './helpers.ts'; const task = (id: string) => tasks.find((t) => t.id === id)!; async function grade(id: string, steps: Parameters<typeof scriptedClient>[0]) { const state = createLibrary(); task(id).setup?.(state); const run = await runAgent(scriptedClient(steps), state, task(id).prompt); return Promise.all(task(id).graders.map((g) => g(run, state))); } test('a good solution passes “reserve-when-unavailable”', async () => { const grades = await grade('reserve-when-unavailable', [ [toolUse('t1', 'search_books', { query: 'Lighthouse' })], [toolUse('t2', 'reserve_book', { bookId: 'b-101', memberId: 'M-17' })], [text('Reserved.')], ]); assert.ok(grades.every((g) => g.pass), JSON.stringify(grades)); }); test('an agent that reserves an available book fails', async () => { const grades = await grade('no-reservation-when-available', [ [toolUse('t1', 'reserve_book', { bookId: 'b-102', memberId: 'M-17' })], [text('Sorry, that did not work.')], ]); const failed = grades.filter((g) => !g.pass).map((g) => g.grader); assert.deepEqual(failed, ['toolNotCalled(reserve_book)']); }); test('missing member number: asking back passes', async () => { const grades = await grade('member-number-missing', [[text('Sure! What is your member number?')]]); assert.ok(grades.every((g) => g.pass)); }); test('model judge: "unknown" does not count as a pass', async () => { const judge = scriptedClient([[text('{"verdict":"unknown","reason":"The answer is ambiguous"}')]]); const run = await runAgent(scriptedClient([[text('Hm.')]]), createLibrary(), 'Hello'); const result = await llmJudge(judge, 'friendly')(run, createLibrary()); assert.equal(result.pass, false); assert.match(result.detail, /^unknown/); }); // test/metrics.test.ts import assert from 'node:assert/strict'; import { test } from 'node:test'; import { passAtK, passHatK } from '../src/metrics.ts'; test('at k = 1, pass@k and pass^k equal the success rate', () => { assert.equal(passAtK(10, 7, 1), 0.7); assert.equal(passHatK(10, 7, 1), 0.7); }); test('pass@k rises and pass^k falls as k grows', () => { assert.ok(passAtK(10, 7, 3) > passAtK(10, 7, 1)); assert.ok(passHatK(10, 7, 3) < passHatK(10, 7, 1)); }); test('edge cases', () => { assert.equal(passAtK(5, 0, 3), 0); assert.equal(passHatK(5, 5, 3), 1); assert.equal(passHatK(5, 2, 3), 0); assert.throws(() => passAtK(2, 1, 3)); }); Now check types and tests. Both run without an API key. npm run typecheck npm test Expected result: tsc reports nothing, and the test runner ends with # pass 7 and # fail 0. 8. The first real run Set the API key and the model and start the evals. EVAL_TRIALS sets how often each task runs, EVAL_K sets k for the metrics. export ANTHROPIC_API_KEY="your-key" EVAL_MODEL=claude-sonnet-5-5 EVAL_TRIALS=5 EVAL_K=3 npm run evals With Amazon Bedrock the call looks like this. As the model, enter the model ID or inference profile that is enabled in your region. export AWS_REGION=eu-central-1 EVAL_PROVIDER=bedrock EVAL_MODEL="<your-bedrock-model-id>" npm run evals You get a table with one row per task: successes, partial credit, pass@3, pass^3, average turns and tokens. Below it is the path to the JSONL file with all transcripts. I am not giving you numbers here, because they depend on the model, the day and the tool descriptions. I do have one expectation, though: the regression tasks should score very high with a current model, while a low value for “two-books-at-once” would not surprise me. That is exactly what a capability task is for. If the key is missing, every trial fails as an infrastructure error and the script exits with code 2. That is intentional: it keeps you from reading a broken run as a bad agent. Which metric counts, pass@k or pass^k? Both metrics are based on k trials but answer opposite questions. pass@k is the probability that at least one of k trials succeeds. It rises with k and fits tools where one good solution is enough, for example when a developer picks from several suggestions. pass^k asks how likely it is that every single one of the k trials works. This value drops as k grows and fits everything customers use directly. A lending assistant that reserves nonsense every third time is useless in practice, even if it “mostly” works. Anthropic works through an example: at 75 percent success per trial, the chance that three out of three trials succeed drops to about 42 percent (0.75 to the power of 3). The chart shows how quickly the two curves drift apart. !Line chart: pass@k rises from 75% to almost 100%, pass^k falls from 75% to 24% for k from 1 to 5 At 75% success per trial, pass@k quickly approaches 100%, while pass^k drops to about 42% at k = 3. In metrics.ts I estimate both values from n trials with c successes. For pass@k I use the estimator from OpenAI's Codex paper: one minus the probability of drawing k trials only from the failures. For pass^k, which comes from the agent benchmark τ-bench, I draw k trials only from the successes. A practical rule follows: n must be at least as large as k, and the larger n, the more stable the estimate. With five trials per task and k = 3 you get a useful first impression. Reading transcripts instead of just percentages The table tells you that something is wrong. Why it is wrong is only in the transcript. Open the JSONL file and read at least the failed trials. You will find two kinds of errors, and you need to tell them apart. The first kind are real agent errors. The agent calls reserve_book with a guessed member number or misses that a copy is available. A better tool description, a clearer system prompt or a different model helps here. The second kind are errors in the eval itself. An example from my own code: asksQuestion passes as soon as there is a question mark anywhere. If the agent answers “Did you see our new opening hours?” and makes a wrong reservation at the same time, only the other two graders of that task catch it. That is why I always combine soft checks with a hard check of the outcome. If a task fails in almost every trial, look at the task and the grader first, then at the agent. Often the task is simply ambiguous. Evaluating AI agents in CI It gets really valuable once you are evaluating AI agents automatically, without having to remember it. I would hook the evals in at two points: on every pull request that changes prompts, tools or agent code, and once per working day against the current model. The daily run catches drift, meaning behavior changes that happen without a commit of your own, for example when a model alias points to a new version. The following GitHub Actions workflow first runs the free tests and then the evals. The transcripts are attached to the run as an artifact, so you can read them even when it fails. # .github/workflows/evals.yml name: Agent-Evals on: pull_request: paths: ['src/**'] schedule: - cron: '17 3 * * 1-5' # early on weekdays so model drift shows up workflow_dispatch: jobs: evals: runs-on: ubuntu-latest timeout-minutes: 20 steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 24 cache: npm - run: npm ci - run: npm test - run: npm run evals env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} EVAL_MODEL: claude-sonnet-5-5 EVAL_TRIALS: '5' EVAL_K: '3' - uses: actions/upload-artifact@v7 if: always() with: name: eval-transcripts path: results/ Create the repository secret ANTHROPIC_API_KEY for this. Pull requests from forks do not get secrets on GitHub by default. The eval step fails there with exit code 2, and that is how it should be: outside code should not run with your API key. Common mistakes when evaluating agents Only one trial per task One trial is an anecdote. Only with several trials can you see whether a result is stable. Five trials are a good start; use more for important regression tasks. Checking the path instead of the outcome If you prescribe that the agent must first search, then check and then reserve, you punish valid shortcuts. Check the final state. I make one exception for forbidden side effects: requiring that reserve_book is not called at all for an available book is a business rule, not a prescribed path. Testing only positive cases If you only test whether the agent reserves, sooner or later it reserves everything. Every “do X” task needs at least one “do not do X here” task. This balance keeps your agent from overreacting or underreacting. Shared state between trials If data from the previous trial is still lying around, you are measuring chance. Every trial starts clean, whether in memory like here or with a freshly provisioned test database. Evals that no longer show anything If a suite sits at 100 percent permanently, it no longer measures progress. For regressions that is the goal. For capability evals it means you need harder tasks. Cost and security Every run costs tokens. The math is simple: tasks times trials times turns per trial. With six tasks, five trials and three to four turns, you quickly end up with around a hundred model calls. Keep an eye on the “Tokens” column in the table, and feel free to use a smaller, cheaper model as the judge if it grades reliably. Check that first against a few cases you graded yourself. On security: evals run against a test environment, never against real systems or real customer data. Write tools in particular can be called hundreds of times in an eval. The API key belongs in secrets, never in the code. And because rate limits can hit, the harness limits concurrency to three simultaneous trials. My take To me, evals are to agents what unit tests are to regular code: the precondition for changing things with confidence. If you can evaluate AI agents, you can afford to experiment. If you run an agent without evals, at some point you no longer dare to switch the model or touch the prompt. That is the opposite of what we want to achieve with AI. At the same time, I do not think much of starting with a big framework and hundreds of tasks. Twenty tasks from real failures beat two hundred invented ones. And I would build code graders first. The model judge is handy, but it costs money, varies itself and has to be calibrated against human judgment. I use it where code really cannot go further. Then there is the human side. The best tasks rarely come from the development team alone. They come from support, from the business side, from the people who know what goes wrong in daily use. When you introduce evals, invite these people to contribute tasks. That improves the agent and builds trust in what it does. FAQ What is the difference between evals and unit tests? Unit tests check deterministic code and always return the same result for the same input. Evals check a system with a language model whose answers vary. That is why evals run several times and return rates instead of a simple green or red. How many tasks do I need to start? If you are evaluating AI agents for the first time, Anthropic recommends starting with roughly 20 to 50 simple tasks from real failures. You do not need hundreds of tasks at the beginning. What matters more is that every task is unambiguous and that two experts would independently reach the same verdict. Do I need an eval framework? Not to get started. A harness like the one in this article has a few hundred lines and shows you exactly what happens. A framework pays off once you need many suites, dashboards or comparisons between models. Can I use a language model as a judge? Yes, for soft criteria such as tone or completeness. Give the judge one clear criterion, allow the answer “unknown” and regularly compare its verdicts with human ratings. For hard facts such as “does the reservation exist”, code is more reliable. How much does an eval run cost? That depends on the model and the number of tasks, trials and turns. Multiply tasks by trials and average turns to get the number of model calls. The harness shows you the tokens per task after the first run. If you want to not only test your agent but also run it on AWS, AgentCore CLI 1.0: Deploy an AI Agent with TypeScript shows the way into the AgentCore Runtime. Your next step Take the last three cases where your agent got it wrong and turn them into tasks with a grader that checks the outcome. Run them five times and read the transcripts. That is the smallest sensible start for evaluating AI agents. I bet at least one of the three cases leads you to a better tool description. You will find more about AI in everyday development on the blog. Now I am curious: how do you check today whether your agent still does what it should after a change? Let me know in the comments. Sources - Anthropic Engineering: Demystifying evals for AI agents (Jan 9, 2026) - Anthropic Engineering: Writing effective tools for agents (Sep 11, 2025) - Claude documentation: Define tools - Anthropic TypeScript SDK on GitHub - npm: @anthropic-ai/bedrock-sdk - Chen et al. (2021): Evaluating Large Language Models Trained on Code (pass@k estimator) - Yao et al. (2024): τ-bench, A Benchmark for Tool-Agent-User Interaction (pass^k) - Node.js documentation: Test runner - GitHub Docs: Using secrets in GitHub Actions This post was created with AI assistance. --- 👉 Next step: New here? My start page collects the best articles, tools and learning paths in one place. Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de
How to never pay a generative API twice: a durable task table, not a retrying job queue Short answer: the moment you might have been charged is not the provider's response, it is the instant before the request leaves your process. Write that instant to a database row before the HTTP call, and a worker that dies mid-request becomes recoverable without guessing: nothing on the row means nobody paid and the task can go back to the queue; a saved request id means poll and never resubmit; the boundary written with no id means a third terminal state that no automation is allowed to retry. TL;DR - Fix: own task table with a lease, and a submit_started_at column written before the paid call. A queue retries by design; pg-boss says so in its own docs: "the job can run twice and handlers should be idempotent." - Alternative: a provider-side idempotency key, if your provider has one. Stripe does. The OpenAI and Anthropic clients ship maxRetries: 2 and send no idempotency header at all. - Trap: treating an expired deadline as a failure. A poll that ran out of patience is "I don't know", and the difference costs money every time you get it wrong. Why a job queue is the wrong unit of recovery for a paid call Because retrying is the entire point of a job queue, and for a paid POST a retry is a second charge. I started by running generation steps as ordinary jobs on the queue the rest of the system already used, pg-boss, because that is where every other background task lived. That lasted until I read the queue's own documentation on what happens when a worker looks dead: > A worker that only looked dead (a network partition, a stalled event loop) keeps running its handler after its job is failed this way, so the job can run twice and handlers should be idempotent. That is the pg-boss queue reference, and it is being honest rather than apologetic. Every durable queue has the same sentence somewhere, because a queue cannot tell a dead worker from a slow one. The contract it offers is at-least-once, and it tells you to make the handler idempotent. For sending an email, fine. For a handler whose first line spends money at an external provider, "make it idempotent" is the hard part being handed back to you. The defaults make it concrete. retryLimit is 2, so a failing job runs three times. expireInSeconds defaults to 15 minutes, after which an active job is retried or failed regardless of what the worker is doing. Image generation lands in tens of seconds and video generation in minutes, so a long clip can sail past the expiration while it is perfectly healthy, get failed by the monitor, and get retried into a second charge. You can raise the expiration to 24 hours, the documented maximum, but then a genuinely dead worker sits undetected for a day. There is a second problem that is not about money. A job that spends twenty minutes polling a remote id occupies a worker slot for twenty minutes. Concurrency on the queue is shared with everything else in the system, and a handful of slow generations will starve the short jobs that the queue exists for. The unit of recovery had to stop being a job and become a row. What RFC 9110 actually says about retrying a POST It says do not, unless you can prove one of two things, and the second one is the design brief for the whole task table. RFC 9110, section 9.2.2: > A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied. Two escape hatches. The first one, knowing the semantics are idempotent, is what a provider gives you with an idempotency key. Most generative APIs do not give you one, which I will get to. The second, "some means to detect that the original request was never applied," is not a property of the protocol at all. It is a property of what you wrote down locally before you made the call. The RFC also has a line about clients that skip both: > Some clients take a riskier approach and attempt to guess when an automatic retry is possible. For example, a client might automatically retry a POST request if the underlying transport connection closed before any part of a response is received, particularly if an idle persistent connection was used. That paragraph describes the official clients of the largest paid APIs in the world, which is the subject of the next section. Do the generative APIs give you an idempotency key? Mostly no, and the clients retry your POSTs anyway. I went looking for this expecting a boring answer and got a much sharper one by reading the SDK sources instead of the marketing pages. | API | Idempotency key on a POST | Official client retries a POST by default | |---|---|---| | Stripe | Idempotency-Key, first result stored and replayed | not examined | | OpenAI | none sent by openai-node, see below | yes, maxRetries: 2 on 408, 409, 429, ≥500, timeouts | | Anthropic | no occurrence of the word in anthropic-sdk-typescript | yes, maxRetries: 2 on 408, 409, 429, ≥500, timeouts | | Replicate | no occurrence in the published OpenAPI document | not examined | Stripe is the benchmark for how this is supposed to work, and its wording is worth quoting because it names the property that matters: > Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors. That is the Stripe API reference on idempotent requests. Note the retention rule in the same page: keys can be pruned once they are at least 24 hours old, and a key reused after pruning generates a new request. Even the gold standard has an expiry after which your safety net is gone, which is a good reminder that the client side still has to keep its own record. The OpenAI side is where it gets strange. The OpenAI Node client carries the full apparatus for idempotency keys. It declares the field, it has a key generator whose default value announces where the machinery came from, and it builds the header: protected idempotencyHeader?: string; protected defaultIdempotencyKey(): string { return `stainless-node-retry-${uuid4()}`; } // ...later, while assembling headers: let idempotencyHeaders: HeadersLike = {}; if (this.idempotencyHeader && method !== 'get') { if (!options.idempotencyKey) options.idempotencyKey = this.defaultIdempotencyKey(); idempotencyHeaders[this.idempotencyHeader] = options.idempotencyKey; } idempotencyHeader is declared and never assigned anywhere in the file. The condition is therefore always false and the header is never sent. RequestOptions even exposes idempotencyKey with the comment "A unique key for this request to enable idempotency", and setting it accomplishes nothing, because there is no header name to put it in. I spent a while convinced I had missed an assignment in a subclass. There is none. Meanwhile the retry predicate in the same file does not look at the method at all: private async shouldRetry(response: Response): Promise<boolean> { const shouldRetryHeader = response.headers.get('x-should-retry'); if (shouldRetryHeader === 'true') return true; if (shouldRetryHeader === 'false') return false; if (response.status === 408) return true; // request timeouts if (response.status === 409) return true; // lock timeouts if (response.status === 429) return true; // rate limits if (response.status >= 500) return true; // internal errors return false; } DEFAULT_TIMEOUT in that client is 600000, ten minutes, and a timeout goes down the same retry path. The Anthropic TypeScript client has a byte-for-byte equivalent predicate, the same maxRetries: 2 and the same ten-minute default, and zero occurrences of the string idempot in the entire file. The default behaviour of both clients, then: send a paid POST, and if the server returns 500 or the socket goes quiet for ten minutes, send it again, up to two more times, with no way for the server to recognise it as the same request. Whether the first attempt was billed is between you and the provider's meter. The client has no way to ask, and neither do you. This is not a scandal. It is the reasonable default for an API where most calls are cheap and a dropped connection is more likely than a silently successful one. It stops being reasonable the moment a single call costs real money and the user can see the balance. The boundary is before the request, not after the response The instant that matters is the last moment before the send, not the first moment after the answer. This is the one idea the whole design rests on, and it took me an embarrassing amount of staring at a state machine to see it. I had been drawing the boundary in the obvious place: the task is safe until the provider answers, and becomes committed when it answers. That is wrong, and the reason is that the answer is exactly the thing you do not have when the worker dies. A crash between "socket write completed" and "response parsed" leaves no evidence at all on the client side, and it is the most likely crash window there is, because that window is where all the waiting happens. Flip it. The task is safe until you begin the request, and becomes unknown the instant you do. That boundary is knowable in advance, so it can be written down in advance: async function runImageTask(task: Task, ports: Ports): Promise<void> { if (task.status === "SUBMITTED" && task.remoteRequestId) { return pollUntilDone(task, ports); // already paid for, never resubmit } await ports.repo.markSubmitStarted(task.id); // the boundary const { requestId } = await ports.provider.submit(task.input); await ports.repo.recordSubmitted(task.id, requestId); // now it is addressable return pollUntilDone(task, ports); } Three writes, and every crash window between them maps to exactly one recovery action. markSubmitStarted is a separate round trip to the database before a call that is about to take thirty seconds, so the cost of the extra write is not worth discussing. The one rule this imposes on the rest of the code is that nothing may sit between markSubmitStarted and the actual send. No input validation, no prompt assembly, no lazy credential fetch. Anything that can fail there produces a task parked in the unknown state for no reason, and parking a task costs a human's attention later. Build the request fully, then mark, then send. Three recovery branches after an expired lease A supervisor picking up a task whose lease has expired has exactly three cases, and the row tells it which one without any guessing: | What is on the row | What happened | Recovery | Money | |---|---|---|---| | status = RUNNING, no submit_started_at | Died before the request went out | Back to PENDING | Nothing spent, safe to redo | | status = SUBMITTED, remote_request_id set | Died after the provider acknowledged | Poll the saved id, never submit | Already spent, result is retrievable | | status = RUNNING, submit_started_at set, no id | Died inside the request | OUTCOME_UNKNOWN, stop | Might be spent, nobody can tell | In code it is short enough to fit in a screenshot, which is the point: type Recovery = | { action: "requeue" } | { action: "poll" } | { action: "park"; status: "OUTCOME_UNKNOWN" }; function recoveryFor(task: Task): Recovery { if (task.status === "SUBMITTED" && task.remoteRequestId) return { action: "poll" }; if (task.submitStartedAt === null) return { action: "requeue" }; return { action: "park", status: "OUTCOME_UNKNOWN" }; } Order matters in that function and it is not the order I wrote first. My first version checked submitStartedAt before the request id, which sends an already-submitted task back through the submit path whenever both columns are set, which is always for a healthy submitted task. The request id is the strongest evidence on the row, so it gets tested first. Why OUTCOME_UNKNOWN has to be terminal Because every automatic way out of it is a bet placed with someone else's money. This was the design decision I argued with myself about longest, since a state that requires a human is an operational cost forever. The two tempting exits both fail. Retrying assumes the request never landed, and if it did land you have paid twice and the user sees one result. Failing assumes the request never landed too, just with a nicer label: the task disappears from the pipeline, the charge does not disappear from the invoice, and the reconciliation happens weeks later when someone asks why the numbers do not match. Both are the same guess wearing different clothes. The state is therefore terminal, the UI says the outcome could not be determined and suggests checking the provider's dashboard, and a human presses the button. In exchange, the system never spends money on its own initiative. For a paid API that is the correct side of the error to be on, and the volume is low enough that it stays correct: the branch only fires when a process dies inside the send window, which is a few seconds out of a generation that takes minutes. The same logic applies to deadlines, and this is where I got it wrong first. Put a cap on polling, because a remote id that never resolves cannot hold a slot forever. My original code marked the task FAILED when the cap was reached. That is the same lie as before: the provider might still be working, or might have finished and billed while my poll loop gave up. An exhausted deadline is not a failure, it is an unknown outcome that happens to be slow. It goes to the same terminal state, with a different reason string. Claiming work with FOR UPDATE SKIP LOCKED The claim is a single statement, and PostgreSQL documents this exact use case. From the SELECT reference: > With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped. Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table. "Inconsistent view of the data" is the point, not a caveat. Two supervisors claiming work must not see the same row, and neither needs an accurate count of what is pending. update generation_task t set lease_owner = $1, lease_expires_at = now() + $2::interval, status = case when t.status = 'PENDING' then 'RUNNING' else t.status end where t.id in ( select id from generation_task where kind = $3 and (status = 'PENDING' or (status in ('RUNNING', 'SUBMITTED') and lease_expires_at < now())) order by created_at limit $4 for update skip locked ) returning t.*; The case matters, and this is the part I got wrong. An earlier version of this statement set status = 'RUNNING' unconditionally, which erases the one signal that tells the executor to poll instead of submit. I rebuilt both versions on a scratch PostgreSQL 17.11 database to be sure I was not misremembering, and the buggy claim does exactly what you would fear: status | remote_request_id ---------+------------------- RUNNING | req-abc The remote_request_id survives, the status does not. A recovered task that had already been paid for comes back looking like a fresh one. The guard at the top of runImageTask still catches it, because that guard checks the id as well, but at that point the query had quietly become the only thing standing between a restart and a double charge, and I would rather two things stood there. On the same scratch database, SKIP LOCKED behaves as advertised: with three eligible rows and one of them held by a concurrent SELECT ... FOR UPDATE in another session, the claim returned 2 rows immediately instead of blocking on the third. That is the whole reason the clause exists, and it is worth confirming with your own eyes once, because the alternative failure mode, a supervisor tick that silently waits on a lock, looks identical to a supervisor tick that has nothing to do. Concurrency is per kind, not global, because the kinds have nothing in common. Take four concurrent image slots, three for clips, two for speech and exactly one for the final local render: the remote ones are limited by the provider's rate limits and the local one is limited by CPU. Those particular numbers are arbitrary and yours will differ. The structure is not arbitrary: one global limit for all four would either starve the cheap fast work or oversubscribe the machine. Heartbeat, because not every step polls A lease needs its own renewal, separate from whatever the work is doing. I learned this from a step that had no polling at all. Remote generation renews the lease for free as a side effect: the poll loop runs every few seconds, and touching the row at the same time costs nothing. Local work has no such loop. The final render is one long call into ffmpeg that returns when it returns, and speech synthesis for a short passage is a single request that either answers or does not. Both of them held a lease that nobody was extending, so the supervisor watched the lease expire on a task that was working perfectly, decided the worker was dead, and started recovery on a row that was still being written to. The fix is a timer independent of the work: renew at one third of the lease duration, so two consecutive misses still leave a margin before expiry. The ratio is what matters, not the absolute value. pg-boss reaches the same conclusion from the other direction and documents it as a separate mechanism from expiration, with a floor of 10 seconds on heartbeatSeconds and a monitor that only notices staleness on its own interval, 60 seconds by default. Detection time is the sum of the two, which is worth internalising before you tune the lease down to something clever. Pinning the order with a test The order of those three writes is load-bearing and invisible, so it gets asserted directly rather than implied by behaviour. The executor takes its ports as an interface, the test passes recording fakes, and the assertion is on the call log: expect(calls).toEqual([ "markSubmitStarted", "submitImage", "recordSubmitted", "poll", "complete", ]); Then the recovery case, which is the one that actually costs money if it regresses: // a task restored in SUBMITTED state with a saved request id expect(calls).not.toContain("submitImage"); expect(calls).toEqual(["poll", "complete"]); A test that asserts on a sequence of call names is usually a smell, because it pins implementation detail rather than behaviour. Here the sequence is the behaviour. Any refactor that moves markSubmitStarted after submitImage still passes every functional test, produces identical output on the happy path, and reintroduces the exact bug the design exists to prevent. The only way to catch it is to say out loud that the write comes first. The price in the docs is not the price on the invoice Store two numbers per task, estimated and actual, because the first one will be wrong. I built the cost display off a model catalogue populated from provider documentation, showed users an estimate before they pressed the button, and felt good about it. The first real run billed noticeably more than the catalogue said. Take a step documented at four cents and invoiced at six: that is half again as much, multiplied by every step in a job. An estimate that far off is worse than no estimate, because people plan around it. The catalogue was not lying, it was quoting a headline number that did not include what the actual call resolved to. The correction is structural rather than a better catalogue: cost_estimated comes from the provider's quote endpoint at submit time, cost_actual comes from the billing figure that arrives with the result, and the user-facing total is the sum of whichever is available per task. When the two disagree the actual value wins and the catalogue entry gets fixed. Documentation prices are for capacity planning. Billing figures are for arithmetic. Bottom line If a background task spends money at an external provider, the queue's retry is no longer a reliability feature and the row, not the job, has to be the unit of recovery. Write the "I am about to spend money" boundary to the database before the request goes out, save the provider's request id the moment it exists, and accept a third terminal state for the narrow window in between, where neither retrying nor failing is honest. The RFC has been describing this for years under "some means to detect that the original request was never applied", and the means is a column. Primary sources worth reading in full: RFC 9110 §9.2.2 on idempotent methods, the PostgreSQL SELECT reference on SKIP LOCKED, the pg-boss queue documentation on heartbeats versus expiration, Stripe on idempotent requests, and the retry predicates in openai-node and anthropic-sdk-typescript, which are worth twenty minutes of anyone's time who is sending paid POSTs through them. --- Originally published at bitpage.me — BitPage, a technical blog on backend, databases, infrastructure and incident post-mortems.
State of AI Instruction Quality: Coding Agent Comparison Claude Code, Codex, Copilot, Cursor and Gemini are the five coding agents in the Q3 report. We compared them across the 10,000 most-starred repos that ship an instruction file, and put the companies that make them through the same check, using their own repos. Claude Code is everywhere. It's set up in 69 of the 100 most-starred repos and in 6 of the top 10. Copilot is in 26 of the top 100, Gemini in 11, Cursor in 7 and Codex in 3. The file Claude Code reads is losing anyway. One CLAUDE.md in five is a line or two telling the agent to go read AGENTS.md. Which coding agents the top repos use | Agent | Top 10 | Top 100 | Top 1,000 | All 10,000 | |---|--:|--:|--:|--:| | Claude Code | 6 | 69 | 646 | 6,030 | | AGENTS.md (shared) | 6 | 72 | 678 | 6,125 | | Copilot | 2 | 26 | 225 | 1,756 | | Gemini | 2 | 11 | 51 | 430 | | Cursor | 1 | 7 | 96 | 734 | | Codex | 0 | 3 | 32 | 162 | !Share of repos that set up each agent, in the 100 most-starred repos against the 9,784 of the 10,000 with a setup: Claude Code 69% against 61.6%, AGENTS.md 72% against 62.6%, Copilot 26% against 17.9%, Gemini 11% against 4.4%, Cursor 7% against 7.5%, Codex 3% against 1.7%. The most-starred repo in the set, freeCodeCamp, uses Copilot. React uses Claude Code. Gemini punches above its size at the top: 430 of the 10,000 set it up, and 11 of the top 100 do. Many repos don't pick one. 4,213 of the 9,784 with a setup (43.1%) use two or more coding agents, and the most common pairing is Claude Code plus AGENTS.md, in 2,399 repos. AGENTS.md is winning 998 of the 5,464 CLAUDE.md files in the 10,000 (18.3%) are 25 words or fewer and point at AGENTS.md. Going the other way, 39 of 6,128 AGENTS.md files point at CLAUDE.md. That's 26 to 1. 11.3% of GEMINI.md files point to AGENTS.md too, and 4.0% of Copilot's instructions files do. The CLAUDE.md files people do write are the least instruction-like main files of any agent. About a quarter of what's in them tells the agent what to do. The rest describes the project. !Median share of items in each agent's main file that are directives or constraints: CLAUDE.md 26.1%, GEMINI.md 35.2%, AGENTS.md 36.4%, Codex's AGENTS.md 38.1%, copilot-instructions.md 38.9%, .cursorrules 46.0%. Pointer files left out. Claude Code setups hedge less. The "should" and "try to" kind runs at a median 3.3 per 100 instructions, against 4.7 for the other agents. The instructions moved into skills While the debate is still ongoing between AGENTS.md and other provider-specific files, most of the instructions moved to a different location. Skills hold 42.3% of every directive and constraint in the set. All the main files together hold 31.3%. !Share of all directives and constraints in the 10,000: skills 42.3% (13,190 SKILL.md files plus their reference files), main files 31.3% (13,375 CLAUDE.md, AGENTS.md and other main files), everything else 26.4% (rules, sub-agents, commands). Peculiar oddity, but only a quarter of repos (2,348) have any skills, and 249 repos hold half of all skill instructions between them. There are almost as many SKILL.md files as main files: 13,190 against 13,375. They spread like packages, without a package manager. 977 SKILL.md files are byte-for-byte copies of one in another repo. The most-copied, frontend-design, sits unchanged in 14 repos. A fix upstream reaches none of the copies. Two names, one file, and copies drifting apart 3,562 repos carry main files for two or more agents. A third of them (1,246) keep the same text under two names, mostly a CLAUDE.md identical to the AGENTS.md beside it. Another third use a pointer. !How the 3,562 repos with main files for two or more agents relate the second file to the first: the same text under two names 1,246 repos (35.0%), a pointer to the other file 1,158 (32.5%), different files 937 (26.3%), copies drifting apart 221 (6.2%). 221 repos keep two copies that have already drifted. The typical pair shares 32 lines, and each side has two or three lines the other lacks. In 92 of 430 Gemini setups (21.4%), GEMINI.md and AGENTS.md hold the same text, and Gemini reads both. For the other agents, the same shape turns up in 1.3% of setups. Gemini setups are also the heaviest at startup: 8.1% hand the agent more than 5,000 words before the first prompt, against 2.3% across the field. Cursor goes the other way. 66.3% of Cursor setups carry rule files, and a third (33.5%) have no main file at all. The other agents' setups almost always have one. The makers, by their own repos | Company | Agent | Repos in the 10,000 | Median score | Share of repos with custom sub-agents | |---|---|--:|--:|--:| | Anthropic | Claude Code | 11 | 7.0 | 27% | | OpenAI | Codex | 13 | 6.8 | 0% | | Google | Gemini | 28 | 7.2 | 4% | | GitHub | Copilot | 11 | 7.9 | 45% | | Cursor | Cursor | 2 | 8.6 | 0% | | All repos | | 10,000 | 7.5 | 5.7% | The scores cluster around the field's. What each company builds is where they split. !Which agents each company sets up in its own repos: Anthropic, 11 repos: Claude Code 11, AGENTS.md 1, Copilot 0, Gemini 0, Cursor 0. OpenAI, 13 repos: Claude Code 2, AGENTS.md 12, Copilot 1, Gemini 0, Cursor 0. Google with its Gemini orgs, 28 repos: Claude Code 5, AGENTS.md 15, Copilot 1, Gemini 18, Cursor 1. GitHub, 11 repos: Claude Code 1, AGENTS.md 5, Copilot 10, Gemini 0, Cursor 0. Cursor, 2 repos: Claude Code 0, AGENTS.md 1, Copilot 0, Gemini 0, Cursor 2. GitHub builds its own agents. 5 of its 11 repos ship custom sub-agents, eight times the field's rate of 5.7%. One of them is spreading: an agentic-workflows sub-agent for GitHub's own workflow tool sits in 44 repos across the 10,000, the second most common sub-agent role after code-reviewer. The agents' own repos shout too. A third of the 10,000 write rules in capitals. Anthropic's prompting guide tells users of its newest models to dial back that kind of language: "Where you might have said 'CRITICAL: You MUST use this tool when...', you can use more normal prompting like 'Use this tool when...'." Claude Code's own repo ships a triage command that says "IMPORTANT: Don't post any comments or messages to the issue." Codex's own AGENTS.md says "Features that change the agent logic MUST add an integration test", and a Gemini CLI command tells the agent "you MUST first consult the getting started documentation". OpenAI's repos leave the agent guessing. Of every writing habit we compared across the makers, only one difference held up statistically. OpenAI's median repo leaves 48.7 of every 100 instructions without a file, command or tool to act on, against 30.8 for the field. One of its SDKs opens a section with: > Solve the narrow problem with the simplest implementation. Avoid unrelated refactors, reformatting, dependency or lockfile churn, generated-file edits, speculative abstractions, compatibility shims, and capabilities not requested. A senior engineer would agree with every word. An agent gets two judgment calls, "the narrow problem" and "the simplest implementation", and nothing to check its work against. Cursor's two repos score highest, 8.6, with the fewest instructions left vague. One still carries the hedged kind, a skill that says "subtask_prompt should read like a standalone request". Who checks the makers These companies write the agents, the documentation and the prompting guides. Their engineers know their codebases far better than any tool does, and that's how the gaps get in. To the person who wrote it, "the narrow problem" has an obvious answer. The agent reading the file has the file, and nothing else. The agent can't flag the gap either. It does what the line says, and when the line names nothing, it guesses. A company reviewing its own files reads them with the same knowledge that left the gap there. ails check reads every instruction file with the same deterministic rules, whoever wrote it. The 10,000 most-starred repos and the companies that build the agents get the same treatment, and the findings above are what came back. !Lines 26 to 41 of the AGENTS.md in OpenAI's openai-node repo, the section "Keep changes small and coherent", with ails check's Specificity Gap finding beside seven of its lines: 28, 31, 32, 34, 36, 37 and 38. Line 28 reads "Solve the narrow problem with the simplest implementation." Check one yourself Every repo in this post is public. Clone one and run the same check: npx @reporails/cli check It lists each finding by file, so you can read the instructions and decide whether we counted fairly. This is the last post from the Q3 report. The data behind all of them is in the 2026-Q3 folder of reporails/reports. --- From The State of AI Instruction Quality: Q3 2026: 10,000 of the most-starred GitHub repos that ship an instruction file, September 2026 snapshot; 9,784 have a setup we could map. "Top 100" and "top 1,000" rank the 10,000 by stars at the snapshot. A setup is one repo and one agent; repos often have several, so agent counts add up to more than the repos. Codex counts repos with a Codex marker in .codex/ (a config.toml, rules/, agents/ or hooks.json) or an AGENTS.override.md; a plain AGENTS.md setup sits in the shared row. A pointer is a main file of 25 words or fewer that names or imports another agent's main file. Items are what the instrument maps from a file; directives and constraints are the items that tell the agent what to do. "Skills" include the reference files beside SKILL.md; setups above the batch's 90-file cap are outside the run, so the skills share is a floor. "The same text" cannot tell a symlink from a copy; only the drifted pairs are certainly two files. "Gemini reads both" is the instrument's model of Gemini, which concatenates AGENTS.md and GEMINI.md. Companies are the GitHub organizations that ship each agent: anthropics, openai, google with google-gemini and gemini-cli-extensions, github and cursor; Cursor's two repos carry no test. Group differences are two-sided Mann-Whitney or Fisher's exact tests, read against a Bonferroni threshold for all 209 comparisons in the run; the top-100 shares are counts. Scores are each company's median repo score on ails check's 0 to 10 scale; for Anthropic, OpenAI, Google and GitHub the 95% bootstrap interval includes the field's 7.5. Vague instructions name no file, command or tool (CORE:C:0042), per 100 directives and constraints, in repos with at least 5. Repos measured. anthropics: anthropic-sdk-typescript, buffa, claude-agent-sdk-python, claude-code, claude-code-action, claude-code-base-action, claude-for-legal, commerce-agents, defending-code-reference-harness, financial-services, launch-your-agent. openai: apps-sdk-ui, codex, codex-security, harmony, openai-agents-js, openai-cli, openai-cookbook, openai-dotnet, openai-go, openai-java, openai-node, openai-python, realtime-voice-component. google: adk-go, adk-java, adk-python, benchmark, clusterfuzz, comprehensive-rust, docsy, dotprompt, filament, go-github, gvisor, heir, mantis, nomulus, oss-fuzz, osv-scanner, osv.dev, perfetto, play-services-plugins, sam, site-kit-wp, syzkaller, xls. google-gemini: gemini-cli. gemini-cli-extensions: code-review, nanobanana, security, workspace. github: branch-deploy, codeql, codeql-action, copilot-cli-for-beginners, copilot-sdk, docs, gh-aw, gh-skyline, gh-stack, github-mcp-server, spec-kit. cursor: community-plugins, cookbook. --- I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the instruction files your coding agent loads, from CLAUDE.md to Copilot and Cursor rules, measures which instructions couple to behavior, and points to the lines worth tightening.
Streamed LLM Responses Hide Their Token Count. Where Usage Lives in 3 TypeScript APIs - Book: AI That Answers - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- You ship the streaming chat box. Text appears word by word, the UI feels fast, and the cost log you wired up last sprint shows a flat line of zeros. Nothing is broken on the network. The tokens were billed. Your code just looked for them in the place a non-streamed response keeps them, and a streamed response keeps them somewhere else. Each API hides them in a different spot. OpenAI's Chat Completions sends usage only in one extra chunk at the end, and only if you ask for it. The Responses API attaches it to the response.completed event. Anthropic's Messages API splits it across two events, input on message_start and a running output count on message_delta. This tutorial writes one small function per API. Each one streams text to a callback and returns the same typed object: input tokens, cached input tokens, output tokens, cost in dollars, and a flag that says whether the numbers are final. Every field name below was checked against the current source of the openai and @anthropic-ai/sdk packages. Setup and the shared type You need Node 22 or later (the openai README lists 22 as the minimum), both SDKs, and tsx to run TypeScript without a build step: npm install openai @anthropic-ai/sdk npm install -D tsx export OPENAI_API_KEY=... export ANTHROPIC_API_KEY=... All three functions return this shape. Put it in usage.ts: export type Price = { inputPerMTok: number; cachedInputPerMTok: number; outputPerMTok: number; }; export type StreamUsage = { inputTokens: number; // all input, cached included cachedInputTokens: number; // subset of inputTokens outputTokens: number; costUsd: number; final: boolean; // came from the closing usage event }; export type OnText = (text: string) => void; export function priceUsage( u: Omit<StreamUsage, "costUsd">, p: Price, ): StreamUsage { const fresh = u.inputTokens - u.cachedInputTokens; const usd = fresh * p.inputPerMTok + u.cachedInputTokens * p.cachedInputPerMTok + u.outputTokens * p.outputPerMTok; return { ...u, costUsd: usd / 1_000_000 }; } The final flag is the part people skip. A streamed call can end three ways: the provider sends its closing usage event, the user closes the tab, or the network drops. Only the first gives you a billing-grade number. The flag lets your logger tell "this call cost $0" apart from "this call ended before anyone told us what it cost". OpenAI Chat Completions: the opt-in last chunk Chat Completions does not send usage on a stream unless you set stream_options: { include_usage: true }. The SDK's own doc comment spells out what happens then: one more chunk arrives before data: [DONE], its usage field covers the whole request, its choices array is always empty, and every other chunk carries usage: null (source). That empty choices array matters. Code that does chunk.choices[0].delta.content without optional chaining throws on the very chunk you wanted. // openai-chat.ts import OpenAI from "openai"; import { priceUsage, type OnText } from "./usage"; const MODEL = "gpt-5.4-mini"; const PRICE = { inputPerMTok: 0.75, cachedInputPerMTok: 0.075, outputPerMTok: 4.5, }; export async function streamChat( prompt: string, onText: OnText, signal?: AbortSignal, ) { const client = new OpenAI(); let input = 0, cached = 0, output = 0; let final = false; try { const stream = await client.chat.completions.create( { model: MODEL, messages: [{ role: "user", content: prompt }], stream: true, stream_options: { include_usage: true }, }, { signal }, ); for await (const chunk of stream) { const text = chunk.choices[0]?.delta?.content; if (text) onText(text); if (chunk.usage) { const u = chunk.usage; input = u.prompt_tokens; cached = u.prompt_tokens_details?.cached_tokens ?? 0; output = u.completion_tokens; final = true; } } } catch (err) { if (!signal?.aborted) throw err; } return priceUsage( { inputTokens: input, cachedInputTokens: cached, outputTokens: output, final, }, PRICE, ); } The prices are the standard rates for gpt-5.4-mini on OpenAI's pricing page as of 2026-10-17: $0.75 per million input tokens, $0.075 cached, $4.50 output. Copy yours from the same page; they change. OpenAI counts cached tokens inside prompt_tokens; its prompt caching guide treats ordinary input as input minus cached minus cache writes, which is why priceUsage subtracts the cached part before applying the full rate. !Three pipes carry streamed text toward a user, each with a small token meter mounted at a different point: one at the far end, one on a closing valve, one split between the start and the middle Same idea, three places: where each API puts the token count on a stream. The Responses API: read response.completed The newer Responses API streams typed events instead of chunks. Text arrives as response.output_text.delta, and the run ends with response.completed, which carries the full Response object (OpenAI streaming guide). Usage sits on event.response.usage, with input_tokens, output_tokens and a required input_tokens_details.cached_tokens in the SDK types (source). No opt-in flag here. The catch is a second terminal event: response.incomplete fires when a response finishes early, for example after hitting max_output_tokens (streaming events reference), and it carries the same response object. If you only listen for completed, every truncated answer logs zero. // openai-responses.ts import OpenAI from "openai"; import { priceUsage, type OnText } from "./usage"; const MODEL = "gpt-5.4-mini"; const PRICE = { inputPerMTok: 0.75, cachedInputPerMTok: 0.075, outputPerMTok: 4.5, }; export async function streamResponses( prompt: string, onText: OnText, signal?: AbortSignal, ) { const client = new OpenAI(); let input = 0, cached = 0, output = 0; let final = false; try { const stream = await client.responses.create( { model: MODEL, input: prompt, stream: true }, { signal }, ); for await (const event of stream) { if (event.type === "response.output_text.delta") { onText(event.delta); } else if ( event.type === "response.completed" || event.type === "response.incomplete" ) { const u = event.response.usage; if (u) { input = u.input_tokens; cached = u.input_tokens_details.cached_tokens; output = u.output_tokens; final = true; } } } } catch (err) { if (!signal?.aborted) throw err; } return priceUsage( { inputTokens: input, cachedInputTokens: cached, outputTokens: output, final, }, PRICE, ); } usage is optional on the Response type, hence the if (u) guard. TypeScript narrows event inside each branch, so event.delta and event.response type-check without casts. Anthropic: input first, output last Anthropic's stream opens with message_start, whose message.usage holds the input count. Output arrives later on message_delta, and the streaming docs put it in a warning box: the token counts in message_delta usage are cumulative. Overwrite, never add. The docs' own example shows why the first event is a trap. Its message_start reports "input_tokens": 25, "output_tokens": 1, and the closing message_delta reports "output_tokens": 15. Log at the start and you record one output token for a fifteen-token reply. In the SDK's MessageDeltaUsage type, input_tokens and both cache fields are number | null and described as cumulative (source). With server tools they move: the docs' web search example starts at 2,679 input tokens on message_start and ends at 10,682 on message_delta. Take the delta's value when it is not null. // anthropic.ts import Anthropic from "@anthropic-ai/sdk"; import { priceUsage, type OnText } from "./usage"; const MODEL = "claude-sonnet-5-5"; const PRICE = { inputPerMTok: 2, cachedInputPerMTok: 0.1, outputPerMTok: 10, }; export async function streamClaude( prompt: string, onText: OnText, signal?: AbortSignal, ) { const client = new Anthropic(); let input = 0, read = 0, write = 0, output = 0; let final = false; try { const stream = await client.messages.create( { model: MODEL, max_tokens: 1024, messages: [{ role: "user", content: prompt }], stream: true, }, { signal }, ); for await (const event of stream) { if (event.type === "message_start") { const u = event.message.usage; input = u.input_tokens; read = u.cache_read_input_tokens ?? 0; write = u.cache_creation_input_tokens ?? 0; output = u.output_tokens; } else if (event.type === "content_block_delta") { if (event.delta.type === "text_delta") { onText(event.delta.text); } } else if (event.type === "message_delta") { const u = event.usage; output = u.output_tokens; input = u.input_tokens ?? input; read = u.cache_read_input_tokens ?? read; write = u.cache_creation_input_tokens ?? write; } else if (event.type === "message_stop") { final = true; } } } catch (err) { if (!signal?.aborted) throw err; } return priceUsage( { inputTokens: input + read + write, cachedInputTokens: read, outputTokens: output, final, }, PRICE, ); } Note the sum at the bottom. Anthropic's input_tokens is not the total: per the prompt caching docs, total input is cache_read_input_tokens + cache_creation_input_tokens + input_tokens. OpenAI includes cached tokens in its input count and Anthropic excludes them, so one normalised type saves you from comparing two different numbers on the same dashboard. The prices are Claude Sonnet 5.5's rates from Anthropic's pricing page as of 2026-10-17: $2 per million input, $0.10 for cache hits, $10 output. Cache writes cost more than base input there ($2.50 per million for the 5-minute tier), and OpenAI now reports cache_write_tokens too. priceUsage bills writes at the base rate, so if you use explicit caching, add a fourth rate. If you prefer the client.messages.stream() helper, await stream.finalMessage() gives you the accumulated message with its usage. The SDK helpers doc says that promise throws if no message exists. It doesn't say what happens on abort, so don't count on finalMessage() for billing after a cancel. The abort case: when the last event never comes Users close tabs. Your server cancels a request when the client disconnects. Both SDKs accept an AbortSignal in the request options (openai README, Anthropic RequestOptions), and aborting makes the for await loop throw. The OpenAI doc comment is blunt about what you lose: "If the stream is interrupted or cancelled, you may not receive the final usage chunk…" On Anthropic you keep the input count from message_start and the last cumulative output count you saw. The real bill is at least that. That is why each function catches only when signal.aborted is true and returns final: false instead of throwing. A real network error still throws. Here is a runner, demo.ts, that can cut a stream short on purpose: // demo.ts import { streamChat } from "./openai-chat"; import { streamResponses } from "./openai-responses"; import { streamClaude } from "./anthropic"; const runners = { chat: streamChat, responses: streamResponses, claude: streamClaude, }; async function main() { type Name = keyof typeof runners; const name = (process.argv[2] ?? "chat") as Name; const abortAfterMs = Number(process.argv[3] ?? 0); const ac = new AbortController(); if (abortAfterMs > 0) setTimeout(() => ac.abort(), abortAfterMs); const usage = await runners[name]( "Explain backpressure in two short paragraphs.", (t) => process.stdout.write(t), ac.signal, ); console.log("\n", usage); } main().catch((err) => { console.error(err); process.exit(1); }); Run npx tsx demo.ts claude for a full call, then npx tsx demo.ts claude 400 to abort after 400 ms. The second run should print final: false. I haven't published output here because your token counts depend on the model version and the prompt; run it and read your own. !A stream of text being cut by scissors partway along, with the token meter at the far end left blank An aborted stream never reaches the event that carries the bill. What to log, and what not to Log every call, including the ones with final: false. Dropping them hides real spend; recording them as zero lies about it. Store the flag next to the numbers and let your cost dashboard sum only final rows, with a separate count of non-final calls so a spike in aborts is visible. And normalise input to "total, cached included" before any row reaches that sum, because the two vendors disagree on what input_tokens means. --- If this was useful Token cost is the first thing that surprises people once an LLM feature leaves the prototype stage, and streaming is where it hides. AI That Answers is the first book in my AI in TypeScript series: your first LLM app in TypeScript, prompts, structured output, and keeping track of what each call costs, written for developers who already ship Node code.  The rest of the series takes the same app through RAG, tool calling, agents and production: all five books here.