hermes-ide

同时开几个 agent，哪个等你批、哪个跑完了？ Hermes 把 Claude Code、Codex、Copilot 收进一个控制台，每任务独立分支和 worktree，等你的进收件箱。 适合多 agent 并行干活、切窗口切到晕的人。 你同时跑过最多几个 agent？ #AI编程 #Agent工具 @grok hermes-hq/hermes-ide

Build a BPE Tokenizer in 80 Lines of TypeScript and Watch JSON Eat Your Tokens - Book: AI That Answers - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- The maintainers of the TOON format ran three flat datasets through OpenAI's o200k_base encoding and counted 164,643 tokens for the pretty-printed JSON and 104,908 for the same data minified. Nothing was dropped. The minified version holds every key and every value. It stops paying for indentation and line breaks, and that alone takes about 36% off the count. Tokens are the unit your LLM bill is written in. Anthropic's pricing table, for one, quotes input and output separately, per million tokens. Yet most developers who pay that bill have never watched a string turn into tokens. You can build the machine that does it in 80 lines of TypeScript and train it on your own JSON. What happens to your prompt before the model sees it OpenAI's tokenizers use byte-pair encoding. BPE began as a compression algorithm, and Sennrich, Haddow and Birch adapted it to split words into subword units for neural machine translation (ACL 2016). The tiktoken README says the encoding is reversible and lossless and works on any text, even text the tokenizer never saw in training. On average, a token covers about 4 bytes. Encoding runs in three steps. 1. A regex cuts the text into chunks, so letters, digits, punctuation and whitespace end up in separate runs. Karpathy's minbpe credits this to the GPT-2 paper and notes it "ensures that no merges will happen across category boundaries." 2. Each chunk becomes its UTF-8 bytes. The starting vocabulary is the 256 possible byte values, so nothing is ever out of vocabulary. 3. Learned merges glue adjacent byte pairs into bigger tokens, in the order they were learned. Training is where the merges come from. Count every adjacent pair across the corpus, take the pair with the highest count, give it the next free id (256, then 257, and so on), replace it everywhere, repeat. The minbpe README walks through aaabdaaabac: three merges (aa, then ab, then the pair of those two new tokens) turn 11 bytes into five tokens, [258, 100, 258, 97, 99]. !Rows of small square byte tiles fusing pairwise into larger tiles, layer by layer Byte-pair encoding: start from single bytes, then fuse the most frequent neighbours, one merge at a time. The tokenizer in 80 lines This is the whole trainer and encoder. It uses GPT-2's pre-split pattern, minus the contraction rules ('s, 'll, and friends), copied from minbpe's regex.py. // bpe.ts: a toy byte-level BPE trainer and encoder type Pair = [number, number]; // GPT-2's pre-split pattern, minus its contraction rules const SPLIT = / ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+/gu; const utf8 = new TextEncoder(); const text8 = new TextDecoder(); const key = (a: number, b: number) => `${a},${b}`; const show = (b: Uint8Array) => JSON.stringify(text8.decode(b)); const chunksOf = (text: string) => (text.match(SPLIT) ?? []).map((c) => Array.from(utf8.encode(c))); function countPairs(ids: number[], counts: Map<string, number>) { for (let i = 0; i < ids.length - 1; i++) { const k = key(ids[i], ids[i + 1]); counts.set(k, (counts.get(k) ?? 0) + 1); } } function merge(ids: number[], [a, b]: Pair, id: number) { const out: number[] = []; for (let i = 0; i < ids.length; i++) { if (ids[i] === a && ids[i + 1] === b) { out.push(id); i++; // the pair's second half is consumed too } else out.push(ids[i]); } return out; } export function train(text: string, numMerges: number) { let chunks = chunksOf(text); const merges = new Map<string, number>(); const vocab = new Map<number, Uint8Array>(); for (let b = 0; b < 256; b++) vocab.set(b, Uint8Array.of(b)); for (let m = 0; m < numMerges; m++) { const counts = new Map<string, number>(); for (const ids of chunks) countPairs(ids, counts); let best = ""; let bestCount = 1; // a pair seen once earns no token for (const [k, c] of counts) { if (c > bestCount) { best = k; bestCount = c; } } if (!best) break; const [a, b] = best.split(",").map(Number) as Pair; const id = 256 + m; chunks = chunks.map((ids) => merge(ids, [a, b], id)); merges.set(best, id); const l = vocab.get(a)!, r = vocab.get(b)!; const bytes = new Uint8Array([...l, ...r]); vocab.set(id, bytes); console.log(`#${m}`, show(l), "+", show(r), "->", show(bytes), `${bestCount}x`); } return { merges, vocab }; } export function encode(text: string, merges: Map<string, number>) { const out: number[] = []; for (let ids of chunksOf(text)) { while (ids.length > 1) { // earliest-learned merge first, the order training used let pair: Pair | null = null; let rank = Infinity; for (let i = 0; i < ids.length - 1; i++) { const r = merges.get(key(ids[i], ids[i + 1])); if (r !== undefined && r < rank) { rank = r; pair = [ids[i], ids[i + 1]]; } } if (!pair) break; ids = merge(ids, pair, rank); } out.push(...ids); } return out; } Pairs are keyed as "a,b" strings because a Map compares array keys by reference, so [1, 2] would never find [1, 2]. Ties go to the pair seen first, since Map iterates in insertion order and the comparison is strict. Training works chunk by chunk, so a merge can never span a letter and a digit, or a word and the punctuation after it, which matches the real thing. And encode always applies the lowest-ranked merge available, the same rule minbpe's _encode_chunk uses: replay the merges in the order training learned them. The show helper decodes a token's bytes for printing. When a token holds only part of a multi-byte character, TextDecoder prints �, because the token boundary falls inside the character. The Japanese sample further down does the same. Train it on JSON and read the merges Feed it the kind of text your app actually sends: an array of records, pretty-printed. // demo.ts import { train, encode } from "./bpe"; const records = Array.from({ length: 40 }, (_, i) => ({ customer_id: 1000 + i, status: i % 4 ? "active" : "suspended", country: ["DE", "BR", "JP", "US"][i % 4], })); const pretty = JSON.stringify(records, null, 2); const { merges } = train(pretty, 40); const one = records[0]; const asPretty = encode(JSON.stringify(one, null, 2), merges); const asMin = encode(JSON.stringify(one), merges); console.log({ pretty: asPretty.length, minified: asMin.length }); Run it with tsx: npx tsx demo.ts. Each log line is one merge, with the two parts, the new token, and how often the pair appeared. Read all 40 and sort them in your head: how many are fragments of the keys customer_id, status and country, how many are values and digits, and how many are quotes, colons and runs of spaces? On a corpus made of nothing but JSON, every merge has to come from one of those. The keys repeat in every single record, and that repetition is what earns them merges. A production tokenizer learns its merges from a huge general corpus instead, so common English words have a good chance of getting a token of their own (the next script shows you whether customer did). Your acctBalanceAdjFlagV2 pays for every piece. Then try the experiments the script makes cheap. Train on JSON.stringify(records) instead of the pretty version and compare the merge log. Rename customer_id to cid and see how many merges it frees up. Compare the two counts in the last line, and remember that this toy only knows 40 merges, so the ratio between them tells you more than either number. Ask a real encoder: o200k_base Your toy has 40 merges. o200k_base has a vocabulary of about 200,000 tokens, roughly double GPT-4's ~100,000, according to Simon Willison's notes from the GPT-4o launch. tiktoken's model table maps gpt-4o, gpt-4.1, o3, o4-mini and gpt-5 to it. js-tiktoken loads it in Node; the lite import pulls in only the one rank file you need. // compare.ts import { Tiktoken } from "js-tiktoken/lite"; import o200k_base from "js-tiktoken/ranks/o200k_base"; const enc = new Tiktoken(o200k_base); const pieces = (s: string) => enc.encode(s).map((id) => enc.decode([id])); const samples = [ '{\n "customer_id": 1000\n}', '{"customer_id":1000}', "customerShippingAddressLine2", "1730505600000", "Your order has been cancelled.", "ご注文はキャンセルされました。", ]; for (const s of samples) { const p = pieces(s); console.log(p.length, JSON.stringify(p)); } You can predict part of what it prints by reading the o200k pre-split regex in tiktoken's openai_public.py. Every chunk the regex produces costs at least one token, so the regex sets a floor. - camelCase splits at every capital. The letter rule matches a run of uppercase letters (possibly none) followed by lowercase, so customerShippingAddressLine2 cuts into customer, Shipping, Address, Line and 2 before BPE starts. Five chunks, at least five tokens. - Digits go in groups of three. The number rule is \p{N}{1,3}. A 13-digit millisecond timestamp is five chunks, so at least five tokens, every time it appears in a log line you paste. - Whitespace is its own chunk. Indentation runs, minus their last space, are cut out separately from the keys they sit in front of. One small quirk: the space after "customer_id": in the pretty version cannot attach to a following number, because the digit rule (\p{N}{1,3}) takes no leading character and the punctuation rule's optional space needs a symbol after it. It becomes a chunk on its own. Before a string value it rides along with the opening quote instead. - Non-English text gets fewer merges. Byte-level BPE can encode anything, but the merges go to whatever was common in training data. Petrov et al. (NeurIPS 2023) found the same text translated into different languages could differ in token length by up to 15 times on the tokenizers they studied. OpenAI's larger vocabulary cut non-English counts: its launch figures, as quoted in Willison's notes, put Gujarati at 4.4x fewer tokens and Japanese at 1.4x fewer than the previous tokenizer. When a token boundary still lands inside a Japanese character, pieces prints � for those fragments. !A long accordion of curly braces and indentation blocks next to a compact stack of the same blocks Same data, same keys and values; only the whitespace and repetition change, and that is where the tokens go. One payload, three shapes Now measure the thing you actually send. This script builds one payload and counts it as pretty JSON, minified JSON and TOON, a tabular encoding of the JSON data model that declares the keys of a uniform array once (@toon-format/toon on npm). // shapes.ts import { Tiktoken } from "js-tiktoken/lite"; import o200k_base from "js-tiktoken/ranks/o200k_base"; import { encode as toToon } from "@toon-format/toon"; const enc = new Tiktoken(o200k_base); const orders = Array.from({ length: 50 }, (_, i) => ({ orderId: 5000 + i, customerId: 70 + (i % 9), status: i % 3 ? "shipped" : "pending", totalCents: 1999 + i * 37, })); const payload = { orders }; const shapes: Record<string, string> = { pretty: JSON.stringify(payload, null, 2), minified: JSON.stringify(payload), toon: toToon(payload), }; for (const [name, text] of Object.entries(shapes)) { console.log(name.padEnd(9), enc.encode(text).length); } Swap payload for a real request body from your logs and you have your own numbers. I can't give you numbers for your data, so here are published ones for theirs. TOON's benchmark, counted with o200k_base: | Track | Pretty JSON | Minified JSON | TOON | |---|---|---|---| | Flat data, 3 datasets | 164,643 | 104,908 | 68,030 | | Mixed structure, 5 datasets | 393,637 | 260,451 | 264,734 | On flat, uniform records TOON came in 58.7% under pretty JSON and 35.2% under minified. On mixed and nested data the picture flips: minified JSON beat TOON by 1.6% overall, and the README says plainly that "for deeply nested or non-uniform data, JSON may be more efficient." It also notes that absolute counts are tokenizer-specific, since other providers tokenize differently, and that you have to measure latency yourself. Fewer tokens is also not the same as better answers, so run your evals on the new shape before you ship it. One win held on both tracks: minifying the JSON cut about a third of the tokens, so do that to anything a program produced before it goes into a prompt. Dropping keys the model does not need helps whatever the shape. Save the tabular format for long arrays of identical objects, where TOON won, and keep JSON for nested data, where it lost. Then put shapes.ts next to your prompt builder and check the count every time someone adds a field. --- If this was useful Token cost is the first bill an LLM app runs up, and it is one of the threads in AI That Answers, the first book of my AI in TypeScript series. It covers your first LLM app in TypeScript end to end: prompts, structured output, and keeping an eye on what each call costs before the invoice does it for you.  The other four books go from retrieval and tool calling to agents in production: the whole AI in TypeScript series is here.

Claude Code's Safety Telemetry Event Lived Six Days. Audit Your Agents With Git - Book: Observability for LLM Applications - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- claude_code.safety_event existed for six days. Claude Code 2.1.307 added it on October 21, a telemetry event a security team would want, logged "for each tool call a permission rule, auto mode, a safety check, a hook or the sandbox stopped, prompted or sent back" (changelog). Version 2.1.312 carried one line about it on October 27: "Changed Claude Code to stop sending the claude_code.safety_event OpenTelemetry event" (same changelog). If you wired a dashboard or an alert to that event last week, it went quiet on the day your developers updated. Nothing errors when an event stops arriving. The panel shows zero, and zero looks like good news. Vendor telemetry from an agent CLI is useful, and you should collect it. But it is a schema that moves every few days, so it can't be the only record of what your agents did. The record that holds still is the one you own: git, and hooks you wrote. A CLI that ships 13 times in 14 days Look at the release cadence first. The Claude Code changelog lists 2.1.300 on October 14 and 2.1.312 on October 27, with a release on most days in between (changelog). Thirteen versions in fourteen days. Many of those touched the OpenTelemetry output. In that same window, the changelog records: - 2.1.300 split cache writes into cache_creation_5m_tokens and cache_creation_1h_tokens on the api_request event, and added GenAI semantic-convention attributes to trace spans. - 2.1.301 added classifier_verdict and permission_mode to tool_decision events. - 2.1.302 fixed new tool_decision attributes "pushing out tool_use_id, tool_source and tool_parameters" when OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT is low. - 2.1.312 changed mcp_server.name to report real names for some plugin servers "instead of custom", and stopped sending safety_event. Every one of those is a reasonable change, and most are improvements. Anthropic writes them down, which is more than many vendors do. But read 2.1.302 again: for a while, on some configurations, adding attributes could push out the ID you would join on. And after 2.1.312, a query grouping MCP calls by mcp_server.name can split what used to be one custom bucket into several names. A query written on October 20 and the same query on October 28 are not measuring the same thing. The docs say as much about one corner of it. On joining OTel data to transcripts, the monitoring page warns that the transcript format "changes between versions, so a pipeline that joins on these fields can break on any release; treat the joins as version-specific rather than a stable contract." !A security dashboard panel reading flat zero while the pipe feeding it has been quietly disconnected upstream An alert on a missing event doesn't fire. It just reads zero. The docs still point at the event Here is the part that makes it easy to miss. As of October 29, the monitoring page still has a full "Safety event" section for claude_code.safety_event, with its category, decision, decision_source and reason_code attributes, marked "Requires Claude Code v2.1.307 or later." It does not say the event stopped in 2.1.312. It goes further. The tool_decision event's source attribute says "config" without telling you which rule matched, and the page's answer is: "To find which one refused a call, look for the Safety event with the same tool_use_id and read its category and rule_source attributes." The security signals table on the same page lists safety_event as the event for "Which check refused a tool call or required a permission prompt for it." So a careful engineer who reads the docs today, builds the query the docs describe, and points it at a current Claude Code gets nothing back. Docs and code drift, and a team shipping that fast will drift more. It is exactly why you don't make a fast-moving vendor stream your only source of truth. To be fair to the telemetry: tool_decision is still documented, with decision set to "accept" or "reject" and a source that separates config, hooks and the user's own answers. Keep collecting it. Just treat it like a third-party API: pin the Claude Code version you validated, alert when the event volume for a type drops to zero, and re-read the changelog before you upgrade. What doesn't move: hooks you own Claude Code's hooks are a contract you control. A command hook gets a JSON payload on stdin with session_id, cwd, permission_mode and hook_event_name, plus tool_name, tool_input and tool_use_id for tool events like the three below. The fields can still change between versions, but the file they land in, its location and its retention are yours. Three events get you most of the way back. PreToolUse fires before every tool call. PermissionRequest runs "when Claude Code is about to ask you for permission to use a tool." PermissionDenied runs when auto mode denies a call, with a reason field, and only then: the docs say it doesn't run for a manual denial, a PreToolUse hook block or a matching deny rule (hooks docs). So a call a deny rule blocked without a prompt shows up in your log as a PreToolUse line with no outcome. For the outcome, pair that line with tool_decision on the same tool_use_id. A logger that appends each payload as one JSON line: #!/bin/sh # .claude/hooks/audit.sh # Append the hook payload to a JSONL audit log. log="$CLAUDE_PROJECT_DIR/.claude/audit/tool-calls.jsonl" mkdir -p "$(dirname "$log")" jq -c '. + {logged_at: (now | todate)}' >> "$log" exit 0 It needs jq and chmod +x. Register it for all three events in .claude/settings.json: { "hooks": { "PreToolUse": [{ "matcher": "*", "hooks": [ { "type": "command", "command": "\"${CLAUDE_PROJECT_DIR}\"/.claude/hooks/audit.sh" } ]}], "PermissionRequest": [{ "matcher": "*", "hooks": [ { "type": "command", "command": "\"${CLAUDE_PROJECT_DIR}\"/.claude/hooks/audit.sh" } ]}], "PermissionDenied": [{ "matcher": "*", "hooks": [ { "type": "command", "command": "\"${CLAUDE_PROJECT_DIR}\"/.claude/hooks/audit.sh" } ]}] } } The script prints nothing and exits 0. The docs are explicit about what that means for PreToolUse: "Exit code 0 with no output means the hook has no decision to report, so the tool call continues through the normal permission flow." The hook records; it doesn't approve. Commit the hook and the settings so every clone gets them, and add .claude/audit/ to .gitignore if you'd rather not commit the log itself. If a future release renames a field, your log still holds the raw payload, so you can read it and adjust. And if a hook stops firing, you notice from a file you own going stale, which is a check you can write today. And git, which no agent vendor owns Hooks tell you what the agent tried. Git tells you what it actually changed, and git doesn't change when your agent ships a new version. git diff, git log -p and git diff main...agent-branch show every line an agent left behind, whatever the agent's version, whatever its vendor. When the question is "did the agent touch the CI workflow" or "did a lockfile change", the answer is in the diff. Point your alerts at the things that matter in the repo: a pre-commit hook or a CI job that fails when .github/workflows/ or a lockfile changes on an agent branch keeps working whether the agent sent an event or not. This is also how I build Hermes IDE, so weigh this section with that in mind. Hermes runs Claude Code, Codex and other agents each in its own terminal and its own worktree. Its README says it "observes what agents report on purpose and what git shows," and it "never answers a permission prompt for you and never types into a terminal on its own." Every agent turn is saved, so you can diff or restore any single turn, with undo. The Review Desk (⌘G) shows the changes by turn or by file, with risk flags (README). !The Hermes Review Desk in By turn view: two turns of a Claude Code task, turn 1's diff of src/db.ts, the changed files and a drafted commit message The Review Desk (⌘G) in By turn view for a Claude Code task: two turns, turn 1's diff of src/db.ts, the changed files and a commit message drafted from the turns. None of that depends on which telemetry event a vendor sent this week. If Claude Code renames an OTel attribute tomorrow, the turn diff looks the same, because it comes from git. What to do this week If you collect Claude Code telemetry, search your dashboards and alert rules for safety_event. Anything built on it has been empty since your fleet moved to 2.1.312 or later. Rebuild what you can on tool_decision, and add a "this event type went silent" alert for every event you depend on. Then put the durable layer under it: a hook log you own, and a CI check that fails when an agent branch touches .github/workflows/ or a lockfile. --- If this was useful The problem here isn't specific to Claude Code: any LLM system you watch through someone else's event stream can change shape under your alerts. Observability for LLM Applications is about building tracing, evals and cost tracking that survive that, with your own instrumentation at the core and vendor data as one input. If you run several coding agents, Hermes IDE is source-available under BSL 1.1 and runs on macOS, Windows and Linux. The code is on GitHub. 

Your Postgres Get-or-Create Has a Race. Postgres 19's ON CONFLICT DO SELECT Closes It - Book: Database Playbook - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Two requests to tag an article postgres arrive a few milliseconds apart. Both look up the tag. Neither finds it. Both insert it. One gets an id back. The other gets SQLSTATE 23505, unique_violation, and your API returns a 500 for a request that did nothing wrong. Get-or-create is the most boring query in your codebase, and the obvious ways to write it on PostgreSQL 18 are all wrong somewhere. Some race. Some return nothing. One writes to the table every time you read from it. On 12 February 2026, commit 88327092ff added INSERT ... ON CONFLICT DO SELECT, and it is in PostgreSQL 19 RC 1, released 15 October 2026. GA is planned for 19 November 2026, barring delays. Until then it is a release candidate, so don't plan a production migration around it yet. Everything below uses the default READ COMMITTED isolation level and one table. Open two psql sessions, A and B, and follow along. CREATE TABLE tags ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL UNIQUE ); SELECT, then INSERT: the race you write first This is the version most code starts with. Look, and insert if missing. -- Session A BEGIN; SELECT id FROM tags WHERE name = 'postgres'; -- no rows -- Session B BEGIN; SELECT id FROM tags WHERE name = 'postgres'; -- no rows -- Session A INSERT INTO tags (name) VALUES ('postgres') RETURNING id; -- Session B INSERT INTO tags (name) VALUES ('postgres') RETURNING id; -- blocks: A holds an uncommitted entry in the unique index -- Session A COMMIT; -- Session B now fails with unique_violation (23505) The BEGIN does not save you. A SELECT that finds nothing locks nothing, so the transaction has no row to protect. A 2020 write-up on concurrency-safe upserts reaches the same verdict: wrapping the read and the write in a transaction "is not going to help in any way here." The unique constraint keeps the data correct. Your code is what breaks. DO NOTHING RETURNING goes quiet on conflict ON CONFLICT has been around since 9.5 and fixes the error. It also takes away the row you wanted. INSERT INTO tags (name) VALUES ('postgres') ON CONFLICT (name) DO NOTHING RETURNING id; When the tag is new, you get its id. When it already exists, you get zero rows. The PostgreSQL 18 INSERT docs are explicit: "Only rows that were successfully inserted or updated will be returned." A skipped row is neither. A 2020 pgsql-hackers thread that proposed a SELECT conflict action put it plainly: "in case of conflict no value is returned." So you add a second statement, a SELECT by name. That is two round trips, and if a DELETE lands between them the SELECT comes back empty as well. Correct code ends up wrapping both in a retry loop. DO UPDATE writes a row so it can read one The common workaround is a no-op update, because DO UPDATE always returns a row. INSERT INTO tags (name) VALUES ('postgres') ON CONFLICT (name) DO UPDATE SET name = EXCLUDED.name RETURNING id; It works. It also turns every "get" into a write, and Postgres does not skip a write because the values match. The trigger functions docs describe the "normal behavior which always performs a physical row update regardless of whether or not the data has changed." Each of those writes costs you something: - Every lookup leaves a dead tuple behind. "An UPDATE or DELETE of a row does not immediately remove the old version of the row," per the vacuuming docs, so VACUUM ends up cleaning after your reads. - Changes to table and index files are logged to WAL first, so replicas and backups carry the no-op writes too. Unless the update qualifies as a heap-only tuple (HOT) update, the new row version also needs new index entries. - With DO UPDATE, row-level BEFORE UPDATE triggers can run, and statement-level UPDATE triggers run whether or not any row was updated. An updated_at trigger now bumps the timestamp on every lookup. - The INSERT docs note that "all rows will be locked when the ON CONFLICT DO UPDATE action is taken." Two transactions fetching the same hot tag queue behind each other until the first one commits. Laurenz Albe's write-up at CYBERTEC lists the same two headline costs: a dead row for VACUUM and, without HOT, an update to every index "even though all data stay the same." The CTE + UNION trick misses a row it waited for The cleverer pre-19 answer does it in one statement: try the insert in a data-modifying CTE, and union in a lookup for the case where it was skipped. WITH ins AS ( INSERT INTO tags (name) VALUES ('mvcc') ON CONFLICT (name) DO NOTHING RETURNING id ) SELECT id FROM ins UNION ALL SELECT id FROM tags WHERE name = 'mvcc' LIMIT 1; No write on the conflict path, one round trip. Now race it. -- Session A BEGIN; INSERT INTO tags (name) VALUES ('mvcc'); -- Session B: run the CTE query above -- blocks on A's uncommitted index entry -- Session A COMMIT; -- Session B returns zero rows Two documented rules combine here. The PostgreSQL 18 concurrency docs say that in READ COMMITTED, DO NOTHING "may have insertion not proceed for a row due to the outcome of another transaction whose effects are not visible to the INSERT snapshot." And the WITH docs say all sub-statements "are executed with the same snapshot." B's snapshot was taken before A committed. The insert waits for A, sees the conflict, and skips. The fallback SELECT reads from that same older snapshot, where the row does not exist yet. Both branches come back empty, and you are back to a retry loop. Postgres 19's DO SELECT, and what to ship before GA The new conflict action returns the existing row and writes nothing. INSERT INTO tags (name) VALUES ('postgres') ON CONFLICT (name) DO SELECT RETURNING id; Run the same two-session race against it. The PostgreSQL 19 concurrency docs say that if the conflict comes from a transaction not yet visible to the INSERT, "the command will wait for that transaction to commit or roll back, then return the conflicting row if it was committed (even though that row was not visible when the command started)." That is the exact case the CTE trick gets wrong. In READ COMMITTED, each row is "guaranteed to either insert or return the conflicting row." Nothing is modified on the select path. The 19 INSERT docs say "the new values will be identical to the old values, since no modification takes place." So you get no new row version, no dead tuple for VACUUM, and no row-level UPDATE triggers. You also get no row lock, unless you ask for one: INSERT INTO tags (name) VALUES ('postgres') ON CONFLICT (name) DO SELECT FOR UPDATE RETURNING id, (old.id IS NULL) AS created; FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE and FOR KEY SHARE lock the conflicting row, for when you are about to modify it in the same transaction. The created column uses the old/new aliases in RETURNING that arrived in PostgreSQL 18. The 19 docs say old values can be non-NULL when the proposed row conflicts with an existing one. A freshly inserted row has no old version, so old.id IS NULL means this call created the tag. The rules, from the same INSERT page: - RETURNING is mandatory, and so is a conflict target such as (name). - An optional WHERE filters which existing rows come back. With FOR UPDATE/SHARE, rows that fail the WHERE are still locked. - Only unique indexes and NOT DEFERRABLE constraints can be arbiters. Exclusion constraints are not supported. - FOR UPDATE/SHARE needs UPDATE privilege on at least one column. Client libraries are starting to pick it up. Sequel 5.108.0, the Ruby toolkit, added :select and :select_where options to insert_conflict for PostgreSQL 19+. As of today PostgreSQL 19 is a release candidate. The RC 1 announcement says it "will be mostly identical to the initial release," plans RC 2 for 12 November and GA for 19 November 2026. The release notes still read "Release date: 2026-??-??". Test against RC 1. Do not run it in production, and do not assume the feature is on your managed Postgres until your provider ships 19. Until then, put the retry loop in a function. In PL/pgSQL, a VOLATILE function (the default) takes a fresh snapshot for each query, so the second pass of the loop sees the row a concurrent transaction just committed. CREATE FUNCTION tag_id(p_name text) RETURNS bigint LANGUAGE plpgsql AS $$ DECLARE v_id bigint; BEGIN LOOP SELECT id INTO v_id FROM tags WHERE name = p_name; IF FOUND THEN RETURN v_id; END IF; INSERT INTO tags (name) VALUES (p_name) ON CONFLICT (name) DO NOTHING RETURNING id INTO v_id; IF FOUND THEN RETURN v_id; END IF; END LOOP; END $$; When you move to 19, the whole function collapses into the three-line DO SELECT statement at the start of this section. --- If this was useful Get-or-create is the kind of query that looks finished in review and fails under load, and the fix depends on how your store handles snapshots, locks and row versions. The Database Playbook is about that layer: choosing a store for the workload you have, and knowing which of its guarantees your code is leaning on. If this post had you rereading your upsert helpers, that is the book I wrote for it. 

Type a pipe() That Checks Every Step: Variadic Tuples, infer and One Recursive Type - Book: The TypeScript Type System - Also by me: The TypeScript Library — the 5-book collection - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Open Effect's Function.ts and count the signatures of pipe. There are 20: one for a bare value, then one more for every extra function, up to 19. RxJS 7's standalone pipe types nine functions and then falls back to a catch-all signature that returns UnaryFunction<T, unknown>. Hand it a tenth function and the output type is gone. Those libraries know exactly what they are doing. When TypeScript 4.0 shipped variadic tuple types, the release notes called this pattern "death by a thousand overloads", and overloads are still how the big libraries type pipe. The reason is that a single generic signature that checks every step is harder to write than it looks, and it gives some things up. You're going to write that single signature anyway: about 40 lines of types, checked by tsc --noEmit, and then the four places it breaks so you know when overloads win. Step 1: read the ends with infer Start with four small helpers. In and Out pull the parameter and return type out of a one-argument function. First and Last pull the first and last element out of a tuple. // pipe.ts export type Fn = (arg: any) => any; type In<F> = F extends (arg: infer A) => any ? A : never; type Out<F> = F extends (arg: any) => infer R ? R : never; type First<T extends unknown[]> = T extends [infer F, ...unknown[]] ? F : never; type Last<T extends unknown[]> = T extends [...unknown[], infer L] ? L : never; Last matches with a leading rest element, [...unknown[], infer L], which TypeScript has accepted since 4.2. With these you can write the obvious pipe type: the input of the first function, the output of the last. type NaivePipe<Fns extends Fn[]> = (input: In<First<Fns>>) => Out<Last<Fns>>; It reads the ends and never looks at the middle. Give it trim (string to string), len (string to number) and shout (string to string) and it reports (input: string) => string, even though shout would receive a number at runtime. A pipe type that only reads the ends is an as cast with better marketing. !A pipeline of joined segments where only the first and last joints are inspected and a mismatched joint in the middle goes unnoticed Reading only the first input and the last output leaves every joint in the middle unchecked. Step 2: walk the list with one recursive type The real check has to visit every adjacent pair. Recursive conditional types (TypeScript 4.1) let a type refer to itself in its own branches, so you peel one function off the front, compare its input with the value flowing in, and recurse on the rest. type PipeError<Step, Got, Want> = { readonly error: "pipe(): a step got the wrong input"; readonly step: Step; readonly got: Got; readonly want: Want; }; type Check< Fns extends Fn[], Input, Cur, Done extends unknown[] = [] > = Fns extends [infer F, ...infer Rest extends Fn[]] ? [Cur] extends [In<F>] ? Check<Rest, Input, Out<F>, [...Done, F]> : PipeError<[...Done, F]["length"], Cur, In<F>> : (input: Input) => Cur; export type Pipe<Fns extends [Fn, ...Fn[]]> = Check< Fns, In<First<Fns>>, In<First<Fns>> >; Four details carry the weight here. infer Rest extends Fn[] puts a constraint on the inferred rest, a feature from TypeScript 4.7. Without it, Rest can't be passed back into Check, whose first parameter demands Fn[]. [Cur] extends [In<F>] wraps both sides in brackets on purpose. A bare Cur extends ... on a type parameter is distributive: if a step returns string | number, the check would split into one branch per member and hand you a union of a function and an error. The brackets compare the whole union at once. Done collects the functions already visited. Its only job is to number the failing step: [...Done, F]["length"] is the 1-based position of F as a number literal type. The recursive call is the entire branch result. Nothing wraps it, and that makes Check tail-recursive, which matters when chains get long. The runtime part is short: export function pipe<Fns extends [Fn, ...Fn[]]>( ...fns: Fns ): Pipe<Fns> { const run = (input: unknown) => fns.reduce((acc: unknown, fn) => fn(acc), input); return run as unknown as Pipe<Fns>; } Fns extends [Fn, ...Fn[]] on a rest parameter makes TypeScript infer the arguments as a tuple, one element per function, which is what Check walks. The double cast is the honest part: the body can't prove a deferred conditional type, so you assert once at the boundary and callers get the checked type. The overloaded libraries pay the same toll in their own way; Effect's implementation signature is pipe(a: unknown, ...args: Array<any>): unknown. Step 3: prove it with @ts-expect-error A type you can't test is a type you will break by accident. // @ts-expect-error (TypeScript 3.9) suppresses the error on the next line, and if that line has no error, tsc reports Unused '@ts-expect-error' directive. That turns every expected failure into an assertion. // pipe.types.ts import { pipe } from "./pipe"; type Equal<A, B> = (<T>() => T extends A ? 1 : 2) extends (<T>() => T extends B ? 1 : 2) ? true : false; type Expect<T extends true> = T; const trim = (s: string) => s.trim(); const len = (s: string) => s.length; const isLong = (n: number) => n > 10; const shout = (s: string) => s.toUpperCase(); const ok = pipe(trim, len, isLong); export type OkIsTyped = Expect< Equal<typeof ok, (input: string) => boolean> >; // @ts-expect-error the input must be a string ok(42); const broken = pipe(trim, len, shout); // @ts-expect-error step 3 wants a string, gets a number broken(" hello "); // @ts-expect-error a broken pipe is not a function const f: (s: string) => string = broken; // @ts-expect-error pipe() needs at least one step pipe(); Equal compares two types strictly, so a result that drifts to any or to a wider type fails it; Expect fails to compile when handed false. A minimal tsconfig.json next to the two files: { "compilerOptions": { "strict": true, "noEmit": true, "target": "es2022", "module": "esnext", "moduleResolution": "bundler" }, "include": ["pipe.ts", "pipe.types.ts"] } Install the compiler with npm i -D typescript, then run npx tsc. The config needs TypeScript 5.0 or newer for "moduleResolution": "bundler"; the types themselves need 4.7. If any expectation stops holding, either an Expect line or an unused directive fails the build, locally or in CI. Where the recursive version breaks Look again at the broken test. The pipe(trim, len, shout) line compiles. The type of broken is PipeError<3, number, string>, and the error only appears where you call it or assign it to a function type. Hovering broken tells you the step, what it got and what it wanted, which beats a silent unknown, but the red squiggle sits on the wrong line. Overloads don't have this problem: each step is its own parameter, so a mismatch is an argument error on the exact function that's wrong. There is a variant that rewrites the rest parameter to the expected tuple so the error moves onto the bad argument; ecyrbe's write-up shows one. I haven't run it, and a commenter there reports a gap in its first-to-second step check, so test it the same way before you trust it. Generic steps lose their type parameter. Take const wrap = <T>(x: T): T[] => [x]. When In and Out match a generic signature against infer, the type parameter doesn't survive; TypeScript issue #38964 shows ReturnType of <T = string>() => T coming out as unknown. So pipe(trim, wrap, join), with join taking string[], fails with got: unknown[]. The fix is an instantiation expression (4.7): pass wrap<string> as the step and the chain checks. Overloads get this for free, because each step is matched against (a: A) => B with A already known, and since TypeScript 3.4 a composer can even propagate a generic step's type parameters into the result. Inline lambdas aren't typed from the previous step. In Effect, pipe(" hi ", (s) => s.trim(), (s) => s.length) types each s from the step before it, because every overload names the intermediate types as separate type parameters. In ...fns: Fns, nothing ties the second slot's parameter to the first slot's return, so an unannotated (s) => s.length has no way to learn that s is a string. The best it gets is the any from the Fn constraint, and an any step checks nothing. Annotate every inline step, or write the steps as named functions. Long chains hit the depth limit, depending on how you wrote the recursion. The TypeScript PR that added tail-recursion elimination lets tail-recursive conditional types loop 1,000 times before giving up, and lowered the general limit for the rest to 100. Check is tail-recursive, so a 30-step pipe is nowhere near the ceiling. Rewrite it so the recursive call sits inside something else, a tuple spread or a union, and you are back under the general limit, where the failure is the familiar Type instantiation is excessively deep and possibly infinite from the 4.5 release notes. Even well under the limit, the 4.1 notes warn that recursive types can increase type-checking time, so keep the step types themselves simple. When overloads are still the better choice !A tall stack of near-identical cards on one side and a single looping arrow passing through a row of function blocks on the other Overloads write one signature per chain length. The recursive type writes one signature that walks the list. The rule that falls out of the four breaks: choose by who writes the steps. If callers write steps inline, as lambdas, or pass generic helpers like map and filter, use overloads. That is the RxJS and Effect situation, where most steps are operators with type parameters and most callbacks are unannotated. Contextual typing per step and per-argument errors are worth more than an unlimited chain length, and very few real pipelines need more than the 19 functions Effect covers. If callers pass named, already-typed, non-generic functions, use the recursive type. Validation chains, middleware lists and data transforms assembled from a registry all fit: the steps already carry their own types, chains can be longer than any overload stack, and the PipeError type names the failing step. Ship it with the @ts-expect-error file so the next refactor can't quietly break the check. --- If this was useful The recursive Check type is a small version of a pattern that keeps coming back: walk a tuple with infer, carry state in an accumulator, keep the recursion in tail position. The TypeScript Type System goes deep on that toolkit: generics, mapped and conditional types, infer, template literals, branded types. If you already write TypeScript daily, that's the one to read next; if you're still getting comfortable with narrowing and modules, start with TypeScript Essentials. - TypeScript Essentials — types, narrowing, modules, async, daily-driver tooling - The TypeScript Type System — generics, mapped/conditional types, infer, template literals, branded types - Kotlin and Java to TypeScript — JVM developers moving to TypeScript - PHP to TypeScript — PHP 8+ developers moving to TypeScript - TypeScript in Production — tsconfig, build tools, monorepos, library authoring, dual ESM/CJS, JSR Coming from Kotlin, Java or PHP? The two bridge books replace the first two for you. Shipping TypeScript at work, with build tooling and published packages? TypeScript in Production covers that layer. All five books ship in ebook, paperback, and hardcover. 

Claude Took Anthropic From 2,500 to 30,000 Merges a Day. Then It Moved Off Git - Book: AI Agents Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- In May 2026, Anthropic merged around 2,500 code changes a day. By September it was merging more than 30,000 a day, according to the company's engineering blog. Anthropic names the cause itself. Claude's ability to write code, the post says, is "what put us in this situation in the first place." It also names where things gave way first: "The bottleneck, now, is all of the developer infrastructure built for human scale: source control, builds, and CI." On October 13, Anthropic published what it did about it. A team of seven built Scion, an in-house version control system, plus a code review tool called Antique, and fully migrated the monorepo off git in five months. The post drew a busy Hacker News thread within hours. What broke at 30,000 merges a day The scale numbers in Anthropic's write-up explain why. "The history of our main branch alone is almost 100 GB, and a compressed working copy is over 25 GB." Anthropic runs "on the order of 25 million code checkouts per day for CI." Then the line that ends the argument: "Fetched naively, that would be over 600 PB of git traffic per day." Anthropic's diagnosis is short: "Git assumes that clients work offline and need a complete copy of the repository." That design is why git is pleasant on a laptop. It is also why every CI job that wants one directory still has to negotiate with history it will never read. Anthropic had been absorbing that load with a fleet of git mirrors, which it now calls "our previous git mirroring infrastructure." !A classic branching version-control graph drawn as a tree, overwhelmed by a dense swarm of small orange commit nodes pouring in from robotic hands, with a narrow pipe at the base straining to carry them When agents write the code, the repository's plumbing becomes the slow part. How Scion breaks git's assumptions Scion starts from the opposite premise, which Anthropic states in four words: "Clients are always online." The rest of the design follows from that: - The server owns history, blame, commits, merges and rebases. The client keeps a bounded local cache and the working files. - Checkouts are sparse by default. Scion's protocol "is centered around sparse checkouts, which only contain the directories the client asks for." Some jobs skip the checkout entirely: "a CI job that needs the diff between a commit and its merge base, for example, can ask the server for it." - Fetches go through tiers: the scn client talks to a per-node scion-daemon, which talks to regional scion-cache servers, which talk to scion-server instances backed by Spanner or cloud storage. Anthropic reports over 99% cache hits at each tier. - Branches are mostly gone. Scion "discourages refs (pointers to commits, such as branches and tags)," except for the main ref and release branches, because shared global state is what stops scaling. The results Anthropic reports: an infrastructure footprint "200x smaller than our previous git mirroring infrastructure," checkouts for CI and research jobs that take "seconds rather than minutes," and "a 25% CPU capacity win" for CI alone. These are Anthropic's own numbers on its own systems, and Anthropic hasn't released anything that would let outsiders reproduce them. None of this is new as a category. Anthropic itself points to Google's Piper and Meta's Sapling as earlier server-centric systems. What is new is the reason: Anthropic built one because its own coding model outran its tooling. The process changes matter more than the VCS The part most teams can learn from has nothing to do with storage. Start with the merge queue. "A proper merge queue, for example, doesn't make much sense at agent scale," Anthropic writes. It moved the safety net later instead: "we've invested in post-merge and pre-deploy automated testing." Review changed too. "Our team members still review every change request for the system," but "they now examine the intent of the change as opposed to reading every line of code." Anthropic does not claim the model got everything right. "Agents will sometimes violate core system constraints to satisfy a pattern." One adversarial review "was overly defensive about trusting another internal service," and Claude piled on extra validation to satisfy it. The upside was large too. The team handed an early internal snapshot of Claude Mythos Preview the whole ingestion pipeline to rewrite, and the new version, per Anthropic, "consumed 20x less memory and was 4x faster." Every engineer who worked on the system would now rate Claude as better at writing code than themselves, the post adds. !Abstract illustration of a request travelling down a tiered stack of cache layers, from a small client node through a middle cache ring to a large central server cylinder, with most requests stopping early at the upper tiers Scion's fetch path: client, node daemon, regional cache, server. Most requests never reach the bottom. Should you leave git? Anthropic says no Anthropic is blunt about this. "Scion is by no means a suitable replacement for git for general use." The concepts "really only make sense for truly enormous monorepos." Anthropic has not announced a public release, and it describes its internal systems as highly bespoke. What it asks for instead is better git infrastructure: "Git platforms can provide comprehensive server-side RPCs like Scion," and "A git server doesn't have to work the same way as a git client does." Two of Scion's ideas already exist in stock git at a smaller scale. A partial clone skips file contents until you need them, and sparse checkout only materialises the directories you name. Both are flags on git clone. For a CI job that touches one service: git clone --filter=blob:none --sparse \ https://github.com/your-org/your-repo.git cd your-repo git sparse-checkout set services/billing It won't get you anywhere near Anthropic's numbers, but the instinct is the same: stop shipping the whole repository to a job that reads one folder. My reading of the post: most teams will not hit 30,000 merges a day, but plenty will hit the moment when agents open changes faster than CI and review can absorb them. That is the moment to look at your merge queue and your review rules, long before you look at your VCS. Several agents on one repo, at laptop scale I build Hermes IDE, so weigh this section with that in mind. Hermes is not affiliated with Anthropic. At Anthropic's volume, branches are shared global state worth avoiding. On one machine with three agents, the problem is smaller and different: two agents in one checkout overwrite each other. In Hermes, ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch in its own git worktree, with the task as the agent's first prompt. Claude Code, Codex, Antigravity CLI and other command-line agents each run in a real terminal. The review side is closer to what Anthropic describes. The Review Desk (⌘G) shows an agent's changes by turn or by file, with risk flags, and sends a line comment back to the agent that wrote it. Reviewing one turn at a time makes it easier to judge the intent behind a change. When you are done, Land commits, opens a pull request or merges locally, and can be undone. How many agent-written changes a day can your CI take before it becomes the queue? Tell me in the comments. --- If this was useful Hermes IDE is free, source-available (BSL 1.1), and runs on macOS, Windows and Linux. Download it and give two agents the same ticket on separate branches, or star it on GitHub if keeping agents apart in one repo matters to you. If you build agents yourself, the AI Agents Pocket Guide covers the patterns behind agents that write and review code, like the ones in Anthropic's post.