hyperfine

Your Agent Says It Made It Faster. Benchmark Both Branches Before You Land - Book: The Complete Guide to Go Programming - Also by me: Pocket Guides for Developers — the 8-book series - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- About 3x faster. That is the whole claim, and it sits in the last line of the agent's summary: "Replaced the regex with a hand-rolled scanner. ParseLine is now about 3x faster, all tests pass." Above it you have a diff and a green test run. The number is a sentence. Nothing in the session timed the old code and the new code under the same conditions and compared them. Maybe the agent ran a benchmark once, on a warm cache, while your other agent was compiling in the next terminal. Maybe it ran nothing and estimated. Published research says you should check. When the authors of the GSO benchmark gave leading coding agents 102 real optimization tasks from 10 codebases, the agents succeeded less than 5% of the time. A success there means a patch that passes the correctness tests and reaches at least 95% of the speedup an expert developer got (GSO). The paper names "lazy optimization" and trouble finding the actual bottleneck among the failure modes. It dates from 2025 and the models have moved since, but the lesson holds: a claimed speedup is a hypothesis until you measure it. The fix is a measurement, and most people skip its last step: reading whether the difference is real. Put the base and the agent's commit side by side Say the agent worked on a branch called hermes/speed-up-parseline. You want two folders: one with the exact commit the agent started from, one with the agent's latest commit. git worktree gives you both without a second clone. cd ~/code/app BR=hermes/speed-up-parseline BASE=$(git merge-base main "$BR") git worktree add --detach ../app-base "$BASE" git worktree add --detach ../app-agent "$BR" Use the merge base, not main. If main moved while the agent worked, comparing against it mixes someone else's changes into your result. The merge base is the commit the agent's branch split from, so the only difference between the two folders is the agent's work. The --detach on the agent side matters. By default, git worktree add refuses to check out a branch that another worktree already has, and the agent's own worktree has that branch. A detached checkout of the same commit gets around that, and your builds in ../app-agent never touch the files the agent is still editing. One limit: this measures what is committed. If the agent left its change uncommitted, ask it to commit first, or build the agent side from its own worktree. Time the whole program with hyperfine If the claim is about a command (a CLI, a build step, a converter), build both sides and let hyperfine run them. These examples assume a Go program with its entry point in ./cmd/app. Swap in your own build command. mkdir -p ~/bench (cd ../app-base && go build -o ~/bench/base ./cmd/app) (cd ../app-agent && go build -o ~/bench/agent ./cmd/app) Then run both against the same input: hyperfine --warmup 3 --runs 30 \ --prepare 'rm -f out.json' \ --export-markdown bench.md \ --export-json bench.json \ -n base "$HOME/bench/base convert big.csv out.json" \ -n agent "$HOME/bench/agent convert big.csv out.json" What each flag does, from the hyperfine docs: - --warmup 3 runs each command three times before measuring, which fills disk caches so the first timed run isn't the odd one out. - --prepare runs before every timed run. Here it deletes the output file so each run starts from the same state. To measure cold-cache behavior on Linux instead, the README's example drops the page cache in --prepare. - --runs 30 fixes the sample size. Without it, hyperfine does at least 10 runs and aims for about three seconds of measuring. - -n gives each command a short name for the report. - --export-markdown writes a table with the mean, the ± spread, and a "Change" and "Factor" column against the reference. Order matters. Since hyperfine 2.0.0, released on October 7, 2026, the first command is always the reference and --reference has been removed for now, so put base first. Version 2 also runs commands without a shell by default, which is why $HOME sits inside double quotes: your own shell expands it before hyperfine sees the string. --prepare still runs through a shell. The mean and ± alone don't tell you whether the gap is noise. The hyperfine repo ships analysis scripts for the JSON export (scripts README), including a Welch's t-test for exactly two benchmarks. With uv installed: uvx --from 'git+https://github.com/sharkdp/hyperfine' \ welch_ttest bench.json It ends with one of two lines, either "A statistically significant difference in mean value was detected (p < 0.05)." or "No statistically significant difference in mean value was detected (p >= 0.05)." The second one means the agent's "3x faster" didn't survive 30 runs. Benchmark the Go function with benchstat When the claim is about one function, time that function. Write the benchmark with b.Loop, which the testing package added in Go 1.24. It resets the timer on the first call, so setup above the loop isn't measured, and it keeps the compiler from optimizing the call away. package csvx import "testing" var line = []byte("42,acme,2026-11-02,paid,19.99") func BenchmarkParseLine(b *testing.B) { b.ReportAllocs() for b.Loop() { ParseLine(line) } } The same benchmark file has to exist on both sides. If the agent wrote it as part of its change, copy it into ../app-base/csvx/ before you build. A benchmark that differs between the two folders compares two different things. The benchstat docs suggest at least 10 runs per side, ideally 20, and interleaving the old and new runs so noise from your machine spreads evenly across both. Compiling each side once with go test -c makes interleaving cheap: (cd ../app-base && go test -c -o ~/bench/base.test ./csvx) (cd ../app-agent && go test -c -o ~/bench/agent.test ./csvx) rm -f old.txt new.txt for i in $(seq 20); do ~/bench/base.test -test.run='^$' \ -test.bench=ParseLine -test.benchmem >> old.txt ~/bench/agent.test -test.run='^$' \ -test.bench=ParseLine -test.benchmem >> new.txt done benchstat old.txt new.txt -test.run='^$' skips the unit tests so only the benchmark runs, and -test.benchmem adds allocation counts. Each pass of the loop adds one sample per side, so you end with 20 each. If you don't have benchstat yet, go install golang.org/x/perf/cmd/benchstat@latest installs it. Read the interval before the headline This is one line from the benchstat documentation's own example, with the column spacing trimmed: Encode/format=json-48 1.718µ ± 1% 1.423µ ± 1% -17.20% (p=0.000 n=10) Each column is the median for that file with the ± range of a 95% confidence interval. The percentage is the change from the first file to the second. p is how likely a difference this large is to come from noise alone, and n is the sample count. When benchstat prints ~ instead of a percentage, it found no statistically significant difference. The default threshold is p < 0.05. !Two runners crossing a finish line almost together, with a magnifying glass held over the photo-finish strip A photo finish needs a closer look before anyone gets the medal. The same docs give a few rules for reading the result: 1. ~ means no measurable speedup. Whatever the agent wrote, the data doesn't support it. 2. Don't rerun until it turns significant. Rerunning until you get the answer you wanted is multiple testing, and it biases the result. 3. Expect false alarms. At p < 0.05, about 5% of benchmarks show a significant change that isn't real, so if you compare 20 benchmarks at once, one of them will probably light up by chance. The docs also warn that significant isn't the same as large: with enough low-noise data, a tiny change can pass the test. So decide what size of win is worth the extra code before you look, and then check the claim itself. "About 3x faster" is −67% on time. A real result of −17% means the code got faster and the agent's sentence was wrong. Note both, because you will read that agent's next claim too. And speed only counts if the code is still correct: GSO counts a patch as a success only when it also passes the correctness tests, and you should hold your branch to the same bar. Send the numbers back in Hermes I build Hermes IDE, so weigh this part with that in mind. Nothing above needs it. The git, hyperfine and benchstat steps work in any terminal. What Hermes changes is where the task already lives. When you start a task with ⌘N (Ctrl+Shift+N on Windows and Linux), the agent runs on its own hermes/ branch in its own worktree, which is the $BR you passed to git merge-base above. The Worktrees tab in the Review Desk shows which folder belongs to which task, so you know what to build. !The Hermes Review Desk Worktrees tab listing one worktree per task with its hermes/ branch, task, age and disk size The Review Desk's Worktrees tab: one Hermes worktree each for orders-api, web-app and orders-types under ~/code, with each worktree's hermes/ branch, task, age and disk size, and the free disk space. Once you have the numbers, open the Review Desk with ⌘G (Ctrl+Shift+G) and switch to By turn. Find the turn where the agent rewrote ParseLine, and put a line comment on the changed line with the benchstat output pasted in. The Review Desk sends that comment to the agent that wrote the line, so it lands in the right session even with three agents running. !The Hermes Review Desk in By turn view, showing two turns of a Claude Code task, a diff of src/db.ts and a drafted commit message The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns listed (T1: 3 files +69 −1, T2: 1 file +1 −0), turn 1's diff of src/db.ts adding a refunded status, the changed-files list, and a commit message drafted from the turns. A comment that works, with your own numbers filled in: "benchstat shows ~ for BenchmarkParseLine at n=20 per side, output below. The scanner adds code and no measurable gain. Revert this turn unless you can show a benchmark where it wins." The agent gets a measurement to answer, and the next claim it makes has to come with one. If the win is real, use Land: it commits, opens a pull request or merges locally into the branch the task started from, and you can undo it. Either way, remove the two measurement folders when you're done. If you copied the benchmark file into ../app-base/csvx/, delete that copy first, or remove will refuse: git worktree remove ../app-base git worktree remove ../app-agent The same check covers modified files. Leave --force off and git worktree remove won't delete work you forgot about. Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. Download it here, and if you find something that doesn't behave the way I described, open an issue on GitHub. --- If this was useful Most of this post is Go's own tooling doing the work: the testing package, b.Loop, go test -c and the flags behind them. If you want the language, the runtime and the standard library underneath those tools explained from the ground up, that is what The Complete Guide to Go Programming covers. It is the first book of the Thinking in Go series. 

Sources: cloc v2.08 (AlDanial/cloc), scc v4.1.0 (boyter/scc), hyperfine v2.0.0 (sharkdp/hyperfine). All runs on my own machine, Nov 1.

Free GitHub tool: time a command with hyperfine Hyperfine is a free, open-source command-line benchmarking tool. It runs a command repeatedly and reports timing statistics, which is more useful than timing it once. On macOS with Homebrew, install it using `brew install hyperfine`. Try `hyperfine "sleep 0.3"` in Terminal. For a real comparison, give it two commands that do the same job, such as two versions of a script. Keep inputs and conditions the same, and remember that caching or background work can change the result. This local use needs no subscription or paid API. The project is MIT/Apache-2.0 licensed. Official repository and examples: sharkdp/hyperfine

Free GitHub tool: time repeatable commands with hyperfine hyperfine is a free, open-source command-line benchmarking tool. It runs a command multiple times and reports timing statistics, which is more useful than judging speed from a single run. On macOS with Homebrew, install it with `brew install hyperfine`. Try a harmless example: `hyperfine --runs 5 "sleep 0.3"`. After that, substitute a repeatable, read-only command of your own. Important: hyperfine executes what you give it repeatedly. Do not benchmark a command that deletes files, sends messages, spends money or changes live data. It is MIT or Apache-2.0 licensed and this use needs no account, subscription or paid API. Repository: sharkdp/hyperfine

两个命令哪个快，别靠感觉猜。⏱️ hyperfine 自动多次运行命令并统计耗时，还支持预热与结果导出。做性能对比很方便。📊 sharkdp/hyperfine #Benchmark #CLI #浇朋友 #蓝朋友 #互关 #fo

Don't Tell Your Coding Agent 'Make It Faster.' Hand It a CPU Profile - Book: AI That Answers - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- You type "make the report job faster" and go refill your coffee. The agent comes back with a Map cache around a date formatter, a for loop where a reduce used to be, and a confident summary. The tests are green. Nobody measured anything, so nobody knows whether any of it helped. That isn't laziness on the agent's part. Anthropic's Claude Code best practices say it plainly: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available." For speed, the check is a timing, and the map of where to look is a CPU profile. Node writes one with a single flag. The loop below goes profile, condense, brief, one attempt per branch, measure, land. Steps 1 to 3 are plain Node.js and hyperfine. From step 4 on I use Hermes IDE, which I build, so weigh that part accordingly. Step 1: profile one workload you can rerun Profile the job you care about, the same way every time. A script that exits on its own is easiest, because --cpu-prof "writes the CPU profile to disk before exit" (Node.js CLI docs). // bench.mjs: one fixed workload, run the same way every time import { buildReport } from './src/report.js'; import { makeOrders } from './test/fixtures.js'; const orders = makeOrders(200_000, { seed: 42 }); const report = buildReport(orders); // Same input, same answer. A faster version must print this too. console.log(report.totalCents); buildReport and makeOrders stand in for your hot path and a fixture builder you already have. The last line gives every attempt a cheap correctness check on top of your tests. Record the profile into a folder you keep out of git: mkdir -p perf node --cpu-prof --cpu-prof-dir=perf \ --cpu-prof-name=before.cpuprofile bench.mjs Without --cpu-prof-name, Node names the file CPU.${yyyymmdd}.${hhmmss}.${pid}.${tid}.${seq}.cpuprofile. A fixed name means each run overwrites the last one. The profiler samples every 1000 microseconds by default (--cpu-prof-interval changes that), and the --cpu-prof flags have been stable since Node v22.4.0 and v20.16.0, per the same docs. Add perf/ to .gitignore. Profiles are evidence for this one job and don't belong in the repo. Step 2: condense the profile into a top-20 table Don't hand the agent the .cpuprofile file. It's JSON with every node of the call tree plus one entry per sample, and the agent pays for every token it reads. The same best-practices page warns that "Claude's context window fills up fast, and performance degrades as it fills." What the agent needs is where the time went. The format is the Chrome DevTools Protocol's Profile type. nodes holds the call tree, each node with a callFrame. samples holds the "Ids of samples top nodes", and timeDeltas the "time intervals between adjacent samples in microseconds." The node on top of the stack when a sample fires is where the CPU was at that moment. Add up the deltas per top node and you get self time. // top-self.mjs: top N functions by self time in a .cpuprofile // usage: node top-self.mjs perf/before.cpuprofile 20 import { readFileSync } from 'node:fs'; const [file, n = '20'] = process.argv.slice(2); const prof = JSON.parse(readFileSync(file, 'utf8')); const nodes = new Map(prof.nodes.map((nd) => [nd.id, nd])); const cwd = `${process.cwd()}/`; const rows = new Map(); // timeDeltas[i]: microseconds between sample i-1 and sample i prof.samples.forEach((id, i) => { const { functionName, url, lineNumber } = nodes.get(id).callFrame; const path = url.replace('file://', '').replace(cwd, ''); // lineNumber is 0-based in V8 call frames const where = url ? `${path}:${lineNumber + 1}` : '-'; const name = functionName || '(anonymous)'; const key = `${name} ${where}`; const row = rows.get(key) ?? { name, where, us: 0 }; row.us += prof.timeDeltas[i]; rows.set(key, row); }); const total = prof.endTime - prof.startTime; const top = [...rows.values()] .sort((a, b) => b.us - a.us) .slice(0, Number(n)); console.log('| self ms | self % | function | where |'); console.log('|---:|---:|---|---|'); for (const r of top) { const ms = (r.us / 1000).toFixed(1); const pct = ((100 * r.us) / total).toFixed(1); console.log(`| ${ms} | ${pct} | ${r.name} | ${r.where} |`); } node top-self.mjs perf/before.cpuprofile 20 > perf/top.md The script groups rows by function name and location, so a function reached from five different callers shows up once, with its total. The +1 is there because Runtime.CallFrame line numbers are 0-based, and you want the line your editor shows. Self time here is an estimate at the profiler's sampling resolution, which is plenty for ranking. Rows with - in the where column have no script URL. A V8 profile also records the engine's own work, garbage collection included (JetBrains' V8 profiling docs), and that time is not in any one function you wrote. If those rows sit at the top, say so in the brief. Step 3: write the brief with numbers in it Take a baseline wall time first. hyperfine runs a command at least 10 times by default, aiming for about 3 seconds of measurement, and --warmup adds untimed runs before it starts: hyperfine --warmup 3 --export-markdown perf/baseline.md \ 'node bench.mjs' hyperfine 2.0, released October 7, 2026, runs benchmarked commands without a shell by default (changelog). Keep pipes and && out of the command, or add -S to get the shell back. Now the brief. It follows the pattern from the Claude Code docs: give the symptom, the likely location, and what "fixed" looks like. bench.mjs is too slow. Make it faster without changing what it prints. Evidence (read both before editing): - perf/top.md: top 20 functions by self time, from `node --cpu-prof` on bench.mjs. Approximate, 1 ms samples. - perf/baseline.md: hyperfine wall time for `node bench.mjs`. Rules: - Start from the top rows. For every change, name the row it targets. - No caching or memoization unless you can show the same call repeating with the same input. - Don't edit test/fixtures.js or bench.mjs. - After each change, run npm test, then re-profile: node --cpu-prof --cpu-prof-dir=perf \ --cpu-prof-name=after.cpuprofile bench.mjs node top-self.mjs perf/after.cpuprofile 20 - Report the before and after rows for the function you changed, and the hyperfine result. - If a top row is outside our code, say so and stop there. Each rule closes off a guess. "Name the row" makes every change traceable to the table. The memoization rule blocks the cache-everything reflex. makeOrders runs inside the profile too, so the fixture rule stops the agent from speeding up the benchmark instead of your code. Step 4: one attempt per task, each on its own branch Two attempts at the same brief are worth more than one, and they must not share a folder. First, commit bench.mjs, top-self.mjs and the .gitignore line. A worktree starts from a commit, so anything uncommitted won't be in it. In Hermes, a small recipe prepares each task's worktree. This one follows the format in the worktree recipe docs: # .hermes/worktree.toml setup = ["npm ci"] copy = ["perf/*.md"] done_when = ["npm test"] copy brings over files git ignores, which is what perf/top.md and perf/baseline.md are, so every attempt starts with the same evidence. done_when is the default check for when a task counts as done, so a faster attempt that breaks the tests fails its done check. The first time Hermes sees the recipe, and whenever it changes, it shows you the commands, and nothing runs until you choose Run setup. Press ⌘N (Ctrl+Shift+N on Windows and Linux), paste the brief, and pick Claude Code. The task starts on its own hermes/ branch in its own worktree, with the brief as the agent's first prompt. Press ⌘N again with the same brief and pick Codex, or the same agent with a narrower brief that only allows changes to the top row. Hermes lets you pick the agent, model, effort and account per task. !The Hermes New task sheet with a task typed, chips for agent, project, a new hermes worktree, approval mode, model and effort, and a preview of the command Hermes will run The ⌘N New task sheet in Hermes 2.3.4, shown with a demo task: Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort chips, and the "Hermes will run" preview of the claude command. Wait until both sessions show done before you measure anything. Two agents running test suites in neighbouring worktrees share your CPU, and a benchmark taken while they run picks up their load along with your code's. Step 5: measure both, read by turn, land one Measure every attempt in one hyperfine run, from your main checkout, with the machine otherwise quiet. Don't rely on the numbers the agents reported. Their runs happened at different times, under different load. git worktree list A=../path-to-attempt-a # paste the paths it printed B=../path-to-attempt-b hyperfine --warmup 3 'node bench.mjs' \ "node $A/bench.mjs" "node $B/bench.mjs" diff <(node bench.mjs) <(node "$A/bench.mjs") diff <(node bench.mjs) <(node "$B/bench.mjs") The first command is the reference, and each later result shows its change from it (hyperfine README). Each diff prints nothing when that attempt prints the same total as your main checkout. Then profile the faster attempt and run top-self.mjs on it. The row the agent said it targeted should have shrunk. If the wall time dropped and that row didn't move, you don't yet know why it got faster, so read the change before you trust it. Reading is where per-turn history pays off. Hermes saves every agent turn. The "Agent replies" strip under the session shows one chip per reply with the lines it changed, and a click shows that turn's changes or restores your files to that point. If turn 3 undid the gain from turn 2, you can go back to turn 2. ⌘G (Ctrl+Shift+G) opens the Review Desk, where you read the attempt by turn or by file. A line comment there goes back to the agent that wrote the line, so "this cache never hits, remove it" reaches the right session. !The Hermes Review Desk in By turn view with two turns listed, the first turn's diff, the changed files list and a drafted commit message The Review Desk (⌘G) in By turn view: two turns listed, turn 1's diff of src/db.ts, the changed-files list, and a commit message drafted from the turns. Land the attempt the numbers back. Land commits, opens a pull request, or merges locally into the branch the task started from, and it can be undone. The other attempt never lands. Its branch and worktree are plain git: while its session is open, or if you keep the worktree when you close the session, git worktree list still shows it. What the profile won't tell your agent A CPU profile shows where the CPU went. If bench.mjs spends most of its time waiting on a database or the network, the table will look thin, and it's the wrong tool for that job. The benchmark is not production either. Seeded fixture orders can have a different shape from your real traffic, so a win here is a hypothesis for production. Sampling has limits too. At one sample per millisecond, very short functions may never land in a sample. Lower --cpu-prof-interval if the table looks too coarse. And when two attempts land within the spread hyperfine reports, call it a tie and keep the simpler diff. None of this needs Hermes. Two git worktree add commands, a copy of perf/*.md into each new worktree, two terminals and the same brief run the same loop. --- If this was useful I build Hermes IDE. It's free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. 2.3.4 is the latest release. It isn't affiliated with Nous Research, Anthropic, OpenAI or any agent vendor. Download it, and if the per-task worktrees save you a bad merge, star it on GitHub. The brief in step 3 is a prompt built from evidence, with a stated output format. That's the habit AI That Answers teaches for your own LLM calls in TypeScript: prompts the model can't misread, structured output you can check, and what each token costs you.  It's book 1 of five. The whole AI in TypeScript series, from first LLM call to agents in production, is here.

Bun vs Node.js in 2026: Bun 1.4 vs Node 24 and 26, Benchmarked I ran these benchmarks myself on one machine, and every number below comes from those runs. The original, with any updates, lives on fabwaseem.com. Bun vs Node in 2026 comes down to this: Bun is a JavaScript runtime, package manager, bundler and test runner in one binary, built on JavaScriptCore and now written in Rust. Node.js is the V8-based runtime almost every server, library and hosting platform targets. Bun 1.4 reports Node.js 26 compatibility, so for many apps switching is now a runtime swap rather than a rewrite. I benchmarked Bun 1.4.3 against Node.js 24.22 (the current LTS) and Node.js 26.12 (LTS from October 28) on one machine. I measured HTTP throughput with Hono, Express and plain node:http, a real Postgres-backed API, a large in-memory heap, startup, TypeScript execution, npm install vs bun install, next build, and native addons like sharp and bcrypt. Every number below comes from those runs. TL;DR: should I switch from Node to Bun? Use Bun as your package manager and script runner today: bun install was 13 to 29 times faster than npm in my tests and changes nothing at runtime. Consider Bun as your production runtime for new or dependency-light APIs, CLIs and serverless functions, where it was 1.4 to 1.8 times faster and used 25-70% less memory under load. Stay on Node.js for large existing services that depend on Node-specific APM, long-running stability or exotic native addons, and move to Node 26, which was 8-16% faster than Node 24 on most of my HTTP tests. | | Node.js 24.22 | Node.js 26.12 | Bun 1.4.3 | | --- | --- | --- | --- | | Engine | V8 | V8 | JavaScriptCore | | Status (Oct 2026) | LTS, maintenance from Oct 20 | Current, LTS from Oct 28 | Stable, 1.x | | Hono JSON API, req/s | 53,408 | 60,431 | 98,145 | | Express 5 JSON, req/s | 35,764 | 38,530 | 55,382 | | Hono + Postgres, req/s | 21,749 | 21,008 | 32,254 (pg), 33,694 (Bun.sql) | | RSS under load, Hono JSON | 105 MB | 105 MB | 36 MB | | Startup (-e 0) | 28.5 ms | 26.3 ms | 3.3 ms | | Run a .ts file | 62.6 ms | 59.1 ms | 3.8 ms | | Install a Next.js app (cold) | 15.4 s (npm) | | 1.15 s (bun) | | Native addons tested | All work | All work | All work (sharp, better-sqlite3, bcrypt, argon2) | What is Bun, and how is it different from Node.js? Bun is an all-in-one toolkit: a JavaScript and TypeScript runtime, an npm-compatible package manager, a bundler and a Jest-style test runner in one executable. Node.js is just the runtime and leans on npm, tsc or a loader, a bundler and a test framework from the ecosystem. Bun runs on Apple's JavaScriptCore engine; Node runs on Google's V8. Bun aims to be a drop-in replacement. It implements Node's built-in modules (node:http, node:fs, node:crypto and the rest), reads package.json and node_modules the same way, and runs N-API native addons. It also adds its own faster APIs such as Bun.serve, Bun.sql and bun:sqlite. What's new in Bun 1.4? Bun 1.4 shipped on August 20, 2026, and it is the first Bun written in Rust instead of Zig. Per the Bun 1.4 release post, it adds "+1,517 tests from the Node.js test suite", fixes "over 2,900 issues", cuts idle CPU by 5x and memory by up to 35%, and "starts 50% faster on Linux." Bun now reports itself as Node.js 26 (process.versions.modules is 147). It also absorbs common dependencies into the runtime: Bun.Image (image processing), Bun.WebView (headless browser), Bun.markdown, Bun.cron(), Bun.Terminal, plus bun run --parallel and bun test --parallel. Upgrading from 1.3 isn't free: the breaking changes list includes stricter TLS verification, YAML 1.2 parsing and no .env auto-loading when Bun is invoked as node. Vercel made 1.4 an explicit opt-in on its platform for the same reason. How I benchmarked Bun vs Node.js 1. Machine. An 8 vCPU Intel Xeon VM with 15 GB RAM. Each server was pinned to 2 cores with taskset. The load generator, oha 1.16.0, ran on the other 6 cores with 64 connections. 2. Runtimes. Node.js 24.22.0, Node.js 26.12.0 and Bun 1.4.3, all official Linux x64 builds. 3. Apps. The same files ran on every runtime: a node:http hello server, a Bun.serve hello server (Bun only), a Hono 4.13 app, an Express 5.3 app, and a Hono route that reads one row from PostgreSQL 18 through pg 8.23 (or Bun.sql on Bun) with a 10-connection pool, across 10,000 users at random IDs. 4. Method. For every configuration: start the server, warm up for 4 seconds, measure for 10 seconds, read RSS from /proc. Each configuration ran 3 times and I report the median. Every request returned 200. 5. CLI timings used hyperfine 2.0.0 (40 runs for startup, 3 to 5 for installs and builds). Postgres ran on the same machine without pinning, so the database test also measures a shared CPU. That's closer to a small VPS than to a tuned cluster. Is Bun faster than Node.js? For HTTP servers, yes, clearly. On the Hono JSON route Bun served 98,145 requests per second against 53,408 on Node 24 and 60,431 on Node 26: 1.84 times Node 24 and 1.62 times Node 26. Bun's advantage held on Express (1.55x Node 24) and on the plain node:http module (1.42x), and Bun.serve reached 128,461 requests per second. !Bun vs Node HTTP benchmark: requests per second for node:http, Hono JSON, Express and Hono with Postgres on Node 24.22, Node 26.12 and Bun 1.4.3; Bun leads every workload, 98,145 vs 53,408 req/s on Hono JSON | Server (median of 3 x 10 s, 64 connections) | Node 24.22 | Node 26.12 | Bun 1.4.3 | Bun vs Node 24 | | --- | --- | --- | --- | --- | | node:http hello | 74,200 req/s | 83,301 req/s | 105,434 req/s | 1.42x | | Bun.serve hello | | | 128,461 req/s | 1.73x | | Hono, 10-item JSON | 53,408 req/s | 60,431 req/s | 98,145 req/s | 1.84x | | Express 5, same JSON | 35,764 req/s | 38,530 req/s | 55,382 req/s | 1.55x | | Hono + Postgres (pg) | 21,749 req/s | 21,008 req/s | 32,254 req/s | 1.48x | | Hono + Postgres (Bun.sql) | | | 33,694 req/s | 1.55x | | 600k-entry in-memory cache | 53,314 req/s | 61,908 req/s | 68,134 req/s | 1.28x | Latency followed throughput. On the Hono JSON route p99 was 2.10 ms on Node 24, 1.95 ms on Node 26 and 1.17 ms on Bun. On the Postgres route p99 was 4.75 ms, 6.95 ms and 3.98 ms, and 3.50 ms with Bun.sql. Two results surprised me. First, Bun stayed about 1.5 times faster even with a real database query in every request, which is where many "it's all the database anyway" arguments end. My database was local and fast, though. Over a real network, round trips will shrink that gap. Second, Node 26 alone was 8-16% faster than Node 24 on four of the five HTTP tests (the Postgres route was a tie), so upgrading Node is the cheapest speedup available. Does Bun use less memory than Node.js? Yes. Idle, a node:http server used 58 MB RSS on Node 24 and 23 MB on Bun, while Bun.serve used 16 MB. Under load the Hono app held 105 MB on both Node versions and 36 MB on Bun. On the Postgres route Node 24 grew to 206 MB against 61 MB for Bun with pg and 56 MB with Bun.sql. | RSS (MB): idle / after 10 s load / peak | Node 24.22 | Node 26.12 | Bun 1.4.3 | | --- | --- | --- | --- | | node:http hello | 58 / 95 / 95 | 60 / 97 / 97 | 23 / 55 / 74 | | Hono JSON | 70 / 105 / 105 | 66 / 105 / 105 | 28 / 36 / 45 | | Express JSON | 65 / 97 / 97 | 66 / 96 / 96 | 40 / 72 / 81 | | Hono + Postgres | 70 / 206 / 206 | 66 / 151 / 151 | 28 / 61 / 69 | | 600k-entry cache | 411 / 455 / 455 | 370 / 410 / 410 | 261 / 300 / 491 | The cache test is the exception worth reading. Bun started with less memory but briefly peaked higher (491 MB) than Node (455 MB). Bun's garbage collector is a known weak spot under large, long-lived heaps. A Bun pull request from this year described a GC "storm" that made allocation-heavy handlers run "at roughly half of Node.js" with a 150 MB live set. That PR #35431 was closed without merging. In my 1.4.3 run with a bigger heap Bun was still 28% faster than Node 24, but test your own workload, not a hello world. How fast does Bun start compared to Node.js? Bun starts about 8 times faster. An empty script took a median 3.3 ms on Bun against 28.5 ms on Node 24 and 26.3 ms on Node 26. Running a TypeScript file took 3.8 ms on Bun against 62.6 ms on Node 24. Node's built-in type stripping uses a WebAssembly-based parser, and here it added about 34 ms over an empty script. bun run launched a package.json script in 3.1 ms against 95.1 ms for npm run. | Command (hyperfine median) | Node 24.22 | Node 26.12 | Bun 1.4.3 | | --- | --- | --- | --- | | Empty script (-e 0) | 28.5 ms | 26.3 ms | 3.3 ms | | hello.ts (type annotations) | 62.6 ms | 59.1 ms | 3.8 ms | | import('express') | 94.1 ms | | 65.3 ms | | Run a no-op package.json script | 95.1 ms (npm run) | | 3.1 ms (bun run) | Startup matters for CLIs, cron jobs, serverless cold starts and test watchers. It barely matters for a long-running API server. Both runtimes run TypeScript directly now: Node's type stripping has been stable since v24.12.0 but only removes erasable syntax and ignores tsconfig.json, while Bun transpiles full TypeScript, including enums. Is bun install faster than npm? Much faster. On a Next.js 16.4.2 app with Tailwind, ESLint and TypeScript (458 packages), a cold install with empty caches and no lockfile took 15.4 seconds with npm and 1.15 seconds with Bun, 13 times faster. A CI-style install from a lockfile with a warm cache took 4.6 seconds with npm ci and 0.16 seconds with bun install --frozen-lockfile, 29 times faster. | Install, Next.js 16.4.2 app (458 packages) | npm 11.19 | bun 1.4.3 | Speedup | | --- | --- | --- | --- | | Cold: no cache, no lockfile (median of 3) | 15.4 s | 1.15 s | 13x | | Warm cache + lockfile (median of 5) | 4.6 s | 0.16 s | 29x | | Peak memory, warm install | 758 MB | 13.5 MB | | This is the safest win in the whole comparison. bun install writes a normal node_modules, so you can keep running the app on Node. The cold number includes downloads over my server's network, so yours will vary. # use Bun only as the package manager and script runner bun install # writes bun.lock and a normal node_modules bun run build # runs your package.json script with Node, as before node server.js # production stays on Node.js Does Next.js work with Bun? Yes. A Next.js 16.4.2 production build ran fine with bun --bun next build, in a median 3.03 seconds against 3.22 seconds on Node 24, about 6% faster. Turbopack does the heavy lifting in Rust either way, so the runtime matters less for builds than for serving requests. Without --bun, bun run build launches the Next.js CLI with Node because of its shebang. bun install bun --bun next build # build with the Bun runtime bun --bun next start # serve with the Bun runtime If you deploy with Docker, the base image changes from node:24-slim to oven/bun, and the rest of a production Next.js Dockerfile stays the same. Do native modules work in Bun? The common ones do. On Bun 1.4.3, sharp (image resize), better-sqlite3, bcrypt, argon2 and pg all installed from npm and passed the same smoke test as on Node 24. Bun runs N-API addons, which covers most popular native packages. Packages that pick a prebuilt binary by NODE_MODULE_VERSION need a build for 147, the ABI Bun 1.4 reports. What still bites people is less about compatibility tables and more about operations: APM agents that patch Node internals, rare Node APIs, and memory behavior under months of traffic. The Bun Node.js compatibility page lists module-by-module status. Is Bun faster at raw JavaScript? Sometimes. CPU-bound micro-benchmarks were mixed, so don't expect a blanket speedup. Bun was about 3 times faster at parsing small JSON and 1.4 times faster than Node 26 on a 67 KB document, but JSON.stringify tied Node 26. Node 26 made a big jump over Node 24 on JSON, 1.8 times faster on stringify. | Single core, ops/s (mean of 2 runs) | Node 24.22 | Node 26.12 | Bun 1.4.3 | | --- | --- | --- | --- | | JSON.parse, 110 B | 1,711,027 | 1,881,418 | 5,226,338 | | JSON.parse, 67 KB | 3,854 | 5,224 | 7,118 | | JSON.stringify, 67 KB | 6,539 | 11,634 | 11,644 | | SHA-256 of 1 KB (node:crypto) | 751,142 | 751,912 | 853,054 | | Sort 100,000 numbers | 55 | 60 | 90 | | Map set + get, 100,000 keys | 102 | 134 | 168 | I dropped one test: a regex replace on a constant string ran at 196 million operations per second on Bun, which means JavaScriptCore optimized the work away. That's the trouble with micro-benchmarks. Trust the HTTP and real-app numbers more than these. Should I use Bun in production in 2026? For new services, it's a reasonable default now. Bun 1.4 is fast, frugal and broadly compatible, and platforms such as Vercel Functions run it. For existing Node services, migrate only when you can measure the benefit and you have tested memory over days, not minutes. Trigger.dev reported a 5x throughput gain moving a service to Bun and also hit a Bun-only memory leak on the way. | Situation | My pick | | --- | --- | | New API, CLI or worker with mainstream dependencies | Bun (runtime + package manager) | | Serverless or cron jobs where cold start dominates | Bun | | Existing large Node service with APM and on-call | Node 26 + bun install in CI | | Heavy use of exotic native addons or Node internals | Node.js | | NestJS or Next.js app | Node in production for now, Bun for installs and scripts; test Bun in staging | !Bun or Node decision guide: use bun install everywhere, Bun runtime for new APIs, CLIs and serverless, Node 26 for large existing services with APM and native addons How do I migrate a Node.js app to Bun? Move in three steps and keep a way back at each one. First swap the package manager, then run your tests on Bun, then switch the production runtime behind a canary. # 1. package manager only (zero runtime risk) bun install # creates bun.lock next to package-lock.json # 2. run tests and scripts on Bun bun run test # your existing test script bun --bun run test # force the Bun runtime even for "node" shebangs # 3. run the server on Bun, keep the code portable bun src/server.ts // portable entry point: Bun.serve on Bun, @hono/node-server on Node import { Hono } from 'hono' const app = new Hono() app.get('/health', (c) => c.json({ ok: true })) if (typeof Bun !== 'undefined') { Bun.serve({ port: 3000, fetch: app.fetch }) } else { const { serve } = await import('@hono/node-server') serve({ fetch: app.fetch, port: 3000 }) } Frameworks built on Web standards, like Hono, make this painless because the same fetch handler runs on both. For database access, pg worked unchanged on Bun in my test, and Bun.sql is the built-in Postgres client that was 4% faster than pg on Bun in my test. If you're choosing an ORM, see Drizzle vs Prisma. If you self-host, Coolify runs a Bun Dockerfile like any other. Common mistakes when comparing Bun and Node.js - Benchmarking hello world only. Bun.serve vs node:http showed a 1.73x gap; Express showed 1.55x and a large-heap handler 1.28x. - Comparing against an old Node. Node 26 was 8-16% faster than Node 24 on four of five HTTP tests. Compare against the Node you would actually run. - Ignoring memory over time. A 10-second run says nothing about a slow leak. Watch RSS for days in staging. - Running the load generator on the same cores. Pin the server and the load generator to separate CPUs, or the numbers mostly measure contention. - Forgetting --bun. bun run executes scripts whose CLI has a #!/usr/bin/env node shebang with Node unless you pass --bun. - Switching runtime and dependencies at once. Swap the package manager first; it gives most of the day-to-day win with none of the runtime risk. Related: I also ran a NestJS 12 app on Bun 1.4.3: it served 1.49 times Node 24's requests per second on a hello route and started in 264 ms vs 361 ms. See the NestJS 12 migration guide. Related: framework choice matters on Bun too. In Hono vs Express vs Fastify, Hono served 118,432 JSON requests per second on Bun 1.4.3, 1.62× its Node 24 result, while Express and Fastify gained less. Key takeaways - Bun 1.4.3 served 1.84 times Node 24's requests per second on a Hono JSON API, and 1.48 times with a Postgres query in every request. - Bun used 25-70% less memory under load, for example 36 MB vs 105 MB RSS for the Hono app. - Bun started in 3.3 ms versus 26-29 ms for Node and ran a TypeScript file in 3.8 ms versus about 60 ms. - bun install was 13 times faster than npm cold and 29 times faster with a warm cache and lockfile. - Node 26 alone was 8-16% faster than Node 24 on four of five HTTP tests, a free upgrade when it becomes LTS on October 28. - sharp, better-sqlite3, bcrypt, argon2, pg and a Next.js 16.4 build all worked on Bun 1.4.3. - Adopt Bun as package manager first, then as runtime for new services; keep large existing Node services on Node 26 until you've tested memory over days. FAQ Is Bun faster than Node.js? For HTTP servers, yes. In my tests Bun 1.4.3 served 98,145 requests per second on a Hono JSON API against 53,408 on Node.js 24 and 60,431 on Node.js 26. With a Postgres query in every request Bun was 1.48 times faster than Node 24. CPU-bound micro-benchmarks were mixed, so the gap depends on the workload. Does Bun use less memory than Node.js? Usually. Under load my Hono app used 36 MB RSS on Bun versus 105 MB on Node 24 and 26, and the Postgres route used 61 MB versus 206 MB on Node 24. With a large in-memory cache Bun idled lower but briefly peaked higher, at 491 MB against 455 MB, so test memory with your own data. Is Bun production ready in 2026? For many services, yes. Bun 1.4 passes 1,517 more Node.js test-suite tests than 1.3, runs common native addons, and is offered on platforms such as Vercel Functions. Teams still report Bun-specific memory issues under long-running load, so canary it and watch memory for days before moving a large existing service. Is bun install faster than npm? Yes. On a Next.js 16.4.2 app with 458 packages, a cold install took 15.4 seconds with npm and 1.15 seconds with Bun. A lockfile install with a warm cache took 4.6 seconds with npm ci and 0.16 seconds with bun install --frozen-lockfile. It writes a normal node_modules, so the app can still run on Node. Does Next.js work with Bun? Yes. A Next.js 16.4.2 production build succeeded with bun --bun next build in 3.03 seconds against 3.22 seconds on Node 24. Use the --bun flag, because bun run otherwise launches the Next.js CLI with Node due to its node shebang. Turbopack compiles in Rust either way, so builds gain little. Do native modules like sharp and bcrypt work in Bun? The popular ones do. On Bun 1.4.3, sharp, better-sqlite3, bcrypt, argon2 and pg installed from npm and passed the same smoke test as on Node 24. Bun runs N-API addons and reports NODE_MODULE_VERSION 147, so packages that select a prebuilt binary by ABI need a build for 147. Should I upgrade to Node 26 or switch to Bun? If you have a large existing service, upgrade to Node 26 first: it was 8 to 16 percent faster than Node 24 on most of my HTTP tests and becomes LTS on October 28, 2026. Switch the runtime to Bun when you start a new service or can measure a clear gain in staging. What changed in Bun 1.4? Bun 1.4, released August 20, 2026, is the first Bun written in Rust instead of Zig. It reports Node.js 26 compatibility, adds 1,517 Node.js test-suite tests, fixes over 2,900 issues, starts 50 percent faster on Linux, and adds built-ins such as Bun.Image, Bun.WebView, Bun.markdown and Bun.cron. It also has breaking changes worth reviewing. --- Originally published on fabwaseem.com.

How Long Before Your Agent Says Anything? Time 4 Coding Agent CLIs With hyperfine - Book: Prompt Engineering Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Use --mcp-config to give Claude Code an MCP server that never answers, and a claude -p run waits for it before the first turn: up to MCP_TIMEOUT, 30 seconds by default, on v2.1.221 or later (CLI reference). Codex waits 1,000 milliseconds for optional servers while it builds its first tool list (Codex MCP docs). OpenCode gives a server 5,000 milliseconds to hand over its tools (OpenCode MCP docs). Those are ceilings for a broken server. A healthy session is faster, and how much faster is a number only your own setup can produce. hyperfine measures it. It runs a command many times and reports the mean and the spread, with optional warmup runs before it starts measuring. Below is one hyperfine command per layer of startup, driving each CLI's headless mode: Claude Code's -p, Codex's codex exec, GitHub Copilot CLI's -p and OpenCode's run. I haven't run them myself. The only numbers in this post are documented limits; the measured ones will be yours. What runs before the first token A fresh session does four things before your prompt reaches a model. 1. The process starts. If you installed Codex from npm, the codex on your PATH is a Node script that spawns the native binary (codex.js), so its start includes Node's. 2. Config and memory files load. Claude Code's docs say a plain claude -p discovers hooks, skills, custom commands, subagents, plugins, MCP servers, auto memory and CLAUDE.md "as an interactive session would." The --bare flag skips all of that "to reduce startup time" (headless docs). Codex "builds an instruction chain when it starts," from AGENTS.md files up to 32 KiB by default (AGENTS.md docs). 3. MCP servers start. Each CLI waits a different amount of time for them, per its docs (table below). 4. The model call. Network, queueing, reading the whole prefix, writing the answer. | CLI | Wait for a slow MCP server | Source | |---|---|---| | Claude Code, -p with --mcp-config | until pending servers connect, up to MCP_TIMEOUT (30 s) | CLI reference | | Codex | mcp_optional_startup_grace_ms (1,000 ms) for optional servers; required ones use startup_timeout_sec (10 s) | MCP docs | | OpenCode | timeout for fetching tools (5,000 ms) | MCP docs | | Copilot CLI | not stated in the pages I read | MCP docs | Each run below peels off one layer. !A row of four terminal windows, each with a stopwatch on top, and a queue of small gears, plugs and folded notes waiting in front of a single speech bubble Everything in the queue runs before the model hears your prompt. Setup: hyperfine and an empty repo Install hyperfine with brew install hyperfine, apt install hyperfine or cargo install --locked hyperfine (README), then check the version. hyperfine 2.0.0, published October 7, 2026, runs benchmarked commands directly, without a shell, by default (release notes). That's what you want here: you're timing the agent, not sh. On 1.x, add -N to each hyperfine call below for the same behavior. 2.0 also adds an --env flag for environment variables, since there's no shell to read FOO=bar cmd. hyperfine --version mkdir -p ~/tmp/agent-timing && cd ~/tmp/agent-timing git init -q for c in claude codex copilot opencode; do printf '%s: %s\n' "$c" "$(command -v "$c")" "$c" --version done Codex refuses to run outside a git repository unless you pass --skip-git-repo-check (docs), hence the git init. The loop records which binary you're timing and its version. Paste both into your notes, because next month's release changes the numbers. Install the MCP test server once, globally, so your timing doesn't include npx resolving a package: npm install -g @modelcontextprotocol/server-everything That gives you a mcp-server-everything command (package.json) that speaks stdio when started with no arguments. Its README says it "attempts to exercise all the features of the MCP protocol." One more habit before any benchmark: run each prompt command once by hand. A CLI that stops at a sign-in screen, or exits early with an error, looks fast in a table. Run 1: process start hyperfine --warmup 3 --runs 30 \ --export-markdown run1-version.md \ 'claude --version' 'codex --version' \ 'copilot --version' 'opencode --version' --warmup runs each command before measuring, so the first read of the binary from disk stays out of the result. This is your floor: the binary loads, prints a string and exits. According to its release notes, hyperfine 2.0 reports peak memory next to wall-clock time by default, so you get both from this one command. Run 2: one word to a full answer hyperfine --warmup 1 --runs 10 \ --export-markdown run2-hello.md \ 'claude -p hello' 'codex exec hello' \ 'copilot -p hello' 'opencode run hello' That's 44 real model calls, and each one counts against your plan or your API bill. Drop to --runs 5 if that matters. Each CLI uses its default model and effort, so you're timing a CLI and a model together. To compare the CLIs alone, set each one to a similar model with its model flag. Run 2 minus Run 1 is config discovery, the MCP servers you already have, the network and the model. To pull the local part out, time a stripped-down run next to the normal one: | CLI | Stripped-down run | What it skips | |---|---|---| | Claude Code | claude --bare -p hello | hooks, skills, plugins, MCP servers, auto memory, CLAUDE.md; needs ANTHROPIC_API_KEY (docs) | | Codex | codex exec --ignore-user-config hello | $CODEX_HOME/config.toml, so your MCP servers; sign-in still works (reference) | | Copilot CLI | copilot -p hello --no-custom-instructions --disable-builtin-mcps | AGENTS.md and related files, the built-in GitHub MCP server (reference) | | OpenCode | opencode run --attach <url> hello against opencode serve | the boot of each run (CLI docs) | Bare mode signs in with an API key instead of your subscription, so the Claude row is an approximation. OpenCode's docs suggest the server route themselves, "to avoid MCP server cold boot times on every run": # terminal 1 opencode serve --port 4096 # terminal 2 hyperfine --warmup 1 --runs 10 \ 'opencode run hello' \ 'opencode run --attach http://localhost:4096 hello' Claude Code also reports its own timing in the JSON result, which jq can pull out: claude -p hello --output-format json | jq '{ duration_ms, duration_api_ms, ttft_ms, ttft_stream_ms, num_turns }' The Agent SDK's TypeScript reference defines the optional ttft_ms as time to first token, "measured when the first complete assistant message arrives," on successful runs only, and ttft_stream_ms as the time until the response stream opens. It lists duration_ms and duration_api_ms without defining them, so set those next to hyperfine's mean and remember that hyperfine's number is the one that includes process start and exit. Check num_turns too: if the agent ran a tool to answer "hello", you timed more than one model call. Codex's --json events carry token usage and no timing fields (docs), so for Codex hyperfine is the clock. Run 3: one MCP server, one AGENTS.md, one server that hangs Three small config files, one per format. mcp.json for Claude Code: { "mcpServers": { "everything": { "type": "stdio", "command": "mcp-server-everything" } } } copilot-mcp.json is the same block with "type": "local" and "tools": ["*"], the format in GitHub's MCP docs. opencode-mcp.json: { "$schema": "https://opencode.ai/config.json", "mcp": { "everything": { "type": "local", "command": ["mcp-server-everything"], "enabled": true } } } Then time all four with the server added: hyperfine --warmup 1 --runs 10 \ --export-markdown run3-mcp.md \ 'claude -p hello --mcp-config mcp.json' \ 'copilot -p hello --additional-mcp-config @copilot-mcp.json' hyperfine --warmup 1 --runs 10 \ --env OPENCODE_CONFIG=opencode-mcp.json \ 'opencode run hello' codex mcp add everything -- mcp-server-everything hyperfine --warmup 1 --runs 10 'codex exec hello' codex mcp remove everything Copilot's --additional-mcp-config adds a server "for this session only" and takes a file path prefixed with @ (reference). OpenCode merges OPENCODE_CONFIG between your global and project config (config docs), so nothing else changes. codex mcp add writes to your Codex config, which is why codex mcp remove follows it. On hyperfine 1.x, drop --env and write 'env OPENCODE_CONFIG=opencode-mcp.json opencode run hello', which works with or without -N. The gap between Run 2 and Run 3 is one server starting, plus its tool definitions riding in the prompt. The token side of that has its own walkthrough in What an Empty Prompt Costs; this is the clock side. Now copy a real AGENTS.md from a work repo into the bench folder and rerun Run 2. All four CLIs read it: - Claude Code: when there is no CLAUDE.md or CLAUDE.local.md in the folder or above it, on v2.1.277 or later (memory docs). - Codex: from the git root down (docs). - OpenCode: by walking up from the current directory (rules). - Copilot CLI: unless you pass --no-custom-instructions (reference). A longer file is more text for the model to read before it answers. Delete the file when you're done, or it skews the next run. Last, a server that never answers. Point a config at sleep 60: cat > mcp-hang.json <<'EOF' {"mcpServers": {"hang": {"type": "stdio", "command": "sleep", "args": ["60"]}}} EOF hyperfine --runs 3 \ 'claude -p hello --mcp-config mcp-hang.json' Do the same for the others: codex mcp add hang -- sleep 60, "command": ["sleep", "60"] in an OpenCode config, "type": "local" for Copilot. The table at the top says roughly what to expect from three of them. If your numbers disagree with the docs, or a run takes the full 60 seconds (Claude Code should move on after about 30), you've found something worth an issue on that CLI's tracker. Ten tasks a day in Hermes IDE I build Hermes IDE, so weigh this section with that in mind. In Hermes, ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch, in its own worktree, with the task as the agent's first prompt. Every agent runs its own interface in a real terminal, "exactly as it would anywhere else" (README). So every task is a fresh start of the vendor's CLI, and it pays the startup you just measured. !The Hermes IDE New task sheet with a refund-endpoint task typed in, chips for Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort, and a preview of the claude command Hermes will run Hermes 2.3.4: the ⌘N New task sheet, with the "Hermes will run" preview of the claude command. That preview is the command to put in hyperfine. Then do the arithmetic with your own table: Run 3 minus the stripped-down run, times the tasks you start in a day. If your repo commits a .mcp.json or an opencode.json, every new worktree has a copy, because it's a tracked file, and every task starts that server again. Hermes runs the same command you would type, so these numbers don't shrink. The saving is elsewhere: on macOS and Linux, agents keep working while Hermes quits, updates or crashes, and you reattach where they left off, so closing the app doesn't make a running agent boot again. !The Hermes IDE agent picker listing Claude Code installed at 2.1.299 and Codex installed at 0.145.0, with Antigravity CLI, Gemini CLI, GitHub Copilot CLI, OpenCode, goose, Hermes Agent, Aider and Kiro CLI marked Not installed, plus Custom agent Hermes 2.3.4: the agent picker in the ⌘N sheet, with each installed agent's version. The picker shows the version Hermes found for each agent, which is the number to paste next to your hyperfine table. Hermes is free to download, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It isn't affiliated with Nous Research, Anthropic, OpenAI, GitHub or any other agent vendor. Download it or read the source on GitHub. Post your four rows in the comments, with versions and OS. I'd like to see how far apart they land on different machines. --- If this was useful Everything Run 3 adds before your prompt is context you chose: an MCP server's tool list, an AGENTS.md, a memory file. Deciding what belongs in that prefix, and what can wait until the agent asks for it, is a big part of the Prompt Engineering Pocket Guide. If you run several agents a day, Hermes IDE puts them side by side, each in its own worktree. 

Free GitHub tool: measure command speed with hyperfine Want to know whether a command is actually faster, rather than judging by feel? Hyperfine runs it repeatedly and reports timing statistics. On a Mac with Homebrew, install it with `brew install hyperfine`. Try `hyperfine 'sleep 0.3'` to see the output, then benchmark two real commands that do the same job. Mind warm caches and other work running on your computer when comparing results. It runs locally, is free and open source, and needs no account or paid API. Official repo and setup: sharkdp/hyperfine

Arguments about which command is faster end quickly with hyperfine. It warms up, runs each one a bunch of times and shows the spread, so one lucky run doesn't get to win. sharkdp/hyperfine