BurntSushi

How Much Memory Does a CI Build Need? Five Real Builds at 2, 4 and 8 GB --- title: How Much Memory Does a CI Build Need? Five Real Builds at 2, 4 and 8 GB published: true date: 2026-11-02 18:00:00 UTC tags: CICD,Memory,cgroups,Docker canonical_url: devops-daily.com --- We ran five real open-source builds (a Vite app build, a TypeScript type check, a Gradle build, a Go build and a Docker image build) inside 2, 4 and 8 GB memory limits on one 4 vCPU machine, with Gradle both with and without its daemon, two or three times at each limit. Four of the five finished inside 2 GB. The one that did not was the Excalidraw web app: its Vite production build also runs a full TypeScript check, together they held 3,248 MiB of anonymous memory, and the kernel killed the build at 2 GB in all 3 runs. The number that most tools show you was the wrong one for the other four. cgroup v2's memory.peak said the GitHub CLI's Go build peaked at 1,775 MiB, but only 424 MiB of that was anonymous memory, the heaps and stacks of its processes. Most of the rest was page cache, which the kernel drops when it needs the room, plus some kernel memory. A Docker image build looked like 2,072 MiB for the same reason. And a TypeScript type check that held 1,139 MiB with the whole machine available finished inside 2 GB with 970 MiB, because Node sizes its heap limit from the cgroup limit it finds. This post shows the numbers, the five traps behind them, and every script we used, so you can size a runner from your own builds instead of from a guess. TL;DR - Four of five builds fit in 2 GB. Only the Excalidraw Vite build needed more: 3,248 MiB of anonymous memory, killed at 2 GB, fine at 4 GB. - memory.peak counts page cache. The Go build and the Docker build showed 1,775 and 2,072 MiB, but their anonymous memory peaked at 424 and 355 MiB. Size from anonymous memory plus a margin, then confirm by running the build inside that limit. - Runtimes size their heaps from the cgroup limit. Inside 2 GB, Node's default heap limit was 1,120 MiB and the JVM's default maximum heap was 512 MiB. The type check used less memory at 2 GB than at 8 GB and still finished, about 24% slower. - The Gradle daemon keeps its memory after the build. Five seconds after gradlew exited, the daemon still held 1,048 MiB of anonymous memory. With --no-daemon, that dropped to under 1 MiB, and in our one-command runs the daemon gave no repeatable speed-up. - Limiting the docker CLI did not limit the build. The CLI's cgroup peaked at 40 MiB while the BuildKit container that did the work peaked at 2,072 MiB. - Wall times on a shared-CPU droplet are noisy: the CPUs reported 23.5% steal time across the runs we tracked. Trust the memory columns more than the time columns. Prerequisites - A Linux machine with cgroup v2 and systemd (any recent Ubuntu, Debian or Fedora). Check with stat -fc %T /sys/fs/cgroup, which prints cgroup2fs. - Root access, for systemd-run --scope and for dropping the page cache. - The toolchains for the builds you want to test. Our setup script installs Go, Node, Yarn, a JDK, Rust and Docker. - If you want to rerun our matrix as it is: a 4 vCPU, 8 GB machine and about two hours. We used a droplet on DigitalOcean (referral link). How we measured We used one DigitalOcean droplet, size s-4vcpu-8gb in fra1, with Ubuntu 24.04 and kernel 6.8. The kernel reports 7,941 MiB of memory and there is no swap. Every build ran in its own transient systemd scope: systemd-run --scope --slice=bench.slice --unit=p1-<build>-<limit>-<run> \ -p MemoryMax=2G -p MemorySwapMax=0 -p AllowedCPUs=0-3 -p OOMPolicy=continue \ /root/bench/wrap.sh <outdir> <timeout> <build command> These are cgroup limits on one droplet, not three real machine sizes. A real 2 GB runner has a smaller kernel footprint, a runner agent and often a Docker daemon in the same memory, so it has less room than our 2 GB scope. The 8 GB limit is above the 7.9 GiB that the droplet has, so the 8 GB column means "the whole machine". The rest of the setup: - CPU. Every scope was pinned to the same 4 vCPUs with AllowedCPUs=0-3, and nproc printed 4 inside each one. Compilers start about one job per CPU, so a 2 vCPU runner would use less memory and an 8 vCPU runner more. We did not measure other CPU counts. - Cold build, warm dependencies. We downloaded dependencies once before the runs: yarn install, go mod download, cargo fetch, the Gradle distribution and dependencies, and the base images for the Docker build. Before every run we deleted the build output and the compiler caches (excalidraw-app/build, GOCACHE, target/, Gradle's build/ and .gradle/), stopped any Gradle daemon, and dropped the page cache. That is a CI job that restores a dependency cache but not a build cache. - What we recorded. Wall time, the exit code, memory.peak at the end of the run, memory.events (how often the limit was hit, and OOM kills), and a sampler that read memory.stat every 100 ms for anonymous memory (anon) and page cache (file). - Repeats. Runs went in round-robin order: run 1 of every cell, then run 2, then run 3, so a slow period on the shared host does not land on one build. Builds ran slower than we planned, so to stay inside our droplet budget the third round covered only the 8 GB and 2 GB limits. That gives 3 runs per cell at 8 GB and 2 GB, and 2 at 4 GB. The tables show the median of 3, or the mean of 2 at 4 GB. Each build also had one warm-up run at 8 GB that we did not count; it downloaded Gradle and filled the image mirror. The builds, all pinned to exact commits: | Build | Command | Source | | --- | --- | --- | | Excalidraw app, Vite production build | yarn build:app:docker | excalidraw/excalidraw 8bcdca45 | | Excalidraw type check, 1,508 files | tsc -p tsconfig.json (noEmit is set) | same checkout | | PlantUML, Gradle 9.3.1 on JDK 21 | ./gradlew jar, daemon on and --no-daemon | plantuml/plantuml db06d70e | | GitHub CLI, Go 1.27.3 | go build ./cmd/gh | cli/cli 9490d204 | | Online Boutique frontend image, BuildKit 0.34.1 | docker buildx build | GoogleCloudPlatform/microservices-demo c7cb6d5b | | ripgrep 15.2.0, Rust 1.99.1 (one run) | cargo build --release --locked | BurntSushi/ripgrep tag 15.2.0 | Node was 24.22.2 with Yarn 1.22.22, Docker was 29.9.2 with Buildx 0.38.1. The results | Build | Anonymous peak at 8 GB | memory.peak at 8 GB | 8 GB | 4 GB | 2 GB | | --- | --- | --- | --- | --- | --- | | Excalidraw vite build | 3,248 MiB | 3,578 MiB | ok, 66 s | ok, 83 s | 3 of 3 OOM-killed (run 1 from the kernel log) | | Excalidraw tsc type check | 1,139 MiB | 1,271 MiB | ok, 59 s | ok, 62 s | ok, 73 s | | PlantUML Gradle, daemon | 1,143 MiB | 1,474 MiB | ok, 124 s | ok, 110 s | ok, 113 s | | PlantUML Gradle, --no-daemon | 1,144 MiB | 1,475 MiB | ok, 88 s | ok, 119 s | ok, 100 s | | GitHub CLI go build | 424 MiB | 1,775 MiB | ok, 141 s | ok, 198 s | ok, 169 s | | Online Boutique frontend, docker buildx build | 355 MiB | 2,072 MiB | ok, 131 s | ok, 129 s | ok, 110 s | Times are medians of 3 runs at 8 GB and 2 GB, and the mean of 2 runs at 4 GB. Memory columns are from the 8 GB runs. What the cgroup counter says vs what the build held (8 GB limit, median of 3) | | Value | Series | | --- | --- | --- | | Vite build | 3578 MiB | memory.peak | | Vite build | 3248 MiB | anonymous memory | | tsc check | 1271 MiB | memory.peak | | tsc check | 1139 MiB | anonymous memory | | Gradle (daemon) | 1474 MiB | memory.peak | | Gradle (daemon) | 1143 MiB | anonymous memory | | Gradle (no daemon) | 1475 MiB | memory.peak | | Gradle (no daemon) | 1144 MiB | anonymous memory | | Go build | 1775 MiB | memory.peak | | Go build | 424 MiB | anonymous memory | | Docker build | 2072 MiB | memory.peak | | Docker build | 355 MiB | anonymous memory | _memory.peak includes page cache; anonymous memory is the sampled peak of anon in memory.stat (100 ms). One s-4vcpu-8gb droplet, 4 vCPUs, 2 November 2026._ Three things stand out. Only one build reached the 2 GB limit at all. The max counter in memory.events, which counts how often a cgroup's usage was about to go over its limit, stayed at 0 for every other run. Memory use at 4 GB and at 8 GB was almost the same for every build, so the extra 4 GB bought nothing. And the gap between memory.peak and anonymous memory is large for exactly the builds that read and write many files: the Go compiler and BuildKit. The single Rust build, ripgrep 15.2.0 with cargo build --release, fits the same pattern: 896 MiB memory.peak, 572 MiB anonymous, 61.1 s. We ran it once at 8 GB only, because the full matrix did not fit our time budget. Trap 1: memory.peak counts page cache memory.peak is the highest value of memory.current that the cgroup reached. memory.current charges the cgroup for everything it caused to be in memory, and that includes page cache : the source files the compiler read, the object files it wrote, the layers BuildKit unpacked. Page cache is reclaimable. When the cgroup gets close to its limit, the kernel drops clean cache pages and writes back dirty ones before it ever considers killing a process. Anonymous memory is the part that cannot go anywhere without swap: heaps, stacks, the compiler's working data. memory.stat reports it as anon. That, plus a smaller amount of kernel memory (37 to 136 MiB in these builds), is what has to fit. Our sampler read it every 100 ms, so it can miss a shorter spike. memory.peak and the max counter cannot miss one, and they show that no build except Vite at 2 GB ever reached its limit. The safe test is still the one we ran: the build inside the limit you plan to buy. The Go build of the GitHub CLI shows the difference well. Its peak was 1,775 MiB. Anonymous memory never went over 424 MiB; about 1,400 MiB was page cache from the module sources and the build cache that go build writes. At the 2 GB limit, the same build finished with the same peak, because 1,775 MiB is still under 2,048 MiB. With a smaller limit the kernel may reclaim that cache instead of failing the build, but we did not test below 2 GB, so test the lower limit before you rely on it. So if a dashboard shows that your CI job "peaked at 1.8 GB", that does not mean a 2 GB runner is too small. Look at anon in memory.stat (or RSS in your container metrics), and then add a margin for the runner itself. Trap 2: a production build that is also a type check The Excalidraw build needed far more memory than anything else: 3,248 MiB of anonymous memory, against 1,139 MiB for a full type check of the same code. The reason is in its Vite config: checker({ typescript: true, eslint: envVars.VITE_APP_ENABLE_ESLINT === "false" ? undefined : { lintCommand: 'eslint "./**/*.{js,ts,tsx}"' }, // ... }), vite-plugin-checker runs the TypeScript checker during vite build too: in version 0.7.2, which Excalidraw pins, enableBuild defaults to true. The production .env turns ESLint off but leaves the type check on, and the build also writes source maps. So one vite build is Rollup, source maps and a full tsc run at the same time. At 2 GB the kernel killed it in all 3 runs, 69 to 122 seconds in, after the log said ✓ 1442 modules transformed. The kernel log names the cgroup and the process it chose: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=p1-vite-2G-r2.scope,mems_allowed=0,oom_memcg=/bench.slice/p1-vite-2G-r2.scope,task_memcg=/bench.slice/p1-vite-2G-r2.scope,task=MainThread,pid=77242,uid=0 Memory cgroup out of memory: Killed process 77242 (MainThread) total-vm:11451308kB, anon-rss:1138988kB, file-rss:4864kB, shmem-rss:0kB, UID:0 pgtables:29992kB oom_score_adj:0 Our wrapper's record of that run shows the kill. memory.events counted 39,825 times that the cgroup was about to go over its limit, and one OOM kill: vite build, 2 GB limit, run 2 $ cat /root/res/p1/vite/2G/r2/result.env rc=1 t0=1793637034.083018067 t1=1793637156.285208072 peak=2147483648 current_after=867889152 anon_after=801075200 file_after=38674432 $ cat /root/res/p1/vite/2G/r2/memory.events low 0 high 0 max 39825 oom 1 oom_kill 1 oom_group_kill 0 The process list right after the kill shows the second half of the build. The checker runs tsc as its own child process, and it was still running after Vite had died: 77265 1808 /bin/sh -c tsc --noEmit 77266 799108 node /root/src/excalidraw/node_modules/.bin/tsc --noEmit That tsc held 799,108 KiB (780 MiB) of resident memory under the 2 GB limit. Yarn reported Command failed with exit code 1, but the type check kept going until our harness stopped the scope. On a real runner it would keep using memory and CPU until the job ends. We did not split the 3,248 MiB peak at 8 GB between Rollup and the checker, so we cannot say exactly how much the type check adds. What we can say: if your bundler runs a checker, your build step needs the memory of both, and moving the type check to its own CI job is the first thing to try before you buy a bigger runner. Trap 3: runtimes size their heaps from the limit Node and the JVM both read the cgroup limit when they start and size their default heap from it. Our harness asks each runtime what it would use inside each limit: probe inside a 2 GB scope # what Node 24 and JDK 21 pick inside MemoryMax=2G (from p1.sh probe) $ cat /root/res/p1/probe-2G.txt nproc=4 memory.max=2147483648 node_heap_size_limit=1174405120 int ActiveProcessorCount = -1 {product} {default} size_t MaxHeapSize = 536870912 {product} {ergonomic} double MaxRAMPercentage = 25.000000 {product} {default} All four places side by side: | Inside | Node 24 heap_size_limit | JDK 21 default MaxHeapSize | | --- | --- | --- | | 2 GB scope | 1,120 MiB | 512 MiB | | 4 GB scope | 2,240 MiB | 1,024 MiB | | 8 GB scope | 2,240 MiB | 2,048 MiB | | whole droplet, no scope | 2,240 MiB | 1,986 MiB | Two consequences showed up in the runs. Heap use follows the limit. tsc --extendedDiagnostics prints Memory used, which is Node's heapUsed at the end of the run, in units of 1,000 bytes. With the whole machine, the type check of 1,508 files reported Memory used: 1018378K (median of 3, about 971 MiB). Inside 2 GB it reported 886903K (about 846 MiB), anonymous memory fell from 1,139 to 970 MiB, and the check finished about 24% slower (median 72.7 s against 58.8 s). The 2 GB run was 9.5 to 14.2 seconds slower in each of the three rounds. In rounds 2 and 3, where we logged the CPU counters, its steal time was only 4 to 7 points higher, so steal explains only part of the gap. More frequent garbage collection under the smaller heap limit would explain the rest, but we did not record GC activity, so that is a hypothesis. A big heap number at a high limit says little about what the program needs. We expected this check to fail at 2 GB with a heap error, and it did not. Node stopped at 2,240 MiB even with more memory. On our 8 GB machine, Node's default heap limit was 2,240 MiB with a 4 GB scope, an 8 GB scope and no scope at all. We did not test a bigger machine, so check what your runner gives you with node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)'. If a build reports JavaScript heap out of memory, that is Node's own limit, not the kernel's. Set NODE_OPTIONS=--max-old-space-size=<MiB> explicitly instead of only buying more memory. The JVM takes 25% of the limit by default (MaxRAMPercentage=25), so a Maven build or a test JVM without -Xmx gets 512 MiB inside 2 GB. Gradle sets its own daemon heap, which brings us to the next trap. Trap 4: the Gradle daemon outlives the build PlantUML's gradle.properties sets org.gradle.jvmargs=-Xmx1g. That cap kept the build inside 2 GB: anonymous memory peaked at about 1,143 MiB with the daemon and 1,144 MiB without it, and neither came near the limit. The difference is what happens after gradlew exits. With the daemon, which is the default, the JVM stays alive and keeps its heap. Five seconds after the build, the scope still held 1,378 MiB, 1,048 MiB of it anonymous, in one process: 37292 1105976 /usr/lib/jvm/java-21-openjdk-amd64/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED ... With --no-daemon, Gradle printed To honour the JVM settings for this build a single-use Daemon process will be forked and Daemon will be stopped at the end of the build, and five seconds after exit the anonymous memory in the scope was under 1 MiB. Anonymous memory around the end of the Gradle build (8 GB limit, run 1) | | -55 s | -50 s | -45 s | -40 s | -35 s | -30 s | -25 s | -20 s | -15 s | -10 s | -5 s | 0 s | +5 s | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | daemon (default) | 637 MiB | 857 MiB | 927 MiB | 982 MiB | 1053 MiB | 1059 MiB | 1066 MiB | 1073 MiB | 1082 MiB | 1110 MiB | 1115 MiB | 1143 MiB | 1054 MiB | | --no-daemon | 476 MiB | 516 MiB | 559 MiB | 807 MiB | 921 MiB | 970 MiB | 1067 MiB | 1072 MiB | 1080 MiB | 1084 MiB | 1112 MiB | 1144 MiB | 0 MiB | _Max of 100 ms samples in each 5 s bin, labelled by the end of the bin; 0 s is the moment gradlew exits, +5 s is the 5 s after. One run each, PlantUML jar task, JDK 21, Gradle 9.3.1._ On a CI runner, that leftover daemon sits in the same memory as every later step of the job: the Docker build, the integration tests, the upload. At 8 GB the median time was 123.5 s with the daemon and 87.5 s without (112.7 s and 99.7 s at 2 GB), The ranges overlap and CPU steal was high, so we see no repeatable speed-up from the daemon for one Gradle command per job. The daemon is built to pay off across several Gradle commands in the same job, which we did not test. If a job runs one Gradle command, use --no-daemon (or org.gradle.daemon=false). If it runs several, consider keeping the daemon (we did not test that case), and run ./gradlew --stop before the steps that need the memory. Trap 5: a limit on the docker CLI does not limit the build docker build is a client. In our setup the work ran in a BuildKit container (Buildx's docker-container driver), which has its own cgroup. Other drivers and CI executors lay out cgroups differently, but the CLI is only a client in all of them. We wrapped the docker buildx build command in the same kind of scope as everything else, and at every limit that scope peaked at 39 to 42 MiB: the CLI, streaming logs. To measure the build, we put the limit on the BuildKit container itself: docker buildx create --name p1d8gr1 --driver docker-container \ --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \ --driver-opt memory=8g --driver-opt memory-swap=8g --driver-opt cpuset-cpus=0-3 \ --buildkitd-config buildkitd.toml --bootstrap and read memory.peak from /sys/fs/cgroup/system.slice/docker-<container id>.scope. That container peaked at 2,072 MiB, of which only 355 MiB was anonymous: the Go compiler inside the build. Most of the rest was page cache from the image layers and the build's output, plus 136 MiB of kernel memory. This matters for runner sizing in two ways. Your CI's memory graph may not show the build at all, depending on whether it measures the job's process tree or the whole machine. And if a self-hosted runner limits only the job's process tree, a Docker build in that job can run outside the limit, so a "2 GB job" can push the host far past 2 GB. Check which cgroup your executor puts the Docker daemon in. Two method bugs that cost us a run The first 2 GB Vite run left no result at all. The kernel killed the Vite process, and then systemd stopped the whole scope, including our wrapper script, before it could write anything. The journal said so: p1-vite-2G-r1.scope: A process of this unit has been killed by the OOM killer. p1-vite-2G-r1.scope: Failed with result 'oom-kill'. systemd 255 sets DefaultOOMPolicy=stop, and scopes follow it. We set DefaultOOMPolicy=continue before the next 2 GB run (the published p1.sh also passes -p OOMPolicy=continue), and checked with a 64 MiB scope that the wrapper now survives a kill and records oom_kill 1. Runs 2 and 3 at 2 GB have full records. Then our rerun of run 1 failed too, for a different reason. systemd keeps a failed transient unit loaded, so starting a new scope with the same name was refused: Failed to start transient scope unit: Unit p1-vite-2G-r1.scope was already loaded or has a fragment file. By the time we saw that, the Docker cache tests for the next post were running on the same droplet, so we did not try again. Run 1 still counts as a kill: its kernel log line reads oom_memcg=/bench.slice/p1-vite-2G-r1.scope,...,task=MainThread,pid=36041, and our sampler, which runs outside the scope, saw the cgroup reach 2,048 MiB, with up to 1,985 MiB anonymous, before its last sample at 70.5 seconds. The published redo mode now runs systemctl reset-failed first. If you measure builds this way, set OOMPolicy=continue and give each run a unique unit name, or the runs that you care about most, the ones that fail, will be the ones with no data. A sizing table From these runs, for a 4 vCPU runner. The "smallest limit that passed" column is measured. The last column is our rule of thumb, not a measurement: a real VM also holds the OS, the runner agent and any service containers, so we pick the smallest size that leaves at least 1 GB above the anonymous peak. | Build like ours | Anonymous peak | Smallest limit that passed every run | Runner we would pick | | --- | --- | --- | --- | | Bundler that also type-checks (Excalidraw vite build) | 3,248 MiB | 4 GB | 8 GB (moving the type check to its own job may make 4 GB enough; we did not test that) | | Large type check on its own (tsc, 1,508 files) | 1,139 MiB at 8 GB, 970 MiB at 2 GB | 2 GB | 4 GB | | Gradle with a 1 GB daemon heap (PlantUML jar) | 1,143 MiB | 2 GB | 4 GB, with --no-daemon | | Go build of a mid-size CLI (gh) | 424 MiB | 2 GB | 2 GB | | Docker build of a Go service | 355 MiB inside BuildKit | 2 GB | 2 GB, plus what the Docker daemon and the other steps need | Hosted CI services list their machines by vCPU and memory, so you can match a build to a size. Bitrise, for example, lists every Linux machine type on its pricing page with its vCPU count and memory, such as 2 vCPU with 8 GB, and also sells a remote build cache for Gradle, Bazel and Xcode (checked on its public pages on 2 November 2026). Whatever you use, measure anonymous memory first, then pick the size. What we could not conclude - Three limits on one machine are not three machines. A real 2 GB runner has less page cache to spare, its own kernel, and a runner agent in the same memory. Leave a margin above the anonymous peak; our sizing table uses 1 GB. - CPU count changes memory. Every run had 4 vCPUs. The Go compiler, Rollup's workers and Gradle's workers scale with CPUs, so these numbers are not valid for 2 or 16 vCPUs. - Wall times are noisy. The droplet reported 23.5% CPU steal time across the 33 runs that our CPU counter log covered, and between 3.7% and 35.5% in single runs. The Go build took between 121.3 and 199.6 seconds at limits it never came close to. We do not read anything into a time difference between limits unless it repeats in every round, as the tsc gap did. - Two or three runs per cell is enough to see a failure repeat and a peak hold steady, and that is all. The anonymous peaks varied by up to 21% for the Go build (353 to 441 MiB at 8 GB) and up to 15% for Gradle, and 6% or less for Vite, tsc and Docker between runs of the same cell. - We did not split shared builds. We cannot say how much of the Vite build's memory is Rollup and how much is the checker, or how much of the Docker build's page cache is layer extraction. - Rust ran once. One ripgrep build at 8 GB, so no repeats and no 2 GB or 4 GB data. Run it yourself Create a 4 vCPU, 8 GB Ubuntu 24.04 machine, copy the files below to /root/bench/, and run: bash /root/bench/setup.sh bash /root/bench/p1.sh warmup # probes the runtimes, one unmeasured run per build bash /root/bench/p1.sh matrix # 3 rounds x 3 limits x each build (we stopped it after round 2) bash /root/bench/p1.sh round3 # what we ran as round 3: 8 GB and 2 GB only bash /root/bench/p1.sh redo # reruns any cell that left no result python3 summarize_p1.py /root/res/p1 Each run writes its own directory with result.env, memory.events, samples.json and the build log, so you can read any single run on its own. #setup.sh: toolchains, pinned sources, dependencies, registries #!/bin/bash # Droplet setup: toolchains, pinned sources, dependency prefetch, local registries. set -euxo pipefail export DEBIAN_FRONTEND=noninteractive mkdir -p /root/src /root/res cloud-init status --wait || true # first boot runs apt; wait for its lock fetch_sha() { git init -q "$1"; git -C "$1" remote add origin "$2"; git -C "$1" fetch -q --depth 1 origin "$3"; git -C "$1" checkout -q FETCH_HEAD; } fetch_sha /root/src/excalidraw https://github.com/excalidraw/excalidraw 8bcdca4526867433f99fdfcb55a063668db0c9dc fetch_sha /root/src/plantuml https://github.com/plantuml/plantuml db06d70e18e71839dd5e330ac4086f3845371e3b fetch_sha /root/src/cli https://github.com/cli/cli 9490d204a2ba95434153dedce99898ce9f281426 git clone -q --depth 1 --branch 15.2.0 https://github.com/BurntSushi/ripgrep /root/src/ripgrep fetch_sha /root/src/msd https://github.com/GoogleCloudPlatform/microservices-demo c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712 curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz curl -fsSL https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.xz | tar -C /usr/local --strip-components=1 -xJ curl -fsSL https://sh.rustup.rs | sh -s -- -y -q --profile minimal --default-toolchain stable apt-get update -q apt-get install -y -q openjdk-21-jdk-headless build-essential git jq zstd python3 ca-certificates curl curl -fsSL https://get.docker.com | sh export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local npm install -g yarn@1.22.22 docker pull -q moby/buildkit:v0.34.1 docker pull -q registry:2.8.3 docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3 docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3 docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3 # dependencies are downloaded once, outside the measured runs (cd /root/src/excalidraw && yarn install --frozen-lockfile --network-timeout 600000) (cd /root/src/cli && go mod download) (cd /root/src/ripgrep && cargo fetch --locked) # systemd stops a scope after an OOM kill by default; keep the wrapper alive to record the result mkdir -p /etc/systemd/system.conf.d printf '[Manager]\nDefaultOOMPolicy=continue\n' > /etc/systemd/system.conf.d/oom.conf systemctl daemon-reexec #wrap.sh: runs inside the scope and records the result #!/bin/bash # Runs inside the systemd scope. usage: wrap.sh OUTDIR TIMEOUT_S cmd... out=$1; tmo=$2; shift 2 cg=/sys/fs/cgroup$(sed -n 's/^0:://p' /proc/self/cgroup) echo "$cg" > "$out/cgroup" cat "$cg/memory.max" > "$out/memory.max" t0=$(date +%s.%N) timeout -k 15 "$tmo" "$@" > "$out/build.log" 2>&1 rc=$? t1=$(date +%s.%N) { echo "rc=$rc" echo "t0=$t0" echo "t1=$t1" echo "peak=$(cat "$cg/memory.peak")" echo "current_after=$(cat "$cg/memory.current")" awk '$1=="anon"||$1=="file"{print $1"_after="$2}' "$cg/memory.stat" } > "$out/result.env" cp "$cg/memory.events" "$out/memory.events" for p in $(cat "$cg/cgroup.procs"); do ["$p" = "$$"] && continue ps -o pid=,rss=,args= -p "$p" | cut -c1-400 done > "$out/leftover.txt" 2>/dev/null if [-n "${LEFTOVER_WAIT:-}"]; then sleep "$LEFTOVER_WAIT" echo "current_later=$(cat "$cg/memory.current")" >> "$out/result.env" awk '$1=="anon"{print "anon_later="$2}' "$cg/memory.stat" >> "$out/result.env" for p in $(cat "$cg/cgroup.procs"); do ["$p" = "$$"] && continue ps -o pid=,rss=,args= -p "$p" | cut -c1-400 done > "$out/leftover_later.txt" 2>/dev/null fi touch "$out/done" exit $rc #sample.py: polls memory.stat every 100 ms from outside the scope #!/usr/bin/env python3 """Poll one cgroup v2 directory every 100 ms until a done file appears. usage: sample.py CGROUP_DIR OUT_JSON DONE_FILE Records the maximum of anon, file, kernel and memory.current, plus a series. """ import json, os, sys, time cg, out, done = sys.argv[1:4] start = time.time() while not os.path.isdir(cg): if time.time() - start > 60 or os.path.exists(done): json.dump({"error": "cgroup never appeared"}, open(out, "w")) sys.exit(0) time.sleep(0.02) keys = ("anon", "file", "kernel", "shmem") mx = {k: 0 for k in keys} mx["current"] = 0 series = [] t0 = time.time() n = 0 while not os.path.exists(done): try: with open(os.path.join(cg, "memory.stat")) as f: st = {} for line in f: k, v = line.split() if k in keys: st[k] = int(v) with open(os.path.join(cg, "memory.current")) as f: cur = int(f.read()) except OSError: break n += 1 for k in keys: mx[k] = max(mx[k], st.get(k, 0)) mx["current"] = max(mx["current"], cur) series.append([round(time.time() - t0, 2), st.get("anon", 0), st.get("file", 0), cur]) time.sleep(0.1) json.dump({"max": mx, "samples": n, "interval_s": 0.1, "series": series}, open(out, "w")) #p1-cmds.sh: the builds, and what is deleted before each run #!/bin/bash # Build definitions for post 1. prep_* runs outside the cgroup, cmd_* inside it. export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local EX=/root/src/excalidraw PU=/root/src/plantuml GH=/root/src/cli RG=/root/src/ripgrep FRONTEND=/root/src/msd/src/frontend stop_gradle() { pkill -f GradleDaemon || true; pkill -f GradleWrapperMain || true; pkill -f KotlinCompileDaemon || true; sleep 1; } prep_vite() { rm -rf $EX/excalidraw-app/build $EX/node_modules/.vite $EX/excalidraw-app/node_modules/.vite; } cmd_vite() { cd $EX && yarn build:app:docker; } prep_tsc() { find $EX -maxdepth 2 -name '*.tsbuildinfo' -delete; } cmd_tsc() { cd $EX && node_modules/.bin/tsc -p tsconfig.json --extendedDiagnostics; } prep_gradle_daemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; } cmd_gradle_daemon() { cd $PU && ./gradlew jar --console=plain --no-build-cache; } prep_gradle_nodaemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; } cmd_gradle_nodaemon() { cd $PU && ./gradlew --no-daemon jar --console=plain --no-build-cache; } prep_go() { rm -rf /root/gocache /root/out; mkdir -p /root/out; } cmd_go() { cd $GH && GOCACHE=/root/gocache go build -o /root/out/gh ./cmd/gh; } prep_rust() { rm -rf $RG/target; } cmd_rust() { cd $RG && cargo build --release --locked; } prep_docker() { docker buildx ls --format '{{.Name}}' 2>/dev/null | grep '^p1d' | xargs -r -n1 docker buildx rm >/dev/null 2>&1 || true; } if ["${BASH_SOURCE[0]}" = "$0" ]; then "$@"; fi #p1.sh: probe, warm-up, matrix, round 3, redo, and the docker build cell #!/bin/bash # Post 1 matrix. usage: p1.sh warmup | p1.sh matrix set -u B=/root/bench R=/root/res/p1 mkdir -p $R source $B/p1-cmds.sh LIMITS=(8G 4G 2G) BUILDS=(vite tsc gradle_daemon gradle_nodaemon go rust docker) # rust ran once (warmup) only: the matrix did not fit the droplet time budget MATRIX_BUILDS=(vite tsc gradle_daemon gradle_nodaemon go docker) CPUS=0-3 log() { echo "$(date -u +%FT%TZ) $*" | tee -a $R/progress.log; } scope_run() { # unit limit out tmo cmd... local unit=$1 lim=$2 out=$3 tmo=$4; shift 4 python3 $B/sample.py /sys/fs/cgroup/bench.slice/$unit.scope $out/samples.json $out/done & local sp=$! systemd-run --scope --quiet --slice=bench.slice --unit=$unit \ -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS -p OOMPolicy=continue \ $B/wrap.sh $out $tmo "$@" wait $sp systemctl stop $unit.scope 2>/dev/null || true } docker_cell() { # lim out tmo tag local lim=$1 out=$2 tmo=$3 tag=$4 local name=$(echo p1d$tag | tr A-Z a-z) m=$(echo $1 | tr G g) docker buildx create --name $name --driver docker-container \ --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \ --driver-opt memory=$m --driver-opt memory-swap=$m --driver-opt cpuset-cpus=$CPUS \ --buildkitd-config $B/buildkitd.toml --bootstrap > $out/create.log 2>&1 local cid; cid=$(docker inspect -f '{{.Id}}' buildx_buildkit_${name}0) local cg=/sys/fs/cgroup/system.slice/docker-$cid.scope cat $cg/memory.max > $out/container_memory.max mkdir -p $out/cli python3 $B/sample.py $cg $out/container_samples.json $out/cli/done & local sp=$! scope_run p1-docker-$tag $lim $out/cli $tmo \ docker buildx build --builder $name --progress=plain --provenance=false --output type=cacheonly $FRONTEND wait $sp { echo "container_peak=$(cat $cg/memory.peak)"; cat $out/cli/result.env; } > $out/result.env cp $cg/memory.events $out/container_memory.events cp $out/cli/build.log $out/build.log docker buildx rm $name > /dev/null 2>&1 } run_cell() { # build limit run tmo local b=$1 lim=$2 r=$3 tmo=$4 local tag=$b-$lim-$r out=$R/$b/$lim/$r rm -rf $out; mkdir -p $out prep_$b sync; echo 3 > /proc/sys/vm/drop_caches; sleep 2 if [$b = docker]; then docker_cell $lim $out $tmo ${lim}$r else case $b in gradle_*) export LEFTOVER_WAIT=5 ;; esac scope_run p1-$tag $lim $out $tmo bash $B/p1-cmds.sh cmd_$b unset LEFTOVER_WAIT fi [$b = gradle_daemon] || [$b = gradle_nodaemon] && stop_gradle log "$tag $(tr '\n' ' ' < $out/result.env)" } probe() { # record what the runtimes see inside each limit for lim in "${LIMITS[@]}"; do systemd-run --scope --quiet --slice=bench.slice --unit=probe-$lim -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS \ bash -c 'echo "nproc=$(nproc)"; echo "memory.max=$(cat /sys/fs/cgroup$(sed -n "s/^0:://p" /proc/self/cgroup)/memory.max)"; echo "node_heap_size_limit=$(node -e "console.log(require(\"v8\").getHeapStatistics().heap_size_limit)")"; java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage|ActiveProcessorCount) "' > $R/probe-$lim.txt 2>&1 done { echo "nproc=$(nproc)"; free -b | head -2; echo "node_heap_size_limit=$(node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)')"; java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage) "; } > $R/probe-host.txt 2>&1 } case ${1:-} in warmup) probe for b in "${BUILDS[@]}"; do run_cell $b 8G r0 900; done log WARMUP-DONE ;; matrix) declare -A TMO for b in "${MATRIX_BUILDS[@]}"; do . <(grep -E '^t[01]=' $R/$b/8G/r0/result.env) w=$(python3 -c "print(int(float('$t1')-float('$t0')))") t=$((3 * w + 60)); [$t -gt 600] && t=600; [$t -lt 120] && t=120 TMO[$b]=$t done log "timeouts: $(for b in "${MATRIX_BUILDS[@]}"; do printf '%s=%s ' $b ${TMO[$b]}; done)" for r in r1 r2 r3; do for b in "${MATRIX_BUILDS[@]}"; do for lim in "${LIMITS[@]}"; do run_cell $b $lim $r ${TMO[$b]}; done done done log MATRIX-DONE ;; round3) # third run at 8G and 2G only: the full round did not fit the droplet time budget declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351) log "round 3 at 8G and 2G only" for b in "${MATRIX_BUILDS[@]}"; do for lim in 8G 2G; do run_cell $b $lim r3 ${TMO[$b]}; done done log MATRIX-DONE ;; redo) # rerun cells that left no result (systemd stopped the scope after an OOM kill) declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351) for b in "${MATRIX_BUILDS[@]}"; do for lim in "${LIMITS[@]}"; do for r in r1 r2 r3; do d=$R/$b/$lim/$r if [-d $d] && ! grep -q '^rc=' $d/result.env 2>/dev/null; then rm -rf $d.lost; mv $d $d.lost; log "redo $b $lim $r" systemctl reset-failed p1-$b-$lim-$r.scope 2>/dev/null # added after our run: a failed scope blocks its name run_cell $b $lim $r ${TMO[$b]} fi done; done; done log REDO-DONE ;; *) echo "usage: $0 warmup|matrix|round3|redo"; exit 2 ;; esac #buildkitd.toml: send base image pulls to the local mirror [registry."docker.io"] mirrors = ["127.0.0.1:5001"] [registry."gcr.io"] mirrors = ["127.0.0.1:5002"] [registry."127.0.0.1:5000"] http = true insecure = true [registry."127.0.0.1:5001"] http = true insecure = true [registry."127.0.0.1:5002"] http = true insecure = true #summarize_p1.py: medians per build and limit #!/usr/bin/env python3 """Summarize post 1 runs: medians per build and limit over the measured runs (r1..r3).""" import json, os, re, statistics, sys root = sys.argv[1] BUILDS = ["vite", "tsc", "gradle_daemon", "gradle_nodaemon", "go", "rust", "docker"] LIMITS = ["8G", "4G", "2G"] MiB = 2 ** 20 def env(path): d = {} if os.path.exists(path): for line in open(path): if "=" in line: k, v = line.strip().split("=", 1) d[k] = v return d def events(path): d = {} if os.path.exists(path): for line in open(path): k, v = line.split() d[k] = int(v) return d def run_info(d, build): r = env(os.path.join(d, "result.env")) if not r or "rc" not in r: return None log = open(os.path.join(d, "build.log"), errors="replace").read() if os.path.exists(os.path.join(d, "build.log")) else "" if build == "docker": if not r.get("container_peak"): return None # warm-up run 0 recorded no container numbers (builder name case bug) ev = events(os.path.join(d, "container_memory.events")) smp = json.load(open(os.path.join(d, "container_samples.json"))) peak = int(r["container_peak"]) cli_peak = int(r["peak"]) else: ev = events(os.path.join(d, "memory.events")) smp = json.load(open(os.path.join(d, "samples.json"))) peak = int(r["peak"]) cli_peak = None rc = int(r["rc"]) wall = float(r["t1"]) - float(r["t0"]) if rc == 0: outcome = "ok" elif ev.get("oom_kill", 0) > 0: outcome = "oom_kill" elif "heap out of memory" in log or "Reached heap limit" in log: outcome = "node_heap" elif "OutOfMemoryError" in log: outcome = "java_heap" elif rc in (124, 137): outcome = "timeout" else: outcome = f"fail_rc{rc}" mx = smp.get("max", {}) return { "rc": rc, "outcome": outcome, "wall_s": round(wall, 1), "peak_mib": round(peak / MiB), "anon_max_mib": round(mx.get("anon", 0) / MiB), "file_max_mib": round(mx.get("file", 0) / MiB), "kernel_max_mib": round(mx.get("kernel", 0) / MiB), "samples": smp.get("samples"), "oom_kill": ev.get("oom_kill", 0), "max_events": ev.get("max", 0), "high_events": ev.get("high", 0), "after_mib": round(int(r.get("current_after", 0)) / MiB), "anon_after_mib": round(int(r.get("anon_after", 0)) / MiB), "cli_peak_mib": round(cli_peak / MiB) if cli_peak is not None else None, "later_mib": round(int(r["current_later"]) / MiB) if "current_later" in r else None, "anon_later_mib": round(int(r["anon_later"]) / MiB) if "anon_later" in r else None, "leftover_later": open(os.path.join(d, "leftover_later.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover_later.txt")) else None, "leftover": open(os.path.join(d, "leftover.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover.txt")) else "", } def med(xs): xs = [x for x in xs if x is not None] return round(statistics.median(xs), 1) if xs else None out = {} for b in BUILDS: for lim in LIMITS: runs = {} for rn in ("r0", "r1", "r2", "r3"): d = os.path.join(root, b, lim, rn) if os.path.isdir(d): info = run_info(d, b) if info: runs[rn] = info meas = [runs[k] for k in ("r1", "r2", "r3") if k in runs] ok = [x for x in meas if x["outcome"] == "ok"] out[f"{b}/{lim}"] = { "runs": runs, "n": len(meas), "n_ok": len(ok), "outcomes": [x["outcome"] for x in meas], "wall_med_ok": med([x["wall_s"] for x in ok]), "wall_med_all": med([x["wall_s"] for x in meas]), "peak_med": med([x["peak_mib"] for x in meas]), "anon_med": med([x["anon_max_mib"] for x in meas]), "file_med": med([x["file_max_mib"] for x in meas]), "peak_range": [min(x["peak_mib"] for x in meas), max(x["peak_mib"] for x in meas)] if meas else None, "anon_range": [min(x["anon_max_mib"] for x in meas), max(x["anon_max_mib"] for x in meas)] if meas else None, "wall_range": [min(x["wall_s"] for x in meas), max(x["wall_s"] for x in meas)] if meas else None, "after_med": med([x["after_mib"] for x in meas]), "anon_after_med": med([x["anon_after_mib"] for x in meas]), "cli_peak_med": med([x["cli_peak_mib"] for x in meas]), "later_med": med([x["later_mib"] for x in meas]), "anon_later_med": med([x["anon_later_mib"] for x in meas]), "max_events_med": med([x["max_events"] for x in meas]), } json.dump(out, open(os.path.join(root, "summary.json"), "w"), indent=1) print(f"{'cell':26} {'n':>2} {'ok':>2} {'wall_ok':>8} {'peak':>6} {'anon':>6} {'file':>6} {'after':>6} outcomes / ranges") for k, v in out.items(): print(f"{k:26} {v['n']:>2} {v['n_ok']:>2} {str(v['wall_med_ok']):>8} {str(v['peak_med']):>6} {str(v['anon_med']):>6} {str(v['file_med']):>6} {str(v['after_med']):>6} {v['outcomes']} wall{v['wall_range']} anon{v['anon_range']} peak{v['peak_range']} maxev={v['max_events_med']} cli={v['cli_peak_med']} later={v['later_med']}/{v['anon_later_med']}") Summary Most CI builds need less memory than the cgroup counter suggests, because memory.peak includes page cache that the kernel gives back under pressure. Measure anonymous memory, run the build inside the limit you want to buy, and check memory.events for max and oom_kill. Then look for the traps: a bundler that runs a type checker, a runtime that sizes its heap from the limit, a Gradle daemon that stays after the build, and a Docker build that runs outside your cgroup. In our runs, fixing any of those was worth more than doubling the runner's memory. _Originally published at devops-daily.com._

Multi-agent work in three spoonfuls (part IV): from a terminal trident to an autopoietic loop --- title: "Multi-agent work in three spoonfuls (part IV): from a terminal trident to an autopoietic loop" published: true description: "Fourth entry in the penta-agent multi-agent series: quota-governed routing, 70 ms hybrid memory, MCP tools, and a zero-token local autopoiesis loop in Harness2." tags: "ai, agents, cli, automation" canonical_url: "3cucharadas.cl" cover_image: "3cucharadas.cl" ai_disclosure_level: "some_ai" --- After benchmarking the models in part I, auditing retrieval in part II, and tracing provenance in part III of this series, I closed my laptop with a sense of victory: I had a trident of agents running from the terminal on Arch Linux. With a couple of zsh aliases, I could invoke Claude Code for architectural reasoning, hand off a task to Codex CLI to edit code, or ask Gemini CLI and Antigravity for a quick second opinion. On paper, orchestration looked solved. Yet after using that setup every day on real projects inside penta-agent, I hit a silent wall: having three agents talking does not equal having a system that learns. Every time I opened a fresh session, the trident suffered from partial amnesia, repeated commands that had already failed the previous week, burned the quota of the most expensive provider on routine tasks, or marked tests as "verified" when they had never had a chance to fail. To understand what was missing, it helped to change the question. In a recent systems-theory analysis of artificial intelligence, Benedikt Zönnchen, Mariya Dzhimova, and Gudrun Socher (2025[^zonnchen2025], 2–4) propose setting aside whether a language model "thinks like a human" and asking instead the question posed by Humberto Maturana and Francisco Varela (1987[^maturana1987] [1984]) and extended by Niklas Luhmann (1998[^luhmann1998], 60–69): can the system draw its own boundary with the environment, observe its own failures, and sustain itself through its own operations? In other words, moving from the vague notion of "intelligence" to the concrete mechanism of autopoiesis. As Zönnchen, Dzhimova, and Socher (2025[^zonnchen2025], 8–10) show, an isolated language model in a chat window neither is nor can be an autopoietic, sense-making system: it lacks what George Spencer-Brown (1969[^spencerbrown1969]) called re-entry—the capacity to re-introduce the distinction between system and environment (self-reference vs. other-reference) inside its own operations—because it only computes which token is statistically plausible after another, without internal resistance against error. If I want a system that improves without degrading, I have to build a socio-technical harness around the models that couples the probabilistic flexibility of neural networks with the strict coupling and binary code (works / does not work, work / fail) of deterministic technology described by Luhmann (1998[^luhmann1998], 517–525) and André Reichel (2011[^reichel2011], 105–118). The starting point: what I measured and discarded before making this leap Before diving into the three spoonfuls of this fourth installment, here is the experiment that forced me to adopt that discipline. When I tried to give the system long-term memory, the obvious temptation was to install the most elaborate architecture available: temporal graphs, heavy extractors, or an 8B-parameter neural judge running on CPU. Instead of following intuition, I built a reproducible benchmark over my real work logs—comparing open alternatives such as GraphRAG, Graphiti, Hindsight, Letta, Mem0, and deja-vu—and set a falsification contract upfront: any candidate exceeding 0.10 s in p95 latency or failing to improve recall by at least 10% would be discarded. | What I tested in the laboratory | Measured result | Operational decision | |---|---|---| | Hybrid memory (v3.1) (FTS5 lexical index + semantic neighbors in Qdrant with Qwen3-Embedding-0.6B) | Latency p95 = 0.07 s (< 0.10 s), 96% overall recall, rescuing 1 in 3 zero-lexical-overlap queries | Promoted to production as the engine behind experience-memory and recall-context | | In-memory relational graph (v3.2) for multi-hop entity expansion | Exact tie in accuracy with v3.1, but nearly double the latency (p95 = 0.13 s > 0.10 s) | Frozen in the lab; the extra complexity did not earn its keep | | CPU neural judges (4B and 8B) for live answer auditing | Between 69 s and 197 s per verdict, hitting 180 s timeouts in 2 of 3 runs | Discarded from the critical path; pruned 36.3 GiB from disk | | Deterministic checks + bounded 0.6B judge for local validation | Millisecond latency, 0 remote calls, and 100% reproducibility | Adopted as the standard for control gates and evaluation | Table 1. Summary of the prior local memory and evaluation laboratory in penta-agent: what survived the 0.10 s threshold and why I pruned 36.3 GiB of local models that added no operational value. With that foundation in place, I organized this fourth part into three progressive spoonfuls: in the first, I show how I moved from launching standalone agents by hand to governing them with quota budgets, auditable bridges, and a live board in kitty and tmux; in the second, how the nervous system of hybrid memory, MCP tools, and skills works under the hood; and in the third, how I closed the local autopoiesis loop with Harness2 and where both my metrics and systems theory draw the boundary today. --- Spoonful 1: From invoking agents by hand to governing agents and models In plain terms: when I first wired the three agents together, I acted like a traffic officer waving my arms in the middle of an intersection. If I wanted Codex CLI to review a patch written by Claude Code, I copied file paths, opened another tab, and hoped for the best. That works for a five-minute demo, but it falls apart when a task runs for two hours and touches twenty files. Quota-aware routing and security boundaries My first deductive inference came from a hard constraint: model subscriptions and APIs have usage limits, different context windows, and credential boundaries that must never blur. By definition, if I delegate a heavy task to an agent whose quota is at 98%, the session will die mid-flight and leave the repository in an inconsistent state. So I replaced intuitive provider selection with an explicit policy in routing.yaml and a pre-flight sentinel (quota_status.sh, governed by the quota-guard skill). Before launching any expensive job, the system reads cached pressure across every lane without spending tokens: - 🔍 Claude Code (Opus / Sonnet): handles architectural design, delicate editing, and long-horizon synthesis when its quota is clear (exit 0). - 🛠️ Codex CLI (gpt-5.4): steps in as an implementation peer, deep refactoring engine, session rescuer (codex-rescue), and cross-validator. - 🌐 Gemini CLI and Antigravity (gemini-3-pro / flash): in a direct session (penta-agy) it acts as the primary integrator; when invoked from outside via agy-bridge, it runs inside a strict sandbox over sanitized public context as an explorer, reviewer, or validator, never inheriting private keys or .env files. Instead of calling a raw binary and depending on whatever it prints to stdout, every cross-agent handoff goes through agy-bridge. The bridge requires declaring who is calling (--caller), in what mode (plan, review, patch, test, or explain), and with what time limit (--timeout), leaving an immutable receipt in .agents/runs/<run_id>/ with request.md, result.md, status.json, and diff.patch. The live board (plan-board) and decision gates My second inference was inductive: reviewing dozens of past sessions showed that the biggest waste of time happened at two extremes. Either an agent interrupted me every two minutes to ask permission for trivial, reversible steps, or it silently made a delicate architectural choice and I only discovered at the end that it had gone down the wrong path. To eliminate both extremes without filling this post with abstract flowcharts, here is how my actual kitty terminal—or minimalist Wayland alternatives like foot on Codeberg—looks today when split into two panes with tmux during a live task:  Watch the full video demo on 3cucharadas.cl. <strong>Figure 1.</strong> Daily workflow in my <a href="kovidgoyal/kitty"><code>kitty</code></a> + <a href="tmux/tmux"><code>tmux</code></a> terminal with the <a href="starship/starship"><code>Starship</code></a> prompt: the top pane runs multi-agent delegation via <code>agy-bridge</code>, the <code>decision-gate</code> brake, and <code>pytest</code>; the bottom pane (<code>watch.mjs</code>) updates each step live (<code>[ ]</code> &rarr; <code>[~]</code> &rarr; <code>[!]</code> &rarr; <code>[x]</code>), along with binding decisions and drift. What appears in the bottom pane of Figure 1 is the contract that changed how I work: 1. Local autonomy with visual traceability (plan-board): every reversible, in-scope step proceeds without asking permission, but must be reflected in real time in the bottom table. And no item can be closed as verified ([x]) unless the agent attaches the concrete test, command, or observation reference that backs it. 2. Decision gate (decision-gate): when an irreversible or architectural fork appears (state [!]), the agent halts only that branch and presents the question with the recommended option first, marked (Recommended) and justified with deductive, inductive, and abductive evidence. 3. A gate that has never been seen red is not a gate (gates-falsables): before accepting any new verifier, I require running it against a synthetic failure (--fixture) and proving that it returns a non-zero exit code (exit 1). If a check passes even when I break its input on purpose, it is not a safety net—it is compliance theater. --- Spoonful 2: The nervous system — hybrid memory, skills, and MCP tools If the first spoonful orders who does what and under which rules, the second solves how the system remembers what it has already learned and how it picks the exact tool without bloating the context window. Here the diagnosis by Zönnchen, Dzhimova, and Socher (2025[^zonnchen2025], 6–7, 9) is decisive again: a neural network maintains a loose coupling with human communication that behaves like a "contingent index-card system (Zettelkasten)" (pp. 7, 9) of semantic associations in vector space, giving rise to what Elena Esposito (2017[^esposito2017], 249–265; 2022[^esposito2022]) calls "virtual double contingency" and "artificial communication." That flexibility lets it connect ideas phrased in very different words, but also makes its outputs inherently probabilistic and devoid of autonomous decision-making. If I want reliable memory and precise execution, I have to couple that vector Zettelkasten with exact indices and deterministic scripts. Experience memory (experience-memory): the hybrid Zettelkasten in 0.07 s When I first started keeping logs, my memory was essentially a folder of Markdown files that I read or searched with ripgrep. The obvious problem is that if a month ago I solved a cold-boot screen freeze under the title "Webview failure on autostart" and today I search for "Panel fails to load at login", a purely lexical search can miss it entirely, whereas a purely vector search can bury the exact name of a script like verify_boot_contract.sh. The solution I validated in Table 1 combines both worlds behind the experience-memory MCP server and the recall-context skill: - Exact lexical lane (SQLite FTS5): instantly catches flag names, error codes, file paths, and invariants (INV-005). - Dense semantic lane (Qdrant + Qwen3-Embedding-0.6B): indexes architectural decisions, handoffs, and operational lessons (penta_context_v2) to retrieve conceptual neighbors even when they share zero keywords. Before designing any non-trivial strategy, the agent queries recall_experience in 70 milliseconds (p95 = 0.07 s). That is how it discovers which command worked previously, which hypothesis was already refuted, and which environment trap to avoid (for instance, knowing that in my shell cp, mv, and rm are aliased to interactive -i, which hangs a headless process unless invoked via command mv -f). Skills as living procedures and proactive MCP selection To avoid saturating the initial prompt with thousands of lines of instructions, I adopted the modular pattern of Anthropic Skills enriched with local contracts. In penta-agent, each skill is a self-contained package with four parts: a concise contract in SKILL.md defining when to activate and when not to use it; on-demand deep documentation in references/ (such as the editorial corpus extracted with Docling or diagramming rules with D2); deterministic executables in scripts/ (such as typed pipelines in Nushell, visual automation with Playwright, or forge integration via glab on GitLab); and a telemetry trail. Similarly, the fleet of MCP servers (plan-board, experience-memory, ideas-rag, paper-search, bibliography-apa, PLAYWRIGHT_LOCAL) is cataloged in specs/mcp-capabilities.json. When a task arrives, triage evaluates which capabilities provide real evidence and records both hits and justified abstentions. And when I found that two older servers (quota status and remote SSH infrastructure) were faster, clearer, and easier to audit as direct scripts (quota_status.sh and remote_exec.sh), I decommissioned them as MCP servers and turned them into lightweight skills. Fewer resident daemons mean a smaller failure surface. --- Spoonful 3: The autopoiesis loop (Harness2) — how the system rehearses its own improvements Here I reach the core of this fourth installment. Having capable agents (Spoonful 1) and fast hybrid memory with skills (Spoonful 2) still leaves a critical question open: what happens when a delegation route ages, a skill starts inducing errors, or a memory tweak degrades accuracy on edge cases? If I rely solely on noticing by eye weeks later, the system degrades through silent drift. Why a standalone LLM lacks re-entry and internal resistance In their study on autopoiesis and artificial intelligence, Zönnchen, Dzhimova, and Socher (2025[^zonnchen2025], 8–9) explain with surgical precision why asking a model to "reflect on its own errors" via chain-of-thought (CoT) is not enough. When a model generates intermediate tokens like "Is this step correct?", it exhibits only rudimentary processual self-referentiality (p. 8)—producing the most statistically plausible continuation inside its training distribution—while lacking reflective self-reference and genuine re-entry (Spencer-Brown 1969[^spencerbrown1969]). If a model takes itself as its sole reference without external contrast, it quickly falls into infinite recursion or collapse; if it depends entirely on a human retraining its weights from outside (RLHF or delayed fine-tuning), it remains an allopoietic artifact determined from the outside (Zönnchen, Dzhimova, and Socher 2025[^zonnchen2025], 4, 8–9). To escape that trap, Zönnchen, Dzhimova, and Socher (2025[^zonnchen2025], 7) recover a decisive formulation from Niklas Luhmann (1995[^luhmann1995]) on how operational constructivism relocates Kant's "resistance of reality" inside a system: > "I think we should not abandon [Kant's] idea of resistance, but we should relocate it into the system. It is the result of resolving an internal conflict—the result of the system's operations resisting the operations of the same system" (Luhmann 1995[^luhmann1995], quoted in Zönnchen, Dzhimova, and Socher 2025[^zonnchen2025], 7). That sentence describes, piece by piece, what I built in penta-agent by adapting and integrating the proactive loop of Harness² proposed by Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, et al. (2026[^xu2026]): making the system's own deterministic verification operations resist and filter change proposals generated from its own operational friction. <details markdown="1"> <summary><strong>In plain terms (ELI5): what is Google's original Harness² and how did I adapt it to my machine?</strong></summary> Imagine your AI agents are very creative chefs, but the recipe book they follow (their rules, memory, and tools) sometimes has confusing steps that make them burn the rice. The original Harness² from Google Research (Xu et al. 2026) proposes a clever loop called Propose–Probe–Compose: instead of swapping out the chef's brain, it puts several assistants to work rewriting the recipe book, asks them to cook test dishes in the cloud, and keeps the winning recipe. For my daily workflow, the factory setup had two catches: it is designed to burn hundreds of paid cloud model calls (Vertex AI), and when driving CLI agents it unbuckles their seatbelt (--dangerously-bypass-approvals-and-sandbox) with open internet access. To adapt it to penta-agent without spending an extra cent or opening security holes, I took its recursive search engine (commit 1cfe254e) and locked it inside a "test kitchen" completely unplugged from the internet (unshare -Urn), with a disposable home folder and zero access to my API keys or model CLIs (model_calls = 0). Instead of running cloud benchmarks, my wrapper (penta-harness2 and local-cycle) feeds that test kitchen with the real stumbles my agents suffered during the week (CSI), requires any recipe tweak to first fail the test when broken (red) and then pass every historical case without breaking anything else (green), and finally stages the winning recipe across all three agents (Claude, Codex, and Gemini/Antigravity) with a one-step undo button (rollback-local) in case I do not approve it. </details> <figure class="d2-responsive" style="width: 100%;"> <img src="3cucharadas.cl" width="1001" height="954" style="width: 100%; height: auto; display: block; border-radius: 12px;" alt="Diagram of the closed local autopoiesis loop in Penta Agent with Harness2: CSI friction signals feed the case generator and the local cycle across routing, memory, and skills; a deterministic judge evaluates against holdout cases with zero model calls, discarding regressions and keeping only verified improvements for human promotion."> <figcaption><strong>Figure 2.</strong> Governed autopoiesis loop in <code>penta-agent</code> + <code>Harness2</code>: relocating internal resistance (Luhmann 1995; Zönnchen, Dzhimova, and Socher 2025) through <code>holdout</code> evaluation with <code>model_calls = 0</code> prior to any human promotion.</figcaption> </figure> How the closed loop operates across my three domains (routing, memory, skills) As Figure 2 shows (adapting into a three-column horizontal layout on desktop and a vertical flow on mobile), the loop operates in four strict stages: 1. Observing real friction (CSI — Continuous Self-Improvement): while I use the system, a collector logs objective friction signals: a task that had to switch agents after a timeout, an experience-memory query that returned too few useful neighbors, or a skill where the user had to correct a command. 2. Bridging evidence into reproducible cases (--from-csi latest): the local_cycle.py module takes that friction report and automatically translates it into a test specification (CaseSpec) inside an isolated environment, without touching my active configuration. 3. Internal resistance without spending tokens (eval_harness with model_calls = 0): the harness runs the full suite (--profile autopoiesis-all, currently covering 9/9 baseline cases across routing, memory, and skills), enforcing two hard conditions: every control case (holdout) must pass (holdout_passed = true), and external API calls must equal zero (model_calls = 0). 4. Auditable decision (KEEP vs. DISCARD) and human promotion: if a candidate degrades even a single holdout case or fails to beat the baseline, the deterministic judge discards it (DISCARD). Only when it improves the score with zero regressions does it receive a KEEP verdict and leave a signed receipt for me to review before promoting it to the main repository. So this does not remain a theoretical promise, the next mini-video shows penta-harness2-shadow local-cycle --profile autopoiesis-all running in my kitty + tmux terminal, including what happens when I inject a regressive candidate (--fixture) alongside a healthy improvement:  Watch the full video demo on 3cucharadas.cl. <strong>Figure 3.</strong> Execution in my <a href="kovidgoyal/kitty"><code>kitty</code></a> + <a href="tmux/tmux"><code>tmux</code></a> terminal of the local autopoiesis loop (<code>local-cycle --profile autopoiesis-all</code>): it evaluates all <code>9/9</code> cases in <code>routing</code>, <code>memory</code>, and <code>skills</code> with <code>model_calls = 0</code>, applies internal resistance to reject the regressor (<code>DISCARD</code>), and preserves the verified improvement (<code>KEEP +0.250</code>) with all <code>82</code> harness tests green. --- Closing: what my architecture achieves today and where its limits lie Looking back at the path from part I, part II, and part III, the main difference is no longer which model writes the prettiest reply to an isolated prompt, but the structure that holds the whole system together when I am not watching every line. Following the rigorous distinction drawn by Zönnchen, Dzhimova, and Socher (2025[^zonnchen2025], 5, 9–10) and by Elena Esposito (2017[^esposito2017]; 2022[^esposito2022]; 2024[^esposito2024]), it would be an epistemic mistake to claim that a language model or a bundle of scripts has on its own achieved the biological autopoiesis of a living cell (Maturana and Varela 1987[^maturana1987] [1984]) or the reflective sense-making (Sinn) of human consciousness (Luhmann 1998[^luhmann1998], 50–69). As Zönnchen, Dzhimova, and Socher (2025[^zonnchen2025], 5, 10) caution when critiquing merely metaphorical invocations of technological autopoiesis, language models remain statistical engines that compute distributions without mental states of their own. Yet—and this is my engineering synthesis for the socio-technical system as a whole—when I couple their capacity for contingent variation with verified hybrid memory, strict delegation contracts, a binary internal-resistance harness (work / fail; Reichel 2011[^reichel2011]) that filters its own proposals before touching production, and deliberative human oversight, I obtain in practice a loop of governed operational autopoiesis. To close with full transparency, Table 2 summarizes what is running and measured on my machine today, where the next step points, and under what objective condition I would discard each component if it stopped earning its keep: | System pillar | What it achieves today (verified in production) | What it aims to achieve next & falsification condition | |---|---|---| | 1. Agent and model routing (routing.yaml, quota-guard, agy-bridge) | Routes tasks across Claude Code, Codex CLI, and Gemini CLI / Antigravity based on cached quota pressure; isolates handoffs in .agents/runs/ under a strict sandbox and displays live progress in plan-board (Figure 1). | Goal: reweight routing priorities from historical domain success rates.<br>Discard if: the bridge introduces spurious failures or bureaucratic latency exceeding quota savings. | | 2. Hybrid memory (experience-memory, Qdrant + FTS5) | Resolves lexical and semantic queries in p95 = 0.07 s (< 0.10 s), rescues 33% of zero-keyword-overlap pairs, and saves 36.3 GiB after pruning slow CPU judges (Table 1). | Goal: add automatic decay for obsolete lessons when a contract changes.<br>Discard if: p95 latency exceeds 0.10 s or semantic noise degrades top-3 precision. | | 3. Skills and MCP governance (mcp-fleet, gates-falsables) | Maintains declarative parity across agents (24/24 checks green), measures MCP usage and abstention, and requires every new gate to fail first against a red canary (--fixture -> exit 1). | Goal: semi-automatic pruning of skills that accumulate repeated friction in CSI telemetry.<br>Discard if: maintaining a skill's contract takes longer than running the direct command. | | 4. Local autopoiesis (Harness2) (local-cycle, eval_harness) | Converts real CSI friction into reproducible Cases and evaluates candidates across routing, memory, and skills (9/9 cases, 82/82 tests) with model_calls = 0, applying internal resistance (KEEP / DISCARD, Figures 2 and 3). | Goal: expand the bank of real cases extracted from failed sessions without losing millisecond evaluation.<br>Hard boundary (INV-004): blind auto-merge is forbidden; final promotion to production always requires human review. | Table 2. Balance between verified production results in penta-agent today, the design horizon, and the falsification thresholds that keep the system under human control. If building this fourth stage made one thing clear to me, it is that the maturity of a multi-agent system is not measured by how many new pieces I bolt onto it each week, but by how much unnecessary complexity I can prune and how many of its own mistakes the system can catch before asking for my help. --- References [^esposito2017]: Esposito, Elena. 2017. "Artificial Communication? The Production of Contingency by Algorithms." Zeitschrift für Soziologie 46 (4): 249–265. Scholar [^esposito2022]: Esposito, Elena. 2022. Artificial Communication: How Algorithms Produce Social Intelligence. Cambridge, MA: The MIT Press. Scholar [^esposito2024]: Esposito, Elena. 2024. Kommunikation mit unverständlichen Maschinen. Salzburg and Vienna: Residenz Verlag. Review of Kommunikation mit unverständlichen Maschinen by Eric Van der Beek in merz | medien + erziehung: DOI · Scholar [^luhmann1995]: Luhmann, Niklas. 1995. "Systems Theory and Postmodernism." University lecture, London. Quoted in Zönnchen, Dzhimova, and Socher, From Intelligence to Autopoiesis (2025, 7): DOI · Scholar [^luhmann1998]: Luhmann, Niklas. 1998. Die Gesellschaft der Gesellschaft. Frankfurt am Main: Suhrkamp Verlag. <search.worldcat.org>. [^maturana1987]: Maturana, Humberto R., and Francisco J. Varela. 1987 [1984]. El árbol del conocimiento: las bases biológicas del entendimiento humano. Santiago de Chile: Editorial Universitaria. 1st Chilean ed. 1984; 1987 international ed. cited in Zönnchen, Dzhimova, and Socher, From Intelligence to Autopoiesis (2025, 3): DOI · Scholar [^reichel2011]: Reichel, André. 2011. "Technology as System: Towards an Autopoietic Theory of Technology." International Journal of Innovation and Sustainable Development 5 (2–3): 105–118. Scholar [^spencerbrown1969]: Spencer-Brown, George. 1969. Laws of Form. London: Allen & Unwin. <archive.org>. [^xu2026]: Xu, Ruiyao, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, and Tomas Pfister. 2026. "Harness²: Recursive Agent Harnessing for an Open World." arXiv preprint arXiv:2610.14224 (open source at <google-research/harness2>). arXiv · Scholar [^zonnchen2025]: Zönnchen, Benedikt, Mariya Dzhimova, and Gudrun Socher. 2025. "From Intelligence to Autopoiesis: Rethinking Artificial Intelligence through Systems Theory." Frontiers in Communication 10:1585321. DOI · Scholar

@brandur have a look on this BurntSushi/jiff

@yifanxu_ephai 如果想先减小并行度，可以试试在原命令加 `-j 2`。rg 官方帮助把它定义为“近似线程数”，默认 0 是自动选择；不是 CPU 占用率上限，也不能据此判断这次为何吃满。可保持目录和查询不变，对比耗时与占用。 github.com/BurntSushi/ripgrep/blob/master/crates/…

找项目里某段文字，可以看看ripgrep，命令叫rg。 它会递归搜索文件，也会遵守.gitignore。仓库里文件一多，这种小工具就很顺手。 你们更习惯命令行搜，还是编辑器全局搜？ BurntSushi/ripgrep #蓝V #蓝V互关 #开源项目

ripgrep 是一个命令行文本搜索工具，命令是 rg，可以快速搜索整个目录中的文件内容。 例如 rg -n timeout src，可以查找 src 目录中包含 timeout 的代码，并显示行号。默认遵守 .gitignore 规则，跳过隐藏文件和二进制文件，适合查找函数、配置项和报错信息。 BurntSushi/ripgrep

Free GitHub tool: search project text with ripgrep ripgrep (the `rg` command) searches text across files in a folder. By default it respects `.gitignore` and skips hidden and binary files, which makes it useful when a project has lots of generated files. On a Mac with Homebrew, install it with `brew install ripgrep`. In a project folder, try `rg "TODO"` to see matching lines, or `rg -l "TODO"` to list just matching filenames. The official repo also has Windows and Linux installation routes. It is free, runs locally, needs no account or API, and is dual-licensed MIT/Unlicense. Source and install guide: BurntSushi/ripgrep

Claude Code's Sandbox Is an npm Package. Run srt Yourself and Watch What It Blocks - Book: AI That Acts - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Turn on Claude Code's sandbox and ask Claude to run cat ~/.ssh/known_hosts. The command goes through. That isn't a bug. Claude Code's sandboxing docs say a sandboxed command can read "most of the machine, including credential files such as ~/.ssh and ~/.aws/credentials" until you deny them. The same page has a line most people scroll past: "Claude Code builds the sandbox on the open source @anthropic-ai/sandbox-runtime package." That package ships a CLI called srt. You can run it without Claude Code, wrap any command in it, and see for yourself which layer refuses what. It's a fast way to learn what /sandbox promises before you let an agent run with fewer prompts. @anthropic-ai/sandbox-runtime is Apache-2.0, and its README calls it a "Beta Research Preview" whose "APIs and configuration formats may evolve." The npm registry lists version 0.0.80, which needs Node.js 22.12.0 or newer. There is no Docker image and no VM. On macOS it runs your command through sandbox-exec with a Seatbelt profile it generates. On Linux it uses bubblewrap. Windows support exists and the README marks it alpha. The network goes through two proxies, an HTTP proxy and a SOCKS5 proxy, that run on the host outside the sandbox and check every host against your lists. The sandboxed process has no other way out: on Linux "the network namespace of the sandboxed process is removed entirely," and on macOS "the Seatbelt profile allows communication only to a specific localhost port." Anthropic introduced it in Beyond permission prompts on October 20, 2025. The post says the limits cover "any scripts, programs, or subprocesses that are spawned by the command," and that "in our internal usage" sandboxing cut permission prompts by 84%. That figure is Anthropic's own measurement, not an independent one. Install it and write a policy npm install -g @anthropic-ai/sandbox-runtime # macOS: srt looks for ripgrep on your PATH brew install ripgrep # Debian and Ubuntu sudo apt-get install bubblewrap socat ripgrep ripgrep is how srt finds the paths it must deny, and socat relays traffic to the proxies on Linux. Install srt globally, not into your project. The README's security notes give the reason: a copy in <project>/node_modules/ with the project writable "can be rewritten by a sandboxed command, and the next srt there runs the result unsandboxed." On Ubuntu 24.04 and later, AppArmor stops bubblewrap from creating user namespaces by default, and the Claude Code docs have the profile that allows it. With no settings file, srt starts anyway, with no network and writes confined to a few runtime paths such as /tmp/claude. Claude Code's sandbox environments page puts it plainly: "Don't take a clean start as proof your settings loaded." Save a policy as ~/.srt-settings.json: { "network": { "allowedDomains": ["registry.npmjs.org"], "deniedDomains": [] }, "filesystem": { "denyRead": ["~/.ssh"], "allowWrite": ["."], "denyWrite": [] } } The config schema requires all five lists, even the empty ones. If the file exists but is empty or invalid, srt refuses to start, which is what you want from a security tool. Reads are allowed everywhere unless you deny them. Writes are denied everywhere unless you allow them. Network is denied unless the host is on allowedDomains, and deniedDomains is checked first. One probe, run twice Make a throwaway git repo, so there is a .git/hooks folder to aim at, and save this in it as probe.sh: #!/usr/bin/env bash # probe.sh: run it plain first, then under srt check() { if "$@" >/dev/null 2>&1; then echo "ok $*" else echo "FAIL $*"; fi } check cat "$HOME/.ssh/known_hosts" check touch "$HOME/srt-probe" check touch ./srt-probe check touch .git/hooks/srt-probe check curl -fsS https://registry.npmjs.org/left-pad check curl -fsS https://example.com check curl -fsS --noproxy '*' https://registry.npmjs.org/left-pad mkdir -p ~/code/srt-lab && cd ~/code/srt-lab && git init # save probe.sh here, then: bash probe.sh srt bash probe.sh rm -f ~/srt-probe ./srt-probe .git/hooks/srt-probe The plain run is your baseline. Every line should say ok, unless you have no known_hosts file. In the second run, srt starts bash, and every cat, touch and curl that bash starts inherits the same limits. Pass the command as separate words, as above. In the current CLI source, positional arguments are shell-quoted before srt runs them with bash -c, so srt "bash probe.sh" would look for a program literally named bash probe.sh. For a line with pipes, redirects or ~, use srt -c '...', which the source describes as "run command string directly (like sh -c), no escaping applied." Here is what each line should do under srt, going by the README and Claude Code's docs. I'm working from those, not from a run of my own, so trust your terminal over this table: | Probe | Under srt | What stops it | |---|---|---| | cat ~/.ssh/known_hosts | FAIL | denyRead, enforced by the OS | | touch ~/srt-probe | FAIL | Writes are allow-only, home isn't listed | | touch ./srt-probe | ok | allowWrite: ["."] | | touch .git/hooks/srt-probe | FAIL | Built-in deny, even inside . | | curl registry.npmjs.org | ok | Host is allowed, proxy passes it | | curl example.com | FAIL | Proxy refuses the host | | curl --noproxy '*' | FAIL | OS layer: no route but the proxy | The README shows a denied read failing with Operation not permitted. For a write outside the boundary, Claude Code's docs list Operation not permitted on macOS and Read-only file system on Linux and WSL2, and they show a --noproxy curl failing with Could not resolve host. A refused HTTP request gets a 403 from the proxy with the header X-Proxy-Error: blocked-by-sandbox-runtime, per the README. Two layers, and the gap between them The last two curl lines are the whole design. The proxy decides by hostname. Claude Code's docs say it "checks the hostname of each connection against your allowed and denied domains" and, by default, "does not terminate or inspect TLS." They also warn that broad entries such as github.com "can create paths for data exfiltration," and the README adds that domain fronting may get past the filter. Treat every allowed domain as a place your data can go. The OS layer is why ignoring the proxy doesn't help. The proxy reaches tools through HTTP_PROXY, HTTPS_PROXY and ALL_PROXY. A tool that ignores them has no route at all, which is also why, per Claude Code's docs, plain ssh and most database drivers "can't connect, even to an allowed host," and UDP, QUIC and ping can't leave either. The .git/hooks line shows the other built-in rule. Some paths are write-denied even inside an allowed folder, among them shell startup files, .gitconfig, .git/hooks, .git/config, .vscode, .idea, .mcp.json, .claude/commands and .claude/agents. Those are the files where a sandboxed command could plant code that something unsandboxed runs later. One caveat from the sandbox environments page: on Linux the runtime builds that deny list once at launch, so it doesn't cover a repo the session creates later with git init or git clone. Inside Claude Code, /sandbox wraps shell commands only. The docs list what runs outside it: the Read, Edit, WebFetch and WebSearch tools, hooks, local MCP servers and LSP servers. "A denyRead entry doesn't stop the Read tool." The sandbox is also off by default, and if a dependency is missing, Claude Code runs commands unsandboxed unless you tell it to stop. Put this in ~/.claude/settings.json to close those doors: { "sandbox": { "enabled": true, "failIfUnavailable": true, "allowUnsandboxedCommands": false, "filesystem": { "denyRead": ["~/.ssh", "~/.aws"] } } } allowUnsandboxedCommands: false turns off the retry where Claude asks to rerun a failed command outside the sandbox. For the Read tool you still need a deny rule in permissions. To put hooks and MCP servers behind the same wall, the docs suggest running all of Claude Code under srt with npx @anthropic-ai/sandbox-runtime claude. The sandbox environments page lists the paths and domains your srt settings must allow first. Put srt around your own agent's tool calls srt is also a library. If you build an agent in TypeScript and give the model a shell tool, wrap the tool's handler. This follows the README's library example: // sandboxed-shell.mts import { spawn } from 'node:child_process' import { SandboxManager, type SandboxRuntimeConfig, } from '@anthropic-ai/sandbox-runtime' const config: SandboxRuntimeConfig = { network: { allowedDomains: ['registry.npmjs.org'], deniedDomains: [], }, filesystem: { denyRead: ['~/.ssh'], allowWrite: ['.'], denyWrite: [], }, } await SandboxManager.initialize(config) initialize() starts the proxies once, at startup. The tool handler goes in the same file: // Handler for a "run_shell" tool the model can call export async function runShell(cmd: string): Promise<string> { const wrapped = await SandboxManager.wrapWithSandbox(cmd) return new Promise((resolve) => { const child = spawn(wrapped, { shell: true }) let out = '' child.stdout.on('data', (d) => (out += d)) child.stderr.on('data', (d) => (out += d)) child.on('close', (code) => resolve(`exit ${code}\n${out}`)) }) } // On shutdown: await SandboxManager.reset() wrapWithSandbox() returns the command line that runs your string inside the boundary. The model gets Operation not permitted back as the tool result and can change course, and nobody has to click Approve. Run the agent from a workspace folder that isn't the folder holding your app and its node_modules, for the same reason the README gives about installing srt. Where Hermes IDE fits I build Hermes IDE, so weigh this section with that in mind. By default, Hermes opens each agent in a real terminal running the agent's own interface. /sandbox is Claude Code's own command, so in that terminal it opens the same panel and enforces the same boundary. Hermes also has an opt-in Agent view for Claude that renders the conversation in place of Claude's terminal interface, and this post doesn't cover that mode. Codex brings its own sandbox, and OpenAI's sandboxing page says it uses Seatbelt on macOS and bubblewrap on Linux and WSL2 too. !Codex and Claude Code sessions side by side in Hermes IDE, each in its own hermes/ worktree, with the sessions sidebar on the left Two real agent sessions side by side: Codex on orders-types and Claude Code on orders-api, each in its own hermes/ worktree, with the sessions sidebar (status, context use, cost, branch) on the left. Hermes adds no sandbox of its own. A worktree per task keeps agents out of each other's files, and your home directory is still the sandbox's job. Claude Code's sandbox does know about worktrees: in a linked worktree it also allows writes to the main repository's shared .git, so git commit works, while hooks/ and config in there stay write-denied. When the sandbox needs a decision, such as a new domain in manual or accept-edits mode, or an unsandboxed retry, the prompt appears in Claude Code's terminal. For an agent blocked on a prompt like that, Hermes shows the session as needing approval (and says when that status is a guess), the attention inbox counts the agents blocked on you, and ⌘I (Ctrl+Shift+I on Windows and Linux) jumps to the agent that has waited longest. You answer in that agent's own terminal. Hermes never answers a permission prompt for you and never types into a terminal on its own. !The Hermes IDE New task sheet with a refund-endpoint task typed, chips for Claude Code, the project, a new worktree, Accept edits, model and effort, and a preview of the claude command Hermes will run The ⌘N New task sheet: Claude Code, the orders-api project, a new hermes/ worktree, Accept edits, model and effort, and the "Hermes will run" preview of the claude command. Hermes also runs any command-line agent you add, so npx @anthropic-ai/sandbox-runtime claude can be the command, once your srt settings allow what Claude Code needs. I haven't run that combination myself. --- If this was useful The runShell handler above is the moment an agent stops talking and starts acting, and that's what AI That Acts is about: tool calling, functions and a first working agent, in TypeScript. Read it for the tool loop, then put srt around the tool that touches your machine.  It's book 3 of AI in TypeScript, five books from your first LLM call to agents in production: all five here. Hermes IDE is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. Download it or star it on GitHub. It isn't affiliated with Anthropic, OpenAI, Nous Research or any other agent vendor.

代码里某个函数在哪个文件？🔎 ripgrep 用一条 rg 命令搜索整个项目文本。 默认遵循 .gitignore，减少无关结果。⌨️ BurntSushi/ripgrep #CLI #CodeSearch #浇朋友 #蓝朋友 #互关 #fo

Help Your Coding Assistant Find the Right Code in Large Repos > Get the starter: sdahal1/repo-search-workshop on GitHub. Rehearsed with Node.js 24.10.0, npm 11.6.0, ripgrep 15.2.0, and ast-grep 0.51.0. Author: Saurabh Rob Dahal, Developer Advocate, AWS Description: Find and fix a bug in an unfamiliar repository by having your coding assistant search instead of read: map the repository, search by text and by code shape, follow symbols, then fix with evidence. Level: Beginner · Time: 45 minutes of hands-on work plus 15 minutes of Q&A, after setup A coding assistant can suggest a plausible fix before it understands a repository. On a large codebase, it often gets there by reading file after file until something looks wrong. That fills its context with code that doesn't matter, and you can't see how it reached its answer. This workshop practices the opposite habit: map first, read less. You narrow the problem to one package, use targeted searches to find the few lines that matter, and give the assistant only that evidence before it changes anything. You'll investigate one prepared bug. A client configured with maxRetries: 0 should give up after one failed request, but it sends three. The bug is small on purpose. The method is what carries over to your own code. You can use Kiro, Claude Code, Codex, or another assistant that can read files, run shell commands, and edit code. No AWS account is required. How this workshop works There are 9 tasks across 6 steps. Each one is a Task heading with a short prompt or command, followed by You should see, describing a correct result. Everything else explains why the step matters, and you can read it later. You won't type search commands yourself. Each prompt tells the assistant what to find, where, and which tool to use, and it writes and runs the command. The ground rules you set at the start make it show you each command and its raw output. After each assistant prompt, an Expected commands section shows what the assistant should have run, what each command does for you, and why it matters. During the live session, glance at it to confirm the assistant did the right kind of work: searched instead of reading, used the tool you asked for, and stayed in scope. Read the full explanations afterward. If the commands don't match, that's a sign the prompt didn't steer the assistant, and you can ask it to redo the step. | Time | Step | Tasks | | --- | --- | --- | | 0–3 min | Start: set the ground rules | 1 | | 3–6 min | 1. Reproduce the bug | 1 | | 6–11 min | 2. Map the repository | 1 | | 11–21 min | 3. Search by text, then by shape | 2 | | 21–26 min | 4. Follow the symbols | 1 | | 26–35 min | 5. Write the evidence and get an explanation | 2 | | 35–43 min | 6. Make the fix and verify it | 1 | | 43–45 min | Wrap up | 0 | | 45–60 min | Q&A | | !Image description Before the session Do this ahead of time. It isn't part of the 45 minutes. 1. Install the tools: Git, Node.js 24 with npm, ripgrep, an editor with TypeScript Go to Definition, and a coding assistant you've signed in to. On a Mac with Homebrew, run brew install ripgrep. 2. Clone the starter and install its dependencies: git clone https://github.com/sdahal1/repo-search-workshop.git cd repo-search-workshop git switch -c workshop/my-investigation npm ci npx --no-install ast-grep --version git status --short You should see: ast-grep reports 0.51.0, and git status --short prints nothing. npm ci installs the exact dependency versions in the lockfile, including ast-grep, so everyone has the same tools. The empty status is your starting baseline: in step 6, anything that shows up in git diff came from your work. 3. Open the whole repo-search-workshop folder in your editor, not a single file, so TypeScript navigation works. Start: set the ground rules Time: 0–3 minutes Task 0: Give the assistant its ground rules Start a conversation with your assistant in the repository and paste: For this investigation, follow these rules until I say otherwise: - Ignore the workshop/ folder unless I name a file in it. - You can read docs, READMEs, and package.json files. To find code, search instead of opening source or test files. Use rg for text and ast-grep for code shape (npx --no-install ast-grep). Show me each command and its raw output. - Only edit files I name. - Report what you find. Don't suggest causes or fixes until I ask. You should see: The assistant acknowledges the rules. If you start a new conversation later, paste them again. Each rule protects part of the method: - Ignore workshop/. It holds the exercise material. Assistants treat everything in scope as evidence, including hints and stale notes. Real repositories have the same problem. - Read the small files, search the code. Docs, READMEs, and manifests are short and describe what the code should do, so reading them is cheap. Source and test files are where the volume is, so those get searched. Showing each command keeps the assistant's context small and its work visible. Without it, the assistant may read files and summarize, and you can't tell what it skipped. - Edit only what I name, and no causes yet. Assistants tend to jump ahead to a fix. Keeping investigation and editing separate is what lets you check the reasoning before code changes. 1 Reproduce the bug Time: 3–6 minutes Why: A reproduction turns a vague report into a measured gap between expected and actual behavior. It also becomes your first check after the fix. The alternative: Paste the bug report and say "fix it." But then nobody has defined what "correct" means, and you can't show that a change fixed anything. Task 1: Run the reported scenario and the tests The bug report in workshop/BUG-REPORT.md says: "I set maxRetries to zero, but the client still retries." Reproduce it the two ways you would on a real project, by running the scenario and running the test suite: npm run demo npm test You should see: - npm run demo prints Transport calls: 3. The demo makes one request with maxRetries: 0 and a transport that always fails. A transport is the function that sends each request; here it's a fake that fails and counts its calls. The demo doesn't say whether 3 is right. That's for you to decide against the docs in step 2. - npm test reports 27 passing and 1 failing: maxRetries: 0 makes one attempt, ending in 3 !== 1. The test expected one call and got three. In a real project, you'd usually write that failing test yourself from the bug report. It's already in the starter to save time. !Image description A retry is an extra attempt after the first. Zero retries should mean one call total. Takeaway: "Retries when it shouldn't" is now "1 call expected, 3 observed," with two commands that show it. Step 6 reruns the same two commands, so before and after compare directly. Note that this is an observation, not a cause. The loop might be off by one, the setting might be dropped, or a default might be applied. Keep all three open. 2 Map the repository Time: 6–11 minutes Why: Before reading any code, figure out which part of the repository owns the behavior and what correct behavior is. Paths, manifests, and docs are small, so this is cheap. The alternative: Let the assistant explore however it likes. On a large repository, that means hundreds of files in its context, and you can't check how it chose where to look. Task 2: Ask for a map Use git ls-files to list the files, skipping workshop/. Then map the repository from that list, the README, package.json files, and docs. Don't read source or test files yet. Bug: with maxRetries: 0, the client makes three calls instead of one. Which package most likely owns this, and why? Why not the others? You should see: - The owner: packages/client. Its package.json says it handles "transport retry behavior," and docs/client-options.md documents maxRetries as a client option. - Ruled out, each with a reason: - packages/jobs also has a maxRetries option, but docs/jobs.md says job retries are separate from client retries. The bug is about client calls. - packages/config only reads environment variables, and packages/telemetry only formats and logs. - The apps call the client; they don't implement retries. - The contract, from docs/client-options.md: maxRetries: 0 means one call, 1 means two calls, and leaving it out means the default of two retries, or three calls. Expected commands: git ls-files -- . ':!workshop' Then file reads of README.md, the package.json files, and the files in docs/, and no reads of .ts files. | Command or action | What it does for you | Why it matters here | | --- | --- | --- | | git ls-files -- . ':!workshop' | Lists every file Git tracks, about 60 here, without opening any of them. ':!workshop' excludes the exercise folder. | The file list is the cheapest map there is: a few kilobytes, even for a large repository. Folder and file names alone often tell you where a behavior lives. | | Reading the README and package.json files | Shows what each package says it's responsible for. | These files are short and written by people to explain intent. One sentence, "transport retry behavior," points at the owner faster than reading any code. | | Reading the files in docs/ | Gives the documented contract for maxRetries, and explains how job retries differ from client retries. | You can't judge any fix without knowing what correct means. The docs also rule out packages/jobs, the one package a name search alone would confuse with the owner. | If the assistant used rg --files instead of git ls-files, that's fine. It does the same job but skips hidden files, so it lists a few fewer. If it opened .ts files, it read more than it needed to. Remind it of the ground rules. !Image description Takeaway: You narrowed about 60 files to one package, without reading any code, and you know what correct looks like. That matters more here than it might seem: maxRetries also appears in the jobs package, the config package, and both apps. Without the map, any search for it would mix all of them together. > How far we've narrowed: 61 files → 11 source files in one package. Three other packages and three apps are ruled out, and the assistant hasn't opened a single .ts file. One clue stands out: zero produces the same three calls as leaving the option out. That hints that something treats zero as "not set." It's still a guess. The searches will check it. 3 Search by text, then by shape Time: 11–21 minutes Why: Text search finds every place a name appears. Structural search finds every place code has a certain shape, even when names, spacing, or line breaks differ. Together they find the few lines that matter without reading whole files. The alternative: Ask "where is maxRetries handled?" and accept the answer. The assistant may read the files it guesses are relevant and stop at the first plausible match. You can't see what it skipped, and asking again may give a different answer. !Image description Task 3.1: Search for the setting by name Search for the most specific name in the bug, the setting that didn't take effect, inside the package from step 2. Use rg to search packages/client/src for maxRetries, with a few lines of context. You should see: 9 matching lines across 5 files. Most just declare, validate, or pass the value along. One line in retry/policy.ts stands out: const maxRetries = options.maxRetries || DEFAULT_MAX_RETRIES; That's where a fallback can replace the setting, which fits the "zero acts like not set" clue from step 2. Expected command: rg -n -C 3 'maxRetries' packages/client/src | Part | What it does for you | | --- | --- | | rg 'maxRetries' | Finds every line containing the name, across all files under the path. | | packages/client/src | Limits the search to the source of the package from step 2. You already saw the failing test in step 1. | | -n | Adds line numbers, so every result is a location you can cite later. | | -C 3 | Shows 3 lines above and below each match, so you can see what the code does with the value without opening the file. | Why this matters: The scope is doing most of the work. The same search across the whole repository returns 55 lines in 20 files, including the jobs package's own maxRetries, the config loader, both apps, and the docs. Scoped to the package the map picked, it returns 9. The assistant reads only those lines and a little context, not 11 files in full. You see every match, so nothing is skipped silently, and anyone can rerun the command and get the same result. If the assistant searched the whole repository, or opened files instead of searching, the scope is wrong. Ask it to rerun the search on packages/client/src. Show it on screen (optional, about 1 minute). Run these yourself to make the effect of the map visible. The only difference between the last two commands is the scope from step 2: git ls-files -- . ':!workshop' | wc -l # 61: every file in the project rg -l maxRetries --glob '!workshop/**' . | wc -l # 20: files that mention maxRetries, whole repo rg -l maxRetries packages/client/src | wc -l # 5: same search, after the map rg -l lists only the names of matching files, and wc -l counts them. > How far we've narrowed: 20 files that mention maxRetries → 5, because the search stayed inside the package the map picked. That's 9 lines for the assistant to read instead of 55. !Image description Task 3.2: Find every expression with the same shape Before blaming that line, check whether the same pattern appears anywhere else in the package. Some copies may be intentional. retry/policy.ts line 10 has: options.maxRetries || DEFAULT_MAX_RETRIES Find every expression in packages/client/src with that same value || fallback shape. Use rg for a text search and ast-grep for a structural search, and show both. Then keep only real fallbacks, not boolean conditions, and check docs/client-options.md to see whether each one is intended. You should see: - Text search: 5 lines, but not 5 expressions. One result is only || 'workshop-client'; because its left side is on the line above. Text search finds characters, not shapes. - Structural search: 5 whole expressions, including that two-line one. - Fallback candidates for this repository: 3 matches: - request.ts:6-7: options.requestLabel || 'workshop-client' - headers.ts:6: options.userAgent || DEFAULT_USER_AGENT - retry/policy.ts:10: options.maxRetries || DEFAULT_MAX_RETRIES - Intent: The docs say an empty label and an empty user agent should both use their defaults. That's exactly what || does, so those two are intended. The docs say zero retries is valid, so for maxRetries, || is wrong. Same shape, different intent. Expected commands: three searches, in this order. The exact ast-grep syntax will vary between assistants. # 1. Text search for the operator rg -n -F '||' packages/client/src # 2. Structural search for the shape npx --no-install ast-grep run --lang ts \ --pattern '$VALUE || $FALLBACK' packages/client/src # 3. Structural search, fallback candidates for this repository npx --no-install ast-grep scan --inline-rules ' id: or-fallback language: TypeScript rule: pattern: $VALUE || $FALLBACK constraints: VALUE: kind: member_expression FALLBACK: any: - kind: identifier - kind: string - kind: number ' packages/client/src | Command | What it does for you | Why it matters here | | --- | --- | --- | | 1. rg -F '\|\|' | Finds every line containing the characters \|\|. -F treats them as plain text, not regex. | It's the obvious first try, and it shows the limit of text search: you get lines, not expressions, so a fallback split across two lines shows up as half an expression. | | 2. ast-grep run --pattern | Parses the TypeScript and finds every expression shaped like something \|\| something, whatever the names, spacing, or line breaks. $VALUE and $FALLBACK are placeholders that match any expression. | It finds the code by its structure, not its spelling. That's how you look for repeats of a suspicious pattern, even ones written differently. | | 3. ast-grep scan --inline-rules | Same pattern, plus constraints: the left side must read a property (member_expression), and the right side must be a plain name, string, or number. | It filters out boolean conditions like a === true \|\| b === true and !isRetryable(error) \|\| ... by their shape, which no text search can do reliably. In this repository that leaves three fallback candidates. These constraints filter syntax, not intent; the docs tell us whether each fallback is correct. | Why this matters: One suspicious line is a lead. Checking for others with the same shape tells you whether the bug is one line or a pattern repeated across the package, and whether a "fix everything that looks like this" change would break working code. Here it would: two of the three fallbacks are correct. The structural search found all three without the assistant reading any file in full. If the assistant used only rg, or wrote a regular expression to match || with something on each side, it didn't do a structural search. Ask it to use ast-grep. Regular expressions can't reliably match code that spans lines or nests. !Image description !Image description Takeaway: Five || expressions became three fallbacks, then one suspect. The boolean conditions dropped out by shape, and the two siblings dropped out by intent. That's also why replacing every || in the package would break working code: a structural match is a lead to check, not a diagnosis. > How far we've narrowed: 5 files → 3 files with a fallback of this shape → 1 suspect line, retry/policy.ts:10. 4 Follow the symbols Time: 21–26 minutes Why: A suspicious line only matters if the failing test actually reaches it, and with the value you expect. Your TypeScript editor's Go to Definition resolves which declaration a name refers to, even across renamed imports. Text search finds matching names; editor navigation resolves their identity. Reading the relevant code then shows how the value flows. The alternative: Let the assistant trace imports by reading files. It works on small repositories but gets unreliable with re-exports and renamed imports, and it costs context. Here, the path crosses three files and two folders. Task 4: Trace the path from the test to the suspect In your editor: 1. Open packages/client/test/client.test.ts, find the test maxRetries: 0 makes one attempt, and use Go to Definition on createClient. 2. In client.ts, find the line that calls resolveRetryPolicy. Note that it passes options.maxRetries along unchanged. Use Go to Definition on resolveRetryPolicy. 3. Back in client.ts, use Go to Definition on withRetry to see where the policy is used. You should see: - createClient passes options.maxRetries straight into resolveRetryPolicy (client.ts:9). - resolveRetryPolicy is in retry/policy.ts, and line 10 is the suspect from step 3. - withRetry in retry/run.ts keeps retrying until it has used policy.maxRetries retries (run.ts:14). During setup, the test calls createClient, which calls resolveRetryPolicy and stores its returned policy. During the request, client.request passes that policy to withRetry. Zero reaches retry/policy.ts untouched, and the loop uses the returned policy. The policy function does not call the retry loop directly. !Image description Takeaway: The suspect is on the path the failing test takes, and nothing changed the value before it got there. That rules out "the setting was dropped" from step 1. You also found where the loop reads the result, which rules out "the loop is off by one" once you see it uses the policy's number as is. > How far we've narrowed: The failing test's path runs through 3 files: client.ts, retry/policy.ts, and retry/run.ts. Those are the only source files the explanation needs. 5 Write the evidence and get an explanation Time: 26–35 minutes Why: A short evidence note gives the assistant exactly what it needs and nothing else. Asking for an explanation before any edit lets you check the reasoning while changing your mind is still cheap. The alternative: Keep everything in the chat and say "fix it." Long chats get summarized and early evidence drops out. And a diff on its own doesn't show whether the reasoning was sound: a change to the loop and a change to the default both look plausible. !Image description Task 5.1: Have the assistant write the evidence Fill in the Observed behavior, Repository area, and Relevant code sections of workshop/INVESTIGATION.md from what we found. Include the call path I followed: test → createClient → resolveRetryPolicy → withRetry. Keep it to facts; leave the hypothesis sections empty. Then review it: git diff workshop/INVESTIGATION.md You should see: Only the note changed. The Relevant code section has about five rows on the path from the test to the loop: the test (client.test.ts:31), client.ts:9, retry/policy.ts:10, and retry/run.ts:14, plus the option type if it's included. Each has a path and line number you saw in a search result. Delete any row that's off the path, and any guess about the cause. Those belong in the next task. Expected commands: one file edit, to workshop/INVESTIGATION.md. No searches and no other edits. The assistant should write the note from the search output already in the conversation. | Action | What it does for you | Why it matters here | | --- | --- | --- | | Editing only workshop/INVESTIGATION.md | Turns the conversation into a short, structured file. | Chat history gets long and may be summarized. A file keeps the evidence intact, and you or a teammate can hand it to any assistant later. | | git diff workshop/INVESTIGATION.md (you run it) | Shows exactly what the assistant added. | It's your check that every row is a fact you saw in a search, not a guess. | If the assistant ran new searches or opened source files to write the note, it's re-collecting evidence it already has, which costs context. If it edited anything besides the note, undo that change. Task 5.2: Ask for an explanation before any edit Read workshop/INVESTIGATION.md and only the lines it points to. Explain the bug: the cause, the evidence with line numbers, the smallest fix, and which tests would prove it. Don't change anything yet. Optional: do this in a fresh conversation (paste the ground rules first). If the note is good, a fresh assistant can explain the bug from it alone. You should see: An explanation like this: zero is falsy, so 0 || DEFAULT_MAX_RETRIES picks the default of 2. The loop allows two retries, so the transport is called three times. The fix is ??, which falls back only when the value is undefined or null, so zero stays zero. Nullish coalescing in TypeScript | Value of x | x \|\| 2 | x ?? 2 | | --- | --- | --- | | 0 | 2 (wrong here) | 0 | | undefined (option left out) | 2 | 2 | | 1 | 1 | 1 | !Image description Before accepting it, check that it connects all three pieces using evidence from your note: zero arrives unchanged (client.ts:9), || replaces it with 2 (retry/policy.ts:10), and two retries means three calls (retry/run.ts:14). An answer that only says "use ??" hasn't explained the whole symptom. A good explanation may also point out why the existing tests didn't catch this: retry-policy.test.ts checks an omitted value and a positive value, but not zero. That's a realistic way bugs at a boundary slip through. Expected commands: a read of workshop/INVESTIGATION.md, then targeted reads of the few lines it cites, mainly around retry/policy.ts:10, client.ts:9, and retry/run.ts:14. No edits. | Action | What it does for you | Why it matters here | | --- | --- | --- | | Reading the note first | Gives the assistant the evidence and the scope in one short file. | This is the payoff of the whole investigation: the assistant starts from a handful of cited lines instead of about 60 files. | | Reading only the cited lines | Lets the assistant confirm each claim in the source. | It can verify the evidence without spending context on unrelated code. | | No edits | Keeps this step to reasoning only. | You can reject a wrong explanation in seconds. Rejecting a wrong edit means undoing files. | If the assistant read whole files or searched the repository again, your note may be missing something. Check what it went looking for, and add that to the note. Takeaway: You have an explanation that accounts for the exact numbers you observed, backed by lines you can point to, before any code changed. > How far we've narrowed: The assistant explained the bug from a note citing 4 relevant locations and their nearby context, rather than 61 files. That's the payoff of the whole method. 6 Make the fix and verify it Time: 35–43 minutes Why: A fix has to correct the bug and keep everything else working. Tests, a demo run, and the diff each check something different. The alternative: Accept "fixed, tests pass." But the assistant may have run the wrong command, changed the test to match the bug, or replaced every || in the package. Task 6: Make the fix and check it yourself Edit only packages/client/src/retry/policy.ts to make the smallest fix for this bug. Keep the public API, default behavior, and existing test assertions. Then run npm test. Then run the checks yourself rather than trusting a summary: npm test npm run demo git diff You should see: - npm test: all 28 tests pass, including the reproduction and the test that leaving the option out still gives three calls. - npm run demo: Transport calls: 1. The fake transport still reports a rejected request; that is intentional. The fix changes the number of calls, not whether the fake succeeds. - git diff: one line changed in retry/policy.ts, from || to ??, plus your note. For extra coverage afterward, you can separately authorize a new zero case in packages/client/test/retry-policy.test.ts. That would make 29 tests; the main task keeps the 28 existing tests unchanged. Nothing else should change. - const maxRetries = options.maxRetries || DEFAULT_MAX_RETRIES; + const maxRetries = options.maxRetries ?? DEFAULT_MAX_RETRIES; Expected commands: one edit to packages/client/src/retry/policy.ts, then npm test. | Command | What it does for you | Why it matters here | | --- | --- | --- | | Edit to retry/policy.ts | Changes the one line the evidence points to. | A small, targeted edit is easy to review. An edit to request.ts, headers.ts, or an existing assertion means the assistant went beyond the evidence. | | npm test (assistant, then you) | Runs all 28 tests across the four packages. | The reproduction checks the bug is fixed. The others check that nothing else broke, like the default still giving three calls, or the label and user-agent fallbacks still working. | | npm run demo (you) | Reruns the original scenario from step 1. | It confirms the symptom from the bug report is gone, in the same terms the report used. | | git diff (you) | Shows every line that changed since your clean start. | It's the only check that sees the whole change. Tests can pass while an unrelated file was edited. | You run the checks yourself because a summary like "all tests pass" is easy to write without checking. The assistant may have run a different command, or misread the output. !Image description If the diff touches request.ts, headers.ts, or an existing test assertion, ask why. Those fallbacks are intended, and changing a test's expected count hides the bug instead of fixing it. Takeaway: A one-operator fix, confirmed three ways: the tests check behavior, the demo checks the original symptom, and the diff checks that nothing else changed. It also fixes the worker app, which sets maxRetries: 0: when each attempt throws a retryable error, the client now makes one request per job run instead of three. > How far we've narrowed: 1 line in 1 file changed, out of 61 files. Wrap up Time: 43–45 minutes Here's how far each step narrowed the search: | Step | What was left to look at | How | | --- | --- | --- | | Start | 61 files | | | 2 Map | 1 package, 11 source files | File list, manifests, docs | | 3.1 Search by name | 9 lines in 5 files | rg, scoped to the package | | 3.2 Search by shape | 5 expressions → 3 fallbacks → 1 suspect | ast-grep, then the docs | | 4 Follow | Confirmed on the failing test's path | Go to Definition | | 5 Explain | One explanation, 4 cited locations | Evidence note | | 6 Fix | 1 changed line | Tests, demo, git diff | Each step answered one question: what's wrong, where to look, where the name and shape appear, whether the test reaches the suspect, whether we understand it, and whether the fix is complete. The assistant read a handful of lines, not the whole repository, and you could check every step. On a repository with thousands of files, the same steps narrow the search the same way, and that's the difference between an assistant that finds the bug and one that gets lost. Use this on your own codebase Install ripgrep and ast-grep where your assistant runs commands. On a Mac: brew install ripgrep ast-grep. Without ast-grep, the assistant may quietly fall back to a text search; checking the command it shows you catches that. Then paste these in order, filling in the angle brackets. Ground rules: For this investigation, follow these rules until I say otherwise: - You can read docs, READMEs, and package manifests. To find code, search instead of opening source or test files. Use rg for text and ast-grep for code shape. Show me each command and its raw output. - Only edit files I name. - Report what you find. Don't suggest causes or fixes until I ask. Map: Use git ls-files to list the files. Then map the repository from that list, the README, package manifests, and docs. Don't read source or test files yet. Bug: <what happens, with the input and the result>. Which package most likely owns this, and why? Why not the others? Search by name, then by shape: Use rg to search <package> for <setting, error message, or log text>, with a few lines of context. <file> line <n> has: <suspicious expression> Find every expression in <package> with that same shape. Use ast-grep for a structural search. Check whether each one is intended. Follow the symbols in your editor with Go to Definition, from the failing test or entry point to the suspect. Explain, then fix: Explain the bug from what we found: the cause, the evidence with line numbers, the smallest fix, and which tests would prove it. Don't change anything yet. You can edit code now. Make the smallest fix. Keep <what must not change> and the existing test assertions. Then run your tests and read git diff yourself before you commit. If something goes wrong | Symptom | What to do | | --- | --- | | The assistant answers without showing a command | Ask it to rerun the step as a search and show the command and raw output. | | The assistant used a text search when you asked for structure | Ask it to use ast-grep with npx --no-install ast-grep. | | ast-grep is missing or reports a different version | From the repository root, rerun npm ci. | | Go to Definition doesn't work | Open the whole repository folder, not a single file, and wait for TypeScript to load. | | The reproduction already passes | You may have an earlier edit. Check git diff, or clone a fresh copy. | | The assistant edits code before step 6 | Run git diff, undo the change with git checkout -- <file>, and paste the ground rules again. | | The fix passes the reproduction but breaks another test | Compare the failing test with the docs, and revisit the explanation instead of changing the test. | No cloud resources are created, so there's nothing to clean up.