microservices-demo

Docker Build Cache on Ephemeral CI Runners: inline, registry and local, Measured --- title: Docker Build Cache on Ephemeral CI Runners: inline, registry and local, Measured published: true date: 2026-11-02 19:00:00 UTC tags: Docker,BuildKit,CICD,Caching canonical_url: devops-daily.com --- A CI runner that starts empty has no BuildKit cache, so every docker build is a cold build unless you export the cache somewhere and import it on the next run. We rebuilt a real multi-stage Node service, the payment service from Google's Online Boutique demo, on a brand-new BuildKit builder every time, with five setups: no cache, inline, registry with mode=min, registry with mode=max, and a local cache directory. For a one-line source change, every cache worked. The rebuild went from 79.0 s with no cache to 19.2 to 26.3 s with any registry-backed cache, because BuildKit skipped the whole builder stage, npm install included. For this Dockerfile, the cheapest backends were the best ones. Inline and registry mode=min cost 0.0 and 1.2 s to export. Inline adds only metadata to the image, and mode=min stored a separate 47.7 MB cache with the same layers as the image. mode=max stored 259.9 MB, added 29.8 s of export to every cold build, and made the dependency bump slower, not faster: the cached compiler toolchain saved no time once its layers had to be pulled, and the export added about 20 s. The local directory cache, written fresh each run, was the slowest of all for the one-line change. TL;DR - Source change: all caches skip the builder stage. Median 19.2 to 26.3 s with inline or registry, against 79.0 s with no cache. In our three runs each, inline and mode=min did as well as mode=max, because BuildKit only loads the cache for the last step that matches. - Export cost is mostly compression. mode=max spent 27.9 s in preparing build cache for export on a cold build, and about 2 s writing 259.9 MB to the registry. mode=min exported in 1.2 s, inline in 0.0 s. - Dependency bump: mode=max did not help. It kept the compiler toolchain, but pulling and unpacking the cached base and toolchain layers took 15.0 to 20.6 s per build, about what mode=min spent pulling the base image and running apk add again (13.4 to 19.8 s). Its export then added 18.4 to 24.7 s. Median 132.4 s for mode=max against 110.8 s for mode=min. - Local cache, fresh directory each run: 74.7 s for a one-line change, because it repeats the full mode=max export every time. - What did not change: npm install ran on every dependency bump with every backend. A cache saves the steps before the change, never the step that changed. - Our registry ran on the same machine, so these times leave out the network. The sizes carry over: 47.7 MB pulled per warm source build, 259.9 MB per mode=max cache. Prerequisites - Docker with Buildx (we used Docker 29.9.2, Buildx 0.38.1 and BuildKit 0.34.1). - A registry you can push to. We ran registry:2.8.3 on the build machine; GHCR, ECR, Docker Hub or your own registry work the same way. - Node.js and npm, only to create the dependency change (npm install --package-lock-only). - If you want to rerun the whole matrix: a 4 vCPU Linux machine and about 80 minutes. We used a droplet on DigitalOcean (referral link). Why a fresh builder changes everything On a laptop, BuildKit keeps every layer it built in its own store, so the second docker build reuses them without any configuration. A CI runner that starts from a clean VM or container has no store. Unless you tell BuildKit where to import a cache from and where to export it to, every build is a cold build. BuildKit can export its cache in several ways. inline writes cache metadata into the image you push. registry writes a separate cache manifest to a registry. local writes an OCI layout to a directory that your CI then saves and restores. gha and s3 write to GitHub's cache service and to S3. mode=min, the default, exports only the layers of the final image. mode=max also exports the layers of every intermediate stage, such as a builder stage. The question for a CI pipeline is which of those pays for itself, for the changes that you actually push. How we measured The Dockerfile. We used src/paymentservice from Google's Online Boutique demo at commit c7cb6d5b, without changes. It is a real multi-stage Node service, and both base images are pinned by digest. This is the whole Dockerfile without its license header: # Define a default value so it's not empty if the builder fails to provide it ARG BUILDPLATFORM=linux/amd64 FROM --platform=$BUILDPLATFORM node:24.20.0-alpine@sha256:e67514e5d0f6c46656005e1b693b2ec9d52e80b641307de684d4a015ba7a4eaf AS builder # Some packages (e.g. @google-cloud/profiler) require additional # deps for post-install scripts RUN apk add --update --no-cache \ python3 \ make \ g++ WORKDIR /usr/src/app COPY package*.json ./ RUN npm install --only=production FROM alpine:3.24.1@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b RUN apk add --no-cache nodejs WORKDIR /usr/src/app COPY --from=builder /usr/src/app/node_modules ./node_modules COPY . . EXPOSE 50051 ENTRYPOINT ["node", "index.js"] The expensive work is in the builder stage: a compiler toolchain from Alpine's mirror, then npm install. The final stage installs Node from Alpine, copies node_modules, and copies the source last. An ephemeral runner, every time. Each build got a new BuildKit builder (docker buildx create --driver docker-container --bootstrap), and we removed it after the build. Nothing survived in the builder's store between builds except what the cache backend exported. The base image mirror and the Docker host itself did persist. We recorded builder start-up separately; it is the same for every backend. Three scenarios: - Cold: an empty cache. The build exports its cache to the backend. - Source change: one line in server.js (a log message). The build imports the cache that the last cold run wrote. - Dependency change: @grpc/grpc-js from 1.14.4 to 1.14.7, made with npm install --package-lock-only --save-exact, the size of a typical Dependabot patch bump. The lockfile diff changes only that package (npm also re-sorted two keys in package.json). It imports the same cache. Warm builds import from the cold cache and export to a separate scratch location, so every warm run starts from the same cache and still pays the export cost that a real pipeline pays. The backends: | Name | Import | Export | | --- | --- | --- | | none | nothing | nothing | | inline | type=registry,ref=<image> | type=inline (cache metadata inside the pushed image) | | registry, min | type=registry,ref=<cache> | type=registry,ref=<cache>,mode=min | | registry, max | type=registry,ref=<cache> | type=registry,ref=<cache>,mode=max | | local | type=local,src=<dir> | type=local,dest=<dir>,mode=max | Every build pushed its image with --provenance=false --sbom=false, so attestations do not add noise. Repeats. We planned five runs of every backend and scenario. The shared droplet ran slower than planned and we had a fixed budget, so we got 5 cold runs per backend, 3 source-change runs, and 3 (2 for the local cache) for the dependency bump. Runs went in round-robin order inside each scenario (run 1 of every backend, then run 2, and so on), which spreads slow periods on the shared host across the backends but does not remove order effects (see the caveats). The tables show the median and the number of runs (with 2 runs, the median is their mean). We dropped the page cache before each build and logged the CPU counters around each one, because the droplet's vCPUs are shared and steal time moves build times. What we recorded. Wall time of docker buildx build, BuildKit's own step timings from --progress=plain (which steps were CACHED, how long the export took), the cache size from the registry manifest or the cache directory, and the bytes BuildKit pulled from the cache registry, from the registry's access log. Where the cache lived. The registry for inline and type=registry was a registry:2.8.3 container on the same droplet, at 127.0.0.1:5000. We did not have a container registry we could use at no cost, and DigitalOcean's Spaces object storage has a monthly minimum, so there was no remote store that cost nothing. On loopback, writing the 259.9 MB mode=max cache took about 2 s, so our export and import times are close to the best case. On a real runner the same bytes cross a network to GHCR, ECR or your own registry, and the time grows with the size. That is why the tables show bytes next to every time: the bytes carry over to your setup, our seconds do not. Base images. Every fresh builder resolved node:24.20.0-alpine and alpine:3.24.1, and downloaded their layers whenever a step needed them. We pointed BuildKit at a pull-through mirror (registry:2.8.3 with REGISTRY_PROXY_REMOTEURL, also on the droplet), so Docker Hub's rate limits and speed stay out of the numbers. A real ephemeral runner pulls them from Docker Hub or a mirror on every build that needs them, so add that time to the rows where the FROM step ran. Package downloads were real. apk add fetched from Alpine's CDN and npm install fetched from the npm registry whenever those steps ran. That is the cost a cache saves, so we left it in. Why there is no type=s3 or type=gha. type=s3 needs an S3 bucket. The only free option here was an S3 emulator on the same droplet, which would measure the emulator, not S3. type=gha works only inside GitHub Actions. We left both out. Cold builds and what the export costs | Backend | Runs | Cold build | Cache export | of which preparing | Cache size | | --- | --- | --- | --- | --- | --- | | none | 5 | 79.0 s | - | - | - | | inline | 5 | 85.4 s | 0.0 s | - | in the image | | registry, mode=min | 5 | 89.4 s | 1.2 s | 0.1 s | 47.7 MB | | registry, mode=max | 5 | 128.0 s | 29.8 s | 27.9 s | 259.9 MB | | local, mode=max | 5 | 122.8 s | 39.1 s | 30.7 s | 259.9 MB | Cold builds: median of 5 runs. Export is the exporting cache step in the BuildKit log; for inline, the preparing layers for inline cache line. With no cache, a cold build took a median 79.0 s on a fresh builder. npm install was 45.3 s of that, the compiler toolchain (apk add python3 make g++) 6.5 s, and compressing and pushing the image about 9.5 s. The totals in the table are noisier than the export column. npm install alone took between 29.2 and 70.9 s across the cold runs, because it downloads from the npm registry, and the droplet reported 8 to 27% CPU steal per build. Read the export column for what each backend costs: - inline costs nothing measurable. BuildKit logged preparing layers for inline cache 0.0s done. The cache is metadata in the image config, and its layers are the image's own. - registry, mode=min added 1.2 s for a 47.7 MB cache: the same 5 layers as the final image, stored under the cache tag. - registry, mode=max added 29.8 s for 259.9 MB in 12 layers. 27.9 s of that was one line in the BuildKit log, preparing build cache for export: BuildKit compressing the builder stage's layers (the Node base image, the compiler toolchain and node_modules), which a mode=min export never touches. Writing the layers to the registry took about 2 s, on loopback. - local, mode=max added 39.1 s for the same 259.9 MB, written as an OCI layout on disk. Preparing took 30.7 s here too. So the expensive part of a mode=max export is compression, and it happens before a byte leaves the machine. On a real network, the upload of 259.9 MB comes on top. A one-line source change | Backend | Runs | Source change | Steps cached (of 8) | Pulling cached layers | apk add (builder) | npm install | Pulled from registry | | --- | --- | --- | --- | --- | --- | --- | --- | | none (a cold build) | 5 | 79.0 s | 0 | - | 6.5 s | 45.3 s | - | | inline | 3 | 26.3 s | 7 | 16.9 s | 0.0 s | 0.0 s | 47.7 MB | | registry, mode=min | 3 | 19.2 s | 7 | 12.4 s | 0.0 s | 0.0 s | 47.7 MB | | registry, mode=max | 3 | 22.9 s | 7 | 15.7 s | 0.0 s | 0.0 s | 47.7 MB | | local, mode=max | 3 | 74.7 s | 7 | - | 0.0 s | 0.0 s | - | The none row is the cold build. Pulling cached layers is the time BuildKit reported on the cached step that had to be unpacked; pulled from registry is from the cache registry's access log. Every backend skipped the whole builder stage. BuildKit marked 7 of the 8 build steps CACHED, including RUN npm install, and only COPY . . ran. That was true for inline and registry mode=min too, although those caches do not contain a single layer of the builder stage. That works because of how BuildKit finds a cache hit. A step's cache key comes from its instruction and the keys of the steps before it, and a COPY from the build context adds a checksum of the files it copies. In this graph, that let BuildKit match the final stage's COPY --from=builder step without running the builder stage. Its solver documentation says it "will not require the cache to be loaded or exists for all the vertexes in the graph but will only load it for the final node that is determined to match cache". Here, that node is COPY --from=builder, which is a layer of the final image, and every backend exported it. The builder stage never ran, and none of its layers were downloaded: every registry build pulled the same 47.7 MB, the final image's layers, even from the 259.9 MB mode=max cache. What a warm build still paid for was pulling and unpacking those cached layers to put COPY . . on top. BuildKit reports that time on the cached COPY --from step: a median of 12.4 to 16.9 s with the registry backends, out of 19.2 to 26.3 s for the whole build. The local backend was the exception: 74.7 s. Importing from the directory was quick, but exporting to a fresh directory took 52.4 s, because every run wrote the full mode=max cache again. The registry backends exported to a new tag in the same repository, and that took 1.4 s or less, because the layers they referenced were already in the registry. We exported to a new directory on purpose, to stop the cache from growing; the section on the local cache below explains the trade. A dependency bump | Backend | Runs | Dependency bump | Steps cached (of 8) | Pulling cached layers | apk add (builder) | npm install | Pulled from registry | | --- | --- | --- | --- | --- | --- | --- | --- | | none (a cold build) | 5 | 79.0 s | 0 | - | 6.5 s | 45.3 s | - | | inline | 3 | 105.7 s | 2 | 2.8 s | 9.1 s | 60.0 s | 26.0 MB | | registry, mode=min | 3 | 110.8 s | 2 | 2.3 s | 9.9 s | 61.5 s | 26.0 MB | | registry, mode=max | 3 | 132.4 s | 4 | 18.0 s | 0.0 s | 66.9 s | 196.8 MB | | local, mode=max | 2 | 132.1 s | 4 | - | 0.0 s | 51.9 s | - | A changed lockfile invalidates COPY package*.json in the builder stage, so npm install ran in every build, with every backend. The backends differ only in the step before it, the compiler toolchain: - inline and registry mode=min do not hold the builder stage, so BuildKit pulled the Node base image from the mirror (4.3 to 10.2 s) and ran apk add python3 make g++ again (a median 9.9 s). Only the final stage's RUN apk add --no-cache nodejs and WORKDIR came from the cache. - registry mode=max and local held it, and BuildKit marked the toolchain step CACHED. But to run npm install on top of it, BuildKit first had to pull and unpack the cached layers, the Node base image and the toolchain, including one layer of 112.3 MB. With the registry, that took 15.0 to 20.6 s per build, about the same as the 13.4 to 19.8 s that mode=min spent on the base image and apk add. The cache hit saved nothing. We expected mode=max to win this scenario, and wrote that down before the runs. It did not. Then the export. With mode=max, BuildKit also had to compress the builder stage's new npm install layer, which the image does not contain, and preparing build cache for export took 19.0 s, against 0.7 s for the whole mode=min export. Put together, the dependency bump took a median 132.4 s with mode=max and 110.8 s with mode=min, and mode=max was slower in each of the three rounds. The export accounts for most of the gap: mode=max took 18.4 to 24.7 s to export, against under 1 s for mode=min, while the toolchain stage cost about the same either way. mode=max also pulled 196.8 MB from the cache registry per build, against 26.0 MB for mode=min. The local cache, also mode=max, took 132.1 s (2 runs). With a remote registry the gap would grow, because those 196.8 MB and the new npm install layer would cross the network. Build time on a fresh builder (median) | | Value | Series | | --- | --- | --- | | none | 79 s | source change | | none | 79 s | dependency bump | | inline | 26.3 s | source change | | inline | 105.7 s | dependency bump | | registry, mode=min | 19.2 s | source change | | registry, mode=min | 110.8 s | dependency bump | | registry, mode=max | 22.9 s | source change | | registry, mode=max | 132.4 s | dependency bump | | local, mode=max | 74.7 s | source change | | local, mode=max | 132.1 s | dependency bump | _One new BuildKit builder per build; cache registry on the same droplet (loopback). Medians; the none bars are the cold build; run counts are in the tables. s-4vcpu-8gb, 2 November 2026._ The local cache: an expensive export type=local writes the cache as an OCI layout in a directory, and your CI saves and restores that directory between runs, for example with a cache action. In our runs the import side was cheap: reading the cache manifest took 0.1 s or less, and the cached layers came from local disk. The export side was the most expensive of all the backends. A cold build spent 39.1 s on it, and a one-line change spent 52.4 s, because each run wrote the full mode=max cache into a new, empty directory. The registry backends wrote their warm cache to a new tag in a registry that already held the layers, and finished in 1.4 s or less. We wrote to a fresh directory on purpose. When you export to the same directory you import from, BuildKit by default keeps every old blob. Docker's GitHub Actions cache guide puts it plainly: exporting to a directory that already holds a cache "leaves the blobs of the previous export behind, so the directory grows with every run". BuildKit 0.34.1 documents an export option for this, reset=true, which removes "any blobs in the cache directory that are not referenced by the current manifests in index.json" and is "useful for keeping the local cache directory from growing indefinitely". We planned to measure both the growth and reset=true, and ran out of droplet time first, so we cannot tell you how much either one changes. Which backend to use For this Dockerfile, on a runner that starts empty: - Use inline or registry mode=min. They gave the same warm builds as mode=max for a source change, cost 0 to 1.2 s to export, and stored no more than the image's own 47.7 MB of layers. Inline needs no extra tag at all, but it only works when you push the image, and its cache is only as old as the last image you pushed. - Use mode=max only when a builder step is slow to run and cheap to pull. BuildKit pulls and unpacks a cached layer before it can build on top of it. Here the toolchain step took about 10 s to run, and pulling its cached layers with the base image took about as long as pulling the base image and running it again. A long compile that produces a small output layer is the opposite case, and that is where mode=max earns its export cost. - For a slow Gradle or Bazel step, look past the layer cache. BuildKit reuses a RUN step whole or runs it again, so a source change before that step reruns the whole build. A tool's own remote build cache works per task and skips the tasks whose inputs did not change. Bitrise, for example, sells one for Gradle, Bazel, Xcode and React Native that it says works on any CI (checked on its public pages on 2 November 2026). We did not test one here. - Price the export, not just the hit. mode=max added about 30 s of compression to every cold build and 19.0 s to every dependency bump, before any upload. - Be careful with type=local. Writing to a fresh directory each run, the older way to keep the cache small, repeats the full mode=max export every time, which took 52.4 s on a one-line change. Docker's guide now reuses the directory with reset=true, which needs Buildx 0.35 or later and removes blobs that the new cache no longer references. We did not get to test it. To check your own Dockerfile, build it twice on a fresh builder with --progress=plain and read three lines per step: whether it says CACHED, how long its DONE takes after that (the pull), and how long preparing build cache for export takes. Those three numbers decide which backend wins. What we could not conclude - The cache registry was on loopback. Our export and import times leave out the network. With a remote registry, add the transfer of 47.7 MB for a source change with any backend, and of the 259.9 MB mode=max cache on every cold export that writes new layers. - One Dockerfile. In this one, the expensive work comes before COPY . ., so any cache skips it on a source change. In a Dockerfile whose expensive step comes after COPY . ., no backend can help with a source change. The Go frontend of the same demo is like that: its go build step comes after COPY . . and took 66.0 to 105.8 s in three uncached runs of it that we made on the same droplet for our CI build memory post. - No no-cache rerun of the changes. With no cache, a source change or a dependency bump is a cold build, so we used the cold median (79.0 s) as the reference and did not rerun it with the changed files. The cold builds ran earlier in the session, when steal was lower (8 to 27%, against 15 to 39% for the warm builds), so if anything that reference is a little fast. - Fewer warm runs than planned. Builds on the shared droplet ran slower than we planned, and we had a fixed budget. Cold builds have 5 runs per backend, source changes 3, and dependency bumps 3 (2 for the local cache). The direction of every comparison we draw held in every round; the exact medians would move with more runs. - Fixed order inside each round. The backends ran in the same order in every round, so a backend's position could bias its time. The host-wide steal we logged per build does not show a pattern by position, but it rose over the session (from 8% to 39% per build). Dependency round 3 ran in the opposite order, mode=max first, and mode=max was still the slowest registry backend. We point at the step timings because they depend less on any of this. - The local result is about a fresh destination. We exported each local warm build to a new, empty directory, while the registry backends exported next to layers that were already stored. That shows that a fresh local destination is expensive, not that type=local is slow in general, and we did not measure the cache action's save and restore of the directory. - Untested: type=gha and type=s3, RUN --mount=type=cache cache mounts, two pipelines writing the same cache tag at once, and the reset=true option for local caches. Run it yourself Create a 4 vCPU Ubuntu machine with Docker, copy the files below to /root/bench/, and run: bash /root/bench/registries.sh STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh prep # contexts for the source and dependency changes STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh cold # 5 rounds x 5 backends STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh warm # 5 source rounds, then dependency rounds while time allows python3 summarize_p2.py /root/res/p2 The script below is the one we ran. Our run did not use warm to the end: to fit the budget we stopped it after three source rounds, ran two dependency rounds with warm2, and a third with dep3, which starts with mode=max. packnow (a tar and zstd test of the local cache) and grow (the same directory in and out, with and without reset=true) are in the script too; we ran out of time before them. Each build writes its own directory with the BuildKit log, the registry access log for that build, and the sizes, so you can check any single number. #registries.sh: cache registry, pull-through mirrors, pinned source #!/bin/bash # The cache registry and the two pull-through mirrors, all on the build machine. docker pull -q moby/buildkit:v0.34.1 docker pull -q registry:2.8.3 docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3 docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3 docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3 git init -q /root/src/msd git -C /root/src/msd remote add origin https://github.com/GoogleCloudPlatform/microservices-demo git -C /root/src/msd fetch -q --depth 1 origin c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712 git -C /root/src/msd checkout -q FETCH_HEAD #buildkitd.toml: base image pulls go to the local mirror [registry."docker.io"] mirrors = ["127.0.0.1:5001"] [registry."gcr.io"] mirrors = ["127.0.0.1:5002"] [registry."127.0.0.1:5000"] http = true insecure = true [registry."127.0.0.1:5001"] http = true insecure = true [registry."127.0.0.1:5002"] http = true insecure = true #p2.sh: one fresh builder per build, every backend and scenario #!/bin/bash # Post 2: BuildKit cache backends on a fresh builder per build. # usage: p2.sh prep | smoke | cold | pack | warm | grow | warmnone set -u export PATH=/usr/local/bin:$PATH B=/root/bench R=/root/res/p2 C=/root/p2 SVC=/root/src/msd/src/paymentservice # stop starting new builds after this time, to stay inside a machine budget (our last stages used 2026-11-02T18:50:00Z) STOP_AT=$(date -u -d "${STOP_AT_UTC:-tomorrow}" +%s) REG=127.0.0.1:5000 BK=moby/buildkit:v0.34.1 mkdir -p $R $C/cache log() { echo "$(date -u +%FT%TZ) $*" | tee -a $R/progress.log; } prep() { rm -rf $C/ctx-* cp -a $SVC $C/ctx-base cp -a $SVC $C/ctx-src sed -i 's/PaymentService gRPC server started on port/PaymentService gRPC server listening on port/' $C/ctx-src/server.js cp -a $SVC $C/ctx-dep ( cd $C/ctx-dep && npm install --package-lock-only --save-exact --ignore-scripts --no-audit --no-fund @grpc/grpc-js@1.14.7 ) for i in 1 2 3 4 5; do cp -a $SVC $C/ctx-grow$i sed -i "s/PaymentService gRPC server started on port/PaymentService gRPC server (build $i) started on port/" $C/ctx-grow$i/server.js done diff -u $SVC/server.js $C/ctx-src/server.js > $R/src-change.diff diff -u $SVC/package.json $C/ctx-dep/package.json > $R/dep-change.diff diff -u $SVC/package-lock.json $C/ctx-dep/package-lock.json >> $R/dep-change.diff grep -c "listening on port" $C/ctx-src/server.js head -60 $R/dep-change.diff } # build CFG SCEN RUN [CTX] build() { local cfg=$1 scen=$2 r=$3 ctx=${4:-} local out=$R/$scen/$cfg/$r name=p2-$cfg-$scen-$r if [$(date +%s) -gt $STOP_AT]; then log "$scen $cfg $r SKIPPED (time budget)"; return; fi rm -rf $out; mkdir -p $out [-z "$ctx"] && case $scen in cold) ctx=$C/ctx-base ;; src) ctx=$C/ctx-src ;; dep) ctx=$C/ctx-dep ;; esac local img=$REG/paymentservice:$cfg-$scen from=() to=() case $cfg in none) ;; inline) to=(--cache-to type=inline) if [$scen = cold]; then img=$REG/paymentservice:inline-base else from=(--cache-from type=registry,ref=$REG/paymentservice:inline-base); fi ;; regmin|regmax) local m=${cfg#reg} if [$scen = cold]; then to=(--cache-to type=registry,ref=$REG/cache:$m,mode=$m) else from=(--cache-from type=registry,ref=$REG/cache:$m); to=(--cache-to type=registry,ref=$REG/cache:$m-scratch,mode=$m); fi ;; local) if [$scen = cold]; then rm -rf $C/cache/local; to=(--cache-to type=local,dest=$C/cache/local,mode=max) else rm -rf $C/cache/scratch; from=(--cache-from type=local,src=$C/cache/local); to=(--cache-to type=local,dest=$C/cache/scratch,mode=max); fi ;; grow) from=(--cache-from type=local,src=$C/cache/grow); to=(--cache-to type=local,dest=$C/cache/grow,mode=max) ;; growr) from=(--cache-from type=local,src=$C/cache/growr); to=(--cache-to type=local,dest=$C/cache/growr,mode=max,reset=true) ;; esac sync; echo 3 > /proc/sys/vm/drop_caches local t0 t1 t2 t3 rc t0=$(date +%s.%N) docker buildx create --name $name --driver docker-container --driver-opt network=host \ --driver-opt image=$BK --buildkitd-config $B/buildkitd.toml --bootstrap > $out/create.log 2>&1 t1=$(date +%s.%N) local cpu0; cpu0=$(grep '^cpu ' /proc/stat) docker buildx build --builder $name --progress=plain --provenance=false --sbom=false \ --metadata-file $out/meta.json "${from[@]}" "${to[@]}" \ --output type=image,name=$img,push=true,registry.insecure=true $ctx > $out/build.log 2>&1 rc=$? t2=$(date +%s.%N) local cpu1; cpu1=$(grep '^cpu ' /proc/stat) docker buildx rm $name > /dev/null 2>&1 t3=$(date +%s.%N) docker logs --since $t1 --until $t2 reg-cache > $out/registry.log 2>&1 { echo "rc=$rc"; echo "t0=$t0"; echo "t1=$t1"; echo "t2=$t2"; echo "t3=$t3"; echo "ctx=$ctx"; echo "cpu0=$cpu0"; echo "cpu1=$cpu1"; } > $out/result.env case $cfg in local) if [$scen = cold]; then d=$C/cache/local; else d=$C/cache/scratch; fi echo "cache_bytes=$(du -sb $d | cut -f1)" >> $out/result.env echo "cache_blobs=$(ls $d/blobs/sha256 | wc -l)" >> $out/result.env [$scen = cold] || rm -rf $C/cache/scratch ;; grow|growr) echo "cache_bytes=$(du -sb $C/cache/$cfg | cut -f1)" >> $out/result.env echo "cache_blobs=$(ls $C/cache/$cfg/blobs/sha256 | wc -l)" >> $out/result.env ;; esac case $cfg-$scen in regmin-cold) python3 $B/regsize.py $REG cache min > $out/cache-size.json ;; regmax-cold) python3 $B/regsize.py $REG cache max > $out/cache-size.json ;; regmin-*) python3 $B/regsize.py $REG cache min-scratch > $out/cache-size.json ;; regmax-*) python3 $B/regsize.py $REG cache max-scratch > $out/cache-size.json ;; esac python3 $B/regsize.py $REG paymentservice ${img##*:} > $out/image-size.json log "$scen $cfg $r rc=$rc build=$(python3 -c "print(round($t2-$t1,1))")s" } pack_test() { for r in r1 r2 r3 r4 r5; do sync; echo 3 > /proc/sys/vm/drop_caches t0=$(date +%s.%N); tar -C $C/cache/local -cf - . | zstd -T0 -q -f -o $C/local.tar.zst; sync; t1=$(date +%s.%N) rm -rf $C/unpack; mkdir -p $C/unpack; sync; echo 3 > /proc/sys/vm/drop_caches t2=$(date +%s.%N); zstd -dc -q $C/local.tar.zst | tar -C $C/unpack -xf -; sync; t3=$(date +%s.%N) echo "$r pack_s=$(python3 -c "print(round($t1-$t0,2))") unpack_s=$(python3 -c "print(round($t3-$t2,2))") dir_bytes=$(du -sb $C/cache/local | cut -f1) tar_zst_bytes=$(stat -c %s $C/local.tar.zst)" | tee -a $R/pack.txt done rm -rf $C/unpack $C/local.tar.zst; log PACK-DONE } case ${1:-} in prep) prep ;; smoke) for c in none inline regmin regmax local; do build $c cold s0; done; log SMOKE-DONE ;; cold) for r in r1 r2 r3 r4 r5; do for c in none inline regmin regmax local; do build $c cold $r; done; done; log COLD-DONE ;; warm) # all source-change rounds first, then whole dependency rounds while they fit before STOP_AT for r in r1 r2 r3 r4 r5; do for c in inline regmin regmax local; do build $c src $r; done; done last=400 for r in r1 r2 r3 r4 r5; do t=$(date +%s) if [$((t + last)) -gt $STOP_AT]; then log "dep $r and later skipped: a round needs about ${last}s"; break; fi for c in inline regmin regmax local; do build $c dep $r; done last=$(( $(date +%s) - t )) done log WARM-DONE ;; warm2) # switched to after source round 3: dependency rounds first, then the last source rounds, while they fit last=400 for r in r1 r2 r3 r4 r5; do t=$(date +%s) if [$((t + last)) -gt $STOP_AT]; then log "dep $r and later skipped: a round needs about ${last}s"; break; fi for c in inline regmin regmax local; do build $c dep $r; done last=$(( $(date +%s) - t )) [$r = r3] && break done for r in r4 r5; do t=$(date +%s) if [$((t + 150)) -gt $STOP_AT]; then log "src $r skipped: time budget"; break; fi for c in inline regmin regmax local; do build $c src $r; done done log WARM-DONE ;; dep3) # third dependency round, started after round 2 (time budget) for c in regmax regmin inline local; do build $c dep r3; done log WARM-DONE ;; packnow) pack_test ;; warmnone) for r in r1 r2 r3 r4 r5; do for s in src dep; do build none $s $r; done; done; log WARMNONE-DONE ;; pack) log "pack deferred until after the warm builds" ;; grow) # local cache pack/unpack test, then the same-directory growth test, without and with reset=true if [$(date +%s) -lt $((STOP_AT + 120))]; then pack_test; fi rm -rf $C/cache/grow $C/cache/growr; cp -a $C/cache/local $C/cache/grow; cp -a $C/cache/local $C/cache/growr echo "start cache_bytes=$(du -sb $C/cache/grow | cut -f1) blobs=$(ls $C/cache/grow/blobs/sha256 | wc -l)" > $R/grow-start.txt for i in 1 2 3 4 5; do build grow src g$i $C/ctx-grow$i; build growr src g$i $C/ctx-grow$i; done; log GROW-DONE ;; *) echo "usage"; exit 2 ;; esac #regsize.py: compressed bytes behind a registry tag #!/usr/bin/env python3 """Sum the compressed layer bytes behind a registry tag (image or BuildKit cache).""" import json, sys, urllib.request reg, repo, tag = sys.argv[1:4] ACCEPT = ", ".join([ "application/vnd.oci.image.index.v1+json", "application/vnd.oci.image.manifest.v1+json", "application/vnd.docker.distribution.manifest.list.v2+json", "application/vnd.docker.distribution.manifest.v2+json", ]) def get(ref): req = urllib.request.Request(f"http://{reg}/v2/{repo}/manifests/{ref}", headers={"Accept": ACCEPT}) with urllib.request.urlopen(req, timeout=30) as r: return json.load(r) def walk(m, layers): for d in m.get("manifests", []): mt = d.get("mediaType", "") if "manifest" in mt or "index" in mt: walk(get(d["digest"]), layers) elif "layer" in mt: layers[d["digest"]] = d["size"] for d in m.get("layers", []): layers[d["digest"]] = d["size"] try: top = get(tag) except Exception as e: print(json.dumps({"error": str(e)})); sys.exit(0) layers = {} walk(top, layers) print(json.dumps({"tag": f"{repo}:{tag}", "mediaType": top.get("mediaType"), "layers": len(layers), "bytes": sum(layers.values()), "sizes": sorted(layers.values(), reverse=True)})) #summarize_p2.py: parse BuildKit logs, registry logs and sizes into medians #!/usr/bin/env python3 """Summarize post 2 runs: parse BuildKit plain progress, registry logs and sizes; medians per cell.""" import json, os, re, statistics, sys root = sys.argv[1] VERTEX = re.compile(r"^#(\d+) (.*)$") DONE = re.compile(r"^DONE ([\d.]+)s$") SUBDONE = re.compile(r"^(preparing layers for inline cache|preparing build cache for export|exporting layers|pushing layers|exporting manifest list|exporting config).*?([\d.]+)s done$") def env(path): d = {} if os.path.exists(path): for line in open(path): if "=" in line: k, v = line.strip().split("=", 1) d[k] = v return d def parse_build(path): v = {} order = [] for line in open(path, errors="replace"): line = line.rstrip("\n") m = VERTEX.match(line) if not m: continue vid, rest = int(m.group(1)), m.group(2) if vid not in v: v[vid] = {"name": rest, "done": None, "cached": False, "error": False, "sub": {}} order.append(vid) continue if rest == "CACHED": v[vid]["cached"] = True elif rest.startswith("ERROR"): v[vid]["error"] = True else: d = DONE.match(rest) if d: v[vid]["done"] = float(d.group(1)) s = SUBDONE.match(rest) if s: v[vid]["sub"][s.group(1)] = v[vid]["sub"].get(s.group(1), 0) + float(s.group(2)) return [v[i] for i in order] def registry_bytes(path): got = put = 0 seen_logrus = False lines = open(path, errors="replace").read().splitlines() if os.path.exists(path) else [] for line in lines: if 'msg="response completed"' in line: seen_logrus = True meth = re.search(r"http\.request\.method=(\w+)", line) uri = re.search(r'http\.request\.uri="([^"]+)"', line) w = re.search(r"http\.response\.written=(\d+)", line) if meth and uri and w and meth.group(1) == "GET" and "/blobs/" in uri.group(1): got += int(w.group(1)) if not seen_logrus: for line in lines: m = re.search(r'"GET (/v2/[^]*/blobs/[^]*) HTTP/[\d.]+" (\d+) (\d+)', line) if m and m.group(2) == "200": got += int(m.group(3)) return got def step_time(vs, pattern): for x in vs: if re.search(pattern, x["name"]): return 0.0 if x["cached"] else x["done"] return None def run(d): r = env(os.path.join(d, "result.env")) if not r: return None vs = parse_build(os.path.join(d, "build.log")) steps = [x for x in vs if re.match(r"^\[(builder|stage-\d+) \d+/\d+\]", x["name"])] exp = [x for x in vs if x["name"].startswith("exporting cache to")] img = [x for x in vs if x["name"] == "exporting to image"] imp = [x for x in vs if x["name"].startswith("importing cache manifest")] info = { "rc": int(r["rc"]), "bootstrap_s": round(float(r["t1"]) - float(r["t0"]), 2), "build_s": round(float(r["t2"]) - float(r["t1"]), 2), "steps": len(steps), "cached_steps": sum(1 for x in steps if x["cached"]), "cached_names": [x["name"] for x in steps if x["cached"]], "apk_builder_s": step_time(vs, r"RUN apk add --update"), "npm_install_s": step_time(vs, r"RUN npm install"), "apk_final_s": step_time(vs, r"RUN apk add --no-cache nodejs"), "copy_modules_s": step_time(vs, r"COPY --from=builder"), "export_cache_s": sum(x["done"] or 0 for x in exp) if exp else None, "export_image_s": img[0]["done"] if img else None, "inline_prep_s": img[0]["sub"].get("preparing layers for inline cache") if img else None, "prep_export_s": sum(x["sub"].get("preparing build cache for export", 0) for x in exp) if exp else None, "import_manifest_s": sum(x["done"] or 0 for x in imp) if imp else None, "pulled_from_cache_registry_bytes": registry_bytes(os.path.join(d, "registry.log")), "errors": [x["name"] for x in vs if x["error"]], # a CACHED step still reports DONE time when its layers must be pulled and unpacked for a later step "cached_pull_s": max([x["done"] or 0 for x in steps if x["cached"]] or [0]), } if "cpu0" in r and "cpu1" in r: a = [int(x) for x in r["cpu0"].split()[1:9]] b = [int(x) for x in r["cpu1"].split()[1:9]] dd = [y - x for x, y in zip(a, b)] info["steal_pct"] = round(100 * dd[7] / sum(dd), 1) if sum(dd) else None if "cache_bytes" in r: info["cache_bytes"] = int(r["cache_bytes"]) info["cache_blobs"] = int(r["cache_blobs"]) for f in ("cache-size.json", "image-size.json"): p = os.path.join(d, f) if os.path.exists(p): try: j = json.load(open(p)) info[f.split(".")[0].replace("-", "_") + "_bytes"] = j.get("bytes") info[f.split(".")[0].replace("-", "_") + "_layers"] = j.get("layers") except Exception: pass return info def med(xs): xs = [x for x in xs if x is not None] return round(statistics.median(xs), 2) if xs else None out = {} for scen in ("cold", "src", "dep"): for cfg in ("none", "inline", "regmin", "regmax", "local", "grow", "growr"): d0 = os.path.join(root, scen, cfg) if not os.path.isdir(d0): continue runs = {rn: run(os.path.join(d0, rn)) for rn in sorted(os.listdir(d0))} runs = {k: v for k, v in runs.items() if v} meas = [v for k, v in runs.items() if k.startswith("r")] keys = ["build_s", "bootstrap_s", "cached_steps", "apk_builder_s", "npm_install_s", "apk_final_s", "copy_modules_s", "export_cache_s", "export_image_s", "inline_prep_s", "prep_export_s", "import_manifest_s", "cached_pull_s", "pulled_from_cache_registry_bytes", "steal_pct", "cache_bytes", "cache_size_bytes", "image_size_bytes"] cell = {"n": len(meas), "rcs": [x["rc"] for x in meas]} for k in keys: cell[k] = med([x.get(k) for x in meas]) vals = [x.get(k) for x in meas if x.get(k) is not None] if vals and k in ("build_s", "export_cache_s", "import_manifest_s"): cell[k + "_range"] = [min(vals), max(vals)] cell["runs"] = runs out[f"{scen}/{cfg}"] = cell json.dump(out, open(os.path.join(root, "summary.json"), "w"), indent=1) MB = 1e6 print(f"{'cell':14} {'n':>2} {'build':>7} {'range':>14} {'cached':>6} {'apkB':>6} {'npm':>6} {'expC':>6} {'expI':>6} {'impM':>5} {'pullMB':>7} {'cacheMB':>8} {'imgMB':>6}") for k, c in out.items(): cb = c.get("cache_size_bytes") or c.get("cache_bytes") print(f"{k:14} {c['n']:>2} {str(c['build_s']):>7} {str(c.get('build_s_range')):>14} {str(c['cached_steps']):>6} {str(c['apk_builder_s']):>6} {str(c['npm_install_s']):>6} {str(c['export_cache_s']):>6} {str(c['export_image_s']):>6} {str(c['import_manifest_s']):>5} {round((c['pulled_from_cache_registry_bytes'] or 0)/MB,1):>7} {round(cb/MB,1) if cb else '-':>8} {round(c['image_size_bytes']/MB,1) if c['image_size_bytes'] else '-':>6} rcs={c['rcs']}") Summary A fresh CI runner has no BuildKit cache, so the backend you pick decides what gets reused. For this multi-stage Node service, the cheapest backends did the most: inline and registry mode=min cut a one-line rebuild from 79.0 s to about 20 s and cost almost nothing to export, because BuildKit could skip the whole builder stage by matching cache keys alone. mode=max stored five times as much, added about 30 s of compression to every cold build, and did not make the dependency bump faster: pulling the cached toolchain took as long as installing it, and the export came on top. Before you switch on mode=max because a guide says so, read --progress=plain for your own build and compare the pull time of each cached step with the time it takes to run. _Originally published at devops-daily.com._

How Much Memory Does a CI Build Need? Five Real Builds at 2, 4 and 8 GB --- title: How Much Memory Does a CI Build Need? Five Real Builds at 2, 4 and 8 GB published: true date: 2026-11-02 18:00:00 UTC tags: CICD,Memory,cgroups,Docker canonical_url: devops-daily.com --- We ran five real open-source builds (a Vite app build, a TypeScript type check, a Gradle build, a Go build and a Docker image build) inside 2, 4 and 8 GB memory limits on one 4 vCPU machine, with Gradle both with and without its daemon, two or three times at each limit. Four of the five finished inside 2 GB. The one that did not was the Excalidraw web app: its Vite production build also runs a full TypeScript check, together they held 3,248 MiB of anonymous memory, and the kernel killed the build at 2 GB in all 3 runs. The number that most tools show you was the wrong one for the other four. cgroup v2's memory.peak said the GitHub CLI's Go build peaked at 1,775 MiB, but only 424 MiB of that was anonymous memory, the heaps and stacks of its processes. Most of the rest was page cache, which the kernel drops when it needs the room, plus some kernel memory. A Docker image build looked like 2,072 MiB for the same reason. And a TypeScript type check that held 1,139 MiB with the whole machine available finished inside 2 GB with 970 MiB, because Node sizes its heap limit from the cgroup limit it finds. This post shows the numbers, the five traps behind them, and every script we used, so you can size a runner from your own builds instead of from a guess. TL;DR - Four of five builds fit in 2 GB. Only the Excalidraw Vite build needed more: 3,248 MiB of anonymous memory, killed at 2 GB, fine at 4 GB. - memory.peak counts page cache. The Go build and the Docker build showed 1,775 and 2,072 MiB, but their anonymous memory peaked at 424 and 355 MiB. Size from anonymous memory plus a margin, then confirm by running the build inside that limit. - Runtimes size their heaps from the cgroup limit. Inside 2 GB, Node's default heap limit was 1,120 MiB and the JVM's default maximum heap was 512 MiB. The type check used less memory at 2 GB than at 8 GB and still finished, about 24% slower. - The Gradle daemon keeps its memory after the build. Five seconds after gradlew exited, the daemon still held 1,048 MiB of anonymous memory. With --no-daemon, that dropped to under 1 MiB, and in our one-command runs the daemon gave no repeatable speed-up. - Limiting the docker CLI did not limit the build. The CLI's cgroup peaked at 40 MiB while the BuildKit container that did the work peaked at 2,072 MiB. - Wall times on a shared-CPU droplet are noisy: the CPUs reported 23.5% steal time across the runs we tracked. Trust the memory columns more than the time columns. Prerequisites - A Linux machine with cgroup v2 and systemd (any recent Ubuntu, Debian or Fedora). Check with stat -fc %T /sys/fs/cgroup, which prints cgroup2fs. - Root access, for systemd-run --scope and for dropping the page cache. - The toolchains for the builds you want to test. Our setup script installs Go, Node, Yarn, a JDK, Rust and Docker. - If you want to rerun our matrix as it is: a 4 vCPU, 8 GB machine and about two hours. We used a droplet on DigitalOcean (referral link). How we measured We used one DigitalOcean droplet, size s-4vcpu-8gb in fra1, with Ubuntu 24.04 and kernel 6.8. The kernel reports 7,941 MiB of memory and there is no swap. Every build ran in its own transient systemd scope: systemd-run --scope --slice=bench.slice --unit=p1-<build>-<limit>-<run> \ -p MemoryMax=2G -p MemorySwapMax=0 -p AllowedCPUs=0-3 -p OOMPolicy=continue \ /root/bench/wrap.sh <outdir> <timeout> <build command> These are cgroup limits on one droplet, not three real machine sizes. A real 2 GB runner has a smaller kernel footprint, a runner agent and often a Docker daemon in the same memory, so it has less room than our 2 GB scope. The 8 GB limit is above the 7.9 GiB that the droplet has, so the 8 GB column means "the whole machine". The rest of the setup: - CPU. Every scope was pinned to the same 4 vCPUs with AllowedCPUs=0-3, and nproc printed 4 inside each one. Compilers start about one job per CPU, so a 2 vCPU runner would use less memory and an 8 vCPU runner more. We did not measure other CPU counts. - Cold build, warm dependencies. We downloaded dependencies once before the runs: yarn install, go mod download, cargo fetch, the Gradle distribution and dependencies, and the base images for the Docker build. Before every run we deleted the build output and the compiler caches (excalidraw-app/build, GOCACHE, target/, Gradle's build/ and .gradle/), stopped any Gradle daemon, and dropped the page cache. That is a CI job that restores a dependency cache but not a build cache. - What we recorded. Wall time, the exit code, memory.peak at the end of the run, memory.events (how often the limit was hit, and OOM kills), and a sampler that read memory.stat every 100 ms for anonymous memory (anon) and page cache (file). - Repeats. Runs went in round-robin order: run 1 of every cell, then run 2, then run 3, so a slow period on the shared host does not land on one build. Builds ran slower than we planned, so to stay inside our droplet budget the third round covered only the 8 GB and 2 GB limits. That gives 3 runs per cell at 8 GB and 2 GB, and 2 at 4 GB. The tables show the median of 3, or the mean of 2 at 4 GB. Each build also had one warm-up run at 8 GB that we did not count; it downloaded Gradle and filled the image mirror. The builds, all pinned to exact commits: | Build | Command | Source | | --- | --- | --- | | Excalidraw app, Vite production build | yarn build:app:docker | excalidraw/excalidraw 8bcdca45 | | Excalidraw type check, 1,508 files | tsc -p tsconfig.json (noEmit is set) | same checkout | | PlantUML, Gradle 9.3.1 on JDK 21 | ./gradlew jar, daemon on and --no-daemon | plantuml/plantuml db06d70e | | GitHub CLI, Go 1.27.3 | go build ./cmd/gh | cli/cli 9490d204 | | Online Boutique frontend image, BuildKit 0.34.1 | docker buildx build | GoogleCloudPlatform/microservices-demo c7cb6d5b | | ripgrep 15.2.0, Rust 1.99.1 (one run) | cargo build --release --locked | BurntSushi/ripgrep tag 15.2.0 | Node was 24.22.2 with Yarn 1.22.22, Docker was 29.9.2 with Buildx 0.38.1. The results | Build | Anonymous peak at 8 GB | memory.peak at 8 GB | 8 GB | 4 GB | 2 GB | | --- | --- | --- | --- | --- | --- | | Excalidraw vite build | 3,248 MiB | 3,578 MiB | ok, 66 s | ok, 83 s | 3 of 3 OOM-killed (run 1 from the kernel log) | | Excalidraw tsc type check | 1,139 MiB | 1,271 MiB | ok, 59 s | ok, 62 s | ok, 73 s | | PlantUML Gradle, daemon | 1,143 MiB | 1,474 MiB | ok, 124 s | ok, 110 s | ok, 113 s | | PlantUML Gradle, --no-daemon | 1,144 MiB | 1,475 MiB | ok, 88 s | ok, 119 s | ok, 100 s | | GitHub CLI go build | 424 MiB | 1,775 MiB | ok, 141 s | ok, 198 s | ok, 169 s | | Online Boutique frontend, docker buildx build | 355 MiB | 2,072 MiB | ok, 131 s | ok, 129 s | ok, 110 s | Times are medians of 3 runs at 8 GB and 2 GB, and the mean of 2 runs at 4 GB. Memory columns are from the 8 GB runs. What the cgroup counter says vs what the build held (8 GB limit, median of 3) | | Value | Series | | --- | --- | --- | | Vite build | 3578 MiB | memory.peak | | Vite build | 3248 MiB | anonymous memory | | tsc check | 1271 MiB | memory.peak | | tsc check | 1139 MiB | anonymous memory | | Gradle (daemon) | 1474 MiB | memory.peak | | Gradle (daemon) | 1143 MiB | anonymous memory | | Gradle (no daemon) | 1475 MiB | memory.peak | | Gradle (no daemon) | 1144 MiB | anonymous memory | | Go build | 1775 MiB | memory.peak | | Go build | 424 MiB | anonymous memory | | Docker build | 2072 MiB | memory.peak | | Docker build | 355 MiB | anonymous memory | _memory.peak includes page cache; anonymous memory is the sampled peak of anon in memory.stat (100 ms). One s-4vcpu-8gb droplet, 4 vCPUs, 2 November 2026._ Three things stand out. Only one build reached the 2 GB limit at all. The max counter in memory.events, which counts how often a cgroup's usage was about to go over its limit, stayed at 0 for every other run. Memory use at 4 GB and at 8 GB was almost the same for every build, so the extra 4 GB bought nothing. And the gap between memory.peak and anonymous memory is large for exactly the builds that read and write many files: the Go compiler and BuildKit. The single Rust build, ripgrep 15.2.0 with cargo build --release, fits the same pattern: 896 MiB memory.peak, 572 MiB anonymous, 61.1 s. We ran it once at 8 GB only, because the full matrix did not fit our time budget. Trap 1: memory.peak counts page cache memory.peak is the highest value of memory.current that the cgroup reached. memory.current charges the cgroup for everything it caused to be in memory, and that includes page cache : the source files the compiler read, the object files it wrote, the layers BuildKit unpacked. Page cache is reclaimable. When the cgroup gets close to its limit, the kernel drops clean cache pages and writes back dirty ones before it ever considers killing a process. Anonymous memory is the part that cannot go anywhere without swap: heaps, stacks, the compiler's working data. memory.stat reports it as anon. That, plus a smaller amount of kernel memory (37 to 136 MiB in these builds), is what has to fit. Our sampler read it every 100 ms, so it can miss a shorter spike. memory.peak and the max counter cannot miss one, and they show that no build except Vite at 2 GB ever reached its limit. The safe test is still the one we ran: the build inside the limit you plan to buy. The Go build of the GitHub CLI shows the difference well. Its peak was 1,775 MiB. Anonymous memory never went over 424 MiB; about 1,400 MiB was page cache from the module sources and the build cache that go build writes. At the 2 GB limit, the same build finished with the same peak, because 1,775 MiB is still under 2,048 MiB. With a smaller limit the kernel may reclaim that cache instead of failing the build, but we did not test below 2 GB, so test the lower limit before you rely on it. So if a dashboard shows that your CI job "peaked at 1.8 GB", that does not mean a 2 GB runner is too small. Look at anon in memory.stat (or RSS in your container metrics), and then add a margin for the runner itself. Trap 2: a production build that is also a type check The Excalidraw build needed far more memory than anything else: 3,248 MiB of anonymous memory, against 1,139 MiB for a full type check of the same code. The reason is in its Vite config: checker({ typescript: true, eslint: envVars.VITE_APP_ENABLE_ESLINT === "false" ? undefined : { lintCommand: 'eslint "./**/*.{js,ts,tsx}"' }, // ... }), vite-plugin-checker runs the TypeScript checker during vite build too: in version 0.7.2, which Excalidraw pins, enableBuild defaults to true. The production .env turns ESLint off but leaves the type check on, and the build also writes source maps. So one vite build is Rollup, source maps and a full tsc run at the same time. At 2 GB the kernel killed it in all 3 runs, 69 to 122 seconds in, after the log said ✓ 1442 modules transformed. The kernel log names the cgroup and the process it chose: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=p1-vite-2G-r2.scope,mems_allowed=0,oom_memcg=/bench.slice/p1-vite-2G-r2.scope,task_memcg=/bench.slice/p1-vite-2G-r2.scope,task=MainThread,pid=77242,uid=0 Memory cgroup out of memory: Killed process 77242 (MainThread) total-vm:11451308kB, anon-rss:1138988kB, file-rss:4864kB, shmem-rss:0kB, UID:0 pgtables:29992kB oom_score_adj:0 Our wrapper's record of that run shows the kill. memory.events counted 39,825 times that the cgroup was about to go over its limit, and one OOM kill: vite build, 2 GB limit, run 2 $ cat /root/res/p1/vite/2G/r2/result.env rc=1 t0=1793637034.083018067 t1=1793637156.285208072 peak=2147483648 current_after=867889152 anon_after=801075200 file_after=38674432 $ cat /root/res/p1/vite/2G/r2/memory.events low 0 high 0 max 39825 oom 1 oom_kill 1 oom_group_kill 0 The process list right after the kill shows the second half of the build. The checker runs tsc as its own child process, and it was still running after Vite had died: 77265 1808 /bin/sh -c tsc --noEmit 77266 799108 node /root/src/excalidraw/node_modules/.bin/tsc --noEmit That tsc held 799,108 KiB (780 MiB) of resident memory under the 2 GB limit. Yarn reported Command failed with exit code 1, but the type check kept going until our harness stopped the scope. On a real runner it would keep using memory and CPU until the job ends. We did not split the 3,248 MiB peak at 8 GB between Rollup and the checker, so we cannot say exactly how much the type check adds. What we can say: if your bundler runs a checker, your build step needs the memory of both, and moving the type check to its own CI job is the first thing to try before you buy a bigger runner. Trap 3: runtimes size their heaps from the limit Node and the JVM both read the cgroup limit when they start and size their default heap from it. Our harness asks each runtime what it would use inside each limit: probe inside a 2 GB scope # what Node 24 and JDK 21 pick inside MemoryMax=2G (from p1.sh probe) $ cat /root/res/p1/probe-2G.txt nproc=4 memory.max=2147483648 node_heap_size_limit=1174405120 int ActiveProcessorCount = -1 {product} {default} size_t MaxHeapSize = 536870912 {product} {ergonomic} double MaxRAMPercentage = 25.000000 {product} {default} All four places side by side: | Inside | Node 24 heap_size_limit | JDK 21 default MaxHeapSize | | --- | --- | --- | | 2 GB scope | 1,120 MiB | 512 MiB | | 4 GB scope | 2,240 MiB | 1,024 MiB | | 8 GB scope | 2,240 MiB | 2,048 MiB | | whole droplet, no scope | 2,240 MiB | 1,986 MiB | Two consequences showed up in the runs. Heap use follows the limit. tsc --extendedDiagnostics prints Memory used, which is Node's heapUsed at the end of the run, in units of 1,000 bytes. With the whole machine, the type check of 1,508 files reported Memory used: 1018378K (median of 3, about 971 MiB). Inside 2 GB it reported 886903K (about 846 MiB), anonymous memory fell from 1,139 to 970 MiB, and the check finished about 24% slower (median 72.7 s against 58.8 s). The 2 GB run was 9.5 to 14.2 seconds slower in each of the three rounds. In rounds 2 and 3, where we logged the CPU counters, its steal time was only 4 to 7 points higher, so steal explains only part of the gap. More frequent garbage collection under the smaller heap limit would explain the rest, but we did not record GC activity, so that is a hypothesis. A big heap number at a high limit says little about what the program needs. We expected this check to fail at 2 GB with a heap error, and it did not. Node stopped at 2,240 MiB even with more memory. On our 8 GB machine, Node's default heap limit was 2,240 MiB with a 4 GB scope, an 8 GB scope and no scope at all. We did not test a bigger machine, so check what your runner gives you with node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)'. If a build reports JavaScript heap out of memory, that is Node's own limit, not the kernel's. Set NODE_OPTIONS=--max-old-space-size=<MiB> explicitly instead of only buying more memory. The JVM takes 25% of the limit by default (MaxRAMPercentage=25), so a Maven build or a test JVM without -Xmx gets 512 MiB inside 2 GB. Gradle sets its own daemon heap, which brings us to the next trap. Trap 4: the Gradle daemon outlives the build PlantUML's gradle.properties sets org.gradle.jvmargs=-Xmx1g. That cap kept the build inside 2 GB: anonymous memory peaked at about 1,143 MiB with the daemon and 1,144 MiB without it, and neither came near the limit. The difference is what happens after gradlew exits. With the daemon, which is the default, the JVM stays alive and keeps its heap. Five seconds after the build, the scope still held 1,378 MiB, 1,048 MiB of it anonymous, in one process: 37292 1105976 /usr/lib/jvm/java-21-openjdk-amd64/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED ... With --no-daemon, Gradle printed To honour the JVM settings for this build a single-use Daemon process will be forked and Daemon will be stopped at the end of the build, and five seconds after exit the anonymous memory in the scope was under 1 MiB. Anonymous memory around the end of the Gradle build (8 GB limit, run 1) | | -55 s | -50 s | -45 s | -40 s | -35 s | -30 s | -25 s | -20 s | -15 s | -10 s | -5 s | 0 s | +5 s | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | daemon (default) | 637 MiB | 857 MiB | 927 MiB | 982 MiB | 1053 MiB | 1059 MiB | 1066 MiB | 1073 MiB | 1082 MiB | 1110 MiB | 1115 MiB | 1143 MiB | 1054 MiB | | --no-daemon | 476 MiB | 516 MiB | 559 MiB | 807 MiB | 921 MiB | 970 MiB | 1067 MiB | 1072 MiB | 1080 MiB | 1084 MiB | 1112 MiB | 1144 MiB | 0 MiB | _Max of 100 ms samples in each 5 s bin, labelled by the end of the bin; 0 s is the moment gradlew exits, +5 s is the 5 s after. One run each, PlantUML jar task, JDK 21, Gradle 9.3.1._ On a CI runner, that leftover daemon sits in the same memory as every later step of the job: the Docker build, the integration tests, the upload. At 8 GB the median time was 123.5 s with the daemon and 87.5 s without (112.7 s and 99.7 s at 2 GB), The ranges overlap and CPU steal was high, so we see no repeatable speed-up from the daemon for one Gradle command per job. The daemon is built to pay off across several Gradle commands in the same job, which we did not test. If a job runs one Gradle command, use --no-daemon (or org.gradle.daemon=false). If it runs several, consider keeping the daemon (we did not test that case), and run ./gradlew --stop before the steps that need the memory. Trap 5: a limit on the docker CLI does not limit the build docker build is a client. In our setup the work ran in a BuildKit container (Buildx's docker-container driver), which has its own cgroup. Other drivers and CI executors lay out cgroups differently, but the CLI is only a client in all of them. We wrapped the docker buildx build command in the same kind of scope as everything else, and at every limit that scope peaked at 39 to 42 MiB: the CLI, streaming logs. To measure the build, we put the limit on the BuildKit container itself: docker buildx create --name p1d8gr1 --driver docker-container \ --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \ --driver-opt memory=8g --driver-opt memory-swap=8g --driver-opt cpuset-cpus=0-3 \ --buildkitd-config buildkitd.toml --bootstrap and read memory.peak from /sys/fs/cgroup/system.slice/docker-<container id>.scope. That container peaked at 2,072 MiB, of which only 355 MiB was anonymous: the Go compiler inside the build. Most of the rest was page cache from the image layers and the build's output, plus 136 MiB of kernel memory. This matters for runner sizing in two ways. Your CI's memory graph may not show the build at all, depending on whether it measures the job's process tree or the whole machine. And if a self-hosted runner limits only the job's process tree, a Docker build in that job can run outside the limit, so a "2 GB job" can push the host far past 2 GB. Check which cgroup your executor puts the Docker daemon in. Two method bugs that cost us a run The first 2 GB Vite run left no result at all. The kernel killed the Vite process, and then systemd stopped the whole scope, including our wrapper script, before it could write anything. The journal said so: p1-vite-2G-r1.scope: A process of this unit has been killed by the OOM killer. p1-vite-2G-r1.scope: Failed with result 'oom-kill'. systemd 255 sets DefaultOOMPolicy=stop, and scopes follow it. We set DefaultOOMPolicy=continue before the next 2 GB run (the published p1.sh also passes -p OOMPolicy=continue), and checked with a 64 MiB scope that the wrapper now survives a kill and records oom_kill 1. Runs 2 and 3 at 2 GB have full records. Then our rerun of run 1 failed too, for a different reason. systemd keeps a failed transient unit loaded, so starting a new scope with the same name was refused: Failed to start transient scope unit: Unit p1-vite-2G-r1.scope was already loaded or has a fragment file. By the time we saw that, the Docker cache tests for the next post were running on the same droplet, so we did not try again. Run 1 still counts as a kill: its kernel log line reads oom_memcg=/bench.slice/p1-vite-2G-r1.scope,...,task=MainThread,pid=36041, and our sampler, which runs outside the scope, saw the cgroup reach 2,048 MiB, with up to 1,985 MiB anonymous, before its last sample at 70.5 seconds. The published redo mode now runs systemctl reset-failed first. If you measure builds this way, set OOMPolicy=continue and give each run a unique unit name, or the runs that you care about most, the ones that fail, will be the ones with no data. A sizing table From these runs, for a 4 vCPU runner. The "smallest limit that passed" column is measured. The last column is our rule of thumb, not a measurement: a real VM also holds the OS, the runner agent and any service containers, so we pick the smallest size that leaves at least 1 GB above the anonymous peak. | Build like ours | Anonymous peak | Smallest limit that passed every run | Runner we would pick | | --- | --- | --- | --- | | Bundler that also type-checks (Excalidraw vite build) | 3,248 MiB | 4 GB | 8 GB (moving the type check to its own job may make 4 GB enough; we did not test that) | | Large type check on its own (tsc, 1,508 files) | 1,139 MiB at 8 GB, 970 MiB at 2 GB | 2 GB | 4 GB | | Gradle with a 1 GB daemon heap (PlantUML jar) | 1,143 MiB | 2 GB | 4 GB, with --no-daemon | | Go build of a mid-size CLI (gh) | 424 MiB | 2 GB | 2 GB | | Docker build of a Go service | 355 MiB inside BuildKit | 2 GB | 2 GB, plus what the Docker daemon and the other steps need | Hosted CI services list their machines by vCPU and memory, so you can match a build to a size. Bitrise, for example, lists every Linux machine type on its pricing page with its vCPU count and memory, such as 2 vCPU with 8 GB, and also sells a remote build cache for Gradle, Bazel and Xcode (checked on its public pages on 2 November 2026). Whatever you use, measure anonymous memory first, then pick the size. What we could not conclude - Three limits on one machine are not three machines. A real 2 GB runner has less page cache to spare, its own kernel, and a runner agent in the same memory. Leave a margin above the anonymous peak; our sizing table uses 1 GB. - CPU count changes memory. Every run had 4 vCPUs. The Go compiler, Rollup's workers and Gradle's workers scale with CPUs, so these numbers are not valid for 2 or 16 vCPUs. - Wall times are noisy. The droplet reported 23.5% CPU steal time across the 33 runs that our CPU counter log covered, and between 3.7% and 35.5% in single runs. The Go build took between 121.3 and 199.6 seconds at limits it never came close to. We do not read anything into a time difference between limits unless it repeats in every round, as the tsc gap did. - Two or three runs per cell is enough to see a failure repeat and a peak hold steady, and that is all. The anonymous peaks varied by up to 21% for the Go build (353 to 441 MiB at 8 GB) and up to 15% for Gradle, and 6% or less for Vite, tsc and Docker between runs of the same cell. - We did not split shared builds. We cannot say how much of the Vite build's memory is Rollup and how much is the checker, or how much of the Docker build's page cache is layer extraction. - Rust ran once. One ripgrep build at 8 GB, so no repeats and no 2 GB or 4 GB data. Run it yourself Create a 4 vCPU, 8 GB Ubuntu 24.04 machine, copy the files below to /root/bench/, and run: bash /root/bench/setup.sh bash /root/bench/p1.sh warmup # probes the runtimes, one unmeasured run per build bash /root/bench/p1.sh matrix # 3 rounds x 3 limits x each build (we stopped it after round 2) bash /root/bench/p1.sh round3 # what we ran as round 3: 8 GB and 2 GB only bash /root/bench/p1.sh redo # reruns any cell that left no result python3 summarize_p1.py /root/res/p1 Each run writes its own directory with result.env, memory.events, samples.json and the build log, so you can read any single run on its own. #setup.sh: toolchains, pinned sources, dependencies, registries #!/bin/bash # Droplet setup: toolchains, pinned sources, dependency prefetch, local registries. set -euxo pipefail export DEBIAN_FRONTEND=noninteractive mkdir -p /root/src /root/res cloud-init status --wait || true # first boot runs apt; wait for its lock fetch_sha() { git init -q "$1"; git -C "$1" remote add origin "$2"; git -C "$1" fetch -q --depth 1 origin "$3"; git -C "$1" checkout -q FETCH_HEAD; } fetch_sha /root/src/excalidraw https://github.com/excalidraw/excalidraw 8bcdca4526867433f99fdfcb55a063668db0c9dc fetch_sha /root/src/plantuml https://github.com/plantuml/plantuml db06d70e18e71839dd5e330ac4086f3845371e3b fetch_sha /root/src/cli https://github.com/cli/cli 9490d204a2ba95434153dedce99898ce9f281426 git clone -q --depth 1 --branch 15.2.0 https://github.com/BurntSushi/ripgrep /root/src/ripgrep fetch_sha /root/src/msd https://github.com/GoogleCloudPlatform/microservices-demo c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712 curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz curl -fsSL https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.xz | tar -C /usr/local --strip-components=1 -xJ curl -fsSL https://sh.rustup.rs | sh -s -- -y -q --profile minimal --default-toolchain stable apt-get update -q apt-get install -y -q openjdk-21-jdk-headless build-essential git jq zstd python3 ca-certificates curl curl -fsSL https://get.docker.com | sh export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local npm install -g yarn@1.22.22 docker pull -q moby/buildkit:v0.34.1 docker pull -q registry:2.8.3 docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3 docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3 docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3 # dependencies are downloaded once, outside the measured runs (cd /root/src/excalidraw && yarn install --frozen-lockfile --network-timeout 600000) (cd /root/src/cli && go mod download) (cd /root/src/ripgrep && cargo fetch --locked) # systemd stops a scope after an OOM kill by default; keep the wrapper alive to record the result mkdir -p /etc/systemd/system.conf.d printf '[Manager]\nDefaultOOMPolicy=continue\n' > /etc/systemd/system.conf.d/oom.conf systemctl daemon-reexec #wrap.sh: runs inside the scope and records the result #!/bin/bash # Runs inside the systemd scope. usage: wrap.sh OUTDIR TIMEOUT_S cmd... out=$1; tmo=$2; shift 2 cg=/sys/fs/cgroup$(sed -n 's/^0:://p' /proc/self/cgroup) echo "$cg" > "$out/cgroup" cat "$cg/memory.max" > "$out/memory.max" t0=$(date +%s.%N) timeout -k 15 "$tmo" "$@" > "$out/build.log" 2>&1 rc=$? t1=$(date +%s.%N) { echo "rc=$rc" echo "t0=$t0" echo "t1=$t1" echo "peak=$(cat "$cg/memory.peak")" echo "current_after=$(cat "$cg/memory.current")" awk '$1=="anon"||$1=="file"{print $1"_after="$2}' "$cg/memory.stat" } > "$out/result.env" cp "$cg/memory.events" "$out/memory.events" for p in $(cat "$cg/cgroup.procs"); do ["$p" = "$$"] && continue ps -o pid=,rss=,args= -p "$p" | cut -c1-400 done > "$out/leftover.txt" 2>/dev/null if [-n "${LEFTOVER_WAIT:-}"]; then sleep "$LEFTOVER_WAIT" echo "current_later=$(cat "$cg/memory.current")" >> "$out/result.env" awk '$1=="anon"{print "anon_later="$2}' "$cg/memory.stat" >> "$out/result.env" for p in $(cat "$cg/cgroup.procs"); do ["$p" = "$$"] && continue ps -o pid=,rss=,args= -p "$p" | cut -c1-400 done > "$out/leftover_later.txt" 2>/dev/null fi touch "$out/done" exit $rc #sample.py: polls memory.stat every 100 ms from outside the scope #!/usr/bin/env python3 """Poll one cgroup v2 directory every 100 ms until a done file appears. usage: sample.py CGROUP_DIR OUT_JSON DONE_FILE Records the maximum of anon, file, kernel and memory.current, plus a series. """ import json, os, sys, time cg, out, done = sys.argv[1:4] start = time.time() while not os.path.isdir(cg): if time.time() - start > 60 or os.path.exists(done): json.dump({"error": "cgroup never appeared"}, open(out, "w")) sys.exit(0) time.sleep(0.02) keys = ("anon", "file", "kernel", "shmem") mx = {k: 0 for k in keys} mx["current"] = 0 series = [] t0 = time.time() n = 0 while not os.path.exists(done): try: with open(os.path.join(cg, "memory.stat")) as f: st = {} for line in f: k, v = line.split() if k in keys: st[k] = int(v) with open(os.path.join(cg, "memory.current")) as f: cur = int(f.read()) except OSError: break n += 1 for k in keys: mx[k] = max(mx[k], st.get(k, 0)) mx["current"] = max(mx["current"], cur) series.append([round(time.time() - t0, 2), st.get("anon", 0), st.get("file", 0), cur]) time.sleep(0.1) json.dump({"max": mx, "samples": n, "interval_s": 0.1, "series": series}, open(out, "w")) #p1-cmds.sh: the builds, and what is deleted before each run #!/bin/bash # Build definitions for post 1. prep_* runs outside the cgroup, cmd_* inside it. export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local EX=/root/src/excalidraw PU=/root/src/plantuml GH=/root/src/cli RG=/root/src/ripgrep FRONTEND=/root/src/msd/src/frontend stop_gradle() { pkill -f GradleDaemon || true; pkill -f GradleWrapperMain || true; pkill -f KotlinCompileDaemon || true; sleep 1; } prep_vite() { rm -rf $EX/excalidraw-app/build $EX/node_modules/.vite $EX/excalidraw-app/node_modules/.vite; } cmd_vite() { cd $EX && yarn build:app:docker; } prep_tsc() { find $EX -maxdepth 2 -name '*.tsbuildinfo' -delete; } cmd_tsc() { cd $EX && node_modules/.bin/tsc -p tsconfig.json --extendedDiagnostics; } prep_gradle_daemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; } cmd_gradle_daemon() { cd $PU && ./gradlew jar --console=plain --no-build-cache; } prep_gradle_nodaemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; } cmd_gradle_nodaemon() { cd $PU && ./gradlew --no-daemon jar --console=plain --no-build-cache; } prep_go() { rm -rf /root/gocache /root/out; mkdir -p /root/out; } cmd_go() { cd $GH && GOCACHE=/root/gocache go build -o /root/out/gh ./cmd/gh; } prep_rust() { rm -rf $RG/target; } cmd_rust() { cd $RG && cargo build --release --locked; } prep_docker() { docker buildx ls --format '{{.Name}}' 2>/dev/null | grep '^p1d' | xargs -r -n1 docker buildx rm >/dev/null 2>&1 || true; } if ["${BASH_SOURCE[0]}" = "$0" ]; then "$@"; fi #p1.sh: probe, warm-up, matrix, round 3, redo, and the docker build cell #!/bin/bash # Post 1 matrix. usage: p1.sh warmup | p1.sh matrix set -u B=/root/bench R=/root/res/p1 mkdir -p $R source $B/p1-cmds.sh LIMITS=(8G 4G 2G) BUILDS=(vite tsc gradle_daemon gradle_nodaemon go rust docker) # rust ran once (warmup) only: the matrix did not fit the droplet time budget MATRIX_BUILDS=(vite tsc gradle_daemon gradle_nodaemon go docker) CPUS=0-3 log() { echo "$(date -u +%FT%TZ) $*" | tee -a $R/progress.log; } scope_run() { # unit limit out tmo cmd... local unit=$1 lim=$2 out=$3 tmo=$4; shift 4 python3 $B/sample.py /sys/fs/cgroup/bench.slice/$unit.scope $out/samples.json $out/done & local sp=$! systemd-run --scope --quiet --slice=bench.slice --unit=$unit \ -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS -p OOMPolicy=continue \ $B/wrap.sh $out $tmo "$@" wait $sp systemctl stop $unit.scope 2>/dev/null || true } docker_cell() { # lim out tmo tag local lim=$1 out=$2 tmo=$3 tag=$4 local name=$(echo p1d$tag | tr A-Z a-z) m=$(echo $1 | tr G g) docker buildx create --name $name --driver docker-container \ --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \ --driver-opt memory=$m --driver-opt memory-swap=$m --driver-opt cpuset-cpus=$CPUS \ --buildkitd-config $B/buildkitd.toml --bootstrap > $out/create.log 2>&1 local cid; cid=$(docker inspect -f '{{.Id}}' buildx_buildkit_${name}0) local cg=/sys/fs/cgroup/system.slice/docker-$cid.scope cat $cg/memory.max > $out/container_memory.max mkdir -p $out/cli python3 $B/sample.py $cg $out/container_samples.json $out/cli/done & local sp=$! scope_run p1-docker-$tag $lim $out/cli $tmo \ docker buildx build --builder $name --progress=plain --provenance=false --output type=cacheonly $FRONTEND wait $sp { echo "container_peak=$(cat $cg/memory.peak)"; cat $out/cli/result.env; } > $out/result.env cp $cg/memory.events $out/container_memory.events cp $out/cli/build.log $out/build.log docker buildx rm $name > /dev/null 2>&1 } run_cell() { # build limit run tmo local b=$1 lim=$2 r=$3 tmo=$4 local tag=$b-$lim-$r out=$R/$b/$lim/$r rm -rf $out; mkdir -p $out prep_$b sync; echo 3 > /proc/sys/vm/drop_caches; sleep 2 if [$b = docker]; then docker_cell $lim $out $tmo ${lim}$r else case $b in gradle_*) export LEFTOVER_WAIT=5 ;; esac scope_run p1-$tag $lim $out $tmo bash $B/p1-cmds.sh cmd_$b unset LEFTOVER_WAIT fi [$b = gradle_daemon] || [$b = gradle_nodaemon] && stop_gradle log "$tag $(tr '\n' ' ' < $out/result.env)" } probe() { # record what the runtimes see inside each limit for lim in "${LIMITS[@]}"; do systemd-run --scope --quiet --slice=bench.slice --unit=probe-$lim -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS \ bash -c 'echo "nproc=$(nproc)"; echo "memory.max=$(cat /sys/fs/cgroup$(sed -n "s/^0:://p" /proc/self/cgroup)/memory.max)"; echo "node_heap_size_limit=$(node -e "console.log(require(\"v8\").getHeapStatistics().heap_size_limit)")"; java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage|ActiveProcessorCount) "' > $R/probe-$lim.txt 2>&1 done { echo "nproc=$(nproc)"; free -b | head -2; echo "node_heap_size_limit=$(node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)')"; java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage) "; } > $R/probe-host.txt 2>&1 } case ${1:-} in warmup) probe for b in "${BUILDS[@]}"; do run_cell $b 8G r0 900; done log WARMUP-DONE ;; matrix) declare -A TMO for b in "${MATRIX_BUILDS[@]}"; do . <(grep -E '^t[01]=' $R/$b/8G/r0/result.env) w=$(python3 -c "print(int(float('$t1')-float('$t0')))") t=$((3 * w + 60)); [$t -gt 600] && t=600; [$t -lt 120] && t=120 TMO[$b]=$t done log "timeouts: $(for b in "${MATRIX_BUILDS[@]}"; do printf '%s=%s ' $b ${TMO[$b]}; done)" for r in r1 r2 r3; do for b in "${MATRIX_BUILDS[@]}"; do for lim in "${LIMITS[@]}"; do run_cell $b $lim $r ${TMO[$b]}; done done done log MATRIX-DONE ;; round3) # third run at 8G and 2G only: the full round did not fit the droplet time budget declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351) log "round 3 at 8G and 2G only" for b in "${MATRIX_BUILDS[@]}"; do for lim in 8G 2G; do run_cell $b $lim r3 ${TMO[$b]}; done done log MATRIX-DONE ;; redo) # rerun cells that left no result (systemd stopped the scope after an OOM kill) declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351) for b in "${MATRIX_BUILDS[@]}"; do for lim in "${LIMITS[@]}"; do for r in r1 r2 r3; do d=$R/$b/$lim/$r if [-d $d] && ! grep -q '^rc=' $d/result.env 2>/dev/null; then rm -rf $d.lost; mv $d $d.lost; log "redo $b $lim $r" systemctl reset-failed p1-$b-$lim-$r.scope 2>/dev/null # added after our run: a failed scope blocks its name run_cell $b $lim $r ${TMO[$b]} fi done; done; done log REDO-DONE ;; *) echo "usage: $0 warmup|matrix|round3|redo"; exit 2 ;; esac #buildkitd.toml: send base image pulls to the local mirror [registry."docker.io"] mirrors = ["127.0.0.1:5001"] [registry."gcr.io"] mirrors = ["127.0.0.1:5002"] [registry."127.0.0.1:5000"] http = true insecure = true [registry."127.0.0.1:5001"] http = true insecure = true [registry."127.0.0.1:5002"] http = true insecure = true #summarize_p1.py: medians per build and limit #!/usr/bin/env python3 """Summarize post 1 runs: medians per build and limit over the measured runs (r1..r3).""" import json, os, re, statistics, sys root = sys.argv[1] BUILDS = ["vite", "tsc", "gradle_daemon", "gradle_nodaemon", "go", "rust", "docker"] LIMITS = ["8G", "4G", "2G"] MiB = 2 ** 20 def env(path): d = {} if os.path.exists(path): for line in open(path): if "=" in line: k, v = line.strip().split("=", 1) d[k] = v return d def events(path): d = {} if os.path.exists(path): for line in open(path): k, v = line.split() d[k] = int(v) return d def run_info(d, build): r = env(os.path.join(d, "result.env")) if not r or "rc" not in r: return None log = open(os.path.join(d, "build.log"), errors="replace").read() if os.path.exists(os.path.join(d, "build.log")) else "" if build == "docker": if not r.get("container_peak"): return None # warm-up run 0 recorded no container numbers (builder name case bug) ev = events(os.path.join(d, "container_memory.events")) smp = json.load(open(os.path.join(d, "container_samples.json"))) peak = int(r["container_peak"]) cli_peak = int(r["peak"]) else: ev = events(os.path.join(d, "memory.events")) smp = json.load(open(os.path.join(d, "samples.json"))) peak = int(r["peak"]) cli_peak = None rc = int(r["rc"]) wall = float(r["t1"]) - float(r["t0"]) if rc == 0: outcome = "ok" elif ev.get("oom_kill", 0) > 0: outcome = "oom_kill" elif "heap out of memory" in log or "Reached heap limit" in log: outcome = "node_heap" elif "OutOfMemoryError" in log: outcome = "java_heap" elif rc in (124, 137): outcome = "timeout" else: outcome = f"fail_rc{rc}" mx = smp.get("max", {}) return { "rc": rc, "outcome": outcome, "wall_s": round(wall, 1), "peak_mib": round(peak / MiB), "anon_max_mib": round(mx.get("anon", 0) / MiB), "file_max_mib": round(mx.get("file", 0) / MiB), "kernel_max_mib": round(mx.get("kernel", 0) / MiB), "samples": smp.get("samples"), "oom_kill": ev.get("oom_kill", 0), "max_events": ev.get("max", 0), "high_events": ev.get("high", 0), "after_mib": round(int(r.get("current_after", 0)) / MiB), "anon_after_mib": round(int(r.get("anon_after", 0)) / MiB), "cli_peak_mib": round(cli_peak / MiB) if cli_peak is not None else None, "later_mib": round(int(r["current_later"]) / MiB) if "current_later" in r else None, "anon_later_mib": round(int(r["anon_later"]) / MiB) if "anon_later" in r else None, "leftover_later": open(os.path.join(d, "leftover_later.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover_later.txt")) else None, "leftover": open(os.path.join(d, "leftover.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover.txt")) else "", } def med(xs): xs = [x for x in xs if x is not None] return round(statistics.median(xs), 1) if xs else None out = {} for b in BUILDS: for lim in LIMITS: runs = {} for rn in ("r0", "r1", "r2", "r3"): d = os.path.join(root, b, lim, rn) if os.path.isdir(d): info = run_info(d, b) if info: runs[rn] = info meas = [runs[k] for k in ("r1", "r2", "r3") if k in runs] ok = [x for x in meas if x["outcome"] == "ok"] out[f"{b}/{lim}"] = { "runs": runs, "n": len(meas), "n_ok": len(ok), "outcomes": [x["outcome"] for x in meas], "wall_med_ok": med([x["wall_s"] for x in ok]), "wall_med_all": med([x["wall_s"] for x in meas]), "peak_med": med([x["peak_mib"] for x in meas]), "anon_med": med([x["anon_max_mib"] for x in meas]), "file_med": med([x["file_max_mib"] for x in meas]), "peak_range": [min(x["peak_mib"] for x in meas), max(x["peak_mib"] for x in meas)] if meas else None, "anon_range": [min(x["anon_max_mib"] for x in meas), max(x["anon_max_mib"] for x in meas)] if meas else None, "wall_range": [min(x["wall_s"] for x in meas), max(x["wall_s"] for x in meas)] if meas else None, "after_med": med([x["after_mib"] for x in meas]), "anon_after_med": med([x["anon_after_mib"] for x in meas]), "cli_peak_med": med([x["cli_peak_mib"] for x in meas]), "later_med": med([x["later_mib"] for x in meas]), "anon_later_med": med([x["anon_later_mib"] for x in meas]), "max_events_med": med([x["max_events"] for x in meas]), } json.dump(out, open(os.path.join(root, "summary.json"), "w"), indent=1) print(f"{'cell':26} {'n':>2} {'ok':>2} {'wall_ok':>8} {'peak':>6} {'anon':>6} {'file':>6} {'after':>6} outcomes / ranges") for k, v in out.items(): print(f"{k:26} {v['n']:>2} {v['n_ok']:>2} {str(v['wall_med_ok']):>8} {str(v['peak_med']):>6} {str(v['anon_med']):>6} {str(v['file_med']):>6} {str(v['after_med']):>6} {v['outcomes']} wall{v['wall_range']} anon{v['anon_range']} peak{v['peak_range']} maxev={v['max_events_med']} cli={v['cli_peak_med']} later={v['later_med']}/{v['anon_later_med']}") Summary Most CI builds need less memory than the cgroup counter suggests, because memory.peak includes page cache that the kernel gives back under pressure. Measure anonymous memory, run the build inside the limit you want to buy, and check memory.events for max and oom_kill. Then look for the traps: a bundler that runs a type checker, a runtime that sizes its heap from the limit, a Gradle daemon that stays after the build, and a Docker build that runs outside your cgroup. In our runs, fixing any of those was worth more than doubling the runner's memory. _Originally published at devops-daily.com._

🖥️ GoogleCloudPlatform/microservices-demo Sample cloud-first application with 10 #microservices showcasing #Kubernetes, #Istio, and #gRPC. GoogleCloudPlatform/microservices-demo #GoogleCloud #GKE

How to collect metrics from a service exposing grpc port Hello, So I have deployed this micro services demo in my cluster using flux. I want to collect metrics from services so I have deployed the monitoring stack using kube-Prometheus-stack and want to use service monitors to identity the services but I see most of the services expose grpc port but Prometheus collect over http right so what can I do in this situation? Thanks

How I Used FalkorDB to Trace Software Security Risks in Warden You’re right—you want the article body, based on the actual README. Use the title separately, then paste everything under Post content into Dev.to. Title: Warden: Graph-Based Security Analysis with FalkorDB --- Warden helps software teams investigate security risks in a repository by connecting services, dependencies, vulnerabilities, ownership, and change history in a graph. Instead of looking at vulnerability findings in isolation, engineers can explore how a package relates to a service, how services are modeled as communicating, and what sensitive data may be connected downstream. Warden uses those relationships to show evidence paths, help prioritize investigation, and rehearse possible dependency fixes before a person approves a change. Project: Warden on GitHub Live demo: warden-bice.vercel.app Demo video: Watch the demo The problem Warden explores A vulnerability scanner can identify a package with a known advisory. But a finding by itself may leave important questions unanswered: - Which service uses the package? - Is that service connected to an internet-facing entry point? - What other services or data are connected to it? - Who owns the service? - What could change if we upgrade the package? Warden brings repository structure and security information together so a team can explore these questions with graph queries and visible evidence. A graph path is evidence of modeled reachability. It does not prove that vulnerable code is called or that an application is exploitable. Warden does not currently perform runtime traffic analysis or function-level vulnerability reachability. Why FalkorDB? The relationships between services, dependencies, vulnerabilities, and data are central to Warden’s workflow. FalkorDB provides the graph store Warden uses to represent and query those connections. Warden loads normalized repository and security information into FalkorDB, then uses graph queries to: - Find vulnerability findings connected to modeled exposure. - Explore paths between services and other graph entities. - Calculate blast radius, rankings, and potential choke points. - Show evidence nodes, edges, sources, and query details. - Compare the modeled graph before and after a candidate dependency upgrade. For fix planning, Warden rehearses proposed upgrades against a temporary copy of the graph. It reruns exposure analysis and safety checks, then presents the plan and gate result for review. A successful graph rehearsal demonstrates modeled graph impact; it does not guarantee that an upgrade preserves application behavior. The target project’s own tests and CI still need to be reviewed. From a repository to a graph When a project is built, Warden reads the selected repository and branch or tag into a project-specific checkout. It discovers services and topology from Kubernetes manifests, Docker Compose files, or supported code and dependency layouts. It parses supported dependency files into package nodes and dependency relationships, looks up package advisories through OSV, and enriches findings with CISA Known Exploited Vulnerabilities (KEV) and FIRST EPSS data. Optional NVD lookups can provide CVSS information. Where available, Warden also reads Git history and adds curated owner, policy, incident, and data-sensitivity context for the shared sample project. The ingest pipeline records provenance so users can see whether information came from a repository or feed, was inferred from configuration, or was curated for the sample project. Supported dependency inputs include npm lockfiles, Python requirements and common Poetry/uv/Pipenv locks, go.mod, NuGet .csproj, Maven POM and Gradle files, Cargo, Ruby Bundler, and Composer lockfiles. The importer supports common repository layouts, not every possible project structure. Results depend on the files and configuration present. What you can do in Warden Explore the attack surface The Home screen shows an attack-surface map and exposure summaries. Select a service to inspect its modeled connections to callers, dependencies, and sensitive data. The Secure [service] interaction is a visual counterfactual. It highlights which currently modeled internet-to-sensitive-data routes would be cut if a service were secured. It does not modify the repository or graph. Ask questions and inspect evidence The Ask screen lets you ask graph-backed questions, such as: - “What should we fix first?” - “Show the path for CVE-2023-44487.” - “Is paymentservice reachable from the internet? Show the evidence.” - “What does this service read, and who owns it?” The agent selects graph tools through MCP and drafts an answer using their results. The interface can show the tool trace, evidence path, source, Cypher query, and parameters. A deterministic verifier checks important claims against the tools’ results for that turn. Model-backed answers require an OpenAI-compatible LLM endpoint. Graph tools and non-LLM views can still be used without an LLM key. Compare and propose fixes The plan builder can compare candidate dependency upgrade strategies. You can set a maximum number of upgrades, specify frozen services, and disallow major-version upgrades. Warden rehearses candidates on graph copies and displays changes, risk removed, new advisories, residual KEV, and gate results. A blocked plan cannot be filed. A passing plan can be filed for approval. Filing creates a pending approval; it does not immediately make a GitHub change. Review pull requests and approvals The PR impact screen analyzes affected services, graph blast radius, owners, related history, policies, and suggested review feedback. Plans and PR analyses can be sent to the Approvals screen for human review. Warden does not automatically merge pull requests. With GitHub configured, approved actions can operate on the selected repository target. Review the proposed changes and the pull request’s CI results before merging. The Watcher also has a clearly labeled demo simulation that can commit a one-line topology change to its configured repository. Use a disposable fork for that demonstration. Use tools, sync, watch, and evaluate The Tools screen runs graph tools directly and displays summaries, rows, evidence paths, source details, queries, parameters, and duration. The graph tools are read-only; GitHub write actions use separate approval workflows. Sync re-reads the configured repository and reports graph changes. The Watcher can poll for branch changes and create pending proposals when it detects new modeled exposure. The Evaluation screen runs designed scenarios against the tools and reports whether their expected outcomes match. These scenarios are a regression suite, not an independent benchmark across arbitrary repositories. Warden also exposes its graph tool registry through an MCP server for use by Warden’s agent and compatible external MCP clients. Architecture Warden uses: - FastAPI for the backend API. - React, TypeScript, and Vite for the frontend. - FalkorDB for repository and security graph data. - PostgreSQL for application records such as accounts, sessions, teams, projects, and conversations. - Cytoscape.js for graph visualizations. - MCP to expose graph tools to the agent and compatible clients. The backend contains the API, authentication, repository ingestion, graph access, security and topology tools, agent, evaluation, planning, approvals, sync, watcher, and MCP server. The frontend contains screens for Home, Ask, PR impact, Approvals, Tools, Sync, Evaluation, Settings, authentication, and the public landing page. The basic request path is: 1. The browser sends an authenticated request to the FastAPI backend. 2. The API resolves the user’s current project. 3. The agent or screen calls project-scoped graph tools. 4. The tools query FalkorDB and return rows and, where available, evidence paths. 5. The interface displays results and supporting details. PostgreSQL stores application data; FalkorDB stores repository and security relationships. Run Warden locally Requirements You’ll need Docker Desktop or Docker Engine with Compose, Python 3.11 or newer, Node.js 20 or newer with npm, Git, and internet access for the first repository build and vulnerability-feed requests. Quick start Clone the repository, open a terminal in the project directory, and copy the environment template: Windows PowerShell: Copy-Item .env.example .env macOS or Linux: cp .env.example .env Edit .env if needed. The backend reads this file from the repository root. Never commit your .env file or put backend secrets in frontend variables. Then run the project launcher: Windows: .\start.ps1 macOS/Linux/Git Bash: ./start.sh The launcher starts FalkorDB and PostgreSQL, prepares the Python environment, installs dependencies if needed, initializes the sample graph if it is empty, and starts the API and Vite development server. The first ingest can take several minutes. The local addresses are: - UI: http://localhost:5173 - API: http://localhost:8000 - API docs: http://localhost:8000/docs - FalkorDB Browser: http://localhost:3000 - FalkorDB graph port: localhost:6380 - PostgreSQL: localhost:5433 Sign up at /signup. New users can access the shared sample project. You can create another project to build a graph from a GitHub repository and selected branch. Manual development setup If you prefer to run the services in separate terminals, start the graph and database from the repository root: docker compose up -d falkordb postgres Then install and initialize the backend: cd backend python -m venv .venv .\.venv\Scripts\Activate.ps1 python -m pip install -r requirements.txt python -m warden.ingest.bootstrap python -m uvicorn warden.api.app:app --host 127.0.0.1 --port 8000 On macOS/Linux, activate the environment with: source .venv/bin/activate In another terminal, start the frontend: cd frontend npm install npm run dev Vite proxies /api to http://127.0.0.1:8000. Set WARDEN_API if the backend is running on a different local port. Environment variables The .env.example file documents the available settings. The main ones are: - FalkorDB: FALKORDB_HOST, FALKORDB_PORT, and optional username, password, and TLS settings. - PostgreSQL: DATABASE_URL for user, team, project, session, and conversation data. - Repository ingestion: DEMO_REPO_URL, DEMO_REF, HISTORY_SINCE, and optional WARDEN_NVD. - LLM: LLM_BASE_URL, LLM_API_KEY, and LLM_MODEL for model-backed Ask answers. - GitHub: GITHUB_TOKEN, GITHUB_OWNER, GITHUB_REPO, and GITHUB_BASE for repository access and approved actions. - Sessions and deployment: SESSION_SECRET, COOKIE_SAMESITE, COOKIE_SECURE, CORS_ORIGINS, WARDEN_URL, and VITE_API_URL. The local defaults in .env.example are for development. Before exposing an API publicly, use a long random SESSION_SECRET, HTTPS, secure cookies, exact CORS origins, and appropriately scoped credentials. Keep GitHub tokens on the backend. Without GitHub credentials, the sample graph and many read-only views remain available, but private repository access and GitHub actions are unavailable. Without an LLM key, graph-backed pages and tools remain usable, but Ask cannot generate model answers. GitHub integration and human approval A fine-grained GitHub token enables private repository access, GitHub history and pull request metadata, and approved repository actions. Grant only the permissions needed for the target repository, configure the project’s repository in Settings, and verify the target shown on the Approvals screen. Warden does not automatically merge pull requests. Plans and PR analyses wait for human approval before GitHub writes. Use a fork or dedicated test repository when trying write actions. Data, privacy, and limitations - The shared sample project uses Google’s Online Boutique repository. - Vulnerability data comes from OSV, CISA KEV, and FIRST EPSS; optional NVD lookups can be enabled. Feed data is cached locally. - Sample ownership, incidents, policies, and data sensitivity are curated demo context, not facts about an actual company. - Repository clones and feed caches are runtime data stored under backend/data/raw/ and ignored by Git. - LLM prompts include the user’s question and the tool data needed to answer it. Check your model provider’s data-handling terms before using private repository data. - Service calls can be inferred from manifests, environment variables, or Compose files. The interface labels inferred and curated evidence. - EPSS and KEV data are time-sensitive and should be refreshed for operational decisions. - A graph path is not proof of exploitability, and a fix-plan rehearsal is not a replacement for application tests. - Repository discovery supports common formats and layouts, not every possible repository. Testing and production deployment Run the backend tests from backend/: python -m pytest -q tests Build and type-check the frontend from frontend/: npm run build For production, deploy the API, FalkorDB, and Caddy on a server with HTTPS, then deploy the frontend/ directory to Vercel as a static site. Set VITE_API_URL to the API’s HTTPS origin in Vercel. Configure the backend’s WARDEN_URL and CORS_ORIGINS to match the frontend origin. Store secrets in the hosting provider, not in committed files. Redeploy the frontend after changing VITE_API_URL. Credits Warden is built with FastAPI, React, Vite, TypeScript, Cytoscape.js, FalkorDB, PostgreSQL, and MCP. It uses public vulnerability information from OSV, CISA KEV, and FIRST EPSS. The sample service repository is Google Online demo.

"I built a multiverse for AI agents: they rehearse every action in hundreds of graph copies before touching anything" !home AI agents are starting to act on real systems. They clean up databases, rotate keys, move cron jobs, scale services. The action itself is rarely the dangerous part. The danger is everything that silently depends on the thing being changed. So for Graph Hacks (WeMakeDevs × FalkorDB) I built Multiverse: before an agent does anything, it rehearses the action in hundreds of copies of the company, counts how often something important breaks, and only lets it through if the answer is clearly safe. - 🎬 Demo video: below - 🔗 Live demo: multiverse-cqzs.onrender.com (free server, the first visit may take a minute to wake up) - 💻 Code: TarunMhanta30/multiverse {% youtube xh497r3bZTY %} --- The problem: "nobody uses this table" Here is the scenario the whole project is built around. A cleanup agent looks at orders_archive. No service declares a dependency on it. No config file mentions it. By every static check, it is safe to delete. But the access log knows something the catalog doesn't: tax_filing_job read the archive in 4 of its last 5 runs. Delete it, and the next quarterly tax filing breaks. Real companies are full of links like this. Some are certain (a Kubernetes deployment is configured with another service's address). Many are only probably real: a job that used to read a table, a script from an old runbook that may still use a secret. A yes/no dependency check can't handle "probably". So I stopped asking "does anything depend on it?" and started asking "in how many possible futures does something important break?" The idea: rehearse, don't guess Multiverse treats the company as a graph in FalkorDB, and every uncertain link as a coin with a known bias. To rehearse a plan: 1. Snapshot the company graph with GRAPH.COPY. 2. For each future, make a fresh copy of the snapshot. 3. In that copy, roll the dice for every uncertain link: is it real in this future? Is the hand-run job used during the window? 4. Apply the change and let the failure ripple outward, one hop at a time, in Cypher. 5. Read back what broke, then delete the copy. Each future becomes one square on screen. Orange means something business-critical broke. After enough futures, Multiverse knows the chance of breaking something critical, and if it's 5% or more, the plan is blocked. !A blocked rehearsal: every square is one copy of the company Building the graph from evidence, not opinions The company in the demo is Boutique Inc. Its 12 services and Redis come straight from Google's real Online Boutique Kubernetes manifests, unmodified. Every *_SERVICE_ADDR environment variable becomes a CALLS edge. The back office (billing, tax and analytics jobs, databases, secrets, owners and a 13-month access log) is synthetic, and labelled as synthetic everywhere in the app. The rule I held myself to: no confidence number is typed in by hand. Every link gets a rule and a confidence computed from its evidence: | Rule | Example | Confidence | |---|---|---| | k8s | frontend has CART_SERVICE_ADDR=cartservice:7070 | 100% | | verified | declared in the catalog and signed off by the owner | 100% | | logs | tax_filing_job read the archive in 4 of its last 5 runs | 48% | | declared | an old runbook says refund_script uses the payment key | 50% | The 48% is checkable by hand. Recent runs count more: a run's weight halves every 90 days. The five runs weigh 0.05, 0.11, 0.22, 0.44 and 0.88. The four that read the archive add up to 0.81 out of 1.70, and 0.81 ÷ 1.70 = 48%. In the app you can click that link and see every run, its weight, and the exact file lines it came from. !The evidence behind one link The Cypher that does the work The heart of each future is a hop-by-hop ripple, run inside that future's own copy: UNWIND $frontier AS f MATCH (x:Asset)-[r:CALLS|READS|WRITES|USES_SECRET|DEPENDS_ON|FED_BY]->(y:Asset {name: f}) WHERE NOT r.key IN $inactive RETURN x.name, type(r), r.soft, r.key, y.name Before the ripple, links the dice said are not real in this future are simply deleted from the copy: MATCH ()-[r]->() WHERE r.key IN $keys DELETE r Hard links break the dependent asset, soft links only degrade it, and data feeds go stale. The statuses are written back into the copy, read out, and the copy is dropped. Launch day uses the same graph differently. The load on every service is the traffic at the front door multiplied along every path, which turns out to be a single query: MATCH p = (e:Asset {name: $entry})-[:CALLS|READS|WRITES*0..8]->(y:Asset) WHERE all(r IN relationships(p) WHERE r.calls_per_request IS NOT NULL) RETURN y.name, sum(reduce(f = 1.0, r IN relationships(p) | f * r.calls_per_request)) Divide capacity by that and you get each service's breaking point, for example "cartservice runs out at 3.5× normal traffic". Knowing when to stop Running a fixed 1,000 futures would be slow and pointless. Multiverse adds futures 25 at a time and computes a 95% Wilson interval after each batch. As soon as the whole interval is clearly above or below the 5% safety line, it stops. A clear BLOCK usually takes 25 to 50 futures. And if nothing uncertain can reach the plan's effects, every future would be identical, so it plays exactly one and says so. "Exact" is a real answer, not a shortcut. Replay: no black boxes Every future is reproducible. The same plan with the same replay code rolls the same dice in the same order. Press Replay on any square and it runs again in a fresh copy, with every FalkorDB command shown as it executes and the failure spreading across the graph in step with the real query results. !Replaying one future, command by command Memory: ask once, never guess again When a verdict hinges on an uncertain link, Multiverse asks the team that owns it. Here, Finance gets "Does tax_filing_job still read orders_archive?". The answer goes into a second graph, the memory (boutique_memory), which survives rebuilds and sessions: (:Question)-[:ASKS]->(:Team) (:Fact)-[:ANSWERS]->(:Question) (:Rehearsal)-[:HAS_FUTURE]->(:Future) (:Plan)-[:CONFLICTS_WITH]->(:Plan) Every later rehearsal uses the answer instead of rolling dice. In the demo, once Finance and Data answer, the same delete becomes exact and safe, and only then does execute let it through. Execution itself re-runs a fresh rehearsal first, so a stale SAFE can't sneak past. Two agents, one company This is the part I find most interesting. A finance agent moves billing to read from the replica: safe alone. A platform agent takes the replica offline on the 1st from 01:00 to 04:00 for maintenance: also safe alone. Together, billing runs at 02:00 against a database that is down, and payouts break. Agents in Multiverse propose plans instead of acting. Each proposal is rehearsed alone and together with every other active plan that overlaps it in time and in the graph. The second plan is held, and Multiverse looks for the earliest safe slot. Checking 168 hourly start times naively means 168 rehearsals. But start times that see the same running jobs and the same overlapping plans produce identical futures, so Multiverse groups them and rehearses each group once: 168 start times, 3 rehearsals. !Agents: holds, the schedule and the safe-slot heatmap A company brain you can question The same graph answers questions about the company itself, using FalkorDB's built-in algorithms: - algo.betweenness finds chokepoints (checkoutservice, which most dependency paths run through) - algo.pageRank finds foundations, the assets everything ultimately leans on - algo.WCC finds islands, groups connected to nothing else, which feed a cleanup finder that then proves each candidate with a real rehearsal - shortestPath explains why one asset depends on another - a full-text index powers the Ctrl K search - "Ask the graph" turns a plain-English question into Cypher and runs it with GRAPH.RO_QUERY and a timeout, so the model physically cannot write Real agents, through MCP Multiverse exposes 13 MCP tools (rehearse, propose_plan, execute_approved_plan, find_safe_window, evidence, recall and more). Claude Desktop or any MCP client can use them, so a real AI agent rehearses before it acts. Every tool call is logged into the memory graph, which is also how the web console shows agent activity live: the memory graph doubles as a shared bus between the console, the MCP server and the command line. What I learned about FalkorDB - GRAPH.COPY is the whole trick. Giving every future its own complete, writable copy of the company made the engine simple. A future is just "copy, delete some edges, ripple, read, drop". No undo logic, no risk to the real graph. A fingerprint check before and after every rehearsal proves the real company is untouched. - Copies are sequential. FalkorDB forks to copy, so futures run one after another, about 0.1 s each on my laptop, most of it the copy itself. That is exactly why adaptive stopping and "exact" mode mattered so much. - Two graphs beat one. Keeping the company (rebuildable from evidence) separate from memory (permanent) made resets safe and made "remembered across sessions" literally true. - Read-only queries are a safety feature. GRAPH.RO_QUERY is what made it reasonable to run LLM-written Cypher at all. Proving it A self-test (python selftest.py) runs 17 checks against its own graphs. For example: - 300 futures land within 3 standard errors of what the evidence predicts - 12 replayed futures match their recordings exactly - adding one edge changes the answer, so nothing is precomputed - fixing a dependency turns BLOCK into SAFE - two plans that are each safe alone are blocked together - every rehearsal leaves the real graph unchanged and deletes all of its copies Honest limits - The back office is synthetic. Only the Online Boutique part is real. - The ripple models hard, soft and stale dependencies, not retries, timeouts or "either A or B is enough" redundancy. - "Ask the graph" is only as good as the Cypher the model writes, so the Cypher is always shown. Try it - Live: multiverse-cqzs.onrender.com. Press Take the 2-minute tour: it runs the real engine and explains each step. - Locally: clone the repo, then run.ps1 on Windows or bash run.sh on macOS/Linux. It starts FalkorDB in Docker, builds the graph and opens the console on localhost:8765. {% github TarunMhanta30/multiverse %} Built solo for Graph Hacks 2026 with FalkorDB, Python, vanilla JavaScript, SVG and three.js. I used an AI coding assistant for parts of the code and docs; the idea, the decisions and the testing are mine. If you're building agents that touch real systems, I'd love to hear how you're handling the "nobody uses this table" problem. Let me know in the comments.

Microsoft-Decision-1 in practice: probabilities for AIOps decisions Microsoft-Decision-1 isn't a chat model. You send a state plus typed questions, and it returns numbers: a probability for yes/no (noul), a probability per option (choice), or a position on an ordered scale (score). No prose, no rationale, about 1 output token per call. That makes it a natural candidate for the decisions around an operations stack: "is this block of logs anomalous?", "which service is the root cause?". We tested it on two public AIOps datasets against gpt-4.1-nano, a reasoning model (gpt-6.1-sol) and non-LLM baselines. This post walks through the signals, how we framed what to detect, and the results, including a first attempt that did not work. The code is on GitHub. !Decision-1 results on root-cause analysis and HDFS anomaly detection Highlights - Root-cause ranking was a strong fit. Decision-1 picked the root-cause service in 94% of 90 failure cases, beating a "largest metric deviation" heuristic (82%, p=0.013), at about a third of the chat baseline's latency. - Log anomaly detection needed work. The first framing scored 49%. With a computed summary, a better question and a calibrated threshold it reached 86% and led the other LLMs. - A small trained classifier still wins when you have labels: 95% with 50 labeled blocks. - Cheap and fast: about 240 ms p50, and cents for a full run at the prices we found (unverified). Part 1: HDFS log anomaly detection The signal: what a block's log looks like The Loghub HDFS_v1 dataset (free for research use; Xu et al., SOSP'09; Zhu et al., ISSRE'23) groups the logs of a Hadoop distributed file system by block: every block gets an identifier, and everything the cluster logged about it forms one session. Parsed into 29 event templates, the sequence for one block tells a story: 1. The NameNode allocates the block. 2. Three DataNodes (replication factor 3) each start receiving it and, when done, log that they received it. 3. Each replica's PacketResponder terminates, acknowledging the write. 4. Each DataNode reports the block to the NameNode, which logs the block map update (addStoredBlock). 5. Later, deletions may appear. We do not reproduce raw log lines: the dataset is unsanitized and licensed for research. We work with event templates and counts only. The dataset also has a Type column derived from the label, so we do not use it (it would leak the answer). Labels come from the dataset's own rules plus manual review. Sampling matters: real prevalence of anomalies is about 2.9%. Our test set is balanced (600 normal, 600 anomalous), so accuracy here is not what you would see in production. What "anomalous" really means Looking at 300 blocks from our calibration split (150 normal, 150 anomalous) shows that "contains an error" is a bad definition: | Pattern | Normal | Anomalous | |---|---|---| | Complete lifecycle (1 allocation, 3 receives, 3 acks), no error message | 112 | 14 | | Complete receive/ack, but an error message is present | 38 | 45 | | Truncated: 1 or 2 receives, no acks | 0 | 65 | | Extra receive attempts (4 or 5) | 0 | 26 | - 38 of 150 normal blocks contain a "got exception while serving" message: a benign event. - 14 anomalous blocks look complete and log no error at all. In the ones we inspected, the NameNode block-map update appears once instead of three times, meaning only one replica was registered. - Truncated blocks are short, often 4 events, with a "writeBlock received exception". So the real signal is the lifecycle: expected steps, repeated three times, and a count of what is missing. A rule like "any exception event" gets 71% on the test split (it flags the benign exceptions and misses the quiet anomalies). A first attempt that failed Our first prompt was the obvious one. The state was the list of event templates with counts, and the question was "does this block's activity indicate an anomaly?". Decision-1 scored 49% (AUC 0.68), worse than the 71% rule. Looking at the saved outputs: - Its probability correlated 0.43 with the number of events, but only 0.14 with the label. - Short blocks (10 events or fewer), all anomalies, got a mean P(anomaly) of 0.45. Blocks of 15 to 20 events, 99% normal, got 0.66. - Error keywords ("exception") pushed the probability up, whether or not they mattered. The model reacts to lexical cues and volume, and treats missing events as healthy. That is a framing problem, not a reason to discard the model. Modeling what we want to detect We changed two things, and tested them on the calibration split (never the test split): 1. State: we added a computed lifecycle_summary (expected replicas = 3; allocations; replica writes started and completed; terminated acks; replicas registered with the NameNode; missing lifecycle steps; error events). This is feature engineering from HDFS domain knowledge, not something the model derives. 2. Question: instead of "is it abnormal?", ask the positive, structural one: "is the block's lifecycle complete?" Here is a real (short) block as sent. The first-attempt request carried only the part up to event_counts; the enriched one adds lifecycle_summary. This is a hard one: the lifecycle looks complete, and there is one "Unexpected error" event, but the dataset labels the block anomalous. The same pattern is also common among normal blocks (see the table above), which is why error keywords alone are a weak signal. { "model": "Decision-1", "state": { "system": "HDFS (Hadoop distributed file system), log events for a single data block", "total_events": 20, "event_counts": { "BLOCK* NameSystem allocateBlock:": 1, "Receiving block src: dest:": 3, "Received block of size from": 3, "PacketResponder for block terminating": 3, "BLOCK* NameSystem addStoredBlock: blockMap updated: is added to size": 3, "BLOCK* NameSystem delete: is added to invalidSet of": 3, "Deleting block file": 3, "Unexpected error trying to delete block BlockInfo not found in volumeMap": 1 }, "lifecycle_summary": { "expected_replicas": 3, "block_allocations": 1, "replica_writes_started": 3, "replica_writes_completed": 3, "pipeline_acks_terminated": 3, "replicas_registered_with_namenode": 3, "missing_lifecycle_steps": [], "exception_or_error_events": 1 } }, "questions": { "q": { "type": "noul", "instructions": "Did this block complete the normal HDFS write lifecycle on all expected replicas (allocated, written, acknowledged and registered)?" } } } The answer is a single probability that the lifecycle is complete; the anomaly score is 1 - P(complete). For a truncated block, missing_lifecycle_steps would instead list entries such as "write completed on every replica". | Variant (200 calibration blocks) | Accuracy @0.5 | AUC | |---|---|---| | "abnormal?", raw counts | 49.5% | 0.65 | | "is the lifecycle complete?", raw counts | 60.5% | 0.82 | | "abnormal?", counts + lifecycle summary | 73.5% | 0.91 | | choice normal / anomalous / uncertain + summary | 79.0% | n/a | | "lifecycle complete?" + summary | 70.5% | 0.97 | Both the state and the wording matter. The last variant ranks best (AUC 0.97, mean P(complete) 0.96 for normal and 0.48 for anomalous blocks), but 0.5 is the wrong threshold for it, so it needs calibrating. Results on 1,200 held-out blocks We picked the best variants on the calibration split and scored them on the full test split, with the same enriched state for every LLM and the threshold chosen on the calibration split only: | Model | "Complete?" @0.5 | "Complete?", calibrated | Choice with abstain | AUC | |---|---|---|---|---| | Decision-1 | 69% | 86% (thr 0.15) | 79% | 0.964 | | gpt-4.1-nano | 65% | 79% | 66% | 0.839 | | gpt-6.1-sol (n=100) | 66% | 66% | 79% | 0.790 | The choice variant never abstained (0 of 1,200). The reasoning model ran on only 100 blocks (CI roughly 70 to 86%), and its self-reported confidence was too coarse for calibration to help. Against non-LLM references on the same blocks: | Reference | Accuracy | |---|---| | Rule: any exception event | 70.8% | | Logistic regression, 50 labeled blocks | 95.1% | | Logistic regression, 200 labeled blocks | 97.5% | | Logistic regression, 8,000 labeled blocks | 99.7% | Note: the logistic regression is not a like-for-like comparison. It is a trained classifier: it needs labeled examples (the rows above are trained on 50, 200 and 8,000 labeled blocks), a training step, and a stable event schema, and it only sees raw event counts. Decision-1 and the other LLMs ran zero-shot with no training; we used labels only to pick a threshold (150 + 150 blocks), and they received our computed lifecycle summary on top of the counts. Decision-1 beat the rule (p<0.0001) but not a logistic regression on event counts. If you have labels and a stable schema, train a classifier. A zero-shot decision model fits when you do not have labels, or when the schema changes too often to retrain. Part 2: root-cause analysis on microservices The signal RCAEval RE2-OB (MIT license) injects faults into the Online Boutique demo (11 microservices on Kubernetes) and records three kinds of telemetry while it does: - Metrics: CPU, memory, sockets, disk, request latency (p50/p90) and error rate per service. - Traces: how long each service's spans take. - Logs: how many log lines per second each service emits. Each case injects one fault into one service, at a known moment. There are 5 root-cause services (checkout, currency, email, product catalog, recommendation) and 6 fault types (CPU, memory, disk, socket exhaustion, added delay, packet loss), each repeated 3 times: 90 cases. What an on-call engineer would see We do not hand the model raw time series. A script reduces each case's telemetry to a compact JSON state, and that state is the only thing Decision-1 sees: - Window: the 12 minutes before and the 12 minutes after the injection time. The state says an alert fired at the start of the "after" window; in a real incident you would use your alert time, which is less exact than the dataset's. - Metrics: for every service and metric (CPU, memory, sockets, disk, latency p50/p90, errors) we compare the after-mean with the before-mean, scaled by how much the metric varied before the fault. Only the 10 largest shifts are kept, each with its before, after and % change. - Traces: mean span duration per service before and after, keeping the 6 largest relative changes. - Logs: log lines per second per service before and after, keeping the 6 largest relative changes. We compute these deltas ourselves because Decision-1 is weak at arithmetic over raw numbers. The metric list is already sorted by that shift score, so the first entry is exactly the pick of the baseline rule below. What Decision-1 adds is combining that list with the trace and log evidence. A clear case: CPU fault on checkout. The top change is the checkout service's CPU going from 0.43 to 18.3 (+4,160%), followed by its memory (+38%) and sockets (+22%); the entries that follow are changes of a few percent. One service lights up on its own metrics: the signature is easy to read. A harder case: added delay on the email service. The email service's own p90 latency rises +4,500%, but the service that calls it, checkout, also slows down (p90 +106%, p50 +195%), and checkout's span time triples (14 to 43 ms) while email's own span time barely moves (0.17 to 0.18 ms). Both services appear in the top changes, and a reader can mistake the caller for the culprit. Packet loss is noisy. Loss on the product catalog service pushes latency up on the services that depend on it (recommendation p90 +68,900%), and the frontend starts reporting errors, so the loudest symptoms are not on the service at fault. What we ask Two questions per case, and each is a choice question: 1. Which of the 11 services is the root cause? 2. Which of the 6 fault types was injected? This is the whole request for the first question (lists shortened). The case id, which contains the true service and fault, is never sent: { "model": "Decision-1", "state": { "system": "Online Boutique microservices on Kubernetes (11 services). An alert fired at the start of the 'after' window; telemetry is summarized as before vs after.", "minutes_before_and_after": [12.0, 12.0], "top_metric_changes": [ {"service": "checkoutservice", "metric": "cpu", "before": 0.429, "after": 18.3, "change_pct": 4160.0} ], "trace_span_duration_by_service": [ {"service": "checkoutservice", "mean_span_ms_before": 12.1, "mean_span_ms_after": 340.5, "change_pct": 2714.0} ], "log_volume_by_service": [ {"service": "frontend", "log_lines_per_s_before": 9.8, "log_lines_per_s_after": 14.2, "change_pct": 44.9} ] }, "questions": { "q": { "type": "choice", "instructions": "Which service is the root cause of the incident?", "criteria": { "adservice": "The ad service is the origin of the fault", "checkoutservice": "The checkout service is the origin of the fault" } } } } The second call reuses the same state and asks "What kind of fault was injected into the root-cause service?" with the options cpu, mem, disk, delay, loss and socket. The response is a probability per option, and we take the highest. The chat baselines receive the same state and options as prompt text. The non-LLM reference is a "largest shift" rule: pick the service of the single metric that moved most relative to its pre-fault variability (the first entry in the state). It needs no training, and it is right 82% of the time. Results (90 cases) | Task | Decision-1 | gpt-4.1-nano | gpt-6.1-sol | Reference rule | |---|---|---|---|---| | Root-cause service (11 options) | 94% | 91% | 100% | 82% | | Fault type (6 options) | 51% | 47% | 54% | 40% | | p50 latency per call | 240 ms | 728 ms | 2,052 ms | n/a | Decision-1 beat the heuristic on the service (McNemar p=0.013). Differences against the two LLMs are not significant (vs gpt-4.1-nano p=0.55, vs gpt-6.1-sol p=0.06), and fault-type differences are not significant for anyone. Only 5 of the 11 services ever appear as a true root cause, which makes the task somewhat easier than it looks. Where the model helps, and where it does not Service (85 of 90 right). The heuristic and Decision-1 disagree on 17 cases. In 14, Decision-1 is right and the rule is wrong: the rule had picked a symptom (the frontend in 9 of them, an email or other service in the rest), mostly on packet-loss and delay faults, where the loudest numbers sit downstream of the fault. In 3, the rule is right and Decision-1 is wrong: all three are the email-delay case above, where Decision-1 named checkout, the caller. Its other two errors were a packet-loss case (it chose the frontend) and a product-catalog loss case (it chose recommendation). Those are the same "symptom versus cause" confusions. Fault type (46 of 90 right). Per-type results show a clear split: | Injected fault | Correct (of 15) | Most common wrong answer | |---|---|---| | CPU | 13 | disk (2) | | Delay | 13 | memory, disk (1 each) | | Memory | 9 | CPU (4) | | Disk | 9 | memory (4) | | Packet loss | 0 | memory (9), delay (5) | | Socket | 2 | CPU (8) | CPU and delay leave a distinctive trace. Packet loss and socket exhaustion do not: the same latency and error symptoms show up as other faults, and Decision-1 never answered "loss" once and chose "socket" only twice. Nothing in the state we give it says "packets were dropped", so this is as much a limit of our summary as of the model. The other two LLMs landed at 47% and 54%, so this is a hard task for all of them. Takeaway for practice. Use it to rank where to look, not to name the fault. It narrowed 11 services to the right one 94% of the time in a quarter of a second; telling you why is a job for a human, or for richer state. Cost and latency Token counts from the main run (1,380 calls for Decision-1 and gpt-4.1-nano, 280 for the reasoning model; calibration calls excluded), at list rates we have not verified (Decision-1 as reported by third-party aggregators, gpt-6.1-sol from OpenAI's list; Azure prices may differ, and hidden reasoning tokens may add to the reasoning model's bill): | Model | Input / output tokens | Approx. cost | Per call | |---|---|---|---| | Decision-1 (about $0.042 per 1M input, output free) | 353k / 1.4k | ~$0.015 | ~$0.00001 | | gpt-4.1-nano (~$0.10 in / $0.40 out per 1M) | 402k / 18k | ~$0.05 to 0.07 | ~$0.00004 | | gpt-6.1-sol ($2 in / $10 out per 1M) | 192k / 18k | ~$0.56 | ~$0.002 | At equal volume the reasoning model would cost about two orders of magnitude more than Decision-1, plus 2 s of latency per call versus 0.24 s. Check Azure Cost Management for the real bill. Caveats - The HDFS lifecycle summary was written by us from domain knowledge (replication factor 3), so it is feature engineering. We tried five variants and chose after seeing calibration results, which makes the headline 86% somewhat optimistic. The sample is balanced and the reasoning model ran on 100 blocks only. - Our pattern table comes from a 300-block calibration split; the 14 "quiet" anomalies were inspected by us and are not proof of a general rule. - RCA has 90 cases and only 5 true root-cause services; samples are small and margins wide. The state is pre-ranked by the same shift score the baseline rule uses, and uses the dataset's exact injection time. - One baseline per model family, temperature 0, our prompt, no few-shot examples. The chat models' "probability" is self-reported confidence, so it is coarser than Decision-1's. - We did not verify published scores for these datasets, so we cite none. Prices are unverified. - Decision-1 returns no explanation, so keep a human in the loop for consequential decisions. Takeaways - Decision-1 works best when the state already contains the facts that matter. Compute lifecycle checks and deltas instead of handing it raw counts. - Ask the structural question ("is the lifecycle complete?"), not the vague one ("is this abnormal?"). - Calibrate the threshold on held-out labeled data; 0.5 was a poor default. - With labeled data and a stable schema, a small trained classifier beat every LLM on HDFS. Use a decision model where you have no labels, and for fast gates like root-cause ranking. - Test every gate on your own data before you automate it. Links - Code and reproduction steps: pedrosakuma/decision-1-aiops - Loghub HDFS_v1: Loghub on GitHub, HDFS_v1 on Zenodo - Xu et al., Detecting Large-Scale System Problems by Mining Console Logs (SOSP 2009) - Zhu et al., Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics (ISSRE 2023) - RCAEval: paper (WWW 2025), code, dataset on Hugging Face - Online Boutique demo