GoogleCloudPlatform

Docker Build Cache on Ephemeral CI Runners: inline, registry and local, Measured --- title: Docker Build Cache on Ephemeral CI Runners: inline, registry and local, Measured published: true date: 2026-11-02 19:00:00 UTC tags: Docker,BuildKit,CICD,Caching canonical_url: devops-daily.com --- A CI runner that starts empty has no BuildKit cache, so every docker build is a cold build unless you export the cache somewhere and import it on the next run. We rebuilt a real multi-stage Node service, the payment service from Google's Online Boutique demo, on a brand-new BuildKit builder every time, with five setups: no cache, inline, registry with mode=min, registry with mode=max, and a local cache directory. For a one-line source change, every cache worked. The rebuild went from 79.0 s with no cache to 19.2 to 26.3 s with any registry-backed cache, because BuildKit skipped the whole builder stage, npm install included. For this Dockerfile, the cheapest backends were the best ones. Inline and registry mode=min cost 0.0 and 1.2 s to export. Inline adds only metadata to the image, and mode=min stored a separate 47.7 MB cache with the same layers as the image. mode=max stored 259.9 MB, added 29.8 s of export to every cold build, and made the dependency bump slower, not faster: the cached compiler toolchain saved no time once its layers had to be pulled, and the export added about 20 s. The local directory cache, written fresh each run, was the slowest of all for the one-line change. TL;DR - Source change: all caches skip the builder stage. Median 19.2 to 26.3 s with inline or registry, against 79.0 s with no cache. In our three runs each, inline and mode=min did as well as mode=max, because BuildKit only loads the cache for the last step that matches. - Export cost is mostly compression. mode=max spent 27.9 s in preparing build cache for export on a cold build, and about 2 s writing 259.9 MB to the registry. mode=min exported in 1.2 s, inline in 0.0 s. - Dependency bump: mode=max did not help. It kept the compiler toolchain, but pulling and unpacking the cached base and toolchain layers took 15.0 to 20.6 s per build, about what mode=min spent pulling the base image and running apk add again (13.4 to 19.8 s). Its export then added 18.4 to 24.7 s. Median 132.4 s for mode=max against 110.8 s for mode=min. - Local cache, fresh directory each run: 74.7 s for a one-line change, because it repeats the full mode=max export every time. - What did not change: npm install ran on every dependency bump with every backend. A cache saves the steps before the change, never the step that changed. - Our registry ran on the same machine, so these times leave out the network. The sizes carry over: 47.7 MB pulled per warm source build, 259.9 MB per mode=max cache. Prerequisites - Docker with Buildx (we used Docker 29.9.2, Buildx 0.38.1 and BuildKit 0.34.1). - A registry you can push to. We ran registry:2.8.3 on the build machine; GHCR, ECR, Docker Hub or your own registry work the same way. - Node.js and npm, only to create the dependency change (npm install --package-lock-only). - If you want to rerun the whole matrix: a 4 vCPU Linux machine and about 80 minutes. We used a droplet on DigitalOcean (referral link). Why a fresh builder changes everything On a laptop, BuildKit keeps every layer it built in its own store, so the second docker build reuses them without any configuration. A CI runner that starts from a clean VM or container has no store. Unless you tell BuildKit where to import a cache from and where to export it to, every build is a cold build. BuildKit can export its cache in several ways. inline writes cache metadata into the image you push. registry writes a separate cache manifest to a registry. local writes an OCI layout to a directory that your CI then saves and restores. gha and s3 write to GitHub's cache service and to S3. mode=min, the default, exports only the layers of the final image. mode=max also exports the layers of every intermediate stage, such as a builder stage. The question for a CI pipeline is which of those pays for itself, for the changes that you actually push. How we measured The Dockerfile. We used src/paymentservice from Google's Online Boutique demo at commit c7cb6d5b, without changes. It is a real multi-stage Node service, and both base images are pinned by digest. This is the whole Dockerfile without its license header: # Define a default value so it's not empty if the builder fails to provide it ARG BUILDPLATFORM=linux/amd64 FROM --platform=$BUILDPLATFORM node:24.20.0-alpine@sha256:e67514e5d0f6c46656005e1b693b2ec9d52e80b641307de684d4a015ba7a4eaf AS builder # Some packages (e.g. @google-cloud/profiler) require additional # deps for post-install scripts RUN apk add --update --no-cache \ python3 \ make \ g++ WORKDIR /usr/src/app COPY package*.json ./ RUN npm install --only=production FROM alpine:3.24.1@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b RUN apk add --no-cache nodejs WORKDIR /usr/src/app COPY --from=builder /usr/src/app/node_modules ./node_modules COPY . . EXPOSE 50051 ENTRYPOINT ["node", "index.js"] The expensive work is in the builder stage: a compiler toolchain from Alpine's mirror, then npm install. The final stage installs Node from Alpine, copies node_modules, and copies the source last. An ephemeral runner, every time. Each build got a new BuildKit builder (docker buildx create --driver docker-container --bootstrap), and we removed it after the build. Nothing survived in the builder's store between builds except what the cache backend exported. The base image mirror and the Docker host itself did persist. We recorded builder start-up separately; it is the same for every backend. Three scenarios: - Cold: an empty cache. The build exports its cache to the backend. - Source change: one line in server.js (a log message). The build imports the cache that the last cold run wrote. - Dependency change: @grpc/grpc-js from 1.14.4 to 1.14.7, made with npm install --package-lock-only --save-exact, the size of a typical Dependabot patch bump. The lockfile diff changes only that package (npm also re-sorted two keys in package.json). It imports the same cache. Warm builds import from the cold cache and export to a separate scratch location, so every warm run starts from the same cache and still pays the export cost that a real pipeline pays. The backends: | Name | Import | Export | | --- | --- | --- | | none | nothing | nothing | | inline | type=registry,ref=<image> | type=inline (cache metadata inside the pushed image) | | registry, min | type=registry,ref=<cache> | type=registry,ref=<cache>,mode=min | | registry, max | type=registry,ref=<cache> | type=registry,ref=<cache>,mode=max | | local | type=local,src=<dir> | type=local,dest=<dir>,mode=max | Every build pushed its image with --provenance=false --sbom=false, so attestations do not add noise. Repeats. We planned five runs of every backend and scenario. The shared droplet ran slower than planned and we had a fixed budget, so we got 5 cold runs per backend, 3 source-change runs, and 3 (2 for the local cache) for the dependency bump. Runs went in round-robin order inside each scenario (run 1 of every backend, then run 2, and so on), which spreads slow periods on the shared host across the backends but does not remove order effects (see the caveats). The tables show the median and the number of runs (with 2 runs, the median is their mean). We dropped the page cache before each build and logged the CPU counters around each one, because the droplet's vCPUs are shared and steal time moves build times. What we recorded. Wall time of docker buildx build, BuildKit's own step timings from --progress=plain (which steps were CACHED, how long the export took), the cache size from the registry manifest or the cache directory, and the bytes BuildKit pulled from the cache registry, from the registry's access log. Where the cache lived. The registry for inline and type=registry was a registry:2.8.3 container on the same droplet, at 127.0.0.1:5000. We did not have a container registry we could use at no cost, and DigitalOcean's Spaces object storage has a monthly minimum, so there was no remote store that cost nothing. On loopback, writing the 259.9 MB mode=max cache took about 2 s, so our export and import times are close to the best case. On a real runner the same bytes cross a network to GHCR, ECR or your own registry, and the time grows with the size. That is why the tables show bytes next to every time: the bytes carry over to your setup, our seconds do not. Base images. Every fresh builder resolved node:24.20.0-alpine and alpine:3.24.1, and downloaded their layers whenever a step needed them. We pointed BuildKit at a pull-through mirror (registry:2.8.3 with REGISTRY_PROXY_REMOTEURL, also on the droplet), so Docker Hub's rate limits and speed stay out of the numbers. A real ephemeral runner pulls them from Docker Hub or a mirror on every build that needs them, so add that time to the rows where the FROM step ran. Package downloads were real. apk add fetched from Alpine's CDN and npm install fetched from the npm registry whenever those steps ran. That is the cost a cache saves, so we left it in. Why there is no type=s3 or type=gha. type=s3 needs an S3 bucket. The only free option here was an S3 emulator on the same droplet, which would measure the emulator, not S3. type=gha works only inside GitHub Actions. We left both out. Cold builds and what the export costs | Backend | Runs | Cold build | Cache export | of which preparing | Cache size | | --- | --- | --- | --- | --- | --- | | none | 5 | 79.0 s | - | - | - | | inline | 5 | 85.4 s | 0.0 s | - | in the image | | registry, mode=min | 5 | 89.4 s | 1.2 s | 0.1 s | 47.7 MB | | registry, mode=max | 5 | 128.0 s | 29.8 s | 27.9 s | 259.9 MB | | local, mode=max | 5 | 122.8 s | 39.1 s | 30.7 s | 259.9 MB | Cold builds: median of 5 runs. Export is the exporting cache step in the BuildKit log; for inline, the preparing layers for inline cache line. With no cache, a cold build took a median 79.0 s on a fresh builder. npm install was 45.3 s of that, the compiler toolchain (apk add python3 make g++) 6.5 s, and compressing and pushing the image about 9.5 s. The totals in the table are noisier than the export column. npm install alone took between 29.2 and 70.9 s across the cold runs, because it downloads from the npm registry, and the droplet reported 8 to 27% CPU steal per build. Read the export column for what each backend costs: - inline costs nothing measurable. BuildKit logged preparing layers for inline cache 0.0s done. The cache is metadata in the image config, and its layers are the image's own. - registry, mode=min added 1.2 s for a 47.7 MB cache: the same 5 layers as the final image, stored under the cache tag. - registry, mode=max added 29.8 s for 259.9 MB in 12 layers. 27.9 s of that was one line in the BuildKit log, preparing build cache for export: BuildKit compressing the builder stage's layers (the Node base image, the compiler toolchain and node_modules), which a mode=min export never touches. Writing the layers to the registry took about 2 s, on loopback. - local, mode=max added 39.1 s for the same 259.9 MB, written as an OCI layout on disk. Preparing took 30.7 s here too. So the expensive part of a mode=max export is compression, and it happens before a byte leaves the machine. On a real network, the upload of 259.9 MB comes on top. A one-line source change | Backend | Runs | Source change | Steps cached (of 8) | Pulling cached layers | apk add (builder) | npm install | Pulled from registry | | --- | --- | --- | --- | --- | --- | --- | --- | | none (a cold build) | 5 | 79.0 s | 0 | - | 6.5 s | 45.3 s | - | | inline | 3 | 26.3 s | 7 | 16.9 s | 0.0 s | 0.0 s | 47.7 MB | | registry, mode=min | 3 | 19.2 s | 7 | 12.4 s | 0.0 s | 0.0 s | 47.7 MB | | registry, mode=max | 3 | 22.9 s | 7 | 15.7 s | 0.0 s | 0.0 s | 47.7 MB | | local, mode=max | 3 | 74.7 s | 7 | - | 0.0 s | 0.0 s | - | The none row is the cold build. Pulling cached layers is the time BuildKit reported on the cached step that had to be unpacked; pulled from registry is from the cache registry's access log. Every backend skipped the whole builder stage. BuildKit marked 7 of the 8 build steps CACHED, including RUN npm install, and only COPY . . ran. That was true for inline and registry mode=min too, although those caches do not contain a single layer of the builder stage. That works because of how BuildKit finds a cache hit. A step's cache key comes from its instruction and the keys of the steps before it, and a COPY from the build context adds a checksum of the files it copies. In this graph, that let BuildKit match the final stage's COPY --from=builder step without running the builder stage. Its solver documentation says it "will not require the cache to be loaded or exists for all the vertexes in the graph but will only load it for the final node that is determined to match cache". Here, that node is COPY --from=builder, which is a layer of the final image, and every backend exported it. The builder stage never ran, and none of its layers were downloaded: every registry build pulled the same 47.7 MB, the final image's layers, even from the 259.9 MB mode=max cache. What a warm build still paid for was pulling and unpacking those cached layers to put COPY . . on top. BuildKit reports that time on the cached COPY --from step: a median of 12.4 to 16.9 s with the registry backends, out of 19.2 to 26.3 s for the whole build. The local backend was the exception: 74.7 s. Importing from the directory was quick, but exporting to a fresh directory took 52.4 s, because every run wrote the full mode=max cache again. The registry backends exported to a new tag in the same repository, and that took 1.4 s or less, because the layers they referenced were already in the registry. We exported to a new directory on purpose, to stop the cache from growing; the section on the local cache below explains the trade. A dependency bump | Backend | Runs | Dependency bump | Steps cached (of 8) | Pulling cached layers | apk add (builder) | npm install | Pulled from registry | | --- | --- | --- | --- | --- | --- | --- | --- | | none (a cold build) | 5 | 79.0 s | 0 | - | 6.5 s | 45.3 s | - | | inline | 3 | 105.7 s | 2 | 2.8 s | 9.1 s | 60.0 s | 26.0 MB | | registry, mode=min | 3 | 110.8 s | 2 | 2.3 s | 9.9 s | 61.5 s | 26.0 MB | | registry, mode=max | 3 | 132.4 s | 4 | 18.0 s | 0.0 s | 66.9 s | 196.8 MB | | local, mode=max | 2 | 132.1 s | 4 | - | 0.0 s | 51.9 s | - | A changed lockfile invalidates COPY package*.json in the builder stage, so npm install ran in every build, with every backend. The backends differ only in the step before it, the compiler toolchain: - inline and registry mode=min do not hold the builder stage, so BuildKit pulled the Node base image from the mirror (4.3 to 10.2 s) and ran apk add python3 make g++ again (a median 9.9 s). Only the final stage's RUN apk add --no-cache nodejs and WORKDIR came from the cache. - registry mode=max and local held it, and BuildKit marked the toolchain step CACHED. But to run npm install on top of it, BuildKit first had to pull and unpack the cached layers, the Node base image and the toolchain, including one layer of 112.3 MB. With the registry, that took 15.0 to 20.6 s per build, about the same as the 13.4 to 19.8 s that mode=min spent on the base image and apk add. The cache hit saved nothing. We expected mode=max to win this scenario, and wrote that down before the runs. It did not. Then the export. With mode=max, BuildKit also had to compress the builder stage's new npm install layer, which the image does not contain, and preparing build cache for export took 19.0 s, against 0.7 s for the whole mode=min export. Put together, the dependency bump took a median 132.4 s with mode=max and 110.8 s with mode=min, and mode=max was slower in each of the three rounds. The export accounts for most of the gap: mode=max took 18.4 to 24.7 s to export, against under 1 s for mode=min, while the toolchain stage cost about the same either way. mode=max also pulled 196.8 MB from the cache registry per build, against 26.0 MB for mode=min. The local cache, also mode=max, took 132.1 s (2 runs). With a remote registry the gap would grow, because those 196.8 MB and the new npm install layer would cross the network. Build time on a fresh builder (median) | | Value | Series | | --- | --- | --- | | none | 79 s | source change | | none | 79 s | dependency bump | | inline | 26.3 s | source change | | inline | 105.7 s | dependency bump | | registry, mode=min | 19.2 s | source change | | registry, mode=min | 110.8 s | dependency bump | | registry, mode=max | 22.9 s | source change | | registry, mode=max | 132.4 s | dependency bump | | local, mode=max | 74.7 s | source change | | local, mode=max | 132.1 s | dependency bump | _One new BuildKit builder per build; cache registry on the same droplet (loopback). Medians; the none bars are the cold build; run counts are in the tables. s-4vcpu-8gb, 2 November 2026._ The local cache: an expensive export type=local writes the cache as an OCI layout in a directory, and your CI saves and restores that directory between runs, for example with a cache action. In our runs the import side was cheap: reading the cache manifest took 0.1 s or less, and the cached layers came from local disk. The export side was the most expensive of all the backends. A cold build spent 39.1 s on it, and a one-line change spent 52.4 s, because each run wrote the full mode=max cache into a new, empty directory. The registry backends wrote their warm cache to a new tag in a registry that already held the layers, and finished in 1.4 s or less. We wrote to a fresh directory on purpose. When you export to the same directory you import from, BuildKit by default keeps every old blob. Docker's GitHub Actions cache guide puts it plainly: exporting to a directory that already holds a cache "leaves the blobs of the previous export behind, so the directory grows with every run". BuildKit 0.34.1 documents an export option for this, reset=true, which removes "any blobs in the cache directory that are not referenced by the current manifests in index.json" and is "useful for keeping the local cache directory from growing indefinitely". We planned to measure both the growth and reset=true, and ran out of droplet time first, so we cannot tell you how much either one changes. Which backend to use For this Dockerfile, on a runner that starts empty: - Use inline or registry mode=min. They gave the same warm builds as mode=max for a source change, cost 0 to 1.2 s to export, and stored no more than the image's own 47.7 MB of layers. Inline needs no extra tag at all, but it only works when you push the image, and its cache is only as old as the last image you pushed. - Use mode=max only when a builder step is slow to run and cheap to pull. BuildKit pulls and unpacks a cached layer before it can build on top of it. Here the toolchain step took about 10 s to run, and pulling its cached layers with the base image took about as long as pulling the base image and running it again. A long compile that produces a small output layer is the opposite case, and that is where mode=max earns its export cost. - For a slow Gradle or Bazel step, look past the layer cache. BuildKit reuses a RUN step whole or runs it again, so a source change before that step reruns the whole build. A tool's own remote build cache works per task and skips the tasks whose inputs did not change. Bitrise, for example, sells one for Gradle, Bazel, Xcode and React Native that it says works on any CI (checked on its public pages on 2 November 2026). We did not test one here. - Price the export, not just the hit. mode=max added about 30 s of compression to every cold build and 19.0 s to every dependency bump, before any upload. - Be careful with type=local. Writing to a fresh directory each run, the older way to keep the cache small, repeats the full mode=max export every time, which took 52.4 s on a one-line change. Docker's guide now reuses the directory with reset=true, which needs Buildx 0.35 or later and removes blobs that the new cache no longer references. We did not get to test it. To check your own Dockerfile, build it twice on a fresh builder with --progress=plain and read three lines per step: whether it says CACHED, how long its DONE takes after that (the pull), and how long preparing build cache for export takes. Those three numbers decide which backend wins. What we could not conclude - The cache registry was on loopback. Our export and import times leave out the network. With a remote registry, add the transfer of 47.7 MB for a source change with any backend, and of the 259.9 MB mode=max cache on every cold export that writes new layers. - One Dockerfile. In this one, the expensive work comes before COPY . ., so any cache skips it on a source change. In a Dockerfile whose expensive step comes after COPY . ., no backend can help with a source change. The Go frontend of the same demo is like that: its go build step comes after COPY . . and took 66.0 to 105.8 s in three uncached runs of it that we made on the same droplet for our CI build memory post. - No no-cache rerun of the changes. With no cache, a source change or a dependency bump is a cold build, so we used the cold median (79.0 s) as the reference and did not rerun it with the changed files. The cold builds ran earlier in the session, when steal was lower (8 to 27%, against 15 to 39% for the warm builds), so if anything that reference is a little fast. - Fewer warm runs than planned. Builds on the shared droplet ran slower than we planned, and we had a fixed budget. Cold builds have 5 runs per backend, source changes 3, and dependency bumps 3 (2 for the local cache). The direction of every comparison we draw held in every round; the exact medians would move with more runs. - Fixed order inside each round. The backends ran in the same order in every round, so a backend's position could bias its time. The host-wide steal we logged per build does not show a pattern by position, but it rose over the session (from 8% to 39% per build). Dependency round 3 ran in the opposite order, mode=max first, and mode=max was still the slowest registry backend. We point at the step timings because they depend less on any of this. - The local result is about a fresh destination. We exported each local warm build to a new, empty directory, while the registry backends exported next to layers that were already stored. That shows that a fresh local destination is expensive, not that type=local is slow in general, and we did not measure the cache action's save and restore of the directory. - Untested: type=gha and type=s3, RUN --mount=type=cache cache mounts, two pipelines writing the same cache tag at once, and the reset=true option for local caches. Run it yourself Create a 4 vCPU Ubuntu machine with Docker, copy the files below to /root/bench/, and run: bash /root/bench/registries.sh STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh prep # contexts for the source and dependency changes STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh cold # 5 rounds x 5 backends STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh warm # 5 source rounds, then dependency rounds while time allows python3 summarize_p2.py /root/res/p2 The script below is the one we ran. Our run did not use warm to the end: to fit the budget we stopped it after three source rounds, ran two dependency rounds with warm2, and a third with dep3, which starts with mode=max. packnow (a tar and zstd test of the local cache) and grow (the same directory in and out, with and without reset=true) are in the script too; we ran out of time before them. Each build writes its own directory with the BuildKit log, the registry access log for that build, and the sizes, so you can check any single number. #registries.sh: cache registry, pull-through mirrors, pinned source #!/bin/bash # The cache registry and the two pull-through mirrors, all on the build machine. docker pull -q moby/buildkit:v0.34.1 docker pull -q registry:2.8.3 docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3 docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3 docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3 git init -q /root/src/msd git -C /root/src/msd remote add origin https://github.com/GoogleCloudPlatform/microservices-demo git -C /root/src/msd fetch -q --depth 1 origin c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712 git -C /root/src/msd checkout -q FETCH_HEAD #buildkitd.toml: base image pulls go to the local mirror [registry."docker.io"] mirrors = ["127.0.0.1:5001"] [registry."gcr.io"] mirrors = ["127.0.0.1:5002"] [registry."127.0.0.1:5000"] http = true insecure = true [registry."127.0.0.1:5001"] http = true insecure = true [registry."127.0.0.1:5002"] http = true insecure = true #p2.sh: one fresh builder per build, every backend and scenario #!/bin/bash # Post 2: BuildKit cache backends on a fresh builder per build. # usage: p2.sh prep | smoke | cold | pack | warm | grow | warmnone set -u export PATH=/usr/local/bin:$PATH B=/root/bench R=/root/res/p2 C=/root/p2 SVC=/root/src/msd/src/paymentservice # stop starting new builds after this time, to stay inside a machine budget (our last stages used 2026-11-02T18:50:00Z) STOP_AT=$(date -u -d "${STOP_AT_UTC:-tomorrow}" +%s) REG=127.0.0.1:5000 BK=moby/buildkit:v0.34.1 mkdir -p $R $C/cache log() { echo "$(date -u +%FT%TZ) $*" | tee -a $R/progress.log; } prep() { rm -rf $C/ctx-* cp -a $SVC $C/ctx-base cp -a $SVC $C/ctx-src sed -i 's/PaymentService gRPC server started on port/PaymentService gRPC server listening on port/' $C/ctx-src/server.js cp -a $SVC $C/ctx-dep ( cd $C/ctx-dep && npm install --package-lock-only --save-exact --ignore-scripts --no-audit --no-fund @grpc/grpc-js@1.14.7 ) for i in 1 2 3 4 5; do cp -a $SVC $C/ctx-grow$i sed -i "s/PaymentService gRPC server started on port/PaymentService gRPC server (build $i) started on port/" $C/ctx-grow$i/server.js done diff -u $SVC/server.js $C/ctx-src/server.js > $R/src-change.diff diff -u $SVC/package.json $C/ctx-dep/package.json > $R/dep-change.diff diff -u $SVC/package-lock.json $C/ctx-dep/package-lock.json >> $R/dep-change.diff grep -c "listening on port" $C/ctx-src/server.js head -60 $R/dep-change.diff } # build CFG SCEN RUN [CTX] build() { local cfg=$1 scen=$2 r=$3 ctx=${4:-} local out=$R/$scen/$cfg/$r name=p2-$cfg-$scen-$r if [$(date +%s) -gt $STOP_AT]; then log "$scen $cfg $r SKIPPED (time budget)"; return; fi rm -rf $out; mkdir -p $out [-z "$ctx"] && case $scen in cold) ctx=$C/ctx-base ;; src) ctx=$C/ctx-src ;; dep) ctx=$C/ctx-dep ;; esac local img=$REG/paymentservice:$cfg-$scen from=() to=() case $cfg in none) ;; inline) to=(--cache-to type=inline) if [$scen = cold]; then img=$REG/paymentservice:inline-base else from=(--cache-from type=registry,ref=$REG/paymentservice:inline-base); fi ;; regmin|regmax) local m=${cfg#reg} if [$scen = cold]; then to=(--cache-to type=registry,ref=$REG/cache:$m,mode=$m) else from=(--cache-from type=registry,ref=$REG/cache:$m); to=(--cache-to type=registry,ref=$REG/cache:$m-scratch,mode=$m); fi ;; local) if [$scen = cold]; then rm -rf $C/cache/local; to=(--cache-to type=local,dest=$C/cache/local,mode=max) else rm -rf $C/cache/scratch; from=(--cache-from type=local,src=$C/cache/local); to=(--cache-to type=local,dest=$C/cache/scratch,mode=max); fi ;; grow) from=(--cache-from type=local,src=$C/cache/grow); to=(--cache-to type=local,dest=$C/cache/grow,mode=max) ;; growr) from=(--cache-from type=local,src=$C/cache/growr); to=(--cache-to type=local,dest=$C/cache/growr,mode=max,reset=true) ;; esac sync; echo 3 > /proc/sys/vm/drop_caches local t0 t1 t2 t3 rc t0=$(date +%s.%N) docker buildx create --name $name --driver docker-container --driver-opt network=host \ --driver-opt image=$BK --buildkitd-config $B/buildkitd.toml --bootstrap > $out/create.log 2>&1 t1=$(date +%s.%N) local cpu0; cpu0=$(grep '^cpu ' /proc/stat) docker buildx build --builder $name --progress=plain --provenance=false --sbom=false \ --metadata-file $out/meta.json "${from[@]}" "${to[@]}" \ --output type=image,name=$img,push=true,registry.insecure=true $ctx > $out/build.log 2>&1 rc=$? t2=$(date +%s.%N) local cpu1; cpu1=$(grep '^cpu ' /proc/stat) docker buildx rm $name > /dev/null 2>&1 t3=$(date +%s.%N) docker logs --since $t1 --until $t2 reg-cache > $out/registry.log 2>&1 { echo "rc=$rc"; echo "t0=$t0"; echo "t1=$t1"; echo "t2=$t2"; echo "t3=$t3"; echo "ctx=$ctx"; echo "cpu0=$cpu0"; echo "cpu1=$cpu1"; } > $out/result.env case $cfg in local) if [$scen = cold]; then d=$C/cache/local; else d=$C/cache/scratch; fi echo "cache_bytes=$(du -sb $d | cut -f1)" >> $out/result.env echo "cache_blobs=$(ls $d/blobs/sha256 | wc -l)" >> $out/result.env [$scen = cold] || rm -rf $C/cache/scratch ;; grow|growr) echo "cache_bytes=$(du -sb $C/cache/$cfg | cut -f1)" >> $out/result.env echo "cache_blobs=$(ls $C/cache/$cfg/blobs/sha256 | wc -l)" >> $out/result.env ;; esac case $cfg-$scen in regmin-cold) python3 $B/regsize.py $REG cache min > $out/cache-size.json ;; regmax-cold) python3 $B/regsize.py $REG cache max > $out/cache-size.json ;; regmin-*) python3 $B/regsize.py $REG cache min-scratch > $out/cache-size.json ;; regmax-*) python3 $B/regsize.py $REG cache max-scratch > $out/cache-size.json ;; esac python3 $B/regsize.py $REG paymentservice ${img##*:} > $out/image-size.json log "$scen $cfg $r rc=$rc build=$(python3 -c "print(round($t2-$t1,1))")s" } pack_test() { for r in r1 r2 r3 r4 r5; do sync; echo 3 > /proc/sys/vm/drop_caches t0=$(date +%s.%N); tar -C $C/cache/local -cf - . | zstd -T0 -q -f -o $C/local.tar.zst; sync; t1=$(date +%s.%N) rm -rf $C/unpack; mkdir -p $C/unpack; sync; echo 3 > /proc/sys/vm/drop_caches t2=$(date +%s.%N); zstd -dc -q $C/local.tar.zst | tar -C $C/unpack -xf -; sync; t3=$(date +%s.%N) echo "$r pack_s=$(python3 -c "print(round($t1-$t0,2))") unpack_s=$(python3 -c "print(round($t3-$t2,2))") dir_bytes=$(du -sb $C/cache/local | cut -f1) tar_zst_bytes=$(stat -c %s $C/local.tar.zst)" | tee -a $R/pack.txt done rm -rf $C/unpack $C/local.tar.zst; log PACK-DONE } case ${1:-} in prep) prep ;; smoke) for c in none inline regmin regmax local; do build $c cold s0; done; log SMOKE-DONE ;; cold) for r in r1 r2 r3 r4 r5; do for c in none inline regmin regmax local; do build $c cold $r; done; done; log COLD-DONE ;; warm) # all source-change rounds first, then whole dependency rounds while they fit before STOP_AT for r in r1 r2 r3 r4 r5; do for c in inline regmin regmax local; do build $c src $r; done; done last=400 for r in r1 r2 r3 r4 r5; do t=$(date +%s) if [$((t + last)) -gt $STOP_AT]; then log "dep $r and later skipped: a round needs about ${last}s"; break; fi for c in inline regmin regmax local; do build $c dep $r; done last=$(( $(date +%s) - t )) done log WARM-DONE ;; warm2) # switched to after source round 3: dependency rounds first, then the last source rounds, while they fit last=400 for r in r1 r2 r3 r4 r5; do t=$(date +%s) if [$((t + last)) -gt $STOP_AT]; then log "dep $r and later skipped: a round needs about ${last}s"; break; fi for c in inline regmin regmax local; do build $c dep $r; done last=$(( $(date +%s) - t )) [$r = r3] && break done for r in r4 r5; do t=$(date +%s) if [$((t + 150)) -gt $STOP_AT]; then log "src $r skipped: time budget"; break; fi for c in inline regmin regmax local; do build $c src $r; done done log WARM-DONE ;; dep3) # third dependency round, started after round 2 (time budget) for c in regmax regmin inline local; do build $c dep r3; done log WARM-DONE ;; packnow) pack_test ;; warmnone) for r in r1 r2 r3 r4 r5; do for s in src dep; do build none $s $r; done; done; log WARMNONE-DONE ;; pack) log "pack deferred until after the warm builds" ;; grow) # local cache pack/unpack test, then the same-directory growth test, without and with reset=true if [$(date +%s) -lt $((STOP_AT + 120))]; then pack_test; fi rm -rf $C/cache/grow $C/cache/growr; cp -a $C/cache/local $C/cache/grow; cp -a $C/cache/local $C/cache/growr echo "start cache_bytes=$(du -sb $C/cache/grow | cut -f1) blobs=$(ls $C/cache/grow/blobs/sha256 | wc -l)" > $R/grow-start.txt for i in 1 2 3 4 5; do build grow src g$i $C/ctx-grow$i; build growr src g$i $C/ctx-grow$i; done; log GROW-DONE ;; *) echo "usage"; exit 2 ;; esac #regsize.py: compressed bytes behind a registry tag #!/usr/bin/env python3 """Sum the compressed layer bytes behind a registry tag (image or BuildKit cache).""" import json, sys, urllib.request reg, repo, tag = sys.argv[1:4] ACCEPT = ", ".join([ "application/vnd.oci.image.index.v1+json", "application/vnd.oci.image.manifest.v1+json", "application/vnd.docker.distribution.manifest.list.v2+json", "application/vnd.docker.distribution.manifest.v2+json", ]) def get(ref): req = urllib.request.Request(f"http://{reg}/v2/{repo}/manifests/{ref}", headers={"Accept": ACCEPT}) with urllib.request.urlopen(req, timeout=30) as r: return json.load(r) def walk(m, layers): for d in m.get("manifests", []): mt = d.get("mediaType", "") if "manifest" in mt or "index" in mt: walk(get(d["digest"]), layers) elif "layer" in mt: layers[d["digest"]] = d["size"] for d in m.get("layers", []): layers[d["digest"]] = d["size"] try: top = get(tag) except Exception as e: print(json.dumps({"error": str(e)})); sys.exit(0) layers = {} walk(top, layers) print(json.dumps({"tag": f"{repo}:{tag}", "mediaType": top.get("mediaType"), "layers": len(layers), "bytes": sum(layers.values()), "sizes": sorted(layers.values(), reverse=True)})) #summarize_p2.py: parse BuildKit logs, registry logs and sizes into medians #!/usr/bin/env python3 """Summarize post 2 runs: parse BuildKit plain progress, registry logs and sizes; medians per cell.""" import json, os, re, statistics, sys root = sys.argv[1] VERTEX = re.compile(r"^#(\d+) (.*)$") DONE = re.compile(r"^DONE ([\d.]+)s$") SUBDONE = re.compile(r"^(preparing layers for inline cache|preparing build cache for export|exporting layers|pushing layers|exporting manifest list|exporting config).*?([\d.]+)s done$") def env(path): d = {} if os.path.exists(path): for line in open(path): if "=" in line: k, v = line.strip().split("=", 1) d[k] = v return d def parse_build(path): v = {} order = [] for line in open(path, errors="replace"): line = line.rstrip("\n") m = VERTEX.match(line) if not m: continue vid, rest = int(m.group(1)), m.group(2) if vid not in v: v[vid] = {"name": rest, "done": None, "cached": False, "error": False, "sub": {}} order.append(vid) continue if rest == "CACHED": v[vid]["cached"] = True elif rest.startswith("ERROR"): v[vid]["error"] = True else: d = DONE.match(rest) if d: v[vid]["done"] = float(d.group(1)) s = SUBDONE.match(rest) if s: v[vid]["sub"][s.group(1)] = v[vid]["sub"].get(s.group(1), 0) + float(s.group(2)) return [v[i] for i in order] def registry_bytes(path): got = put = 0 seen_logrus = False lines = open(path, errors="replace").read().splitlines() if os.path.exists(path) else [] for line in lines: if 'msg="response completed"' in line: seen_logrus = True meth = re.search(r"http\.request\.method=(\w+)", line) uri = re.search(r'http\.request\.uri="([^"]+)"', line) w = re.search(r"http\.response\.written=(\d+)", line) if meth and uri and w and meth.group(1) == "GET" and "/blobs/" in uri.group(1): got += int(w.group(1)) if not seen_logrus: for line in lines: m = re.search(r'"GET (/v2/[^]*/blobs/[^]*) HTTP/[\d.]+" (\d+) (\d+)', line) if m and m.group(2) == "200": got += int(m.group(3)) return got def step_time(vs, pattern): for x in vs: if re.search(pattern, x["name"]): return 0.0 if x["cached"] else x["done"] return None def run(d): r = env(os.path.join(d, "result.env")) if not r: return None vs = parse_build(os.path.join(d, "build.log")) steps = [x for x in vs if re.match(r"^\[(builder|stage-\d+) \d+/\d+\]", x["name"])] exp = [x for x in vs if x["name"].startswith("exporting cache to")] img = [x for x in vs if x["name"] == "exporting to image"] imp = [x for x in vs if x["name"].startswith("importing cache manifest")] info = { "rc": int(r["rc"]), "bootstrap_s": round(float(r["t1"]) - float(r["t0"]), 2), "build_s": round(float(r["t2"]) - float(r["t1"]), 2), "steps": len(steps), "cached_steps": sum(1 for x in steps if x["cached"]), "cached_names": [x["name"] for x in steps if x["cached"]], "apk_builder_s": step_time(vs, r"RUN apk add --update"), "npm_install_s": step_time(vs, r"RUN npm install"), "apk_final_s": step_time(vs, r"RUN apk add --no-cache nodejs"), "copy_modules_s": step_time(vs, r"COPY --from=builder"), "export_cache_s": sum(x["done"] or 0 for x in exp) if exp else None, "export_image_s": img[0]["done"] if img else None, "inline_prep_s": img[0]["sub"].get("preparing layers for inline cache") if img else None, "prep_export_s": sum(x["sub"].get("preparing build cache for export", 0) for x in exp) if exp else None, "import_manifest_s": sum(x["done"] or 0 for x in imp) if imp else None, "pulled_from_cache_registry_bytes": registry_bytes(os.path.join(d, "registry.log")), "errors": [x["name"] for x in vs if x["error"]], # a CACHED step still reports DONE time when its layers must be pulled and unpacked for a later step "cached_pull_s": max([x["done"] or 0 for x in steps if x["cached"]] or [0]), } if "cpu0" in r and "cpu1" in r: a = [int(x) for x in r["cpu0"].split()[1:9]] b = [int(x) for x in r["cpu1"].split()[1:9]] dd = [y - x for x, y in zip(a, b)] info["steal_pct"] = round(100 * dd[7] / sum(dd), 1) if sum(dd) else None if "cache_bytes" in r: info["cache_bytes"] = int(r["cache_bytes"]) info["cache_blobs"] = int(r["cache_blobs"]) for f in ("cache-size.json", "image-size.json"): p = os.path.join(d, f) if os.path.exists(p): try: j = json.load(open(p)) info[f.split(".")[0].replace("-", "_") + "_bytes"] = j.get("bytes") info[f.split(".")[0].replace("-", "_") + "_layers"] = j.get("layers") except Exception: pass return info def med(xs): xs = [x for x in xs if x is not None] return round(statistics.median(xs), 2) if xs else None out = {} for scen in ("cold", "src", "dep"): for cfg in ("none", "inline", "regmin", "regmax", "local", "grow", "growr"): d0 = os.path.join(root, scen, cfg) if not os.path.isdir(d0): continue runs = {rn: run(os.path.join(d0, rn)) for rn in sorted(os.listdir(d0))} runs = {k: v for k, v in runs.items() if v} meas = [v for k, v in runs.items() if k.startswith("r")] keys = ["build_s", "bootstrap_s", "cached_steps", "apk_builder_s", "npm_install_s", "apk_final_s", "copy_modules_s", "export_cache_s", "export_image_s", "inline_prep_s", "prep_export_s", "import_manifest_s", "cached_pull_s", "pulled_from_cache_registry_bytes", "steal_pct", "cache_bytes", "cache_size_bytes", "image_size_bytes"] cell = {"n": len(meas), "rcs": [x["rc"] for x in meas]} for k in keys: cell[k] = med([x.get(k) for x in meas]) vals = [x.get(k) for x in meas if x.get(k) is not None] if vals and k in ("build_s", "export_cache_s", "import_manifest_s"): cell[k + "_range"] = [min(vals), max(vals)] cell["runs"] = runs out[f"{scen}/{cfg}"] = cell json.dump(out, open(os.path.join(root, "summary.json"), "w"), indent=1) MB = 1e6 print(f"{'cell':14} {'n':>2} {'build':>7} {'range':>14} {'cached':>6} {'apkB':>6} {'npm':>6} {'expC':>6} {'expI':>6} {'impM':>5} {'pullMB':>7} {'cacheMB':>8} {'imgMB':>6}") for k, c in out.items(): cb = c.get("cache_size_bytes") or c.get("cache_bytes") print(f"{k:14} {c['n']:>2} {str(c['build_s']):>7} {str(c.get('build_s_range')):>14} {str(c['cached_steps']):>6} {str(c['apk_builder_s']):>6} {str(c['npm_install_s']):>6} {str(c['export_cache_s']):>6} {str(c['export_image_s']):>6} {str(c['import_manifest_s']):>5} {round((c['pulled_from_cache_registry_bytes'] or 0)/MB,1):>7} {round(cb/MB,1) if cb else '-':>8} {round(c['image_size_bytes']/MB,1) if c['image_size_bytes'] else '-':>6} rcs={c['rcs']}") Summary A fresh CI runner has no BuildKit cache, so the backend you pick decides what gets reused. For this multi-stage Node service, the cheapest backends did the most: inline and registry mode=min cut a one-line rebuild from 79.0 s to about 20 s and cost almost nothing to export, because BuildKit could skip the whole builder stage by matching cache keys alone. mode=max stored five times as much, added about 30 s of compression to every cold build, and did not make the dependency bump faster: pulling the cached toolchain took as long as installing it, and the export came on top. Before you switch on mode=max because a guide says so, read --progress=plain for your own build and compare the pull time of each cached step with the time it takes to run. _Originally published at devops-daily.com._

How Much Memory Does a CI Build Need? Five Real Builds at 2, 4 and 8 GB --- title: How Much Memory Does a CI Build Need? Five Real Builds at 2, 4 and 8 GB published: true date: 2026-11-02 18:00:00 UTC tags: CICD,Memory,cgroups,Docker canonical_url: devops-daily.com --- We ran five real open-source builds (a Vite app build, a TypeScript type check, a Gradle build, a Go build and a Docker image build) inside 2, 4 and 8 GB memory limits on one 4 vCPU machine, with Gradle both with and without its daemon, two or three times at each limit. Four of the five finished inside 2 GB. The one that did not was the Excalidraw web app: its Vite production build also runs a full TypeScript check, together they held 3,248 MiB of anonymous memory, and the kernel killed the build at 2 GB in all 3 runs. The number that most tools show you was the wrong one for the other four. cgroup v2's memory.peak said the GitHub CLI's Go build peaked at 1,775 MiB, but only 424 MiB of that was anonymous memory, the heaps and stacks of its processes. Most of the rest was page cache, which the kernel drops when it needs the room, plus some kernel memory. A Docker image build looked like 2,072 MiB for the same reason. And a TypeScript type check that held 1,139 MiB with the whole machine available finished inside 2 GB with 970 MiB, because Node sizes its heap limit from the cgroup limit it finds. This post shows the numbers, the five traps behind them, and every script we used, so you can size a runner from your own builds instead of from a guess. TL;DR - Four of five builds fit in 2 GB. Only the Excalidraw Vite build needed more: 3,248 MiB of anonymous memory, killed at 2 GB, fine at 4 GB. - memory.peak counts page cache. The Go build and the Docker build showed 1,775 and 2,072 MiB, but their anonymous memory peaked at 424 and 355 MiB. Size from anonymous memory plus a margin, then confirm by running the build inside that limit. - Runtimes size their heaps from the cgroup limit. Inside 2 GB, Node's default heap limit was 1,120 MiB and the JVM's default maximum heap was 512 MiB. The type check used less memory at 2 GB than at 8 GB and still finished, about 24% slower. - The Gradle daemon keeps its memory after the build. Five seconds after gradlew exited, the daemon still held 1,048 MiB of anonymous memory. With --no-daemon, that dropped to under 1 MiB, and in our one-command runs the daemon gave no repeatable speed-up. - Limiting the docker CLI did not limit the build. The CLI's cgroup peaked at 40 MiB while the BuildKit container that did the work peaked at 2,072 MiB. - Wall times on a shared-CPU droplet are noisy: the CPUs reported 23.5% steal time across the runs we tracked. Trust the memory columns more than the time columns. Prerequisites - A Linux machine with cgroup v2 and systemd (any recent Ubuntu, Debian or Fedora). Check with stat -fc %T /sys/fs/cgroup, which prints cgroup2fs. - Root access, for systemd-run --scope and for dropping the page cache. - The toolchains for the builds you want to test. Our setup script installs Go, Node, Yarn, a JDK, Rust and Docker. - If you want to rerun our matrix as it is: a 4 vCPU, 8 GB machine and about two hours. We used a droplet on DigitalOcean (referral link). How we measured We used one DigitalOcean droplet, size s-4vcpu-8gb in fra1, with Ubuntu 24.04 and kernel 6.8. The kernel reports 7,941 MiB of memory and there is no swap. Every build ran in its own transient systemd scope: systemd-run --scope --slice=bench.slice --unit=p1-<build>-<limit>-<run> \ -p MemoryMax=2G -p MemorySwapMax=0 -p AllowedCPUs=0-3 -p OOMPolicy=continue \ /root/bench/wrap.sh <outdir> <timeout> <build command> These are cgroup limits on one droplet, not three real machine sizes. A real 2 GB runner has a smaller kernel footprint, a runner agent and often a Docker daemon in the same memory, so it has less room than our 2 GB scope. The 8 GB limit is above the 7.9 GiB that the droplet has, so the 8 GB column means "the whole machine". The rest of the setup: - CPU. Every scope was pinned to the same 4 vCPUs with AllowedCPUs=0-3, and nproc printed 4 inside each one. Compilers start about one job per CPU, so a 2 vCPU runner would use less memory and an 8 vCPU runner more. We did not measure other CPU counts. - Cold build, warm dependencies. We downloaded dependencies once before the runs: yarn install, go mod download, cargo fetch, the Gradle distribution and dependencies, and the base images for the Docker build. Before every run we deleted the build output and the compiler caches (excalidraw-app/build, GOCACHE, target/, Gradle's build/ and .gradle/), stopped any Gradle daemon, and dropped the page cache. That is a CI job that restores a dependency cache but not a build cache. - What we recorded. Wall time, the exit code, memory.peak at the end of the run, memory.events (how often the limit was hit, and OOM kills), and a sampler that read memory.stat every 100 ms for anonymous memory (anon) and page cache (file). - Repeats. Runs went in round-robin order: run 1 of every cell, then run 2, then run 3, so a slow period on the shared host does not land on one build. Builds ran slower than we planned, so to stay inside our droplet budget the third round covered only the 8 GB and 2 GB limits. That gives 3 runs per cell at 8 GB and 2 GB, and 2 at 4 GB. The tables show the median of 3, or the mean of 2 at 4 GB. Each build also had one warm-up run at 8 GB that we did not count; it downloaded Gradle and filled the image mirror. The builds, all pinned to exact commits: | Build | Command | Source | | --- | --- | --- | | Excalidraw app, Vite production build | yarn build:app:docker | excalidraw/excalidraw 8bcdca45 | | Excalidraw type check, 1,508 files | tsc -p tsconfig.json (noEmit is set) | same checkout | | PlantUML, Gradle 9.3.1 on JDK 21 | ./gradlew jar, daemon on and --no-daemon | plantuml/plantuml db06d70e | | GitHub CLI, Go 1.27.3 | go build ./cmd/gh | cli/cli 9490d204 | | Online Boutique frontend image, BuildKit 0.34.1 | docker buildx build | GoogleCloudPlatform/microservices-demo c7cb6d5b | | ripgrep 15.2.0, Rust 1.99.1 (one run) | cargo build --release --locked | BurntSushi/ripgrep tag 15.2.0 | Node was 24.22.2 with Yarn 1.22.22, Docker was 29.9.2 with Buildx 0.38.1. The results | Build | Anonymous peak at 8 GB | memory.peak at 8 GB | 8 GB | 4 GB | 2 GB | | --- | --- | --- | --- | --- | --- | | Excalidraw vite build | 3,248 MiB | 3,578 MiB | ok, 66 s | ok, 83 s | 3 of 3 OOM-killed (run 1 from the kernel log) | | Excalidraw tsc type check | 1,139 MiB | 1,271 MiB | ok, 59 s | ok, 62 s | ok, 73 s | | PlantUML Gradle, daemon | 1,143 MiB | 1,474 MiB | ok, 124 s | ok, 110 s | ok, 113 s | | PlantUML Gradle, --no-daemon | 1,144 MiB | 1,475 MiB | ok, 88 s | ok, 119 s | ok, 100 s | | GitHub CLI go build | 424 MiB | 1,775 MiB | ok, 141 s | ok, 198 s | ok, 169 s | | Online Boutique frontend, docker buildx build | 355 MiB | 2,072 MiB | ok, 131 s | ok, 129 s | ok, 110 s | Times are medians of 3 runs at 8 GB and 2 GB, and the mean of 2 runs at 4 GB. Memory columns are from the 8 GB runs. What the cgroup counter says vs what the build held (8 GB limit, median of 3) | | Value | Series | | --- | --- | --- | | Vite build | 3578 MiB | memory.peak | | Vite build | 3248 MiB | anonymous memory | | tsc check | 1271 MiB | memory.peak | | tsc check | 1139 MiB | anonymous memory | | Gradle (daemon) | 1474 MiB | memory.peak | | Gradle (daemon) | 1143 MiB | anonymous memory | | Gradle (no daemon) | 1475 MiB | memory.peak | | Gradle (no daemon) | 1144 MiB | anonymous memory | | Go build | 1775 MiB | memory.peak | | Go build | 424 MiB | anonymous memory | | Docker build | 2072 MiB | memory.peak | | Docker build | 355 MiB | anonymous memory | _memory.peak includes page cache; anonymous memory is the sampled peak of anon in memory.stat (100 ms). One s-4vcpu-8gb droplet, 4 vCPUs, 2 November 2026._ Three things stand out. Only one build reached the 2 GB limit at all. The max counter in memory.events, which counts how often a cgroup's usage was about to go over its limit, stayed at 0 for every other run. Memory use at 4 GB and at 8 GB was almost the same for every build, so the extra 4 GB bought nothing. And the gap between memory.peak and anonymous memory is large for exactly the builds that read and write many files: the Go compiler and BuildKit. The single Rust build, ripgrep 15.2.0 with cargo build --release, fits the same pattern: 896 MiB memory.peak, 572 MiB anonymous, 61.1 s. We ran it once at 8 GB only, because the full matrix did not fit our time budget. Trap 1: memory.peak counts page cache memory.peak is the highest value of memory.current that the cgroup reached. memory.current charges the cgroup for everything it caused to be in memory, and that includes page cache : the source files the compiler read, the object files it wrote, the layers BuildKit unpacked. Page cache is reclaimable. When the cgroup gets close to its limit, the kernel drops clean cache pages and writes back dirty ones before it ever considers killing a process. Anonymous memory is the part that cannot go anywhere without swap: heaps, stacks, the compiler's working data. memory.stat reports it as anon. That, plus a smaller amount of kernel memory (37 to 136 MiB in these builds), is what has to fit. Our sampler read it every 100 ms, so it can miss a shorter spike. memory.peak and the max counter cannot miss one, and they show that no build except Vite at 2 GB ever reached its limit. The safe test is still the one we ran: the build inside the limit you plan to buy. The Go build of the GitHub CLI shows the difference well. Its peak was 1,775 MiB. Anonymous memory never went over 424 MiB; about 1,400 MiB was page cache from the module sources and the build cache that go build writes. At the 2 GB limit, the same build finished with the same peak, because 1,775 MiB is still under 2,048 MiB. With a smaller limit the kernel may reclaim that cache instead of failing the build, but we did not test below 2 GB, so test the lower limit before you rely on it. So if a dashboard shows that your CI job "peaked at 1.8 GB", that does not mean a 2 GB runner is too small. Look at anon in memory.stat (or RSS in your container metrics), and then add a margin for the runner itself. Trap 2: a production build that is also a type check The Excalidraw build needed far more memory than anything else: 3,248 MiB of anonymous memory, against 1,139 MiB for a full type check of the same code. The reason is in its Vite config: checker({ typescript: true, eslint: envVars.VITE_APP_ENABLE_ESLINT === "false" ? undefined : { lintCommand: 'eslint "./**/*.{js,ts,tsx}"' }, // ... }), vite-plugin-checker runs the TypeScript checker during vite build too: in version 0.7.2, which Excalidraw pins, enableBuild defaults to true. The production .env turns ESLint off but leaves the type check on, and the build also writes source maps. So one vite build is Rollup, source maps and a full tsc run at the same time. At 2 GB the kernel killed it in all 3 runs, 69 to 122 seconds in, after the log said ✓ 1442 modules transformed. The kernel log names the cgroup and the process it chose: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=p1-vite-2G-r2.scope,mems_allowed=0,oom_memcg=/bench.slice/p1-vite-2G-r2.scope,task_memcg=/bench.slice/p1-vite-2G-r2.scope,task=MainThread,pid=77242,uid=0 Memory cgroup out of memory: Killed process 77242 (MainThread) total-vm:11451308kB, anon-rss:1138988kB, file-rss:4864kB, shmem-rss:0kB, UID:0 pgtables:29992kB oom_score_adj:0 Our wrapper's record of that run shows the kill. memory.events counted 39,825 times that the cgroup was about to go over its limit, and one OOM kill: vite build, 2 GB limit, run 2 $ cat /root/res/p1/vite/2G/r2/result.env rc=1 t0=1793637034.083018067 t1=1793637156.285208072 peak=2147483648 current_after=867889152 anon_after=801075200 file_after=38674432 $ cat /root/res/p1/vite/2G/r2/memory.events low 0 high 0 max 39825 oom 1 oom_kill 1 oom_group_kill 0 The process list right after the kill shows the second half of the build. The checker runs tsc as its own child process, and it was still running after Vite had died: 77265 1808 /bin/sh -c tsc --noEmit 77266 799108 node /root/src/excalidraw/node_modules/.bin/tsc --noEmit That tsc held 799,108 KiB (780 MiB) of resident memory under the 2 GB limit. Yarn reported Command failed with exit code 1, but the type check kept going until our harness stopped the scope. On a real runner it would keep using memory and CPU until the job ends. We did not split the 3,248 MiB peak at 8 GB between Rollup and the checker, so we cannot say exactly how much the type check adds. What we can say: if your bundler runs a checker, your build step needs the memory of both, and moving the type check to its own CI job is the first thing to try before you buy a bigger runner. Trap 3: runtimes size their heaps from the limit Node and the JVM both read the cgroup limit when they start and size their default heap from it. Our harness asks each runtime what it would use inside each limit: probe inside a 2 GB scope # what Node 24 and JDK 21 pick inside MemoryMax=2G (from p1.sh probe) $ cat /root/res/p1/probe-2G.txt nproc=4 memory.max=2147483648 node_heap_size_limit=1174405120 int ActiveProcessorCount = -1 {product} {default} size_t MaxHeapSize = 536870912 {product} {ergonomic} double MaxRAMPercentage = 25.000000 {product} {default} All four places side by side: | Inside | Node 24 heap_size_limit | JDK 21 default MaxHeapSize | | --- | --- | --- | | 2 GB scope | 1,120 MiB | 512 MiB | | 4 GB scope | 2,240 MiB | 1,024 MiB | | 8 GB scope | 2,240 MiB | 2,048 MiB | | whole droplet, no scope | 2,240 MiB | 1,986 MiB | Two consequences showed up in the runs. Heap use follows the limit. tsc --extendedDiagnostics prints Memory used, which is Node's heapUsed at the end of the run, in units of 1,000 bytes. With the whole machine, the type check of 1,508 files reported Memory used: 1018378K (median of 3, about 971 MiB). Inside 2 GB it reported 886903K (about 846 MiB), anonymous memory fell from 1,139 to 970 MiB, and the check finished about 24% slower (median 72.7 s against 58.8 s). The 2 GB run was 9.5 to 14.2 seconds slower in each of the three rounds. In rounds 2 and 3, where we logged the CPU counters, its steal time was only 4 to 7 points higher, so steal explains only part of the gap. More frequent garbage collection under the smaller heap limit would explain the rest, but we did not record GC activity, so that is a hypothesis. A big heap number at a high limit says little about what the program needs. We expected this check to fail at 2 GB with a heap error, and it did not. Node stopped at 2,240 MiB even with more memory. On our 8 GB machine, Node's default heap limit was 2,240 MiB with a 4 GB scope, an 8 GB scope and no scope at all. We did not test a bigger machine, so check what your runner gives you with node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)'. If a build reports JavaScript heap out of memory, that is Node's own limit, not the kernel's. Set NODE_OPTIONS=--max-old-space-size=<MiB> explicitly instead of only buying more memory. The JVM takes 25% of the limit by default (MaxRAMPercentage=25), so a Maven build or a test JVM without -Xmx gets 512 MiB inside 2 GB. Gradle sets its own daemon heap, which brings us to the next trap. Trap 4: the Gradle daemon outlives the build PlantUML's gradle.properties sets org.gradle.jvmargs=-Xmx1g. That cap kept the build inside 2 GB: anonymous memory peaked at about 1,143 MiB with the daemon and 1,144 MiB without it, and neither came near the limit. The difference is what happens after gradlew exits. With the daemon, which is the default, the JVM stays alive and keeps its heap. Five seconds after the build, the scope still held 1,378 MiB, 1,048 MiB of it anonymous, in one process: 37292 1105976 /usr/lib/jvm/java-21-openjdk-amd64/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED ... With --no-daemon, Gradle printed To honour the JVM settings for this build a single-use Daemon process will be forked and Daemon will be stopped at the end of the build, and five seconds after exit the anonymous memory in the scope was under 1 MiB. Anonymous memory around the end of the Gradle build (8 GB limit, run 1) | | -55 s | -50 s | -45 s | -40 s | -35 s | -30 s | -25 s | -20 s | -15 s | -10 s | -5 s | 0 s | +5 s | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | daemon (default) | 637 MiB | 857 MiB | 927 MiB | 982 MiB | 1053 MiB | 1059 MiB | 1066 MiB | 1073 MiB | 1082 MiB | 1110 MiB | 1115 MiB | 1143 MiB | 1054 MiB | | --no-daemon | 476 MiB | 516 MiB | 559 MiB | 807 MiB | 921 MiB | 970 MiB | 1067 MiB | 1072 MiB | 1080 MiB | 1084 MiB | 1112 MiB | 1144 MiB | 0 MiB | _Max of 100 ms samples in each 5 s bin, labelled by the end of the bin; 0 s is the moment gradlew exits, +5 s is the 5 s after. One run each, PlantUML jar task, JDK 21, Gradle 9.3.1._ On a CI runner, that leftover daemon sits in the same memory as every later step of the job: the Docker build, the integration tests, the upload. At 8 GB the median time was 123.5 s with the daemon and 87.5 s without (112.7 s and 99.7 s at 2 GB), The ranges overlap and CPU steal was high, so we see no repeatable speed-up from the daemon for one Gradle command per job. The daemon is built to pay off across several Gradle commands in the same job, which we did not test. If a job runs one Gradle command, use --no-daemon (or org.gradle.daemon=false). If it runs several, consider keeping the daemon (we did not test that case), and run ./gradlew --stop before the steps that need the memory. Trap 5: a limit on the docker CLI does not limit the build docker build is a client. In our setup the work ran in a BuildKit container (Buildx's docker-container driver), which has its own cgroup. Other drivers and CI executors lay out cgroups differently, but the CLI is only a client in all of them. We wrapped the docker buildx build command in the same kind of scope as everything else, and at every limit that scope peaked at 39 to 42 MiB: the CLI, streaming logs. To measure the build, we put the limit on the BuildKit container itself: docker buildx create --name p1d8gr1 --driver docker-container \ --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \ --driver-opt memory=8g --driver-opt memory-swap=8g --driver-opt cpuset-cpus=0-3 \ --buildkitd-config buildkitd.toml --bootstrap and read memory.peak from /sys/fs/cgroup/system.slice/docker-<container id>.scope. That container peaked at 2,072 MiB, of which only 355 MiB was anonymous: the Go compiler inside the build. Most of the rest was page cache from the image layers and the build's output, plus 136 MiB of kernel memory. This matters for runner sizing in two ways. Your CI's memory graph may not show the build at all, depending on whether it measures the job's process tree or the whole machine. And if a self-hosted runner limits only the job's process tree, a Docker build in that job can run outside the limit, so a "2 GB job" can push the host far past 2 GB. Check which cgroup your executor puts the Docker daemon in. Two method bugs that cost us a run The first 2 GB Vite run left no result at all. The kernel killed the Vite process, and then systemd stopped the whole scope, including our wrapper script, before it could write anything. The journal said so: p1-vite-2G-r1.scope: A process of this unit has been killed by the OOM killer. p1-vite-2G-r1.scope: Failed with result 'oom-kill'. systemd 255 sets DefaultOOMPolicy=stop, and scopes follow it. We set DefaultOOMPolicy=continue before the next 2 GB run (the published p1.sh also passes -p OOMPolicy=continue), and checked with a 64 MiB scope that the wrapper now survives a kill and records oom_kill 1. Runs 2 and 3 at 2 GB have full records. Then our rerun of run 1 failed too, for a different reason. systemd keeps a failed transient unit loaded, so starting a new scope with the same name was refused: Failed to start transient scope unit: Unit p1-vite-2G-r1.scope was already loaded or has a fragment file. By the time we saw that, the Docker cache tests for the next post were running on the same droplet, so we did not try again. Run 1 still counts as a kill: its kernel log line reads oom_memcg=/bench.slice/p1-vite-2G-r1.scope,...,task=MainThread,pid=36041, and our sampler, which runs outside the scope, saw the cgroup reach 2,048 MiB, with up to 1,985 MiB anonymous, before its last sample at 70.5 seconds. The published redo mode now runs systemctl reset-failed first. If you measure builds this way, set OOMPolicy=continue and give each run a unique unit name, or the runs that you care about most, the ones that fail, will be the ones with no data. A sizing table From these runs, for a 4 vCPU runner. The "smallest limit that passed" column is measured. The last column is our rule of thumb, not a measurement: a real VM also holds the OS, the runner agent and any service containers, so we pick the smallest size that leaves at least 1 GB above the anonymous peak. | Build like ours | Anonymous peak | Smallest limit that passed every run | Runner we would pick | | --- | --- | --- | --- | | Bundler that also type-checks (Excalidraw vite build) | 3,248 MiB | 4 GB | 8 GB (moving the type check to its own job may make 4 GB enough; we did not test that) | | Large type check on its own (tsc, 1,508 files) | 1,139 MiB at 8 GB, 970 MiB at 2 GB | 2 GB | 4 GB | | Gradle with a 1 GB daemon heap (PlantUML jar) | 1,143 MiB | 2 GB | 4 GB, with --no-daemon | | Go build of a mid-size CLI (gh) | 424 MiB | 2 GB | 2 GB | | Docker build of a Go service | 355 MiB inside BuildKit | 2 GB | 2 GB, plus what the Docker daemon and the other steps need | Hosted CI services list their machines by vCPU and memory, so you can match a build to a size. Bitrise, for example, lists every Linux machine type on its pricing page with its vCPU count and memory, such as 2 vCPU with 8 GB, and also sells a remote build cache for Gradle, Bazel and Xcode (checked on its public pages on 2 November 2026). Whatever you use, measure anonymous memory first, then pick the size. What we could not conclude - Three limits on one machine are not three machines. A real 2 GB runner has less page cache to spare, its own kernel, and a runner agent in the same memory. Leave a margin above the anonymous peak; our sizing table uses 1 GB. - CPU count changes memory. Every run had 4 vCPUs. The Go compiler, Rollup's workers and Gradle's workers scale with CPUs, so these numbers are not valid for 2 or 16 vCPUs. - Wall times are noisy. The droplet reported 23.5% CPU steal time across the 33 runs that our CPU counter log covered, and between 3.7% and 35.5% in single runs. The Go build took between 121.3 and 199.6 seconds at limits it never came close to. We do not read anything into a time difference between limits unless it repeats in every round, as the tsc gap did. - Two or three runs per cell is enough to see a failure repeat and a peak hold steady, and that is all. The anonymous peaks varied by up to 21% for the Go build (353 to 441 MiB at 8 GB) and up to 15% for Gradle, and 6% or less for Vite, tsc and Docker between runs of the same cell. - We did not split shared builds. We cannot say how much of the Vite build's memory is Rollup and how much is the checker, or how much of the Docker build's page cache is layer extraction. - Rust ran once. One ripgrep build at 8 GB, so no repeats and no 2 GB or 4 GB data. Run it yourself Create a 4 vCPU, 8 GB Ubuntu 24.04 machine, copy the files below to /root/bench/, and run: bash /root/bench/setup.sh bash /root/bench/p1.sh warmup # probes the runtimes, one unmeasured run per build bash /root/bench/p1.sh matrix # 3 rounds x 3 limits x each build (we stopped it after round 2) bash /root/bench/p1.sh round3 # what we ran as round 3: 8 GB and 2 GB only bash /root/bench/p1.sh redo # reruns any cell that left no result python3 summarize_p1.py /root/res/p1 Each run writes its own directory with result.env, memory.events, samples.json and the build log, so you can read any single run on its own. #setup.sh: toolchains, pinned sources, dependencies, registries #!/bin/bash # Droplet setup: toolchains, pinned sources, dependency prefetch, local registries. set -euxo pipefail export DEBIAN_FRONTEND=noninteractive mkdir -p /root/src /root/res cloud-init status --wait || true # first boot runs apt; wait for its lock fetch_sha() { git init -q "$1"; git -C "$1" remote add origin "$2"; git -C "$1" fetch -q --depth 1 origin "$3"; git -C "$1" checkout -q FETCH_HEAD; } fetch_sha /root/src/excalidraw https://github.com/excalidraw/excalidraw 8bcdca4526867433f99fdfcb55a063668db0c9dc fetch_sha /root/src/plantuml https://github.com/plantuml/plantuml db06d70e18e71839dd5e330ac4086f3845371e3b fetch_sha /root/src/cli https://github.com/cli/cli 9490d204a2ba95434153dedce99898ce9f281426 git clone -q --depth 1 --branch 15.2.0 https://github.com/BurntSushi/ripgrep /root/src/ripgrep fetch_sha /root/src/msd https://github.com/GoogleCloudPlatform/microservices-demo c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712 curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz curl -fsSL https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.xz | tar -C /usr/local --strip-components=1 -xJ curl -fsSL https://sh.rustup.rs | sh -s -- -y -q --profile minimal --default-toolchain stable apt-get update -q apt-get install -y -q openjdk-21-jdk-headless build-essential git jq zstd python3 ca-certificates curl curl -fsSL https://get.docker.com | sh export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local npm install -g yarn@1.22.22 docker pull -q moby/buildkit:v0.34.1 docker pull -q registry:2.8.3 docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3 docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3 docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3 # dependencies are downloaded once, outside the measured runs (cd /root/src/excalidraw && yarn install --frozen-lockfile --network-timeout 600000) (cd /root/src/cli && go mod download) (cd /root/src/ripgrep && cargo fetch --locked) # systemd stops a scope after an OOM kill by default; keep the wrapper alive to record the result mkdir -p /etc/systemd/system.conf.d printf '[Manager]\nDefaultOOMPolicy=continue\n' > /etc/systemd/system.conf.d/oom.conf systemctl daemon-reexec #wrap.sh: runs inside the scope and records the result #!/bin/bash # Runs inside the systemd scope. usage: wrap.sh OUTDIR TIMEOUT_S cmd... out=$1; tmo=$2; shift 2 cg=/sys/fs/cgroup$(sed -n 's/^0:://p' /proc/self/cgroup) echo "$cg" > "$out/cgroup" cat "$cg/memory.max" > "$out/memory.max" t0=$(date +%s.%N) timeout -k 15 "$tmo" "$@" > "$out/build.log" 2>&1 rc=$? t1=$(date +%s.%N) { echo "rc=$rc" echo "t0=$t0" echo "t1=$t1" echo "peak=$(cat "$cg/memory.peak")" echo "current_after=$(cat "$cg/memory.current")" awk '$1=="anon"||$1=="file"{print $1"_after="$2}' "$cg/memory.stat" } > "$out/result.env" cp "$cg/memory.events" "$out/memory.events" for p in $(cat "$cg/cgroup.procs"); do ["$p" = "$$"] && continue ps -o pid=,rss=,args= -p "$p" | cut -c1-400 done > "$out/leftover.txt" 2>/dev/null if [-n "${LEFTOVER_WAIT:-}"]; then sleep "$LEFTOVER_WAIT" echo "current_later=$(cat "$cg/memory.current")" >> "$out/result.env" awk '$1=="anon"{print "anon_later="$2}' "$cg/memory.stat" >> "$out/result.env" for p in $(cat "$cg/cgroup.procs"); do ["$p" = "$$"] && continue ps -o pid=,rss=,args= -p "$p" | cut -c1-400 done > "$out/leftover_later.txt" 2>/dev/null fi touch "$out/done" exit $rc #sample.py: polls memory.stat every 100 ms from outside the scope #!/usr/bin/env python3 """Poll one cgroup v2 directory every 100 ms until a done file appears. usage: sample.py CGROUP_DIR OUT_JSON DONE_FILE Records the maximum of anon, file, kernel and memory.current, plus a series. """ import json, os, sys, time cg, out, done = sys.argv[1:4] start = time.time() while not os.path.isdir(cg): if time.time() - start > 60 or os.path.exists(done): json.dump({"error": "cgroup never appeared"}, open(out, "w")) sys.exit(0) time.sleep(0.02) keys = ("anon", "file", "kernel", "shmem") mx = {k: 0 for k in keys} mx["current"] = 0 series = [] t0 = time.time() n = 0 while not os.path.exists(done): try: with open(os.path.join(cg, "memory.stat")) as f: st = {} for line in f: k, v = line.split() if k in keys: st[k] = int(v) with open(os.path.join(cg, "memory.current")) as f: cur = int(f.read()) except OSError: break n += 1 for k in keys: mx[k] = max(mx[k], st.get(k, 0)) mx["current"] = max(mx["current"], cur) series.append([round(time.time() - t0, 2), st.get("anon", 0), st.get("file", 0), cur]) time.sleep(0.1) json.dump({"max": mx, "samples": n, "interval_s": 0.1, "series": series}, open(out, "w")) #p1-cmds.sh: the builds, and what is deleted before each run #!/bin/bash # Build definitions for post 1. prep_* runs outside the cgroup, cmd_* inside it. export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local EX=/root/src/excalidraw PU=/root/src/plantuml GH=/root/src/cli RG=/root/src/ripgrep FRONTEND=/root/src/msd/src/frontend stop_gradle() { pkill -f GradleDaemon || true; pkill -f GradleWrapperMain || true; pkill -f KotlinCompileDaemon || true; sleep 1; } prep_vite() { rm -rf $EX/excalidraw-app/build $EX/node_modules/.vite $EX/excalidraw-app/node_modules/.vite; } cmd_vite() { cd $EX && yarn build:app:docker; } prep_tsc() { find $EX -maxdepth 2 -name '*.tsbuildinfo' -delete; } cmd_tsc() { cd $EX && node_modules/.bin/tsc -p tsconfig.json --extendedDiagnostics; } prep_gradle_daemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; } cmd_gradle_daemon() { cd $PU && ./gradlew jar --console=plain --no-build-cache; } prep_gradle_nodaemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; } cmd_gradle_nodaemon() { cd $PU && ./gradlew --no-daemon jar --console=plain --no-build-cache; } prep_go() { rm -rf /root/gocache /root/out; mkdir -p /root/out; } cmd_go() { cd $GH && GOCACHE=/root/gocache go build -o /root/out/gh ./cmd/gh; } prep_rust() { rm -rf $RG/target; } cmd_rust() { cd $RG && cargo build --release --locked; } prep_docker() { docker buildx ls --format '{{.Name}}' 2>/dev/null | grep '^p1d' | xargs -r -n1 docker buildx rm >/dev/null 2>&1 || true; } if ["${BASH_SOURCE[0]}" = "$0" ]; then "$@"; fi #p1.sh: probe, warm-up, matrix, round 3, redo, and the docker build cell #!/bin/bash # Post 1 matrix. usage: p1.sh warmup | p1.sh matrix set -u B=/root/bench R=/root/res/p1 mkdir -p $R source $B/p1-cmds.sh LIMITS=(8G 4G 2G) BUILDS=(vite tsc gradle_daemon gradle_nodaemon go rust docker) # rust ran once (warmup) only: the matrix did not fit the droplet time budget MATRIX_BUILDS=(vite tsc gradle_daemon gradle_nodaemon go docker) CPUS=0-3 log() { echo "$(date -u +%FT%TZ) $*" | tee -a $R/progress.log; } scope_run() { # unit limit out tmo cmd... local unit=$1 lim=$2 out=$3 tmo=$4; shift 4 python3 $B/sample.py /sys/fs/cgroup/bench.slice/$unit.scope $out/samples.json $out/done & local sp=$! systemd-run --scope --quiet --slice=bench.slice --unit=$unit \ -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS -p OOMPolicy=continue \ $B/wrap.sh $out $tmo "$@" wait $sp systemctl stop $unit.scope 2>/dev/null || true } docker_cell() { # lim out tmo tag local lim=$1 out=$2 tmo=$3 tag=$4 local name=$(echo p1d$tag | tr A-Z a-z) m=$(echo $1 | tr G g) docker buildx create --name $name --driver docker-container \ --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \ --driver-opt memory=$m --driver-opt memory-swap=$m --driver-opt cpuset-cpus=$CPUS \ --buildkitd-config $B/buildkitd.toml --bootstrap > $out/create.log 2>&1 local cid; cid=$(docker inspect -f '{{.Id}}' buildx_buildkit_${name}0) local cg=/sys/fs/cgroup/system.slice/docker-$cid.scope cat $cg/memory.max > $out/container_memory.max mkdir -p $out/cli python3 $B/sample.py $cg $out/container_samples.json $out/cli/done & local sp=$! scope_run p1-docker-$tag $lim $out/cli $tmo \ docker buildx build --builder $name --progress=plain --provenance=false --output type=cacheonly $FRONTEND wait $sp { echo "container_peak=$(cat $cg/memory.peak)"; cat $out/cli/result.env; } > $out/result.env cp $cg/memory.events $out/container_memory.events cp $out/cli/build.log $out/build.log docker buildx rm $name > /dev/null 2>&1 } run_cell() { # build limit run tmo local b=$1 lim=$2 r=$3 tmo=$4 local tag=$b-$lim-$r out=$R/$b/$lim/$r rm -rf $out; mkdir -p $out prep_$b sync; echo 3 > /proc/sys/vm/drop_caches; sleep 2 if [$b = docker]; then docker_cell $lim $out $tmo ${lim}$r else case $b in gradle_*) export LEFTOVER_WAIT=5 ;; esac scope_run p1-$tag $lim $out $tmo bash $B/p1-cmds.sh cmd_$b unset LEFTOVER_WAIT fi [$b = gradle_daemon] || [$b = gradle_nodaemon] && stop_gradle log "$tag $(tr '\n' ' ' < $out/result.env)" } probe() { # record what the runtimes see inside each limit for lim in "${LIMITS[@]}"; do systemd-run --scope --quiet --slice=bench.slice --unit=probe-$lim -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS \ bash -c 'echo "nproc=$(nproc)"; echo "memory.max=$(cat /sys/fs/cgroup$(sed -n "s/^0:://p" /proc/self/cgroup)/memory.max)"; echo "node_heap_size_limit=$(node -e "console.log(require(\"v8\").getHeapStatistics().heap_size_limit)")"; java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage|ActiveProcessorCount) "' > $R/probe-$lim.txt 2>&1 done { echo "nproc=$(nproc)"; free -b | head -2; echo "node_heap_size_limit=$(node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)')"; java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage) "; } > $R/probe-host.txt 2>&1 } case ${1:-} in warmup) probe for b in "${BUILDS[@]}"; do run_cell $b 8G r0 900; done log WARMUP-DONE ;; matrix) declare -A TMO for b in "${MATRIX_BUILDS[@]}"; do . <(grep -E '^t[01]=' $R/$b/8G/r0/result.env) w=$(python3 -c "print(int(float('$t1')-float('$t0')))") t=$((3 * w + 60)); [$t -gt 600] && t=600; [$t -lt 120] && t=120 TMO[$b]=$t done log "timeouts: $(for b in "${MATRIX_BUILDS[@]}"; do printf '%s=%s ' $b ${TMO[$b]}; done)" for r in r1 r2 r3; do for b in "${MATRIX_BUILDS[@]}"; do for lim in "${LIMITS[@]}"; do run_cell $b $lim $r ${TMO[$b]}; done done done log MATRIX-DONE ;; round3) # third run at 8G and 2G only: the full round did not fit the droplet time budget declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351) log "round 3 at 8G and 2G only" for b in "${MATRIX_BUILDS[@]}"; do for lim in 8G 2G; do run_cell $b $lim r3 ${TMO[$b]}; done done log MATRIX-DONE ;; redo) # rerun cells that left no result (systemd stopped the scope after an OOM kill) declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351) for b in "${MATRIX_BUILDS[@]}"; do for lim in "${LIMITS[@]}"; do for r in r1 r2 r3; do d=$R/$b/$lim/$r if [-d $d] && ! grep -q '^rc=' $d/result.env 2>/dev/null; then rm -rf $d.lost; mv $d $d.lost; log "redo $b $lim $r" systemctl reset-failed p1-$b-$lim-$r.scope 2>/dev/null # added after our run: a failed scope blocks its name run_cell $b $lim $r ${TMO[$b]} fi done; done; done log REDO-DONE ;; *) echo "usage: $0 warmup|matrix|round3|redo"; exit 2 ;; esac #buildkitd.toml: send base image pulls to the local mirror [registry."docker.io"] mirrors = ["127.0.0.1:5001"] [registry."gcr.io"] mirrors = ["127.0.0.1:5002"] [registry."127.0.0.1:5000"] http = true insecure = true [registry."127.0.0.1:5001"] http = true insecure = true [registry."127.0.0.1:5002"] http = true insecure = true #summarize_p1.py: medians per build and limit #!/usr/bin/env python3 """Summarize post 1 runs: medians per build and limit over the measured runs (r1..r3).""" import json, os, re, statistics, sys root = sys.argv[1] BUILDS = ["vite", "tsc", "gradle_daemon", "gradle_nodaemon", "go", "rust", "docker"] LIMITS = ["8G", "4G", "2G"] MiB = 2 ** 20 def env(path): d = {} if os.path.exists(path): for line in open(path): if "=" in line: k, v = line.strip().split("=", 1) d[k] = v return d def events(path): d = {} if os.path.exists(path): for line in open(path): k, v = line.split() d[k] = int(v) return d def run_info(d, build): r = env(os.path.join(d, "result.env")) if not r or "rc" not in r: return None log = open(os.path.join(d, "build.log"), errors="replace").read() if os.path.exists(os.path.join(d, "build.log")) else "" if build == "docker": if not r.get("container_peak"): return None # warm-up run 0 recorded no container numbers (builder name case bug) ev = events(os.path.join(d, "container_memory.events")) smp = json.load(open(os.path.join(d, "container_samples.json"))) peak = int(r["container_peak"]) cli_peak = int(r["peak"]) else: ev = events(os.path.join(d, "memory.events")) smp = json.load(open(os.path.join(d, "samples.json"))) peak = int(r["peak"]) cli_peak = None rc = int(r["rc"]) wall = float(r["t1"]) - float(r["t0"]) if rc == 0: outcome = "ok" elif ev.get("oom_kill", 0) > 0: outcome = "oom_kill" elif "heap out of memory" in log or "Reached heap limit" in log: outcome = "node_heap" elif "OutOfMemoryError" in log: outcome = "java_heap" elif rc in (124, 137): outcome = "timeout" else: outcome = f"fail_rc{rc}" mx = smp.get("max", {}) return { "rc": rc, "outcome": outcome, "wall_s": round(wall, 1), "peak_mib": round(peak / MiB), "anon_max_mib": round(mx.get("anon", 0) / MiB), "file_max_mib": round(mx.get("file", 0) / MiB), "kernel_max_mib": round(mx.get("kernel", 0) / MiB), "samples": smp.get("samples"), "oom_kill": ev.get("oom_kill", 0), "max_events": ev.get("max", 0), "high_events": ev.get("high", 0), "after_mib": round(int(r.get("current_after", 0)) / MiB), "anon_after_mib": round(int(r.get("anon_after", 0)) / MiB), "cli_peak_mib": round(cli_peak / MiB) if cli_peak is not None else None, "later_mib": round(int(r["current_later"]) / MiB) if "current_later" in r else None, "anon_later_mib": round(int(r["anon_later"]) / MiB) if "anon_later" in r else None, "leftover_later": open(os.path.join(d, "leftover_later.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover_later.txt")) else None, "leftover": open(os.path.join(d, "leftover.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover.txt")) else "", } def med(xs): xs = [x for x in xs if x is not None] return round(statistics.median(xs), 1) if xs else None out = {} for b in BUILDS: for lim in LIMITS: runs = {} for rn in ("r0", "r1", "r2", "r3"): d = os.path.join(root, b, lim, rn) if os.path.isdir(d): info = run_info(d, b) if info: runs[rn] = info meas = [runs[k] for k in ("r1", "r2", "r3") if k in runs] ok = [x for x in meas if x["outcome"] == "ok"] out[f"{b}/{lim}"] = { "runs": runs, "n": len(meas), "n_ok": len(ok), "outcomes": [x["outcome"] for x in meas], "wall_med_ok": med([x["wall_s"] for x in ok]), "wall_med_all": med([x["wall_s"] for x in meas]), "peak_med": med([x["peak_mib"] for x in meas]), "anon_med": med([x["anon_max_mib"] for x in meas]), "file_med": med([x["file_max_mib"] for x in meas]), "peak_range": [min(x["peak_mib"] for x in meas), max(x["peak_mib"] for x in meas)] if meas else None, "anon_range": [min(x["anon_max_mib"] for x in meas), max(x["anon_max_mib"] for x in meas)] if meas else None, "wall_range": [min(x["wall_s"] for x in meas), max(x["wall_s"] for x in meas)] if meas else None, "after_med": med([x["after_mib"] for x in meas]), "anon_after_med": med([x["anon_after_mib"] for x in meas]), "cli_peak_med": med([x["cli_peak_mib"] for x in meas]), "later_med": med([x["later_mib"] for x in meas]), "anon_later_med": med([x["anon_later_mib"] for x in meas]), "max_events_med": med([x["max_events"] for x in meas]), } json.dump(out, open(os.path.join(root, "summary.json"), "w"), indent=1) print(f"{'cell':26} {'n':>2} {'ok':>2} {'wall_ok':>8} {'peak':>6} {'anon':>6} {'file':>6} {'after':>6} outcomes / ranges") for k, v in out.items(): print(f"{k:26} {v['n']:>2} {v['n_ok']:>2} {str(v['wall_med_ok']):>8} {str(v['peak_med']):>6} {str(v['anon_med']):>6} {str(v['file_med']):>6} {str(v['after_med']):>6} {v['outcomes']} wall{v['wall_range']} anon{v['anon_range']} peak{v['peak_range']} maxev={v['max_events_med']} cli={v['cli_peak_med']} later={v['later_med']}/{v['anon_later_med']}") Summary Most CI builds need less memory than the cgroup counter suggests, because memory.peak includes page cache that the kernel gives back under pressure. Measure anonymous memory, run the build inside the limit you want to buy, and check memory.events for max and oom_kill. Then look for the traps: a bundler that runs a type checker, a runtime that sizes its heap from the limit, a Gradle daemon that stays after the build, and a Docker build that runs outside your cgroup. In our runs, fixing any of those was worth more than doubling the runner's memory. _Originally published at devops-daily.com._

🖥️ GoogleCloudPlatform/microservices-demo Sample cloud-first application with 10 #microservices showcasing #Kubernetes, #Istio, and #gRPC. GoogleCloudPlatform/microservices-demo #GoogleCloud #GKE

Generative AI on Google Cloud: notebooks, code samples, sample apps, and other resources (18k stars) What it does All text below is quoted directly from the project README. >Sample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform Generative AI on Google Cloud: - gemini/: Discover Gemini through starter notebooks, use cases, function calling, sample apps, and more. - search/: Use this folder if you're interested in using Agent Search, a Google-managed solution to help you rapidly build search engines for websites and across enterprise data. - rag-grounding/: Use this folder for information on Retrieval Augmented Generation (RAG) and Grounding. - vision/: Use this folder if you're interested in building your own solutions from scratch using features from Imagen and Veo. - audio/: Use this folder if you're interested in building your own solutions from scratch using features from Chirp, a version of Google's Universal Speech Model (USM). - setup-env/: Instructions on how to set up Google Cloud, the Gen AI Python SDK, and notebook environments on Google Colab and Workbench. >The latest Gemini models are available! Try out Gemini 4 Argon. Repository GoogleCloudPlatform/generative-ai Repository stats |Metric|Value| |:-|:-| |Stars|17,827| |Forks|4.5k| |Last release|None| |Languages|Jupyter Notebook 84.3%, Python 8.1%, HTML 2.3%| |Contributors|322| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|None| |AI co-authored commits|None| Main contributor |Field|Value| |:-|:-| |Name|Holt Skinner| |GitHub|@holtskinner| |Commits to this repo|910 (top contributor)| |Bio|Developer Advocate, Google Cloud AI| |Company / Location|Austin, TX| |Followers|1.2k| ────────────────────────────── 📌 If you want to browse more repos like this, we've organized them by category and rankings here: bestgithubrepositories.com 🗂️ We're also archiving repos where possible, so useful projects don't disappear when they're removed from GitHub.

@_mctrinh Have you tried OKF, was hoping this catches up as the standard. github.com/GoogleCloudPlatform/knowledge-catalog/…

Google's Open Knowledge Format (OKF), an implementation that addresses real life problems Google published the Open Knowledge Format this summer. It is a way to write knowledge down as a folder of Markdown files with a YAML header, so that people and AI models read the same files. Each file is a concept (a rule, a procedure, a metric, a role) and the header says where it comes from, who checked it, when it expires and whether it is still current. No server, no database. If you can cat a file, you can read it. Great concept, but the spec stops at the format. It does not say how knowledge gets into a bundle, how to know a bundle is right, what to do when two documents disagree, or how a correction found by a user gets back in. In a company, that is the hard part. This project tries to answer real organizations problems using OKF with skills for coding agents (Claude Code and Pi): - okf-produce reads the documents, agrees with their owner on what the bundle is for, writes it with a source behind each claim, and keeps it current. - okf-validate checks a bundle against the spec. Plain Python, runs in CI too. - okf-answer answers questions from one or several bundles, with a reference after each claim and a warning when a fact is unchecked, expired or not covered. The same documents can give different bundles for different readers (angle), the skill will ask: an employee, an auditor and a documentation officer do not want the same thing from the same quality manual. The steps that need no judgment are code, not model: hashing, indexes, trust tiers, dates. The model never marks its own work as verified, a person does. Readers that did not write the bundle check it. When a user finds a mistake, the consumer writes a report and the producer picks it up at the next update. I tested it in a real organization, with several bundles. Token usage and accuracy were satisfactory, good enough that the org wants to keep going. The repo has three sample bundles made from the same 240 Wikipedia pages on Greek mythology, each from a different angle: the family tree, the stories by theme, and the English words that come from the myths. The family tree bundle includes a calculation (how is Perseus related to Heracles?) with a small checker that proves the approved script ran. Repo: wmahfoudh/okf Spec: GoogleCloudPlatform/open-knowledge-format Are you using OKF, where do you think it breaks, and how you handle trust and sources when you let a model answer from company documents.

Build a smart search bot for your phone I was on my phone the other day and wanted to find everything and anything related to a hamster I used to know named Stinky. His memories remain in various photos, videos, journal entries and songs, but Stinky is no longer with us (he died a few years ago of old age). My phone's built-in search wasn't much help – it just looked for titles of files – so I decided to see what I could build myself without coding. I conducted some research First, I did a quick Google search to see what on-device models were available on Hugging Face: > I’m looking for a model on Hugging Face that I can put on my phone that lets me context search for video, audio, text, and photos. And I received three strong leads: - Google Embeddinggemma 2: 740M parameters, 1.5 GB storage, 2GB RAM - Meta ImageBind: ~1.2B parameters, 2.5GB storage, 4GB RAM - NVIDIA Omni-Embed-Nemotron: ~3B–4.7B parameters, 9.5 GB storage, 8GB RAM I chose to work with Embeddinggemma 2 since its size was small enough to run on a smartphone. Finally, I thought about how I was going to get the app on my phone. I chose to use Flutter, a multi-platform mobile framework that is easy and fun to use. With my toolbox complete, I set out to build the personal search bot of my dreams. I used these steps to build the app On my laptop, I loaded up my agent command center, Antigravity, and passed it a prompt to create the app: > Create an app that can search my phone for a theme of my choice. The app should be able to search photos, videos, documents, and voice recordings on my phone and return a list of them that match my theme. I want to use Embeddinggemma 2 as my model and Flutter as my framework. Call the app Lark. Design it to be cheerful and fun. 45 minutes later, the first version of an app called Lark was complete, no coding required. !Image description When I launched the app though, it didn't quite work. The AI agent had mocked the search experience and hadn't actually connected my app to a model. After some back and forth over a few hours, the model was properly connected and the mocked data removed. The resulting app was quite nice! When I was finished, I wondered how easy it might be to connect this to Model Garden so I could free up 1.5GB on my device and use my app on all of my devices. I prompted Antigravity to do this and in 30 minutes I had a working gig! 3 things I learned building the app Building a fully functional semantic search app without writing code was wild, but it definitely wasn't magic. If you're looking to build your own AI tools, here are my biggest takeaways: 1. Don't fall for the mock data trap. Antigravity built a beautiful UI in just 45 minutes, but it was just a facade. AI had mocked the search experience instead of connecting the model. I had to explicitly request that it not use mock data to get what I needed. 2. Pivoting from an on device model to a cloud hosted model is surprisingly easy. You don't have to lock yourself in one architecture. Switching stacks is much easier these days. 3. Simple math is your friend on mobile. EmbeddingGemma 2 uses clever math optimizations that eliminate the need for heavy native dependencies that would have drained my phone's battery. Barriers are vanishing Projects like Lark prove that the barrier to entry for highly personalized software is vanishing. You no longer need an engineering team to solve a hyper-specific personal problem, you just need a bit of patience and the right prompts. P.S. Want to see what I built? Download the source code for the Lark app here. Take what you want, modify it into what you need, and if you build something cool, let me know! ~ Stinky (2021-2023) ~ Gone but not forgotten ~ !Image description

PR merged, officially a GCP open source contributor! 🚀 #opensource cxas-scrapi#486

Potential Bug: GKE anetd MTU shrinking leads to network failure (Dataplane V2) Cluster Info: - Cluster note: GKE Dataplane V2 with WireGuard inter-node encryption (UDP 51871) - Versions: Control plane and all live pools 1.35.8-gke.1796000, channel REGULAR (changed to STABLE during this incident) - Anetd Image: cni-writer v0.1.40-gke.6, cilium v1.18.7-gke1.35-gke.39 I have also posted on discuss.google.dev but wanted to get more insights from everyone here that I've been following. Please let me know if this is not the right place. Summary: My team just recovered from a downtime started from a GKE control plane upgrade last night. Our observation so far points to a suspected bug somewhere in anetd cni-writer. I'm hoping for anyone who may have experienced something similar and can give pointers to the right direction. - GKE's anetd cni-writer v0.1.40-gke.6 lowers a node's MTU by 80 bytes on every anetd restart. - Pods created earlier keep the larger MTU, and their full-size packets vanish. - We recover by recreating pods that predate anetd restart, then to be on the safer side, we replace every node to reset the MTU to baseline. - Current measures my team is taking: monitor anetd pods for any restart/delete. Every node MTU is fresh at 1460, so one anetd restart drops that node MTU to 1380 while its running pods keep 1460. We observe that mismatch to be the outage. - Logs from our apps:App -> RabbitMQ: Broker closed AMQP connections for missed heartbeats; unhandled ECONNRESET exits the processApp -> Valkey: Valkey replies dropped; ioredis waited for the kernel retry limit, then failed every queued command So, the suspected bug, step-by-step: 1. A fresh node starts at MTU 1460: NIC, CNI configuration, host-side pod interfaces, and pod eth0. WireGuard cilium_wg0 is 1380, leaving 80 bytes for encryption. 2. Every anetd pod start reruns its cni-writer init container. Control-plane upgrades roll anetd on every node, including pools with node auto-upgrade off. 3. cni-writer reads cilium_wg0 (1380) as the base MTU and writes it as the CNI pod MTU. 4. Host-side pod interfaces shrink to 1380. Running pods keep eth0 at 1460 and send packets the host side drops, with no ICMP back. Large replies stall; clients time out. 5. Cilium subtracts 80 from the smaller base: WireGuard becomes 1300. The next restart copies 1300, and so on. Logs from cni-writer anetd pods: # anetd-9j66s on 3fc1dd1c-6nrh, second anetd start on this node I1025 19:18:02.916461 cni_writer.go:378] Checking priority interface "cilium_wg0" for MTU. I1025 19:18:02.916725 cni_writer.go:383] Using MTU 1380 obtained from priority interface cilium_wg0 I1025 19:18:02.916738 cni_writer.go:518] Successfully determined and cached MTU: 1380 (from cilium_wg0) # anetd-cdmvs on 521c6876-gk4s I1025 19:04:35.214705 cni_writer.go:383] Using MTU 1220 obtained from priority interface cilium_wg0 # anetd-l8mf6 on 56b8fd8a-mdxw, control-plane upgrade roll I1025 17:23:25.846768 cni_writer.go:383] Using MTU 1300 obtained from priority interface cilium_wg0 --- Update, I also ran an LLM to check upstream repos for clues, here goes: - GoogleCloudPlatform/netd PR #199 (commit ac67942e, 2023-02-27) - Prefers cilium_wg0 over the default NIC when writing pod MTU. - Matches the policy in our cni-writer logs; legacy shell source, not proven to be our binary. - GoogleCloudPlatform/netd PR #202 (commit c50a56a0, 2023-02-28) - Refactors MTU selection into a loop, keeping WireGuard first. - The policy was kept, not reverted. - cilium/cilium commit e99bee54 (2019-05-27) - CNI configuration MTU overrides the agent's configured MTU. - Why --mtu=1460 alone does not hold the base. - cilium/cilium commit 6ae08fdc (2021-05-05) - WireGuard MTU is device MTU minus WireguardOverhead, 80 bytes. - Normal accounting becomes a loop when the base is copied from WireGuard. - GoogleCloudPlatform/netd commit 38ff1139 (2025-01-22) - Skips legacy CNI writing on unified-CNI nodes, handing it to anetd. - Marks the boundary; does not date the defect. - GoogleCloudPlatform/netd issue #33 (closed 2025-12-04) - Asks to move the shell CNI generator to Go; closed as 'CNI writer is there now'. - The Go writer exists; no public source or build mapping. cilium-agent startup: + mtu=1460 + exec /usr/bin/cilium-agent --k8s-kubeconfig-path=/var/lib/kubelet/kubeconfig --config-dir=/tmp/cilium/config-map --mtu=1460 {"level":"info","msg":"Overwriting MTU based on CNI configuration","module":"agent.datapath.mtu","mtu":1460} {"level":"info","msg":"Using configured MTU","module":"agent.datapath.mtu","mtu":1460} Reconstructed loop: | Startup | Writer's input | CNI base MTU | WireGuard MTU | | ---------------------- | ----------------------------------- | ------------ | ------------- | | Fresh node | Node NIC, no WireGuard interface yet | 1460 | 1380 | | First affected restart | Existing cilium_wg0 at 1380 | 1380 | 1300 | | Second affected restart | Existing cilium_wg0 at 1300 | 1300 | 1220 | - The writer reuses an encapsulation-adjusted MTU as the unadjusted base. Cilium accepts it and subtracts the overhead again. The observed 80-byte steps match. - We found no public cni_writer.go for v0.1.40-gke.6 and no build-to-commit mapping. The deployed Cilium revision b8f2f41d06 does not resolve publicly; upstream v1.18.7 is commit 2adae214. - The public netd MTU test disables the Cilium plugin and mocks NICs, so it never exercises the restart loop. - No public issue confirms this ratchet or a fixed GKE version.

Apache 2.0, open source: github.com/GoogleCloudPlatform/LifeSciences/blob/… #GoogleCloud #opensource #AgenticAI #MultiAgent #DrugDiscovery #TechBio #Pharma #Biotech #LifeSciences #AIforScience #Oncology #GenAI #AIAgents #DataSharing #BioPharma #MachineLearning #CompBio #HealthTech