CI/CD Optimizations, Part 1: PR Checks
---
title: CI/CD Optimizations, Part 1: PR Checks
cover_image: logicalbytes.dev
canonical_url: logicalbytes.dev
---
!CI/CD Optimizations, Part 1: PR Checks cover art
Statio's monorepo has a Go backend, a React web app, a desktop client, and a pile of GitHub Actions workflows that run on every pull request. I've seen a lot of talk on X recently about optimizing (usually a rust migration). So I decided that I might as well take a crack at what I have going on. Over the last week I spent a few evenings making those workflows faster. Go Build & Test went from 5.2 minutes to 1.8. Go Lint went from 6.0 to 3.0. The time from opening a PR to seeing all green went from about 5.3 to 2.5 which was slightly better than my 3 minute target which was based on looking away for a minute then every check had completed.
All numbers below are medians of successful jobs, pulled from the GitHub API across 493 workflow runs between August 5 and October 7, plus 125 PR runs from October 7 to 9 for the last section. Runners are the standard ubuntu-latest hosted ones.
The PR pipeline
Three workflows trigger on a pull request:
- CI (ci.yml): Go Build & Test, Go Lint, Frontend Check, OpenAPI codegen drift, a contract test against platform-core, deploy config validation, and a PR title check.
- Security Scan : Grype against the Go modules and against the Node lockfile.
- E2E MCP Pipeline : builds the gateway, stands up a test MCP server, and runs the pipeline end to end.
Everything runs in parallel. The important thing to remember is that the PR is done when the slowest job is done. Shaving time off any other job changes nothing you can feel.
Mid-September
In mid-September the slowest jobs were the Go ones. Go Lint ran 3.8 minutes, Go Build & Test 3.6, and Frontend Check about a minute. A typical PR finished in about 3.6 to 4 minutes, and Go Lint or Go Build was the last job to finish in 47 of 49 runs. This was good enough so I didn't worry about it. However the number began climbing as I added deeper checks.
Frontend tests landed in CI (#118, Sep 17). Before this, Frontend Check ran tsc and nothing else, so the vitest suite could go red and no check would notice. After it, the job also ran vitest and an eslint ratchet (--max-warnings 441, a number that's only allowed to go down). At the time the suite had 9 test files and the job went to about 2 minutes. That's a fine trade.
Lint lost its cache (#124, Oct 1). golangci-lint caches analysis results between runs. Ours flapped: nolintlint would report a //nolint directive as unused on one run and fine on the next, depending on what the cache held. A lint check that fails at random teaches everyone to hit re-run. I set skip-cache: true and Go Lint went from 3.8 to 5.0 minutes. I'd make that call again. A slow honest check beats a fast one nobody trusts.
The web app grew. This one wasn't a CI change at all. Between October 1 and October 6 we shipped onboarding, a help menu, container registry UI, new pricing pages, and tests for all of it. Test files went from 22 to 42. Frontend Check's daily median went 2.5, 2.8, 3.6, 3.8, 4.4, 5.0 minutes. A good consequence of ensuring we had testing coverage and ensured that nothing would break in the future, bad because now we were at over 5 minutes, not where I wanted to be.
By October 6 a PR took about 5.3 minutes and every job felt slow. So I started with what I knew best, Go, because Go was the part I was sure was broken.
Bug one: a cache key that can hit never updates
The Go jobs used actions/setup-go with its built-in cache. It keys the cache on a hash of go.sum. Which seemed right until I saw cache misses and looked into how actions/cache decides whether to save:
It only saves when the primary key missed.
So the first run after a go.sum change misses, builds, and saves. Every run after that hits the same key, restores that snapshot, and never saves again. The module cache stays fine because modules only change when go.sum does. The build cache, ~/.cache/go-build, goes stale. Every commit after the snapshot recompiles the packages that changed since then, and the list grows with every PR until someone bumps a dependency.
The fix (#166) turns off setup-go's cache and uses actions/cache directly, with github.run_id at the end of the key:
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: false
- name: Go cache
uses: actions/cache@v4
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}-${{ github.run_id }}
restore-keys: |
go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}-
go-${{ runner.os }}-
The run ID guarantees the primary key misses every time, so every run saves. restore-keys does prefix matching and returns the newest entry, so every run starts from the most recent cache. Go's build cache is content-addressed, so a stale entry is never used for a file that changed, it sits there until GitHub evicts it.
Proud of myself for the fix I merged it and the numbers got worse. Go Build went from 4.6 to 5.2 minutes and Go Lint from 5.0 to 6.0. That was expected because we had some other bugs that were preventing this fix from fully working.
Bug two: seven jobs, one key, and the fastest one wins
Every job in ci.yml that touches Go used that same cache step, so every job in a run computed the same key. GitHub's cache won't let two writers reserve one key. The first job to reach its post step saves. The rest log this and move on:
Failed to save: Unable to reserve cache with key go-Linux-<hash>-<run_id>,
another job may be creating this cache.
It's a warning, so the job stays green.
Which job gets to its post step first? The fastest one, which is also the job that compiles the least. In our workflow that's a job like OpenAPI codegen drift or deploy config validation, both done in under a minute. The cache that won was 8 KB. That became the newest entry, so restore-keys handed 8 KB to Go Build, Go Lint, and everything else on every run. They all logged "Cache restored," then compiled the whole tree from scratch, then lost the race to save.
The fix (#170) is one more variable in the key:
key: go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}-${{ github.run_id }}
restore-keys: |
go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}-
go-${{ runner.os }}-${{ github.job }}-
Each job now has its own cache, and that makes sense for more than the race. Build & Test compiles the whole module and its test binaries, Lint loads packages for analysis, and codegen builds one tool. Sharing one cache between them was never going to work and something I had missed previously. However, that fix that made things slower before would now come into play.
Results:
Go jobs, median minutes
| | Before #166 | After #166 | After #170 |
|---|---|---|---|
| Go Build & Test | 4.6m | 5.2m | 1.8m |
| Go Lint | 5.0m | 6.0m | 3.0m |
| OpenAPI Codegen | 0.8m | 1.0m | 0.5m |
Go Lint still runs with skip-cache: true, so its 3.0 minutes is a warm Go build cache plus a cold lint analysis. That's the expected trade-off from #124.
When to use a per-job, run-keyed cache: any job whose build output changes often, which is most Go projects. When not to: if a job's cache is only downloaded dependencies, a key on the lockfile hash is fine because the cache only needs to change when the lockfile does. The run ID also means one cache entry per run. GitHub caps a repo at 10 GB and evicts the least recently used entries, so keep an eye on size. I thought ours sat well under the cap. It didn't, so more lessons were to be learned in the near future. Also Github, maybe like show when we have caches full in some sort of billing page with more identification around them? Regardless we should have checked our cache sizes because we would have caught that 8kb for a Go monorepo is probably wrong.
Three smaller fixes (#173)
With the main cache working, I went looking for other jobs paying the same costs.
Lint only what changed on PRs. golangci-lint has --new-from-rev, which reports only issues on lines that differ from a given revision. On pull requests we now pass --new-from-rev=origin/main. Push to main still lints the whole tree, so anything the scoped run misses gets caught on merge.
- uses: actions/checkout@v7
with:
fetch-depth: 0 # --new-from-rev needs main's history
- uses: golangci/golangci-lint-action@v9
with:
skip-cache: true
args: >-
--timeout=10m --fix=false
${{ github.event_name == 'pull_request' && '--new-from-rev=origin/main' || '' }}
This flag still type-checks and analyzes every package, because most linters need the whole program. It only filters which issues get reported. The gain is smaller than "lint 5 files instead of 500" sounds. On the first run the golangci step went from about 130 seconds to 102.
When not to use it: if main isn't always clean. Scoped linting assumes the baseline passes. If main has existing failures, PRs will skip them and nobody will fix them.
Cache the Grype vulnerability database. Both Grype jobs downloaded the full vulnerability DB on every run. It's 462 MB and upstream publishes about once a day. Now it's cached with a date in the key:
- name: Date for Grype DB cache key
id: date
run: echo "day=$(date -u +%Y-%m-%d)" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v4
with:
path: ~/.cache/grype/db
key: grype-db-${{ runner.os }}-${{ steps.date.outputs.day }}
restore-keys: grype-db-${{ runner.os }}-
The first job each day downloads the DB and saves it. Every job after that restores it. Grype checks for a newer DB when it starts, so a day-old copy from restore-keys gets updated rather than scanned against as-is. For Grype a lockfile hash would be the wrong key here because the DB changes when new CVEs are published, not when our code does.
Give the E2E workflow the same Go cache fix. The E2E workflow lives in its own file and never got #166 or #170. It was still on setup-go's built-in cache with the stale-snapshot problem. It now has the job-scoped, run-keyed cache plus caches for Bun's install directory and the Grype DB, since the last stage of that pipeline shells out to Grype.
The first run of #173 was cold on purpose: new cache keys, so the second was the test because we had built our caches:
PR #173, cold run vs warm run
| | Before #173 | #173 cold | #173 warm |
|---|---|---|---|
| Grype Go | 2.0m | 2.3m | 1.2m |
| Grype Node | 2.3m | 2.9m | 1.8m |
| E2E pipeline | 4.8m | 5.8m | 3.1m |
| Go Lint | 3.0m | 3.1m | none |
_Go Lint did not rerun on the warm attempt._
The Grype DB cache is 462 MB, and restoring it is not free. It took 13 seconds in one job and 28 in another. That's still well ahead of downloading and unpacking the same file from upstream, but a cache is still a download, from a closer server. E2E restored a 324 MB Go cache and dropped by 1.7 minutes against its pre-#173 number. The lint job didn't rerun on the second attempt; its number from the first run is the one above, and since skip-cache stays on, there's no warm state for it to gain.
The PR didn't get faster
Here's the per-era picture across all the PR-triggered jobs. Every job runs in parallel, so the PR finishes when the longest bar does. Switch between the eras:
PR-triggered jobs, median minutes
| | Mid-Sept | Oct 1 to 6 | After #170 |
|---|---|---|---|---|
| Go Build & Test | 3.6m | 4.6m | 1.8m |
| Go Lint | 3.8m | 5.0m | 3.0m |
| Frontend Check | 1.0m | 3.7m | 5.1m |
| E2E pipeline | 3.6m | 4.6m | 4.8m |
| Grype (slower scan) | 2.0m | 2.2m | 2.3m |
Going by job and it's a big win. However if are focused on the finish line, and it's flat. In mid-September the slowest job was Go Lint at 3.8 minutes. Today it's Frontend Check at 5.1. The Go fixes took two jobs off the critical path, and the frontend job, which had doubled in the same week, was already standing behind them so we saved 12 seconds, not quite good enough but that means we only had one more job to optimize. Some of the jobs we could have left as is, they do not affect our PR times and checks at all but none of that means the Go work was wasted. CI minutes cost money and runner slots, and Go Build failing at 1.8 minutes instead of 5.2 gets you a red X three minutes sooner. Creating earlier signals means faster turn around time on a fix for it. That matters, because a failing test is the most common reason to look at CI at all. The goals was still "my PRs are faster", so despite all the optimizations and minutes saved, we again were at a measurable 12 seconds faster PR times.
A better prioritization would have been to pull the jobs from your last 20 PR runs, find which one finished last in each, and count. If one job is last 80% of the time, that's where we could have started and the GitHub API returns started_at and completed_at for every job, and a 20-line script against /actions/runs/{id}/jobs could have given us a better priority list. However, I think it was better to knock out one languages performance then switch onto the next one. If you are thinking of pure speed, knock down the slowest jobs first, if you are more like me and you want to start with where you are the most familiar, knocking things down is only beneficial in the long run.
The actual bottleneck
Frontend Check runs for 5.1 minutes and the vitest step is 3.9 minutes of it. Vitest's summary from the latest run:
Duration 233.89s (transform 1.94s, setup 2.31s, collect 72.96s, tests 125.55s, environment 21.60s, prepare 4.07s)
Collect is 73 seconds. That's vitest loading and transforming every test file and its imports before anything runs. Our components pull in Tamagui, and the first render in each file pays a one-time config compile. With 43 test files that cost is paid 43 times.
Some tests sit around 2.6 seconds each. EgressPolicyPanel, ServerSuspension, HostedServerSlots, LimitExceeded, and HelpMenu all cluster at that number. My first guess was real timers, since tests that all take the same odd length of time are usually waiting on a debounce or a waitFor running to its timeout. I was just wrong, sad I couldn't get a quick win, there were only two test files touch timers at all, and both already use vi.useFakeTimers(). So the 2.6 seconds was the first render in each file, and almost all of it was jsdom applying Tamagui's styles. More on that below.
We already tried the obvious config change. #171 switched vitest from its default forks pool to threads. On a laptop that cut the suite from 59.5 to 31.5 seconds because the V8 JIT stays warm across files. On CI the job didn't move: 5.1 minutes before, 5.1 after, demoralizing but hey just have to keep digging. A hosted runner on a private repo has 2 vCPUs and the laptop has a lot more, so there was almost no parallelism to gain. We also tried isolate: false and it broke 24 files on state leaking between them, the kind of leak a shared in-memory localStorage mock produces, so isolation stays. A lot of testing and not a lot of results, but again, see what sticks.
From there the plan was:
1. Make the test environment cheaper. Every file pays for a DOM, and the DOM is where the first-render time goes. Low risk if the suite stays green.
2. Shard the suite across jobs. vitest --shard=1/3 splits files across three parallel jobs. It's the one change that scales with suite size instead of fighting it. The cost is three runners instead of one, plus repeating the pnpm install and collect phase per shard.
3. Move lint and type-check out of the test job. They don't depend on each other, so running them in parallel takes them off the critical path entirely.
The target is a 3-minute PR, with 2 as the stretch.
Fixing the tests: happy-dom
Vitest gives every test file a fake browser, and the default is jsdom. jsdom aims to be a faithful browser, and that includes a real CSS engine. Tamagui injects a lot of styles on first render, and jsdom parses and applies all of them. happy-dom is a lighter DOM built for tests that does far less of that work.
The switch was one line in vitest.config.ts, plus a stub for navigator.clipboard in two test files, because happy-dom's clipboard behaves differently. I also told Vite to pre-bundle Tamagui (deps.optimizer.web) so each file doesn't transform the same package again, and aliased react-native to react-native-web so that pre-bundle resolves.
The clearest single number: LimitExceeded's first test took 2,078 ms on jsdom and 48 ms on happy-dom.
Whole suite, 521 tests, median of 3 runs on the same M3 Pro. Every run passed all 521:
vitest, 521 tests, seconds (M3 Pro)
| | jsdom | happy-dom | happy-dom + Tamagui pre-bundle |
|---|---|---|---|
| Node 22 | 30.3s | 19.5s | 16.5s |
| Node 24 | 27.4s | 15.0s | 13.7s |
| Node 26 | 27.4s | 14.7s | 12.4s |
From where CI was (Node 22, jsdom) to where the branch is (Node 24, happy-dom, pre-bundle), the suite went from 30.3 to 13.7 seconds, 55% less. About two thirds of that is happy-dom. Node 24 and the pre-bundle split the rest. Node 26 is a little faster again, but it isn't LTS yet, so CI moves to 24.
Still that is my laptop's number. On CI, with 2 vCPUs instead of 12, the same suite took 3.9 minutes. On the PR (#174) the vitest step took 106 seconds, down from the 233 to 244 range on the last three runs on main. That's 55% less, the same share as on the laptop, which makes sense: the work that went away is per-file setup, and every file pays it no matter how many cores there are. Collect is still the biggest phase at 60 seconds, so it's what to work on next.
While I was in there I removed some older dependencies that nothing imported anymore. I ensured the type check, lint, the full suite, and the production build all pass without them.
Swapping the frontend toolchain
With the tests on a better footing, I wanted to know what the rest of the frontend job would cost with faster tools. Three candidates: TypeScript 7 (the Go port of the compiler), bun check (a new type checker in Bun canary, shipping in 1.4.3), and the Oxc tools, oxlint and oxfmt, for lint and format. Biome is in the chart as the other Rust option.
Every number below is the median of 5 runs on an M3 Pro with 12 cores, against the same 246 files in the web app. "Cold" means every .tsbuildinfo deleted first. The 2-thread bars are my best stand-in for a 2 vCPU hosted runner, not a measurement from one.
Frontend toolchain, seconds (M3 Pro, 246 files)
| | 12 cores | 2 threads |
|---|---|---|
| Type check |
| tsc 6.0.2 -b (cold) | 12.04s | none |
| tsc 6.0.2 -b (warm) | 6.73s | none |
| tsc 7.0.2 -b (cold) | 1.86s | 2.13s |
| tsc 7.0.2 -b (warm) | 1.16s | none |
| bun check -b (canary) | 1.10s | 1.87s |
| Lint |
| ESLint (current config) | 4.44s | none |
| ESLint (project off) | 2.42s | none |
| oxlint | 0.08s | 0.09s |
| oxlint --type-aware | 0.40s | none |
| Biome (recommended rules) | 0.14s | none |
| Format check |
| Prettier 3.2.5 | 2.26s | none |
| oxfmt | 0.08s | 0.10s |
| Biome | 0.14s | none |
_Median of 5 runs. tsc 6 is single-threaded, so it has no 2-thread number. Blank 2-thread cells were not measured._
Before trusting a faster type checker I wanted to know it reports the same errors. On our config all three report zero. That proves less than it sounds, so I turned on --strict for the web app, which produces 12 errors. tsc 6, tsc 7, and bun check reported the same 12, at the same file, line, column, and error code. Bun's issue tracker has a run of open false-positive reports against bun check from this week, and none of them hit this codebase. That's one codebase, so I'm treating it as "works for us," not "works."
ESLint was paying for type information it never used. The config sets parserOptions.project, which makes typescript-eslint build a full TypeScript program, but it only enables tseslint.configs.recommended, which has no type-aware rules. Turning project off cut lint from 4.4 to 2.4 seconds and changed nothing it reports. Finally got my quick win with a one-line fix that needs no new tool or testing!
oxlint is about 50 times faster than ESLint here. @oxlint/migrate converted 83 rules from the existing config. The one that matters and didn't come across is import/order: Oxc moved import sorting into the formatter. On a probe file with known problems, oxlint caught everything ESLint did except an unused React import. So my one line fix was now out the door again, we can see that quick wins and CI/CD are just not meshing well.
The format check wasn't enforcing anything. Prettier says 199 of 246 files aren't formatted. Prettier only ran from a pre-commit hook, and that hook listed its file types under types, which pre-commit ANDs together. No file is JSON and CSS at the same time, so the hook never matched a single file. Nobody noticed because a hook that matches nothing passes.
Here's how much of the CI job these steps account for. The main column is the last three successful runs before the change. The PR column is the first run of #174, with happy-dom, Node 24, tsc 7, oxlint, and oxfmt all in:
Frontend Check on CI, seconds
| | main | PR #174 |
|---|---|---|
| Install | 14 to 16s | 23s |
| Type check | 21 to 23s | 6s |
| Tests | 233 to 244s | 106s |
| Lint | 13 to 14s | 1s |
| Format check | none | 0.1s |
| Frontend Check job | 323s | 162s |
_main is tsc 6, jsdom and ESLint, with no format check; ranges span the last three runs. PR #174 is tsc 7, happy-dom, oxlint and oxfmt._
The job went from 5.4 minutes to 2.7, half the time. Install is slower because the lockfile changed and the pnpm cache missed. It should be back to about 15 seconds once main saves a new cache.
On a 2 vCPU runner tsc 7 ran in 6 seconds against tsc 6's 22. oxlint reported "Finished in 106ms on 246 files using 2 threads." Type check and lint went from 35 seconds to 7, so the toolchain saved about half a minute. The biggest win was the tests saving more than two minutes. The toolchain swap is worth it for the local loop, where tsc -b dropping from 12 seconds to under 2 changes how often I run it. The PR got faster because of the tests.
So did the PR, this time. All of #174's checks started at 17:15:00, and the last one finished at 17:19:13: 4.2 minutes, down from about 5.1. Frontend Check finished fourth, behind Go Lint at 3.6 minutes and e2e at 4.2. That's the lesson from the first half of this post working the other way around. Cutting the frontend job in half bought 0.9 minutes, because that's how far ahead of e2e it was. e2e was now our critical path now, and its cache fix is sitting in #173.
Where the faster tools do pay off:
- The local loop. Type check, lint, and format check together drop from about 19 seconds cold (12.0 + 4.4 + 2.3) to about 2 (1.9 + 0.1 + 0.1). That's the difference between running all three on every save and running them before a push.
- Pre-commit. The hook runs ESLint over the whole web app on every commit. At 0.1 seconds instead of 4.4 it stops being the hook people skip, and the format check can actually be enforced.
- The CI job shape. Plan item 3 was to move lint and type check into their own job so they run in parallel with tests. At 2 to 3 seconds combined there's no point paying for a second runner and a second pnpm install. They can stay in the test job and add almost nothing.
What I switched to
oxlint for lint, oxfmt for format, and tsc 7 for type checking. bun check was the fastest checker, but it's a canary build with open false-positive bugs, and it's under a second ahead of tsc 7. It gets another look once 1.4.3 is stable.
Dropping ESLint is what made tsc 7 easy. TypeScript 7 doesn't ship the compiler's JavaScript API until 7.1, and typescript-eslint is built on that API, so keeping ESLint would have meant installing TypeScript 6 next to 7. With ESLint gone, one package still needs the API: packages/api generates its clients with openapi-typescript, which builds its output with the TypeScript compiler's factory functions. That one package stays on TypeScript 6. Everything else is on 7.
The switch is six commits:
1. Replace ESLint with oxlint, using the config @oxlint/migrate produced. CI goes from a stale 421-warning ratchet to --deny-warnings.
2. Replace Prettier with oxfmt. oxfmt --migrate=prettier carried over every option except the import-sorting plugin, and sortImports with one custom group for react reproduces the old order: React, third party, @/, relative.
3. Reformat apps/web/src. 197 files, mechanical, nothing else in the commit.
4. Add that commit to .git-blame-ignore-revs, so git blame and GitHub's blame view skip it.
5. Move to TypeScript 7, except packages/api.
6. Fix two lint suppressions the reformat broke.
The last one is worth knowing about if you do this. Two components had a one-line effect with a suppression above it:
// eslint-disable-next-line react-hooks/exhaustive-deps
useEffect(() => { refreshAll(); }, []);
The formatter split that into three lines. The warning is reported on the dependency array, which was now two lines below the comment, so the suppression stopped applying. Tests passed and type checks passed. --deny-warnings is the only thing that caught it, which is a decent argument for making lint warnings fail the build.
To check the new gates actually fail, I added a file with an any and an unused variable, and another with unsorted imports. oxlint and oxfmt each exited non-zero, and both went back to zero once the files were deleted.
e2e: one stage removed, one cache borrowed
With #173 merged, e2e was the slowest check now. We could reuse some of what we learned and apply two changes got it to about 2.
Dropping the scan stage
The last stage of the e2e pipeline ran Grype against the Docker image the test had just built, and failed on any high-severity finding. Grype is a vulnerability scanner that checks an image's packages against a CVE database. In the warm #173 run, that stage cost about 41 seconds: 31 to restore the 462 MB database cache, 4 to install Grype, and 6 for the scan itself.
I asked what the stage was actually testing, and the answer was "nothing in this PR." It called the Grype CLI directly from bash. No product code ran. The image it scanned is a generated MCP server on oven/bun:1-alpine, so the result depended on whether Alpine had an unpatched CVE that day. Our exceptions file had two entries for exactly that: OpenSSL and zlib, no fix available upstream, and main failing the same way as every PR.
The feature, which is Statio scanning images customers push and refusing to deploy one with critical findings, is tested elsewhere. Go unit tests feed the scan pipeline fake Grype reports and check that a critical blocks the deploy. Those run in seconds, need no database, and give the same answer every day.
So scan became an opt-in stage, like the two registry stages already were. The first run after that took 2m4s.
A cache no PR could ever restore
Then #174's e2e took 3m59s. The log said:
Cache not found for input keys: go-Linux-e2e-a6a22cc...
The cause is how GitHub scopes caches. A workflow run can restore caches saved by its own branch or by the base branch, main. It can't see caches from other PRs. ci.yml runs on pushes to main, so its jobs always have a main-scoped cache for a new PR to start from. The e2e workflow only ran on pull_request. Each PR saved a cache that only that PR could use, so re-runs on the same branch were warm, but the first run on every new PR compiled Go from scratch. The deploy stage alone went from 46 seconds to 1m43s.
My first fix was to run e2e on pushes to main as well, so main would save a cache. It works, and it's the wrong trade. It adds a full 4-minute e2e run to every merge just to produce a cache, and that cache holds packages another job had already compiled.
Go Build & Test runs go build ./... and go test ./... on every push to main. The repo is a single Go module, so that job's cache already contains every package e2e compiles. It's 558 MB and keyed on the same go.sum hash. So e2e now restores it without saving anything:
- name: Go cache (restore from ci.yml go job)
uses: actions/cache/restore@v4
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: go-${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}-
restore-keys: |
go-${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}-
go-${{ runner.os }}-go-
That looks like it breaks the rule from bug two: give every job its own cache. It doesn't, and the reason is the rule exists because jobs that share a key also race to _save_ it, and the fastest one overwrites everyone else's with 8 KB. actions/cache/restore is the restore half of the action with no post step. It can never write, so it can't race and it can't pollute anything. It also skips the 14 seconds e2e used to spend uploading its own cache at the end.
e2e, cold vs borrowed cache
| | #174 cold | #176 borrowed cache |
|---|---|---|
| gen stage | 24s | 1s |
| deploy stage | 1m43s | 43s |
| Pipeline step | 3m9s | 1m23s |
| e2e total | 3m59s | 2m9s |
That was the first run of #176, on a branch that had never run e2e. Restoring 558 MB takes about 19 seconds. That's the biggest fixed cost left in the job, and it buys back about 1m40s of compiling (gen + deploy vs restore cache).
When to borrow a cache read-only: the other job builds a superset of what you build, with the same flags, and it runs on your default branch. When not to: if your job compiles things the other job doesn't, like a different module, different build tags, or -race. A restore-only job never saves, so anything it compiles that isn't in the borrowed cache gets compiled again on every run.
A cache key that hashed nothing
While I was in that file, the Bun cache logged its key as bun-Linux-. That key was built from hashFiles('**/bun.lock'), and hashFiles returns an empty string when nothing matches. The Bun project in e2e is the MCP server the test generates at runtime, so there's no lockfile in the repo for it to hash. The key never changed, and the bun install it was meant to speed up takes 3 seconds. I removed it.
Two days later: 2.5 minutes
After #174 and #176, the first PR run took 4.2 minutes. Two more PRs went in that evening, one for lint and one for the cache budget. Here's every PR run since, 125 of them, against each job's worst median from the week before:
Medians, worst this week vs now
| | worst | now |
|---|---|---|
| PR, open to all green | 5.3m | 2.5m |
| Go Lint | 6.0m | 1.4m |
| Go Build & Test | 5.2m | 1.8m |
| Frontend Check | 5.4m | 2.4m |
| e2e | 4.8m | 2.3m |
_"Worst" is each job's highest median during the week: after #166 for the Go jobs, the last main runs before #174 for Frontend Check, before #173 for e2e. "Now" is successful PR runs from October 7 evening to October 9._
The PR number is the time from the first job starting to the last one finishing, first attempts only, on the 44 commits that ran the full set of jobs. Median 2.48 minutes, p90 3.2, worst 3.6. 24 of the 44 finished in under 2.5. The target earlier in this post was 3 minutes with 2 as the stretch. The median is past the target and the p90 is close to it.
Profiling the linter (#177)
Go Lint was still the job with skip-cache: true from #124, so every run did a cold analysis of the whole tree. Before deciding what to cut, I ran golangci-lint with --cpu-profile-path on a cold full-tree run and read the profile. golangci-lint runs every analyzer inside one pass, so pprof -top lumps them together. Summing the profile by analyzer gives this:
golangci-lint CPU by analyzer, cold full tree, seconds
| | flat CPU |
|---|---|
| SSA/IR build (shared) | 56.1s |
| wrapcheck | 16.6s |
| gosec | 13.8s |
| whitespace | 9.4s |
| revive | 4.6s |
| staticcheck | 3.5s |
| unused | 2.8s |
_The SSA/IR build is shared by staticcheck, unused, gosec and nilness. errcheck, goconst, dupl, musttag and exhaustruct were all near zero._
What I found was that the biggest cost isn't a linter. It's building the SSA form of the program, and four linters share it, so disabling any one of them saves nothing, it was determined that those checks were needed so we tried to take a quick look to see if we could speed this one up a bit as well.
The analysis cache is back on. The reason it was off was nolintlint flipping between "this directive is required" and "this directive is unused." The config already had a comment about the same failure from a different cause: golangci-lint caps how many identical issues it reports, and when the issue a //nolint covers gets dropped by that cap, the directive looks unused. max-same-issues: 0 was already in the config to turn that cap off, which is very likely why the flapping stopped reproducing. I checked it rather than assume it: a cold run, then five warm runs and three with a file touched, and all eight reported zero nolintlint issues. That's evidence, not proof, so the workflow comment says what to do if it comes back: put skip-cache: true back, one line. Warm, lint takes 3.1 seconds. One file touched, 7.4. Cold, 36 to 56.
Generated code is no longer a lint target. linters.exclusions.paths looked like it was skipping our generated code. It wasn't. It filters _results_, so every analyzer still walked 1.82 million generated lines, 48% of the tree, and the findings were thrown away at the end. The only way to skip that work is to leave those packages off the command line. A cold run at 2 CPUs, the runner's shape, went from 72.5 to 37.5 seconds with identical output. The generated packages still get type-checked, because the code we lint imports them. What goes away is running analyzers over them.
whitespace is off. 9.4 seconds to check for stray blank lines inside blocks, which is the formatter's job.
Removing that exclusion turned up something else. db/ent was an unanchored path pattern, so it matched more than the generated ent client. It also silenced the 27 hand-written schema files in db/ent/schema/, plus db/ent_client.go and db/ent_test.go, which just happen to contain db/ent in their paths. Both generated directories were already covered by generated: strict, which skips files with a DO NOT EDIT header, so the pattern added nothing for generated code and hid real code. Un-excluding those files surfaced two findings, both fixed in the PR.
Go Lint went from 3.2 to 3.6 minutes on the last runs before #177 to a median of 1.4 after it. The golangci-lint step itself is now 27 to 33 seconds.
A cache that saves on every PR (#181)
The run-keyed cache from #170 has a cost I waved off earlier. Every run saves a new entry, and in this repo each Go cache is about 0.56 GiB. PRs included, that's a lot of half-gigabyte writes. The repo's cache usage was at 12.5 GiB against GitHub's 10 GiB limit, so eviction was running all the time, and it removes the least recently used entries first. Whatever runs least often loses its cache. For us that was the release pipeline, which is its own story and the next post. For PRs, the fact that matters is that every run was paying to upload half a gigabyte that only that branch could ever use.
The fix is to restore on every run and save only on pushes to main:
- name: Go cache (restore)
uses: actions/cache/restore@v4
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}-
restore-keys: |
go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}-
go-${{ runner.os }}-${{ github.job }}-
# ... the job's Go work ...
- name: Go cache (save)
if: github.event_name == 'push'
uses: actions/cache/save@v4
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}-${{ github.run_id }}
PRs read and never write. Main writes on every merge, which is often enough to stay warm. That's 5 to 6 times fewer Go cache writes. The trade is that a PR starts from the last merge instead of its own last run, so a re-run on the same branch compiles a little more than it used to. With the branch scoping rules from the e2e section, the first run of a new PR was starting from main's cache anyway.
Where the time goes now
PR-triggered jobs, median minutes
| | Oct 1 to 6 | After #170 | Now |
|---|---|---|---|
| Go Build & Test | 4.6m | 1.8m | 1.8m |
| Go Lint | 5.0m | 3.0m | 1.4m |
| Frontend Check | 3.7m | 5.1m | 2.4m |
| E2E pipeline | 4.6m | 4.8m | 2.3m |
| Grype (slower scan) | 2.2m | 2.3m | 1.8m |
Every job now finishes within about a minute of the others. Frontend Check is still the last to finish most often, 25 of 44 runs, then Grype Node 7, e2e 4, Grype Go 3, Go Lint 3, Go Build 2. That spread is what a balanced pipeline looks like. It also means no single fix moves the PR much from here. Cutting a minute off Frontend Check would hand the finish line to e2e or Grype Node, which are within seconds of it.
The fixed costs are now bigger than the work. In a recent Go Lint run, the golangci-lint step took 27 to 33 seconds and restoring the Go cache took 34 to 39. Checkout and setup-go add another 16 to 24. Lint is cheaper to run than its cache is to download, and the next round of work is shrinking what each job has to restore.
Self-hosted runners
I have a Proxmox box at home, and it's tempting. A persistent runner keeps its disk between jobs, so the Go build cache, the lint cache, the Grype DB, and node_modules are always warm, and nothing competes for a 10 GB cache budget. I used to think it was the only way to get the lint cache back. #177 did that on hosted runners.
I'm not doing it yet, for two reasons. One box means jobs queue behind each other, and this pipeline is fast because it's parallel. And a self-hosted runner executes whatever code is in the PR on hardware I own, on my network. For a security product that needs a real isolation story (ephemeral VMs per job, no access to the LAN) before it gets anywhere near a pull request.
Notes for next time
- Find the critical path before optimizing anything. Make a table of which job finished last. That's a 20-minute script and it would have changed what I worked on first.
- Check cache sizes, not cache hits."Cache restored" with an 8 KB payload is a miss.
- A cache key that can hit never updates. For build output, put something unique in the key and let restore-keys find the newest entry.
- Jobs that share a cache key race to save it, and the fastest one wins. Put github.job in the key.
- A job that only runs on PRs never has a cache on main. Every new PR starts cold. Borrow a main job's cache read-only rather than adding a main run to seed one.
- Print your cache keys. hashFiles on a glob that matches nothing returns an empty string, and the key quietly becomes a constant.
- Ask what a CI stage tests. A scan of today's CVE database against a base image tests the database, not the PR.
- Profile the linter before tuning it. The biggest cost was a shared SSA build that no single linter owns. Without the profile I'd have disabled staticcheck and saved nothing.
- Exclusions filter results, they don't skip work. To stop analyzing generated code, leave it off the lint targets.
- Anchor your path patterns. db/ent also matched 27 hand-written files and two that only had db/ent in their names.
- A cache that saves on every PR spends the whole repo's budget. Eviction hits the caches that run least often. Save on main, restore everywhere.
- Watch the job you aren't working on. Frontend Check doubled in a week from normal feature work. A weekly look at job medians would have caught it before it was the bottleneck.
And as a final note, if you go down optimizing your CI pipeline, which I think Github would GREATLY appreciate, don't expect quick wins like I was hoping for!
Resources
- actions/cache: how cache keys and restore-keys match
- actions/cache/restore: restore-only action
- actions/cache/save: save-only action
- GitHub docs: restrictions for accessing a cache (branch scope)
- GitHub docs: caching dependencies, usage limits and eviction
- golangci-lint: --new-from-rev and related flags
- Grype: database update behavior
- Vitest: test sharding
- Vitest: fake timers
Mentions
CIを最大4.6倍高速化する方法 ! この記事は2023年11月1日に執筆されたものです。 こんにちは、InflabのDevOpsエンジニアのJakeです。 皆さんは、CIパイプラインの実行に時間がかかりすぎて、PRレビューに影響が出たことはありませんか？ プロジェクトの規模が大きくなるにつれて長くなっていくCI/CDパイプラインの所要時間に、頭を悩ませたことはありませんか？ 本記事では、CIを最大4.6倍まで高速化できたノウハウを共有します。 依存関係とビルドのキャッシュがヒットした場合の数値 最適化の結果 この記事のポイント CI（継続的インテグレーション）の所要時間を最大79%削減 Jenkins Pipelineの最適化: ビルド、テストを実行するDockerコンテナの依存関係キャッシュのパスを設定 モノレポのCIにインクリメンタルビルドを適用: turborepo remote cache シンプルながら非常に効果的で、簡単に適用できるCI/CDパイプライン最適化のノウハウを共有します。 最適化の背景 Inflabの開発チームではモノレポを導入し、パッケージ単位でコードを管理しています。プロジェクトの規模が大きくなるにつれてコードベースとパッケージが増え、CIの所要時間が急増しました。 プロジェクト初期: パッケージ1個、4分14秒 プロジェクト後期: パッケージ9個、12分4秒 プロジェクトの成長とともにコードで使うライブラリがどんどん増え、依存関係（dependency）のインストール時間が長くなりました。 また、ビルドとテストを順番に実行していたため、所要時間はどんどん積み上がっていきました。ビルドパイプライン側で直接並列実行させるなどの手も試しましたが、依存するパッケージ間の複雑な関係や並行性の問題があり、別の解決策を探す必要がありました。 そこで pnpm と turborepo を導入し、モノレポプロジェクトの生産性を改善することになりました。 Yarnとpnpmの比較 次のグラフは、npm、pnpm、yarn、yarn PnPのベンチマークです。 画像出典: pnpm.io グラフのとおり、依存関係のインストールでは条件によって約2倍速くなっています。私たちがまとめたそれぞれの特徴は次のとおりです。 yarn 依存関係のインストール時間: 最大120秒 複雑な node_modules の構造により、ディスクI/OとCPUリソースを消費 pnpm 依存関係のインストール時間: 最大63秒 symlinkでグローバルキャッシュのパス（.pnpm-store）にリンク -> 不要なファイルコピーが減少 -> ディスクI/Oが減少 node_modules └── .pnpm ├── bar@1.0.0 │ └── node_modules │ └── bar -> <store>/bar │ ├── index.js │ └── package.json └── foo@1.0.0 └── node_modules └── foo -> <store>/foo ├── index.js └── package.json これまで使っていた yarn には、依存関係をインストールすると複雑な node_modules ディレクトリ構造のせいでディスクI/OとCPUリソースが無駄に消費されるという欠点がありました。そのため、より高スペックなビルドサーバーが必要になり、所要時間も長くなっていました。 pnpm の導入後は、すべての依存パッケージを .pnpm-store にインストールし、各パッケージ内の node_modules ディレクトリにはsymlinkを張るだけの形になりました。これによりディスクへの書き込みが減り、依存関係のインストール時間を約60秒短縮できました。 Turborepo Turborepoを説明する前に、モノレポについて少しおさらいしておきます。 Monorepoとは？ 画像出典: monorepo.tools モノレポ（Monorepo）とは？ A monorepo is a single repository containing multiple distinct projects, with well-defined relationships. モノレポとは、関係が明確に定義された複数の個別プロジェクトを含むリポジトリを指す。 モノレポを導入すると、毎回多数のリポジトリを作成・管理する手間を減らせます。詳しくはリンク先をご参照ください。 関係が明確に定義されておらず、カプセル化もされていない単なるコードの寄せ集めはモノレポではなく、Monolithとも異なると説明されています。 このようにモノレポでは、複数のアプリやライブラリを1つのリポジトリで管理するため、同じ作業を多数のリポジトリに繰り返し適用する必要がありません。 しかし、モノレポのコードベースの規模が大きくなるとどうなるでしょうか？ ビルドとテストを順次実行することによる所要時間の増加 変更されていない部分まで、不要なビルド & テストを繰り返し実行 次の例は、モノレポのCIパイプラインで、変更されたパッケージのビルドだけを並列実行するよう構成したものです。 stage('Docker Build') { parallel { stage('Package A') { when { anyOf { changeset "cmd/package-alpha//*" changeset "docker/package-alpha//*" } } steps { sh 'docker build -t package-alpha -f docker/package-alpha/Dockerfile .' } } stage('Package B') { when { anyOf { changeset "cmd/package-beta//*" changeset "docker/package-beta//*" } } steps { sh "docker build -t package-beta -f docker/package-beta/Dockerfile ." } } ...(中略)... プロダクトや組織の規模に応じて適切にリポジトリを分ける方法もありますが、結局のところ、モノレポを専用のツールなしで上記の例のようにCIパイプラインで直接管理しようとすると、かなりの工数がかかるでしょう。 パッケージが追加されるたびに手作業でCIパイプラインを修正するのは、本当に大変です。（とにかく手間がかかります） パッケージ同士が複雑な依存関係で絡み合っている場合、それをすべてCIパイプラインで実装するのは困難です。 こんなときに必要になるのが、turborepo や nx のようなモノレポツールです。 さまざまなモノレポツールがありますが、本記事では私たちが導入した turborepo を中心に説明します。 主な機能 turborepo は、JavaScriptおよびTypeScriptのコード向けの高性能なビルドシステムです。公式ページで説明されている特徴は次のとおりです。 Incremental Builds Content-aware hashing Parallel execution Remote Caching Zero runtime overhead Pruned subsets Task pipelines Meets you where you’re at Profile in your browser ここで強調したいのは、Incremental Builds、Content-aware hashing、Parallel execution、Remote Cachingです。順番に見ていきましょう。 Incremental Builds 日本語に直訳すると「増分ビルド」です。 追加・変更された部分だけをテスト & ビルドすることで、大規模なモノレポのCI所要時間を大幅に短縮します。 Detecting affected projects/packages Content-aware hashing コードの内容からハッシュ値を計算し、変更を検知します。 Gitに依存するchangesetとは対照的です。 Parallel execution 従来はビルドを順番に実行していたため、CPUリソースに無駄がありました。 ビルドを並列で実行することで、CPUコアを効率的に使い、ビルドをより早く終わらせることができます。 Local Task Orchestration Remote Caching Local Computation Caching taskの処理結果をローカルに保存して再利用する機能です。同じマシンでは、同じ状態のコードに対するビルドやテストを2回以上行いません。 Distributed Computation Caching Local Computation Cachingを拡張した概念で、taskの処理結果を1台のマシンではなく複数の環境にまたがって共有します。 CIエージェントのいずれかがすでにそのコードの状態をビルド・テストしていれば、ほかのエージェントがそれを再実行することは決してありません。 Remote Cachingは、Distributed Computation Cachingと似た概念です。 同じコードベースを修正する作業者が複数人いると仮定しましょう。 作業者Aが先にビルドとテストを実行していた場合、その結果は共用のキャッシュサーバーに保存されます。 続いて作業者Bがビルドとテストを実行するときは、共用キャッシュサーバーにキャッシュされた作業者Aの処理結果を読み込み、不要なビルドの実行をスキップします。 共用キャッシュサーバーにはビルドの成果物ファイルもあわせて保存されるため、それをすぐにデプロイに活用できます。同じ作業が繰り返し実行されるCIパイプラインを効果的に改善できます。詳しい数値は後ほど共有します。 上記の4つの機能をうまく活用すれば、大規模なプロジェクトもぐっと楽に管理できます。 ところで、なぜ本記事のタイトルはTurborepoの導入ではなく、CIの改善なのでしょうか？ いくつか見落としていた点があり、これらのメリットを活かしきれていなかったからです。ここから説明していきます。 CI所要時間の最適化 Inflabでは、Jenkinsを使ってCI/CDパイプラインを構築しています。 CIジョブがトリガーされると、JenkinsのEC2エージェント上でDockerイメージを起動し、次の順序でジョブを実行します。 依存関係のインストール（Install Dependencies） テスト（Test） ビルド（Build） 静的解析（Static Analysis） 依存関係のインストール 依存関係のインストール処理をまとめると、次のとおりです。 pnpm i --frozen-lockfile 必要な依存ライブラリのバージョンを確認 --frozen-lockfile: 必要な依存関係のマニフェストファイル（pnpm-lock.yaml）が存在するかを確認し、記載されたバージョンに従って依存関係をインストール npmjs.com、GitHub Registry のようなパッケージリポジトリから依存関係をダウンロードしてインストール post-installスクリプトを実行 先ほど、pnpm を導入しただけで依存関係のインストール時間を約60秒短縮できたとお伝えしました。 実行環境が毎回変わるCI環境では、依存関係のキャッシュを導入すれば、さらに多くの時間を節約できます。 環境ごとの依存関係インストールの結果 ローカル環境では、pnpm i を実行するとグローバルな依存関係キャッシュのディレクトリからキャッシュを読み込み、不要なインストールをスキップします。一方、CI環境では実行サーバーが毎回変わる可能性があるため、依存関係のキャッシュを別途設定する必要があります。 ローカル環境: インストールしたすべての依存関係はグローバルキャッシュのディレクトリ（~/Library/pnpm/store/*）に蓄積 一度インストールした依存関係をほかのプロジェクトでも参照し、不要なインストールを省略（キャッシュヒット時は3秒前後） CI環境: 実行サーバーが毎回変わる可能性があるため、依存関係のキャッシュを別途設定しなければ毎回インストールが走る（約60秒） CI環境での依存関係のキャッシュ CI環境で依存関係をキャッシュする方法には、主に次のようなものがあります。 圧縮: node_modules のパスをtar & gzipで圧縮 CI用のキャッシュイメージを作る: 依存関係をあらかじめインストールしたDockerイメージを作成し、それをビルドに活用する方法 キャッシュボリュームのマウント: build agentに残っているglobal cacheを活用する方法 圧縮 GitHub Actions actions/cache@v3 の例 ... - id: cache uses: actions/cache@v3 with: path: '/node_modules' key: ${{ runner.os }}-node-${{ hashFiles(format('/{0}', steps.hash-file.outputs.hash-file)) }} ... Jenkins Job Cacherプラグインの活用例 cache(maxCacheSize: 250, defaultBranch: 'develop', caches: [ arbitraryFileCache(path: 'node_modules', cacheValidityDecidingFile: 'package-lock.json') ]) { // ... } 依存関係が入った node_modules ディレクトリをtarballにまとめ、gzipやxzなどで圧縮する方法です。 キャッシュは pnpm-lock.yaml のようなマニフェストファイルのハッシュで区別されるため、必要な依存関係が変わるたびにインストールし直す必要があるのが欠点です。 また、node_modules ディレクトリの圧縮は基本的にシングルスレッドで行われるため、時間がかかります。 プロジェクトの規模が大きくなると node_modules ディレクトリも大きく複雑になるため、キャッシュファイルのダウンロードと展開だけで2分以上かかることもあります。 これでは本末転倒なので、いっそ圧縮しないほうが良い場合もあります。 プロジェクトの規模が大きい場合 pnpm install の所要時間: 60秒 キャッシュファイルのダウンロードと展開の所要時間: 120秒以上 場合によっては、実際に所要時間を計測・比較したうえで判断する必要あり GitHub Actionsを使う場合は、pnpmの公式ドキュメントのガイドに従い、cache: 'pnpm' を設定するだけで簡単に適用できます。 pnpm公式ドキュメントで推奨されているGitHub Actionsパイプラインの例 name: pnpm Example Workflow on: push: jobs: build: runs-on: ubuntu-20.04 strategy: matrix: node-version: [15] steps: - uses: actions/checkout@v3 - uses: pnpm/action-setup@v2 with: version: 8 - name: Use Node.js ${{ matrix.node-version }} uses: actions/setup-node@v3 with: node-version: ${{ matrix.node-version }} cache: 'pnpm' - name: Install dependencies run: pnpm install Jenkinsパイプラインを使う場合は、次のように設定することが推奨されています。 pipeline { agent { docker { image 'node:lts-bullseye-slim' args '-p 3000:3000' } } stages { stage('Build') { steps { sh 'corepack enable' sh 'corepack prepare pnpm@latest-8 --activate' sh 'pnpm install' } } } } 上記の例ではpnpmのstore pathを別途指定していないため、JENKINS_WORKSPACE の配下に .pnpm-store のパスを作成して依存関係をキャッシュします。 しかし、Jenkins Multibranch Pipelineの場合は、ブランチやPull Requestごとに別々のWorkspaceのパスを持ちます。 つまり、同じプロジェクトであっても、異なるブランチやPR間で依存関係のキャッシュを共有できません。新しいPRが作成されるたびに、依存関係を最初からインストールし直すことになります。 これを解決する方法は、キャッシュボリュームのマウントで後ほど説明します。 CI用のキャッシュイメージを作る 次に紹介するのは、CI用のキャッシュイメージを作る方法です。 次の例は、pnpm fetch を使って依存関係のキャッシュイメージを作る Dockerfile です。 FROM node:18-slim pnpmのインストール手順は省略 pnpm fetch does require only lockfile COPY pnpm-lock.yaml ./ RUN pnpm fetch --prod 上記のイメージをpullしてから依存関係をインストールすれば、キャッシュを活用してすぐにビルドできます。 イメージタグにはマニフェストファイル（pnpm-lock.yaml）のハッシュ値を指定します。そのため先ほどの方法と同じく、このファイルが変更されたらイメージを再ビルドする必要があります。 when cache miss, build and push into image registry like ECR docker build -t rallit-frontend-dep:02d93ab24 -f dep.Dockerfile . docker push rallit-frontend-dep:02d93ab24 when build docker run -i --rm -v $WORKSPACE:/workspace rallit-frontend-dep:02d93ab24 'pnpm i --frozen-lockfile' ただし、pnpm fetch --prod コマンドでインストールした依存関係をイメージとしてビルドしてpullするため、その過程でディスクのオーバーヘッドが発生します。 社内で所要時間を計測した結果、実際のCI環境ではイメージをpullして実行するのに約168秒かかることが分かりました。 キャッシュイメージを作る方法も、プロジェクトの規模が大きいほど時間がかかり、通常の pnpm install より約108秒も遅くなりました。 pnpm-lock.yaml が変更されるたびにこうしたキャッシュイメージをビルドし、コンテナイメージレジストリにアップロードするのは非効率だと判断しました。 Dockerのマルチステージビルドを使ってイメージをビルドしている方は、pnpmの公式ドキュメントを参考にcache type mountingを適用することをおすすめします。 FROM base AS prod-deps RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm install --prod --frozen-lockfile FROM base AS build RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm install --frozen-lockfile RUN pnpm run build DockerのRUN専用キャッシュを活用する パッケージをインストールするたびに、毎回インターネットからすべてのパッケージを取得する必要はありません。 --mount フラグとともに明示的なキャッシュを使うと、ビルド間で対象ディレクトリの内容が保持されます。 このレイヤーを再ビルドする必要がある場合は、そのパスにあるキャッシュを活用します。 私たちはDockerのマルチステージビルドを使わず、ビルドの段階ごとに個別に最適化したイメージを使っているため、この方法は適用できませんでした。 続いて、依存関係のインストール時間を大きく短縮できた、キャッシュボリュームのマウントについて説明します。 キャッシュボリュームのマウント 先ほど、Jenkinsパイプラインでpnpmを使って依存関係をインストールする場合は、キャッシュを別途設定する必要があると述べました。 つまり、同じプロジェクトであっても、異なるブランチやPR間で依存関係のキャッシュを共有できません。新しいPRが作成されるたびに、依存関係を最初からインストールし直すことになります。 実際の .pnpm-store のパスを比較しながら見ていきましょう。 従来（AS-IS） CACHE_PATH のパスが JENKINS_WORKSPACE の中にある 異なるプロジェクト、ブランチ、PR間で依存関係のキャッシュを共有できない パスの比較 JENKINS_WORKSPACE=/workspace/rallit_frontend_PR_15242 CACHE_PATH=/workspace/rallit_frontend_PR_15242/.pnpm-store 例えば、rallit_frontendのPR#15242のCIジョブが完了したとしましょう。 /workspace/rallit_frontend_PR_15242/.pnpm-store には、先ほどインストールした依存関係が残っています。 続いてPR#15243が作成されると、どうなるでしょうか？ PR#15243は /workspace/rallit_frontend_PR_15243/.pnpm-store を参照することになります。 このフォルダは実際には存在しないので、依存関係を最初からインストールすることになり、その分時間がかかります。 先にインストールしたPR#15242の依存関係キャッシュは、まったく使えません。 たった1行の変更でも約60秒余計に待たされるので、かなり不便です。 変更後（TO-BE） エージェント内にGlobal Dependency Cacheのパスを設定 同じエージェントで実行するビルドはDependency Cacheを活用 パスの比較 JENKINS_WORKSPACE=/workspace/rallit_frontend_PR_15242 CACHE_PATH=/tmp/.pnpm-store CIジョブを実行するすべてのコンテナで、pnpm store-path オプションに同じパスを指定しました。 これにより、rallit_frontendのPR#15242、PR#15243はどちらも同じパス（/tmp/.pnpm-store）を参照するようになります。 PR#15243のCIジョブでは、PR#15242ですでにインストールされた依存関係を再利用することになり、約60秒かかっていた依存関係のインストール時間をわずか3秒に短縮できました。 pnpm store-path オプションは、ビルド用のベースイメージを作る Dockerfile で次のように指定しました。 set global node cache directory ARG CACHE_PATH=/cache/node RUN yarn config set cache-folder ${CACHE_PATH}/.yarn RUN pnpm config set store-dir ${CACHE_PATH}/.pnpm-store パッケージマネージャーにYarnを使っているプロジェクトでも依存関係のキャッシュパスを共有できるよう、Yarnにも同様の設定を入れています。 続いて、このDockerイメージを実行する Jenkinsfile 側でも CACHE_PATH 環境変数を設定します。 env.DOCKER_OPTION = '-v /tmp:/cache/node -e CI=1' (中略) stage('Install') { steps { script { sh "docker run --rm -i -v $WORKSPACE:/workspace $DOCKER_OPTION $NODE_IMAGE 'pnpm i --frozen-lockfile'" } } } JenkinsfileのDocker拡張機能を使わず、コマンドで直接Dockerコンテナを実行する場合 pnpmがCI環境かどうかの判定に使う環境変数の1つである BUILD_NUMBER が渡されないため、CI=1 のデフォルト値が適用されません。 -e BUILD_NUMBER=$BUILD_NUMBER で渡すか、-e CI=1 環境変数を直接設定するのがおすすめです。 詳しくは公式ドキュメントをご参照ください。 CI=trueの場合、pnpmはロックファイル（pnpm-lock.yaml）を生成せず、ロックファイルがマニフェストと同期していない場合、更新が必要な場合、または存在しない場合はインストールに失敗します。 この対応によって、直前のビルドで使われた依存関係のキャッシュを、ほかのリポジトリのCIでも再利用できるようになりました。 適用前 - pnpm i --frozen-lockfile Scope: all 7 workspace projects Lockfile is up to date, resolution step is skipped Progress: resolved 1, reused 0, downloaded 0, added 0 Packages: +3357 (中略) Progress: resolved 3357, reused 1253, downloaded 2025, added 3357, done Done in 1m 28.6s 適用後 - pnpm i --frozen-lockfile Scope: all 7 workspace projects Lockfile is up to date, resolution step is skipped Packages: -11 ----------- Done in 2.6s 依存関係インストール処理の最適化結果 依存関係キャッシュMiss: 1分28秒 依存関係キャッシュHit: 2.6秒 依存関係のインストール時間を最大97%短縮 依存関係のインストールは従来の88秒から2.6秒へと85.4秒減り、約97%短縮できました。 もちろん、ビルドエージェントが入れ替わって依存関係のキャッシュパスが初期化されると、最初の1回だけは1分28秒のインストールが必要です。 それでも、ビルドのたびに依存関係をインストールしていた従来とは違い、エージェント内で依存関係のキャッシュを共有できるので、後続のビルドではインストール時間を97%以上短縮できます。 複数のリポジトリが同じライブラリを使っていても、それぞれバージョンが異なる場合があります。 この場合、使用するバージョンを1つに統一するとキャッシュヒット率（Cache Hit Rate）が上がり、上記と同様に依存関係のインストール時間を大幅に短縮できました。 例: inflearn-frontend と rallit-frontend で使っている Datadog RUM（Real User Monitoring） エージェントのバージョンが一致している場合、npm registryからの依存関係のダウンロードとインストールをスキップし、ビルドエージェントにあるグローバルな依存関係キャッシュを活用して時間を短縮できる。 （rallit-frontend で先にインストールしていれば inflearn-frontend でそれを利用でき、その逆も可能） テストとビルドの所要時間の短縮 先に述べたとおり、モノレポでは一部のパッケージを修正しただけでも、すべてのパッケージをテスト・ビルドしなければなりません。 Turborepoを使えば、変更されたパッケージと、それに依存するパッケージだけをテスト・ビルドするので、CIの所要時間を大幅に短縮できます。これをTask Cacheと呼びます。 Inflabのフロントエンドチームは早くからTurborepoをローカル開発環境に導入していたため、実際に変更されたコードだけをテスト・ビルドして、作業内容を素早く検証できていました。 これはコンテキストスイッチの頻度を減らし、生産性の向上につながりました。（実験：遅いビルド時間の隠れたコスト in GeekNews） Turborepo Task Caching package.json にある build、test、lint などをtaskと呼びます。 Turborepoはtaskの実行結果とログをキャッシュし、遅いtaskを高速化します。 Content-aware hashingでソースコードの変更を検知し、変更されたパッケージに対してのみ build、test、lint などのtaskを実行します。 テストとビルドの結果は、まずLocal Filesystem Cacheとして ./node_modules/.cache/turbo/78awdk123.tar.zst のようなパスに保存されます。 Taskキャッシュがヒット（hit）したパッケージはビルドをスキップし、変更されたパッケージだけを新たにテスト・ビルドします。 実行結果をもとにTaskキャッシュを更新し、常に最新の状態に保ちます。 Taskキャッシュがミス（miss）した場合は、次のようにtaskを実行し、その結果でLocal Filesystem cacheを更新します。 /lib:build: cache miss, executing 86fb10d5583f361c /app:build: cache miss, executing 7804de963ab93a0e (中略) Tasks: 5 successful, 5 total Cached: 0 cached, 5 total Time: 4m36.208s ...writing to cache... Taskキャッシュがヒット（hit）した場合は、キャッシュ作成時のログを再生（replay）し、ビルド結果をキャッシュサーバーからダウンロードします。 /lib:build: cache hit, replaying logs 86fb10d5583f361c /ds:build: cache hit, replaying logs 7804de963ab93a0e /a:build: cache hit, replaying logs 8be7f0056074cbc3 /b:build: cache hit, replaying logs d01733e33b71a814 Tasks: 5 successful, 5 total Cached: 5 cached, 5 total Time: 4.114s >>> FULL TURBO Taskキャッシュの適用結果 TaskキャッシュMiss: 4分36秒 TaskキャッシュHit: 4.114秒 Taskキャッシュの導入で所要時間を約98.5%改善 Taskキャッシュも依存関係のキャッシュと同じく、キャッシュがない初回は時間がかかります。しかしそれ以降のビルドでは変更されたパッケージに対してのみTaskを実行するため、かなりの時間を節約できました。 問題は、別途設定をしなければLocal Filesystem Cacheしか使えないことです。 Jenkinsエージェントではworkspaceをブランチ・PR単位で分けているため、検証済みのPRを develop ブランチにマージすると、テストとビルドを最初からやり直す必要がありました。 CIパイプラインで一度ビルドしたものは再ビルドせずに済ませたいところです。どうすれば改善できるでしょうか？ Turborepo Remote Caching Turborepo Remote Caching機能を使えば、この問題を解決できます。 Taskキャッシュを複数のビルドエージェント間で共有でき、インクリメンタルビルドを積極的に活用できるようになります。 Remote Cachingを使うには、追加費用が発生するVercelのクラウドを使うか、Vercelが公開しているAPI仕様に従ってTurborepo custom remote cache serverを自前で実装する必要があります。 私たちは、オープンソースとして公開されているducktors/turborepo-remote-cacheを使いました。 ducktors/turborepo-remote-cache Turborepo custom remote cache serverのオープンソース実装で、self-hosted環境で使うのに適しています。 シンプルなWebサーバーアプリケーションで、次の実行方法をサポートしています。 Dockerコンテナとして実行 AWS Lambdaなどのサーバーレス関数として実行 ストレージについては、次のようなさまざまなオプションが用意されています。 Amazon S3（または互換ストレージ） ファイルシステム Lambda + S3の組み合わせで完全にサーバーレスで構築することもできます。それぞれのメリット・デメリットを踏まえて、環境に合った構成を選べば良いでしょう。 ここでは、Docker + ファイルシステムでカスタムキャッシュサーバーを手軽に構築する手順を紹介します。 次のように compose.yaml ファイルを作成します。${} の中に入る変数は .env ファイルで定義します。 services: turborepo-remote-cache: image: ducktors/turborepo-remote-cache:1.14.2 ports: - 3000:3000 volumes: - ./tmp:/tmp environment: - NODE_ENV=production - PORT=3000 - TURBO_TOKEN=${TURBO_TOKEN} - LOG_LEVEL=info - STORAGE_PROVIDER=local - AWS_REGION=${AWS_REGION} 続いてガイドに従い、Turborepoのカスタムキャッシュサーバーのアドレスといくつかのパラメーターを、Dockerの環境変数として渡します。 TURBO_API: キャッシュサーバーのアドレス TURBO_TOKEN: 認証トークン TURBO_TEAM: キャッシュを使うteam（主にプロジェクト単位で指定） stage('Build') { steps { script { env.DOCKER_OPTION = "$DOCKER_OPTION -e TURBO_API=$TURBO_API -e TURBO_TOKEN=$TURBO_TOKEN -e TURBO_TEAM=$TURBO_TEAM" sh "docker run --rm -i -v $WORKSPACE:/workspace $DOCKER_OPTION $NODE_IMAGE 'pnpm build'" } } } Dockerの環境変数ではなくturbo.jsonなどの設定ファイルに直接書く方法もありますが、今回はCI専用のキャッシュサーバーとの連携なので、上記のようにパイプライン側に記述しました。 適用後にビルドを何度か実行すると Remote caching enabled というログが出力され、Taskキャッシュが蓄積・再利用されていることが確認できます。 - pnpm build:development • Packages in scope: ...... • Running build:development in 6 packages • Remote caching enabled a:build: cache hit, replaying logs 83da8d8fb24dd1a9 b:build: cache hit, replaying logs 249bb66c4e03301b c:build: cache hit, replaying logs 8be7f0056074cbc3 (中略) Tasks: 5 successful, 5 total Cached: 5 cached, 5 total Time: 4.114s >>> FULL TURBO ハッシュはコードの状態をもとに計算されるので、ブランチやPRが違っても、コードを修正していなければTaskが無駄に再実行されることはありません。 では、Continuous Deploymentのパイプラインでビルドキャッシュを使う場合、ビルド成果物はどうデプロイすれば良いでしょうか？ turbo.jsonでコマンドごとにoutputsのパスを設定すると、そのパスがビルド成果物として扱われ、Turborepoのリモートキャッシュサーバーに保存されます。キャッシュヒット時には、それがダウンロードされます。 そのため、デプロイ時の混乱を防ぐには、outputs オプションを正しく設定しておく必要があります。（関連する公式ドキュメント） turbo.json { "$schema": "turbo.build", "pipeline": { "app#build": { "outputs": ["dist/"] }, "util#build": { "outputs": ["lib/"] } } } このオプションを正しく設定していないと、デプロイパイプラインを経由しない緊急ロールバックのあとにパイプラインを再実行しても、本番環境に反映されないことがあるので注意してください。 便利さの裏には慎重さと正確さが欠かせないことを、心に留めておきましょう。 ソースコードの静的解析 依存関係のインストールと、テスト・ビルド。この2つを最適化した結果、すべてのキャッシュがヒットすれば、ここまでの処理は10秒前後で終わります。 静的解析では、コードのセキュリティ脆弱性から、重複やパフォーマンス上の問題が起こりうる箇所まで見つけられるため、SonarQubeを積極的に導入・活用しています。 しかし、CIのたびに静的解析を実行していたため、コードの規模によっては40秒から最大2分以上かかることもありました。 これを短縮するため、解析が不要なファイルを sonar-project.properties で除外しました。 sonar.exclusions=packages//public/mockServiceWorker.js, packages//src/@types//, packages//src/mocks//, packages//vite.config.ts, sonar.analysis.mode=incrementalは使用不可 以前は -Dsonar.analysis.mode=incremental パラメーターを追加して変更箇所だけを解析する機能がありましたが、現在はサポートが終了しています。 代わりに、変更されたディレクトリだけを解析する仕組みをパイプラインに自前で実装する方法や、CIのポリシーを変更して develop ブランチへのマージ時にのみ解析する方法があります。 銀の弾丸（silver-bullet）は存在しないので、皆さんの環境に合った方法を選んで改善してください。 詳しくはSonarQubeのドキュメントをご参照ください。 私たちのContinuous Deploymentパイプラインには、コンテナのデプロイ完了を待つ処理がありました。その待ち時間にSonarQubeの解析を並行して実行することで、約40秒〜最大2分かかっていた静的解析の時間を短縮できました。 CI最適化の結果 キャッシュHitを想定（install、buildがともにヒットした場合） pnpmの依存関係キャッシュとTurborepoのTaskキャッシュを適用したことで、CIの所要時間を最大79%短縮できました。 モノレポの中でも、パッケージ数とコード量が多いプロジェクトほど最適化の効果が際立ちました。パッケージが多いほど、修正による影響範囲が小さくなるためです。 さらに、turborepo もモノレポも導入していないプロジェクトE、Fでも、依存関係のインストールを最適化しただけでCIが約1.6〜2倍速くなりました。 調査対象の平均CI所要時間は6分3秒から2分4秒になり、最大66%改善できました。静的解析まで最適化すれば、さらに時間を節約できると期待しています。 おわりに 以上、簡単な修正でCIの所要時間を劇的に改善した経験を共有しました。皆さんもぜひ試してみて、得られた成果やより良い方法をコメントで教えていただけるとうれしいです。 最適化を進めるなかで私を信頼し、込み入った質問にも丁寧に答えてくださったInflab開発チームの皆さんのおかげで、無事に最適化を終えることができました。ありがとうございました。 次回の記事では、JenkinsのEC2エージェントのコンピューティングコストを最大70%節約できたスポットインスタンスの導入をテーマにお届けします。 Inflab DevOpsチームの最適化はまだまだ続きます。今後もご期待ください！ 参考資料 monorepo.tools pnpm.io
GitHub Actions: If you deploy to GitHub pages speed up page builds by using caching: actions/cache. Furthermore split into build and deploy stages to have build validation for PRs if you work with branches. t.co
@adam_chal @huntharo not a concrete example, but all the docs are here...LLMs are great at generating this stuff actions/cache and here is a more concrete guide: blacksmith.sh
CI/CD Optimizations, Part 1: PR Checks --- title: CI/CD Optimizations, Part 1: PR Checks cover_image: logicalbytes.dev canonical_url: logicalbytes.dev --- !CI/CD Optimizations, Part 1: PR Checks cover art Statio's monorepo has a Go backend, a React web app, a desktop client, and a pile of GitHub Actions workflows that run on every pull request. I've seen a lot of talk on X recently about optimizing (usually a rust migration). So I decided that I might as well take a crack at what I have going on. Over the last week I spent a few evenings making those workflows faster. Go Build & Test went from 5.2 minutes to 1.8. Go Lint went from 6.0 to 3.0. The time from opening a PR to seeing all green went from about 5.3 to 2.5 which was slightly better than my 3 minute target which was based on looking away for a minute then every check had completed. All numbers below are medians of successful jobs, pulled from the GitHub API across 493 workflow runs between August 5 and October 7, plus 125 PR runs from October 7 to 9 for the last section. Runners are the standard ubuntu-latest hosted ones. The PR pipeline Three workflows trigger on a pull request: - CI (ci.yml): Go Build & Test, Go Lint, Frontend Check, OpenAPI codegen drift, a contract test against platform-core, deploy config validation, and a PR title check. - Security Scan : Grype against the Go modules and against the Node lockfile. - E2E MCP Pipeline : builds the gateway, stands up a test MCP server, and runs the pipeline end to end. Everything runs in parallel. The important thing to remember is that the PR is done when the slowest job is done. Shaving time off any other job changes nothing you can feel. Mid-September In mid-September the slowest jobs were the Go ones. Go Lint ran 3.8 minutes, Go Build & Test 3.6, and Frontend Check about a minute. A typical PR finished in about 3.6 to 4 minutes, and Go Lint or Go Build was the last job to finish in 47 of 49 runs. This was good enough so I didn't worry about it. However the number began climbing as I added deeper checks. Frontend tests landed in CI (#118, Sep 17). Before this, Frontend Check ran tsc and nothing else, so the vitest suite could go red and no check would notice. After it, the job also ran vitest and an eslint ratchet (--max-warnings 441, a number that's only allowed to go down). At the time the suite had 9 test files and the job went to about 2 minutes. That's a fine trade. Lint lost its cache (#124, Oct 1). golangci-lint caches analysis results between runs. Ours flapped: nolintlint would report a //nolint directive as unused on one run and fine on the next, depending on what the cache held. A lint check that fails at random teaches everyone to hit re-run. I set skip-cache: true and Go Lint went from 3.8 to 5.0 minutes. I'd make that call again. A slow honest check beats a fast one nobody trusts. The web app grew. This one wasn't a CI change at all. Between October 1 and October 6 we shipped onboarding, a help menu, container registry UI, new pricing pages, and tests for all of it. Test files went from 22 to 42. Frontend Check's daily median went 2.5, 2.8, 3.6, 3.8, 4.4, 5.0 minutes. A good consequence of ensuring we had testing coverage and ensured that nothing would break in the future, bad because now we were at over 5 minutes, not where I wanted to be. By October 6 a PR took about 5.3 minutes and every job felt slow. So I started with what I knew best, Go, because Go was the part I was sure was broken. Bug one: a cache key that can hit never updates The Go jobs used actions/setup-go with its built-in cache. It keys the cache on a hash of go.sum. Which seemed right until I saw cache misses and looked into how actions/cache decides whether to save: It only saves when the primary key missed. So the first run after a go.sum change misses, builds, and saves. Every run after that hits the same key, restores that snapshot, and never saves again. The module cache stays fine because modules only change when go.sum does. The build cache, ~/.cache/go-build, goes stale. Every commit after the snapshot recompiles the packages that changed since then, and the list grows with every PR until someone bumps a dependency. The fix (#166) turns off setup-go's cache and uses actions/cache directly, with github.run_id at the end of the key: - uses: actions/setup-go@v7 with: go-version-file: go.mod cache: false - name: Go cache uses: actions/cache@v4 with: path: | ~/.cache/go-build ~/go/pkg/mod key: go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}-${{ github.run_id }} restore-keys: | go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}- go-${{ runner.os }}- The run ID guarantees the primary key misses every time, so every run saves. restore-keys does prefix matching and returns the newest entry, so every run starts from the most recent cache. Go's build cache is content-addressed, so a stale entry is never used for a file that changed, it sits there until GitHub evicts it. Proud of myself for the fix I merged it and the numbers got worse. Go Build went from 4.6 to 5.2 minutes and Go Lint from 5.0 to 6.0. That was expected because we had some other bugs that were preventing this fix from fully working. Bug two: seven jobs, one key, and the fastest one wins Every job in ci.yml that touches Go used that same cache step, so every job in a run computed the same key. GitHub's cache won't let two writers reserve one key. The first job to reach its post step saves. The rest log this and move on: Failed to save: Unable to reserve cache with key go-Linux-<hash>-<run_id>, another job may be creating this cache. It's a warning, so the job stays green. Which job gets to its post step first? The fastest one, which is also the job that compiles the least. In our workflow that's a job like OpenAPI codegen drift or deploy config validation, both done in under a minute. The cache that won was 8 KB. That became the newest entry, so restore-keys handed 8 KB to Go Build, Go Lint, and everything else on every run. They all logged "Cache restored," then compiled the whole tree from scratch, then lost the race to save. The fix (#170) is one more variable in the key: key: go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}-${{ github.run_id }} restore-keys: | go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}- go-${{ runner.os }}-${{ github.job }}- Each job now has its own cache, and that makes sense for more than the race. Build & Test compiles the whole module and its test binaries, Lint loads packages for analysis, and codegen builds one tool. Sharing one cache between them was never going to work and something I had missed previously. However, that fix that made things slower before would now come into play. Results: Go jobs, median minutes | | Before #166 | After #166 | After #170 | |---|---|---|---| | Go Build & Test | 4.6m | 5.2m | 1.8m | | Go Lint | 5.0m | 6.0m | 3.0m | | OpenAPI Codegen | 0.8m | 1.0m | 0.5m | Go Lint still runs with skip-cache: true, so its 3.0 minutes is a warm Go build cache plus a cold lint analysis. That's the expected trade-off from #124. When to use a per-job, run-keyed cache: any job whose build output changes often, which is most Go projects. When not to: if a job's cache is only downloaded dependencies, a key on the lockfile hash is fine because the cache only needs to change when the lockfile does. The run ID also means one cache entry per run. GitHub caps a repo at 10 GB and evicts the least recently used entries, so keep an eye on size. I thought ours sat well under the cap. It didn't, so more lessons were to be learned in the near future. Also Github, maybe like show when we have caches full in some sort of billing page with more identification around them? Regardless we should have checked our cache sizes because we would have caught that 8kb for a Go monorepo is probably wrong. Three smaller fixes (#173) With the main cache working, I went looking for other jobs paying the same costs. Lint only what changed on PRs. golangci-lint has --new-from-rev, which reports only issues on lines that differ from a given revision. On pull requests we now pass --new-from-rev=origin/main. Push to main still lints the whole tree, so anything the scoped run misses gets caught on merge. - uses: actions/checkout@v7 with: fetch-depth: 0 # --new-from-rev needs main's history - uses: golangci/golangci-lint-action@v9 with: skip-cache: true args: >- --timeout=10m --fix=false ${{ github.event_name == 'pull_request' && '--new-from-rev=origin/main' || '' }} This flag still type-checks and analyzes every package, because most linters need the whole program. It only filters which issues get reported. The gain is smaller than "lint 5 files instead of 500" sounds. On the first run the golangci step went from about 130 seconds to 102. When not to use it: if main isn't always clean. Scoped linting assumes the baseline passes. If main has existing failures, PRs will skip them and nobody will fix them. Cache the Grype vulnerability database. Both Grype jobs downloaded the full vulnerability DB on every run. It's 462 MB and upstream publishes about once a day. Now it's cached with a date in the key: - name: Date for Grype DB cache key id: date run: echo "day=$(date -u +%Y-%m-%d)" >> "$GITHUB_OUTPUT" - uses: actions/cache@v4 with: path: ~/.cache/grype/db key: grype-db-${{ runner.os }}-${{ steps.date.outputs.day }} restore-keys: grype-db-${{ runner.os }}- The first job each day downloads the DB and saves it. Every job after that restores it. Grype checks for a newer DB when it starts, so a day-old copy from restore-keys gets updated rather than scanned against as-is. For Grype a lockfile hash would be the wrong key here because the DB changes when new CVEs are published, not when our code does. Give the E2E workflow the same Go cache fix. The E2E workflow lives in its own file and never got #166 or #170. It was still on setup-go's built-in cache with the stale-snapshot problem. It now has the job-scoped, run-keyed cache plus caches for Bun's install directory and the Grype DB, since the last stage of that pipeline shells out to Grype. The first run of #173 was cold on purpose: new cache keys, so the second was the test because we had built our caches: PR #173, cold run vs warm run | | Before #173 | #173 cold | #173 warm | |---|---|---|---| | Grype Go | 2.0m | 2.3m | 1.2m | | Grype Node | 2.3m | 2.9m | 1.8m | | E2E pipeline | 4.8m | 5.8m | 3.1m | | Go Lint | 3.0m | 3.1m | none | _Go Lint did not rerun on the warm attempt._ The Grype DB cache is 462 MB, and restoring it is not free. It took 13 seconds in one job and 28 in another. That's still well ahead of downloading and unpacking the same file from upstream, but a cache is still a download, from a closer server. E2E restored a 324 MB Go cache and dropped by 1.7 minutes against its pre-#173 number. The lint job didn't rerun on the second attempt; its number from the first run is the one above, and since skip-cache stays on, there's no warm state for it to gain. The PR didn't get faster Here's the per-era picture across all the PR-triggered jobs. Every job runs in parallel, so the PR finishes when the longest bar does. Switch between the eras: PR-triggered jobs, median minutes | | Mid-Sept | Oct 1 to 6 | After #170 | |---|---|---|---|---| | Go Build & Test | 3.6m | 4.6m | 1.8m | | Go Lint | 3.8m | 5.0m | 3.0m | | Frontend Check | 1.0m | 3.7m | 5.1m | | E2E pipeline | 3.6m | 4.6m | 4.8m | | Grype (slower scan) | 2.0m | 2.2m | 2.3m | Going by job and it's a big win. However if are focused on the finish line, and it's flat. In mid-September the slowest job was Go Lint at 3.8 minutes. Today it's Frontend Check at 5.1. The Go fixes took two jobs off the critical path, and the frontend job, which had doubled in the same week, was already standing behind them so we saved 12 seconds, not quite good enough but that means we only had one more job to optimize. Some of the jobs we could have left as is, they do not affect our PR times and checks at all but none of that means the Go work was wasted. CI minutes cost money and runner slots, and Go Build failing at 1.8 minutes instead of 5.2 gets you a red X three minutes sooner. Creating earlier signals means faster turn around time on a fix for it. That matters, because a failing test is the most common reason to look at CI at all. The goals was still "my PRs are faster", so despite all the optimizations and minutes saved, we again were at a measurable 12 seconds faster PR times. A better prioritization would have been to pull the jobs from your last 20 PR runs, find which one finished last in each, and count. If one job is last 80% of the time, that's where we could have started and the GitHub API returns started_at and completed_at for every job, and a 20-line script against /actions/runs/{id}/jobs could have given us a better priority list. However, I think it was better to knock out one languages performance then switch onto the next one. If you are thinking of pure speed, knock down the slowest jobs first, if you are more like me and you want to start with where you are the most familiar, knocking things down is only beneficial in the long run. The actual bottleneck Frontend Check runs for 5.1 minutes and the vitest step is 3.9 minutes of it. Vitest's summary from the latest run: Duration 233.89s (transform 1.94s, setup 2.31s, collect 72.96s, tests 125.55s, environment 21.60s, prepare 4.07s) Collect is 73 seconds. That's vitest loading and transforming every test file and its imports before anything runs. Our components pull in Tamagui, and the first render in each file pays a one-time config compile. With 43 test files that cost is paid 43 times. Some tests sit around 2.6 seconds each. EgressPolicyPanel, ServerSuspension, HostedServerSlots, LimitExceeded, and HelpMenu all cluster at that number. My first guess was real timers, since tests that all take the same odd length of time are usually waiting on a debounce or a waitFor running to its timeout. I was just wrong, sad I couldn't get a quick win, there were only two test files touch timers at all, and both already use vi.useFakeTimers(). So the 2.6 seconds was the first render in each file, and almost all of it was jsdom applying Tamagui's styles. More on that below. We already tried the obvious config change. #171 switched vitest from its default forks pool to threads. On a laptop that cut the suite from 59.5 to 31.5 seconds because the V8 JIT stays warm across files. On CI the job didn't move: 5.1 minutes before, 5.1 after, demoralizing but hey just have to keep digging. A hosted runner on a private repo has 2 vCPUs and the laptop has a lot more, so there was almost no parallelism to gain. We also tried isolate: false and it broke 24 files on state leaking between them, the kind of leak a shared in-memory localStorage mock produces, so isolation stays. A lot of testing and not a lot of results, but again, see what sticks. From there the plan was: 1. Make the test environment cheaper. Every file pays for a DOM, and the DOM is where the first-render time goes. Low risk if the suite stays green. 2. Shard the suite across jobs. vitest --shard=1/3 splits files across three parallel jobs. It's the one change that scales with suite size instead of fighting it. The cost is three runners instead of one, plus repeating the pnpm install and collect phase per shard. 3. Move lint and type-check out of the test job. They don't depend on each other, so running them in parallel takes them off the critical path entirely. The target is a 3-minute PR, with 2 as the stretch. Fixing the tests: happy-dom Vitest gives every test file a fake browser, and the default is jsdom. jsdom aims to be a faithful browser, and that includes a real CSS engine. Tamagui injects a lot of styles on first render, and jsdom parses and applies all of them. happy-dom is a lighter DOM built for tests that does far less of that work. The switch was one line in vitest.config.ts, plus a stub for navigator.clipboard in two test files, because happy-dom's clipboard behaves differently. I also told Vite to pre-bundle Tamagui (deps.optimizer.web) so each file doesn't transform the same package again, and aliased react-native to react-native-web so that pre-bundle resolves. The clearest single number: LimitExceeded's first test took 2,078 ms on jsdom and 48 ms on happy-dom. Whole suite, 521 tests, median of 3 runs on the same M3 Pro. Every run passed all 521: vitest, 521 tests, seconds (M3 Pro) | | jsdom | happy-dom | happy-dom + Tamagui pre-bundle | |---|---|---|---| | Node 22 | 30.3s | 19.5s | 16.5s | | Node 24 | 27.4s | 15.0s | 13.7s | | Node 26 | 27.4s | 14.7s | 12.4s | From where CI was (Node 22, jsdom) to where the branch is (Node 24, happy-dom, pre-bundle), the suite went from 30.3 to 13.7 seconds, 55% less. About two thirds of that is happy-dom. Node 24 and the pre-bundle split the rest. Node 26 is a little faster again, but it isn't LTS yet, so CI moves to 24. Still that is my laptop's number. On CI, with 2 vCPUs instead of 12, the same suite took 3.9 minutes. On the PR (#174) the vitest step took 106 seconds, down from the 233 to 244 range on the last three runs on main. That's 55% less, the same share as on the laptop, which makes sense: the work that went away is per-file setup, and every file pays it no matter how many cores there are. Collect is still the biggest phase at 60 seconds, so it's what to work on next. While I was in there I removed some older dependencies that nothing imported anymore. I ensured the type check, lint, the full suite, and the production build all pass without them. Swapping the frontend toolchain With the tests on a better footing, I wanted to know what the rest of the frontend job would cost with faster tools. Three candidates: TypeScript 7 (the Go port of the compiler), bun check (a new type checker in Bun canary, shipping in 1.4.3), and the Oxc tools, oxlint and oxfmt, for lint and format. Biome is in the chart as the other Rust option. Every number below is the median of 5 runs on an M3 Pro with 12 cores, against the same 246 files in the web app. "Cold" means every .tsbuildinfo deleted first. The 2-thread bars are my best stand-in for a 2 vCPU hosted runner, not a measurement from one. Frontend toolchain, seconds (M3 Pro, 246 files) | | 12 cores | 2 threads | |---|---|---| | Type check | | tsc 6.0.2 -b (cold) | 12.04s | none | | tsc 6.0.2 -b (warm) | 6.73s | none | | tsc 7.0.2 -b (cold) | 1.86s | 2.13s | | tsc 7.0.2 -b (warm) | 1.16s | none | | bun check -b (canary) | 1.10s | 1.87s | | Lint | | ESLint (current config) | 4.44s | none | | ESLint (project off) | 2.42s | none | | oxlint | 0.08s | 0.09s | | oxlint --type-aware | 0.40s | none | | Biome (recommended rules) | 0.14s | none | | Format check | | Prettier 3.2.5 | 2.26s | none | | oxfmt | 0.08s | 0.10s | | Biome | 0.14s | none | _Median of 5 runs. tsc 6 is single-threaded, so it has no 2-thread number. Blank 2-thread cells were not measured._ Before trusting a faster type checker I wanted to know it reports the same errors. On our config all three report zero. That proves less than it sounds, so I turned on --strict for the web app, which produces 12 errors. tsc 6, tsc 7, and bun check reported the same 12, at the same file, line, column, and error code. Bun's issue tracker has a run of open false-positive reports against bun check from this week, and none of them hit this codebase. That's one codebase, so I'm treating it as "works for us," not "works." ESLint was paying for type information it never used. The config sets parserOptions.project, which makes typescript-eslint build a full TypeScript program, but it only enables tseslint.configs.recommended, which has no type-aware rules. Turning project off cut lint from 4.4 to 2.4 seconds and changed nothing it reports. Finally got my quick win with a one-line fix that needs no new tool or testing! oxlint is about 50 times faster than ESLint here. @oxlint/migrate converted 83 rules from the existing config. The one that matters and didn't come across is import/order: Oxc moved import sorting into the formatter. On a probe file with known problems, oxlint caught everything ESLint did except an unused React import. So my one line fix was now out the door again, we can see that quick wins and CI/CD are just not meshing well. The format check wasn't enforcing anything. Prettier says 199 of 246 files aren't formatted. Prettier only ran from a pre-commit hook, and that hook listed its file types under types, which pre-commit ANDs together. No file is JSON and CSS at the same time, so the hook never matched a single file. Nobody noticed because a hook that matches nothing passes. Here's how much of the CI job these steps account for. The main column is the last three successful runs before the change. The PR column is the first run of #174, with happy-dom, Node 24, tsc 7, oxlint, and oxfmt all in: Frontend Check on CI, seconds | | main | PR #174 | |---|---|---| | Install | 14 to 16s | 23s | | Type check | 21 to 23s | 6s | | Tests | 233 to 244s | 106s | | Lint | 13 to 14s | 1s | | Format check | none | 0.1s | | Frontend Check job | 323s | 162s | _main is tsc 6, jsdom and ESLint, with no format check; ranges span the last three runs. PR #174 is tsc 7, happy-dom, oxlint and oxfmt._ The job went from 5.4 minutes to 2.7, half the time. Install is slower because the lockfile changed and the pnpm cache missed. It should be back to about 15 seconds once main saves a new cache. On a 2 vCPU runner tsc 7 ran in 6 seconds against tsc 6's 22. oxlint reported "Finished in 106ms on 246 files using 2 threads." Type check and lint went from 35 seconds to 7, so the toolchain saved about half a minute. The biggest win was the tests saving more than two minutes. The toolchain swap is worth it for the local loop, where tsc -b dropping from 12 seconds to under 2 changes how often I run it. The PR got faster because of the tests. So did the PR, this time. All of #174's checks started at 17:15:00, and the last one finished at 17:19:13: 4.2 minutes, down from about 5.1. Frontend Check finished fourth, behind Go Lint at 3.6 minutes and e2e at 4.2. That's the lesson from the first half of this post working the other way around. Cutting the frontend job in half bought 0.9 minutes, because that's how far ahead of e2e it was. e2e was now our critical path now, and its cache fix is sitting in #173. Where the faster tools do pay off: - The local loop. Type check, lint, and format check together drop from about 19 seconds cold (12.0 + 4.4 + 2.3) to about 2 (1.9 + 0.1 + 0.1). That's the difference between running all three on every save and running them before a push. - Pre-commit. The hook runs ESLint over the whole web app on every commit. At 0.1 seconds instead of 4.4 it stops being the hook people skip, and the format check can actually be enforced. - The CI job shape. Plan item 3 was to move lint and type check into their own job so they run in parallel with tests. At 2 to 3 seconds combined there's no point paying for a second runner and a second pnpm install. They can stay in the test job and add almost nothing. What I switched to oxlint for lint, oxfmt for format, and tsc 7 for type checking. bun check was the fastest checker, but it's a canary build with open false-positive bugs, and it's under a second ahead of tsc 7. It gets another look once 1.4.3 is stable. Dropping ESLint is what made tsc 7 easy. TypeScript 7 doesn't ship the compiler's JavaScript API until 7.1, and typescript-eslint is built on that API, so keeping ESLint would have meant installing TypeScript 6 next to 7. With ESLint gone, one package still needs the API: packages/api generates its clients with openapi-typescript, which builds its output with the TypeScript compiler's factory functions. That one package stays on TypeScript 6. Everything else is on 7. The switch is six commits: 1. Replace ESLint with oxlint, using the config @oxlint/migrate produced. CI goes from a stale 421-warning ratchet to --deny-warnings. 2. Replace Prettier with oxfmt. oxfmt --migrate=prettier carried over every option except the import-sorting plugin, and sortImports with one custom group for react reproduces the old order: React, third party, @/, relative. 3. Reformat apps/web/src. 197 files, mechanical, nothing else in the commit. 4. Add that commit to .git-blame-ignore-revs, so git blame and GitHub's blame view skip it. 5. Move to TypeScript 7, except packages/api. 6. Fix two lint suppressions the reformat broke. The last one is worth knowing about if you do this. Two components had a one-line effect with a suppression above it: // eslint-disable-next-line react-hooks/exhaustive-deps useEffect(() => { refreshAll(); }, []); The formatter split that into three lines. The warning is reported on the dependency array, which was now two lines below the comment, so the suppression stopped applying. Tests passed and type checks passed. --deny-warnings is the only thing that caught it, which is a decent argument for making lint warnings fail the build. To check the new gates actually fail, I added a file with an any and an unused variable, and another with unsorted imports. oxlint and oxfmt each exited non-zero, and both went back to zero once the files were deleted. e2e: one stage removed, one cache borrowed With #173 merged, e2e was the slowest check now. We could reuse some of what we learned and apply two changes got it to about 2. Dropping the scan stage The last stage of the e2e pipeline ran Grype against the Docker image the test had just built, and failed on any high-severity finding. Grype is a vulnerability scanner that checks an image's packages against a CVE database. In the warm #173 run, that stage cost about 41 seconds: 31 to restore the 462 MB database cache, 4 to install Grype, and 6 for the scan itself. I asked what the stage was actually testing, and the answer was "nothing in this PR." It called the Grype CLI directly from bash. No product code ran. The image it scanned is a generated MCP server on oven/bun:1-alpine, so the result depended on whether Alpine had an unpatched CVE that day. Our exceptions file had two entries for exactly that: OpenSSL and zlib, no fix available upstream, and main failing the same way as every PR. The feature, which is Statio scanning images customers push and refusing to deploy one with critical findings, is tested elsewhere. Go unit tests feed the scan pipeline fake Grype reports and check that a critical blocks the deploy. Those run in seconds, need no database, and give the same answer every day. So scan became an opt-in stage, like the two registry stages already were. The first run after that took 2m4s. A cache no PR could ever restore Then #174's e2e took 3m59s. The log said: Cache not found for input keys: go-Linux-e2e-a6a22cc... The cause is how GitHub scopes caches. A workflow run can restore caches saved by its own branch or by the base branch, main. It can't see caches from other PRs. ci.yml runs on pushes to main, so its jobs always have a main-scoped cache for a new PR to start from. The e2e workflow only ran on pull_request. Each PR saved a cache that only that PR could use, so re-runs on the same branch were warm, but the first run on every new PR compiled Go from scratch. The deploy stage alone went from 46 seconds to 1m43s. My first fix was to run e2e on pushes to main as well, so main would save a cache. It works, and it's the wrong trade. It adds a full 4-minute e2e run to every merge just to produce a cache, and that cache holds packages another job had already compiled. Go Build & Test runs go build ./... and go test ./... on every push to main. The repo is a single Go module, so that job's cache already contains every package e2e compiles. It's 558 MB and keyed on the same go.sum hash. So e2e now restores it without saving anything: - name: Go cache (restore from ci.yml go job) uses: actions/cache/restore@v4 with: path: | ~/.cache/go-build ~/go/pkg/mod key: go-${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}- restore-keys: | go-${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}- go-${{ runner.os }}-go- That looks like it breaks the rule from bug two: give every job its own cache. It doesn't, and the reason is the rule exists because jobs that share a key also race to _save_ it, and the fastest one overwrites everyone else's with 8 KB. actions/cache/restore is the restore half of the action with no post step. It can never write, so it can't race and it can't pollute anything. It also skips the 14 seconds e2e used to spend uploading its own cache at the end. e2e, cold vs borrowed cache | | #174 cold | #176 borrowed cache | |---|---|---| | gen stage | 24s | 1s | | deploy stage | 1m43s | 43s | | Pipeline step | 3m9s | 1m23s | | e2e total | 3m59s | 2m9s | That was the first run of #176, on a branch that had never run e2e. Restoring 558 MB takes about 19 seconds. That's the biggest fixed cost left in the job, and it buys back about 1m40s of compiling (gen + deploy vs restore cache). When to borrow a cache read-only: the other job builds a superset of what you build, with the same flags, and it runs on your default branch. When not to: if your job compiles things the other job doesn't, like a different module, different build tags, or -race. A restore-only job never saves, so anything it compiles that isn't in the borrowed cache gets compiled again on every run. A cache key that hashed nothing While I was in that file, the Bun cache logged its key as bun-Linux-. That key was built from hashFiles('**/bun.lock'), and hashFiles returns an empty string when nothing matches. The Bun project in e2e is the MCP server the test generates at runtime, so there's no lockfile in the repo for it to hash. The key never changed, and the bun install it was meant to speed up takes 3 seconds. I removed it. Two days later: 2.5 minutes After #174 and #176, the first PR run took 4.2 minutes. Two more PRs went in that evening, one for lint and one for the cache budget. Here's every PR run since, 125 of them, against each job's worst median from the week before: Medians, worst this week vs now | | worst | now | |---|---|---| | PR, open to all green | 5.3m | 2.5m | | Go Lint | 6.0m | 1.4m | | Go Build & Test | 5.2m | 1.8m | | Frontend Check | 5.4m | 2.4m | | e2e | 4.8m | 2.3m | _"Worst" is each job's highest median during the week: after #166 for the Go jobs, the last main runs before #174 for Frontend Check, before #173 for e2e. "Now" is successful PR runs from October 7 evening to October 9._ The PR number is the time from the first job starting to the last one finishing, first attempts only, on the 44 commits that ran the full set of jobs. Median 2.48 minutes, p90 3.2, worst 3.6. 24 of the 44 finished in under 2.5. The target earlier in this post was 3 minutes with 2 as the stretch. The median is past the target and the p90 is close to it. Profiling the linter (#177) Go Lint was still the job with skip-cache: true from #124, so every run did a cold analysis of the whole tree. Before deciding what to cut, I ran golangci-lint with --cpu-profile-path on a cold full-tree run and read the profile. golangci-lint runs every analyzer inside one pass, so pprof -top lumps them together. Summing the profile by analyzer gives this: golangci-lint CPU by analyzer, cold full tree, seconds | | flat CPU | |---|---| | SSA/IR build (shared) | 56.1s | | wrapcheck | 16.6s | | gosec | 13.8s | | whitespace | 9.4s | | revive | 4.6s | | staticcheck | 3.5s | | unused | 2.8s | _The SSA/IR build is shared by staticcheck, unused, gosec and nilness. errcheck, goconst, dupl, musttag and exhaustruct were all near zero._ What I found was that the biggest cost isn't a linter. It's building the SSA form of the program, and four linters share it, so disabling any one of them saves nothing, it was determined that those checks were needed so we tried to take a quick look to see if we could speed this one up a bit as well. The analysis cache is back on. The reason it was off was nolintlint flipping between "this directive is required" and "this directive is unused." The config already had a comment about the same failure from a different cause: golangci-lint caps how many identical issues it reports, and when the issue a //nolint covers gets dropped by that cap, the directive looks unused. max-same-issues: 0 was already in the config to turn that cap off, which is very likely why the flapping stopped reproducing. I checked it rather than assume it: a cold run, then five warm runs and three with a file touched, and all eight reported zero nolintlint issues. That's evidence, not proof, so the workflow comment says what to do if it comes back: put skip-cache: true back, one line. Warm, lint takes 3.1 seconds. One file touched, 7.4. Cold, 36 to 56. Generated code is no longer a lint target. linters.exclusions.paths looked like it was skipping our generated code. It wasn't. It filters _results_, so every analyzer still walked 1.82 million generated lines, 48% of the tree, and the findings were thrown away at the end. The only way to skip that work is to leave those packages off the command line. A cold run at 2 CPUs, the runner's shape, went from 72.5 to 37.5 seconds with identical output. The generated packages still get type-checked, because the code we lint imports them. What goes away is running analyzers over them. whitespace is off. 9.4 seconds to check for stray blank lines inside blocks, which is the formatter's job. Removing that exclusion turned up something else. db/ent was an unanchored path pattern, so it matched more than the generated ent client. It also silenced the 27 hand-written schema files in db/ent/schema/, plus db/ent_client.go and db/ent_test.go, which just happen to contain db/ent in their paths. Both generated directories were already covered by generated: strict, which skips files with a DO NOT EDIT header, so the pattern added nothing for generated code and hid real code. Un-excluding those files surfaced two findings, both fixed in the PR. Go Lint went from 3.2 to 3.6 minutes on the last runs before #177 to a median of 1.4 after it. The golangci-lint step itself is now 27 to 33 seconds. A cache that saves on every PR (#181) The run-keyed cache from #170 has a cost I waved off earlier. Every run saves a new entry, and in this repo each Go cache is about 0.56 GiB. PRs included, that's a lot of half-gigabyte writes. The repo's cache usage was at 12.5 GiB against GitHub's 10 GiB limit, so eviction was running all the time, and it removes the least recently used entries first. Whatever runs least often loses its cache. For us that was the release pipeline, which is its own story and the next post. For PRs, the fact that matters is that every run was paying to upload half a gigabyte that only that branch could ever use. The fix is to restore on every run and save only on pushes to main: - name: Go cache (restore) uses: actions/cache/restore@v4 with: path: | ~/.cache/go-build ~/go/pkg/mod key: go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}- restore-keys: | go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}- go-${{ runner.os }}-${{ github.job }}- # ... the job's Go work ... - name: Go cache (save) if: github.event_name == 'push' uses: actions/cache/save@v4 with: path: | ~/.cache/go-build ~/go/pkg/mod key: go-${{ runner.os }}-${{ github.job }}-${{ hashFiles('**/go.sum') }}-${{ github.run_id }} PRs read and never write. Main writes on every merge, which is often enough to stay warm. That's 5 to 6 times fewer Go cache writes. The trade is that a PR starts from the last merge instead of its own last run, so a re-run on the same branch compiles a little more than it used to. With the branch scoping rules from the e2e section, the first run of a new PR was starting from main's cache anyway. Where the time goes now PR-triggered jobs, median minutes | | Oct 1 to 6 | After #170 | Now | |---|---|---|---| | Go Build & Test | 4.6m | 1.8m | 1.8m | | Go Lint | 5.0m | 3.0m | 1.4m | | Frontend Check | 3.7m | 5.1m | 2.4m | | E2E pipeline | 4.6m | 4.8m | 2.3m | | Grype (slower scan) | 2.2m | 2.3m | 1.8m | Every job now finishes within about a minute of the others. Frontend Check is still the last to finish most often, 25 of 44 runs, then Grype Node 7, e2e 4, Grype Go 3, Go Lint 3, Go Build 2. That spread is what a balanced pipeline looks like. It also means no single fix moves the PR much from here. Cutting a minute off Frontend Check would hand the finish line to e2e or Grype Node, which are within seconds of it. The fixed costs are now bigger than the work. In a recent Go Lint run, the golangci-lint step took 27 to 33 seconds and restoring the Go cache took 34 to 39. Checkout and setup-go add another 16 to 24. Lint is cheaper to run than its cache is to download, and the next round of work is shrinking what each job has to restore. Self-hosted runners I have a Proxmox box at home, and it's tempting. A persistent runner keeps its disk between jobs, so the Go build cache, the lint cache, the Grype DB, and node_modules are always warm, and nothing competes for a 10 GB cache budget. I used to think it was the only way to get the lint cache back. #177 did that on hosted runners. I'm not doing it yet, for two reasons. One box means jobs queue behind each other, and this pipeline is fast because it's parallel. And a self-hosted runner executes whatever code is in the PR on hardware I own, on my network. For a security product that needs a real isolation story (ephemeral VMs per job, no access to the LAN) before it gets anywhere near a pull request. Notes for next time - Find the critical path before optimizing anything. Make a table of which job finished last. That's a 20-minute script and it would have changed what I worked on first. - Check cache sizes, not cache hits."Cache restored" with an 8 KB payload is a miss. - A cache key that can hit never updates. For build output, put something unique in the key and let restore-keys find the newest entry. - Jobs that share a cache key race to save it, and the fastest one wins. Put github.job in the key. - A job that only runs on PRs never has a cache on main. Every new PR starts cold. Borrow a main job's cache read-only rather than adding a main run to seed one. - Print your cache keys. hashFiles on a glob that matches nothing returns an empty string, and the key quietly becomes a constant. - Ask what a CI stage tests. A scan of today's CVE database against a base image tests the database, not the PR. - Profile the linter before tuning it. The biggest cost was a shared SSA build that no single linter owns. Without the profile I'd have disabled staticcheck and saved nothing. - Exclusions filter results, they don't skip work. To stop analyzing generated code, leave it off the lint targets. - Anchor your path patterns. db/ent also matched 27 hand-written files and two that only had db/ent in their names. - A cache that saves on every PR spends the whole repo's budget. Eviction hits the caches that run least often. Save on main, restore everywhere. - Watch the job you aren't working on. Frontend Check doubled in a week from normal feature work. A weekly look at job medians would have caught it before it was the bottleneck. And as a final note, if you go down optimizing your CI pipeline, which I think Github would GREATLY appreciate, don't expect quick wins like I was hoping for! Resources - actions/cache: how cache keys and restore-keys match - actions/cache/restore: restore-only action - actions/cache/save: save-only action - GitHub docs: restrictions for accessing a cache (branch scope) - GitHub docs: caching dependencies, usage limits and eviction - golangci-lint: --new-from-rev and related flags - Grype: database update behavior - Vitest: test sharding - Vitest: fake timers
Optimize Test Execution: Parallelization, Caching & Scheduling - [Why faster test runs are the single biggest lever on lead time] - [How to shard tests and run parallel test runners without breaking things] - [Cache the right layers: dependencies, artifacts, and Docker images that actually save time] - [Schedule smart, retry selectively, and size resources to minimize flake and cost] - [Actionable checklist: implement parallelization, caching, and smart scheduling] Fast CI feedback is the production-quality gatekeeper: every minute you shave off test execution multiplies developer throughput and reduces the blast radius of context-switching. Shorter, predictable test runs keep changes small, reviews fast, and your team in a flow state — that’s measurable business leverage, not just a nice-to-have. Slow, noisy CI looks the same across companies: long PR queues, blocked merges, developers waiting hours for green checks, flaky failures that waste triage time, and runaway cloud costs from inefficient runners. The direct consequences are longer lead time for changes, lower confidence in CI signals, and developer context-switch tax that compounds across teams and sprints. Why faster test runs are the single biggest lever on lead time Shortening test execution time directly reduces the critical path from commit to feedback, which improves your Lead Time for Changes — a core DORA metric tied to business performance. High-performing teams routinely compress that lead time and gain outsized benefits in stability and feature throughput. - Hard-won lesson: reduce the critical path first. That means identify what executes in the PR gate and optimize it before trying to micro-optimize marginal tests. - Measure, then act: collect per-test timings and failure rates for the last N runs — those numbers let you target the top 20% of tests that consume ~80% of runtime. > Important: Parallelization without data turns into wasted cost and flakiness. Use runtime data to balance shards and reserve parallel runs for tests that are actually on the critical path. Table — quick comparison of common sharding strategies | Strategy | Strength | When to use | Major caveat | |---|---:|---|---| | Time-based sharding (historic timings) | Best balanced runtime | Large suites with timing history | Requires reliable historic JUnit/JUnit-like timings. | | File or name-based sharding | Simple to implement | Small-to-medium suites | Can create skewed shards if test durations vary widely. | | Round-robin / modulo by index | Deterministic & cheap | No timing data available | Poor balance for skewed distributions. | | Runner-local parallelism (pytest-xdist, Playwright workers) | Fast, minimal infra setup | When infra is constrained to one machine | Still subject to single-host resource contention. | How to shard tests and run parallel test runners without breaking things Start by classifying tests into fast unit, slow integration, and expensive e2e suites; run different classes with different strategies. Practical sharding patterns - Local parallelism: use a parallel test runner (example: pytest-xdist with pytest -n auto) to split work across CPU cores; this is the lowest-friction speedup for Python tests. Use --dist loadscope or --dist loadfile to reduce fixture reinitialization when needed. - CI-level sharding across machines: use CI platform features to split the suite by time or file lists (CircleCI’s tests split --split-by=timings is an example of timing-based splitting). That produces balanced shards and minimizes tail latency. - Runner matrix / job matrix: use job matrices to create N shards as matrix entries, controlling max-parallel on GitHub Actions or parallel:matrix on GitLab to throttle concurrency and avoid resource overload. Example: balanced test sharding on CircleCI (conceptual) # CircleCI CLI splits using previous timings to create balanced nodes circleci tests glob "tests/**/*_test.py" \ | circleci tests split --split-by=timings --timings-type=name \ | xargs -n 1 -I {} pytest {} CircleCI automatically uses uploaded JUnit/XML timings to compute splits; the first run will be unbalanced but subsequent runs converge. Example: lightweight cross-machine sharder (pattern) # scripts/generate-test-list.sh # output: tests-list.txt (one test per line) # split into N shards (shard index 1..N) python ci/split_tests.py --tests-file tests-list.txt --shard-index $SHARD_INDEX --total-shards $TOTAL # run tests for this shard: xargs -a shard-tests.txt -n1 -P1 pytest -q Provide ci/split_tests.py that reads a timings cache and assigns tests to shards using a greedy bin-packing algorithm (example below). Greedy bin-packing shard script (Python — simplified) # ci/split_tests.py # usage: python ci/split_tests.py --timings timings.json --total 4 --shard-index 1 import json, argparse parser=argparse.ArgumentParser() parser.add_argument('--timings', required=True) parser.add_argument('--total', type=int, required=True) parser.add_argument('--shard-index', type=int, required=True) args=parser.parse_args() times=json.load(open(args.timings)) # {"tests/test_a.py::test_foo": 3.2, ...} items=sorted(times.items(), key=lambda t: -t) bins=[[] for _ in range(args.total)] bin_times=*args.total for name, t in items: i=bin_times.index(min(bin_times)) bins[i].append(name) bin_times[i]+=t shard=bins[args.shard_index-1] print('\n'.join(shard)) Use historic timings for accurate balance; falling back to file-based modulo sharding when no history exists is acceptable short-term. Tooling notes - Use test frameworks’ native parallel features where available (Playwright has --shard and workers options; prefer those for UI/browser tests). - For JVM-based suites, enable JUnit 5’s parallel execution carefully (junit.jupiter.execution.parallel.enabled=true) and use @ResourceLock for shared resources. Verify thread-safety first. Cache the right layers: dependencies, artifacts, and Docker images that actually save time Caching is low-hanging fruit, but frequently misused. Cache what’s expensive to resolve and cheap to restore; avoid caching huge folders that cost more to download than to rebuild. Best-practice cache targets - Language package managers: ~/.cache/pip, ~/.m2/repository, node_modules (with caution). Use lockfile-hash keys to invalidate when dependencies change. GitHub’s actions/cache is the canonical tool on Actions. - Build artifacts: compiled assets, prebuilt binaries, compiled TypeScript artifacts. - Docker layer cache: use BuildKit to persist/export caches between runs (--cache-to / --cache-from) or use registry-backed build cache to avoid re-executing unchanged layers. That speeds repeated image builds dramatically when the Dockerfile is structured for layer reuse. Example: GitHub Actions caching for Python dependencies # .github/workflows/ci.yml (excerpt) - uses: actions/checkout@v4 - name: Cache pip uses: actions/cache@v4 id: pip-cache with: path: ~/.cache/pip key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} - name: Install if: steps.pip-cache.outputs.cache-hit != 'true' run: pip install -r requirements.txt Use cache-hit to skip install steps when a strong cache hit occurs. Be mindful of cache size limits and eviction policies. Example: BuildKit Dockerfile cache mounts (fast image builds) # syntax=docker/dockerfile:1.4 FROM python:3.11-slim WORKDIR /app COPY pyproject.toml poetry.lock ./ RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt COPY . . CMD ["pytest"] BuildKit’s --mount=type=cache preserves pip cache directories across builds without polluting your image, and BuildKit can export/import caches to registries for CI reuse. Cache nuanced rules - Use content-based keys (hash of lockfile + build-tool version) — avoid raw timestamps. - Don’t cache ephemeral files or caches that are faster to re-create (e.g., on some shared runners downloading small packages may be faster than restoring large caches). - Keep caches scoped narrowly (per-language or per-build-step) to avoid unnecessary invalidations and heavy downloads. Schedule smart, retry selectively, and size resources to minimize flake and cost Parallelization and caching cut time — scheduling and retries keep pipelines healthy and trustworthy. Smart scheduling patterns - Gate with small, fast checks: run lint + unit + smoke in the PR gate; run heavy integration and E2E suites on main or nightly. That keeps PR feedback fast while preserving full coverage on merges. - Prioritize critical tests: schedule fast & high-signal tests first; use --failed-first or --last-failed modes where supported so failing tests surface earlier. (pytest supports --lf and --ff modes.) - Isolate resource-sensitive tests: run DB-intensive or flaky network tests on dedicated runners or in serial to avoid noisy neighbors. Retries and flake mitigation - Automatic retries reduce noise from transient infra failures; configure them conservatively. GitLab’s retry lets you limit retries and restrict them to runner/system failures rather than application failures. Use job-level retries to cover infra blips, not test logic errors. - Re-run failing tests selectively: rerun only failed tests a small number of times (pytest-rerunfailures or CI-based rerun tools) to avoid masking real regressions but reduce noise. - Quarantine and triage: detect high-flakiness tests (by frequency and owner) and move them off the blocking path while opening tickets to fix them; Google uses automated quarantining and flakiness dashboards in large fleets. Resource sizing & cost control - Autoscale runners for peak concurrency, and scale down at night — use spot/spot-like instances when acceptable to save cost. - Cap per-job concurrency (strategy.max-parallel in GitHub Actions or parallelism / resource class in CircleCI) to avoid overloading test infra and artificially increasing flakiness. - For browser tests, Playwright recommends limiting worker count in CI and using multiple sharded jobs for parallelism across machines rather than over-subscription on a single host. Operational example: conservative retry policy (GitLab) test: script: - pytest -q retry: max: 1 when: - runner_system_failure This retries only for runner/system failures and limits retries to 1 to avoid hiding test logic problems. Actionable checklist: implement parallelization, caching, and smart scheduling Use this stepwise protocol on a single service or repository; treat it like an experiment — measure before and after. 1. Measure baseline (week 0) - Collect PR median/95th CI time-to-green and per-test runtimes from the last 14–30 runs. - Identify top 20% slow tests and top 10% flakiest tests. 2. Target the critical path (week 1) - Move fastest, highest-signal tests into the PR gate (lint, unit, smoke). - Move expensive E2E/integration tests to merge/train runs or nightly. 3. Add fast wins: caching (days 1–2) - Add actions/cache / GitLab cache: for package managers with keys based on lockfile hash. Validate cache-hit logic to skip installs. - Convert Docker builds to BuildKit and add --mount=type=cache entries for language caches; export cache to registry for cross-run reuse. 4. Add measured parallelism (days 2–7) - Implement pytest -n auto for local parallelism on powerful runners; confirm test independence. - Add CI-level sharding for heavy suites using timing-based splits (CircleCI) or matrix shards (GitHub/GitLab) with max-parallel control. - Use a greedy sharder (example ci/split_tests.py) fed by historic timings to balance shards. 5. Harden flakiness and retries (week 2) - Configure conservative job retries for infra failures only (retry on GitLab). - Use pytest-rerunfailures or CI rerun actions to re-run failing tests a small number of times; track re-run success rate. - Quarantine highest-flake tests and create triage tickets with owners; track metrics and remove from quarantine only after validation. 6. Iterate and optimize (ongoing) - Track PR median/95th time-to-green after each change. - Watch for cost-per-minute trends; increase parallelism only when it reduces wall-clock time proportionally and preserves signal quality. - Automate shard rebalancing when timing data drifts; rebuild caches strategically (not every run). Example CI snippet: GitHub Actions matrix shards + caching name: CI on: [push, pull_request] jobs: tests: runs-on: ubuntu-latest strategy: matrix: shard: [1,2,3,4] max-parallel: 4 steps: - uses: actions/checkout@v4 - name: Cache pip uses: actions/cache@v4 with: path: ~/.cache/pip key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} - name: Install if: steps.cache.outputs.cache-hit != 'true' run: pip install -r requirements.txt - name: Generate shard test list run: python ci/split_tests.py --timings ci/timings.json --total 4 --shard-index ${{ matrix.shard }} > shard-tests.txt - name: Run tests run: xargs -a shard-tests.txt -n1 pytest -q This pattern keeps caching deterministic and uses a timing-based sharder to balance wall-clock time. Sources: Accelerate State of DevOps 2021 - Benchmarks and evidence linking lead time for changes and delivery performance; used to justify why CI speed matters and the impact of lead time improvements. CircleCI: Test splitting and parallelism - Explanation of timing-based test splitting and examples for balanced shards; used for sharding strategies and CLI-based splitting examples. pytest-xdist documentation - Details on pytest -n auto, distribution modes (--dist), and options for worker behavior; used for local parallel runner guidance. actions/cache GitHub action (actions/cache) - Official docs for caching dependencies in GitHub Actions, cache-key strategies, and cache-hit usage; used for caching patterns. Docker BuildKit documentation - BuildKit features, cache mounts, and --cache-to/--cache-from concepts for Docker caching in CI. Google Testing Blog — Flaky Tests at Google and How We Mitigate Them - Industry-scale observations and mitigation tactics for flaky tests; used to justify quarantine, re-runs, and flake dashboards. JUnit 5 User Guide — Parallel Execution - How to enable and configure parallel execution in JUnit 5 and synchronization mechanisms; used for JVM guidance. GitHub Actions: Running variations of jobs in a workflow (matrix) - Matrix strategies, max-parallel, and failure handling for GitHub Actions; used for matrix-based sharding patterns. GitLab CI/CD parallel:matrix documentation - GitLab’s parallel:matrix syntax and behavior for spawning parallel job permutations; used for GitLab sharding examples. GitLab CI retry job keyword documentation - Configuring job retries and controlling when to retry (runner/system failures vs. script failures); used for conservative retry recommendations. Playwright Test — Parallelism and Sharding - workers, --shard, and Playwright’s recommendations for CI worker sizing and sharding; used for browser test best practices.
@ShamashAran And then there’s this where a core part of a platforms service is technically in maintenance mode, still accepts PRs, and just does nothing with them cache#1749