bazelbuild

@xsphi @sakompella you can get Bazel to offload builds to a persistent rustc worker to preserve build incrementally at a sub-crate level: rules_rust#4184

Buildbarn rejecting an execution property? Check the Bazel version and the exact key. Bazel PR #32010 separates client-only requirements from remote platform properties. Merged does not mean released: verify your version before assuming the fix applies. bazel#32010

@SuchirKavi @theodorvaryag @bazelbuild @buildbuddy @tweagio @IntuitiveSurg Lean heavily into bazelbuild/bazel-central-registry and ask the agents to break down the tasks. It's hard to create a generic skill for it because sometimes it's domain-specific.

Build-as-Code, CI Integration, and Build Doctor - Why treat builds as code: eliminate drift and make builds a pure function - CI integration patterns for hermetic builds and remote cache clients - Designing and implementing a Build Doctor diagnostics tool - Rolling out at scale: onboarding, guardrails, and measuring impact - Practical checklists and runbooks for immediate action Treat every build flag, toolchain pin, and cache policy as versioned code — not a local habit. Doing so converts the build from a mutable ritual into a repeatable, auditable function whose outputs are pure and sharable. The pain is specific: slow pull requests because CI redoes work, “works on my machine” debugging, cache-poisoning incidents that invalidate hours of developer effort, and onboarding that takes days because local setups differ. Those symptoms trace back to one root cause: build affordances (flags, toolchains, cache policy, and CI integration) live as handwaves instead of code, so behavior diverges between machines and pipelines. Why treat builds as code: eliminate drift and make builds a pure function Treating the build as code — build-as-code — means storing every decision that influences outputs in version control: WORKSPACE pins, BUILD rules, toolchain stanzas, .bazelrc snippets, CI bazel flags, and the remote-cache client configuration. That discipline enforces hermeticity: the build result is independent of the host machine and therefore reproducible across developer laptops and CI servers. What you get when you do this correctly: - Bit-identical artifacts for the same inputs, eliminating “it works on my machine” debugging. - A cacheable DAG: actions become pure functions of declared inputs, so results can be reused across machines. - Safe experimentation via branches: differing toolchain or flag sets are explicit commits, not env leaks. Practical rails that make this discipline enforceable: - Keep a repo-level .bazelrc that defines the canonical flags used in CI and for canonical local runs (build --remote_cache=..., build --host_force_python=...). - Pin toolchains and third-party deps in WORKSPACE with exact commits or SHA256 checksums. - Treat ci and local modes as two configurations in the build-as-code model; only one (CI) should be allowed to write authoritative cache entries in the early rollout phase. > Important: Hermeticity is an engineering property you can test for; make those tests part of CI so the repository encodes the build’s contract rather than relying on implicit conventions. CI integration patterns for hermetic builds and remote cache clients The CI layer is the single most potent lever for accelerating team builds and protecting the cache. There are three practical patterns you will choose from depending on scale and trust. - CI-as-single-writer, developers-read-only: CI builds (full, canonical builds) write to the remote cache; developer machines read only. This prevents accidental cache poisoning and keeps the authoritative cache consistent. - Combined local + remote cache: Developers use a local disk cache plus a shared remote cache. The local cache improves cold-starts and avoids unnecessary network trips; the remote cache enables cross-machine reuse. - Remote execution (RBE) for speed at scale: CI and some dev flows offload heavy actions to RBE workers and take advantage of both remote execution and the shared CAS. Bazel exposes standard knobs for these patterns; the remote cache stores action metadata and the content-addressable store of outputs, and a build consults the cache before running actions. Example .bazelrc snippets (repo-level vs CI): # .bazelrc (repo - canonical flags) build --remote_cache=grpcs://cache.corp.example:9090 build --remote_download_outputs=minimal build --host_jvm_args=-Xmx2g build --show_progress_rate_limit=30 # .bazelrc.ci (CI-only overrides; kept on CI runner) build --remote_cache=grpcs://cache.corp.example:9090 build --remote_executor=grpcs://rbe.corp.example:8989 build --remote_timeout=180s build --bes_backend=grpcs://bep.corp.example # send BEP to analysis UI CI example (GitHub Actions, illustrating integration with existing cache steps): use the platform cache for language deps and let Bazel use the remote cache for build outputs. The actions/cache action is a common helper for pre-built dependency caches. name: ci on: [push, pull_request] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Restore tool caches uses: actions/cache@v4 with: path: ~/.cache/bazel key: ${{ runner.os }}-bazel-${{ hashFiles('**/WORKSPACE') }} - name: Bazel build (CI canonical) run: bazel build --bazelrc=.bazelrc.ci //... Contrast of caching approaches | Mode | What it shares | Latency impact | Infra complexity | | --- | ---: | --- | --- | | Local disk cache | per-host artifacts | small improvement, not shared | low | | Shared remote cache (HTTP/gRPC) | CAS + action metadata | network-limited, large benefit across team | medium | | Remote execution (RE) | executes actions remotely | minimizes dev wall-clock time | high (workers, auth, scheduling) | Remote execution and remote caching are complementary; RBE focuses on compute scaling while the cache focuses on reuse. The protocol landscape and client/server implementations (e.g., the Bazel Remote Execution APIs) are standardized and supported by several OSS and commercial offerings. Practical CI guardrails to enforce: - Make CI the canonical writer during pilot: developer configs set --remote_upload_local_results=false while CI sets it true. - Lock who can clear the cache and implement a cache-poison rollback plan. - Send BEP (Build Event Protocol) from CI builds to a centralized invocations UI for later troubleshooting and historical metrics. Tools like BuildBuddy ingest BEP and provide cache-hit breakdowns. Designing and implementing a Build Doctor diagnostics tool What a Build Doctor does - Acts like a deterministic, fast diagnostics agent that runs locally and in CI to surface misconfigurations and non-hermetic actions. - Collects structured evidence (Bazel info, BEP, aquery/cquery, profile traces) and returns actionable findings (missing --remote_cache, genrule calling curl, actions with nondeterministic outputs). - Produces machine-readable results (JSON), human-friendly reports, and CI annotations for PRs. Data sources and commands to use - bazel info for environment and output base. - bazel aquery --output=jsonproto 'deps(//my:target)' to retrieve action commandlines and inputs programmatically. This output can be scanned for rogue network calls, writes outside the declared outputs, and suspicious commandline flags. - bazel build --profile=command.profile.gz //... followed by bazel analyze-profile command.profile.gz to get the critical path and per-action durations; the JSON trace profile can be loaded into tracing UIs for deeper analysis. - Build Event Protocol (BEP) / --bes_results_url to stream invocation metadata to a server for long-term analytics. BuildBuddy and similar platforms provide BEP ingestion and a UI for cache-hit debugging. Minimal Build Doctor architecture (three components) 1. Collector — shell or agent that runs the Bazel commands and writes structured files: - bazel info --show_make_env -> doctor/info.json - bazel aquery --output=jsonproto ... -> doctor/aquery.json - bazel build --profile=doctor.prof //... -> doctor/command.profile.gz - optional: fetch BEP or remote cache server logs 2. Analyzer — Python/Go service that: - Parses aquery for suspicious mnemonics or commands (Genrule, ctx.execute) that contain network tools. - Runs bazel analyze-profile doctor.prof and correlates long actions with aquery outputs. - Verifies .bazelrc flags and remote cache client presence. 3. Reporter — emits: - a concise human report - structured JSON for CI pass/fail gating - annotations for PRs (failed hermetic checks, top 5 critical-path actions) Example: a tiny Build Doctor check in Python (skeleton) #!/usr/bin/env python3 import json, subprocess, sys, gzip def run(cmd): print("+", " ".join(cmd)) return subprocess.check_output(cmd).decode() def check_remote_cache(): info = run(["bazel", "info", "--show_make_env"]) if "remote_cache" not in info: return {"ok": False, "msg": "No remote_cache configured in bazel info"} return {"ok": True} def parse_aquery_json(path): with open(path,'rb') as f: return json.load(f) def main(): run(["bazel","aquery","--output=jsonproto","deps(//...)", "--include_commandline=false", "--noshow_progress"]) # analyzer steps would follow... print(json.dumps({"checks":[check_remote_cache()]})) if __name__ == '__main__': main() Diagnostic heuristics you should encode (examples) - Actions whose command lines contain curl, wget, scp, or ssh indicate network access and likely non-hermetic behavior. - Actions that write to $(WORKSPACE) or outside declared outputs indicate source-tree mutation. - Targets tagged no-cache or no-remote deserve review; frequent no-cache usage is a smell. - bazel build outputs that differ across repeated clean runs reveal nondeterminism (timestamps, randomness in build steps). A Build Doctor should avoid hard fails on first rollout. Start with informational severities and escalate rules to warnings and hard-gate checks as confidence grows. Rolling out at scale: onboarding, guardrails, and measuring impact Rollout phases 1. Pilot (2–4 teams): CI writes to cache, developers use read-only cache settings. Run Build Doctor in CI and as a local dev hook. 2. Expand (6–8 weeks): Add more teams, tune heuristics, add tests that detect cache-poisoning patterns. 3. Org-wide: Make CANONICAL .bazelrc and toolchain pins required, add PR checks, and open the cache for a broader set of write clients. Key metrics to instrument and track - P95 build/test times for common developer flows (changes to a single package, full test runs). - Remote cache hit rate: percent of actions served from the remote cache vs executed. Track daily and by repository. Aim high; a >90% hit rate on incremental builds is a realistic, high-leverage target for mature setups. - Time to first successful build (new hire): measure from checkout to successful test run. - Number of hermeticity regressions: count CI-detected non-hermetic checks per week. How to collect these metrics - Use CI BEP exports to compute cache-hit ratios. Bazel prints per-invocation process summaries that indicate remote cache hits; programmatic BEP ingestion gives more reliable metrics. - Push derived metrics to a telemetry system (Prometheus / Datadog) and create dashboards: - Histogram of build times (for P50/P95) - Time-series of remote cache hit rate - Weekly count of Build Doctor violations per team Guardrails and change control - Use a cache-write role: only designated CI runners (and a small set of trusted service accounts) can write to the authoritative cache. - Add a cache clear and rollback playbook to respond to cache poisoning: snapshot the cache state and restore from a pre-poisoned snapshot if necessary. - Gate merges with Build Doctor findings: start with warnings and move to hard failure for core rules once false positives are low. Developer onboarding - Ship a developer start.sh that sets up the repo-level .bazelrc and installs bazelisk to pin bazel versions. - Provide one-page runbook: git clone ... && ./start.sh && bazel build //:all --profile=./first.profile.gz so new hires produce a baseline profile that CI can compare to. - Add a lightweight VSCode/IDE recipe that reuses the same repo-level flags so the dev environment mirrors CI. Practical checklists and runbooks for immediate action Baseline measurement (week 0) 1. Run a canonical CI build for the main branch for seven consecutive runs and collect: - bazel build --profile=ci.prof //... - BEP exports (--bes_results_url or --build_event_json_file) 2. Compute baseline P95 build times and cache-hit rate from BEP/CI logs. Setup remote cache and clients (week 1) 1. Deploy a cache (e.g., bazel-remote, Buildbarn, or managed service). 2. Put canonical flags into repo .bazelrc and a CI-only .bazelrc.ci. 3. Configure CI to be the primary writer; developers set --remote_upload_local_results=false in their per-user bazelrc. Ship the Build Doctor (week 2) 1. Add collector hooks to CI to capture aquery, profile, and BEP. 2. Run Analyzer on CI invocations; surface findings as PR comments and nightly reports. 3. Begin triage for top findings (e.g., genrules with network calls, nonhermetic toolchains). Pilot & expand (weeks 3–8) 1. Pilot with three squads and run Build Doctor in PRs as info-only. 2. Iterate on heuristics and reduce false positives. 3. Convert high-confidence checks into gating rules. Runbook snippet: respond to a cache-poison incident - Step 1: Identify corrupted outputs via BEP and Build Doctor reports. - Step 2: Quarantine suspect cache prefixes and flip CI to write to a fresh cache namespace. - Step 3: Roll back to last-known-good cache snapshot and re-run canonical CI builds to repopulate. > Quick rule: make CI the source of truth for cache writes during rollout and keep destructive cache administration actions auditable. Sources Hermeticity | Bazel - Definition of hermetic builds, benefits, and guidance on identifying non-hermetic behavior. Remote Caching - Bazel Documentation - How Bazel stores action metadata and CAS blobs, flags like --remote_cache and --remote_download_outputs, and disk-cache options. bazelbuild/remote-apis (GitHub) - The Remote Execution API specification and list of clients/servers implementing the protocol. JSON Trace Profile | Bazel - --profile, bazel analyze-profile, and how to generate and inspect JSON trace profiles for critical-path analysis. buildbuddy-io/buildbuddy (GitHub) - An example BEP and remote-cache ingestion solution that demonstrates how build event data and cache metrics can be surfaced to teams. actions/cache (GitHub) - GitHub Actions cache action documentation and guidance for dependency caching in CI workflows. The Bazel Query Reference / aquery - aquery/cquery usage and --output=jsonproto for machine-readable action graph inspection. Treat the build as code, make CI the canonical actor for cache writes, and ship a Build Doctor that codifies the heuristics you already reach for in the hallway — those operational moves convert day-to-day build firefighting into measurable, automatable engineering work.

@corentinanjuna it was related to strengthening this bazel#23167 into an enforced invariant, and using the existing DO_NOT_BUILD_HERE or a new pointer to track all workspace->output root mappings. and then a ui command to gc, similar to nix-store gc

Faster Chromium builds on Linux: local settings and REAPI backend options Disclosure: I maintain expbuild, a Kubernetes platform for managing build caches. For anyone tuning Chromium build times, here’s a short starting point based on the upstream docs. From a checkout with dependencies installed: gn args out/FastDev For development builds where full debugging information isn’t needed: is_debug = true is_component_build = true symbol_level = 1 # Optional, if you aren't debugging these components: blink_symbol_level = 0 v8_symbol_level = 0 Component builds help with incremental linking. Reduced symbols save work but limit debugging. Build with: autoninja -C out/FastDev chrome Adding remote execution The current documented Chromium path uses Siso: use_siso = true use_remoteexec = true You still need backend configuration and authentication. For a non-Google service, the setup uses .gclient settings for reapi_address, reapi_instance, and reapi_backend_config_path, plus a backend-specific backend.star. Remove an old use_reclient = true setting when following this path. See the official setup guide, and check your branch. Which server can you use? REAPI execution backends include: - Buildbarn - Buildfarm - BuildGrid - BuildBuddy - EngFlow These appear in the REAPI implementation list. Protocol support is a starting point; worker environments, platform properties, and authentication still need to match Chromium’s requirements. Where expbuild fits expbuild manages cache services on Kubernetes, including a bazel-remote-based REAPI cache. That service provides caching, not execution workers, so it cannot replace the execution backend above. Chromium/Siso integration with expbuild is not yet validated; contributions and testing in this area are welcome. For performance comparisons, I’d measure clean builds and rebuilds after a source change separately. An unchanged second build may simply skip all work. What backend are you using for Chromium, and what made the biggest difference—linking, cache reuse, or remote execution?

Configuring AOSP with reclient: a few RBE details worth checking I’ve been looking through AOSP’s reclient integration. A few details stood out that are easy to miss when setting up remote build execution. Start with your branch’s configuration. Check build/soong/docs/rbe.json and the reclient binaries under prebuilts/remoteexecution-client/. The main settings are: export USE_RBE=1 export RBE_service="YOUR_GRPC_HOST:PORT" export RBE_instance="YOUR_INSTANCE" export RBE_DIR="prebuilts/remoteexecution-client/live" # Use remote execution for the initial C++ test. export RBE_CXX_EXEC_STRATEGY=remote This is only the connection and strategy configuration. Authentication and worker platform settings still need to match your backend. Verify the client directory exists in your checkout. local still uses the remote cache. In reclient, it means local execution on a cache miss. It does not disable remote caching. remote executes remotely on a miss, while remote_local_fallback can retry locally after a remote failure. A successful build doesn’t prove remote execution works. For initial testing, use a small target with pending compilation work and the remote strategy. Check the action logs and backend activity. Cache hits alone won’t validate your workers, and fallback can hide a broken setup. Check the platform properties. AOSP rules can explicitly pass properties such as container-image and Pool. A self-hosted backend must understand those values and provide a compatible execution environment. Changing the endpoint alone may not be enough. An unchanged second build isn’t a remote-cache benchmark. Ninja might simply skip the work. To measure remote cache reuse, the actions need to run again while the remote cache remains populated. References: AOSP configuration example, reclient execution strategies. For anyone running AOSP with reclient: which backend are you using, and what needed adapting beyond the endpoint and authentication?