How to Preview a Weave Router Benchmark Run Before Launch
A benchmark launch can fan out across tasks, attempts, and model routes. Before spending model or sandbox time, inspect the exact plan.
Weave Router's public weave-bench CLI has a --dry-run option for that step. It resolves the benchmark manifest, builds one Harbor command per comparison arm, prints the commands and version pins, then exits before launching any Harbor jobs. In this walkthrough, the plan selects one Terminal-Bench task and two arms. It produces no benchmark result.
Install the benchmark CLI
The harness requires Python 3.12 or newer and uv. This guide is pinned to Router commit cbe7584, so the source and manifest match the behavior described here:
git clone https://github.com/weave-os/router.git
cd router
git checkout cbe758442460ca68db8f4fe9837db087c50cf1e5
cd bench
uv sync --locked --extra harbor
Cloning the repository and syncing dependencies use the network. After that setup is complete, the planning command below builds and prints a local plan; it does not call the Router or model APIs or start Harbor jobs.
Print a one-task plan
This command compares the Router's router arm with a direct GPT-5.6 Luna control, using the benchmark's pinned smoke task:
uv run --locked --extra harbor weave-bench run terminal-bench-4 \
--arms router,luna \
--smoke \
--dry-run
Representative output:
# terminal-bench-4 run_id=<generated-run-id>
# tasks=1 (manifest has 66); harbor=0.22.0 (published: 0.22.0) codex=0.150.0
...
--include-task-name terminal-bench/cad-model
The CLI generates the run ID. The important details are the selected task and the pinned versions, followed by the Harbor command for each arm.
What the plan means
- One task, not 66: --smoke selects terminal-bench/cad-model from the 66-task manifest.
- Two arms, two Harbor jobs: router sends requests through Weave Router's /beta policy; luna calls GPT-5.6 Luna directly. This is not a same-model comparison because the Router policy chooses a model per request.
- Two attempts per arm: the pinned configuration sets two attempts and up to 12 concurrent tasks per job. For this one-task example, a real run is configured for up to four task attempts in total. An attempt can make multiple model requests, so that number is not an API-call count or a cost estimate.
- No credentials in the plan: dry-run output shows the names of environment variables a real job would use, not their secret values. Keep credentials in the environment; do not paste them into a report.
--smoke only limits task selection. Remove --dry-run and the CLI launches the Harbor jobs, which can make model calls and incur compute and provider costs. Review the generated commands, budget, credentials, and sandbox configuration before launching. Treat a plan as a plan, never as benchmark evidence.
Sources
- cmd_run returns before job launch when --dry-run is set
- Benchmark package and Python version requirement
- Terminal-Bench task, attempt, concurrency, and version pins
- Router and direct Luna arm definitions
- Weave Router benchmark methodology
This walkthrough explains the public CLI at the cited commit. No benchmark was launched and no new performance or cost result is claimed.
Mentions
值得看的不是“自动挑模型”这句口号，而是它把路由做成了可观测、可回退的网关：Weave Router 兼容 Anthropic、OpenAI、Gemini 接口，按请求选择模型，路由耗时目标低于 50ms。 weave-os/router Go 实现，可自托管，支持 Postgres、BYOK 和 OTLP；最近还加了按模型熔断，遇到 529 会自动排除过载模型。省下的模型费用要结合流量和效果实测，部署则要承担数据库与多副本缓存失效的维护成本。
How to Preview a Weave Router Benchmark Run Before Launch A benchmark launch can fan out across tasks, attempts, and model routes. Before spending model or sandbox time, inspect the exact plan. Weave Router's public weave-bench CLI has a --dry-run option for that step. It resolves the benchmark manifest, builds one Harbor command per comparison arm, prints the commands and version pins, then exits before launching any Harbor jobs. In this walkthrough, the plan selects one Terminal-Bench task and two arms. It produces no benchmark result. Install the benchmark CLI The harness requires Python 3.12 or newer and uv. This guide is pinned to Router commit cbe7584, so the source and manifest match the behavior described here: git clone https://github.com/weave-os/router.git cd router git checkout cbe758442460ca68db8f4fe9837db087c50cf1e5 cd bench uv sync --locked --extra harbor Cloning the repository and syncing dependencies use the network. After that setup is complete, the planning command below builds and prints a local plan; it does not call the Router or model APIs or start Harbor jobs. Print a one-task plan This command compares the Router's router arm with a direct GPT-5.6 Luna control, using the benchmark's pinned smoke task: uv run --locked --extra harbor weave-bench run terminal-bench-4 \ --arms router,luna \ --smoke \ --dry-run Representative output: # terminal-bench-4 run_id=<generated-run-id> # tasks=1 (manifest has 66); harbor=0.22.0 (published: 0.22.0) codex=0.150.0 ... --include-task-name terminal-bench/cad-model The CLI generates the run ID. The important details are the selected task and the pinned versions, followed by the Harbor command for each arm. What the plan means - One task, not 66: --smoke selects terminal-bench/cad-model from the 66-task manifest. - Two arms, two Harbor jobs: router sends requests through Weave Router's /beta policy; luna calls GPT-5.6 Luna directly. This is not a same-model comparison because the Router policy chooses a model per request. - Two attempts per arm: the pinned configuration sets two attempts and up to 12 concurrent tasks per job. For this one-task example, a real run is configured for up to four task attempts in total. An attempt can make multiple model requests, so that number is not an API-call count or a cost estimate. - No credentials in the plan: dry-run output shows the names of environment variables a real job would use, not their secret values. Keep credentials in the environment; do not paste them into a report. --smoke only limits task selection. Remove --dry-run and the CLI launches the Harbor jobs, which can make model calls and incur compute and provider costs. Review the generated commands, budget, credentials, and sandbox configuration before launching. Treat a plan as a plan, never as benchmark evidence. Sources - cmd_run returns before job launch when --dry-run is set - Benchmark package and Python version requirement - Terminal-Bench task, attempt, concurrency, and version pins - Router and direct Luna arm definitions - Weave Router benchmark methodology This walkthrough explains the public CLI at the cited commit. No benchmark was launched and no new performance or cost result is claimed.