sgl-project

Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff TL;DR SGLang 0.5.23 (released 2026-11-02) ships an opt-in feature called Paged Experts. It keeps K of the E experts of every MoE layer in GPU memory, holds all E in pinned host RAM, and copies in only the experts a step routes to that are not resident. The expert math still runs on the GPU. - What the PR measured: on Qwen3-30B-A3B, single-stream decode of 205.6, 87.7 and 32.6 tok/s (int4 on a 16 GB card, FP8 on a 16 GB card, int4 on an 8 GB laptop), against 16.6, 2.9 and 1.2 for --cpu-offload-gb. That is 12x, 30x and 27x. - Against llama.cpp tuned to its best --n-cpu-moe: a tie when most of the model fits or the link is slow (206 vs 194 and 33 vs 33 tok/s), and nearly 2x faster in the FP8 case (87.7 vs 45.2). - What I computed from the PR's numbers: the measured speeds are only reachable if consecutive tokens reuse experts. Uniform random routing would cap the FP8 case near 40 tok/s on a 50 GB/s link. - Where it breaks: with 4 concurrent requests the laptop run fell to 11 tok/s against 41 for llama.cpp. The dividing line is K // top_k, the largest decode batch that fits one wave. - I did not run any of this. The measurements come from one PR author's two consumer machines. The arithmetic on top of them is mine. What does Paged Experts actually do? A mixture-of-experts layer routes each token to top_k of E experts. For Qwen3-30B-A3B that is 8 of 128, across 48 layers. I checked the model's config.json on Hugging Face: 48 layers, 128 experts, 8 per token, hidden size 2048, MoE intermediate size 768. The experts are most of the 30B parameters, so a model that is cheap to compute can still be impossible to hold on a small GPU. There are three ways to serve it when the experts do not fit: 1. --cpu-offload-gb in SGLang. Offloaded weights are streamed to the GPU on every forward pass, whatever the router picked. 2. llama.cpp --n-cpu-moe N. The experts of N layers stay in system RAM and are computed on the CPU. No weight traffic, but CPU-speed math. 3. Paged Experts. A K-slot table per layer lives on the GPU. Each step, the experts it needs that are missing get copied from pinned host memory into slots (least recently used out), then the model's own fused-MoE kernel runs on the GPU as usual. The third option is the HiCache idea applied to expert weights: the GPU holds the working set, host RAM holds the rest. PR #42557 (merged 2026-10-19) added bf16, block-FP8 and GPTQ experts, and PR #45787 (merged 2026-10-25) added AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8. Turning it on takes two flags, quoted from the docs: python -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B-GPTQ-Int4 \ --enable-paged-experts \ --paged-experts-num-resident 92 The constraints matter as much as the flags. The docs list single GPU only (tensor, expert, pipeline and data parallelism are rejected at startup), speculative decoding rejected at startup, and a page-locked host store that has to hold every expert of every layer. Under WSL2, pinned memory is capped at about half of system RAM. How fast is it, and against what? The PR benchmarks on two machines: an RTX 4070 Laptop (8 GB, PCIe 4.0 x8, 54 GB RAM, WSL2) and an RTX 5070 Ti desktop (16 GB, PCIe 5.0 x16, 62 GB RAM, native Linux). The workload is ShareGPT prompts with 256 output tokens, prefix caching off in both engines, same KV capacity (10,240 tokens), byte-matched quantization (GPTQ int4 against IQ4_XS, FP8 against Q8_0). llama.cpp's --n-cpu-moe was searched in steps of 1 and only counted when its own memory projection fit physical VRAM. The PR's predecessor compared against --cpu-moe only, which in the author's own words overstated the advantage. !Bar chart of single-stream decode tokens per second for Paged Experts, tuned llama.cpp and SGLang CPU offload on three GPU and quantization setups | Setup (Qwen3-30B-A3B) | Paged Experts | llama.cpp, best | --cpu-offload-gb | vs offload | vs llama.cpp | |---|---|---|---|---|---| | 16 GB, int4, K=92 | 205.6 | 194.4 | 16.6 | 12.4x | 1.06x | | 16 GB, FP8, K=40 | 87.7 | 45.2 | 2.9 | 30.2x | 1.94x | | 8 GB laptop, int4, K=26 | 32.6 | 32.8 | 1.2 | 27.2x | 0.99x | Decode tok/s, single stream, from the PR tables. The two ratio columns are my division. Each headline cell was measured at least twice. The 12x to 30x headline is against the weakest baseline. Against a tuned llama.cpp the result splits in two: when the model nearly fits, or the link is slow, paging and CPU compute land in the same place. When many experts must live in RAM and the link is fast, GPU math wins by a wide margin because llama.cpp's CPU-side layers become the bottleneck. Why is it fast when experts keep crossing PCIe? A token at batch 1 activates 48 x 8 = 384 experts. Per the docs an int4 expert here is 2.43 MB; FP8 is about 4.72 MB (3 x 2048 x 768 weights at one byte, scales ignored). If every activation missed, one token would need 0.93 GB (int4) or 1.81 GB (FP8) over the link. Now compare with the measured decode time. The FP8 run does 87.7 tok/s, so 11.4 ms per token. Copying 1.81 GB in 11.4 ms needs 159 GB/s, more than a PCIe 5.0 x16 link can deliver (about 63 GB/s theoretical). So most activations cannot have been misses. For a number I need an effective bandwidth. The PR implies one: a prefill pages in about 4 GB in about 80 ms at K=92, and (128 - 92) x 117 MB per slot is 4.2 GB, so roughly 50 GB/s. With that, the highest miss rate each measured speed allows is: !Bar chart comparing the miss rate under uniform random routing with the highest miss rate allowed by the measured decode speed | Setup | Link assumed | Miss rate if routing were uniform random (1 - K/E) | Highest miss rate the speed allows | |---|---|---|---| | 16 GB, FP8, K=40 | ~50 GB/s | 69% | 31% | | 8 GB laptop, int4, K=26 | 15.75 GB/s (peak) | 80% | 52% | | 16 GB, int4, K=92 | ~50 GB/s | 28% | 26% | Under uniform random routing the FP8 run would be capped near 40 tok/s, and it measured 87.7. The laptop would be capped near 21 tok/s on a link running at its theoretical peak, and it measured 32.6. So the router must be reusing experts from one token to the next, the premise the docs state. The int4 16 GB row is inconclusive: at 63 GB/s instead of 50 the bound moves to 33%, above the random baseline. Two caveats. These are upper bounds on the miss rate, not measured miss rates; only a counter inside the engine gives the real value. And the reuse was seen on natural-language prompts. The PR warns that synthetic prompts mis-measure paging, so if your traffic is unusual, test on your own. What happens with more than one request? The PR is candid here. At 4 concurrent requests the laptop run produced 11 to 12 tok/s against 41 for llama.cpp, and FP8 on the 16 GB card gave 84 tok/s at K=40 but 31 at K=24 (llama.cpp: 55). !Bar chart of output tokens per second with 4 concurrent requests, Paged Experts against llama.cpp, for four K settings The docs explain the mechanism. A decode step is served by a single wave only if tokens x top_k <= K, which is decode up to K // top_k requests, and that single wave is the case a CUDA graph can capture. Larger batches run without a graph and need several waves, each with its own page-in. Four requests with top-8 routing can touch up to 32 distinct experts per layer, and the table of K values shows the pattern: | Run | K | K // 8 | 4-request result vs llama.cpp | |---|---|---|---| | int4, 16 GB | 92 | 11 | 387 vs 312 tok/s | | FP8, 16 GB | 40 | 5 | 84 vs 55 | | FP8, 16 GB | 24 | 3 | 31 vs 55 | | int4, 8 GB | 26 | 3 | 11 vs 41 | Both runs where K // 8 is below 4 lose; both where it is 5 or more win. Four points do not make a law, and the PR attributes the loss to steps that route to more experts than fit, the same condition seen from the other side. It does give a usable sizing rule: choose K at least batch x top_k for the batch size you plan to serve, or accept that you are outside the captured fast path. For one user this is irrelevant. For anything shared it decides whether the feature fits. What does a resident slot cost in context length? K and the KV cache come out of the same static budget (--mem-fraction-static), and one slot costs per_expert_bytes x num_moe_layers. Here the bf16 KV cache is 2 x 48 layers x 4 KV heads x 128 head dim x 2 bytes = 98,304 bytes (96 KiB) per token. - One int4 slot is 117 MB, about 1,190 tokens of bf16 KV cache (about 2,370 with an FP8 KV cache). - One FP8 slot is about 227 MB, about 2,300 tokens of bf16 KV cache. Dropping K by 10 on the int4 model buys roughly 11,900 tokens of context at the price of more paging. The PR's whole 10,240-token cache is only about 1.0 GB, so the experts dominate the budget. If the KV cache does not fit at startup, the docs suggest lowering K or using --kv-cache-dtype fp8. Why do short prompts get slower first tokens? A prefill touches nearly every expert, so it pages in everything that is not resident, once. For int4 at K=92 that is about 4.2 GB. For FP8 at K=40 it is 88 experts x 227 MB, about 19.9 GB, and on the laptop (K=26) about 11.9 GB over a link that peaks at 15.75 GB/s, at least 750 ms. | TTFT p50 | Paged | llama.cpp best | |---|---|---| | int4 16 GB, short / 2,048 tokens | 89 / 267 ms | 68 / 344 ms | | FP8 16 GB, short / 2,048 tokens | 284 / 443 ms | 226 / 984 ms | | laptop int4, short / 2,048 tokens | 928 / 1,441 ms | 620 / 2,396 ms | The page-in is a fixed cost, while llama.cpp's CPU-side layers scale with prompt length, so paging loses on short prompts and wins on long ones. The FP8 2,048-token TTFT of 443 ms is within about 45 ms of the 400 ms needed to move 19.9 GB at 50 GB/s, which suggests nearly all of it is page-in (arithmetic, not a profile). For short chat turns, TTFT is the number to watch. Can paging change the model's outputs? By design no. Routing is untouched and the paged layer returns the same result as the unpaged one. The docs report bit-identical logprobs for unquantized, FP8 and INT8 experts under deterministic inference, and agreement within Marlin's own batch-to-batch noise for the other formats. On GSM8K (200 questions), gpt-oss-20b MXFP4 scored 0.96 to 0.97 paged and 0.975 unpaged. The more instructive part is how it can go wrong silently, as described in PR #45787: - An expert tensor that was not listed as pageable stays unpaged, so a paged-in expert runs with another expert's values. In an AWQ control with the zero points unpaged, the median logprob error was 1.67 nats, against 0.003 when paged correctly. - gpt-oss-20b's MXFP4 experts are padded by the Marlin kernel (2880 to 2944 or 3072). Before the fix, paged gpt-oss-20b scored GSM8K 0.000 with every answer truncated. After it, 0.960. - The fix for the first class is structural: the pageable tensors are now derived from the layer, and at startup an independence check re-runs one more chunk of experts and fails if the post-load repack mixed experts. None of these crash. They produce fluent wrong text, so run an accuracy smoke test against the unpaged model on your own checkpoint format. A preflight script before you turn it on This is the arithmetic above in a few lines: slot cost, batch cap, and whether a measured speed implies expert reuse. def preflight(name, layers, experts, top_k, expert_mb, K, link_gbps, tok_s, kv_bytes_per_token): slot_mb = expert_mb * layers # GPU memory per slot, all layers batch_cap = K // top_k # largest single-wave decode batch all_miss_gb = layers * top_k * expert_mb / 1e3 # GB per token if every activation missed max_miss = link_gbps / (tok_s * all_miss_gb) # highest miss rate the speed allows random_miss = 1 - K / experts # miss rate with no reuse verdict = "reuse implied" if max_miss < 0.8 * random_miss else "inconclusive" print(f"{name}: slot={slot_mb:.0f} MB (~{slot_mb*1e6/kv_bytes_per_token:,.0f} KV tokens), " f"batch cap={batch_cap}, random-routing miss={random_miss:.0%}, " f"max miss allowed by {tok_s} tok/s={max_miss:.0%}, {verdict}") KV = 2 * 48 * 4 * 128 * 2 # Qwen3-30B-A3B bf16 KV bytes per token preflight("int4 16GB", 48, 128, 8, 2.43, 92, 50.0, 205.6, KV) preflight("FP8 16GB", 48, 128, 8, 4.72, 40, 50.0, 87.7, KV) preflight("int4 8GB", 48, 128, 8, 2.43, 26, 15.75, 32.6, KV) Output from running it: int4 16GB: slot=117 MB (~1,187 KV tokens), batch cap=11, random-routing miss=28%, max miss allowed by 205.6 tok/s=26%, inconclusive FP8 16GB: slot=227 MB (~2,305 KV tokens), batch cap=5, random-routing miss=69%, max miss allowed by 87.7 tok/s=31%, reuse implied int4 8GB: slot=117 MB (~1,187 KV tokens), batch cap=3, random-routing miss=80%, max miss allowed by 32.6 tok/s=52%, reuse implied Use a link speed measured with a pinned-memory copy test, not the spec sheet. The 50 GB/s here is inferred from the PR. A rollout checklist 1. Confirm your format is in the supported table and the MoE runner is Triton or Marlin. Anything else is rejected at startup, which is the good failure. 2. Check that the host can pin the whole expert store (half of RAM under WSL2). 3. Choose K from the docs' formula, then check K // top_k against your real concurrency. 4. Measure host-to-device bandwidth on your machine, and benchmark your own prompts, because the PR warns that synthetic ones mis-measure paging. 5. Run an accuracy smoke test against the unpaged model, and record TTFT for short and long prompts separately. FAQ Does it work across several GPUs or with speculative decoding? No to both in 0.5.23. The docs say single GPU only, with tensor, expert, pipeline and data parallelism rejected at startup, and speculative decoding is rejected too. Does it change accuracy compared with an unpaged server? The docs report bit-identical logprobs for unquantized, FP8 and INT8 experts under deterministic inference, and agreement within Marlin's noise for the other formats. Past bugs in this area failed silently, so run a smoke test on your format. How do I choose K? Use the docs' budget formula, then keep K at least batch x top_k for your target concurrency. Each extra slot costs per_expert_bytes x layers of memory that would otherwise be KV cache. Is it faster than llama.cpp's --n-cpu-moe? It depends. In the PR, single-stream decode tied in two of three setups and was 1.94x faster in the third, while llama.cpp had lower TTFT on short prompts and better 4-request throughput on the laptop. How much host RAM do I need? Enough to pin every expert: about 14.9 GB at int4 and 29 GB at FP8 for Qwen3-30B-A3B, by my arithmetic on the docs' per-expert size. Further reading - KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off - What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You Sources - SGLang v0.5.23 release notes - PR #42557: Paged Experts, serve MoE models whose weights do not fit VRAM - PR #45787: Paged Experts for AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8 experts - Paged Experts documentation at v0.5.23 - Qwen3-30B-A3B config.json

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4 --- title: "Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4" published: true description: "llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one g6.4xlarge, measured with the same prompts, the same 1-to-64-request sweep and the same GSM8K and BFCL questions. Answers are level on every engine; vLLM with Google's drafter is the fastest at every load, up to 4,516 output tokens per second." tags: aws, ec2, vllm, nvidia cover_image: raw.githubusercontent.com --- This article provides a step by step comparison of six inference serving engines on one Amazon EC2 G6 instance: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from one NVIDIA L4. xbill9/gemma4-dev The question is which engine to put on a G6 when one small model has the whole GPU. Each engine gets the same weights, the same prompts, the same 1-to-64-request load sweep and the same 1,319 GSM8K and 400 BFCL questions, and each is measured with and without Google's E2B drafter wherever the engine supports one. Every engine ran on one instance, one at a time, with the card to itself. --- The Results in Brief Answer quality barely moves between engines. GSM8K lands between 0.8954 and 0.9014 on the five CUDA engines with the drafter on or off, and every paired difference against llama.cpp is within the run-to-run range. LiteRT-LM scores level on GSM8K and lower on BFCL. For one request, five engines are within 8% of each other without a drafter (134.7 to 144.6 tokens per second end to end) and LiteRT-LM runs at 0.62 to 0.73x llama.cpp. The drafter decides the order: it gives vLLM 1.96x, llama.cpp 1.53x, SGLang 1.39x and LiteRT-LM 1.15x, and it slows mistral.rs to 0.95x. Under load the batching engines separate. At 64 requests of 128 tokens in and 512 out, vLLM reaches 3,716 tokens per second without a drafter and 4,516 with it, SGLang 2,967, mistral.rs 1,825 and llama.cpp 1,056. PyTorch and LiteRT-LM serve one request at a time. With each engine at its best setting, vLLM is the fastest at every level from 1 to 64 requests, on both shapes, and its drafter never loses. At the g6.4xlarge on-demand price that is $0.08 per million output tokens at full load. --- What Changes and What Stays | Engine | Version | Weights | Installed with | | :--- | :--- | :--- | :--- | | llama.cpp | 7182ef71e | exact Q4_0 GGUF | CMake, CUDA 13.1, sm_89 | | vLLM | 0.33.0 | W4A16, int4 tables | pip, its own torch 2.14.0+cu130 | | SGLang | 0.5.23 | W4A16, int4 tables | pip, cp313 wheel on Python 3.14, two patches | | mistral.rs | v0.9.5 | exact Q4_0 GGUF | cargo install, CUDA 13.1, sm_89 | | PyTorch | this repo's int4 engine | W4A16 | the DLAMI's torch 2.13 | | LiteRT-LM | 0.19.0 | exact Q4_0, and Google's int2/4/8 file | pip, GPU through WebGPU on Vulkan | The same for all six: one g6.4xlarge on-demand in us-east-1d (16 vCPUs, 64 GiB, one NVIDIA L4 with 23,034 MiB, driver 595.112.07), the PyTorch 2.13 / Ubuntu 26.04 Deep Learning AMI, Gemma 4 E2B on Google's QAT Q4_0 grid in every container, greedy decoding, and the system Python 3.14 with no virtual environment. The exact Q4_0 GGUF and the W4A16 checkpoint carry the same 4-bit values: xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf for llama.cpp and mistral.rs, xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 for vLLM and SGLang. --- At This Point You Should Have… - An AWS account with G-family vCPU quota in us-east-1 and an instance profile with SSM access - An S3 bucket for moving files to and from the instance - The system python3 with boto3, and the MCP server in gpu-pytorch-g6-2b-q4w4a16/ - A clone: git clone https://github.com/xbill9/gemma4-dev --- Step 1 — Find a G6 With Capacity The rigs name a g6.2xlarge (8 vCPUs, one L4). On 2026-11-05 it had no spot or on-demand capacity in any us-east-1 zone, and neither did g6.4xlarge spot. The MCP tool create_g6_instance walks every subnet, spot first and then on-demand: subnet-0c2872fe4182b9ec1 spot=True: ❌ AWS InsufficientInstanceCapacity: There is no Spot capacity available that matches your request. subnet-09e0f13c0a5b43092 spot=True: ❌ AWS InsufficientInstanceCapacity: There is no Spot capacity available that matches your request. subnet-0c2872fe4182b9ec1 spot=False: ❌ AWS InsufficientInstanceCapacity: We currently do not have sufficient g6.4xlarge capacity in the Availability Zone you requested (us-east- subnet-0d2b2038965e8d0c0 spot=False: ✅ Launching `i-040ed957b67c2d514` (g6.4xlarge, on-demand, 1x L4) in `us-east-1`. AMI: `ami-0f5139ed9ed52b805` Both sizes carry the same L4. The g6.4xlarge doubles the vCPUs and memory, which only llama.cpp's prompt threads and the build steps use. 🔎 Tip: all access is SSM and S3. The instance has no SSH key and its role has no S3 access. Commands go through SSM Run Command, which returns at most 24,000 characters, so files larger than that move through presigned S3 URLs generated on the workstation. --- Step 2 — One Rig Directory per Engine Each engine has its own directory with a tpu.env that decides every serving flag, a Makefile run on the instance, and its measurements under benchmarks/runs/: gpu-llamacpp-g6-2b-q4_0/ gpu-vllm-g6-2b-q4w4a16emb4/ gpu-sglang-g6-2b-q4w4a16emb4/ gpu-mistralrs-g6-2b-q4_0/ gpu-pytorch-g6-2b-q4w4a16/ gpu-litertlm-g6-2b-int248/ gpu-litertlm-g6-2b-q4w4a16emb4/ gpu-vllm-g6-2b-q4w4a16emb4/flight/setup.sh installs every engine in order and logs each step, so a failed step reruns alone: GEMMA4_DEV=/opt/gemma4-dev MODEL_S3_URI=s3://... bash setup.sh llamacpp vllm sglang mistralrs litert 18:48:44 START llamacpp 18:56:56 END llamacpp 18:56:56 START vllm 18:59:18 END vllm 18:59:18 START sglang 18:59:57 END sglang --- Step 3 — What the Deep Learning AMI Needs The AMI ships the NVIDIA driver and a PyTorch 2.13 environment, and no CUDA toolkit. Four things differ from a desktop install: - vLLM brings its own torch. The system Python has no torch, so pip install vllm==0.33.0 pulls its pinned wheel from PyPI, built for CUDA 13.0, and it runs on driver 595.112.07: torch 2.14.0+cu130 cuda 13.0 NVIDIA L4 (8, 9) - llama.cpp and mistral.rs build against CUDA 13.1 from NVIDIA's apt repository, for sm_89. mistral.rs also links cuRAND, which is a separate package: libcurand-dev-13-1. - LiteRT-LM finds the L4 through Vulkan. The AMI lists the L4 as a Vulkan device, and LiteRT-LM's WebGPU backend serves on it. - pip puts scripts in ~/.local/bin. SSM runs commands as root with HOME=/root, and that directory is off its PATH, so vllm, litert-lm and SGLang's ninja are found only once it is added. 🔎 Tip: SGLang 0.5.23 needs two patches for this checkpoint. Its Gemma 4 attention shim runs only for multimodal configs, and it leaves the int4 embed_tokens and per-layer table as uninitialized bf16. The rig's patches/ keep both tables packed on the card. --- Step 4 — Size Each Engine for the Whole Card The card serves one engine at a time and nothing else, so each engine gets the most the L4 will hold: - vLLM reserves 85% of the card (--gpu-memory-utilization 0.85), 64 running requests, 16,384 tokens each. - SGLang reserves 80% (--mem-fraction-static 0.80), 64 running requests, decode CUDA graphs up to 64. - llama.cpp runs 64 slots over a 65,536-token cache for the load sweep, and one slot of 131,072 for accuracy. - mistral.rs runs 32 requests over a fixed 16,384-token cache. 🔎 Tip: mistral.rs plans activation memory per running request. It sets aside about 395 MiB for each, so 64 requests do not load on the L4's 23 GB: Error: Primary device cuda[0] needs 28292 MiB (2976 MiB of fixed model components, 25316 MiB of activations, 0 MiB of post-load reservations), but only 21192 MiB is usable under the memory goal. At 32 requests a 65,536-token cache leaves too little room; --max-seqs 32 --kv-cache 16384tokens --max-num-batched-tokens 4096 serves in 5,286 MiB. --- Step 5 — One Request at a Time flight/engines.py sends six chat prompts, 256 tokens each, three times per engine, and takes the geometric mean of each prompt's median, end to end. Then 4 requests at once: arm e2e tok/s vs llama decode TTFT ms 4 at once x one memory ready s repeat llama 135.7 1.000 136.9 29 437.6 3.23 2930 MiB 6.1 6/6 llama-d2 207.7 1.531 211.4 30 373.2 1.80 3004 MiB 4.1 6/6 mistralrs 144.6 1.066 144.4 11 500.0 3.46 4664 MiB 24.1 6/6 pytorch 134.7 0.993 - - 134.5 1.00 3248 MiB 236.2 6/6 litert-int248 98.4 0.725 125.2 512 100.5 1.02 1489 MiB 9.4 6/6 vllm 140.4 1.035 140.7 16 537.2 3.83 18228 MiB 167.3 6/6 vllm-d4 275.3 2.030 281.2 19 711.3 2.58 18918 MiB 132.5 6/6 sglang 136.5 1.006 139.7 50 475.4 3.48 18345 MiB 307.2 6/6 sglang-d4 189.6 1.398 198.1 55 508.3 2.68 15191 MiB 61.1 6/6 | Engine | No drafter | Best drafter | Drafter gain | | :--- | :--- | :--- | :--- | | vLLM | 140.4 | 🥇 275.3 (4 tokens) | 1.96x | | llama.cpp | 135.7 | 207.7 (2 tokens) | 1.53x | | SGLang | 136.5 | 189.6 (4 tokens) | 1.39x | | mistral.rs | 144.6 | 137.6 (1 token) | 0.95x | | PyTorch | 134.7 | — | — | | LiteRT-LM | 98.4 | 112.8 (its own) | 1.15x | LiteRT-LM's exact Q4_0 build runs at 84.4 tokens per second, and LiteRT-LM takes about half a second to its first token against 11 to 55 ms for the others. Every engine repeats its own greedy text on all six prompts. --- Step 6 — 1 to 64 Requests sweep.py runs two shapes, 512 tokens in / 128 out and 128 in / 512 out, at 1 to 64 requests at once, five repeats, a unique prompt per request so no prefix cache can hit, output timed from the stream. flight/suite.py runs every sweep and every accuracy job one server at a time, and marks each finished job so a rerun skips it: python3 suite.py /opt/flight-out/2026-11-06-suite-g6 sweep:vllm,sweep:vllm-d1,sweep:vllm-d4,sweep:sglang,... 2026-11-06 04:06:20 START sweep:vllm ready_s=39.1 gpu=52, 2040 MHz, 19296 MiB 2026-11-06 04:07:33 sweep:vllm in512-out128 rc=0 2026-11-06 04:10:07 sweep:vllm in128-out512 rc=0 2026-11-06 04:10:08 END sweep:vllm ok=True Aggregate output tokens per second without a drafter, 128 in / 512 out: | Requests | vLLM | SGLang | mistral.rs | llama.cpp | | :--- | :--- | :--- | :--- | :--- | | 1 | 138.8 | 134.6 | 133.9 | 133.8 | | 8 | 991.0 | 967.4 | 626.5 | 597.5 | | 64 | 🥇 3,715.7 | 2,967.1 | 1,825.2 | 1,056.3 | 512 in / 128 out: | Requests | vLLM | SGLang | mistral.rs | llama.cpp | | :--- | :--- | :--- | :--- | :--- | | 1 | 132.8 | 129.6 | 133.1 | 125.4 | | 8 | 738.3 | 708.3 | 591.2 | 428.1 | | 64 | 🥇 1,593.7 | 1,373.0 | 1,132.0 | 630.8 | All 126 levels completed. vLLM's output at 64 requests is 26.8x its one-request rate on long replies. SGLang reaches 0.80 to 0.86x vLLM at 64, mistral.rs 0.49 to 0.71x and llama.cpp 0.28 to 0.40x. --- Step 7 — Each Engine at Its Best best_per_level.py picks each engine's fastest measured setting at each level, 128 in / 512 out: | Requests | vLLM | SGLang | llama.cpp | mistral.rs | | :--- | :--- | :--- | :--- | :--- | | 1 | 🥇 453 (4 drafted) | 283 (4 drafted) | 271 (2 drafted) | 134 (none) | | 2 | 🥇 856 (4 drafted) | 552 (4 drafted) | 430 (2 drafted) | 254 (none) | | 4 | 🥇 1,483 (4 drafted) | 875 (4 drafted) | 596 (2 drafted) | 468 (none) | | 8 | 🥇 2,399 (4 drafted) | 1,223 (4 drafted) | 791 (2 drafted) | 627 (none) | | 16 | 🥇 2,823 (4 drafted) | 1,673 (none) | 1,021 (2 drafted) | 1,339 (none) | | 32 | 🥇 3,665 (1 drafted) | 2,257 (none) | 1,132 (2 drafted) | 1,843 (none) | | 64 | 🥇 4,516 (4 drafted) | 2,967 (none) | 1,170 (2 drafted) | 1,825 (none) | 512 in / 128 out: | Requests | vLLM | SGLang | llama.cpp | mistral.rs | | :--- | :--- | :--- | :--- | :--- | | 1 | 🥇 345 (4 drafted) | 214 (4 drafted) | 224 (2 drafted) | 133 (none) | | 8 | 🥇 1,217 (4 drafted) | 711 (4 drafted) | 494 (2 drafted) | 591 (none) | | 16 | 🥇 1,389 (1 drafted) | 1,026 (none) | 572 (2 drafted) | 1,121 (none) | | 64 | 🥇 1,710 (1 drafted) | 1,373 (none) | 631 (none) | 1,132 (none) | vLLM's drafter pays at every level: with 4 drafted tokens 2.60x / 3.26x no drafter for one request, still 1.02x / 1.22x at 64. SGLang's drafter pays up to 4 to 8 requests and loses from 16 (0.73x / 0.69x at 64). llama.cpp's 2-token drafter pays at every level on long replies (1.11x at 64) and up to 16 requests on long prompts. --- Step 8 — Answer Quality gen_eval.py asks each engine all 1,319 GSM8K test questions (2,048-token limit) and the 400 BFCL simple function-calling questions (512), one request at a time, greedy, thinking off: | Engine | GSM8K | BFCL | GSM8K with drafter | BFCL with drafter | | :--- | :--- | :--- | :--- | :--- | | llama.cpp | 0.8984 | 0.9250 | 0.8976 | 0.9200 | | vLLM | 0.8969 | 0.9225 | 0.8992 | 0.9225 | | SGLang | 0.8954 | 0.9150 | 0.8992 | 0.9150 | | mistral.rs | 0.9014 | 0.9200 | — | — | | PyTorch | 0.8961 | 0.9225 | — | — | | LiteRT-LM, exact Q4_0 | 0.8999 | 0.8600 | — | — | | LiteRT-LM, Google's file | 0.8908 | 0.8250 | — | — | Paired question by question, every engine is level with llama.cpp on GSM8K, and the drafter changes no score. On BFCL, SGLang is 4 questions behind llama.cpp, a gap bigger than the run-to-run range, and LiteRT-LM is 26 and 40 behind. The drafter cuts the time to finish GSM8K: | Engine | No drafter | With drafter | Speedup | | :--- | :--- | :--- | :--- | | vLLM | 3,918 s | 🥇 1,270 s | 3.08x | | SGLang | 4,035 s | 1,956 s | 2.06x | | llama.cpp | 3,971 s | 2,007 s | 1.98x | | mistral.rs | 3,918 s | — | — | | PyTorch | 3,991 s | — | — | | LiteRT-LM | 5,018 s | — | — | 🔎 Tip: send thinking off explicitly to SGLang and mistral.rs. "chat_template_kwargs": {"enable_thinking": false} keeps replies short on every engine; mistral.rs thinks by default on Gemma 4. --- Step 9 — Cost At the g6.4xlarge on-demand price, $1.3232 an hour in us-east-1 from the AWS Price List API, the measured throughput works out to (arithmetic, from cost.py): | Setting | 128 in / 512 out | 512 in / 128 out | | :--- | :--- | :--- | | vLLM + drafter, 1 request | $0.81 per M tokens | $1.06 per M tokens | | vLLM + drafter, 64 requests | 🥇 $0.08 | 🥇 $0.21 (1 drafted) | | SGLang, 64 requests | $0.12 | $0.27 | | mistral.rs, 64 requests | $0.20 | $0.32 | | llama.cpp + drafter, 64 requests | $0.31 | $0.59 | The whole sweep and accuracy suite ran 12.2 hours, $16.14 on demand. A g6.2xlarge carries the same L4 at $0.9776 an hour; it was not measured here. --- Step 10 — Tear Down Results come off the instance as one tarball through a presigned S3 PUT, and then the instance is terminated: python3 -c "import boto3; print(boto3.client('ec2', region_name='us-east-1').terminate_instances(InstanceIds=['i-040ed957b67c2d514'])['TerminatingInstances'][0]['CurrentState']['Name'])" shutting-down --- Compare and Contrast | | vLLM | SGLang | llama.cpp | mistral.rs | | :--- | :--- | :--- | :--- | :--- | | 1 request, best | 🥇 275.3 | 189.6 | 207.7 | 144.6 | | 64 requests, 128/512, best | 🥇 4,516 | 2,967 | 1,170 | 1,825 | | Drafter under load | 🟢 pays at every level | ❌ 0.69x at 64 | 🟢 to 16–64 | ❌ slower | | GSM8K level | 🟢 yes | 🟢 yes | 🟢 yes | 🟢 yes | | BFCL level | 🟢 yes | ⚠️ 4 behind | 🟢 yes | 🟢 yes | | Runs as installed | 🟢 yes | ❌ two patches | 🟢 yes | ⚠️ needs cuRAND, fixed cache | PyTorch and LiteRT-LM serve one request at a time: | | PyTorch | LiteRT-LM | | :--- | :--- | :--- | | 1 request, best | 134.7 | 112.8 | | GSM8K level | 🟢 yes | 🟢 yes | | BFCL level | 🟢 yes | ❌ 26–40 behind | | Runs as installed | 🟢 yes | ⚠️ Vulkan | --- So, Which One? On a G6 serving one small model, run vLLM with Google's W4A16 drafter at 4 tokens: it is the fastest engine for one request (275.3 tokens per second) and at every load to 64 requests, it finishes GSM8K 3.08x sooner than without the drafter at the same score, and at full load it costs $0.08 per million output tokens on demand. SGLang without a drafter reaches 0.66x of that at 64 requests and needs two patches. llama.cpp with its 2-token drafter is the simplest to build and serves one user at 207.7 tokens per second, but tops out near 1,170 tokens per second. mistral.rs, PyTorch and LiteRT-LM suit one user at a time. --- Summary The goal of this article was to find the inference serving engine to run for Gemma 4 E2B on an Amazon EC2 G6. The key was holding everything else fixed: one instance, the same 4-bit values in every container, the same prompts, load sweep and question sets, and each engine sized for the whole L4. The six-engine results were: - 🟢 All six engines give the same GSM8K answers (0.8908 to 0.9014) with the drafter on or off. - 🟢 vLLM with its drafter is the fastest engine at every level from 1 to 64 requests: 275.3 tokens per second for one request and 4,516 at 64 (128 in / 512 out). - 🟢 vLLM's drafter pays at every load on the L4, and finishes GSM8K 3.08x sooner. - 🟢 At full load vLLM serves a million output tokens for $0.08 on demand. - ⚠️ The drafter's gain depends on the engine: 1.96x vLLM, 1.53x llama.cpp, 1.39x SGLang, 1.15x LiteRT-LM, 0.95x mistral.rs. - ⚠️ g6.2xlarge had no capacity in us-east-1 on the day; g6.4xlarge on-demand carries the same L4. - ❌ mistral.rs loads at most 32 requests on the L4's 23 GB. - ❌ SGLang's drafter cuts output to 0.69 to 0.73x of running without it at 64 requests. Scope: one g6.4xlarge on-demand in us-east-1d, one NVIDIA L4, the Deep Learning AMI with driver 595.112.07, Gemma 4 E2B on Google's QAT Q4_0 grid, one engine at a time, greedy decoding. One-request speed is six prompts of 256 tokens, three repeats; load is two synthetic shapes at 1 to 64 requests, five repeats each; accuracy is one pass of GSM8K (1,319) and BFCL simple (400). The one-request runs were on 2026-11-05 and the sweeps and accuracy on 2026-11-06, on the same instance; costs are arithmetic from the on-demand price. The strategy for comparing inference serving engines on Amazon EC2 G6 was validated with an incremental step by step approach. --- References - gemma4-dev on GitHub - One-request run: 2026-11-05-engines-g6 - Sweeps and accuracy: 2026-11-06-suite-g6 - Amazon EC2 G6 instances - AWS Deep Learning AMIs - llama.cpp - vLLM - SGLang - mistral.rs - LiteRT-LM - xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 - xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf - GSM8K - Berkeley Function Calling Leaderboard

Ripple-65B Is an Open Coder With 1M Context. Its DGX Spark Script Stops at 128K - Book: RAG Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- On November 4, Ricursive Intelligence released Ripple-65B Preview, a 65-billion-parameter dense model it says it pretrained from scratch on 12 trillion tokens. The weights are on Hugging Face under Apache 2.0. The headline is 80.5 on Terminal-Bench 2.1, ahead of Reflection's Beam (501B, 80.1), Meta's Muse Spark Open (816B, 76.4) and Thinking Machines' Inkling (975B, 63.8). Ripple's score is Ricursive's own, set against each rival lab's self-reported figure, and VentureBeat points out these are separate vendor evaluations, not a controlled head-to-head test. The launch page also says the model's size lets it run efficiently on local devices such as NVIDIA's DGX Spark, a desktop box with a GB10 chip and 128 GB of unified memory. Ricursive's own inference scripts show the window depends on the route. Its two-B200 server recipe defaults to 1,048,576 tokens. Its DGX Spark script sets 131,072. What Ricursive shipped Four repos, per the model card: - Ripple-65B-Preview: the BF16 instruct model. - Ripple-65B-Preview-NVFP4: labelled Experimental, a "conservative quantization" with NVFP4 MLP layers and attention kept in BF16 (card). - Ripple-65B-Base-Preview: the BF16 base model. - Ripple-65B-Preview-DFlash2: a draft model the inference scripts use for speculative decoding (spark.py). The card says "up to 1M" context in its intro, while its training table lists a 2M sequence length for the long-context stages. The architecture is new, and it ships as code. The repo's config sets model_type to ricursive and maps the model class to modeling_ricursive.py in the repo (Hub API). Loading that with Transformers takes trust_remote_code=True, which runs the repo's Python on your machine (Hugging Face docs). This is the publisher's own repo, but read the file before you load it. Read the scorecard row by row !Editorial illustration of a balance scale holding one small solid black cube on one side and a tall tower of many hollow cubes on the other, the beam level, burnt orange highlights on the small cube Ricursive's pitch: one 65B dense model against MoE models roughly 8 to 15 times its total size, on its own scorecard. The card's table mixes published numbers with Ricursive's in-house runs, and the harnesses differ: Terminus 2 for SWE-bench Pro v2.0 Hard, DeepSWE 1.1 and one of the two Terminal-Bench columns, Inspect AI for SciCode, and OpenCode wherever no harness is named (card). Ripple's Terminal-Bench lead holds under both columns it reports: 80.5 against Beam's 80.1 self-reported, and 80.5 against Beam's 70.8 under Terminus 2. It does not lead everywhere. On the same table, Beam scores 81.6 to Ripple's 76.5 on SWE-bench Verified, and 66.1 to 61.42 on SWE-bench Pro v1.0. Muse Spark Open leads DeepSWE 1.1 at 60.0 to 44.3 and MCP Atlas at 89.2 to 71.67. On SciCode, Ripple's 39.4 is the lowest of the five models. The numbers also move between Ricursive's own documents. VentureBeat found the launch graphic giving Inkling 39.6 on SWE-bench Pro v2.0 Hard, where the card lists 2.0. "Smaller" needs a footnote too. Beam has 501B total parameters but 23B active per token. Ripple is dense, so every token it generates goes through all 65B, on a Spark that Ricursive's launch page itself describes as low on memory bandwidth next to datacenter accelerators. Three ways to run it, three context windows !Editorial illustration of one cream water tank feeding three pipes of very different widths: a wide pipe into a cloud, a medium pipe into a pair of server racks, a thin pipe into a small desktop box Same weights, three routes. The context window you get depends on which one you take. | Route | Hardware | Context | Requests at once | |---|---|---|---| | OpenRouter, served by GMICloud | none | 1,048,576, max output 131,072 | provider-managed | | Ricursive's server recipe (SGLang 0.5.19) | 2× B200, Linux, CUDA 13 | 1,048,576 | 8 per port | | Ricursive's DGX Spark script | DGX Spark (GB10) | 131,072 | 1 | The OpenRouter row comes from its endpoint listing: BF16, $0.20 per million input tokens, $0.60 per million output, $0.05 for cached input, with tools and tool_choice supported. VentureBeat's article reported a 16,384-token limit there; the listing now shows the full window, so check it before you size a prompt. Together AI lists the model as coming soon. The fastest way to try it is one OpenAI-style call: curl https://openrouter.ai/api/v1/chat/completions \ -H "Authorization: Bearer $OPENROUTER_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "ricursiveai/ripple-65b-preview", "messages": [ {"role": "user", "content": "Explain git rebase --onto with an example"} ], "max_tokens": 4096 }' Give it room. Ricursive's server README says to allow enough output tokens for both the reasoning and the answer. On its own server, reasoning comes back in message.reasoning_content, separate from message.content. The DGX Spark path, line by line spark.py in the NVFP4 repo is short, and it tells you what local means here: - It checks for the Spark's GB10 GPU (compute capability 12.1) and raises an error on anything else. - It fixes the context at 131,072 tokens and runs one request at a time. - It wraps SGLang's engine inside your Python process. There is no HTTP server and no port, and complete() rejects streaming. - chat() defaults to max_tokens=256 and temperature=0. Usage, from inside the repo's inference/local folder: from spark import Spark engine = Spark( "ricursiveAI/Ripple-65B-Preview-NVFP4", "ricursiveAI/Ripple-65B-Preview-DFlash2", ) print(engine.chat("Find files over 100 MB", max_tokens=4096)) A coding agent that expects an OpenAI-compatible endpoint won't find one on this path. You'd put your own server in front of it. I haven't run the script myself. MacBooks? VentureBeat says Ricursive's social announcement names them, but the release doesn't pin down a laptop memory configuration, speed or usable context. The Hub already lists nine community quantizations, including MLX and GGUF builds, all from uploaders other than Ricursive. Ricursive's own cards don't mention them. Running it next to the agent you already use Several of the agentic scores on Ripple's card came from the OpenCode harness, and OpenCode can talk to OpenRouter through its provider config. That makes a fair test cheap: give one ticket to the agent you trust and the same ticket to OpenCode on Ripple, then compare the diffs. I build Hermes IDE, so weigh this with that in mind. Hermes runs OpenCode next to Claude Code, Codex and other CLI agents, each in its own real terminal. ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch and worktree, so the two attempts never touch each other's files. You set OpenRouter up as a provider in OpenCode itself; Hermes uses each agent's own sign-in and never asks for your tokens. Hermes is not affiliated with Ricursive, OpenRouter or any agent vendor. !The Hermes IDE 2.3.4 New task sheet with its agent picker open The ⌘N New task sheet in Hermes 2.3.4 with the agent picker open: Claude Code and Codex installed, and OpenCode, Antigravity CLI and the other supported agents listed as not installed, plus Custom agent. What to watch before you switch The card calls this a Preview release, and the NVFP4 build is marked Experimental. Neither the release nor VentureBeat's coverage cites an independent Terminal-Bench run of the 65B model. Together's listing still says coming soon. Fireworks, which the launch page said would go live later on release day, had no confirmed rate in VentureBeat's report. And the community MLX and GGUF builds still have to prove themselves on real agent tasks. If you own a Spark, measure tokens per second on spark.py with a prompt close to 128K before you plan local agent work around it. --- If this was useful A 128K local window still means choosing what the model sees. The RAG Pocket Guide covers that half: chunking, retrieval and reranking, so the files that matter land in the window and the rest stay out. If you want to race Ripple against your current agent on one repo, I build Hermes IDE for that: free, source-available (GitHub), on macOS, Windows and Linux. 

SGLang's RadixAttention treats the KV cache as a radix tree of tokens across requests. For multi-turn chats and few-shot evals, it reuses prompt KV cache instead of recomputing from scratch. Cache hit rates regularly cross 70%. sgl-project/sglang

Cosyvoice3 TTS is now 1.69x faster on SM120 Consumer Blackwell devices in SGL-Omni! On SM120 it's DiT block was falling back to padded Torch SDPA, I managed to route it to FA4 and unlock torch.compile. You can check it out on your hardware sglang-omni#3383

3/6 Two: with a regex stop condition, the matched text is left in the response and `no_stop_trim` is ignored. The trimmer looks for the regex pattern text literally, so it never matches. 👉sglang#47461

vLLM・SGLang・TensorRT-LLMをクラウドGPUで動かしたら、それぞれの個性が見えてきた この記事で分かること Qwen3-8Bを、L4 24GB 1台の上でvLLM・SGLang・TensorRT-LLMに載せ、同じリクエストを送った。揃えたつもりの条件が、5か所で揃っていなかった。どれもベンチマークの数字に入り込む。 速さの順位は載せない（速さは自分の条件で測ってほしい）。すべてこの版・L4（sm_89）での話である。エンジンを選びたいだけなら「エンジン別の要点」と最後の「私の使い分け」へ。 エンジン別の要点 エンジン 知っておくこと vLLM 0.31.0 L4ではQwen/Qwen3-8B-FP8が重みだけFP8（W8A16）で動く SGLang 0.5.21 ツール定義に項目が足され、入力が増える。L4・8B bf16は--mem-fraction-staticの自動値では起動しない（0.87で動いた） TensorRT-LLM 1.2.1 L4ではQwen/Qwen3-8B-AWQ・-FP8をそのまま読めない。KVが足りないと黙って待たせる。キャッシュを消すAPIが無い 用語 用語 意味 KVキャッシュ 生成中のリクエストが、それまでのトークンの計算結果を置いておくGPUメモリ。容量をトークン数で数え、同時に処理できる本数の上限を決める プリフィル 入力（プロンプト）をまとめて読み込む最初の段階。入力が長いほど重い TTFT Time To First Token。送ってから最初の1トークンが返るまでの時間 W8A16／W8A8 重み（Weight）と計算時の値（Activation）のビット数。W8A16は重みだけ8bitで計算は16bit 追い出し（preemption／retraction） KVが足りなくなったとき、処理中のリクエストをいったん外すこと。後でやり直す RadixAttention SGLangのプレフィックスキャッシュ。木構造で、任意のトークン位置まで一致を使い回す（vLLMは決まった長さのブロック単位） sm_89／Ada／Hopper NVIDIA GPUの世代。L4はAda（sm_89）、H100はHopper（sm_90） ModelOpt NVIDIAの量子化ツール。TensorRT-LLM向けの量子化モデルを作る 条件 項目 値 モデル Qwen3-8B（bf16、AWQ 4bit、FP8） GPU NVIDIA L4 24GB 1台（sm_89、AWS g6.xlarge）。ドライバ 595.112.07、CUDA 13.2 エンジン vLLM 0.31.0 / SGLang 0.5.21 / TensorRT-LLM 1.2.1（trtllm-serve、PyTorchの経路） リクエスト /v1/chat/completions。ツール定義10個（ケースごとに40個から選ぶ）＋日本語の質問。システムプロンプトなし、thinkingオフ、質問の末尾にリクエストごとの識別子 max_tokens／最大の文脈長 512／8,192 測った並行数 AWQ・FP8は1と16。bf16は1（16は載らない：3節）。SGLang bf16は容量の確かめで8も 送ったケース 200ケースのうち20ケースを繰り返し送った（並行数1で80件） プレフィックスキャッシュ 3エンジンとも既定で有効。区切り（エンジン×チェックポイント×並行数×測定の種類）ごとにサーバーを起動し直す エンジン別 vLLM SGLang TensorRT-LLM 入力長 最大／中央値（200ケース） 2,591／2,310 2,691／2,410 2,591／2,310 ツールパーサー hermes qwen（Qwen25Detector） qwen3 バッチの上限 max_num_seqs 256・max_num_batched_tokens 2048 max_running_requests 2048 max_batch_size 2048・max_num_tokens 8192 チャンク化プリフィル 有効 有効（4096） 無効 CUDA graph FULL_AND_PIECEWISE 有効 バッチ1〜32・64・128 バッチの上限はどれも並行数16より大きい。 1. 入力：SGLangだけトークン数に差が出た 同じリクエストでusage.prompt_tokensを比べると、SGLangだけ全ケースで+100トークンだった。 // 送ったもの（{…}は省略） {"type": "function", "function": {"name": "search_master", "description": "…", "parameters": {…}}} // テンプレートに渡るもの {"type": "function", "function": {"description": "…", "name": "search_master", "parameters": {…}, "strict": false}, "defer_loading": null} エンジン toolsの扱い prompt_tokens vLLM 型に通すが、テンプレートに渡す長さは変わらない 送ったまま手元でテンプレートに通した長さと一致 SGLang model_dump()で項目を足し、キーの順が変わる +100 TensorRT-LLM — vLLMと一致 書き換えを再現する関数を書き、1,200件でSGLang自身のmodel_dump()と突き合わせて違いは0件だった。確かめるなら、テンプレート適用後の文字列（vLLMなら/tokenize）の差分を取る。 切り替えるオプションは無い。 揃えるならクライアントから足した形で送る（他エンジンの入力も変わる）。今回はそのまま測った。 効き方：プリフィルが重くなる／KVの見積もりがずれる（3節）／精度にも混ざる。ツールが多いほど差は大きい。パーサーも3つとも別の実装で、読み損ねは精度の差として出る。 2. 量子化：「FP8」が同じ意味ではない エンジン 起動時の警告 vLLM Weight-only FP8 compression will be used leveraging the Marlin kernel. SGLang Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! 「FP8で比べた」表は、重みだけ8bitと計算も8bitが混ざりうる。起動ログのカーネル名で確かめる。 vLLMがL4でW8A16になるのは、カーネルの候補を順に試し、最初に使えるものを選ぶため（vllm/model_executor/kernels/linear/init.pyのinit_fp8_linear_kernel）。 CUTLASSのブロックFP8のカーネルはsm_90・sm_100・sm_120向けしか無い（csrc/.../c3x/scaled_mm_blockwise_sm*_fp8.cu）。--linear-backendで変えられるはずだが試していない。 TensorRT-LLM 1.2.1・L4では、公開の次の2つを読めなかった。 チェックポイント 読めない理由 代わり（ModelOpt 0.37.0） Qwen/Qwen3-8B-AWQ（AutoAWQ形式） load_hf_quant_configにawqの分岐が無い --qformat int4_awq Qwen/Qwen3-8B-FP8（ブロック単位） 対応表で、ブロック単位FP8はAda（sm_89）が対応外（HopperとBlackwell sm100/103は対応）。モデル側の表でもQwen3は対応外なので、H100でも使えない --qformat fp8 --kv_cache_qformat none 2つで約56分かかった。nvidia/Qwen3-8B-FP8はKVまでFP8になるため使わなかった（KVの型の上書きは試していない）。TensorRT-LLMだけチェックポイントが別物なので、量子化した構成はエンジン間で比べず、各エンジン内でbf16との精度差だけ確かめた。 3. 容量：同じGPUでも載る量が違う 名前の似たメモリ設定が、何に対する割合かが違う（L4は約22.5GiB〔23,034MiB〕、Qwen3-8B bf16の重みは15.27GiB）。 | weights | KV | act | rest | GPU |===============|==============|=========|======| vLLM |<---------------- 0.92 ---------------->| SGLang |<----------- 0.87 ----------->| TRT-LLM |<-------- 0.9 x free --------->| act＝活性化など計算中の作業領域（プリフィルの中間結果、CUDA graph）。rest＝エンジンの外で使われる分の余白。模式図で、長さは比率どおりではない。 エンジン 割合の対象 割合に含むもの 値 KVの容量（トークン） 容量を直接指定するなら vLLM GPU全体 重み＋KV＋活性化 0.92（既定） 30,240 --kv-cache-memory-bytes（指定すると0.92は無視される） SGLang GPU全体 重み＋KV（活性化は外） 0.87（手で決めた） 26,813 --max-total-tokens TensorRT-LLM 重みと活性化を確保した後の空き KVだけ 0.9（既定） 約32,800 kv_cache_config.max_tokens（割合と両方なら小さいほう） 揃えたいなら割合ではなくトークン数で直接指定する（オプション名は版で変わりやすい）。今回は既定に近い状態を測るため揃えていない。 KV 1トークン = 36層 × KVヘッド8 × 128次元 × K/V 2 × 2バイト = 144KiB 1本の最大 = 入力2,691（SGLang） + max_tokens 512 + 余裕1 = 3,204 並行数8 = 3,204 × 8 = 25,632（+100前は 3,104 × 8 = 24,832） 並行数16 ≒ 51,300 → bf16は3エンジンとも載らない 全リクエストが最大長まで伸びても入るかで数え、入らない組み合わせは測っていない。 SGLangは自動値（0.684）で起動しなかった。 自動値＝（GPUの容量−作業領域の見積もり）÷GPUの容量（srt/arg_groups/memory_hook.py）。 作業領域の見積もり（L4） 大きさ 固定の分 512MiB プリフィル chunked_prefill_size 4,096トークン×1.5MiB＝6GiB（係数1.5はソースで「経験的な値」） 並列化の分 128MiB（1GPU） CUDA graph デコードの最大バッチ48×2MiB＋プリフィルの分 合計 約7.1GiB（23,034MiB×(1−0.684)） ValueError: Loaded weights leave no GPU memory for the KV cache under --mem-fraction-static=0.684. 並行数8が載る値を0.01刻みで探した（0.01あたり約1,590トークン）。 割合 KVの容量 並行数8（25,632） KV以外の残り 追い出し 0.684（自動） 起動しない — — — 0.84 22,047 載らない 1.82 GB — 0.86 25,224 載らない 1.40 GB — 0.87 26,813 載る 1.11 GB 0回 +100前なら0.86で足りたが、1節の+100で408トークン足りなくなり0.87にした。0.88以上は試していない。 4. 溢れたとき：追い出すか、黙って待たせるか 方式 エンジン 混んだとき遅れる場所 見える指標 追い出す vLLM・SGLang 生成の途中（ストリーミングが止まる） vllm:num_preemptions／sglang:num_retracted_requests_total 待たせる TensorRT-LLM 最初の1トークン（TTFT） 無い。numActiveRequestsで見る TensorRT-LLMは「入力長＋max_tokens」ぶんを先に確保するので、max_tokensを大きく付けるほど同時に受け入れる本数が減る。並行数16で送っても中では8本しか走っていないことがある。空きに合わせて詰め込むMAX_UTILIZATIONもある（処理中のリクエストが一時停止しうる。試していない）。 今回は/metricsを2秒ごとに読み、numActiveRequestsの最大が送った本数に届いたことを確かめた。溢れる組み合わせは測っていない（3節）。 5. 観測：そのまま読むと誤判定になる エンジン 癖 対処 SGLang キャッシュヒットが0だとprompt_tokens_detailsごと出ない 無ければ0 SGLang num_retracted_requests_totalは初回の追い出しまで出ない ゲージnum_retracted_reqsがあれば0回 TensorRT-LLM kvCacheStatsはenable_iter_perf_statsが要り、1件処理するまで空 短いリクエストを1件送ってから読む TensorRT-LLM その1件で、測定の最初のリクエストがcached_tokens=1になった（<|im_start|>を共有。partial reuseが既定で有効なので、32トークンのブロックの一部でも当たる） 書式を通さない/v1/completionsで送る TensorRT-LLM 反復ごとの統計は読むたびに消える 測定中に2秒ごとに読む 「読めなければ0」にしていたら誤った結論を出していた。「読めなければ止める」作りにしていたので気づけた。 エンジン キャッシュの消し方 vLLM POST /reset_prefix_cache（VLLM_SERVER_DEV_MODE=1） SGLang POST /flush_cache TensorRT-LLM APIなし。再起動 キャッシュがどれだけ効くかは、送るケースの数とKVの容量で大きく変わる。今回は20ケース（入力の合計45,957トークン）を繰り返し送った。 チェックポイント KVの容量（トークン） 20ケースぶんが cached_tokens／入力（3エンジンとも同程度） bf16 26,813〜32,800 収まらず、一周する前に追い出される 3.6〜4.7%（当たるのは先頭の書式 約50トークンだけ） AWQ・FP8 76,096〜99,552 収まり、2周目からツール定義ごと当たる 並行数1で約75%、16で約82% 効いているのは並行数ではなく、量子化で空いたKVの容量である。量子化した構成の速さには、計算が軽くなった分とキャッシュに当たった分が混ざりうる。ケースの数・繰り返し方とKVの容量を並べて確かめたい。 6. 動かすまで 版がぶつかるので、仮想環境をエンジンごとに分け、クライアントはHTTPだけで話す。 エンジン torch transformers vLLM 2.13.0+cu132 5.17.0 SGLang 2.13.0 5.12.1 TensorRT-LLM 2.9.1+cu130 4.57.3 問題 対処 TensorRT-LLM環境でModelOptが動かない（NumPy<2固定と最新pyarrowが衝突） pyarrow<22 TensorRT-LLM FP8が起動中にsegfault（原因不明、再実行で通過） — そのとき親プロセスが残り、20分待ち続けた ログにSegfault encountered等が出たら即失敗にして1回だけ再起動 まとめ：比較を作る・読むときのチェックリスト 確かめること 確かめ方 モデルに届く入力は同じか usage.prompt_tokensを並べる。テンプレート適用後の文字列を比べる ツールパーサーは何か 起動オプションを書く 量子化の計算は何か 起動ログのカーネル名・警告 量子化のチェックポイントは同じか 対応表とソースで、使うGPUで読める形式か確かめる KVの容量はいくつか 割合ではなくトークン数で見る。揃えるなら直接指定する 溢れたら何が起きるか 追い出しカウンタと同時実行数。max_tokensも書く 指標が読めないときの扱い 読めなければ止める キャッシュの状態 有効か、区切りごとに消すか、cached_tokensはいくつか。ケースの数・繰り返し方とKVの容量も書く 私の使い分け（推奨案） 速さではなく、導入の手間・設定の素直さ・観測しやすさで決めた。 当てはまる範囲：24GBクラスのGPU1台・8B級・同時8本程度まで。大きいGPUや70B級ではメモリ（3節）と量子化の対応（2節）が変わる。 推奨 想定するユースケース 決め手 気をつけること vLLM 社内チャットやRAGのPoC。公開モデル（量子化版を含む）を差し替えて評価する 公開チェックポイントをそのまま読める（2）。既定の設定で起動し、追い出しがカウンタで見える（3・4） L4のFP8はW8A16（2） SGLang 同じツール定義・システムプロンプトを先頭に付けるエージェントやバッチ処理 RadixAttention（用語）。L4でもFP8がW8A8（2）。※今回は共有部分が短く、実測ではない（5） 入力が増える（1）。割合を手で決める（3）。指標が0で出ない（5）。共有部分の長い順に処理するlpmは既定ではない（既定fcfs） TensorRT-LLM モデルを固定して長く運用する本番API 処理中のリクエストが途中でやり直しにならない（4）。量子化は最初の1回で済む 量子化を作り直す（2）。混むとTTFTが伸びるので、SLAがTTFTなら自分で測る（4）。キャッシュを消すAPIなし（5） 迷ったらvLLMで基準を作り、チェックリストで条件を揃えてから他と比べる。 付録：起動コマンド vLLM 0.31.0（bf16。AWQ・FP8はモデル名だけ替える。VLLM_SERVER_DEV_MODE=1 を付けて起動） vllm serve Qwen/Qwen3-8B --served-model-name Qwen/Qwen3-8B \ --host 127.0.0.1 --port 8000 \ --enable-auto-tool-choice --tool-call-parser hermes \ --enable-prompt-tokens-details --max-model-len 8192 SGLang 0.5.21 python -m sglang.launch_server --model-path Qwen/Qwen3-8B --served-model-name Qwen/Qwen3-8B \ --host 127.0.0.1 --port 30000 \ --tool-call-parser qwen --enable-metrics --enable-cache-report \ --mem-fraction-static 0.87 --context-length 8192 TensorRT-LLM 1.2.1（ModelOptで作ったAWQ。bf16はQwen/Qwen3-8B） echo "enable_iter_perf_stats: true" > trtllm-serve.yaml trtllm-serve ~/models/Qwen3-8B-int4-awq-modelopt \ --host 127.0.0.1 --port 8001 --backend pytorch \ --tool_parser qwen3 --config trtllm-serve.yaml --max_seq_len 8192 出典 SGLang v0.5.21：github.com/sgl-project/sglang/tree/v0.5.21 （srt/entrypoints/openai/serving_chat.py・protocol.py・usage_processor.py、srt/observability/metrics_collector.py、srt/function_call/function_call_parser.py、srt/arg_groups/memory_hook.py） TensorRT-LLM v1.2.1：github.com/NVIDIA/TensorRT-LLM/tree/v1.2.1 （tensorrt_llm/commands/serve.py、llmapi/llm_args.py、_torch/model_config.py、_torch/modules/linear.py、cpp/tensorrt_llm/executor/jsonSerialization.cpp、docs/source/features/quantization.md） ModelOpt 0.37.0：github.com/NVIDIA/Model-Optimizer/tree/0.37.0 （examples/llm_ptq/hf_ptq.py） vLLM v0.31.0：github.com/vllm-project/vllm/tree/v0.31.0 （vllm/config/cache.py、vllm/engine/arg_utils.py、vllm/model_executor/layers/quantization/fp8.py、vllm/model_executor/kernels/linear/init.py・scaled_mm/marlin.py・cutlass.py・flashinfer.py・deep_gemm.py・b12x.py、vllm/utils/flashinfer.py、vllm/platforms/cuda.py、vllm/model_executor/layers/quantization/utils/w8a8_utils.py、csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/、vllm/entrypoints/openai/chat_completion/protocol.py、vllm/v1/metrics/loggers.py）、起動ログ vLLM プレフィックスキャッシュの設計：docs.vllm.ai TensorRT-LLM スケジューラの方針（GUARANTEED_NO_EVICT／MAX_UTILIZATION）：nvidia.github.io SGLang サーバー引数：docs.sglang.io 、new_token_ratioはsrt/managers/schedule_policy.py nvidia/Qwen3-8B-FP8：huggingface.co PagedAttention：Kwon et al., SOSP 2023。RadixAttention：Zheng et al., NeurIPS 2024

155 responses leaked between AI customers: the GLM-5.3 KV cache bug — and the canary test for YOUR provider --- published: true tags: ai, security, cloud, llm description: "On Cloudflare Workers AI, 155 GLM-5.3 responses contained other customers' data — stale KV-cache pages served with HTTP 200. The mechanism, the 25-minute repro, the SGLang bug behind it, and a canary script that tests YOUR provider for cross-request leaks." series: black box --- > Between August 27 and September 9, some responses from GLM-5.3 on Cloudflare Workers AI contained other customers' prompts and partial generations. Not a breach of a database, not a prompt injection — the inference engine itself served stale KV-cache pages from previous, unrelated requests, with a clean HTTP 200. Exactly 155 responses were affected, all triggered by prompts longer than 681,280 tokens. Cloudflare published the full autopsy yesterday. This post is the mechanism in four steps, why the same bug class may sit in other inference stacks, and a canary script that tests your own provider for cross-request leakage. The detail that should stop you: 681,280 is 65% of 1,048,576 — the model's context window. Someone sized the KV pool at roughly two-thirds of the context and shipped it. Everything else follows from that one number. The failure chain, step by step From Cloudflare's disclosure, cross-checked against SGLang issue #30233 (open since July 6): 1. The mismatch. GLM-5.3 launched with a 1,048,576-token context window, but the prefill KV pool held only 681,280 tokens. Cloudflare knew the pool was smaller and believed the engine would cleanly reject oversized prompts. 2. The rejection path. A prompt longer than the pool gets rejected in prefill — but in disaggregated prefill/decode mode, the rejected prompt was "replaced with a one-token placeholder," and prefill transferred just that placeholder's KV pages to decode. No error was returned. 3. The uninitialized read. Decode had reserved 8 KV pages for the request. Only page P0 was ever written. Attention then read all 8 reserved pages — 7 of which still contained data from earlier requests on the same node. Other customers' prompts, other customers' generations. 4. The clean 200. The response streamed out composed partly of someone else's context. No error code, no anomaly — which is why it took an external researcher, not monitoring, to catch it. Two bugs overlapped: a configuration bug (pool < context — Cloudflare's, and only GLM-5.3 was affected there because every other Workers AI model had a pool larger than its context) and an engine bug (rejected requests surviving to the KV handoff — SGLang's, generic to disaggregated deployments, fixed in v0.5.22 after Cloudflare's draft PR #38704 and the merged follow-ups #36988/#38935). And a third factor that isn't a bug at all: KV pages aren't zeroed between requests by design — performance depends on reuse. The entire safety of the architecture rests on the bookkeeping around those pages being airtight. The timeline worth studying Reported by Gadi Evron (Knostic) via HackerOne on September 8, 23:11 UTC. Reproduced by Cloudflare within 25 minutes. Model private at 14:00 the next day, root cause at 14:35, fixed config at 14:52, model restored at 19:40. Coordinated disclosure published October 26 with the reporter and the SGLang maintainers, complete with page diagrams, issue links, and the exact number of affected responses. Set that next to the same month's eMudhra response (revocation in 14 hours, then blaming "social media amplification" for the scrutiny) and you have the two poles of incident handling. Cloudflare's post even ends with the hardening roadmap — KV checksums on every prefill-to-decode transfer, canaries, per-page ownership tags, and formal verification of the RDMA transfer protocol. When the failure mode is "the bookkeeping lied," you eventually stop patching and start proving. The lab: a canary test for YOUR provider You cannot audit a provider's KV pool from the outside. You can detect the symptom class: data from request A surfacing in the response to request B. That's exactly what Cloudflare's post-incident canaries do — here is the userland version, works against any OpenAI-compatible endpoint (point BASE/KEY/MODEL at OpenAI, Anthropic-compatible gateways, vLLM, SGLang, llamacpp, Ollama): # canary.py — cross-request leak smoke test for chat APIs # usage: BASE=https://api.provider.com/v1 KEY=sk-... MODEL=... python3 canary.py import os, random, string, sys, urllib.request BASE = os.environ["BASE"].rstrip("/") # e.g. http://localhost:8000/v1 KEY = os.environ.get("KEY", "none") MODEL = os.environ["MODEL"] canary = "CANARY-" + "".join(random.choices(string.ascii_lowercase + string.digits, k=16)) def chat(messages, max_tokens=200): req = urllib.request.Request( f"{BASE}/chat/completions", data=__import__("json").dumps({"model": MODEL, "messages": messages, "max_tokens": max_tokens, "temperature": 0}).encode(), headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}) return __import__("json").loads(urllib.request.urlopen(req, timeout=120).read()) # A — plant the canary chat([{"role": "user", "content": f"Internal ref for this session: {canary}. Reply with 'noted' only."}]) print(f"[A] canary planted: {canary}") # B — unrelated requests; scan every response for the canary leak = False for i in range(1, 6): r = chat([{"role": "user", "content": f"Write a two-sentence haiku about packet #{i}."}]) text = r["choices"][0]["message"]["content"] hit = canary in text leak |= hit print(f"[B{i}] {'!!! CANARY LEAKED' if hit else 'clean'} — {text[:60]!r}") print("RESULT:", "CROSS-REQUEST LEAK DETECTED" if leak else "no leak in this sample (not a proof)") Honest scope, stated in the script's last line: a clean run is not a proof — the GLM-5.3 bug needed prompts beyond the KV pool on a misconfigured node. This catches gross leakage cheaply; run it after upgrades, against new endpoints, and in CI against your own inference stack. On providers, vary it: repeat with the B requests on a fresh session/conversation id — same-account separation is the weakest fence to test. I ran the smoke test against the API that generates this blog's drafts (Z.ai's own GLM serving — for the record, the incident was Cloudflare's deployment, not Z.ai's): endpoint and auth verified, but my key's balance was empty, so the full run waits on a recharge. The script above is exactly what I'll run the day it has one — and what you can run against your provider tonight, with your key. If you run the engine yourself The operator checklist from the post, in priority order: upgrade to SGLang v0.5.22+ (rejected requests are retired at the admission queue, before any KV transfer); make your KV pool larger than your context window — and add the pre-deployment check that compares measured pool size to context length, because GLM-5.3 shipped without it; enable --disaggregation-enable-kv-checksum so every prefill→decode transfer is verified. The bug you're patching was reported in July, exploited-by-accident in August, and disclosed properly in October — if you deployed SGLang from a summer image, you are the audience for that checklist. TL;DR A KV pool sized at 65% of the context window plus a rejection path that kept going after refusal equals 155 cross-customer leaks with HTTP 200. Root-caused in 25 minutes, fixed in a day, disclosed with page-level diagrams — and the engine bug underneath is generic, so if you self-host inference, this is your patch-week. Everyone else: the canary script above is five minutes against your provider. Zero errors on the wire is not the same as zero leaks. Sources: Cloudflare — KV cache exposure remediation · SGLang issue #30233 · SGLang PR #38704 (Cloudflare draft)

Your LLM router may have a 30-second timeout you never configured A request can have a generous application timeout and still fail while its body is being forwarded. SGLang PR #45696 provides a concrete example: a default in reqwest put a 30-second TCP_USER_TIMEOUT underneath the router's request_timeout. The trigger was backpressure during upload. An engine's HTTP process stopped reading incoming request bodies while it handled a burst of long prompts. Once the sender's buffers filled, the forwarding connection stalled. The kernel eventually returned ETIMEDOUT, and the router answered 504 with x-router-error-code: upstream_timeout. Raising the application timeout did not remove that lower-level timer. The PR author reports roughly 1,000 new 100K-token prompts arriving together, followed by outstanding sends failing around 30.7 seconds after the burst. TCPAbortOnTimeout increased alongside the failures. Those are the author's observations; I have not independently reproduced the workload. The regression test is much smaller and requires no model: accept a connection, read nothing for 40 seconds, then consume a 16 MiB body and return HTTP 200. On the author's Linux aarch64 system, the old build failed at 30.51 seconds. The patched build returned 200 at 40.03 seconds. The slow test is ignored by default, so it must be selected explicitly: cargo test --test proxy -- --ignored reads_nothing The patch calls .tcp_user_timeout(None) when constructing the forwarding client on Linux, Android, and Fuchsia. SGLang's request timeout and stream idle/stale limits still apply. The inspected lockfile resolves reqwest 0.12.28, whose versioned source confirms the 30-second default on those targets. This is specifically a stalled-send problem. It does not imply that every inference request lasting more than 30 seconds will fail. A small body that the engine consumes promptly takes a different path through the timeout conditions. For an incident with a sharp band of failures around one duration, I would inspect socket options alongside application settings. Record when the body stops making progress, check TCPAbortOnTimeout, and distinguish connect, upload, response-header, and stream-idle deadlines. A larger request_timeout cannot override an earlier deadline elsewhere in that path. Disabling TCP_USER_TIMEOUT everywhere would also be a poor general fix: a dead peer still needs a bounded lifetime. Here the router already has application-level limits, and the change removes a competing socket deadline. The fix merged on October 22, 2026 at 04:43:55 UTC. SGLang v0.5.22 was released before it; the confirmed scope here is the affected pre-fix main-branch forwarding client. The PR reports 224 proxy tests passing with the slow test ignored and 36 library tests passing, but its extra CI run failed, so I would not describe the full presubmit as green. Sources: sglang#45696 github.com/seanmonstar/reqwest/blob/v0.12.28/src/… github.com/sgl-project/sglang/commit/80263b558081… Disclosure: AI-assisted writing and source review. All experimental results above are attributed to the upstream author.

Locality of reference helped 94GB Qwen3.8-Flash-Next runs at 68 tok/s on one RTX 5090 llama.cpp merged PR #29887 two weeks ago and I think it is the most interesting thing to land in local inference this year, so I went through the PR, the benchmarks and the review thread and wrote up how it actually works. Short version below, long version linked at the end. The problem Qwen3.8-Flash-Next at Q4_0 is 93.7 GiB, and 65.4 GiB of that is experts. It is 125B total but only 6B active: 512 experts per layer, the router picks 10 (plus 1 shared) per token. One expert is 3 x 2560 x 640 = 4.9M params = about 2.6 MiB at 4-bit. 10 experts x 48 layers = 480 experts per token = about 1.3 GB of weights touched per token. The observation Expert routing has locality of reference. Token N+1 mostly picks the same experts as token N. A handful of experts are hot, most are almost never touched. That is the same property your CPU cache, your OS page cache, your browser cache, every CDN and Redis allkeys-lru all rely on. The fix PR #29887 (by am17an, a port of Tether’s qvac-fabric work, merged by ggerganov on Oct 7) sets aside a slice of VRAM as an LRU cache of experts. Hit: compute from VRAM, zero bytes moved. Miss: copy that one 2.6 MiB expert over PCIe and evict the least recently used one. The log prints the hit rate at exit. Numbers from the PR description, same model file, same machine: GPU |before |with cache |hit rate RTX 4090 24 GB |25.0 tok/s |40.7 tok/s (1.62x) |77% RTX 5090 32 GB |30.8 tok/s |67.8 tok/s (2.20x) |89% AMD R9700 32 GB (a reviewer’s card) |11.0 tok/s |27.6 tok/s |94.6% The 5090 gains more for two reasons: PCIe 5.0 halves the cost of every miss, and 32 GB means a bigger cache and fewer misses. How to use it llama-server -m Qwen3.8-Flash-Next-Q4_0.gguf -fa on -c 32768 -b 2048 -ub 2048 -cmoe --moe-cache-mib 11000 The PR author’s rule of thumb is about 10% of your expert bytes (65.4 GiB -> 6544 MiB). Env var form is LLAMA_ARG_MOE_CACHE_MIB. SGLang merged the same idea 12 days later as “paged experts” in PR #42557 The part I find most interesting is LRU , Locality of reference go back to 1960s. And i find locality of reference principle highly underrated. Full write-up with the memory math script, an expert cache simulator you can run, and the benchmark table: cloudmash.blog llama.cpp PR #29887, MoE expert cache: llama.cpp#29887 SGLang PR #42557, paged experts: sglang#42557