sglang

SGLang's RadixAttention treats the KV cache as a radix tree of tokens across requests. For multi-turn chats and few-shot evals, it reuses prompt KV cache instead of recomputing from scratch. Cache hit rates regularly cross 70%. sgl-project/sglang

When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure TL;DR - What was reported: On 2026-10-26, Cloudflare and the SGLang team disclosed that Cloudflare's GLM-5.3 deployment on Workers AI returned 155 responses containing data from other customers' requests between 2026-08-27 and 2026-09-09. Cloudflare says all 155 were answers to prompts longer than 681,280 tokens, and that no other Workers AI model was affected. - Two bugs, one question: A configuration gap (prefill KV pool of 681,280 tokens against a 1,048,576-token context window) made an error path reachable. Separately, SGLang's disaggregated serving could free a decode-side KV slot while a launched RDMA write was still on its way to it. Both come down to: when is a slot really free? - The invariant: A slot is safe to reuse only when no remote writer can still touch it. The SGLang write-up calls the abort race "a distributed use-after-free." - The fix has a formula: hold time must exceed abort propagation + remaining chunks + the call already in flight. If the acknowledgement path dies, every abort waits out the full timeout, and held capacity is roughly abort rate x hold time x pages per request. - What to do today: Check pool size against context length after every config change, run SGLang v0.5.21 or later for PD disaggregation, pair aborts at the router, and watch the release outcome metric. Scripts below. What actually happened in the Cloudflare and SGLang disclosure? Cloudflare's report describes a model that launched on 2026-08-27 with a configured context window of 1,048,576 tokens. The prefill KV cache pool could hold only 681,280 tokens. That is 65.0% of the window, a gap of 367,296 tokens or 35.0%. A request could therefore fit the context window and still exceed the memory available to process it. In Cloudflare's words, nothing forced the two numbers to agree. On the engine side, the SGLang write-up says that in disaggregated prefill/decode serving, "a request marked for rejection could still proceed through the handoff. Decode could then read cache pages that had not been populated for that request." Cloudflare describes the same thing from the outside: the rejected request was replaced by a one-token placeholder, decode got a "transfer complete" message, and it assumed prefill had overwritten the old cache contents. Prefill had not, so stale pages from earlier requests survived. !Chart: prefill KV pool of 681,280 tokens against a 1,048,576-token context window The timeline, from the two primary documents: | Date (2026) | Event | |---|---| | Aug 27 | Exposure window opens (GLM-5.3 launch on Workers AI) | | Sep 9 | Cloudflare disables public access to the model, notifies SGLang maintainers, restores service that evening after a larger pool and an engine patch | | Sep 16 | SGLang merges the intake admission fix (#38935) and optional KV transfer checksums (#39500) | | Oct 2 | v0.5.21: deferred KV release on by default for Mooncake and NIXL | | Oct 3 | Generation-tagged acks and MORI support (#37077), shipped in v0.5.22 | | Oct 26 | Coordinated disclosure | One scope note matters for your risk assessment: the SGLang post says default PD configurations were affected from the v0.4.x line until v0.5.21 (v0.5.22 for MORI), and that the fix line was not backported to earlier release branches. A single-instance deployment without the cross-engine transfer path is not affected by the abort race. Why can a KV slot be "free" and still get written to? The decode engine pre-reserves destination slots for each incoming request, and its allocator marks them allocated or free. The disclosure's key observation is that "free" has a second, hidden condition. The allocator's bit says nothing about writes that were already posted to a NIC queue. The failure takes four steps: 1. A client disconnect, admission rejection, or transfer fault aborts request A while at least one launched write has not landed. 2. Decode frees A's slots. The allocator marks them free. 3. The allocator hands those slots to request B. 4. A's late write lands inside B's KV cache. B decodes on foreign keys and values, with no error anywhere. The write-up gives four facts that make step 2 to step 4 possible, and each is worth knowing if you run any disaggregated stack: - The allocator can mark a slot free while writes are in flight. It and the NIC do not share state. - Launched RDMA work cannot be cancelled. There is no call to cancel one work request already posted to the queue pair, and Python cannot interrupt a blocking C++ transfer call mid-flight. A pull design does not escape this, because a posted READ is as hard to take back as a posted WRITE. - Router cancellation helps but cannot finish the job. An abort is only safe when both halves of a prefill/decode pair learn about it, yet whatever the transfer engine already launched continues regardless. - The window scales with context length. A short prompt is in flight for milliseconds. A 128K to 1M-token context keeps writes in flight for seconds to minutes, so the same abort rate produces far more exposure on long-context traffic. The corruption is silent by design. The transfer reports success and no component logs an error. A single contaminated slot derails a whole generation because attention reads every position. In the team's 2xH200 reproduction, the victim counted correctly up to 80 and then its output turned into the donor's prompt text. The race is probabilistic in production, but a client can raise the odds by sending long-context requests and disconnecting mid-transfer. The client cannot choose the victim and learns nothing back, so the impact is one-way leakage plus an integrity problem. How likely is corruption if you free slots at abort time? I wrote a small Monte Carlo to see how the three ingredients interact. This is an illustrative model with stated assumptions, not a measurement of SGLang. The core of it is below; the assumptions are: - KV size of 37.5 KB per token, derived from the one data point in the post: a single call for an 800K-token request carried 30 GB (taking GB as 10^9 bytes). - 8,192-token chunks at an effective 25 GB/s, so one chunk write takes about 12.3 ms. - Prefill at 20,000 tokens/s, so 128K tokens take 6.6 s and 1M tokens take 52.4 s. - The allocator reuses freed slots after an exponentially distributed delay. The x-axis is the mean of that delay, standing in for memory pressure. - Every abort lands mid-transfer, as the disclosure's conditions require. import random def p_contam_free_at_abort(tail_fn, mean_reuse, n=200_000): """Slots are freed at abort and reused after Exp(mean_reuse). Corruption if a launched write lands after the reuse.""" bad = 0 for _ in range(n): reuse = random.expovariate(1.0 / mean_reuse) bad += tail_fn() > reuse return bad / n chunk_t = 8192 * 37_500 / 25e9 # one chunk write, about 12.3 ms tail_notify = lambda: chunk_t * random.random() # abort reached prefill tail_silent = lambda ctx: random.random() * ctx / 20_000 # prefill keeps pushing !Chart: share of mid-transfer aborts that corrupt a neighbor when slots are freed at abort Two readings follow. When the abort does not reach prefill, the tail is every chunk prefill still has to push, so corruption is near certain unless reuse takes seconds. Notifying prefill (SGLang's #27372) shrinks the tail to one in-flight chunk and collapses the curve, but at a 10 ms mean reuse delay my model still corrupts 43% of mid-transfer aborts. Notification narrows the window. Only holding the slots closes it. The disclosure frames it the same way: the router decides who dies, and the engine decides when memory is safe. What does the fix hold, and for how long? Decode cannot cancel prefill's launched writes, so it keeps the aborted request's slots reserved for longer than those writes can keep landing: T_hold > t_abort_propagation + t_remaining_chunks + t_in_flight_call The design makes every abort path hold the slots, then bounds each term on the right: 1. Notify prefill on abort (#27372). Decode sends ABORT to every prefill rank, and each rank drops chunks still waiting in its software queue. That drives t_remaining_chunks toward zero. 2. Hold, do not free (#35049, NIXL port #35360). On abort, decode holds the request's KV slots, its row in the request pool, and its metadata buffer entry. Each prefill rank counts outstanding work per room and replies ABORT_ACK only when that count drains to zero. Decode releases when every notified rank has acknowledged, or when a timeout fires. The default timeout in that PR is 30 seconds. 3. Gate by capability, then default on (#37454, #41023). The hold only helps on backends that acknowledge, so it shipped opt-in and became the default in v0.5.21. The timeout is described as "a practical backstop, not a proof", because a call that times out can leave writes the engine is still retrying. That is why #43719 (merged 2026-10-12) makes prefill withhold the drain ack for any room whose send failed. Before it, a failed Mooncake send still acked as drained. The design rule in the conclusion is worth copying: make every degradation lengthen the hold. How long should the timeout be? The post recommends 60 to 120 seconds for SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT, and says a larger value "adds little safety and multiplies held KVCache during abort storms." I checked the shape of that claim with a second illustrative model: 5% of aborts take a degraded path where writes keep retrying past the 30 s transfer timeout (30 s plus an exponential with a 20 s mean), and the release is timeout-only. !Chart: corrupted aborts per 10,000 as a function of hold time Below the transfer timeout the hold is useless on the degraded path (about 500 per 10,000 corrupted, which is just the 5% degraded share). At 60 s it falls to 113, at 120 s to 6, and at 240 s to 0 in 200,000 trials. The tail decays exponentially under my assumption, so the second doubling buys almost nothing while it doubles the cost below. Measure your own retry distribution before copying a number. What does holding cost in capacity? The post gives the sizing rule: if releases fall back to the timeout, held capacity is about abort rate x T_hold x pages per request. With the healthy ack path, a hold is released milliseconds after the drain and costs nothing noticeable. With a dead ack path, every abort pays the full timeout. !Chart: KV held on decode when every release waits out the timeout | Abort rate | T_hold | Held KV at 128K-token reservations (4.8 GB each) | |---|---|---| | 1 per minute | 30 s | 2.4 GB | | 1 per minute | 120 s | 9.6 GB | | 6 per minute | 30 s | 14.4 GB | | 6 per minute | 120 s | 57.6 GB | At the 37.5 KB/token figure, one 800K-token reservation is 30 GB, so a handful of long-context aborts waiting out a 120 s hold can pin a large share of a decode pool. This is why the metric below matters as much as the fix itself: a dead ack path turns a safety feature into an admission-latency problem. The SGLang team measured the steady-state price as small: across 21 runs on 2xH100 (1P1D Mooncake, 1024 requests at concurrency 256), throughput was statistically indistinguishable with the feature on or off, all deltas within plus or minus 2%. The hold only applies on the abort path. Which pre-flight check catches the configuration half? Cloudflare's first recommendation is to measure the KV pool against context length after every configuration change. That check needs no GPU. It needs two numbers from your engine: the token capacity of the prefill KV pool and the configured context length. Field names differ across versions, so read them from your engine's startup log or info endpoint and verify on your build. import re def pool_gate(pool_tokens: int, context_len: int, max_ctx_per_req: int | None = None) -> str: """Fail the deploy if a request that fits the context window cannot fit the prefill KV pool.""" need = max_ctx_per_req or context_len if pool_tokens < need: short = need - pool_tokens return f"FAIL pool={pool_tokens} < context={need} (short by {short}, {short / need:.1%}); lower context_len or grow the pool" return f"OK pool={pool_tokens} covers context={need}" print(pool_gate(681_280, 1_048_576)) # FAIL pool=681280 < context=1048576 (short by 367296, 35.0%); lower context_len or grow the pool Make FAIL block the rollout. It would have flagged the reported configuration on day one. How do you know the protection is actually on? SGLang v0.5.22 ships a counter, sglang:decode_deferred_kv_release_total{outcome}. The post says a healthy deployment releases almost all holds through drained, while a timeout-heavy ratio means the ack path needs attention. Here is a parser and a verdict function; I ran it against the synthetic sample shown, whose numbers are made up. PAT = re.compile(r'sglang:decode_deferred_kv_release_total\{([^}]*)\}\s+([0-9.eE+-]+)') def release_outcomes(metrics_text: str) -> dict: out = {} for labels, val in PAT.findall(metrics_text): m = re.search(r'outcome="([^"]+)"', labels) if m: out[m.group(1)] = out.get(m.group(1), 0.0) + float(val) return out def verdict(outcomes: dict, min_drained: float = 0.95) -> str: total = sum(outcomes.values()) if total == 0: return "NO DATA: no holds yet, or the metric is missing on this build" drained = outcomes.get("drained", 0.0) / total return ("HEALTHY" if drained >= min_drained else "ACK PATH SUSPECT") + f" drained={drained:.1%} of {int(total)} holds" sample = '''sglang:decode_deferred_kv_release_total{outcome="drained",tp_rank="0"} 9712 sglang:decode_deferred_kv_release_total{outcome="timeout",tp_rank="0"} 41 sglang:decode_deferred_kv_release_total{outcome="error",tp_rank="0"} 3''' print(verdict(release_outcomes(sample))) # HEALTHY drained=99.5% of 9756 holds The 0.95 threshold is my choice, not SGLang's. Set yours from your own baseline after rollout. Beyond the counter, the engine offers an opt-in integrity check: --disaggregation-enable-kv-checksum on both prefill and decode (their KV layouts must match). It computes an Adler-32 checksum on the GPU and verifies the received cache before the request joins a running batch. It is off by default because each transfer pays an extra GPU read and a synchronization point, and it catches only late writes that land before the commit point. The hold covers the rest. Operators still on v0.5.19 or v0.5.20 are told to set SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1 on both servers, plus SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES on v0.5.20. The post is explicit that this does not replace upgrading, because failure kinds other than decode-initiated aborts stay unprotected before v0.5.21. Does this apply to engines other than SGLang? The write-up says the four steps apply to "any PD implementation that moves KVCache between engines with asynchronous DMA, push or pull." It does not say other engines are affected, and I have not tested any. Ask three questions of your own stack, whichever engine it runs: - When a request dies mid-transfer, what delays the allocator's reuse of its destination slots? - Is that delay tied to a positive signal that the writer finished, or only to a timer? - If the signal path breaks, does the system get safer (longer hold) or less safe (earlier release)? If you cannot answer these from the code or docs, you have found your next audit item. FAQ What exactly leaked in the Cloudflare incident? Cloudflare reports 155 responses that contained data from other customers' requests, all to prompts longer than 681,280 tokens, between 2026-08-27 and 2026-09-09. Only GLM-5.3 on Workers AI was affected, per its investigation. Is my single-node SGLang deployment affected? The SGLang post says a single-instance deployment without the cross-engine KV transfer path is not affected by the abort race. The exposed path is PD disaggregation with Mooncake, NIXL, MORI, or a custom transfer backend on affected versions. Which version should I run? Per the post, v0.5.21 protects Mooncake and NIXL by default, v0.5.22 covers MORI, and the fix was not backported. Some press coverage says the fix landed earlier; I follow the engine team's own release mapping. Can I tell whether I was exposed before upgrading? Not reliably. The corruption leaves no error or log line, and the opt-in checksum only helps going forward. Does a bigger timeout make it safer? Only until it exceeds the longest realistic write tail. Past that it mostly increases held capacity during abort storms. Are the charts measurements? The cover uses the reported pool and window sizes. The other three come from my own simulation, so read them as shapes, not SGLang numbers. Sources - SGLang security update: improving request isolation (LMSYS, 2026-10-26) - Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang (Cloudflare) - SGLang pull requests cited: #38935, #39500, #32758, #37077 - SGLang releases: v0.5.21, v0.5.22 - Simulation assumptions: KV 37.5 KB/token (30 GB per 800K-token call, from #32758's tests as quoted in the post), 8,192-token chunks at 25 GB/s, prefill 20,000 tokens/s, exponential reuse delay, 5% degraded aborts with a 30 s timeout plus a 20 s-mean retry tail, 200,000 trials per point, seed 7. Further reading - Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off - Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

3/6 Two: with a regex stop condition, the matched text is left in the response and `no_stop_trim` is ignored. The trimmer looks for the regex pattern text literally, so it never matches. 👉sglang#47461

2/6 One: under memory pressure the scheduler automatically retracts a running request and re-prefills it with its sampling-penalty history reset to zero. So it silently produces a different answer. Sibling repos already fixed this; core hasn't. 👉 sglang#47245

NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit Serving large language models at scale is fundamentally a memory problem. During autoregressive decoding, every generated token requires reading the entire weight matrix and the accumulated KV cache from GPU high-bandwidth memory (HBM). On an H100, that bandwidth ceiling is around 3.35 TB/s — fast, but finite. As context windows grow and batch sizes increase, the gap between what the GPU can compute and what it can actually move through memory widens. Lossless weight compression is one lever practitioners can pull to ease this bottleneck. The idea is simple: store weights in a smaller format, decompress on the fly during inference, and net a bandwidth gain. The catch is that decompression itself must be cheap enough not to eat the savings. A new paper from researchers at Stanford and Together AI introduces NyoomFloat12 (NF12), a 12-bit fixed-length format that threads this needle by exploiting a structural property of trained BF16 weights that turns out to be nearly universal. The Redundancy Hidden in BF16 Weights BFloat16 uses 1 sign bit, 8 exponent bits, and 7 mantissa bits. The exponent field is where the redundancy lives. In a trained model, the distribution of weight magnitudes is tightly concentrated — most weights are small, and after applying a per-matrix power-of-two scale to normalize the distribution, the upper four bits of the exponent field are 0111 for more than 99.7% of weights across every tested LLM. Those four bits are constant. Storing them is waste. NF12 drops them. The result: eight BF16 weights (128 bits) pack into 96 bits — 12 bytes instead of 16. That is a 25% reduction in weight storage, achieved without any approximation. For the small fraction of weights whose exponent falls outside the expected range, NF12 uses an escape mechanism. A zero meta byte signals that a group of eight weights is stored verbatim in a per-matrix BF16 buffer. The overhead is negligible, and the format remains strictly lossless. Why Fixed-Length Matters for GPUs The prior art in lossless LLM weight compression — most notably DFloat11, presented at NeurIPS 2025 — uses Huffman coding on the exponent bits. Huffman achieves slightly higher compression (around 10.79 bits per weight versus NF12's 12.02), but it produces variable-length encodings. Variable-length codes are a poor fit for SIMT (Single Instruction, Multiple Threads) GPU architectures. When threads in a warp decode weights of different lengths, they diverge — some finish early and stall while others catch up. The decompression kernel becomes serialized, and the bandwidth savings evaporate into compute overhead. DFloat11's decode kernel takes around 1,771 microseconds on the benchmarks reported in the NF12 paper. NF12's fixed-length decoder takes 548 microseconds — a 3.2× speedup in decompression alone. NF12's decode path is designed around GPU warp execution. A warp reconstructs groups of weights in lockstep: 13 register operations plus a three-instruction escape check. The common path costs less than a standard BF16 memory copy. That is the key property — the decoder is not just fast, it is faster than the alternative of not compressing at all. Two Regimes, Two Wins The NF12 paper evaluates performance across two distinct inference regimes, and the format delivers in both. Bandwidth-bound inference (small batch, short context): Here the bottleneck is how fast weights can be streamed from HBM to the compute units. NF12's fused GEMV (general matrix-vector multiplication) kernel achieves 1.26× the throughput of a custom BF16 GEMV and 1.35× over cuBLAS on an H100. The 25% smaller weight footprint translates almost directly into a proportional bandwidth gain, minus the small decode overhead. Capacity-bound inference (large batch, long context): Here the bottleneck is total GPU memory. The KV cache competes with model weights for HBM space, and as context grows, the cache wins — crowding out concurrency. By shrinking the weight footprint, NF12 frees HBM for KV cache. On Qwen3-32B, NF12 provides 4.5× more KV cache capacity compared to BF16. At high concurrency, this translates to 2.54× higher rollout throughput — the headline number from the paper. That 2.54× figure is not a cherry-picked benchmark. It reflects a realistic serving scenario where many sequences are processed simultaneously and the KV cache is the binding constraint. Practical Implications for Inference Engineers NF12 is a drop-in optimization for existing model weights. No retraining, no fine-tuning, no accuracy degradation — the format is lossless by construction. The compression step is a one-time offline operation. At inference time, the fused GEMV kernel handles decompression transparently. The format is most impactful in two situations practitioners encounter regularly: 1. Fitting larger models on existing hardware. A 32B model in NF12 occupies roughly the same HBM as a 24B model in BF16. That headroom can be used for longer contexts, larger batches, or simply running a model that previously required an extra GPU. 2. Improving throughput at long context. As context windows push into the hundreds of thousands of tokens, KV cache pressure becomes the dominant constraint. NF12's capacity gains compound with other KV cache optimizations like grouped query attention and prefix caching. The comparison with DFloat11 is instructive for practitioners choosing between lossless methods. DFloat11 compresses slightly more aggressively and is well-suited for scenarios where the goal is fitting a model into a fixed memory budget and throughput is secondary. NF12 is the better choice when end-to-end inference speed is the objective — its fixed-length decoder keeps the GPU's SIMT execution efficient. What This Tells Us About the Inference Optimization Landscape NF12 is part of a broader pattern in 2026 inference research: the recognition that the memory wall is the primary constraint, and that solutions must be co-designed with GPU architecture rather than borrowed from general-purpose compression. The Invariant Bit Packing (IBP) work from earlier this year takes a similar approach for PCIe transfers. The common thread is that compression formats designed for CPUs — Huffman, ANS, LZ4 — do not map cleanly onto SIMT execution models. NF12's contribution is demonstrating that a carefully chosen fixed-length format can beat variable-length alternatives on end-to-end throughput even when it compresses less. The 25% size reduction is not the point; the SIMT-aligned decoder is. For teams running inference at scale, NF12 is worth evaluating as a complement to quantization-based approaches. Unlike INT8 or FP8 quantization, it introduces zero approximation error — useful in domains where numerical fidelity matters, or when quantization has already been applied to the KV cache and further lossy compression of weights is undesirable. The paper is available on alphaXiv. The authors are Liberman, Fang, Zhang, Dao, and Fu — several of whom have prior work on FlashAttention and efficient GPU kernels, which explains the focus on SIMT-aligned design. For teams already using vLLM or SGLang for serving, NF12 is designed to integrate at the kernel level without changes to the serving framework's scheduling or batching logic. The authors plan to release the CUDA kernels as open source, which should make adoption straightforward once the code is available.

vLLM・SGLang・TensorRT-LLMをクラウドGPUで動かしたら、それぞれの個性が見えてきた この記事で分かること Qwen3-8Bを、L4 24GB 1台の上でvLLM・SGLang・TensorRT-LLMに載せ、同じリクエストを送った。揃えたつもりの条件が、5か所で揃っていなかった。どれもベンチマークの数字に入り込む。 速さの順位は載せない（速さは自分の条件で測ってほしい）。すべてこの版・L4（sm_89）での話である。エンジンを選びたいだけなら「エンジン別の要点」と最後の「私の使い分け」へ。 エンジン別の要点 エンジン 知っておくこと vLLM 0.31.0 L4ではQwen/Qwen3-8B-FP8が重みだけFP8（W8A16）で動く SGLang 0.5.21 ツール定義に項目が足され、入力が増える。L4・8B bf16は--mem-fraction-staticの自動値では起動しない（0.87で動いた） TensorRT-LLM 1.2.1 L4ではQwen/Qwen3-8B-AWQ・-FP8をそのまま読めない。KVが足りないと黙って待たせる。キャッシュを消すAPIが無い 用語 用語 意味 KVキャッシュ 生成中のリクエストが、それまでのトークンの計算結果を置いておくGPUメモリ。容量をトークン数で数え、同時に処理できる本数の上限を決める プリフィル 入力（プロンプト）をまとめて読み込む最初の段階。入力が長いほど重い TTFT Time To First Token。送ってから最初の1トークンが返るまでの時間 W8A16／W8A8 重み（Weight）と計算時の値（Activation）のビット数。W8A16は重みだけ8bitで計算は16bit 追い出し（preemption／retraction） KVが足りなくなったとき、処理中のリクエストをいったん外すこと。後でやり直す RadixAttention SGLangのプレフィックスキャッシュ。木構造で、任意のトークン位置まで一致を使い回す（vLLMは決まった長さのブロック単位） sm_89／Ada／Hopper NVIDIA GPUの世代。L4はAda（sm_89）、H100はHopper（sm_90） ModelOpt NVIDIAの量子化ツール。TensorRT-LLM向けの量子化モデルを作る 条件 項目 値 モデル Qwen3-8B（bf16、AWQ 4bit、FP8） GPU NVIDIA L4 24GB 1台（sm_89、AWS g6.xlarge）。ドライバ 595.112.07、CUDA 13.2 エンジン vLLM 0.31.0 / SGLang 0.5.21 / TensorRT-LLM 1.2.1（trtllm-serve、PyTorchの経路） リクエスト /v1/chat/completions。ツール定義10個（ケースごとに40個から選ぶ）＋日本語の質問。システムプロンプトなし、thinkingオフ、質問の末尾にリクエストごとの識別子 max_tokens／最大の文脈長 512／8,192 測った並行数 AWQ・FP8は1と16。bf16は1（16は載らない：3節）。SGLang bf16は容量の確かめで8も 送ったケース 200ケースのうち20ケースを繰り返し送った（並行数1で80件） プレフィックスキャッシュ 3エンジンとも既定で有効。区切り（エンジン×チェックポイント×並行数×測定の種類）ごとにサーバーを起動し直す エンジン別 vLLM SGLang TensorRT-LLM 入力長 最大／中央値（200ケース） 2,591／2,310 2,691／2,410 2,591／2,310 ツールパーサー hermes qwen（Qwen25Detector） qwen3 バッチの上限 max_num_seqs 256・max_num_batched_tokens 2048 max_running_requests 2048 max_batch_size 2048・max_num_tokens 8192 チャンク化プリフィル 有効 有効（4096） 無効 CUDA graph FULL_AND_PIECEWISE 有効 バッチ1〜32・64・128 バッチの上限はどれも並行数16より大きい。 1. 入力：SGLangだけトークン数に差が出た 同じリクエストでusage.prompt_tokensを比べると、SGLangだけ全ケースで+100トークンだった。 // 送ったもの（{…}は省略） {"type": "function", "function": {"name": "search_master", "description": "…", "parameters": {…}}} // テンプレートに渡るもの {"type": "function", "function": {"description": "…", "name": "search_master", "parameters": {…}, "strict": false}, "defer_loading": null} エンジン toolsの扱い prompt_tokens vLLM 型に通すが、テンプレートに渡す長さは変わらない 送ったまま手元でテンプレートに通した長さと一致 SGLang model_dump()で項目を足し、キーの順が変わる +100 TensorRT-LLM — vLLMと一致 書き換えを再現する関数を書き、1,200件でSGLang自身のmodel_dump()と突き合わせて違いは0件だった。確かめるなら、テンプレート適用後の文字列（vLLMなら/tokenize）の差分を取る。 切り替えるオプションは無い。 揃えるならクライアントから足した形で送る（他エンジンの入力も変わる）。今回はそのまま測った。 効き方：プリフィルが重くなる／KVの見積もりがずれる（3節）／精度にも混ざる。ツールが多いほど差は大きい。パーサーも3つとも別の実装で、読み損ねは精度の差として出る。 2. 量子化：「FP8」が同じ意味ではない エンジン 起動時の警告 vLLM Weight-only FP8 compression will be used leveraging the Marlin kernel. SGLang Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! 「FP8で比べた」表は、重みだけ8bitと計算も8bitが混ざりうる。起動ログのカーネル名で確かめる。 vLLMがL4でW8A16になるのは、カーネルの候補を順に試し、最初に使えるものを選ぶため（vllm/model_executor/kernels/linear/init.pyのinit_fp8_linear_kernel）。 CUTLASSのブロックFP8のカーネルはsm_90・sm_100・sm_120向けしか無い（csrc/.../c3x/scaled_mm_blockwise_sm*_fp8.cu）。--linear-backendで変えられるはずだが試していない。 TensorRT-LLM 1.2.1・L4では、公開の次の2つを読めなかった。 チェックポイント 読めない理由 代わり（ModelOpt 0.37.0） Qwen/Qwen3-8B-AWQ（AutoAWQ形式） load_hf_quant_configにawqの分岐が無い --qformat int4_awq Qwen/Qwen3-8B-FP8（ブロック単位） 対応表で、ブロック単位FP8はAda（sm_89）が対応外（HopperとBlackwell sm100/103は対応）。モデル側の表でもQwen3は対応外なので、H100でも使えない --qformat fp8 --kv_cache_qformat none 2つで約56分かかった。nvidia/Qwen3-8B-FP8はKVまでFP8になるため使わなかった（KVの型の上書きは試していない）。TensorRT-LLMだけチェックポイントが別物なので、量子化した構成はエンジン間で比べず、各エンジン内でbf16との精度差だけ確かめた。 3. 容量：同じGPUでも載る量が違う 名前の似たメモリ設定が、何に対する割合かが違う（L4は約22.5GiB〔23,034MiB〕、Qwen3-8B bf16の重みは15.27GiB）。 | weights | KV | act | rest | GPU |===============|==============|=========|======| vLLM |<---------------- 0.92 ---------------->| SGLang |<----------- 0.87 ----------->| TRT-LLM |<-------- 0.9 x free --------->| act＝活性化など計算中の作業領域（プリフィルの中間結果、CUDA graph）。rest＝エンジンの外で使われる分の余白。模式図で、長さは比率どおりではない。 エンジン 割合の対象 割合に含むもの 値 KVの容量（トークン） 容量を直接指定するなら vLLM GPU全体 重み＋KV＋活性化 0.92（既定） 30,240 --kv-cache-memory-bytes（指定すると0.92は無視される） SGLang GPU全体 重み＋KV（活性化は外） 0.87（手で決めた） 26,813 --max-total-tokens TensorRT-LLM 重みと活性化を確保した後の空き KVだけ 0.9（既定） 約32,800 kv_cache_config.max_tokens（割合と両方なら小さいほう） 揃えたいなら割合ではなくトークン数で直接指定する（オプション名は版で変わりやすい）。今回は既定に近い状態を測るため揃えていない。 KV 1トークン = 36層 × KVヘッド8 × 128次元 × K/V 2 × 2バイト = 144KiB 1本の最大 = 入力2,691（SGLang） + max_tokens 512 + 余裕1 = 3,204 並行数8 = 3,204 × 8 = 25,632（+100前は 3,104 × 8 = 24,832） 並行数16 ≒ 51,300 → bf16は3エンジンとも載らない 全リクエストが最大長まで伸びても入るかで数え、入らない組み合わせは測っていない。 SGLangは自動値（0.684）で起動しなかった。 自動値＝（GPUの容量−作業領域の見積もり）÷GPUの容量（srt/arg_groups/memory_hook.py）。 作業領域の見積もり（L4） 大きさ 固定の分 512MiB プリフィル chunked_prefill_size 4,096トークン×1.5MiB＝6GiB（係数1.5はソースで「経験的な値」） 並列化の分 128MiB（1GPU） CUDA graph デコードの最大バッチ48×2MiB＋プリフィルの分 合計 約7.1GiB（23,034MiB×(1−0.684)） ValueError: Loaded weights leave no GPU memory for the KV cache under --mem-fraction-static=0.684. 並行数8が載る値を0.01刻みで探した（0.01あたり約1,590トークン）。 割合 KVの容量 並行数8（25,632） KV以外の残り 追い出し 0.684（自動） 起動しない — — — 0.84 22,047 載らない 1.82 GB — 0.86 25,224 載らない 1.40 GB — 0.87 26,813 載る 1.11 GB 0回 +100前なら0.86で足りたが、1節の+100で408トークン足りなくなり0.87にした。0.88以上は試していない。 4. 溢れたとき：追い出すか、黙って待たせるか 方式 エンジン 混んだとき遅れる場所 見える指標 追い出す vLLM・SGLang 生成の途中（ストリーミングが止まる） vllm:num_preemptions／sglang:num_retracted_requests_total 待たせる TensorRT-LLM 最初の1トークン（TTFT） 無い。numActiveRequestsで見る TensorRT-LLMは「入力長＋max_tokens」ぶんを先に確保するので、max_tokensを大きく付けるほど同時に受け入れる本数が減る。並行数16で送っても中では8本しか走っていないことがある。空きに合わせて詰め込むMAX_UTILIZATIONもある（処理中のリクエストが一時停止しうる。試していない）。 今回は/metricsを2秒ごとに読み、numActiveRequestsの最大が送った本数に届いたことを確かめた。溢れる組み合わせは測っていない（3節）。 5. 観測：そのまま読むと誤判定になる エンジン 癖 対処 SGLang キャッシュヒットが0だとprompt_tokens_detailsごと出ない 無ければ0 SGLang num_retracted_requests_totalは初回の追い出しまで出ない ゲージnum_retracted_reqsがあれば0回 TensorRT-LLM kvCacheStatsはenable_iter_perf_statsが要り、1件処理するまで空 短いリクエストを1件送ってから読む TensorRT-LLM その1件で、測定の最初のリクエストがcached_tokens=1になった（<|im_start|>を共有。partial reuseが既定で有効なので、32トークンのブロックの一部でも当たる） 書式を通さない/v1/completionsで送る TensorRT-LLM 反復ごとの統計は読むたびに消える 測定中に2秒ごとに読む 「読めなければ0」にしていたら誤った結論を出していた。「読めなければ止める」作りにしていたので気づけた。 エンジン キャッシュの消し方 vLLM POST /reset_prefix_cache（VLLM_SERVER_DEV_MODE=1） SGLang POST /flush_cache TensorRT-LLM APIなし。再起動 キャッシュがどれだけ効くかは、送るケースの数とKVの容量で大きく変わる。今回は20ケース（入力の合計45,957トークン）を繰り返し送った。 チェックポイント KVの容量（トークン） 20ケースぶんが cached_tokens／入力（3エンジンとも同程度） bf16 26,813〜32,800 収まらず、一周する前に追い出される 3.6〜4.7%（当たるのは先頭の書式 約50トークンだけ） AWQ・FP8 76,096〜99,552 収まり、2周目からツール定義ごと当たる 並行数1で約75%、16で約82% 効いているのは並行数ではなく、量子化で空いたKVの容量である。量子化した構成の速さには、計算が軽くなった分とキャッシュに当たった分が混ざりうる。ケースの数・繰り返し方とKVの容量を並べて確かめたい。 6. 動かすまで 版がぶつかるので、仮想環境をエンジンごとに分け、クライアントはHTTPだけで話す。 エンジン torch transformers vLLM 2.13.0+cu132 5.17.0 SGLang 2.13.0 5.12.1 TensorRT-LLM 2.9.1+cu130 4.57.3 問題 対処 TensorRT-LLM環境でModelOptが動かない（NumPy<2固定と最新pyarrowが衝突） pyarrow<22 TensorRT-LLM FP8が起動中にsegfault（原因不明、再実行で通過） — そのとき親プロセスが残り、20分待ち続けた ログにSegfault encountered等が出たら即失敗にして1回だけ再起動 まとめ：比較を作る・読むときのチェックリスト 確かめること 確かめ方 モデルに届く入力は同じか usage.prompt_tokensを並べる。テンプレート適用後の文字列を比べる ツールパーサーは何か 起動オプションを書く 量子化の計算は何か 起動ログのカーネル名・警告 量子化のチェックポイントは同じか 対応表とソースで、使うGPUで読める形式か確かめる KVの容量はいくつか 割合ではなくトークン数で見る。揃えるなら直接指定する 溢れたら何が起きるか 追い出しカウンタと同時実行数。max_tokensも書く 指標が読めないときの扱い 読めなければ止める キャッシュの状態 有効か、区切りごとに消すか、cached_tokensはいくつか。ケースの数・繰り返し方とKVの容量も書く 私の使い分け（推奨案） 速さではなく、導入の手間・設定の素直さ・観測しやすさで決めた。 当てはまる範囲：24GBクラスのGPU1台・8B級・同時8本程度まで。大きいGPUや70B級ではメモリ（3節）と量子化の対応（2節）が変わる。 推奨 想定するユースケース 決め手 気をつけること vLLM 社内チャットやRAGのPoC。公開モデル（量子化版を含む）を差し替えて評価する 公開チェックポイントをそのまま読める（2）。既定の設定で起動し、追い出しがカウンタで見える（3・4） L4のFP8はW8A16（2） SGLang 同じツール定義・システムプロンプトを先頭に付けるエージェントやバッチ処理 RadixAttention（用語）。L4でもFP8がW8A8（2）。※今回は共有部分が短く、実測ではない（5） 入力が増える（1）。割合を手で決める（3）。指標が0で出ない（5）。共有部分の長い順に処理するlpmは既定ではない（既定fcfs） TensorRT-LLM モデルを固定して長く運用する本番API 処理中のリクエストが途中でやり直しにならない（4）。量子化は最初の1回で済む 量子化を作り直す（2）。混むとTTFTが伸びるので、SLAがTTFTなら自分で測る（4）。キャッシュを消すAPIなし（5） 迷ったらvLLMで基準を作り、チェックリストで条件を揃えてから他と比べる。 付録：起動コマンド vLLM 0.31.0（bf16。AWQ・FP8はモデル名だけ替える。VLLM_SERVER_DEV_MODE=1 を付けて起動） vllm serve Qwen/Qwen3-8B --served-model-name Qwen/Qwen3-8B \ --host 127.0.0.1 --port 8000 \ --enable-auto-tool-choice --tool-call-parser hermes \ --enable-prompt-tokens-details --max-model-len 8192 SGLang 0.5.21 python -m sglang.launch_server --model-path Qwen/Qwen3-8B --served-model-name Qwen/Qwen3-8B \ --host 127.0.0.1 --port 30000 \ --tool-call-parser qwen --enable-metrics --enable-cache-report \ --mem-fraction-static 0.87 --context-length 8192 TensorRT-LLM 1.2.1（ModelOptで作ったAWQ。bf16はQwen/Qwen3-8B） echo "enable_iter_perf_stats: true" > trtllm-serve.yaml trtllm-serve ~/models/Qwen3-8B-int4-awq-modelopt \ --host 127.0.0.1 --port 8001 --backend pytorch \ --tool_parser qwen3 --config trtllm-serve.yaml --max_seq_len 8192 出典 SGLang v0.5.21：github.com/sgl-project/sglang/tree/v0.5.21 （srt/entrypoints/openai/serving_chat.py・protocol.py・usage_processor.py、srt/observability/metrics_collector.py、srt/function_call/function_call_parser.py、srt/arg_groups/memory_hook.py） TensorRT-LLM v1.2.1：github.com/NVIDIA/TensorRT-LLM/tree/v1.2.1 （tensorrt_llm/commands/serve.py、llmapi/llm_args.py、_torch/model_config.py、_torch/modules/linear.py、cpp/tensorrt_llm/executor/jsonSerialization.cpp、docs/source/features/quantization.md） ModelOpt 0.37.0：github.com/NVIDIA/Model-Optimizer/tree/0.37.0 （examples/llm_ptq/hf_ptq.py） vLLM v0.31.0：github.com/vllm-project/vllm/tree/v0.31.0 （vllm/config/cache.py、vllm/engine/arg_utils.py、vllm/model_executor/layers/quantization/fp8.py、vllm/model_executor/kernels/linear/init.py・scaled_mm/marlin.py・cutlass.py・flashinfer.py・deep_gemm.py・b12x.py、vllm/utils/flashinfer.py、vllm/platforms/cuda.py、vllm/model_executor/layers/quantization/utils/w8a8_utils.py、csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/、vllm/entrypoints/openai/chat_completion/protocol.py、vllm/v1/metrics/loggers.py）、起動ログ vLLM プレフィックスキャッシュの設計：docs.vllm.ai TensorRT-LLM スケジューラの方針（GUARANTEED_NO_EVICT／MAX_UTILIZATION）：nvidia.github.io SGLang サーバー引数：docs.sglang.io 、new_token_ratioはsrt/managers/schedule_policy.py nvidia/Qwen3-8B-FP8：huggingface.co PagedAttention：Kwon et al., SOSP 2023。RadixAttention：Zheng et al., NeurIPS 2024

How to Deploy GLM-OCR: Complete Guide for Document Understanding What if you could extract text from complex PDFs, tables, and formulas with a model smaller than most smartphone apps? GLM-OCR achieves state-of-the-art document understanding with just 0.9 billion parameters. It's lightweight enough to run on modest hardware yet accurate enough to top the OmniDocBench V1.5 leaderboard at 94.62 points. {% cta apidog.com %} Try Apidog today {% endcta %} Traditional OCR tools struggle with document structure. They lose table formatting, misread mathematical formulas, and fail on multi-column layouts. Cloud APIs solve these problems but charge per request and send your sensitive documents to third-party servers. GLM-OCR eliminates both issues: it handles complex layouts locally with production-grade accuracy, all under an MIT license that permits commercial use without licensing fees. 💡When building document processing pipelines that need reliable API testing —whether extracting data from invoices, parsing technical documentation , or automating form processing— Apidog streamlines the entire workflow . It provides visual request building, automated documentation generation, and collaborative debugging tools that work seamlessly with your GLM-OCR deployment. Understanding GLM-OCR Architecture GLM-OCR uses a three-component encoder-decoder architecture optimized for document understanding. The CogViT visual encoder processes document images using weights pretrained on billions of image-text pairs. It extracts visual features while preserving spatial relationships critical for understanding layout. A lightweight cross-modal connector sits between encoder and decoder. This component downsamples visual tokens efficiently, reducing computational overhead without sacrificing accuracy. The GLM-0.5B language decoder then generates structured text output, handling everything from plain paragraphs to complex nested tables. The model employs a two-stage inference pipeline. First, PP-DocLayout-V3 analyzes document structure—identifying headers, paragraphs, tables, and figures. Second, parallel recognition processes each region simultaneously. This approach maintains document hierarchy where traditional OCR flattens everything into unstructured text. Training innovations further boost performance. Multi-token prediction loss improves training efficiency by predicting multiple tokens simultaneously. Stable full-task reinforcement learning enhances generalization across diverse document types. The result: 96.5% accuracy on formula recognition, 86.0% on table recognition, and leading performance on information extraction tasks. At inference, GLM-OCR processes 1.86 PDF pages per second on a single GPU—significantly faster than comparable models. The 0.9B parameter count means you deploy on consumer hardware rather than enterprise clusters. !glm-orc model Model Specifications GLM-OCR handles documents up to 8K resolution (7680×4320 pixels). It recognizes 8 languages including English, Chinese, Japanese, and Korean. The model processes both raster images (PNG, JPEG) and vector inputs. Typical inference consumes 4-6GB VRAM at FP16 precision, fitting on consumer GPUs like RTX 3060 or cloud instances like AWS g4dn.xlarge. > | Hardware | VRAM Required | Pages/sec | Use Case | -------------------------------------------------------------------- > | RTX 3060 | 4-6GB | ~1.5 | Development | > | RTX 4090 | 4-6GB | ~2.5 | Production | > | AWS g4dn.xlarge | 16GB | ~1.8 | Cloud deployment | > | 4x A100 (TPS=4) | 80GB | ~7.0 | Enterprise | Local Deployment Options GLM-OCR supports four deployment methods depending on your infrastructure and performance requirements. Each uses the same underlying model weights from Hugging Face but optimizes for different scenarios. - vLLM provides the best balance of throughput and latency for production workloads. It implements PagedAttention for efficient memory management and supports continuous batching for high-concurrency scenarios. - SGLang offers maximum performance through its runtime optimization. It excels at speculative decoding and structured generation, making it ideal when you need the fastest possible inference. - Ollama delivers the simplest setup. One command downloads and runs the model locally—no Python dependencies or configuration files. Perfect for prototyping and personal use. - Transformers enables direct Python integration. Use this for development, debugging, or when you need fine-grained control over the inference pipeline. All methods require the GLM-OCR weights from Hugging Face (zai-org/GLM-OCR). The model runs on NVIDIA GPUs with CUDA support. CPU-only inference works but at significantly reduced speed. Setting Up vLLM for Production vLLM provides production-ready inference with OpenAI-compatible API endpoints. This lets you swap GLM-OCR into existing applications that currently use OpenAI's vision models. Installation Install vLLM with CUDA support: pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly For containerized deployment, use the official Docker image: docker pull vllm/vllm-openai:nightly Install compatible Transformers—vLLM requires the latest development version for GLM-OCR support: pip install git+https://github.com/huggingface/transformers.git Launching the Service Start the vLLM server with GLM-OCR: vllm serve zai-org/GLM-OCR \ --allowed-local-media-path / \ --port 8080 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' The --allowed-local-media-path flag enables the model to access local image files. Set this to your document directory or / for unrestricted access (use with caution in production). The --speculative-config enables Multi-Token Prediction, a GLM-OCR feature that accelerates inference by predicting multiple tokens simultaneously. Client Integration Once running, interact with GLM-OCR through standard HTTP requests: curl --location --request POST 'http://localhost:8080/v1/chat/completions' \ --header 'Content-Type: application/json' \ --data-raw '{ "model": "zai-org/GLM-OCR", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "file:///path/to/document.png"}}, {"type": "text", "text": "Extract all text from this document"} ] } ] }' The OpenAI-compatible response format means existing SDKs work without modification. Point your OpenAI client at http://localhost:8080 and use zai-org/GLM-OCR as the model name. Production Configuration For high-throughput deployments, add tensor parallelism across multiple GPUs: vllm serve zai-org/GLM-OCR \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.95 \ --max-model-len 8192 \ --allowed-local-media-path / \ --port 8080 Adjust --tensor-parallel-size to match your GPU count. Monitor GPU utilization and increase batch sizes to maximize throughput. Monitoring and Scaling Track vLLM performance through its built-in metrics endpoint at /metrics. Prometheus-compatible data includes request latency, queue depth, and GPU utilization. Set up alerts when queue depth exceeds 10 requests or GPU memory hits 90%. For horizontal scaling, deploy multiple vLLM instances behind a load balancer with sticky sessions to maintain context across requests. Consider using Apidog's API monitoring features to track production metrics alongside your model performance. SGLang High-Performance Inference SGLang provides advanced runtime optimizations for maximum inference speed. It excels at speculative decoding and structured generation, making it ideal for latency-sensitive applications. Installation Install SGLang via Docker (recommended for dependency isolation): docker pull lmsysorg/sglang:dev Or install from source: pip install git+https://github.com/sgl-project/sglang.git#subdirectory=python Install compatible Transformers: pip install git+https://github.com/huggingface/transformers.git Launching the Service Start SGLang with optimized speculative decoding: python -m sglang.launch_server \ --model zai-org/GLM-OCR \ --port 8080 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 The speculative decoding parameters accelerate inference by drafting multiple tokens simultaneously and verifying them in parallel. Adjust --speculative-num-steps based on your hardware—higher values increase speed but require more memory. Structured Output SGLang's structured generation ensures GLM-OCR outputs valid JSON or other schemas: import sglang as sgl @sgl.function def extract_invoice(s, image_path): s += sgl.user(sgl.image(image_path) + "Extract invoice data as JSON") s += sgl.assistant(sgl.gen("json_output", json_schema={ "type": "object", "properties": { "invoice_number": {"type": "string"}, "date": {"type": "string"}, "total": {"type": "number"}, "items": { "type": "array", "items": { "type": "object", "properties": { "description": {"type": "string"}, "quantity": {"type": "integer"}, "price": {"type": "number"} } } } } })) result = extract_invoice.run(image_path="invoice.png") print(result["json_output"]) This guarantees machine-readable output without post-processing or retry logic. For API endpoints that serve structured responses, Apidog's schema validation can automatically verify your output formats match expected JSON structures. When to Choose SGLang Over vLLM Select SGLang when you need structured outputs or speculative decoding. Its regex-constrained generation guarantees valid JSON schemas, eliminating retry logic. The speculative algorithm accelerates token generation by 30-40% on GPUs with sufficient memory. > | Feature | vLLM | SGLang | --------------------------------------------------------------- > | Throughput | High | Very High | > | Latency | Good | Excellent | > | OpenAI Compatible | Yes | No | > | Structured Output | Manual | Built-in | > | Community Support | Excellent | Growing | > | Setup Complexity | Medium | High | > | Best For | Production APIs | Speed-critical apps | For standard OCR without strict latency requirements, vLLM provides sufficient performance with simpler configuration and better community support. Transformers Direct Integration For development, debugging, or custom pipelines, use the Transformers library directly. This provides maximum flexibility at the cost of lower throughput compared to vLLM or SGLang. Installation Install the latest Transformers from source: pip install git+https://github.com/huggingface/transformers.git Basic Inference Load and run GLM-OCR in Python: from transformers import AutoProcessor, AutoModelForImageTextToText import torch MODEL_PATH = "zai-org/GLM-OCR" # Prepare input messages = [ { "role": "user", "content": [ {"type": "image", "url": "document.png"}, {"type": "text", "text": "Text Recognition:"} ], } ] # Load model and processor processor = AutoProcessor.from_pretrained(MODEL_PATH) model = AutoModelForImageTextToText.from_pretrained( MODEL_PATH, torch_dtype="auto", device_map="auto", ) # Process input inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt" ).to(model.device) inputs.pop("token_type_ids", None) # Generate output generated_ids = model.generate(**inputs, max_new_tokens=8192) output_text = processor.decode( generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False ) print(output_text) The device_map="auto" automatically distributes model layers across available GPUs. For single-GPU deployment, this loads the full model on one device. For CPU-only inference, change to device_map="cpu"—expect significantly slower performance. Batch Processing Process multiple documents efficiently: import os from pathlib import Path def batch_process(directory, output_file): documents = list(Path(directory).glob("*.png")) + \ list(Path(directory).glob("*.pdf")) results = [] for doc_path in documents: # Convert PDF to images if needed if doc_path.suffix == ".pdf": images = convert_pdf_to_images(doc_path) else: images = [doc_path] for image in images: text = extract_text(image) # Your extraction function results.append({ "file": str(doc_path), "page": image.page_num if hasattr(image, 'page_num') else 1, "text": text }) # Save results with open(output_file, 'w') as f: json.dump(results, f, indent=2) # Usage batch_process("./invoices/", "extracted_data.json") When processing documents in production, Apidog's workspace management helps organize multiple document processing endpoints into logical groups, making it easier to test and monitor different workflows. Memory Optimization For GPUs with limited VRAM, use quantization: from transformers import BitsAndBytesConfig quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16 ) model = AutoModelForImageTextToText.from_pretrained( MODEL_PATH, quantization_config=quantization_config, device_map="auto", ) 4-bit quantization reduces memory usage by 75% with minimal accuracy impact for document understanding tasks. Handling Edge Cases Documents with heavy handwriting or extreme skew angles reduce accuracy. Pre-process images with deskewing algorithms before sending to GLM-OCR. For multi-page PDFs, extract pages as separate images rather than passing the entire file. This enables parallel processing and simplifies error handling when individual pages fail. Watermarked documents occasionally trigger false positives in text regions—experiment with contrast adjustments if you see garbled output in specific areas. Real-World GLM-ORC Use Cases GLM-OCR excels in several production scenarios: Invoice Processing Finance teams extract line items, dates, and totals from scanned invoices. The model maintains table structure, ensuring accurate totals calculation without manual review. Process thousands of invoices per day with local deployment and zero API costs. Technical Documentation Engineering teams convert PDF manuals and specs to searchable text. Formula recognition preserves mathematical equations, making technical content machine-readable. Ideal for legacy documentation modernization projects. !glm-orc example Legal Document Analysis Legal professionals review contracts and agreements with OCR that respects document hierarchy. Multi-column layout handling ensures paragraphs aren't incorrectly merged. Privacy-first approach keeps sensitive data on-premises. Healthcare Records Medical offices digitize patient forms and prescriptions. Recognizes 8 languages, useful for multilingual healthcare environments. Local deployment meets HIPAA compliance requirements by keeping data internal. Conclusion GLM-OCR delivers production-grade document understanding in a 0.9B parameter package. You deploy it locally, maintain data privacy, and achieve throughput rates that rival cloud APIs—all without per-request pricing. The architecture handles complex layouts, tables, and formulas that traditional OCR misses, while the MIT license permits unrestricted commercial use. Choose vLLM for production deployments requiring high throughput and OpenAI compatibility. Use SGLang when maximum inference speed matters. Select Transformers for development and custom pipelines. Each option runs the same underlying model, so you switch deployment methods without retraining or retuning. When building document processing pipelines—whether extracting data from invoices, parsing technical documentation, or automating form processing—streamline your API testing with Apidog. It provides visual request building, automated documentation generation, and collaborative debugging tools that complement your GLM-OCR deployment workflow.

155 responses leaked between AI customers: the GLM-5.3 KV cache bug — and the canary test for YOUR provider --- published: true tags: ai, security, cloud, llm description: "On Cloudflare Workers AI, 155 GLM-5.3 responses contained other customers' data — stale KV-cache pages served with HTTP 200. The mechanism, the 25-minute repro, the SGLang bug behind it, and a canary script that tests YOUR provider for cross-request leaks." series: black box --- > Between August 27 and September 9, some responses from GLM-5.3 on Cloudflare Workers AI contained other customers' prompts and partial generations. Not a breach of a database, not a prompt injection — the inference engine itself served stale KV-cache pages from previous, unrelated requests, with a clean HTTP 200. Exactly 155 responses were affected, all triggered by prompts longer than 681,280 tokens. Cloudflare published the full autopsy yesterday. This post is the mechanism in four steps, why the same bug class may sit in other inference stacks, and a canary script that tests your own provider for cross-request leakage. The detail that should stop you: 681,280 is 65% of 1,048,576 — the model's context window. Someone sized the KV pool at roughly two-thirds of the context and shipped it. Everything else follows from that one number. The failure chain, step by step From Cloudflare's disclosure, cross-checked against SGLang issue #30233 (open since July 6): 1. The mismatch. GLM-5.3 launched with a 1,048,576-token context window, but the prefill KV pool held only 681,280 tokens. Cloudflare knew the pool was smaller and believed the engine would cleanly reject oversized prompts. 2. The rejection path. A prompt longer than the pool gets rejected in prefill — but in disaggregated prefill/decode mode, the rejected prompt was "replaced with a one-token placeholder," and prefill transferred just that placeholder's KV pages to decode. No error was returned. 3. The uninitialized read. Decode had reserved 8 KV pages for the request. Only page P0 was ever written. Attention then read all 8 reserved pages — 7 of which still contained data from earlier requests on the same node. Other customers' prompts, other customers' generations. 4. The clean 200. The response streamed out composed partly of someone else's context. No error code, no anomaly — which is why it took an external researcher, not monitoring, to catch it. Two bugs overlapped: a configuration bug (pool < context — Cloudflare's, and only GLM-5.3 was affected there because every other Workers AI model had a pool larger than its context) and an engine bug (rejected requests surviving to the KV handoff — SGLang's, generic to disaggregated deployments, fixed in v0.5.22 after Cloudflare's draft PR #38704 and the merged follow-ups #36988/#38935). And a third factor that isn't a bug at all: KV pages aren't zeroed between requests by design — performance depends on reuse. The entire safety of the architecture rests on the bookkeeping around those pages being airtight. The timeline worth studying Reported by Gadi Evron (Knostic) via HackerOne on September 8, 23:11 UTC. Reproduced by Cloudflare within 25 minutes. Model private at 14:00 the next day, root cause at 14:35, fixed config at 14:52, model restored at 19:40. Coordinated disclosure published October 26 with the reporter and the SGLang maintainers, complete with page diagrams, issue links, and the exact number of affected responses. Set that next to the same month's eMudhra response (revocation in 14 hours, then blaming "social media amplification" for the scrutiny) and you have the two poles of incident handling. Cloudflare's post even ends with the hardening roadmap — KV checksums on every prefill-to-decode transfer, canaries, per-page ownership tags, and formal verification of the RDMA transfer protocol. When the failure mode is "the bookkeeping lied," you eventually stop patching and start proving. The lab: a canary test for YOUR provider You cannot audit a provider's KV pool from the outside. You can detect the symptom class: data from request A surfacing in the response to request B. That's exactly what Cloudflare's post-incident canaries do — here is the userland version, works against any OpenAI-compatible endpoint (point BASE/KEY/MODEL at OpenAI, Anthropic-compatible gateways, vLLM, SGLang, llamacpp, Ollama): # canary.py — cross-request leak smoke test for chat APIs # usage: BASE=https://api.provider.com/v1 KEY=sk-... MODEL=... python3 canary.py import os, random, string, sys, urllib.request BASE = os.environ["BASE"].rstrip("/") # e.g. http://localhost:8000/v1 KEY = os.environ.get("KEY", "none") MODEL = os.environ["MODEL"] canary = "CANARY-" + "".join(random.choices(string.ascii_lowercase + string.digits, k=16)) def chat(messages, max_tokens=200): req = urllib.request.Request( f"{BASE}/chat/completions", data=__import__("json").dumps({"model": MODEL, "messages": messages, "max_tokens": max_tokens, "temperature": 0}).encode(), headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}) return __import__("json").loads(urllib.request.urlopen(req, timeout=120).read()) # A — plant the canary chat([{"role": "user", "content": f"Internal ref for this session: {canary}. Reply with 'noted' only."}]) print(f"[A] canary planted: {canary}") # B — unrelated requests; scan every response for the canary leak = False for i in range(1, 6): r = chat([{"role": "user", "content": f"Write a two-sentence haiku about packet #{i}."}]) text = r["choices"][0]["message"]["content"] hit = canary in text leak |= hit print(f"[B{i}] {'!!! CANARY LEAKED' if hit else 'clean'} — {text[:60]!r}") print("RESULT:", "CROSS-REQUEST LEAK DETECTED" if leak else "no leak in this sample (not a proof)") Honest scope, stated in the script's last line: a clean run is not a proof — the GLM-5.3 bug needed prompts beyond the KV pool on a misconfigured node. This catches gross leakage cheaply; run it after upgrades, against new endpoints, and in CI against your own inference stack. On providers, vary it: repeat with the B requests on a fresh session/conversation id — same-account separation is the weakest fence to test. I ran the smoke test against the API that generates this blog's drafts (Z.ai's own GLM serving — for the record, the incident was Cloudflare's deployment, not Z.ai's): endpoint and auth verified, but my key's balance was empty, so the full run waits on a recharge. The script above is exactly what I'll run the day it has one — and what you can run against your provider tonight, with your key. If you run the engine yourself The operator checklist from the post, in priority order: upgrade to SGLang v0.5.22+ (rejected requests are retired at the admission queue, before any KV transfer); make your KV pool larger than your context window — and add the pre-deployment check that compares measured pool size to context length, because GLM-5.3 shipped without it; enable --disaggregation-enable-kv-checksum so every prefill→decode transfer is verified. The bug you're patching was reported in July, exploited-by-accident in August, and disclosed properly in October — if you deployed SGLang from a summer image, you are the audience for that checklist. TL;DR A KV pool sized at 65% of the context window plus a rejection path that kept going after refusal equals 155 cross-customer leaks with HTTP 200. Root-caused in 25 minutes, fixed in a day, disclosed with page-level diagrams — and the engine bug underneath is generic, so if you self-host inference, this is your patch-week. Everyone else: the canary script above is five minutes against your provider. Zero errors on the wire is not the same as zero leaks. Sources: Cloudflare — KV cache exposure remediation · SGLang issue #30233 · SGLang PR #38704 (Cloudflare draft)

LLM inferenceも、vision-languageやdiffusionまで同じframeworkで扱う。 SGLangはlanguage、vision-language、diffusion model向けのopen-source inference framework。repositoryにはimage・video generation engineも含まれます。 sgl-project/sglang

Your LLM router may have a 30-second timeout you never configured A request can have a generous application timeout and still fail while its body is being forwarded. SGLang PR #45696 provides a concrete example: a default in reqwest put a 30-second TCP_USER_TIMEOUT underneath the router's request_timeout. The trigger was backpressure during upload. An engine's HTTP process stopped reading incoming request bodies while it handled a burst of long prompts. Once the sender's buffers filled, the forwarding connection stalled. The kernel eventually returned ETIMEDOUT, and the router answered 504 with x-router-error-code: upstream_timeout. Raising the application timeout did not remove that lower-level timer. The PR author reports roughly 1,000 new 100K-token prompts arriving together, followed by outstanding sends failing around 30.7 seconds after the burst. TCPAbortOnTimeout increased alongside the failures. Those are the author's observations; I have not independently reproduced the workload. The regression test is much smaller and requires no model: accept a connection, read nothing for 40 seconds, then consume a 16 MiB body and return HTTP 200. On the author's Linux aarch64 system, the old build failed at 30.51 seconds. The patched build returned 200 at 40.03 seconds. The slow test is ignored by default, so it must be selected explicitly: cargo test --test proxy -- --ignored reads_nothing The patch calls .tcp_user_timeout(None) when constructing the forwarding client on Linux, Android, and Fuchsia. SGLang's request timeout and stream idle/stale limits still apply. The inspected lockfile resolves reqwest 0.12.28, whose versioned source confirms the 30-second default on those targets. This is specifically a stalled-send problem. It does not imply that every inference request lasting more than 30 seconds will fail. A small body that the engine consumes promptly takes a different path through the timeout conditions. For an incident with a sharp band of failures around one duration, I would inspect socket options alongside application settings. Record when the body stops making progress, check TCPAbortOnTimeout, and distinguish connect, upload, response-header, and stream-idle deadlines. A larger request_timeout cannot override an earlier deadline elsewhere in that path. Disabling TCP_USER_TIMEOUT everywhere would also be a poor general fix: a dead peer still needs a bounded lifetime. Here the router already has application-level limits, and the change removes a competing socket deadline. The fix merged on October 22, 2026 at 04:43:55 UTC. SGLang v0.5.22 was released before it; the confirmed scope here is the affected pre-fix main-branch forwarding client. The PR reports 224 proxy tests passing with the slow test ignored and 36 library tests passing, but its extra CI run failed, so I would not describe the full presubmit as green. Sources: sglang#45696 github.com/seanmonstar/reqwest/blob/v0.12.28/src/… github.com/sgl-project/sglang/commit/80263b558081… Disclosure: AI-assisted writing and source review. All experimental results above are attributed to the upstream author.