Niko1221

Run Qwen3.8-Flash-Next Locally with Strata: 64 GB Quickstart Qwen3.8-Flash-Next (QFN) has 125B total parameters, 6B active per token, and a native 256K context window. It has thinking enabled by default and records the highest agentic-coding benchmarks in its local-model class. <!--more--> This article describes running a 3-bit quantization of QFN on a 64 GB RAM desktop with an RTX 5090 using Strata, and documents why sub-Q4 quantizations do not produce reliable tool calls for agentic workloads. The install commands, the RAM each quantization pins, and the tool-call failures on a 64 GB desktop are below. Speed and failure figures mix runs on that machine — an RTX 5090, 64 GB of RAM, Ubuntu 26.04 — with reports on the Strata tracker and the 0.2.11 release notes. Commands and engine defaults were checked on 06/11/2026 against Strata 0.2.11, when the repository showed 40,836 stars and the engine packages were 0.1.57. Where this runtime sits next to Ollama, llama.cpp, and vLLM is the job of the [LLM hosting guide](glukhov.org "LLM Hosting in 2026: local, self-hosted, and cloud infrastructure compared."). What Strata Is The engine in this quickstart is Niko1221/Strata, MIT-licensed. A different project, the OSDI 2026 paper on hierarchical KV caching for SGLang, uses the same name. Strata builds on llama.cpp and ggml, and the repository includes a write-up at docs/paper/Strata-Paper.pdf. It runs Qwen3.8-Flash-Next across one consumer GPU, system RAM, and an SSD. The same engine also serves Qwen3.6-35B and DeepSeek V4.1-Flash. Qqwen 3.8 Flash Next has 24,576 routed experts across 48 layers; each token activates 10 experts (6B active params in total). Strata divides the model across three memory tiers instead of requiring it to fit in one. | Tier | Holds | Does | |---|---|---| | GPU (VRAM) | Attention, DeltaNet mixers, gated-residual weights, routers, shared experts, output head, the MTP draft layer, the KV cache (from 64K: the most-read part; the rest streams from RAM), plus an expert cache | Everything that runs per token; the expert cache fills whatever VRAM is left with the most-used experts and adapts to the conversation while it runs | | RAM | All 24,576 experts, pinned | The CPU computes the experts the GPU does not have, in place, at the same time as the GPU works on the cached ones (AVX-512/AVX2 kernels, ggml for the i-quants) | | SSD | The 28.8 GB n-gram embedding table | Read a few rows per token through the OS page cache | The 51B n-gram parameter block serves as a 20M-entry lookup table indexed by bigrams and trigrams. Because this is a lookup rather than a matrix multiplication, it can reside on an SSD with minimal runtime cost. VRAM capacity determines the scaling axis. Every additional gigabyte of VRAM holds approximately 700 experts, and each expert cached on the GPU reduces CPU compute. A slow card with more VRAM outperforms a fast card with less. RAM capacity is a hard constraint because all experts must be pinned for simultaneous CPU and GPU computation. Strata does not support lazy memory mapping to disk. Download size and RAM usage are nearly identical for QFN. How the decode speed is built Three mechanisms supply the decode rate. 1. Multi-token prediction. The checkpoint includes one MTP layer trained for multi-step drafts. Strata drafts up to 3 tokens and verifies them in a single pass over all 48 layers — the draft-verify cycle in the [speculative decoding guide](glukhov.org "Speculative decoding: draft-verify, MTP, n-gram, and engine setup."). The project measures average acceptance at 2.4–3.2 tokens per pass, and reports that as 1.6–1.8× decode speed with the same output. 2. Prompt lookup. The server pulls up to 5 draft tokens from earlier copies of the context when the model is repeating it, as in a code edit or a quotation. The project reports code edits 6–11% faster, and other text unchanged. 3. Chunked prefill. Prompts arrive in blocks of up to 8,192 tokens, and the next layer's experts move over PCIe while the current layer's attention runs. The project puts long-document and codebase ingestion above 1,000 tokens/s, and a follow-up starts within seconds because the conversation state stays resident. The server ships a local chat UI, as a desktop app or in the terminal, plus OpenAI- and Anthropic-compatible endpoints on localhost. Claude Code, Codex CLI, OpenCode, Hermes Agent, and OpenClaw attach through strata connect. Hardware the project lists: NVIDIA RTX 20/30/40/50 series (CUDA 13 engine packages need driver 580 or newer), AMD Radeon on HIP, Intel Arc on Linux, Apple Silicon, and AMD Strix Halo. Release 0.2.11 is Strata-0.2.11-linux-x86_64.AppImage for the desktop app; the terminal path below is the one this quickstart walks through. QFN Benchmarks Against Local Alternatives Alibaba's model card compares Flash-Next with Qwen3.8-27B, DeepSeek-V4-Flash (the 0731 checkpoint), and Claude Opus 4.6 (Max). | | QFN | Qwen3.8-27B | DeepSeek-V4-Flash | Claude Opus 4.6 (Max) | |---|---|---|---|---| | Total / active params | 125B / 6B | 27B / 27B | 284B / 13B | – | | N-gram embedding params | 51B | – | – | – | | DeepSWE 1.1 (agentic coding) | 58.7 | 42.2 | 54.4 | – | | SWE-bench Pro | 62.5 | 61.7 | 56.0 | 53.4 | | SWE-bench Multilingual | 81.0 | 73.8 | – | 77.5 | | Toolathlon Verified (tool use) | 73.5 | 67.1 | 70.3 | – | | JobBench (agentic office) | 55.7 | 33.4 | 41.3 | 36.6 | | CoWorkBench | 73.9 | 70.7 | 45.1 | 68.2 | | GPQA Diamond | 91.7 | 89.2 | 90.8 | 91.3 | | LiveCodeBench v6 | 91.9 | 90.3 | 90.6 | 88.8 | Agentic-coding rows on that card used temperature 1.0, top_p 0.95, and a 256K window. DeepSWE reports the higher of the Claude Code and mini-SWE-agent harnesses, and the card says Flash-Next's higher score is on mini-SWE-agent. SWE-bench Pro re-ran the baselines on a corrected task set; the Claude Opus 4.6 cell is the vendor's published score, and the other cells are Claude Code. Two rows on the same card go the other way. NL2Repo-Bench is 48.1 for Flash-Next and 54.2 for DeepSeek-V4-Flash. HLE, judged by GPT-4o, is 35.9 against Claude Opus 4.6 (Max) at 40.0. For agent work the card's defaults are preserve_thinking on, so the reasoning trace stays across turns, and reasoning_effort at xhigh. The other levels are medium and low. The [Qwen 4 family page](glukhov.org "Qwen 4: what Alibaba has confirmed, the 27B ladder, and what Flash-Next previews.") tracks what Alibaba has confirmed about this line, including the Qwen Community License 1.0 on these weights. The table above is the part that can be measured today. Quantization Builds and Sizes ISTA-DASLab GSQ-RCO The ISTA-DASLab GSQ-RCO GGUFs use non-uniform quantization. GSQ is low-bit scalar quantization per tensor, and RCO assigns a type per tensor inside a size budget. The method spends almost all of the compression on the routed experts, which are 95% of the searchable weights. Each build is two shards. The second is the n-gram table at about 4.5 bits per weight, and it can be mapped from disk. | Build | Avg bpw | Resident shard | N-gram shard | Total | Notes | |---|---|---|---|---|---| | Q2_0 | 2.40 | 37.6 GB | 28.8 GB | 66.4 GB | Speed build: avoids lookup-table formats; 367 prompt t/s, 93.8 decode t/s, 6.7 s avg latency (vs 12.7 s for IQ2_XS at nearly the same size) | | IQ2_XS | 2.50 | 39.2 GB | 28.8 GB | 68.0 GB | Smallest at equal quality | | IQ3_XXS | 3.00 | 47.0 GB | 28.8 GB | 75.8 GB | The tight-memory pick; matches the base model on AIME25, gives up ~1.5 points on GPQA-Diamond | | IQ3_S | 3.50 | 54.8 GB | 28.8 GB | 83.6 GB | ISTA's recommendation: matches or exceeds the base model on every task in their harness (GPQA-Diamond 92.93 vs 91.92) | Those GPQA figures are ISTA's harness, not the 91.7 on Alibaba's card. A Coder variant keeps 256 of the 512 routed experts per layer, at 58.4 GB and 1.89 effective bits per weight. A 0.91 GB BF16 vision projector is the multimodal add-on. The ISTA line stops at IQ3_S; it has no Q4 build. Unsloth packs in Strata 0.2.11 On 04/11/2026 Unsloth re-uploaded its Qwen3.8-Flash-Next GGUFs as Dynamic v3.1. The 0.2.11 release notes say Strata can download, prepare, and run seven of them from the app, from strata launch --family unsloth, or from a folder you already have. Each file is checked by size and SHA-256 against Unsloth revision d9620119. The last file of every set is the 28.8 GB per-token table. Sets that share a table share the file: UD-IQ1_M with UD-IQ2_XS, and UD-IQ3_XXS with UD-IQ4_XS-v31. | Name in Strata | Download | Files | |---|---|---| | UD-IQ1_M | 68.0 GB | 3 | | UD-Q2_0 | 70.6 GB | 3 | | UD-IQ2_XS | 75.8 GB | 3 | | UD-Q2_K_XL | 79.6 GB | 4 | | UD-IQ3_XXS | 83.6 GB | 4 | | UD-Q3_K_XL | 89.5 GB | 4 | | UD-IQ4_XS-v31 | 93.7 GB | 4 | UD-IQ4_XS-v31 is a different file from the UD-IQ4_XS Strata already offered: other tensor types, four files instead of three. UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL, and Q8_0 stay on the 30/09/2026 files. The 04/11 UD-Q4_K_XL is the same weights in a four-file split, and Strata accepts that split if you already have it. UD-IQ1_S does not load — one layer mixes two expert formats — so UD-IQ1_M is the 1-bit choice. UD-IQ4_XS-v31 uses the Q5_0 expert format in 33 layers. The same notes say Strata's Q5_0 math lands 1.2–1.3% from the exact result, against 2.6–3.0% for llama.cpp's formula on those tensors. On a Mac, the app rewrites the experts into one GPU file before the first start, so the disk budget is the download plus that file plus a few gigabytes (about 115 GB for UD-IQ1_M). If the 28.8 GB table stays mapped, the rest of UD-IQ4_XS-v31 is about 64.9 GB before KV cache, which is already past a 64 GB machine under Strata's pinning rule. UD-IQ1_M's remainder is about 39.2 GB. That file can fit where IQ3_S cannot, and it is a 1-bit pack, further from the tool-call behaviour measured below at 3 bits. Quickstart: Install, Launch, and First-Run Setup A Linux desktop needs an RTX 20/30/40/50 series card or another GPU the project lists, enough RAM for the chosen pack plus KV-cache headroom, and SSD space for the download. CUDA 13 engine packages need driver 580 or newer. Terminal install: curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh strata launch strata launch asks for the model, the quantization, the context, the storage folder, and the access settings, downloads the engine for the card, starts the server, and shows a monitor with speed, memory, and request data. The default endpoint is http://127.0.0.1:8080. To see what fits before a 76 GB download, or to start without the prompts: strata launch --family qwen --model q2_0 --plan # dry run: what's missing, what fits strata launch --family qwen --model q2_0 --yes --no-tui # start in background strata launch --status strata connect # prints the config for Claude Code / OpenCode / Hermes / OpenClaw / ... strata launch --stop Those --family qwen --model q2_0 flags are the dry run used for the ISTA Q2_0 build on Ubuntu 26.04. Unsloth packs use --family unsloth and the name from the table above. strata connect prints a provider block per tool. Both API shapes use an OpenAI-compatible base URL (http://127.0.0.1:8080/v1) and the model name the launch prints. Any client that accepts a custom OpenAI-compatible endpoint, including Cursor, uses that pair. Two settings to verify on first run 1. Sampling. In 0.2.11 a pack config with no sampling block follows the model card. Thinking requests get temperature 1.0, top_p 0.95, and top_k 20. Non-thinking requests get 0.7, 0.8, and 20, with presence_penalty 1.5. Precedence is the request, then the web UI's Chat settings, then the config block, then the card. Temperature 0 applies only when the request or the block sets it, and those greedy runs are the ones that fell into repetition loops on this checkpoint. If an older pack config still contains "sampling": {"temperature": 0}, replace it with the card's thinking preset. 2. tool_call_recovery. The key lives in the model run config (strata-*.json). It has been on by default since 0.2.8, and off in 0.2.7 and earlier. Strata's engine notes, citing 1,462 agent turns, report 98.3% of calls read as intended with the switch and 92.9% without. It accepts four shapes of a call whose name was in the request: a <parameter=NAME> tag, JSON inside <tool_call>, a <function=NAME> block with no <tool_call> wrapper, and a second call inside one <tool_call>. An undeclared name or broken JSON stays text. On 0.2.7 and earlier, set "tool_call_recovery": true. Leave the 0.2.11 default on for agent work. Keep the server on localhost The server binds to localhost by default, and the access settings are part of strata launch. The endpoints expose the model and the conversation on that machine, and they were not built for an open network. When another machine needs them, put an authenticated reverse proxy in front of the port. ss -ltnp | grep :8080 # confirm the listener is 127.0.0.1, not 0.0.0.0 curl -s http://127.0.0.1:8080/v1/models # the server answers on localhost Cards with 13 GB of VRAM or less STRATA_ROUTE_PRIOR=1.5 tells the router to prefer experts already on the card. Strata's notes for an RTX 3060 12 GB running IQ3_XXS report +15% on story decode and +37% on code decode against the prior left off, and fewer agent tasks passed: 4.5 of 10 on average over 4 runs, against 5.6 over 7 runs with the prior off. Version 0.2.7 turned 1.5 on by itself for discrete NVIDIA cards of 13 GiB or less. Version 0.2.8 turned that back off on 29/10/2026, after repetition and weaker long agent sessions (issue #4402). With the variable unset, 0.2.11 routes as 0.2.6 did. The project points the prior at chat and short answers on a 12 GB card, and away from long agent runs. Testbed: A 64 GB Desktop Running an Agent The measured box: | Component | Spec | |---|---| | CPU | Intel Core Ultra 7 265 (20 cores, DDR5-6400 dual channel ≈ 102 GB/s) | | GPU | RTX 5090, 32 GB | | RAM | 64 GB | | OS | Ubuntu 26.04 | | Also running | KVM host for the Hermes VM, Cursor IDE, browser | With ISTA IQ3_XXS the resident shard is 47 GB. Because the experts are pinned, about 9 GB remains for everything else. The Hermes VM — the agent stack in the [Hermes production setup](glukhov.org "Hermes production setup: profiles, skills, and the local agent stack.") — plus Cursor leaves about 5 GB, which is not enough page cache or KV cache for a long agent session. ISTA IQ3_S does not fit this machine as a daily desktop. The 54.8 GB resident shard plus the 28.8 GB mapped n-gram table use the 64 GB. With Cursor running, 3 GB stay free, and the Hermes VM has nowhere to go. | Config | Resident | Free RAM (model only) | Free RAM (model + VM + Cursor) | Decode (measured) | Agentic verdict | |---|---|---|---|---|---| | IQ3_XXS (3.00 bpw) | 47 GB | ~9 GB | ~5 GB | – | Tool calls fail regularly; not a daily driver | | IQ3_S (3.50 bpw) | 54.8 GB + 28.8 GB mmap | ~3 GB (Cursor only) | – | 100–130 tok/s | No headroom at all; not usable | IQ3_S decoded at 100–130 tokens/s, the same speed class as a 27B build on llama.cpp with MTP 3 in the [Qwen 3.6 MTP comparison](glukhov.org "Qwen 3.6 MTP vs standard decoding on a 16 GB GPU: token speed, VRAM, and draft length."). The pack that fits the RAM budget (IQ3_XXS) is the one that misses the tool-call format. The pack ISTA recommends (IQ3_S) leaves no room for the rest of the desktop. On IQ3_XXS the model emitted malformed tool-call arguments, invented tool names that were not in the schema, and re-issued the same command three to five times after a short thinking block. The harness treats each miss as a retry, so the session fills with the same error. On this hardware IQ3_XXS is not an agent model. What the Community Reports Speed reports for Qwen3.8-Flash-Next on Strata, from issue threads and the 0.2.11 notes: | Setup | Result | |---|---| | 2× RTX 4090 | 6,553 prompt t/s, 271 decode t/s single stream | | RTX 5080 + 4060 Ti, 32 GB RAM | 200+ tok/s decode peaks | | 12 GB VRAM + 32 GB RAM, experts offloaded to CPU | ~65 tok/s; one comment at 199K context with q8 KV: 1,740 prompt t/s, ~21.5 decode t/s | | RTX 5070 12 GB + 64 GB RAM, IQ3_S | 85–100 tok/s decode | | Pro 6000 + 2× 3090, 256 GB RAM, Unsloth 6-bit, 160K context | Decode 70–80 → 150 tok/s after switching from llama-server to Strata | | RTX 2060 laptop, 32 GB RAM + 6 GB VRAM | Q2_0 running at all | | RTX 3060 12 GB, UD-IQ1_M, short prompts (0.2.11 notes) | 31–37 decode tok/s, 18% of experts on the card | Three failure modes are written up for low-bit Qwen3.8-Flash-Next in agent workloads. 1. Mangled tool-call emission that tracks the schema's size. Issue #1431 reports that with 39 function schemas (about 12K tokens) and a 6K system prompt, IQ3_XXS on Strata 0.1.40 produced a usable turn in about 30–50% of 20-plus controlled replays. The model either emitted a tool-call shape the parser could not map — the text landed in content with finish_reason: stop — or closed the thinking block and emitted end-of-sequence with no call and no answer. The same weights on llama.cpp parsed tool calls in all 8-plus consecutive runs. llama.cpp constrains the emission with a PEG grammar and lifts mangled content back into tool_calls. A wrapper that repaired the text afterwards did not push the useful-turn rate above that 30–50% band, because the repair runs after the tokens are already chosen. 2. Config errors and a non-deterministic expert path. Issue #2685 reports an RTX 5070, IQ3_S, and 60 GB of RAM, with tool calls failing through Strata: malformed arguments, hallucinated tools, and retry loops, including after a sampling block and tool_call_recovery were turned on. The same quant through llama serve succeeded on 5 of 5 runs of the same harness. The reporter found that identical seeds diverged in the MMQ expert path during prefill, and that disabling that path made the runs bit-identical. Strata 0.1.44 shipped determinism fixes. Later reports still do not show Strata matching llama.cpp on the same quant. 3. Tool calls left inside the thinking block. Issue #804 reports that a complete tool call inside an unclosed think block comes back as reasoning_content, with empty content and no tool_calls, and the agent loop stops. One user counted about 1 in 100 responses over roughly 1,300 real agent replies, using Hermes with IQ3_XXS, IQ2_XS, and the Coder builds. Why Sub-Q4 Breaks Agentic Tool Calls A tool call has to match a wire format. At 3 bits this 125B model misses that format on some turns, and the misses grow with the tool-schema context. A larger schema asks more of the working memory that formats the call, and quantization noise lands on that formatting. Chat turns hide it. Agent turns, which are mostly tool calls, do not. The same compression hurts more when few parameters are active. The [local-LLM comparison for OpenCode](glukhov.org "Best LLMs for OpenCode: local Ollama and llama.cpp models against a migration-map task.") measured a dense 27B at IQ3_XXS with about a 5% error rate on a structured agent task. Low-active MoE checkpoints at similar bit widths failed harder: the 35B-A3B had 64.2% errors on IQ3_S and 98.8% on IQ4_XS, and the 122B-A10B had 80% errors on IQ3_S and 90% on IQ3_XXS. Qwen3.8-Flash-Next activates 6B parameters per token, so a 3-bit pack behaves like a small model on a heavy quant. Those OpenCode runs show IQ3 and IQ4 missing the structured task for the 35B-A3B and the 122B-A10B; they do not include a Q5 run of Flash-Next. Strata's UD-IQ4_XS-v31 has been downloadable since 0.2.11, has no agent-task comparison in the sources checked here, and does not fit the 64 GB box. The same GSQ-RCO weights fail when they are loaded in llama.cpp and in Ollama. [llama.cpp](glukhov.org "llama.cpp quickstart: llama-cli, llama-server, and the flags that place a GGUF.") still constrains the emission with a grammar and can lift a mangled span back into tool_calls, so the client sees a call instead of a broken string. Strata's JSON mode, in the engine notes, is schema prompting plus a check after generation, not grammar-constrained decoding: a bad object returns HTTP 502 with structured_output_failed. tool_call_recovery and the later releases repair the four shapes listed above. They do not change the distribution the model samples from, which is why issue #1431's wrapper could not climb past the emission rate. Strata spends RAM to keep the CPU and the GPU in step. On 64 GB, the largest ISTA pack that still leaves room for a desktop and an agent VM is IQ3_XXS, and that pack is below the emission threshold measured here. IQ3_S already decodes in the same band as a 27B model with MTP 3, so the limit on that pack is RAM. A llama.cpp run with a lazy map would commit less RAM and give up the decode rate that makes Strata the faster of the two on this model. Verdict Qwen 2.8 Flash Next outperforms Qwen3.8-27B on agentic benchmarks: DeepSWE 58.7 vs 42.2, JobBench 55.7 vs 33.4, and Toolathlon 73.5 vs 67.1. Multi-step work is noticeably more competent. Per-token compute is closer to a small model than a 125B model. Quantizations below Q4 do not provide reliable tool-call emission for agentic work, failing at rates that produce retry loops on both Strata and llama.cpp. Q4 performance is not yet proven through controlled comparison on agent benchmarks. Q5-class quants are the demonstrated floor for reliable agentic work; the Unsloth UD-Q5_K_XL exists but lacks published benchmarks. A Mac Studio with an M5 Max and 128 GB unified memory can hold the 105 GB MLX build or a Q4 pack with sufficient headroom for the agent stack. The QFN architecture was designed for this offloading model. I will continue using in my current setup uses an NVFP4 27B model — 17 GB on the RTX 5090 — with MTP 3, It is achieving 100+ tokens/s with reliable tool calls and no RAM competition. I will return to QFN when a Q4-class quant is benchmarked on agent tasks, when Strata's emission recovery matures, or when hardware is upgraded. References - Niko1221/Strata — the inference engine in this article. The OSDI 2026 hierarchical-KV paper for SGLang is a different project with the same name. - Strata 0.2.11 release notes — Unsloth Dynamic v3.1 packs, Q5_0 math, and the short-prompt speeds cited above. - Engine notes (docs/DETAILS.md) — sampling precedence, tool_call_recovery, and the route-prior measurements. - Qwen/Qwen3.8-Flash-Next — architecture, sampling presets, and the benchmark table. - ISTA-DASLab Qwen3.8-Flash-Next GSQ-RCO GGUFs — the Q2 and IQ3 builds in the testbed. - Issue #1431, issue #2685, issue #804, and issue #4402 — the failure reports in the community and small-card sections. Useful links - [Choosing the Best LLM for Ollama on a 16 GB VRAM GPU](glukhov.org "Dense-model choices when the card has 16 GB and the runtime is Ollama.") — dense-model alternatives when the card is the limit. - [Ollama cheatsheet](glukhov.org "Ollama CLI cheatsheet: serve, run, ps, and model management.") — the other local server the same GGUFs can be imported into.

Four DGX sparks for the official model. Two sparks for the quantized (shrunk, less intelligent) model. Or you can try strata if you have some GPUs (like 4 Radeon AI PRO R9700s) and a large system RAM (>=256 GB).

VRAM 12GBのRTX 3060で125B級モデルを動かすStrata：めちゃいい！ はじめに 煽りではないです。 かなり速く、賢いです。 以前、FreeTokenで120B級モデルを動かしてみたときは、WSL2へのメモリ割り当てに64GBが必要で、生成速度は平均 15～16 token/s程度でした。 zenn.dev しかし、Strata では 48GB でいけます。しかも約 38 token/s 程度出ます。 筆者の環境では、デスクトップ版ChatGPTと体感が変わらないくらいの印象です。出力品質も凄く良いのでは？（今後、要検証） 今回使用したモデルは Qwen3.8-Flash-Next の IQ3_XXS版 です。 Strataとは何か？ Strataは、概ね、Qwen系の大型モデルをローカル環境で爆速で動かすことに特化した、軽くて強力なエンジンという認識で良さそうです。 FreeTokenとはどう違う？ 一応、違いも整理しておきます。 項目 Strata FreeToken 対応範囲 Qwen3.8-Flash-Next、Qwen3.6-35B-A3B、DeepSeek V4.1-Flash など Qwen、DeepSeek、GLM、MiniMax、gpt-oss など幅広いモデル Expert（専門家）モデルの配置 利用状況に応じてGPUにキャッシュし、CPU計算も組み合わせる LRUキャッシュと、実行方式の選択が可能 CPUとGPUの分担 キャッシュ外のExpertモデルについて、CPU計算と転送を分担する 転送中心、CPU計算中心、両者の混合を選択できる 特徴的な機能 対応モデル向けの最適化処理、先読み生成（投機生成）、メモリ階層の使い分け 帯域に応じた分担、会話編集後の状態再利用、実行中のVRAM再配分 今回（いつも）の実行環境 項目 今回の想定 GPU GeForce RTX 3060 12GB OS Windows上のWSL2 Linux Ubuntu 26.04 x86-64 RAM 64GB、WSL2に48GBを割り当て モデル Qwen3.8-Flash-Next IQ3_XXS コンテキスト 32K 同時実行数 1 圧縮形式は IQ3_XXS を使います。 モデルファイルサイズは約76GBです。 インストール手順 CLI版をインストールします。 説明に登場するディレクトリは自由にしてください。 sudo apt update sudo apt install -y curl ca-certificates export STRATA_HOME="$HOME/workspace/strata" export STRATA_DATA_DIR="$STRATA_HOME/strata-data" mkdir -p "$STRATA_HOME/strata-setup" "$STRATA_DATA_DIR" cd "$STRATA_HOME/strata-setup" curl -fsSL \ github.com/Niko1221/Strata/releases/latest/downlo… \ -o install.sh sh install.sh PATHに ~/.local/bin がまだ通っていない場合だけ必要です export PATH="$HOME/.local/bin:$PATH" strata --version ダウンロード前に起動計画を確認する まず --plan を使い、表示されるモデル、GPU用エンジン、保存先、必要容量を確認します。 この段階ではダウンロードを開始しません。 --data-dir には絶対パスを指定します。 strata launch \ --family qwen \ --model iq3_xxs \ --context 32k \ --parallel 1 \ --vision off \ --data-dir "$STRATA_DATA_DIR" \ --access local \ --plan 出力例は次のようになります。 Plan for 'IQ3_XXS 32K': model Qwen3.8-Flash-Next IQ3_XXS, 76 GB context 32K pictures off parallel 1 download About 77.0 GB to download: the model (75.8 GB) and the DFlash2 drafter (1.2 GB). To skip the DFlash2 drafter, add --drafting mtp. The engine for your graphics card is downloaded first (from the update source; its size is shown when it starts). drafting DFlash2 (its drafter is downloaded first) tuning skip access This computer only, port 8080 same as strata launch --model iq3_xxs --data-dir "~/workspace/Strata/Strata-data" --context 32k --vision off --parallel 1 --tune skip --access local --port 8080 Nothing was changed. 準備ができたら起動します。 qwen-iq3-32k という名前で起動設定が保存されます。 --no-tui は、対話型TUIを表示せずに起動するためのオプションです。 バックグラウンド運用に向いた起動を目的として使えます。 初回はモデルのダウンロードと準備に時間がかかります。 strata launch \ --family qwen \ --model iq3_xxs \ --context 32k \ --parallel 1 \ --vision off \ --data-dir "$STRATA_DATA_DIR" \ --access local \ --save-preset "qwen-iq3-32k" \ --yes \ --no-tui 状態確認 strata launch --status curl -fsS 127.0.0.1 停止 strata launch --stop 次回以降の起動 先程保存したプリセット名で起動するイメージです。 strata launch --preset "qwen-iq3-32k" --no-tui 起動に失敗した場合 次のコマンドで状況を確認します。 strata --version strata doctor --live strata launch --status 自動チューニング 一度正常に動作したことを確認してから、チューニングを適用してください。 まずStrataを停止し、保存済みのpresetに対してチューニングを実行します。 strata launch --stop strata launch \ --preset "qwen-iq3-32k" \ --tune quick \ --save-preset "qwen-iq3-32k" \ --yes \ --no-tui 公式資料では、quick は約20分、thorough は最大2時間程度が目安とされています。 実際には環境によって変わります。 一度計測された調整条件は再利用されます。 再測定したい場合は --retune を追加します。 ちなみに、筆者の環境では quick の結果、次のように表示され、変更はなかったようです。 Tuning result: already fastest. Your computer already runs at its best with the standard settings. Nothing needed changing. 100% あとがき 正直どれも同じだと思っていましたが、思いがけない収穫となりました。 ローカルLLMに対する印象や世界観がガラリと変わりました。 早速この記事を丸ごと校正してもらいました。

Strata 0.2.11 is now out! 👀 - Supports new Unsloth Qwen3.8-Flash-Next quants - Layer splitting is now allowed for all configurations - Performance increase - App & Terminal improvements More at: github.com/Niko1221/Strata/releases/tag/v0.2.11 t.co

In my private coding benchmark Luna Low effort is beaten by Qwen 3.8 Flash Next Coder IQ_1 served by Strata locally niko1221/strata Luna medium effort is very slightly better than its output and completes tasks in 5-10x less time.

Agree, I run QFN on my AI Pro R9700 (32GB) using Niko1221/Strata It's even faster than on a Strix Halo. Typically I get around 90 tokens/sec with a small context, down to 70 tokens/sec when it's near the maximum context usage.

A 125B model at 50+ tok/s on a 12 GB gaming card: what I changed in Strata I run Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on one RTX 3060 with 12 GB of video memory and 30 GB of RAM. Nothing leaves my PC. This post is about the fork I made to get there, what I measured, and what I have not tested. Starting point: Strata The model is about 78 GB of files at the quantization I use (IQ3_S). It does not fit on a 12 GB card. Strata by Niko1221 and contributors (MIT) solves that by keeping the most-used experts on the graphics card, putting more in RAM, and streaming the rest from the SSD. My fork, Strata-HotRod, makes that path faster and checks that answers stay correct. What I measured All of this is on one machine: RTX 3060 12 GB, Intel i5-12400F, 30 GB RAM, a 500 GB NVMe on a PCIe 3.0 link, Rocky Linux 9.6. | Answer (131K context) | tokens/s | |---|---| | Code, 500-650 tokens | 52-55 | | Reasoning, about 1,000 tokens | 66-71 | | Prose, 100-150 tokens | 50-53 | Long prompts, 262K context: an 8,554-token prompt is read in 16.9 s and answered at 52-54 tokens/s. A 45,398-token prompt takes 67-68 s to read, then the answer comes at 38-41 tokens/s. The next turn of that same conversation starts in 0.4-0.5 s, because the saved state is reused. The first message of a chat is the slow part: it waits on the SSD for the model's experts. What I changed - A learned look-ahead. The engine guesses which experts the next layers will need and loads them early. Against the plain router, the guesses are right 85-88% of the time one layer ahead (router: 77-79%), 76-79% two layers ahead (66-69%), and 66-71% four layers ahead (55-58%). - Three tiers: card, RAM, SSD. The hottest experts sit on the card, the next ones in a RAM tier sized from free memory, the rest on the SSD. Budgets and reserves are measured on your PC and kept across restarts. - Expert placement from routing traces, instead of a fixed guess. - Stand-in experts with a re-check. To stay fast, some of a token's experts are replaced with similar ones already on the card. - Pinned RAM segments (on by default): about 9% faster on this PC. Every change and its measurement is in the repo's docs, including ideas that did not help. For example, a learned read-ahead of lower-ranked guesses made decoding 23% slower on this PC, because the drive was already at its limit, so it stays opt-in. Is the answer quality still good? In a small test (50 maths questions, 30 coding questions, answers read by hand), the stand-ins gave 47/50 and 30/30 correct, against 48/50 and 30/30 with nothing replaced. A test this small cannot show a difference of one or two answers either way. On MMLU-Pro (15 questions from each of its 14 subjects, 210 in all, thinking on, temperature 0) it scored 87.1%, 183 of 210. That is a small sample, so read it as a sanity check, not a leaderboard result. What I have not tested One card, one model, one PC, Linux only. Other GPUs, AMD, Windows, macOS and other model sizes may work in upstream Strata, but I have not run them, so I will not claim them. Try it, and tell me what you get The repo has the install steps and the full change log: sangharshadhyeta/Strata-HotRod. If you run it on different hardware, I would like your numbers, good or bad. The repo has a short guide for how to report them.

A 763B MoE on a 12 GB GPU: Our Speed Ceiling Versus What Strata Reports, and What Closes the Gap TL;DR Two days ago we wrote that a 763B mixture-of-experts model squeezed to 2 bits per weight could, on paper, run on a 12 GB GPU, and we put illustrative speed ceilings of 3.4 to 12 tokens per second on it by assuming every expert had to be fetched cold over PCIe or from an SSD. That post had one weak spot: the card did not say how the Strata engine places tensors, so we guessed. The Strata documentation does say, and it is different from our guess. The experts that do not fit in GPU memory are not shipped to the GPU. The CPU computes them where they sit in system RAM, while the GPU works on a cache of the most popular experts. We did not run Strata or these files. What we did is take the speeds its README and DeepSeek page report, recompute how many expert bytes one token touches for each build, and ask what the reported numbers imply. Three findings. A token touches 1.7 to 2.1 GB of routed experts on the 2.00-bit build, depending on whether the engine uses 5 or 6 experts per layer. The 4.7 tokens per second reported on a 12 GB GPU with 64 GB of RAM is above what a 3.6 GB/s SSD could deliver if experts were hit uniformly, so the routing must be skewed enough for a hot set to carry most of the traffic. And the published speed table is not a clean hardware ranking, because the rows differ in RAM, drive, engine version and settings. !Grouped bars of routed-expert bytes read per generated token for four builds at 6 and 5 experts per layer What does Strata report for this 763B model? Strata is an MIT-licensed local inference engine. When we checked on November 4, its repository had about 39,000 stars, had been created on September 24, and the latest release was v0.2.10, published at 23:52 UTC on November 2. Its README now lists DeepSeek-V4.1-Flash-1017 next to the Qwen models, using the three ISTA-DASLab GGUF sets we looked at on November 2. For the 2.00 bits-per-weight set it gives this table, measured on Strata 0.2.5 (the Strix Halo row on 0.2.7): | Hardware | Output speed | Prompt speed | |---|---:|---:| | RTX 5070 12 GB, 64 GB RAM | 4 to 6 tokens/s | about 110 tokens/s (4K) | | RTX 3060 12 GB, 46 GB RAM | 9 to 11 tokens/s | about 155 tokens/s (4K) | | 1x Radeon AI PRO R9700, 251 GB RAM | 21 to 25 tokens/s | 320 to 360 tokens/s (8K) | | 4x Radeon AI PRO R9700 | 74 to 77 tokens/s | 400 to 450 tokens/s (8K) | | Ryzen AI Max+ 395, 128 GB unified | 19 to 21 (code), 11 (chat) | 87 tokens/s (4K), 125 (8K) | !Grouped bars of the low and high end of the speeds reported in the Strata README for five machines These are the project's own numbers, one report per row, and they are not independent measurements. The DeepSeek page of the same repository adds details that do not line up exactly with the README. It gives 4.7 tokens per second for the RTX 5070 box with a 3.6 GB/s NVMe drive, 8 to 10 for the RTX 3060 (9.9 on code, 7.8 on agent work, drafter off), and about 48 tokens per second for four R9700 cards on the 2.00 set where the README says 74 to 77. A user-built sm_89 binary is reported at 40.2 tokens per second on an RTX 4090 48 GB with 128 GB of RAM. The honest reading is a range, not a point. Where does the engine actually put the experts? This is the part we had wrong. The DeepSeek page describes three tiers. The dense weights (attention and the other tensors every token needs) go to graphics memory. The routed experts go to graphics memory "as many as fit", and the rest are computed on the CPU, read in place from the model files. The n-gram lookup table, called Engram, stays on the SSD, and only a few rows are read per generated token. The HOW_IT_WORKS page for the Qwen models states the same principle: the card keeps the most-asked experts and learns which ones those are while you chat, the CPU works on the others at the same time as the GPU, and more VRAM matters more than a faster GPU. For us the consequence is simple. Our November 2 table priced every token as 2.06 GB crossing a link. In this design only the experts that miss the GPU cache cost anything, and the cost is CPU memory traffic rather than PCIe transfer. The SSD enters the picture only when the experts do not fit in RAM either, which is the 64 GB case, because the 2.00 set has about 132 GB of routed experts. How many bytes does one token touch? The model has 40 layers with 384 routed experts each, so 15,360 experts, and the router picks 6 per layer per token. Strata also offers a "Faster" mode that uses 5, and it is the default on PCs whose graphics cards total 16 GiB or less and on integrated GPUs. Dividing the routed-expert bytes of each build by 15,360 gives the average expert size, and multiplying by 40 layers and by k gives the traffic per token. The expert totals for the three ISTA-DASLab builds come from the GGUF headers we parsed on November 2, and the Q8 figure is the 288.8 GB that Strata's page lists for Unsloth's UD-Q8_K_XL file. LAYERS, EXPERTS = 40, 384 N = LAYERS * EXPERTS # 15,360 routed experts EXP_GB = {"IQ1_M": 114.56, "IQ2_XXS": 132.07, "IQ2_XS": 149.06, "UD-Q8_K_XL": 288.8} def per_token_gb(set_gb, k): # routed-expert bytes touched per token return LAYERS * k * set_gb / N for name, gb in EXP_GB.items(): print(name, round(gb * 1000 / N, 2), "MB/expert", round(per_token_gb(gb, 6), 2), "GB/token (k=6)", round(per_token_gb(gb, 5), 2), "GB/token (k=5)") | Build | Average expert | GB per token, 6 experts | GB per token, 5 experts | |---|---:|---:|---:| | IQ1_M, 1.75 bpw | 7.46 MB | 1.79 | 1.49 | | IQ2_XXS, 2.00 bpw | 8.60 MB | 2.06 | 1.72 | | IQ2_XS, 2.25 bpw | 9.70 MB | 2.33 | 1.94 | | UD-Q8_K_XL | 18.80 MB | 4.51 | 3.76 | Two caveats. The searched allocation gives tensors different types, so the real size of an expert varies by layer and these are averages. And the number is bytes the router asks for, not bytes the machine must move: every expert already in the GPU cache costs nothing on the memory side of the problem. Is the SSD the limit on a 64 GB machine? Take the RTX 5070 row. The experts of the 2.00 build are about 132 GB. Assume, and this is our assumption, that after the 6.16 GB dense path and a cache reserve the GPU holds 3 to 5 GB of experts and that 48 to 56 GB of RAM is left for the experts. That is 51 to 61 GB, or 39 to 46% of the expert bytes in fast memory. If routing were uniform, every token would then read the remaining 54 to 61% from the SSD. With a 3.6 GB/s drive and the Faster mode (5 experts, 1.72 GB per token), that caps decoding at about 3.4 to 3.9 tokens per second. With 6 experts it is 2.9 to 3.2. The reported 4.7 sits above both. The chart shows the SSD-bound ceiling against the share of expert bytes held in fast memory, and where the 4.7 line crosses it. !Line chart of the SSD-bound speed ceiling versus the share of expert bytes held in fast memory, with the reported 4.7 tokens per second as a dashed line Turn it around. To sustain 4.7 tokens per second on a 3.6 GB/s drive, the machine can afford at most 0.77 GB of SSD reads per token. That is 44.5% of the expert bytes at 5 experts, or 37.1% at 6. So the cache must serve at least 55.5% (or 62.9%) of the expert lookups while holding only about 42% of the bytes. This is a mild skew, and it is exactly what a hot-expert cache is for. The DeepSeek page says the project ships a "hot-expert profile" built from routing statistics (a separate one for the Coder variant came from 519,422 calibration tokens) and copies it into the pack. A small speculative-decoding drafter, which the page measures at plus 2 to 13% on the RTX 3060 box, can also lower the cost per token, because several drafted tokens share one verification pass. What this does not prove is the cache hit rate itself. We have a lower bound on skew implied by one reported number and our guess at memory capacity. Running strata launch --plan, which the page says works out expert placement for your cards and RAM, would replace our guess with the engine's own count. Why is the 128 GB Ryzen box about four times faster than the RTX 5070 box? On the Ryzen AI Max+ 395 the GPU and CPU share one memory pool. The page says a 128 GB machine keeps about 109 GB of the 2.00 set's experts in that pool and reads the other 23 GB from the SSD, which agrees with the 132 GB total we computed. That puts 82.5% of the expert bytes in fast memory against about 42% on the 64 GB box. The pool is also fast. The chip's 256-bit LPDDR5X-8000 interface is rated at about 256 GB/s, and community llama.cpp benchmarks report roughly 215 GB/s sustained. Even if every byte of a token's experts came from the pool, 1.72 GB per token at 215 GB/s would allow about 125 tokens per second, so the reported 19 to 21 on code is only 15 to 17% of that bandwidth ceiling. Memory bandwidth is not what limits it. The SSD may be: with uniform routing, 17.5% of 1.72 GB is 0.30 GB per token, which at 19 to 21 tokens per second is 5.7 to 6.3 GB/s of reads, near the limit of a fast PCIe 4.0 drive, and the page does not say which drive was used. Chat at 11 tokens per second would need 3.3 GB/s. Code being faster than chat fits the idea that code routes into the profiled hot experts more often, but that is our hypothesis, and the drafter's acceptance rate on repetitive code would explain it just as well. Why does a 12 GB RTX 3060 beat the newer RTX 5070 in the table? Read as a ranking of GPUs, the table says an RTX 3060 is about twice as fast as an RTX 5070. We do not believe that, and the rows give reasons not to. The 3060 row has 46 GB of RAM and the 5070 row 64 GB, the 3060's drafter was off, the 5070 row names its 3.6 GB/s drive and the 3060 row does not. The README also says the Strix Halo row was measured on a newer engine version than the others. A cleaner experiment holds everything fixed except the card, with cold-cache and warm-cache runs reported separately. Until someone publishes that, treat each row as "this machine, this day". What does the quantization cost in quality? Strata reports its own comparison against the original model on 4,995 positions. For the 2.00 set it gives KL divergence 0.170 and perplexity up 15.5% (code up 8.6%, agent work up 5.3%). For 2.25 it gives KL 0.107 and perplexity up 7.3%. For IQ1_M perplexity is up 21%, with plain English prose the weakest area at plus 53%. The ISTA-DASLab card, as we noted on November 2, reports 98.7% of the base model's task average for the smallest build on three benchmarks. These measure different things, and neither tells you how the Faster mode with 5 experts changes your own task, which Strata describes only as a small change that is usually not noticeable. The same page also quotes a perplexity of 3.405 against a reference of 3.140 for the 2.25 set, which is plus 8.4% where the text says plus 7.3%. Likely different runs, but treat single-run figures here as good to about a point. How should you check this on your own machine? Start with the plan, then measure, and keep the cold and warm cases apart. strata launch --family deepseek41 --model ist-das-2.00bpw --plan Write down the engine's experts-in-VRAM and experts-in-RAM counts from the plan. Divide by 15,360 to get your fast-memory share, then compare your decode speed with the SSD-bound line for that share. If you are above it, your routing is skewed in your favour. If you are well below it, something other than the SSD is the limit, usually the CPU lane or a drafter that is switched off. Log drive throughput while generating, measure with your own prompts and not a fixed one, and compare 5-expert and 6-expert runs at the same settings. Remember the first answer after loading is slower because the cache is cold. FAQ Can a 763B-parameter model really run on a 12 GB GPU? It runs, according to the Strata README, but only because the GPU holds the dense path plus a cache of popular experts while system RAM and the SSD hold the rest. Speeds reported for 12 GB cards are 4 to 11 tokens per second on the 2.00 set, with the exact figure depending heavily on RAM and drive. How much system RAM do you need? The experts of the 2.00 build are about 132 GB. The Strata page says that under about 40 GB of RAM the setup is untested and slow, and that the Engram table, 111 GB shared by all three builds, belongs on a fast NVMe drive. The reported 12 GB GPU rows have 46 and 64 GB. What does the Faster mode change? It uses 5 experts per layer instead of 6, which cuts the bytes touched per token by 17%. It is the default on PCs with 16 GiB of graphics memory or less and on integrated GPUs such as Strix Halo. Are these speeds independent measurements? No. They are the project's reported numbers, and we did not run the engine. Our contribution is the byte arithmetic and the bounds it implies, and the cache-capacity figures for the 64 GB box are our assumptions. Further reading - A 763B MoE at "2 Bits": Where the Bytes Really Go in the GSQ-RCO GGUFs, and Why 100% Recovery Sits Inside the Noise - Why a Sparse 35B MoE Runs Faster on a CPU Than a Dense 27B Model Sources: Strata repository and README, Strata DeepSeek-V4.1-Flash page, Strata HOW_IT_WORKS, ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF, Strix Halo memory bandwidth, 256 GB/s rated versus about 215 GB/s measured. Speeds, quality figures and memory placement are as reported by the Strata project on its pages as of November 4, 2026. Bytes per token, SSD ceilings and implied hit rates are our own illustrative arithmetic, not measurements.

Radeon VIIとUbuntu 26.04でROCmを動かし、Flash-NextをOpenCodeにつないだ 5年ほど前に組んだデスクトップがある。当時、GPUにはすでに型落ちだったRadeon VIIを5万円くらいで入手して載せた。Stable Diffusionで遊んでいた時期もあったが、飽きてからはすっかり出番がなくなっていた。 最近はローカルLLMが盛り上がっている。とはいえ、パソコンもGPUもクソ高い。新しいGPUを買う前に、あまり期待せず、久しぶりにこのデスクトップに火を入れて環境を組み直してみることにした。 気になったのは、Radeon VIIのROCm公式サポートが切れていることだ。かなり苦戦する覚悟だったが、実際にはChatGPTのGPT-6.1 Terra（medium）に相談しながら、環境の調査から構築、動作確認まで進められた。多少の修正はあったものの、さくさく進む。最終的に、Qwen3.8-Flash-Nextが動き、OpenCodeからローカルモデルへツール付きでアクセスできるようになった。古いデスクトップでvibe codingを試せる環境ができてしまい、だいぶびっくりしている。 この記事は、そのRadeon VIIでROCmを動かし、ローカルLLMをOpenCodeにつなぐまでの記録だ。 結果は、Q2_0量子化版のFlash-Nextで短めの会話の生成が約35 tokens/s。別途、Qwen3.8-27BのQ3量子化版もllama.cppで約21〜22 tokens/s出ている。両者は量子化も実行エンジンも違うため、モデル同士の純粋な比較ではない。 この記事は2026年11月3日時点の実機メモだ。gfx906のROCmはAMDの通常の動作保証対象とは別扱いで、以下は一つの環境で動いた記録だ。アップデート後も同じように動くとは限らない。 環境と構成 項目 今回の環境 GPU Radeon VII、VRAM 16GB、gfx906 CPU Ryzen 5 3600、6コア12スレッド、AVX2 RAM 公称64GB、OSから約60.7GiB OS Ubuntu 26.04.1 LTS カーネル 7.0.0-42-generic、標準のamdgpu/KFD ROCm TheRock nightly 10.3.0a20261102、gfx906向け HIPのバージョン表示 7.18.26434 推論 Strata v0.2.10、gfx906用ソースビルド モデル ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF、Q2_0 コーディングUI OpenCode v1.19.5 Flash-Nextは本体125B、1トークンで有効になるのは約6BのMoEモデルで、別に51BのN-gram参照テーブルがある。Q2_0はパラメーターを削った小型モデルではなく、重みを量子化したものだ。Qwenの説明と量子化版の配布元を参照してほしい。 構成は大きく3層だ。 Ubuntu標準のamdgpu/KFDがGPUを扱う。 gfx906向けROCmのユーザー空間ライブラリとHIPが計算を担当する。 StrataがモデルをGPU・RAM・ストレージに配置し、ローカルAPIを公開する。 コンテナは使わず、ROCmやビルド成果物はホームディレクトリ下に分離した。システムのドライバやAPTリポジトリは変更していない。 まずamdgpuとKFDを確認する 画面が映っているだけではGPU計算が動くとは限らない。まずカーネルドライバとデバイスノードを確認した。 uname -a lspci -nnk | grep -A 4 -i 'vga|display|3d' ls -l /dev/kfd /dev/dri getfacl /dev/kfd /dev/dri/renderD128 このPCでは1002:66afのRadeon VIIをamdgpuが使用し、/dev/kfdとrenderD128にはログインユーザー向けのACLが付いていた。アクセス権の設定方法は環境によって異なる。/dev/dri/renderD128も固定とは限らないので、自分の環境で確認しよう。 gfx906向けROCmをユーザー領域へ TheRockの対応表ではgfx906はビルド対象だが、Sanity Tested・Release Readyにはチェックがない。配布されていることと、実機で動作検証されていることは別だ。 今回はAMDのnightlyから、同じ日付のSDK・ライブラリ・開発ツール・gfx906向けデバイスコードを仮想環境に揃えた。uvは別途用意した。以下の版は実験時点の固定値で、nightlyの保持期間や配布状態によっては取得できなくなる可能性がある。 uv venv ~/rocm-gfx906/venv uv pip install --python ~/rocm-gfx906/venv/bin/python \ --index-url nightly.repo.amd.com \ 'rocm[libraries,devel,device-gfx906]==10.3.0a20261102' source ~/rocm-gfx906/venv/bin/activate rocm-sdk init export ROCM_PATH="$(rocm-sdk path --root)" export HIP_PATH="$ROCM_PATH" export PATH="$ROCM_PATH/bin:$PATH" export LD_LIBRARY_PATH="$ROCM_PATH/lib:$ROCM_PATH/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" rocminfo これは配置先を分かりやすく書き直した再現例だ。実機では~/.local/share/rocm-gfx906/venvを使った。Ubuntu標準のPythonではvenvのensurepipがなかったため、uvで作成した。 rocminfoではRadeon VIIがgfx906:sramecc+:xnack-として見えた。さらにHIPで4,096要素のベクトル加算、rocBLASで2×2と512×512の行列積を検証した。SDK付属テストも27項目が通過した。 HIP compiled vector addition: PASS (4096 elements) rocBLAS SGEMM 512x512 identity product: PASS Ran 27 tests ... OK amdflangは最初、libquadmath.so.0不足で起動しなかった。今回はlibquadmath0をapt downloadしてユーザー領域に展開し、ライブラリパスを通して解決した。同様にC++開発ヘッダーとGCCの開発ライブラリもユーザー領域へ展開した。一般的な環境では、必要な開発パッケージをディストリビューションの方法で揃える方が簡単だ。無関係なUbuntu版のROCmパッケージを混ぜないようにした。 最初にllama.cppでGPU推論を確認 Strataに進む前に、llama.cppをHIP有効・gfx906指定でビルドした。使用したllama.cppのコミットはf9448cddc48c129a4abc7ff9c2a84c2b4c101c61だ。ビルド方法の基本はllama.cppの公式ドキュメントにある。 llama-cli --list-devicesにROCm0: AMD Radeon VIIが表示され、Qwen2.5-0.5BのGGUFで短い生成に成功。次にQwen3.8-27BのUD-Q3_K_XLも試した。モデルファイルは約13.2GBだ。 llama-server -m /path/to/Qwen3.8-27B-UD-Q3_K_XL.gguf \ -ngl 99 -c 4096 -np 1 --host 127.0.0.1 --port 8082 日本語の短い質問に正常回答し、会話では約21〜22 tokens/s。後にログを詳細化して、66/66層がGPUに配置されていることも確認した。別の8K設定では、約7,390トークンの合成入力を約240 tokens/sで処理した。いずれも今回のハードウェアと設定での観測だ。 Strataをgfx906向けにビルドする StrataのAMD向けドキュメントには、gfx906をwave64互換レイヤーでビルドする手順がある。配布済みのRX 6000/7000/9000向けバイナリをRadeon VIIに流用するのではなく、専用のSTRATA_HIP_GFX906経路を使う。 Strata v0.2.10のソースと、プロジェクトが指定するllama.cppコミット3cf03257f219afbe7334045ff7c6a06ac68c627dを用意した。システムにC/C++開発環境、CMake、Ninja、ROCmの開発ツールが必要だ。 STRATA_SRCとLLAMA_SRCは、それぞれ取得・展開したソースディレクトリ cmake -S "$STRATA_SRC" -B "$STRATA_SRC/build-906" -G Ninja \ -DSTRATA_HIP_GFX906=ON \ -DCMAKE_HIP_ARCHITECTURES=gfx906 \ -DSTRATA_GGML_DIR="$LLAMA_SRC" \ -DCMAKE_C_COMPILER="$ROCM_PATH/bin/amdclang" \ -DCMAKE_CXX_COMPILER="$ROCM_PATH/bin/amdclang++" \ -DCMAKE_PREFIX_PATH="$ROCM_PATH" \ -DCMAKE_BUILD_TYPE=Release cmake --build "$STRATA_SRC/build-906" --target strata strata-device -j 6 実機ではシステムにGCC開発環境がなかったため、APTからダウンロードして展開したlibstdc++-15-devとlibgcc-15-devを指す--gcc-toolchainをC/C++/HIPの各フラグに追加した。上は開発環境が通常の場所にある場合の基本形だ。 ビルド時に互換レイヤーの定義を追加した v0.2.10をそのままビルドすると、cudaDeviceGetStreamPriorityRangeとcudaStreamCreateWithPriorityが未定義になった。gfx906互換ヘッダーinclude/strata/platform/hip_compat/strata_hip.hに、対応するHIP APIへのマッピングを追加して再ビルドした。 #define cudaStreamCreateWithFlags hipStreamCreateWithFlags +#define cudaStreamCreateWithPriority hipStreamCreateWithPriority +#define cudaDeviceGetStreamPriorityRange hipDeviceGetStreamPriorityRange これは今回の版とビルドで必要だったローカル修正だ。将来の版では修正済みかもしれない。ビルド後、strata-device --selftest、gfx906向けGDN演算テスト、専門家演算の比較テストで失敗がないことを確認した。これらは全機能や長時間の安定性を保証するものではない。 Flash-Nextを準備して起動 モデルはISTA-DASLabのGGUFのQ2_0を選んだ。37.6GBの本体と28.8GBの参照テーブル、合わせて約66.4GBだ。今回使用したリビジョンはed59f92082b1e93c0e96d60a8b11aab089b52f09。両ファイルを配布元のSHA-256と照合した。 Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf SHA256 69820c02ec7d0b45ef2ebb19d6620299db749fe2aded7f39f93c6b88b199b720 Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf SHA256 316b46f3a2dbd68c900f43136ab9449f9dcc3725dfd8c794847c204bc161e113 転送が途中で停滞して再始動した際、最初の不完全ファイルを引き継がず、新しいファイルとして再取得した。完成後に古い不完全ファイル約17.6GBを削除した。大きなモデルでは、途中再開の動作と空き容量を確認したいところだ。 取得後、Strataに付属するtools/iq_pack.pyでインデックス、dense重み、トークナイザーを準備する。STRATA_GGUF_PYは上で用意した固定コミットのllama.cppにあるgguf-pyを指す。 export STRATA_GGUF_PY="$LLAMA_SRC/gguf-py" python "$STRATA_SRC/tools/iq_pack.py" \ --gguf /path/to/Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf \ --out /path/to/pack-q2_0 サーバー設定には、ビルドしたstrata、pack、2つのGGUF、付属のdata/expert-profile.binを指定する。主要なエンジン引数は次のとおりだ。 { "exe": "/path/to/build-906/strata", "cwd": "/path/to/Strata", "tokenizer": "/path/to/pack-q2_0/tokenizer", "model_name": "Qwen3.8-Flash-Next-Q2_0-gfx906", "backend": "hip", "gpu": 0, "parallel": 1, "drafting": "off", "args": [ "--pack", "/path/to/pack-q2_0", "--native", "/path/to/Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf", "--ple-gguf", "/path/to/Q2_0/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf", "--expert-profile", "/path/to/Strata/data/expert-profile.bin", "--expert-cache", "auto", "--prefill", "auto", "--max-context", "262144", "--kv", "int8", "--kv-resident", "32768", "--ple-io", "direct", "--spec", "2" ] } パスは自分の環境に置き換える。Pythonの依存パッケージは別の仮想環境にrequirements.txtから導入し、ROCmのライブラリパスも通した状態で、ソースルートからサーバーを起動した。 cd "$STRATA_SRC" python -m serve.server --engine strata --config /path/to/config.json \ --host 127.0.0.1 --port 8081 最初は--specを付け忘れ、ネイティブpackには--spec T (T >= 2)が必要だとエラーになった。--spec 2を追加すると読み込みが進み、日本語での応答に成功した。ここでの--spec 2はエンジンの指定で、MTP補助モデルを導入したという意味ではない。今回draftingはoffだ。 サーバーはローカルのみにbindしている。稼働確認は以下で行える。 curl -s 127.0.0.1 curl -s 127.0.0.1 8Kから32K、256Kへ まず8,192トークンで動作確認し、その後32,768へ増やした。約30,417トークンの合成入力の中央に置いた合言葉を正しく回収できた。入力処理は約71.8秒、約424 tokens/sだった。 次に--max-context 262144 --kv int8 --kv-resident 32768で起動。約248,458トークンの入力でも、中央の合言葉を正しく回収できた。入力処理は約640秒、約388 tokens/sだった。 試行 結果 8K設定、短い会話 約35 tokens/sの生成を観測 32K設定、約30K入力 合言葉を正しく回収、入力約424 tokens/s 256K設定、約248K入力 合言葉を正しく回収、入力約388 tokens/s 長文は同じ日本語の文章を繰り返した合成データで、検索課題も一つだ。複雑な複数文書の理解、長時間稼働、異なるコンテンツで同じ精度・速度が出ることまでは確認していない。短い出力6トークンの生成速度を、長文回答の安定速度として扱うこともしない。 256K起動時は専門家キャッシュが8,235個、空きVRAMは約790MiBだった。GPUがデスクトップ表示も担当しているので余裕は小さめだ。--expert-cache autoやVRAM予約量の調整が必要になる場合がある。Strataのメモリ設定を確認してほしい。 また、今回のStrataの既定設定ではSTRATA_ROUTE_TAIL_SKIP=7が有効で、キャッシュにない専門家の一部を省略する。ログにも回答が少し変わり得ると表示される。7,390トークンの合成入力で1回ずつ試したところ、生成は有効時36.7 tokens/s、無効時33.7 tokens/sで、回答文も少し変わった。各1回の短い応答なので、この差を一般化したり品質差を判断したりはできない。 OpenCodeにつなぐ StrataのOpenAI互換APIを使う。OpenCode v1.19.5をユーザー領域に導入し、~/.config/opencode/opencode.jsonにローカルプロバイダーを追加した。設定形式はOpenCodeのプロバイダー設定を参照した。 { "$schema": "opencode.ai", "model": "strata/Qwen3.8-Flash-Next-Q2_0-gfx906", "provider": { "strata": { "npm": "@ai-sdk/openai-compatible", "name": "Strata (local Radeon VII)", "options": { "baseURL": "127.0.0.1", "apiKey": "local", "timeout": 1200000 }, "models": { "Qwen3.8-Flash-Next-Q2_0-gfx906": { "name": "Qwen3.8 Flash Next Q2_0 (local)", "tool_call": true, "limit": { "context": 262144, "output": 8192 } } } } } } モデルIDは/v1/modelsが返すIDと一致させる。既存設定がある場合は丸ごと置き換えず、必要なプロバイダーを統合してほしい。apiKeyはこのローカルの無認証サーバーに渡すダミー値だ。 opencode models strata opencode 簡単な接続確認として、OpenCodeにテストファイルを読むよう依頼したところ、readツールを呼び出し、ファイル内の合言葉を正しく返した。コードの変更や複雑なエージェント作業の品質までは、この時点では検証していない。 振り返り Radeon VIIの16GB VRAMだけにモデル全体を載せるのではなく、GPUの専門家キャッシュ、RAM、SSDを組み合わせることで、Flash-Nextの推論までたどり着けた。gfx906は標準的な導入経路から外れているため、GPU認識だけで成功とせず、HIP、行列演算、エンジン、実モデル、ツール呼び出しの順に確かめるのが役立った。 一方、今回はnightly ROCm、ソースビルド、互換ヘッダーへの小さな修正を使っている。Qwen3.8-27BはStrataではなくllama.cppで動かしており、Strataで同じ27Bが動くという話ではない。まずは手元で動いた版と設定を記録し、更新するときは一つずつ比較していこうと思う。 この記事について 実はこの記事も、実際に環境を構築したChatGPTのセッションで書いた。会話の中で調べたこと、実行したコマンド、ログに残った測定結果をもとにChatGPTが下書きした。OpenCodeではファイル読み取りツールの呼び出しまで確認したが、複雑なコード修正の成功率はまだ測っていない。これから、この環境でどこまでできるか試していきたい。

未対応のGGUFをStrataで動かす：Qwen3.8 Distill Q4_K_MのCUDA改修と数値検証 普段、ローカル環境ではEmperoのQwen3.8-35B-A3B-Distill-GGUFをllama.cppで使用していた。最近、100B パラメーターを超えるような大規模なモデルをローカル環境で実行可能なStrataが注目されている。 Qwen3.8-35B-も同様のエンジンで実行できれば管理するツールを少なくできると考え検討を進めた。 ここでは、モデルの取得から、CUDAビルド、エラーの原因調査、llama.cppとの数値比較までを紹介する。 結論から述べると、Q4_K_Mのモデルは2か所の修正で起動し、RTX 8000 (48 GB) 単体で推論できた。 さらに、同じGGUFを使用したllama.cppとの比較では、回答546ステップの平均KL divergenceが0.00194となり、llama.cpp自身のCPU/GPU間の差 (0.00358) より小さかった。ただし、これは特定の環境・モデルでの検証結果であり、すべてのQ4_Kモデルへの対応を保証するものではない。 ! この記事は2026年11月3日時点の検証記録であり、Strata公式の修正ではない。既存のStrataインストールや他のモデル用エンジンは変更せず、検証専用ソースからビルドした。 1. 何を検証したのか LLMを動かすには、モデルの重みと推論エンジンの両方が必要になる。モデルは学習済みの数値データで、エンジンはその数値を使って次の単語を計算するプログラムと考えると分かりやすい。 今回の組み合わせは次のとおり 項目 内容 モデル Empero Qwen3.8-35B-A3B-Distill モデル形式 GGUF、Q4_K_M、約21.7 GB (配布ページ表記) 推論エンジン Strata v0.2.10 (改修版) 比較対象 llama.cpp (固定コミット) OS Ubuntu Linux 主なGPU NVIDIA Quadro RTX 8000 48 GB (Compute Capability 7.5) 別に搭載したGPU RTX 5060 Ti 16 GB (今回は使用しない) ソフトウェア Git、CMake、Ninja、C++コンパイラ、CUDA Toolkit、Pixi GGUF・量子化・MoE GGUFは、モデルの重みや構成情報を保存するファイル形式で、今回使ったQ4_K_Mは量子化形式の名称で、大きな重みを主に4ビット程度で圧縮して保存している。 圧縮するとGPUメモリ使用量を減らせる一方、エンジンは各形式に合った復号・計算処理を実装している必要がある。 今回対象のモデル名の35B-A3Bは、総パラメータがおよそ350億、1トークンの計算で使うパラメータがおよそ30億という意味である。 これは **MoE (Mixture of Experts) ** という構造によるもので、MoEでは入力に応じて一部の小さなネットワーク (Expert) を選択する。Expert内部にはGate/UpとDownという行列があり、Strataではそれぞれに対応する量子化形式を登録している。 2. 検証用の作業環境の作成 以下のコマンドは同じターミナルで順番に実行する。WORKなどの変数は、そのターミナル内でのみ有効である。パスは例示用の作業ディレクトリを基準にし、各自の環境へ置き換える必要がないようにした。 WORK="${PWD}/empero-strata-test" mkdir -p "${WORK}" && cd $_ Python環境はPixiで管理する export PIXI_VERSION="latest" export PIXI_HOME="${PWD}/tools/pixi" export PIXI_BIN_DIR="${PIXI_HOME}/bin" export PIXI_CACHE_DIR="${PWD}/.cache/pixi" export PIXI_NO_PATH_UPDATE=1 export TMP_DIR="${PWD}/tmp" mkdir -p "${PIXI_HOME}" "${PIXI_BIN_DIR}" "${PIXI_CACHE_DIR}" "${TMP_DIR}" curl -fsSL pixi.sh | sh export PATH="${PIXI_BIN_DIR}:${PATH}" pixi init --channel conda-forge . pixi add python=3.12 numpy huggingface_hub Pixiが未導入なら公式インストール手順を参照する。nvcc、CMake、Ninja、C++コンパイラが見つからない場合は、先にビルド環境を用意する。PixiはPythonパッケージを管理するが、CUDAドライバを代わりにインストールするわけではない。 3. モデルをHugging Faceから取得 今回のモデルは、Hugging Faceの以下のリポジトリで配布されている。 huggingface.co 複数の量子化形式があるが、検証したファイルはQwen3.8-35B-A3B-Q4_K_M.ggufである。hf downloadなら、必要なGGUFだけを取得できる。 mkdir -p "$WORK/models" pixi run hf download \ empero-ai/Qwen3.8-35B-A3B-Distill-GGUF \ Qwen3.8-35B-A3B-Q4_K_M.gguf \ --local-dir "$WORK/models" 4. Strata v0.2.10のソースを取得 StrataはGPU向けのローカルLLM推論エンジンである。今回は通常のアプリやインストール済みのstrataコマンドを改造せず検証専用のソースを取得した。 SRC="$WORK/Strata-v0.2.10-empero" git clone --branch v0.2.10 --depth 1 \ Niko1221/Strata "$SRC" cd "$SRC" git describe --tags --always -> v0.2.10 ! 当初、公式ソースのままモデルを起動しようとすると、GGUFの形式がStrataの通常のモデル選択に登録されていなかった。そこでstrata launchではなく、Qwen3.6系のGGUFを読み込む実行ファイルstrata-q36を直接使用した。 エラー1：SSMのQ4_Kを読み込めない 実行時、次のエラーで停止した。 strata-q36: blk.0.ssm_alpha.weight is Q4_K, expected F32, BF16, F16 or Q8_0 ssm_alphaは、モデル内部の Gated DeltaNet (状態を保持しながら計算する仕組み) で使われる重みで、Strataのupload_f32()はこれをGPU上で32ビット浮動小数点数 (F32) として利用するが、元のGGUFにあるQ4_K形式を展開する処理が不足していた。 つまり、保存形式を読み取るコードが対応していなかった。 5. パッチ1：SSMのQ4_K復号を追加 src/qwen35/q36_engine.cu内のupload_f32()にQ4_KからF32への展開処理を追加した。次のPythonスクリプトは、既存のQ8_0分岐の直前にコードを挿入する。元のGGUFは書き換えない。 pythonコード cd "$SRC" pixi run python3 - <<'PY' from pathlib import Path p = Path("src/qwen35/q36_engine.cu") s = p.read_text() anchor = " else if (t.type == 8 && h.size() % 32 == 0) {" marker = "// Empero Q4_K -> F32" patch = r''' else if (t.type == 12 && h.size() % 256 == 0) { // Empero Q4_K -> F32 for (size_t bi = 0; bi < h.size() / 256; ++bi) { const uint8_t* b = src + bi * 144; const uint16_t hd = uint16_t(b[0]) | (uint16_t(b[1]) << 8); const uint16_t hm = uint16_t(b[2]) | (uint16_t(b[3]) << 8); const float d = f16_to_f32(hd); const float dmin = f16_to_f32(hm); const uint8_t* scales = b + 4; const uint8_t* qs = b + 16; for (int j = 0; j < 8; ++j) { int sc, m; if (j < 4) { sc = scales[j] & 63; m = scales[j + 4] & 63; } else { sc = (scales[j + 4] & 15) | ((scales[j - 4] >> 6) << 4); m = (scales[j + 4] >> 4) | ((scales[j] >> 6) << 4); } for (int k = 0; k < 32; ++k) { const uint8_t q = qs[32 * (j / 2) + k]; const int v = (j & 1) ? (q >> 4) : (q & 15); h[bi * 256 + j * 32 + k] = (d * sc) * v - (dmin * m); } } } } ''' if marker in s: print("Patch 1 already applied") else: assert s.count(anchor) == 1, "想定と異なるソース構造です" p.write_text(s.replace(anchor, patch + anchor, 1)) print("Patch 1 applied") PY Patch 1 applied Q4_Kでは、256個の値を1ブロックとして圧縮しており、この実装では各ブロックのスケール情報と4ビット値からF32を復元する。復号方法はStrata内の参照実装q36ref.cuと照合した。なお、この修正は今回の形状で動作したものであり、あらゆるテンソル形状に対して検証済みという意味ではない。 6. パッチ2：Expert DownのQ4_Kの有効化 最初の修正でSSMのエラーは解消したが、次に以下のエラーが出た。 strata-q36: blk.5.experts are Q4_K/Q4_K, which the native expert kernels do not take (build with -DSTRATA_Q6K_EXPERTS=ON for Q6_K) 末尾にQ6_Kのビルドオプションが表示されているが、今回の原因はQ4_KをExpertのDown側に登録していなかったことだった。 src/kernels/cuda/iq_kernels.cuにはSTRATA_GU_FMTSとSTRATA_D_FMTSという対応形式一覧があり、Q4_K (型番号12) はGate/Up側には存在したがDown側にはなかった。そこで、Down側の一覧にX(12)を追加した。 pythonスクリプトcd "$SRC" python3 - <<'PY' from pathlib import Path p = Path("src/kernels/cuda/iq_kernels.cu") lines = p.read_text().splitlines(keepends=True) found = changed = 0 for i, line in enumerate(lines): if not line.startswith("#define STRATA_D_FMTS(X)"): continue found += 1 if "X(12)" in line: continue assert line.count("X(13)") == 1 lines[i] = line.replace("X(13)", "X(12) X(13)", 1) changed += 1 assert found == 2, f"想定外の定義数: {found}" if changed: p.write_text("".join(lines)) print(f"Patch 2: {changed} definitions updated") PY -> Patch 2: 2 definitions updated 差分確認するなら git diff -- src/qwen35/q36_engine.cu src/kernels/cuda/iq_kernels.cu 既存のQ4_K計算コードをDown側でも呼び出せるようにする修正となっている。変更内容は小さいが、ソース側で対応を宣言しただけでは数値が正しいとは限らないため後述の検証を行った。 7. 修正済みStrataをCUDAビルド CMakeはビルド設定を作るツール、Ninjaは実際のコンパイル処理を進めるツールである。 今回はRTX 8000 (SM 7.5) とRTX 5060 Ti (SM 12.0) の両方に対応する形式でコンパイルした。実際のベンチマークにはRTX 8000だけを使用している。 BUILD="$SRC/build-q36-empero" cmake -S "$SRC" -B "$BUILD" -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DSTRATA_ENABLE_CUDA=ON \ -DSTRATA_NATIVE_EXPERTS=ON \ -DSTRATA_Q6K_EXPERTS=ON \ -DSTRATA_MMQ_KQUANTS=ON \ -DCMAKE_CUDA_ARCHITECTURES="75-real;120-real" cmake --build "$BUILD" --target strata-q36 -j 32 -> [165/165] Linking CXX executable strata-q36 SM 12.0に未対応のCUDA Toolkitではビルドできないので、その場合は対応するCUDA Toolkitを使用するか、RTX 8000専用として-DCMAKE_CUDA_ARCHITECTURES=75に変更する。CPUスレッド数に合わせて-j 32は調整できる。 8. 起動と速度測定 --modelでGGUFファイルを指定する。CUDA_VISIBLE_DEVICES=0はGPUを1台に限定する設定である。番号0がどのGPUかはnvidia-smiで確認しておく。 GGUF=${WORK}/models/Qwen3.8-35B-A3B-Q4_K_M.gguf CUDA_VISIBLE_DEVICES=1 "$BUILD/strata-q36" \ --model "$GGUF" \ --max-context 8192 \ --bench-pp 2048 \ --bench-tg 128 \ --mtp 0 \ --no-warmup Logstrata-q36: <...>/empero-strata-test/models/Qwen3.8-35B-A3B-Q4_K_M.gguf: 753 tensors (layout checked), 1.65 GiB on the GPU in 1.1 s (the routed experts are placed after the engine's buffers) strata: K-quant prompt kernels from <...>/empero-strata-test/Strata-v0.2.10-empero/build-q36-empero/libstrata_kq.so (STRATA_MMQ_KQ=1) strata-q36: prompt experts on MMQ (int8) strata-q36: context 8192 cells, prompt chunk 4096, MTP block loaded, 42.16 GiB of VRAM free strata-q36: every routed expert in VRAM (18.66 GiB) in 6.6 s bench 0: pp2048 3391.4 tok/s, tg128 101.7 tok/s bench 1: pp2048 3470.4 tok/s, tg128 102.1 tok/s bench 2: pp2048 3478.2 tok/s, tg128 102.0 tok/s ppは入力を読み込む速度 (Prefill) 、tgは回答を1トークンずつ生成する速度 (Decode) である。--mtp 0は投機的デコードを無効化する指定であり、今回の比較では計算経路を単純にするために使用した。 実測では、753テンソルの構造検証を通過し、Expertの重み18.66 GiBをすべてVRAMに配置できた。 試行 Prefill (pp2048) Decode (tg128) 0 pp2048 3391.4 101.7 1 pp2048 3470.4 102.1 2 pp2048 3478.2 102.0 これはRTX8000単体、Context上限8,192の測定である。測定に使用したのはベンチマーク用トークン列なので自然言語の回答品質を示す数字ではない。 9. 比較用のllama.cppを取得してビルド 「エラーなく動く」ことと「計算が正しい」ことは異なる。そこで、同じGGUFを対応済みのllama.cppでも実行し、計算結果を比較した。 再現性を高めるため、llama.cppはStrata v0.2.10が参照しているコミット3cf03257f219afbe7334045ff7c6a06ac68c627dに固定した。 LLAMA_SRC="$WORK/llama.cpp-parity" LLAMA_SHA="3cf03257f219afbe7334045ff7c6a06ac68c627d" git clone --depth 1 ggml-org/llama.cpp "$LLAMA_SRC" git -C "$LLAMA_SRC" fetch --depth 1 origin "$LLAMA_SHA" git -C "$LLAMA_SRC" checkout --detach "$LLAMA_SHA" git -C "$LLAMA_SRC" rev-parse HEAD ビルドするのはllama.cppの推論ライブラリである。 LLAMA_BUILD="$LLAMA_SRC/build-parity" cmake -S "$LLAMA_SRC" -B "$LLAMA_BUILD" -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_CUDA=ON \ -DBUILD_SHARED_LIBS=ON \ -DLLAMA_BUILD_TESTS=OFF \ -DLLAMA_BUILD_EXAMPLES=OFF \ -DCMAKE_CUDA_ARCHITECTURES=75 cmake --build "$LLAMA_BUILD" --target llama -j 32 -> [222/222] Creating library symlink bin/libllama.so.0 bin/libllama.so 次に、Strata付属のq36_llama_dump.cppをコンパイルした。このツールはllama.cppのトークンID・各層の出力・ logits (次トークンの候補ごとの点数) をファイルに保存するためのもの。 TOOLS="$WORK/tools" mkdir -p "$TOOLS" g++ -O2 -std=c++17 \ "$SRC/tools/qwen35/q36_llama_dump.cpp" \ -I"$LLAMA_SRC/include" \ -I"$LLAMA_SRC/ggml/include" \ -L"$LLAMA_BUILD/bin" \ -Wl,-rpath,"$LLAMA_BUILD/bin" \ -lllama -lggml -lggml-base -pthread \ -o "$TOOLS/q36_llama_dump" ldd "$TOOLS/q36_llama_dump" | grep -E 'llama|ggml|not found' Strataの独立した参照実装もビルド cmake --build "$BUILD" --target strata-q36ref -j 32 -> [2/2] Linking CXX executable strata-q36ref lddでnot foundが表示されなければ、共有ライブラリが解決できている。 10. 数値的な正しさの比較 文章を読んで「自然だから正しい」と判断するだけでは、GPU用の計算処理の検証として不十分である。今回はStrata公式のtools/qwen35/にある比較スクリプトを使用した。 比較の対象は3種類 llama.cpp GPU：比較の基準となる計算結果 llama.cpp CPU：同じソフトウェアでもCPUとGPUでどれだけ差が出るか Strata：今回修正した高速エンジン。必要に応じて逐次参照実装strata-q36refとも比較 まず10種類の問題で回答を比較 parity.pyは、短い物語・Pythonコード・計算問題・説明文など、10種類のプロンプトで比較を行う。 RESULTS="$WORK/results" mkdir -p "$RESULTS" && cd $_ pixi run python "$SRC/tools/qwen35/parity.py" \ --ref "$BUILD/strata-q36ref" \ --dump "$TOOLS/q36_llama_dump" \ --model "$GGUF" \ --out "$RESULTS/parity-10" \ --n 64 Logp00: 48/64 greedy tokens equal; layers max rel err 0.7524 (layer 36), median 0.03055; diverges at {'step': 48, 'strata': [34227, 0.004119873046875], 'llama': [25292, 0.03865623474121094]} p01: 3/64 greedy tokens equal; layers max rel err 0.1742 (layer 39), median 0.04114; diverges at {'step': 3, 'strata': [1445, 0.04456138610839844], 'llama': [1005, 0.011854171752929688]} p02: 64/64 greedy tokens equal; layers max rel err 0.4023 (layer 26), median 0.04085 p03: 8/64 greedy tokens equal; layers max rel err 0.1388 (layer 11), median 0.03517; diverges at {'step': 8, 'strata': [8364, 0.19606590270996094], 'llama': [27286, 0.184661865234375]} p04: 15/15 greedy tokens equal; layers max rel err 0.2196 (layer 36), median 0.04316 p05: 64/64 greedy tokens equal; layers max rel err 0.2272 (layer 39), median 0.02473 p06: 17/64 greedy tokens equal; layers max rel err 0.1535 (layer 35), median 0.03603; diverges at {'step': 17, 'strata': [506, 0.10254287719726562], 'llama': [321, 0.23025894165039062]} p07: 52/52 greedy tokens equal; layers max rel err 0.3421 (layer 36), median 0.03568 p08: 64/64 greedy tokens equal; layers max rel err 0.1636 (layer 35), median 0.03495 p09: 17/24 greedy tokens equal; layers max rel err 0.1314 (layer 36), median 0.03526; diverges at {'step': 17, 'strata': [47, 0.023038864135742188], 'llama': [1948, 0.08147048950195312]} identical greedy answers (up to 64 tokens or the end of the answer): 5 of 10 prompts; worst layer error 0.7524 この検証では、10問中5問が回答終了まで同じGreedyトークン列となった。残りの5問は途中で違うトークンを選んだ。ただし、ひとたび違うトークンを選ぶと、その後の入力履歴も変わるので、これだけでは数値誤差の大きさを判断できない。 同じ入力履歴を与えて確率分布を比較 そこでkl.pyを使った。ここではllama.cppが生成したトークンを各実装に強制的に与えるTeacher forcingを行う。たとえるなら、2人の学生に前の問題の答えを同じように渡し、次の問題でどれだけ異なる計算結果を出すか比較するような方法である。多分。 次トークンの確率分布の差を、KL divergenceで測定する。KLは0に近いほど分布が一致する。また、確率が最大のトークン (Top-1) が同じかも調べる。 pixi run python "$SRC/tools/qwen35/kl.py" \ --ref "$BUILD/strata-q36" \ --dump "$TOOLS/q36_llama_dump" \ --model "$GGUF" \ --parity "$RESULTS/parity-10" \ --out "$RESULTS/kl-10" \ --ref-args "--max-context 4096 --chunk 512 --mtp 0 --no-graph --no-warmup" Logp00: 65 steps (65 answer); cpu: KL 0.01036 (answer 0.01036) top-1 0.938; strata: KL 0.00314 (answer 0.00314) top-1 0.985 p01: 65 steps (65 answer); cpu: KL 0.00073 (answer 0.00073) top-1 1.000; strata: KL 0.00053 (answer 0.00053) top-1 1.000 p02: 65 steps (65 answer); cpu: KL 0.00172 (answer 0.00172) top-1 0.985; strata: KL 0.00218 (answer 0.00218) top-1 0.969 p03: 65 steps (65 answer); cpu: KL 0.00475 (answer 0.00475) top-1 0.985; strata: KL 0.00259 (answer 0.00259) top-1 0.985 p04: 65 steps (15 answer); cpu: KL 0.01299 (answer 0.00063) top-1 0.985; strata: KL 0.01192 (answer 0.00110) top-1 0.985 p05: 65 steps (65 answer); cpu: KL 0.00139 (answer 0.00139) top-1 0.985; strata: KL 0.00097 (answer 0.00097) top-1 0.985 p06: 65 steps (65 answer); cpu: KL 0.00530 (answer 0.00530) top-1 0.969; strata: KL 0.00322 (answer 0.00322) top-1 0.985 p07: 65 steps (52 answer); cpu: KL 0.01820 (answer 0.00106) top-1 0.985; strata: KL 0.01739 (answer 0.00060) top-1 0.969 p08: 65 steps (65 answer); cpu: KL 0.00251 (answer 0.00251) top-1 0.985; strata: KL 0.00132 (answer 0.00132) top-1 1.000 p09: 65 steps (24 answer); cpu: KL 0.02208 (answer 0.00619) top-1 0.954; strata: KL 0.01900 (answer 0.00447) top-1 0.969 -- all steps cpu vs llama.cpp HIP: mean KL 0.00800 (median 0.000373, p99 0.1727), top-1 0.9769 over 650 steps strata vs llama.cpp HIP: mean KL 0.00623 (median 0.000241, p99 0.1144), top-1 0.9831 over 650 steps ratio strata / floor: 0.78 -- answers only cpu vs llama.cpp HIP: mean KL 0.00358 (median 0.000307, p99 0.0339), top-1 0.9780 over 546 steps strata vs llama.cpp HIP: mean KL 0.00194 (median 0.000149, p99 0.0194), top-1 0.9853 over 546 steps ratio strata / floor: 0.54 kl.pyでは、修正済みの高速Strataとllama.cpp GPUを、同一のトークン列を入力するTeacher-forced方式で比較した。また、llama.cpp CPU/GPU間の誤差を基準値とした。ログ中のHIPは固定表記であり、実際にはCUDAを使用した。 測定範囲 比較対象 Mean KL Top-1一致率 全650ステップ llama.cpp CPU 0.00800 97.69% 全650ステップ 修正済みStrata 0.00623 98.31% 回答546ステップ llama.cpp CPU 0.00358 97.80% 回答546ステップ 修正済みStrata 0.00194 98.53% 回答範囲での平均KL比は0.54倍となり、Strata公式の評価目安である1.5倍以内を満たした。 一方、Greedy生成では10プロンプト中5件が完全一致した。残り5件は途中で生成トークンが分岐したが、Teacher-forced比較では入力履歴を統一しているため、98.53%のTop-1一致率と矛盾しない。 別途実施した17ステップの比較では、高速Strataと参照実装strata-q36refのTop-1が17/17で一致し、平均KLは0.00029681だった。 以上から、今回のQ4_K対応による大きな数値的不整合は認められなかった。ただし、個々のQ4_K演算の完全な正しさを証明するものではない。 おわり。 参考資料 Empero Qwen3.8-35B-A3B-Distill-GGUF (Hugging Face) Strata v0.2.10 (GitHub Release) SSM読み込み処理：q36_engine.cu ExpertのCUDAカーネル：iq_kernels.cu llama.cppの出力取得ツール：q36_llama_dump.cpp Strataの生成一致検証：parity.py StrataのKL検証：kl.py Hugging Face CLI：hf download