TheTom

Strix Halo 32 GB laptop: gufo fork hits 99% of memory bandwidth, runs a 27B ~30% faster than upstream gufo and fits 8x the context Written by an AI (Claude). I set the goals and checked the results; every number was measured on my laptop, and the scripts to rerun them are in the repo. This is my first public kernel project: about two months of continuous work. tubo is my fork of gufo. Hardware: a laptop with the Ryzen AI MAX+ 392 (12 cores, Radeon 8060S) and 32 GB, so a smaller and more power-limited box than most 395 / 128 GB setups here. Model: Qwen3.8-27B. TL;DR - tubo reads the 27B weights at 73% of the 256 GB/s limit vs 53% in upstream gufo. A plain read test reaches 99%, so there is still room. - Same model, same settings, only the engine differs: 37.8 vs 29.6 tok/s (+28%), same quality (HumanEval 155 vs 156). - More weight types: loads IQ3_XXS quants that upstream cannot, for example the apex* model. - Apex* model + turbo6 KV cache: near-lossless vs UD-Q4 + fp16 KV (HumanEval 154 vs 155, MATH-500 76.6% vs 77.8%), 15% faster (53.3 vs 46.5 tok/s), 86% faster at 32k context, 17% less memory, and 128k context fits in 32 GB. |Qwen3.8-27B, same laptop|gufo|tubo|tubo + apex* + turbo6| |:-|:-|:-|:-| |weight read, % of memory limit|53%|73%|75%| |decode, video question (1024 tokens)|26.1 tok/s|32.0 tok/s|34.8 tok/s| |decode, 512-token answer, 10 runs|29.6 tok/s|37.8 tok/s|45.1 tok/s| |HumanEval passed (of 164)|156|155|154| |speed over the HumanEval run|36.1 tok/s|46.5 tok/s|53.3 tok/s| |RAM at 4k context|21.7 GiB|21.3 GiB|17.6 GiB| |longest context on 32 GB|4k|32k|128k tested, 262k loads| Memory read efficiency (share of the 256 GB/s theoretical limit, GPU pinned 1700 MHz): |what|GB/s|% of limit| |:-|:-|:-| |plain memory read (test kernel)|252.8|98.8%| |many small reads, launches overlapped (test kernel)|246.9|96.4%| |best real weight kernel (IQ4_XS, 1 token)|231.6|90.5%| |all 27B weights, as the server reads them, 1 token|194|76%| |all 27B weights, 8 tokens per step (speculative decoding)|186|73% (gufo: 53%)| Video, same question on gufo | tubo | tubo + apex* + turbo6, side by side, real time: youtu.be The three answers are not word-for-word the same: tubo adds numbers in a different order than gufo, which flips a rare token at temperature 0, and apex is a different model. Quality is checked below, not by matching text. Quality: upstream gufo (UD-Q4 + fp16 KV) vs tubo + apex* + turbo6 (temperature 0, thinking off): |test|gufo, UD-Q4, fp16 KV|tubo, apex*, turbo6| |:-|:-|:-| |HumanEval (164 problems)|156|154 (difference is noise, p = 0.79)| |MATH-500|not run (tubo, same UD-Q4 + fp16: 389)|383 (within noise, p = 0.41)| |hidden facts found in a long prompt|cannot load 32k|40/40 at 128k| *apex = a smaller 3-4-bit quant of a Qwen3.8-27B fine-tune (IsValorum). turbo6 = a 6-bit KV cache from TheTom/llama-cpp-turboquant, by JakeATX. Numbers are from this laptop, GPU pinned at 2000 MHz unless noted. Vanilla gufo runs slower here than in its own published tables (we have not found why: 12 vs 16 cores, laptop power limit, or 32 vs 128 GB), so compare both engines on the same machine. Prefill: with the same model and settings, tubo is no faster than gufo (tie up to ~1k tokens, +4-6% at 2-3k). The gain on long prompts comes from splitting part of the work onto the chip's NPU (same tubo build, same day): |prompt tokens|GPU only|GPU + NPU| |:-|:-|:-| |2k|449 tok/s|458 tok/s| |4k|440 tok/s|484 tok/s (+10%)| |8k|428 tok/s|472 tok/s (+10%)| If you have a 395 with 64-128 GB, I would love your numbers: both engines on your box, same model and settings. REPRODUCE.md has the exact commands. Issues and corrections welcome. Full tables, method, failures, credits and how to reproduce: BonkusCheemus/tubo EDIT: added the video link, which was missing when the post went up.

@wikiwayne Interesting, thanks for the data. i've been mainly just working on model bringup all night. i think this model deserves an offlabel card. may be some gotchas there (context: TheTom/offlabel)

Your setup is definitely a monster compared to mine so it should give much better results. I also believe you run dense models with decent results with that setup. Not sure what llamacpp build you are using but on mine using this fork: TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw): [*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2 [qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning = on n-predict = 32768 reasoning-format = deepseek reasoning-budget = 16384 reasoning-budget-message = "Proceed to final answer" reasoning-preserve = true swa-full = true np = 2 batch-size = 4096 ubatch-size = 2048 cache-reuse = 256 ngl = auto ctx-checkpoints = 33 ctx-size = 262144 no-mmproj = true no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 2 I am also certain that these flags could be improved somehow depending on your system.