HazyResearch

Research Paper Deep Dive: Flash Attention 2 — Optimizing Transformer Attention Research Paper Deep Dive: Flash Attention 2 — Optimizing Transformer Attention *Original Publication Notice: This article was originally published on the Botmartz Engineering Blog. Canonical URL: blog.botmartz.com --- Paper: "Flash-Attention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (Dao et al., 2022) ArXiv: arxiv.org Key Insight: Attention's bottleneck isn't compute (FLOP), it's memory I/O. Flash Attention reorders operations to minimize memory transfers, achieving 2-4× speedup. The Problem: Attention is Memory-Bound Standard attention: Q, K, V are (seq_len, d) 1. Compute QK^T: (seq_len, seq_len) - stored in slow GPU memory 2. Compute softmax over seq_len 3. Compute attention @ V The (seq_len, seq_len) matrix is huge: 4K context = 16M elements = 64MB at float32. GPU can't keep this in fast cache (SRAM is 10-100MB). Constant memory transfers kill performance. Flash Attention: Tiling & Recomputation Key idea: Block-wise computation 1. Divide Q, K, V into blocks 2. Compute attention per block, store only output (not full attention matrix) 3. Recompute blocks during backward (trade memory for compute) Block size: Fits in SRAM (~20KB per block) Result: 2-4× fewer memory transfers Implementation Sketch import torch def flash_attention_v2(Q, K, V, block_size=128): """ Q, K, V: (batch, seq_len, d_head) Computes attention using tiling to minimize memory I/O """ N, T, d = Q.shape # Output and normalization O = torch.zeros_like(Q) l = torch.zeros(N, T) # Row sums for softmax normalization m = torch.full((N, T), -float('inf')) # Row maxes for numerical stability # Process in blocks for block_start in range(0, T, block_size): block_end = min(block_start + block_size, T) block_len = block_end - block_start Q_block = Q[:, block_start:block_end, :] # (N, block_len, d) # Compute attention with full K, V (this is where flash attention differs) # Flash: compute attention in blocks, recompute during backward for kv_start in range(0, T, block_size): kv_end = min(kv_start + block_size, T) K_block = K[:, kv_start:kv_end, :] V_block = V[:, kv_start:kv_end, :] # Attention scores S = torch.matmul(Q_block, K_block.transpose(-2, -1)) / (d ** 0.5) # Numerical stability: subtract max m_new = torch.max(m[:, block_start:block_end, None], S.max(dim=2, keepdim=True)[0]) exp_S = torch.exp(S - m_new) # Update output (weighted by attention) l_new = torch.exp(m[:, block_start:block_end, None] - m_new) * l[:, block_start:block_end, None] + exp_S.sum(dim=2, keepdim=True) O[:, block_start:block_end, :] = ( torch.exp(m[:, block_start:block_end, None] - m_new) * O[:, block_start:block_end, :] + torch.matmul(exp_S, V_block) ) / l_new m[:, block_start:block_end] = m_new.squeeze(-1) l[:, block_start:block_end] = l_new.squeeze(-1) return O Benchmarks Model: LLaMA 7B (seq_len=4096, flash_attention vs standard) Standard Attention: - Memory: 8GB for attention matrix - Throughput: 100 tok/s - Latency: 10ms per token Flash Attention v2: - Memory: 1GB (no full matrix stored) - Throughput: 330 tok/s - Latency: 3ms per token 3.3× speedup, 8× memory reduction! Our Analysis: Why This Matters Flash Attention is a breakthrough because it proves that theoretical complexity isn't everything. Attention is O(n²) in both FLOP and memory. Flash doesn't reduce FLOP, but reduces memory I/O (the real bottleneck). This is a lesson: profile your code, understand your hardware bottleneck, and optimize accordingly. Practical Implication: Enable Flash Attention in production: from torch.nn.functional import scaled_dot_product_attention # Enable flash attention automatically (PyTorch 2.0+) with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_mem_efficient=False, enable_math=False): output = scaled_dot_product_attention(Q, K, V) References 1. Paper: Flash-Attention (Dao et al., 2022) 2. Flash Attention v2: Even faster with better algorithms (Dao et al., 2023) 3. Code: HazyResearch/flash-attention Conclusion Flash Attention teaches that hardware awareness is critical for optimization. Understanding GPU memory hierarchies (SRAM vs HBM) and I/O patterns enables dramatic speedups. This is the frontier of research: algorithmic innovations that respect hardware constraints. Next: we'll analyze another frontier paper—RoPE (Rotary Position Embeddings). --- Published by Botmartz AI Solutions — Frontier AI Engineering, Custom RAG, and Agentic Systems.

Two PyTorch ops that stopped a CUDA text model on an Apple GPU The Next-Concept Modeling eval scripts would not start on this Mac. PyTorch 2.14.1 reported the Apple GPU as available, and evaluate.py still exited with no GPU visible; pass --gpus. The model itself did not need a rewrite. The released configs already turn FlexAttention off and use scaled dot-product attention, linear layers, embeddings, and RoPE. Three call sites assumed CUDA. Two of those assumptions are real Metal limits, and the third is just device plumbing. The launcher only counted CUDA devices evaluate.py asked torch.cuda.device_count() and then set CUDA_VISIBLE_DEVICES for each shard. On a Mac that count is zero, so the process stopped before it loaded a checkpoint. A short helper keeps the NVIDIA path and adds the Apple GPU when CUDA is absent: def accelerator() -> torch.device: if torch.cuda.is_available(): return torch.device("cuda") if torch.backends.mps.is_available(): return torch.device("mps") return torch.device("cpu") An Apple Silicon Mac has one GPU, so the launcher starts one process and does not set CUDA_VISIBLE_DEVICES. Checkpoint loads and .cuda() calls in the shard and the scorer use that same device. GPT-2, which scores generative perplexity after generation finishes, moves there too. Metal has no float64 XSum and LM1B apply a frequency penalty and an adjacency penalty before the argmax. That loop did logits.double() on the model device. On MPS that raises: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead. The loop is a Python argmax over a short sequence, so the numerics can stay in float64 on the CPU. The transformer forward stays on the GPU. def as_float64(t: torch.Tensor) -> torch.Tensor: if t.device.type == "mps": t = t.detach().to("cpu") return t.double() The unigram tables used by that penalty are loaded on the CPU as well. On CUDA they stay on the GPU, in float64, which is what the original script did. Area pooling requires lengths that divide Generation then died inside the residual vector quantizer: RuntimeError: Adaptive pool MPS: input sizes must be divisible by output sizes. Non-divisible input sizes are not implemented on MPS device yet. F.interpolate(..., mode="area") on a 1D tensor is adaptive average pooling. The Sudoku tokenizer's levels end at 89 and step through 50, 67, and the other lengths in between. 89 is not a multiple of 50, so Metal refuses the op. PyTorch's own error says to run that case on the CPU. The tensor is a handful of codebook vectors, so the copy is cheap and the attention stays on the GPU: def _interpolate(x, size, mode): out_len = size[-1] if x.device.type == "mps" and mode == "area" and x.shape[-1] % out_len != 0: return F.interpolate(x.cpu(), size=size, mode=mode).to( device=x.device, dtype=x.dtype ) return F.interpolate(x, size=size, mode=mode) This is the same ATen op the CUDA path uses, run on the CPU only when Metal cannot take the shape. PyTorch issue 96056 is the tracking bug the error points at. What I left alone FlexAttention has had Metal kernels since PyTorch 2.13. Kiarina's M1 Max measurements are the reason I did not turn it on. A 32,768-token window of 256 was 7.83× faster than SDPA. Causal attention at 8,192 tokens went the other way: SDPA was about 19× faster. NCM's levels are a few hundred tokens, the released configs set use_flex_attention: false, and the MPS FlexAttention path Kiarina tested is forward-only. SDPA is the kernel that matches this model. I also left the weights in float32. The penalty is sensitive to ties, and I wanted the first samples close to the CUDA run before trying fp16. What actually ran On this machine torch.mps.recommended_max_memory() returned 17.76 GiB. Two easy Sudoku boards (--n 2) scored 2/2 solved, 100% cell accuracy, and 100% valid grids. One XSum document produced a real summary ("The number of beds made available for homeless people living in accommodation in Wales has risen in last year...") and the merge step scored it with GPT-2 on the same GPU: ROUGE-1 27.91, ROUGE-2 4.88, ROUGE-L 23.26, generative perplexity 28.34, 26 words. That is one document. It is evidence the scorer ran, not a comparison to the paper. Memory, if you run the large configs Those two CPU copies are small. A large offload is a different story. Nefayran measured a 4 GiB MPS tensor moved with .to("cpu"): the Activity Monitor footprint dropped to about 0.1 GiB until the CPU actually read the copy, and then it was 4.2 GiB again. On Apple Silicon the CPU and GPU share one pool, so parking weights on the CPU does not hand memory back to the GPU. The same series shows why a big eval can get slow without ever raising out-of-memory. On a 48 GB Mac the default MPS high-watermark sits above physical RAM, so the job swaps first. Setting the cap takes both ratios, before the first MPS tensor: PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.8 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8 python evaluate.py ... I would watch swap on LM1B-large. That generator is 36 layers at width 1280, and the scorer is GPT-2 large in the next process. If swap climbs, drop batch_size in the eval config. Generation and scoring do not overlap, which already helps. What I would do differently I would not port the model to MLX to get a first result. The weights are a PyTorch state_dict, and the math already ran once the device and the two unsupported ops were handled. I would run a full split before quoting quality numbers. Two solved Sudoku boards and one summary only prove the graph executes on Metal. --- Next-Concept Modeling is from Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini, and Christopher Ré, arXiv:2610.14162. The code above is a local change to the MIT-licensed repo. This article was drafted with AI assistance.

I was inspired by the NCM work the @Research_Hazy team put out a few days ago! I propose we call it "almost autoregressive" But in any case, I made a little video (with claude) to better understand what they were doing Link if you want to learn more! hazyresearch/ncm t.co

NCM vs LLM: What Next-Concept Modeling Changes, and What It Doesn’t > Short answer > > NCM (Next-Concept Modeling) is a new research method from Stanford and Together AI that writes text coarse-to-fine: it drafts the overall structure first, then fills in detail, predicting many pieces in parallel instead of one token at a time like today’s LLMs. Early results are fast and efficient, but they are measured against diffusion language models at small scale, not against ChatGPT-class LLMs. It is worth watching, not acting on yet. - 10–31× - faster generation than strong diffusion language models, in the authors’ tests - ~4–5× - fewer training tokens, including tokenizer training - 725M - largest model tested: far smaller than frontier LLMs - 0 - hosted APIs: it is a research release with code on GitHub NCM vs LLM is a comparison between two ways of generating text. Today’s large language models are autoregressive: they write one token at a time, each depending on the ones before. Next-Concept Modeling, published by Stanford’s Hazy Research lab and Together AI in October 2026, builds the text as a hierarchy, outlining it first and then refining every part in parallel. That makes it potentially much faster, but it is still early research. Why generation speed is a business issue, not just a technical one It is easy to treat “faster text generation” as an engineering detail. For decision makers it is three business levers at once: cost, response time and what your products can do. - Cost: most of what you pay a model provider, or spend on your own GPUs, goes on generating output. If the same answer needs far fewer sequential steps, the same hardware can serve more requests. - Response time: people abandon slow tools. Support agents, sales teams and customers notice the difference between an answer in one second and an answer in eight. - New products: some ideas are uneconomic today because generation is too slow or expensive, such as drafting a personalised summary for every customer record overnight, or checking every contract clause in real time while someone types. > Illustrative example > > A support team uses an AI assistant to draft replies of around 300 words. An autoregressive model produces those words one token at a time, so a long reply means hundreds of steps in a row. A coarse-to-fine model would produce the reply in a fixed number of levels, each computed in parallel. If that holds up at production scale, the same team could get drafts noticeably faster and the provider could serve more agents per GPU. This is a thought experiment, not a measured result. How today’s LLMs generate text Models such as GPT, Claude and Gemini are autoregressive: they predict the next token, a word or part of a word, append it, and repeat. This is reliable and scales extremely well, which is why it powers nearly every AI product today. Its weakness is speed: a 1,000-word answer needs well over a thousand sequential steps, and each step must wait for the one before. Much of the cost of running AI products comes from that sequential decoding. Researchers have tried alternatives. Diffusion language models start from noise and refine the whole text over many steps in parallel. They avoid strict left-to-right writing but typically need many passes to reach good quality, which is the gap NCM targets. How Next-Concept Modeling works NCM borrows an idea from image generation, where models paint a rough picture and then add detail. It works in two stages: - A multiscale tokenizer learns to represent text as a hierarchy of discrete codes: a few codes for the broad structure at the top level, more codes for finer detail at each level below. - A transformer then generates the text level by level, coarse to fine. All codes within a level are predicted at once, in parallel, and each level is conditioned on the levels above it. - A decoder turns the finished hierarchy back into words. Because each level is produced in one parallel step, the number of sequential steps depends on the number of levels, not the length of the text. That is the source of the speed-up. NCM vs LLM at a glance Autoregressive LLMsNext-Concept ModelingHow text is producedOne token at a time, left to rightCoarse to fine, a whole level in parallelSequential stepsGrows with the length of the outputFixed by the number of levelsMaturityProduction at massive scaleResearch release, October 2026Largest modelsHundreds of billions of parameters725 million parameters in the paperCompared against—Mainly diffusion language models; an autoregressive baseline only on SudokuHow to use itHosted APIs and open modelsResearch code and task-specific checkpoints Where coarse-to-fine generation could fit first The published experiments cover open-ended text generation, summarisation and puzzle-style reasoning. Based on how the method works, the most likely early uses are tasks with a clear overall structure and a predictable length: Use caseWhy coarse-to-fine might suit itBusiness exampleSummariesThe structure is predictable and the length is boundedDaily digests of tickets, calls or incident reportsStructured documentsSections and fields can be planned before wordingDraft reports, product descriptions, form responsesConstraint puzzles and planningThe whole answer must satisfy rules at onceScheduling, configuration and rota problems (the paper’s Sudoku results point this way)Bulk generationThroughput matters more than live streamingOvernight personalisation for thousands of records These are directions to test, not proven applications. Until there are general-purpose NCM models, each of them would need a task-specific model trained in-house. Where today’s LLMs keep winning - Open-ended conversation and very long outputs, where the length and direction of the answer are not known in advance. - Streaming responses: chat interfaces show words as they are generated, which hides latency. A level-by-level method shows nothing until a level completes. - Tool use and agents, where the model calls tools mid-answer and reacts to the results, a pattern explained in our guide to [AI agents vs AI workflows](/blog/ai-agents-vs-ai-workflows-what-businesses-need-to-know-in-2026/). - Breadth and maturity: frontier LLMs are trained on vast data, support many languages and tasks in one model, and come with hosting, safety tooling and support. - Ecosystem: inference servers, fine-tuning tools, evaluation frameworks and security controls are all built around autoregressive models. A worked example: how NCM would build a summary To make the idea concrete, imagine summarising a two-page incident report. This walk-through is a simplified illustration of the coarse-to-fine principle, not a trace of the real model, which works on learned codes rather than readable outlines. An autoregressive LLM would write the same summary strictly left to right, deciding the first word before it has “planned” the last sentence. That works remarkably well in practice, but it is inherently sequential. NCM’s bet is that settling structure first and detail later can be both faster and, for some tasks, more coherent. What the results actually show The authors report that, on a single NVIDIA H100, NCM generated text 21 times faster than the strongest diffusion baseline on LM1B, 10 times faster on OpenWebText and 31 times faster on XSum summarisation while matching its ROUGE-L score. It also needed about 4 to 5 times fewer training tokens, even counting the tokenizer’s training. On Sudoku puzzles it outperformed both autoregressive and diffusion baselines across difficulty levels. Three caveats matter for anyone reading the headlines: - The speed comparisons are against diffusion language models, not against GPT-, Claude- or Gemini-class autoregressive LLMs. - The models are small: the scaling experiments use 97M, 317M and 725M parameters, while frontier LLMs are hundreds of times larger. Whether the advantage holds at that scale is an open question the authors themselves list. - The results are new and author-reported; independent reproductions and general-purpose benchmarks have not yet appeared. > Not the “end of LLMs” > > Social posts calling NCM the end of LLMs go well beyond the paper. It is a promising new way to generate text, tested at small scale against a different family of models. Treat claims of LLM replacement as speculation until NCM is shown working at frontier scale on general tasks. How to judge AI research headlines like this one NCM is a useful case study in reading AI news. A new paper appears, social media calls it a revolution, and leaders are asked whether to change course. Six questions separate a promising result from a reason to act: QuestionWhat to look forNCM todayFaster than what?The baseline it was compared withMainly diffusion language modelsAt what scale?Model size compared with what you useUp to 725M parametersOn which tasks?Benchmarks that resemble your workloadGeneration, summarisation, SudokuWho measured it?Independent reproduction, not only the authorsAuthor-reported so farAt what quality?Accuracy or quality held steady while speed improvedCompetitive with or better than the diffusion baselinesCan I use it?Hosted APIs, general models, supportResearch code and task-specific checkpoints Apply the same checklist to the next “end of transformers” or “10× cheaper AI” headline. Most results that matter take one to three years to reach products, and the ones that do usually arrive through your existing providers. Three scenarios for the next 12–36 months Nobody knows yet how NCM will develop. Planning around scenarios, rather than predictions, keeps the roadmap robust: ScenarioWhat happensWhat it means for youIt stays a research ideaResults do not hold at large scaleNo change: autoregressive LLMs keep improving and getting cheaperIt becomes an ingredientProviders adopt coarse-to-fine or parallel decoding inside their modelsFaster, cheaper APIs with no work on your side, if you can switch models easilyIt becomes a new model familyGeneral-purpose NCM models match LLMs on many tasksOption to move high-volume, structured workloads to a cheaper, faster generator In all three scenarios the winning move is the same: build AI features so that the model behind them can be replaced without rewriting the product. That is cheap to do now and expensive to retrofit later. What it means for business leaders For now, nothing in your AI roadmap needs to change. Every production AI system you can buy or build today runs on autoregressive models, and the practical decisions remain the same: choosing between [AI agents and AI workflows](/blog/ai-agents-vs-ai-workflows-what-businesses-need-to-know-in-2026/), governing data and measuring value. What NCM signals is that generation cost and speed may fall sharply in the next few years if parallel approaches mature. - Avoid long lock-in to a single model architecture: keep model calls behind your own interface so a faster, cheaper generator can be swapped in. - Measure latency and cost per task now, so you can recognise when a new approach genuinely improves them. - Ask vendors how they would adopt faster decoding methods, not whether they use NCM. - Let your AI team experiment with the open code on narrow, structured tasks if speed matters, but keep it out of production. The architectural habits that protect you here are the same ones we recommend for any LLM system in our [10 architectural rules for effective LLM use](/blog/the-prompt-engine-10-architectural-rules-for-effective-llm-use/). For your AI team, the practical priorities do not change: a clean abstraction over model providers, an evaluation set built from your own tasks, and cost and latency tracked per feature. With those in place, testing a new kind of generator becomes a week of work rather than a rebuild, and you can make the decision on your own data instead of on headlines. Teams without them tend to stay locked to whichever model they started with, even when better and cheaper options appear. Questions to ask your AI vendors - What latency and cost per request do we get today for our main use cases, and how do you report it? - How quickly can we switch to a newer or cheaper model if one becomes available? - Are you evaluating faster decoding methods such as parallel or coarse-to-fine generation, and how would that change pricing? - If quality changes when you change models, how will we know before our customers do? What to watch next Signals that NCM is becoming practical - Results at billions of parameters, not only hundreds of millions. - Head-to-head comparisons with autoregressive LLMs on general tasks, not only diffusion models. - Independent reproductions of the speed and quality claims. - A general-purpose checkpoint instead of one model per task, and support in inference servers or hosted APIs. - Major model providers adopting coarse-to-fine or parallel decoding in their own products. AI strategy Want an AI architecture that can adopt better models as they arrive? We review your AI systems for model lock-in, latency and cost, and design the abstraction and measurement that let you switch to faster, cheaper models without a rebuild. [Book a free 30-minute call](/contact-us/?topic=AI%20architecture#book) Sources - Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation (Hazy Research blog, 12 October 2026) - Paper on arXiv (2610.14162) (Stanford University and Together AI) - NCM research code (GitHub) --- Originally published at phpscientist.com.

Qwen 3.8 27B Derivative on a 12GB RTX 3060: Up to 44 Tokens/s, Vision, Tools, and 128K Context Fully in VRAM [Edit] Here is the model working for a moment (while I record the screen): youtu.be Hi. I'm a software developer, and this is my best setup so far for running fast, useful local AI on consumer hardware. The goal is a useful local assistant: good speed, fewer mistakes, tools, vision, and the ability to write small scripts and games. It runs with a 128K-token context window, with the main model and context cache on the GPU, and image processing and context compression on the CPU. I've been using it to manage my calendar, search the web, and solve a more complex task: classifying, organizing, deduplicating, and comparing my old music files against my Spotify favorites playlist. My playlist is finally being updated automatically with all the songs I love <3. So far, it's working well. It even created a basic chair in Blender and wrote a Snake game in HTML/JS. (Note: these examples use tools and integrations configured separately, they are not included here.) My hardware: RTX 3060 with 12 GB of VRAM, Ryzen 7 3700X (8 cores, 16 threads), and 32 GB of RAM. I use vision and tools a lot and find this setup very competitive for daily use. Since I haven't found a better setup for my needs in the last two weeks, I decided to share what I've put together here. I run CachyOS, by the way. It may help inference speed, but I haven't isolated its effect. This guide is for Linux! I don't know how to reproduce the setup on Windows. What I use - Model: Swift Bonsai 2 27B Katana + MTP, based on Bonsai 2 / Ternary Qwen3.8-27B and tuned for shorter reasoning. - Model file: Swift-Bonsai-2-27B-Katana-PTQ1_0-MTP-Q8_0.gguf, about 6.4 GB. Ternary PTQ1_0 backbone, Q8_0 MTP head. - Vision projector: Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf, about 629 MB, loaded separately. - Inference: the Prism fork of llama.cpp, build prism-b10772-e8fd209, CUDA 13.3. - Agent: Hermes in Docker. - History compression: Granite 3.3 2B on the CPU through Ollama, registered locally as granite3.3-compressor:2b. My settings I keep the vision projector and Granite on the CPU to save VRAM. The 3700X handles both well enough for my use. Bonsai's config.env sets the options below. start.sh applies them and enables Flash Attention. |Setting|Value|Purpose| |:-|:-|:-| |Context / slots|CTX_SIZE=128000, PARALLEL=1|One conversation with a large context window| |GPU layers|N_GPU_LAYERS=99|Offload all model layers to the GPU| |K/V cache, including MTP|KV_CACHE_TYPE=q4_0|Reduce VRAM used by the context cache| |MTP draft|SPEC_N_MAX=1|Draft up to one token ahead per decoding step| |Reasoning|REASONING_EFFORT=medium, REASONING_BUDGET=2048|Limit thinking time| |Vision on the CPU|MMPROJ_CPU=1|Save VRAM, with slower image processing| |Image tokens|IMAGE_MAX_TOKENS=1536|My compromise: 1024 lost too much detail, 2048 took too long| |CUDA consistency|KATANA_BATCH_INVARIANT=1|Keep vision output correct with this runtime/GPU| Performance During one test response, generation stayed around 43–44 tokens/s and peaked at 45.74 tokens/s over a three-second window. In daily use, I usually see 28–40 tokens/s after a few prompts and 17–25 tokens/s much later on, with A LOT more context. MTP proposes tokens that the main model verifies to speed up generation. I tried draft limits of 1 and 2 and chose 1, also guided by sudoingx's Bonsai 2 PTQ1_0 + MTP tests on an RTX 3060 12 GB. With a 131K window and Q4 K/V, that setup reached 50.1 tokens/s with MTP versus 39.8 without, and 26.86 versus 22.23 with 41.8K tokens filled. Draft limit 1 beat 2 in both cases. These are the author's measurements, on a different model variant and runtime. 2 can also be a good choice - try SPEC_N_MAX=2 ./scripts/start.sh inside Bonsai-fast/ and compare it on your workload. Anyway, my results with slightly different models and setup were inferior, as I stated, so I'm not sure if I did something wrong here. I chose IMAGE_MAX_TOKENS=1536 for a balance of detail and speed: I got 2 seconds for photos of people near cars, 27 seconds for a complex image, versus up to 40 seconds at 2048 on a complex image. It still makes mistakes, regardless of the value. Prism recommends 1024 for general use and 2048+ or crops for fine text, so 1024 is also an option if speed matters more than detail. I chose q4_0 for my context based on published memory savings and how the complete setup performed in use. A cache-only benchmark on related Qwen 3.6 models found Q4 usable, with the largest output-distribution changes on long documents. A separate Bonsai 2 benchmark used Q4 successfully with 191K tokens filled on a 16 GB RTX 4080 SUPER. These are just supporting references, not measurements of my Katana setup. With 128,000 tokens per slot, Q4 K/V caches, MTP on the GPU, and the vision projector on the CPU, the whole Bonsai process used 9,144 MiB (8.9 GiB) of VRAM. The runtime allocates the context cache at startup, so the configured capacity fits my graphics card. The main attention K/V cache alone (context) is approximately 2,250 MiB (2.2 GiB) at 128K: 128,000 tokens x 16 full-attention layers x 2 K and V x 4 KV heads x 256 values per head x 18/32 bytes per Q4_0 value. With my desktop and other apps open, about 823 MiB (0.8 GiB) remained free. I roughly understand this calculation, but it’s easy to see how much Q4_0 reduces the result: it uses an average of 0.5625 bytes per value, compared with F16’s 2 bytes. For my 128K context, that reduces the main K/V cache from 7.81 GiB to 2.20 GiB, about 72% less memory - and the model still seems smart, so… I also considered q8_0, but q4_0 is still roughly half the size, it’s just off the table. On the model side, I chose ternary PTQ1_0 for its smaller memory footprint, leaving more VRAM for context and MTP. PTQ1_0 packs the same ternary weights more tightly than PQ2_0, at about 1.75 versus 2.13 bits per weight, without another quantization step. It doesn't mean zero quality loss compared with the original FP16 LLM, but is a fair price to pay for usability. The Prism fork of llama.cpp adds PTQ1_0/MTP kernels and attention that reads the Q4/Q8 cache directly, plus optimizations for the RTX 3060's sm_86 architecture. PTQ1_0 uses less memory, but whether it runs faster than PQ2_0 depends on your GPU and the runtime’s kernels. For reasoning, I use medium, following Prism's recommendation for shorter reasoning on the original Bonsai 2. In practice, every reasoning level feels like xhigh to me on Katana, so I also set a 2,048-token reasoning budget to keep thinking loops under control, with the server injecting this message when the budget is reached: “... Time to stop thinking. Give an answer or make the tool call now. Do not waste time.” Bonsai needs clear instructions for complex tasks. Otherwise, it can get stuck in a loop of writing and fixing helper scripts. The guidelines in my SOUL.md have helped it stay on track and finish tasks. To summarize the conversation history, I use Granite. I chose it for its small 2B size and support for long-document and meeting summarization. granite3.3-compressor:2b is my local name for this configuration. The included Modelfile keeps Granite's template and license, sets num_ctx 128000, and adds num_gpu 0 in the CPU variant. Chat compression takes 1.5–5 minutes in my tests, depending on the conversation content and size. Finally, I use Hermes Desktop with the included Docker container as its backend. I noticed the UI was using 15–35% of the GPU while the window was focused. Enabling --force-prefers-reduced-motion reduced animations and brought GPU usage down to 1–5%, leaving more resources available for model inference. Warning! These figures are mostly observations, not controlled benchmarks!! Although published tests guided my choices, I evaluated the complete setup in use rather than measuring each setting's individual gain. Installation Download: local-ai-bonsai.zip. Extract local-ai-bonsai.zip and open the extracted local-ai-bonsai/ folder. It contains all the setup files: |Folder|Contents| |:-|:-| |Bonsai-fast/|Model download and launch scripts| |hermes/|Docker Compose and Desktop UI settings| |granite-compressor/|128K compressor configuration, including CPU mode| |hermes-host-access/|Temporary SSH access from the container to the host, enabled when you choose| I haven't included every dependency. You need to install Docker, Compose, Ollama, and other prerequisites yourself. Bonsai requires Linux x86_64 and NVIDIA drivers and libraries compatible with CUDA 13.3. The ZIP contains scripts and configuration files. Setup downloads the model weights and inference binaries. See each folder's README for details. Start Bonsai Inside Bonsai-fast/: ./scripts/setup.sh ./scripts/start.sh setup.sh downloads and extracts the Prism runtime into bin/ and downloads the Katana model and vision projector into models/. Run it once to prepare the files. start.sh starts the inference server - keep it running while using Hermes. No compilation or Prism Model Manager is needed. Start Hermes Inside hermes/, enter your dashboard credentials and time zone in compose.yaml. These credentials let you sign in through the browser or Hermes Desktop, and a loopback-only setup may require the dashboard session token (HERMES_DASHBOARD_SESSION_TOKEN) to use the container as the Hermes Desktop backend. Then run: docker compose up -d Open http://127.0.0.1:9119 or connect Hermes Desktop to the backend. The container uses network_mode: host to reach Bonsai and Ollama. Its data/ and workspace/ folders persist. To use the container as the Hermes Desktop backend, open it and go to Settings -> Gateways -> Remote gateway, enter http://127.0.0.1:9119, sign in with the required credentials, then save and reconnect. In the web dashboard, open Config -> YAML. Merge this into providers:, keeping the other entries. Edit bonsai-mtp if it already exists: providers: bonsai-mtp: name: Bonsai MTP base_url: 127.0.0.1 model: swift-bonsai2-27b-katana-mtp discover_models: true context_length: 128000 models: swift-bonsai2-27b-katana-mtp: context_length: 128000 Click Save, then Models -> Main model -> Change. Pick Bonsai MTP, select swift-bonsai2-27b-katana-mtp, and click Switch. Use Refresh Models if needed. This sets the model for new sessions. Keep Hermes's context value aligned with Bonsai's CTX_SIZE=128000. The launch script also loads the vision projector. Optional extras Use hermes-host-access/README.md to install the temporary SSH bridge with new keys. This lets the sandboxed AI help with tasks on the host when needed. The bridge expires automatically after three hours in case you forget to close it. SOUL.md contains my general agent instructions. Customize it before copying it to hermes/data/SOUL.md. To reduce Hermes Desktop's animations, merge hermes/desktop-config.example.yaml into the Desktop client's ~/.hermes/config.yaml, then fully quit and reopen Hermes Desktop. CPU context compression With Ollama running, inside granite-compressor/: ollama pull granite3.3:2b ollama create granite3.3-compressor:2b -f ./Modelfile.granite-compressor.cpu Then run these commands inside hermes/: docker compose exec hermes hermes config set auxiliary.compression.provider custom docker compose exec hermes hermes config set auxiliary.compression.model granite3.3-compressor:2b docker compose exec hermes hermes config set auxiliary.compression.base_url 127.0.0.1 docker compose exec hermes hermes config set auxiliary.compression.timeout 300 docker compose restart hermes After a test request, ollama ps should show 100% CPU while Granite is loaded. Bonsai and Granite each have their own 128K context window. See the package README for the API test and the variant that lets Ollama choose CPU/GPU placement. Warning! The compressor has a 128K context limit, including when it summarizes conversations from Claude or GPT. Our Hermes setup is intended for local models, rather than large flagship models, so larger compression inputs still need to fit within that limit and will be handled locally. Next steps I'd like to add local decision tools in the style of TypeSafe's Jev: an open text model such as Laya, plus Qevi-2B for images. Both return structured decisions and probabilities instead of generating text. Bonsai could call either tool for quick decisions and item classification, switching between them depending on the task. If needed, I may lower Bonsai's context to 100K to make room for smaller models. That would free roughly 492 MiB from the main attention cache alone. The total VRAM savings would need to be measured... I'd like to find a compact image generation model that delivers good results with my remaining hardware resources, although I think running it alongside Bonsai is probably impossible with current technology and my hardware limits. I'm also interested in NCM (Next-Concept Modeling), which generates text in parallel, starting with a rough outline and progressively adding detail. Its summarization research already looks promising for context compression. It might also help make larger local models faster and more efficient. This video introduced me to it. Do you have any ideas for improving inference further? Maybe a setting or approach I've missed? I hope this post helps someone, and maybe I’ll learn something new from the replies. Thanks!

NCM is the new interesting language model architecture. Maybe it can shift the industry from the memory-bound LLMs to compute-bound diffusion models, who knows HazyResearch/ncm

gonna start poking at the different levels of hierarchy here and see what kinda fuckin gibberish I can get it to spit out HazyResearch/ncm

Next Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

@kavvata Try it out here. HazyResearch/ncm

Someone shared these videos explaining NCM with me, and I thought they were pretty cool! Some of the titles might be a little exaggerated though 😅, but it's nice to see people taking an interest in and trying out our work! Code: HazyResearch/ncm youtu.be youtu.be youtu.be youtu.be