How to install and tune oMLX on M5 Max 128 GB
---
title: How to install and tune oMLX on M5 Max 128 GB
published: true
description: Recipe for oMLX 0.6.4 on a Mac M5 Max with 128 GB: install, Claude Code and advanced settings with screenshots (Lightning MTP, VLM MTP and DFlash).
tags: macos, llm, ai, apple
canonical_url: jacar.es
cover_image: media.jacar.es
---
oMLX is an LLM inference server built on MLX, the framework Apple shipped in December 2023 for Apple Silicon. It adds continuous batching, two-tier KV cache (RAM + SSD), and an OpenAI- and Anthropic-compatible API.
On a Mac M5 Max with 128 GB of unified memory you can hold three or four large models at once with TurboQuant 3.5-bit on KV cache. That is enough to feed chat, agent and IDE in parallel. This guide collects the configuration tested in May 2026 to get the most out of that combination.
Quick answers
Which oMLX version do I use and how do I install it? Version 0.6.4 (released 29 August 2026, Apache 2.0). The direct route: download the .dmg from GitHub Releases, open it and drag to Applications. Or install via brew tap jundot/omlx https://github.com/jundot/omlx and brew install jundot/omlx/omlx.
The first load of the panel at http://localhost:8000/admin prompts for the API key. The full Homebrew route, including the service that starts on its own, is in installing, updating and uninstalling oMLX with Homebrew.
What TurboQuant setting makes sense on 128 GB? 3.5-bit. vLLM’s independent analysis published 11 May 2026 shows 3.5-bit matches full-precision quality with about 4x less KV-cache memory. On M5 Max that turns 128k context from "blows past available RAM" into "fits alongside other loaded models."
How much faster is it with the advanced 0.6.4 settings? Between 1.19x and 1.88x when generating code, and between 1.07x and 1.47x with 15,000 tokens of context, measured through the API on 15 September 2026. The biggest gain came from Lightning MTP on Qwen3.8-27B oQ4e, from 21.1 to 39.7 tok/s, and it needs a build of the model that keeps the MTP head. The details are in the advanced settings.
Which models can I load concurrently on 128 GB?
Primary: unsloth/Qwen3.6-35B-A3B-MLX-8bit (37.7 GB, MoE with 3B active). Fast helper: Qwen3-14B-Instruct-mlx-4bit (8 GB). Vision: Qwen2.5-VL-32B-mlx-4bit (18 GB).
Embeddings: BGE-M3-mlx (1.2 GB). Reranker: ModernBERT-base-mlx (150 MB). Comfortable sum: ~65 GB with headroom.
How do I point Claude Code at the local endpoint? Since 0.5.0 the short way is omlx launch claude, which exports the variables for you. By hand: export ANTHROPIC_BASE_URL=http://127.0.0.1:8000, ANTHROPIC_AUTH_TOKEN=<your_api_key> and the three ANTHROPIC_DEFAULT_*_MODEL variables (Opus/Sonnet/Haiku).
Launch with claude --bare to drop the system prompt to ~1,795 tokens. The dashboard’s Claude Code with oMLX section assembles the full command for you.
Does this replace Claude Opus 4.7? Not one for one. Claude Code is tuned for Claude’s tool-use format; a non-Claude model behind the endpoint loses reliability in agentic loops. Use this for offline work, sensitive data that should not leave the Mac, or as a fallback when api.anthropic.com is rate-limiting you.
Install
The easiest path: download the .dmg from GitHub Releases, open it and drag the app to Applications. The server starts in the background with a menu-bar icon.
Or install via Homebrew:
brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx
omlx start
omlx serve --model-dir ~/models
The first hit to http://localhost:8000/admin prompts for the API key. By default oMLX ships with no API key set (empty field), which is fine for local testing. If the instance only listens on 127.0.0.1 this is safe; the moment you expose it to the LAN or plan to share the endpoint, go to Settings → Auth & Info and set a long random string. The section accepts multiple keys at once.
!Login screen of the oMLX admin panel at localhost:8000/admin, with the API Key field empty and a "Stay signed in for 30 days" checkbox
Server settings for 128 GB
Settings → Global Settings holds the full server configuration. The decisions that matter on a 128 GB machine:
- Server → Host: Localhost only (127.0.0.1). If you open it to the LAN, put real auth in front first.
- Server → Port: 8000 by default.
- Resource Management → Memory Limit (Total): Auto. oMLX subtracts what macOS reserves; on 128 GB you end up around 110-114 GB for inference.
- Resource Management → Memory Limit (Models Only): Auto. Keeps a percentage for activations, KV cache and auxiliary processes.
- Resource Management → Hot Cache Limit: 10%. Intermediate RAM-tier KV cache. With TurboQuant enabled on the models that support it (see below), 10% is the value people running this setup in practice land on for 128 GB. With TurboQuant off and a single model loaded you can drop to Off, but you stop gaining.
- Resource Management → Cold Cache Limit (SSD Cache): 10%. Around 80 GB of SSD for cold tokens without saturating it.
- Resource Management → Max Concurrent Requests: 16. Comfortable for a single user with an agent, a chat session and an IDE pinging at once. Raise to 32 if you share with a small team.
- Resource Management → Idle Timeout: None. Keeps models warm; the first token arrives in well under a second instead of seconds later.
- Generation Defaults → Max Context Window: 256000 as the global default. The Qwen3.6 family holds up at long context, and with TurboQuant the effective RAM stretches enough to actually use it. Each model can be capped lower in Model Settings.
- Generation Defaults → Max Tokens: 64000. Upper bound per response. Going higher only matters if you plan to generate full books in one shot.
- Generation Defaults → Temperature: 1.0 for general use, 0.2-0.5 for code models.
- Model Settings → Experimental Features → TurboQuant KV Cache: enable at 3.5-bit on the large dense models. vLLM’s independent study published 11 May 2026 of Google’s TurboQuant shows 3.5-bit matches full-precision quality with roughly 4x less KV-cache memory. On M5 Max that makes 128k context fit on models where FP16 does not. On 0.6.4 it cannot be combined with VLM MTP, and in these tests it did not speed up generation; if speed is what you are after, see the advanced settings.
!oMLX Global Settings page: Host set to Localhost only (127.0.0.1), Port 8000, Memory Limit Total Auto (120 GB), Memory Limit Models Only Auto (108 GB), Hot Cache at 9% (12 GB), Cold Cache SSD at 10% (372 GB), Max Concurrent Requests 16, Idle Timeout None, Max Context Window 256000, Max Tokens 64000, Temperature 1.0, Top P 0.95
Model stack for multi-LLM
With 128 GB you have room to load one large primary model, a fast helper, a VLM, embeddings and a reranker concurrently. Download them from Models → Downloader by pasting the Hugging Face repo URL.
The recommendation holding up in May 2026 comes from people actually running Claude Code on oMLX on M5 Max (see Diego R. Baquero’s gist for the source): Unsloth’s Qwen 3.6 35B-A3B family in MoE with ~3B active parameters per token. On 128 GB the 8-bit fits without breaking a sweat:
- Primary (chat + code + reasoning): unsloth/Qwen3.6-35B-A3B-MLX-8bit (~37.7 GB). MoE 35B with 3B active. The all-rounder: long chat, code, agents. On 128 GB it fits alongside everything else without squeezing.
- More compressed alternative: unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit (~21.6 GB) if you want two primary models loaded at once. Loses a notch of quality versus 8-bit but leaves room to experiment.
- Dense reasoning (when MoE falls short): Mistral-Large-2-123B-Instruct-mlx-4bit (~70 GB) or Llama-3.3-70B-Instruct-mlx-4bit (~40 GB). For deep reasoning in a single pass, where a dense architecture beats a small-activation MoE.
- Fast helper: Qwen3-14B-Instruct-mlx-4bit (~8 GB). For cheap tasks in agents, parsing and summaries.
- Vision: Qwen2.5-VL-32B-Instruct-mlx-4bit (~18 GB) covers OCR, image description and multimodal reasoning.
- Embeddings: BGE-M3-mlx (~1.2 GB), dense + sparse + multi-vector in a single model.
- Reranker: ModernBERT-base-mlx (~150 MB) to close the loop on a decent RAG pipeline.
Comfortable concurrent load with Qwen3.6 35B-A3B 8-bit as primary + 14B helper + VL-32B + embeddings + reranker: around 65-70 GB. Add Mistral Large 2 123B on top for dense reasoning and you hit ~135 GB nominal. The LRU policy moves idle models to SSD, so the cohabitation works in practice for sessions that do not pin everything at once. Pin the primary model from Model Manager so it never gets evicted.
Point Claude Code at the local endpoint
oMLX exposes an Anthropic-compatible API at http://127.0.0.1:8000. In current versions the routes live under /v1: POST /v1/messages and POST /v1/messages/count_tokens. Claude Code respects ANTHROPIC_BASE_URL, so pointing the CLI at your Mac is a matter of exporting environment variables. First, turn off the attribution header in Claude Code’s global config so the gateway does not bolt extra noise onto the prompt:
Since 0.5.0 there is a shortcut that does all of this for you and did not exist when this recipe was written:
omlx launch claude
It sets ANTHROPIC_BASE_URL against your server, puts the configured key in ANTHROPIC_AUTH_TOKEN (or omlx if there is none), empties ANTHROPIC_API_KEY to force the base URL to be used, raises API_TIMEOUT_MS to 3,000,000 and disables non-essential traffic. It also sets CLAUDE_CODE_MAX_CONTEXT_TOKENS and CLAUDE_CODE_AUTO_COMPACT_WINDOW to what the engine can actually do, which is what the context-scaling option used to do by hand. It enforces a minimum window of 48,000 tokens and will not start if the chosen model falls short. There are equivalent commands for Codex, OpenCode, OpenClaw, Hermes and Pi, described in the dashboard and command line guide.
The manual recipe below still works and is the one to use if you want fine control over each variable.
~/.claude/settings.json:
{
"env": {
"CLAUDE_CODE_ATTRIBUTION_HEADER": "0"
}
}
Then the startup command:
export ANTHROPIC_BASE_URL=http://127.0.0.1:8000
export ANTHROPIC_AUTH_TOKEN=<your_api_key>
export ANTHROPIC_DEFAULT_OPUS_MODEL=unsloth/Qwen3.6-35B-A3B-MLX-8bit
export ANTHROPIC_DEFAULT_SONNET_MODEL=unsloth/Qwen3.6-35B-A3B-MLX-8bit
export ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen3-14B-Instruct-mlx-4bit
export ANTHROPIC_DEFAULT_MODEL=unsloth/Qwen3.6-35B-A3B-MLX-8bit
export API_TIMEOUT_MS=600000
export CLAUDE_CODE_USE_BEDROCK=0
export DISABLE_NONESSENTIAL_TRAFFIC=1
claude --bare
The --bare flag skips hooks, LSP, plugin sync and auto-memory, dropping Claude Code’s system prompt to around 1,795 tokens (versus the thousands it uses with everything loaded). For offline work against a local model that is the sensible default: every token in the system prompt is bandwidth your Mac would otherwise burn. Drop the flag when you point back at api.anthropic.com.
The oMLX dashboard builds this command for you in the Claude Code with oMLX section: pick Opus, Sonnet and Haiku from three dropdowns and copy the ready-to-paste command. Set Context scaling for Claude Code to 64000: Claude Code requests 200k tokens by default, but a local model behaves better with a 64k ask than a 200k ask it cannot honor. The option scales the reported counts so auto-compact triggers at the target size.
!oMLX dashboard with the Claude Code with oMLX section fully configured: Local mode active, Opus mapped to Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-mlx-8bit, Sonnet to Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-8bit, Haiku to Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT-HERETIC-UNCENSORED-MLX-mxfp8, Context scaling for Claude Code enabled with target 200000, plus pre-built commands for Codex, OpenCode and other integrations below
A real caveat: Claude Code is tuned to Claude’s tool-use format and response patterns. A non-Claude model behind ANTHROPIC_BASE_URL works for autocomplete and reasoning, but you will see drops in tool-call reliability and in agentic loops. Use this for offline work, sensitive data that should not leave the Mac, or as a fallback when api.anthropic.com is rate-limiting you. It is not a 1:1 substitute for Claude Opus 4.7.
SSH from another Mac
If oMLX runs on a Mac Studio and you want to use Claude Code from your MacBook, SSH port forwarding is the bridge:
ssh -L 8000:localhost:8000 user@mac-studio
Once connected, 127.0.0.1:8000 on your laptop points to the Studio’s oMLX. Save this script as claude-local.sh:
#!/bin/bash
export ANTHROPIC_BASE_URL='http://localhost:8000'
export ANTHROPIC_AUTH_TOKEN=''
export ANTHROPIC_DEFAULT_OPUS_MODEL='Qwen3.6-35B-A3B-MLX-8bit'
export ANTHROPIC_DEFAULT_SONNET_MODEL='Qwen3.6-35B-A3B-MLX-8bit'
export ANTHROPIC_DEFAULT_HAIKU_MODEL='Qwen3-14B-MLX-4bit'
export ANTHROPIC_DEFAULT_MODEL='Qwen3.6-35B-A3B-MLX-8bit'
export API_TIMEOUT_MS=600000
export CLAUDE_CODE_USE_BEDROCK=0
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
claude --bare --dangerously-skip-permissions
Make it executable with chmod +x claude-local.sh and run it from the SSH session. The default API key is the empty field. If you set one in Settings → Auth & Info, replace '' with it.
--bare drops Claude Code’s system prompt to ~1,795 tokens. --dangerously-skip-permissions bypasses interactive permission prompts that don’t make sense in an automated pipeline.
Benchmark on real hardware
Bench → Performance runs tests on your actual hardware. The panel covers prefill at eight prompt sizes (pp1024, pp4096, pp8192, pp16384, pp32768, pp65536, pp131072, pp200000) and continuous batching at 2x, 4x and 8x concurrency. Results go to the public oMLX leaderboard, and My Submissions shows yours next to other machines on the same profile. On 0.6.4 the upload happens automatically when each run finishes; how to avoid it is in the 0.6.4 benchmarks.
Reference numbers measured on M5 Max 40-core with 128 GB on oMLX 0.3.8, in May 2026. Not extrapolated (the omlx.ai leaderboard already has M5 Max submissions, and vLLM’s independent TurboQuant study published 11 May 2026 confirms the figures for the large dense models):
- Qwen 3.6 35B-A3B 8-bit (MoE, 3B active): 65-80 tok/s on decode at short context. The model that most changes day-to-day use on this machine. Live benchmarks from this setup:
| Test | TTFT | Decode TPS | E2E | Peak Mem |
| --- | --- | --- | --- | --- |
| pp1024/tg128 | 577 ms | 72.4 tok/s | 2.35s | 38.5 GB |
| pp4096/tg128 | 1,293 ms | 83.1 tok/s | 2.83s | 36.2 GB |
| pp8192/tg128 | 2,995 ms | 82.1 tok/s | 4.55s | 36.5 GB |
| pp32768/tg128 | 13,785 ms | 15.2 tok/s | 22.18s | 40.9 GB |
Continuous batching (pp1024 / tg128):
| Batch | Decode TPS | Speedup |
| --- | --- | --- |
| 1x (baseline) | 72.4 tok/s | 1.00x |
| 2x | 87.1 tok/s | 1.20x |
| 4x | 123.8 tok/s | 1.71x |
| 8x | 237.7 tok/s | 3.28x |
- Llama 3.3 70B 4-bit: 14-18 tok/s on decode.
- Mistral Large 2 123B 4-bit with TurboQuant 3.5-bit on KV cache: 8-11 tok/s on decode. The number that matters at 128k context is peak memory, ~74 GB (FP16 KV blows past 128 GB). dasroot.net’s long-context M5 Max experiment documents the same peaks for the 104B dense model.
- gpt-oss-20b MXFP4-Q4 on M5 Max 40c per the public oMLX benchmark submission lands around 100 tok/s on decode.
The tables above are our own measurement on 0.3.8, and on 15 September 2026 I re-ran the tests on 0.6.4 on this same machine. With no acceleration at all, this model goes from 15.2 to 77.9 tok/s with a 32k prompt. Speculative decoding adds between 1.07x and 1.88x, depending on the model and the context. The new tables are in the 0.6.4 benchmarks.
vLLM measured the TurboQuant KV-cache compression at 4.41x in May 2026. That figure is what separates "I have 128k context but it does not fit" from "I have it and can load two other models alongside it". At 4-bit and 3.5-bit, quality stays close to full precision; at 3-bit you start to feel it in code and long-form reasoning.
!oMLX Bench → Performance Benchmark page with Qwen3.6-35B-A3B-MLX-8bit selected, Single Request Test checkboxes (pp1024, pp4096, pp8192, pp32768) and Continuous Batching Test checkboxes (2x, 4x, 8x), Run Benchmark button, and My Submissions link
Advanced settings for maximum performance on 0.6.4
On 0.6.4, the biggest speed lever on this M5 Max is speculative decoding, and the right method depends on the model. Measured through the API at temperature 0.6, Qwen3.8-27B oQ4e went from 21.1 to 39.7 tok/s with Lightning MTP. Qwen3.6-35B-A3B 8-bit went from 79.7 to 117.5 tok/s with VLM MTP. Everything in this section was measured on 15 September 2026 on the machine this guide is about, and the method and the full tables are in the 0.6.4 benchmarks section.
Which acceleration method to use with each model
oMLX 0.6.4 has three speculative decoding methods and allows only one per model:
- Lightning MTP: uses the multi-token prediction (MTP) head that ships inside the weights. It only works if the conversion kept the mtp.* tensors.
- VLM MTP: uses an external MTP drafter under 2 GB, trained for that base model. It only kicks in when no other request is running.
- DFlash: uses a z-lab block-diffusion drafter that proposes up to 16 tokens per pass. It serves requests one at a time and keeps its own prefix cache.
In all three the large model verifies every proposed token, so the output comes from the large model, not from the drafter. This is what each combination measured on this M5 Max:
| Model | Method | What to download | Size | Decode writing code | Decode with 15,000 tokens of context |
| --- | --- | --- | --- | --- | --- |
| Qwen3.6-35B-A3B 8-bit | VLM MTP | mlx-community/Qwen3.6-35B-A3B-MTP-bf16 | 1.7 GB | 79.7 → 117.5 tok/s | 84.0 → 90.7 tok/s |
| Qwen3.6-35B-A3B oQ4e | Lightning MTP | Jundot/Qwen3.6-35B-A3B-oQ4e-mtp (full model) | 21.6 GB | 100.2 → 119.2 tok/s | 100.2 → 115.5 tok/s |
| Qwen3.8-27B oQ4e | Lightning MTP | Jundot/Qwen3.8-27B-oQ4e-mtp (full model) | 17.0 GB | 21.1 → 39.7 tok/s | 21.4 → 27.2 tok/s |
| Qwen3.8-27B 4-bit | VLM MTP | mlx-community/Qwen3.8-27B-MTP-bf16 | 0.87 GB | 24.1 → 34.5 tok/s | 21.7 → 23.2 tok/s |
| Gemma 4 12B 8-bit | VLM MTP | mlx-community/gemma-4-12B-it-assistant-bf16 | 0.88 GB | 27.4 → 50.5 tok/s | 26.6 → 39.2 tok/s |
| gpt-oss-20b | None | – | – | – | – |
For Qwen3.6-35B-A3B, the oQ4e build with Lightning MTP was the fastest in both tests and takes about 15 GB less than the 8-bit one. With long context and prose answers every gain shrinks, because the large model accepts fewer draft tokens, but no configuration was slower than its baseline.
If a Qwen model’s settings show the warning Config declares MTP layers but the weight files contain neither mtp.* tensors nor native nextn layers, the conversion stripped the MTP head. The default mlx-lm converters do that: it applied to the Qwen3.6-35B-A3B builds from mlx-community and the Qwen3.8-27B builds from lmstudio-community installed on this Mac. You have two ways out: download a build that keeps the head, such as the oQ4e-mtp builds published by the oMLX author, or turn on VLM MTP with an external drafter.
Downloading the drafters
Drafters download like any other model. In Models → Downloader, paste the Hugging Face repository id and start the download. oMLX recognises drafters by their type and marks them as helpers; with Hide Helper Models on, they also stay out of the API’s model list.
!oMLX 0.6.4 Download from HuggingFace form with the repository id mlx-community/Qwen3.6-35B-A3B-MTP-bf16 filled in, the HF Token field empty and the Download button
Turning on Lightning MTP
With a model that keeps its MTP head, open Settings → Model Settings, click the model’s gear icon and turn on Lightning MTP in the Acceleration block. When you save, oMLX reloads the model.
On Qwen the default is 3 draft tokens per cycle, and an adaptive controller adjusts it from each response’s acceptance rate. The oMLX log writes one line per response with that rate: on Qwen3.6-35B-A3B oQ4e it sat around 72-77%, with 2.5 to 2.9 tokens emitted per cycle. With greedy sampling the output matches the model without MTP, apart from tiny numerical differences. With temperature, rejection sampling keeps the large model’s distribution.
!Acceleration block of the Jundot/Qwen3.8-27B-oQ4e-mtp model settings in oMLX 0.6.4, with the Lightning MTP switch turned on
Turning on VLM MTP with an external drafter
In the same Acceleration block, turn on VLM MTP, pick the drafter from the drop-down and save. Three conditions in the 0.6.4 code are worth knowing first:
- The model has to load on the vision engine. The multimodal Qwen3.6 and Qwen3.8 builds do so by default, but on a text-only model the setting is silently ignored.
- It cannot be combined with TurboQuant or with repetition or presence penalties. The built-in Qwen presets set presence_penalty to 1.5, which turns it off.
- It only speeds up a request when nothing else is running. With concurrent requests, oMLX falls back to normal batching.
!VLM MTP card in the mlx-community/Qwen3.6-35B-A3B-8bit settings in oMLX 0.6.4, turned on with the Qwen3.6-35B-A3B-MTP-bf16 drafter and the block size left empty, which means 4 tokens
You can apply the same change through the admin API, which is handier when you configure more than one model or script it. The first command logs in with your key and stores the cookie; the second updates the model’s settings:
curl -c omlx.cookies -X POST http://127.0.0.1:8000/admin/api/login \
-H "Content-Type: application/json" \
-d '{"api_key": "your_api_key_here"}'
curl -b omlx.cookies -X PUT \
http://127.0.0.1:8000/admin/api/models/Qwen3.6-35B-A3B-8bit/settings \
-H "Content-Type: application/json" \
-d '{"vlm_mtp_enabled": true,
"vlm_mtp_draft_model": "Qwen3.6-35B-A3B-MTP-bf16"}'
The model name in the path and the drafter name are the ids shown in the model list. For Lightning MTP the body is {"mtp_enabled": true}. The endpoint rejects any field it does not know, so a typo does not slip through.
DFlash only pays off with short prompts
DFlash was the fastest method with 1,000-token prompts and the worst with long context. In the built-in benchmark, Qwen3.6-35B-A3B 8-bit rose from 92 to 112 tok/s with a 1k prompt, but dropped from 78 to 17 tok/s with 32k. Qwen3.8-27B 4-bit dropped from 23.5 to 6.3 tok/s with 32k. If you use Claude Code, which sends tens of thousands of tokens of context every turn, leave it off.
For short-prompt work such as chat or autocomplete, download the matching z-lab drafter: z-lab/Qwen3.6-35B-A3B-DFlash (0.77 GB), z-lab/Qwen3.8-27B-DFlash2 (3.85 GB) or z-lab/gemma4-12B-it-DFlash (1.46 GB). Turn it on in Acceleration → DFlash and quantise the drafter to 4 bits. On the 27B, the 4-bit drafter gave 48.2 tok/s with a 1k prompt against 36.9 tok/s in bf16, and used 2.5 GiB less.
It has two more limits. It serves one request at a time. And although dflash_max_ctx sends long requests to the normal engine, after the first one DFlash stays off until you reload the model.
!oMLX 0.6.4 DFlash card with the Qwen3.6-35B-A3B-DFlash drafter selected, drafter quantisation on with 4-bit weights, 16-bit activations and group size 64, and the max context threshold left unlimited
Prefill on the Neural Engine: no gain with Lightning MTP
Qwen ANE Prompt Processing splits part of the Qwen3.5, 3.6 and 3.8 prefill between Apple’s Neural Engine (ANE) and the GPU. The Tune for this Mac button measures the best split on your machine and saves nothing until you apply the result. On Qwen3.8-27B oQ4e it recommended sending 35% of the MLP channels and 37.5% of the GDN layers to the ANE. With that split, prefill on an 8,193-token prompt went from 452 to 510 tok/s, 12.8% more.
That 12.8% did not hold with the full model. With the split applied and Lightning MTP on, the built-in benchmark measured 505 tok/s of prefill with a 4k prompt against 499 without the ANE, and 422 against 471 with 32k. Through the API, the first token with 15,100 tokens of context took 34.8 s, against 35.2 s without the ANE. In exchange the loaded model grew from 16.0 to 20.7 GB and loading went from 2 to 20 s, so on this Mac I left it off.
Before tuning it on an M5 Max, turn off Use both ANEs in the model settings. That mode is meant for the M3 Ultra, which joins two chips and has two ANEs. It only speeds up prompts of 2,048 tokens or more; generation stays on the GPU.
The project itself documents three costs. It is not bit-exact, because the ANE runs weights requantised to INT8. It relies on private Apple interfaces that a macOS update can break, and it lengthens model load. If you turn it on anyway, also turn on Reuse Compiled ANE Programs under Global Settings → Advanced so it does not recompile on every start.
!oMLX 0.6.4 Qwen ANE Prompt Processing card with the split the tuner proposed: 2048-token block, padding from 1816, 0.35 of the MLPs and 0.375 of the GDN layers on the ANE, Use both ANEs off and the Tune for this Mac button
The global settings that matter
oMLX’s default global configuration already suits a single user, and only four settings change anything measurable:
- Kernel memory limit: macOS caps the memory the GPU can wire by default, and the dashboard shows a red warning when that cap is below what oMLX could use. Run sudo sysctl iogpu.wired_limit_mb=124518, which on 128 GB is RAM minus 5%, and restart oMLX, which applies it at start-up. The value is lost when the Mac restarts.
- Prefill Priority: set it to Speed. When memory is tight, Max Context shrinks prefill down to 32 tokens per step so the prompt fits, while Speed keeps full speed and rejects what does not fit.
- Hide Helper Models: turn it on so drafters stay out of /v1/models and no client picks one by mistake.
- Max Concurrent Requests: keep it between 8 and 16, and do not lower it to 1. The scheduler admits no new request while as many cache writes are pending as that limit.
Burst Decode can stay on Balanced. Aggressive holds tokens for up to 200 ms before sending them, against 100 ms, so the client sees the first token later. One last warning: clicking Save Settings unloads every loaded model, even if you did not touch the cache, because the dashboard resends every field.
!Resource Management section of the oMLX 0.6.4 global settings: Balanced memory guard, hot cache at 9% (12 GB), SSD cache at 10% (372 GB), 16 concurrent requests, Chunked Prefill off, Prefill Priority set to Speed and no idle timeout
What does not make it faster
Three settings with promising names did not improve single-user speed:
- TurboQuant KV: with Lightning MTP on Qwen3.8-27B oQ4e, turning it on at 4 bits left peak memory unchanged with a 32k prompt (24.4 GiB) and lowered decode from 35.4 to 30.3 tok/s. In these hybrid models only the full-attention layers keep a KV cache, so there is little to compress. It saves memory on huge contexts; it does not make generation faster.
- SpecPrefill: brings the first token forward by having a small model drop the prompt tokens it scores as unimportant. The large model never sees them, so it loses information by design.
- Chunked Prefill: it only cuts time to first token when requests overlap.
Benchmarks on 0.6.4 on this M5 Max
With 0.6.4 and the settings above, this M5 Max generates between 1.19 and 1.88 times more tokens per second when writing code. With 15,000 tokens of context, the gain sits between 1.07 and 1.47 times. Without any acceleration, Qwen3.6-35B-A3B 8-bit now holds 77.9 tok/s with a 32k prompt, where 0.3.8 dropped to 15.2 tok/s. The tests date from 15 September 2026.
How it was measured
There are two test sets, because the built-in benchmark shows how speed changes with context but cannot compare with and without acceleration:
- Through the API: streaming requests to /v1/chat/completions at temperature 0.6 with reasoning off. There are two tests. In the short one, each configuration answers three coding tasks (Python, TypeScript and Go) twice, with 768-token answers. In the long one, it answers two questions about roughly 15,100 tokens of source code, with 512-token answers. Timing is taken on the client.
- Built-in benchmark: started from POST /admin/api/bench/start, three repeats per configuration and the median, prompts of 1,025, 4,097, 16,385 and 32,769 tokens, 256 generated tokens and the code_python corpus. It always samples greedily, which is the best case for speculative decoding.
Every request starts with a unique prefix, so none of them benefits from the prefix cache. Tables show the median, with the minimum and maximum in brackets when they spread more than 10% from the median in the API tests, or more than 25% in the built-in benchmark.
The tests ran back to back for about three hours, with other applications open. The built-in benchmark records macOS thermal pressure, and 194 of its 242 measurements were taken at the Heavy level, the third of five (Nominal, Moderate, Heavy, Trapping and Sleeping). Only the first gpt-oss batch ran cold. Its prefill with a 16k prompt gave 3,140 tok/s cold and 1,755 tok/s hot.
Read the absolute figures as those of a hot Mac. The comparisons with and without each setting were run one after the other, so the relative gains are more reliable than the absolute values.
Two details of the 0.6.4 built-in benchmark change how to read it. First, when it finishes it uploads the results to omlx.ai with the chip, the models and an identifier derived from the Mac, and the dashboard has no option to prevent it. With a local model, the only way to keep the results on the Mac is to tick ANE-aligned prompts (+1 token). That is the checkbox I used, which is why the prompts are one token longer.
Second, it loads vision models on the text-only engine unless MTP is on, so comparing with and without acceleration mixes two engines. With short prompts, Qwen3.6-35B-A3B oQ4e scored 49 tok/s on that engine and 100 tok/s through the API. That is why the gains in this guide come from the API tests.
Speed through the API with coding answers
| Model | Configuration | Decode | First token | 768-token answer |
| --- | --- | --- | --- | --- |
| Qwen3.6-35B-A3B 8-bit | No acceleration | 79.7 tok/s | 0.37 s | 10.0 s |
| | VLM MTP | 117.5 (110.5-125.5) tok/s | 0.37 s | 6.9 s |
| Qwen3.6-35B-A3B oQ4e-mtp | No acceleration | 100.2 tok/s | 0.34 s | 8.0 s |
| | Lightning MTP | 119.2 (108.5-139.1) tok/s | 0.46 s | 6.9 s |
| Qwen3.8-27B 4-bit | No acceleration | 24.1 tok/s | 0.79 s | 32.6 s |
| | VLM MTP | 34.5 tok/s | 0.65 s | 22.8 s |
| Qwen3.8-27B oQ4e-mtp | No acceleration | 21.1 tok/s | 0.82 s | 37.3 s |
| | Lightning MTP | 39.7 tok/s | 0.94 s | 20.4 s |
| Gemma 4 12B 8-bit | No acceleration | 27.4 tok/s | 0.84 s | 28.9 s |
| | VLM MTP | 50.5 (47.5-52.7) tok/s | 0.66 s | 15.9 s |
Speed through the API with 15,000 tokens of context
| Model | Configuration | Decode | First token |
| --- | --- | --- | --- |
| Qwen3.6-35B-A3B 8-bit | No acceleration | 84.0 tok/s | 6.20 s |
| | VLM MTP | 90.7 tok/s | 5.96 s |
| Qwen3.6-35B-A3B oQ4e-mtp | No acceleration | 100.2 tok/s | 5.83 s |
| | Lightning MTP | 115.5 tok/s | 6.14 s |
| Qwen3.8-27B 4-bit | No acceleration | 21.7 tok/s | 34.5 s |
| | VLM MTP | 23.2 tok/s | 32.6 s |
| Qwen3.8-27B oQ4e-mtp | No acceleration | 21.4 tok/s | 31.3 s |
| | Lightning MTP | 27.2 (24.4-30.1) tok/s | 35.2 s |
| Gemma 4 12B 8-bit | No acceleration | 26.6 tok/s | 20.1 s |
| | VLM MTP | 39.2 tok/s | 19.3 s |
Each row is the median of two 512-token answers over the same code context. With this context the large model accepted fewer draft tokens than when writing code: on Qwen3.8-27B 4-bit, the VLM MTP gain fell from 1.43x to 1.07x.
Built-in benchmark by prompt length
Generation speed in tok/s by prompt length, with prefill at 4k and MLX peak memory, which includes the weights:
| Model and configuration | 1k | 4k | 16k | 32k | Prefill 4k | Peak |
| --- | --- | --- | --- | --- | --- | --- |
| Qwen3.6-35B-A3B 8-bit, no acceleration | 92.1 | 91.1 | 84.1 | 77.9 | 2,426 | 38.7 GiB |
| Qwen3.6-35B-A3B 8-bit, VLM MTP | 118.3 | 106.6 | 100.2 | 94.4 | 3,113 | 41.8 GiB |
| Qwen3.6-35B-A3B 8-bit, DFlash (4-bit drafter) | 112.4 | 110.5 | 32.6 (23.3-35.0) | 16.6 | 2,734 | 38.2 GiB |
| Qwen3.6-35B-A3B oQ4e-mtp, no acceleration | 49.3 | 46.1 | 45.2 | 42.2 | 2,258 | 23.2 GiB |
| Qwen3.6-35B-A3B oQ4e-mtp, Lightning MTP | 77.2 (47.0-101.7) | 85.8 | 87.9 (85.7-116.6) | 82.8 (54.7-99.5) | 2,014 | 24.6 GiB |
| Qwen3.6-35B-A3B oQ4e-mtp, DFlash (4-bit drafter) | 142.7 | 86.3 | 39.2 | 17.3 | 3,373 | 22.7 GiB |
| Qwen3.8-27B 4-bit, no acceleration | 30.6 | 27.3 | 25.0 | 23.5 | 525 | 21.7 GiB |
| Qwen3.8-27B 4-bit, VLM MTP | 39.3 | 36.6 | 36.4 | 30.9 | 619 | 25.4 GiB |
| Qwen3.8-27B 4-bit, DFlash2 (4-bit drafter) | 48.2 (38.7-51.1) | 43.7 | 35.8 (22.0-39.0) | 6.3 | 561 | 22.8 GiB |
| Qwen3.8-27B 4-bit, DFlash2 (bf16 drafter) | 36.9 | 35.1 | 22.3 | 5.9 | 568 | 25.3 GiB |
| Qwen3.8-27B oQ4e-mtp, no acceleration | 27.0 | 26.4 | 24.4 | 21.4 | 545 | 22.2 GiB |
| Qwen3.8-27B oQ4e-mtp, Lightning MTP | 48.0 (42.6-58.6) | 37.7 | 41.1 | 35.4 | 499 | 24.4 GiB |
| Qwen3.8-27B oQ4e-mtp, Lightning MTP + TurboQuant 4-bit | 42.2 | 37.7 | 39.7 | 30.3 | 547 | 24.4 GiB |
| Qwen3.8-27B oQ4e-mtp, Lightning MTP + ANE prefill | 35.6 | 33.1 | 35.8 | 30.9 | 505 | 28.3 GiB |
| Qwen3.8-27B oQ4e-mtp, DFlash2 (4-bit drafter) | 41.4 | 33.8 | 18.8 (16.6-21.9) | 5.8 | 458 | 23.3 GiB |
| Gemma 4 12B 8-bit, no acceleration | 31.0 | 30.5 | 30.1 | 28.8 | 955 | 14.5 GiB |
| Gemma 4 12B 8-bit, VLM MTP | 65.6 | 42.7 | 50.6 | 49.0 | 940 | 15.5 GiB |
| Gemma 4 12B 8-bit, DFlash (4-bit drafter) | 92.4 (35.5-158.0) | 34.4 | 53.4 | 51.2 | 997 | 18.1 GiB |
| gpt-oss-20b, no acceleration | 120.2 | 106.5 | 100.4 | 78.8 | 3,491 | 12.4 GiB |
DFlash figures with a 1k prompt varied widely between repeats: on Gemma 4 12B they were 35.5, 92.4 and 158.0 tok/s. Speed depends on how many draft tokens the large model accepts, and that changes with the text it has to continue.
Compared with 0.3.8
The May table and this one are not fully comparable. That one was measured on the Qwen3.6-35B-A3B-MLX-8bit copy with 128 output tokens; this one, on the mlx-community copy with 256. Even so, the long-context improvement is too large to come from those differences:
| Qwen3.6-35B-A3B 8-bit, built-in benchmark | 0.3.8 (May) | 0.6.4 (September) |
| --- | --- | --- |
| Decode with a 1k prompt | 72.4 tok/s | 92.1 tok/s |
| Decode with a 4k prompt | 83.1 tok/s | 91.1 tok/s |
| Decode with a 32k prompt | 15.2 tok/s | 77.9 tok/s |
| First token with a 1k prompt | 577 ms | 298 ms |
End-to-end verification
A curl call to the OpenAI-compatible API to confirm the server is alive and a model responds:
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Authorization: Bearer your_api_key_here" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3-Coder-30B-A3B-Instruct-mlx-8bit",
"messages": [{"role": "user", "content": "Hello from oMLX"}]
}'
And from Claude Code, once you have exported the variables from the previous section:
claude --print "Are you running locally?"
The reply should come back from local without touching api.anthropic.com. To confirm at the network level, open Activity Monitor → Network, filter by the claude process and check that the outbound connection is against 127.0.0.1:8000.
!oMLX Chat view at localhost:8000/admin/chat with the model dropdown open, listing the four chat-completion models available locally (both Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus variants, Qwen3.5-9B-Claude-4.6-HighIQ, and Qwen3.6-40B-Claude-4.6-Opus-Deckard-Thinking); the embeddings and reranker models do not appear here because they are not chat-completion
What has changed since 0.3.8
This guide was written against oMLX 0.3.8 in May 2026. The current version is 0.6.4, released on 29 August 2026, and more than a dozen releases in between touch exactly what is tuned here. What follows is the summary of what changed, with the figures attributed to whoever measured them.
Lightning MTP speculative decoding
This is the change that affects this configuration most. Version 0.5.0, from 10 July 2026, added native depth-k speculative decoding for Qwen3.6-27B, Qwen3.6-35B-A3B, DeepSeek-V4-Flash and GLM-5.2.
On an Apple M3 Ultra the project measured Qwen3.6-35B-A3B going from 89.6 to 140.4 tok/s, a 1.57x gain. Qwen3.6-27B went from 35.0 to 55.1 tok/s. On this M5 Max, measured through the API on 15 September 2026, Lightning MTP took Qwen3.8-27B oQ4e from 21.1 to 39.7 tok/s. How to turn it on, and when another method is the better choice, is in the advanced settings.
Custom kernels, with a note for M5
The same 0.5.0 added custom compute kernels for DeepSeek V4, Qwen3.5/3.6 and GLM-5.2, with prefill improvements measured on M3 Ultra. They are 45% on DeepSeek-V4-Flash (313.3 to 455.8 tok/s at 64k context), 33% on Qwen3.6-27B and 99% on GLM-5.2 at 32k. What matters for this machine is that dispatch is NAX-aware precisely to avoid regressions on the M5 series, so it is worth confirming the effect rather than assuming it.
oQ and oQe quantisation
oMLX no longer depends only on the weights you download: it brings its own quantisation scheme. Version 0.5.0 added oQe, an activation-importance calibration pass with expert-coverage tracking for MoE models. In the project’s measurements oQ4e improves average accuracy over oQ4 while staying in the same disk-size class. That is 83.88% against 82.83% on Qwen3.6-35B-A3B, and 73.09% against 70.01% on Qwen3.5-9B.
The neural engine and cold starts
The 0.6.3 branch worked on the prefill path over Apple’s neural engine. Two of the project’s figures matter day to day. Bank compilation memory dropped from 35.8 GB to 4.7 GB, and an optional persistent compile cache cuts fresh-process startup by 53% to 66%. Version 0.6.4 continued down that road and improved Qwen3.8-Flash-Next prompt processing by 33.5%, with 24.2% less total request time at 32k, again on M3 Ultra.
New endpoints
The OpenAI- and Anthropic-compatible pair has been joined by /v1/rerank for document reranking, /v1/responses for Codex compatibility and /v1/mcp/tools for Model Context Protocol. With an embedding model and a reranker loaded at once, a whole retrieval pipeline fits on the same server. The full breakdown is in the guide to the API, the key and the port.
Sub-key authentication
Where there used to be a list of equivalent keys, there are now three credentials with different powers. A main key opens the API and the panel; sub-keys only call the API and cannot log into the panel or change settings; panel session tokens are held in a signed cookie. If you are handing access to more than one application, hand out sub-keys.
The setting names
The memory configuration in the earlier section still holds as judgement, but the keys are named differently in ~/.omlx/settings.json. The memory ceiling is now memory_guard_tier, with four levels (safe, balanced, aggressive and custom), and the bespoke value goes in memory_guard_custom_ceiling_gb. By default the ceiling is system RAM minus 8 GB. The two cache tiers are hot_cache_max_size and ssd_cache_max_size, and the concurrency limit is max_concurrent_requests.
Watch one of them: hot_cache_max_size ships as "0", which disables the hot cache in RAM. I go through it in benchmarks and the memory ceiling.
Other pieces that did not exist
Profiles save named bundles of settings over a single model, exposed as model:profile and at no extra memory cost. Aliases rename a model as far as the API is concerned. And there is experimental distributed inference, splitting a model’s layers across more than one Mac over SSH according to each machine’s memory, accounting for whether the link is Thunderbolt or 10GbE. The first two are in the dashboard and command line guide.
Sources and further reading
- Diego R. Baquero, "Running Claude Code with a local LLM": the tested configuration recipe this post’s settings are based on (TurboQuant 3.5-bit, 64k context, attribution off, Qwen 3.6 35B-A3B models).
- A First Comprehensive Study of TurboQuant, vLLM blog, 11 May 2026: independent analysis confirming 3.5-bit matches full-precision quality.
- omlx.ai: public leaderboard with real submissions from M5 Max and other machines.
- Maxing Out M5 Max Context Windows: Memory Fragmentation and TurboQuant Benchmarks (dasroot.net, April 2026): the other detailed experiment on long-context M5 Max with TurboQuant.
- Repositories: jundot/omlx, ml-explore/mlx, ml-explore/mlx-lm, huggingface.co, huggingface.co.
If you came here wondering what this actually is and how it differs from Apple’s MLX framework, I separate the two in what is oMLX. For a cross-platform alternative outside Apple Silicon, the Ollama local LLM guide covers the model catalogue and OpenAI-compatible API. To wire the local endpoint into an agent, the Anthropic SDK tutorial and the MCP multi-vendor patterns already talk to this same format with no code changes.
Related resources
- Calculator: which LLM fits your GPU?
- Learning path: Local LLMs
Frequently asked questions
Do I need to set an API key to use oMLX locally only?
No. oMLX ships with the API key field empty, and as long as the instance only listens on 127.0.0.1 that is safe for local testing. The moment you expose it to the LAN or share the endpoint, go to Settings → Auth & Info and set a long random string. Current versions also offer sub-keys that can only call the API and cannot log into the panel or change settings, meant for handing access to more than one application.
Can I use Claude Code from a MacBook against oMLX running on a Mac Studio?
Yes, through SSH port forwarding: ssh -L 8000:localhost:8000 user@mac-studio makes 127.0.0.1:8000 on the laptop point at the Studio's oMLX. Then export the same variables (ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and the ANTHROPIC_DEFAULT_*_MODEL set) in a script such as claude-local.sh, make it executable with chmod +x and run claude --bare from the SSH session. If you set a key in Settings → Auth & Info, replace the empty string with it.
Are the benchmarks in this post still valid if I install 0.6.4?
The May tables, no: they were measured on 0.3.8, and when I re-ran them on 15 September 2026, 0.6.4 was faster on this same machine. Without acceleration, Qwen3.6-35B-A3B 8-bit goes from 15.2 to 77.9 tok/s with a 32k prompt. Lightning MTP or VLM MTP add between 1.07x and 1.88x, depending on the model and the context. If you measure on your own Mac, compare through the API and not only with Bench → Performance, which runs vision models on a different engine when MTP is off.
---
Originally published on jacar.es, where the benchmarks are kept up to date with each oMLX release.
Mentions
oMLX 0.7.3 is out, and Ask AI is the feature I use the most Hey everyone, it's been a while! I don't post here very often, but I read every new post that comes in. Thank you to everyone using oMLX, and to everyone who shares questions, bug reports and tips here. oMLX 0.7.3 is out today, and I wanted to tell you about my personal favorite feature: Ask AI. Coding with big frontier-class open models keeps getting better, and I enjoy that too. But I've started to think the real value of local AI is somewhere else. I built this feature myself, and I use it every day. While I'm working, I press a shortcut and a small window shows up over whatever I'm doing. No copy and paste. I can ask it to look at my screen and explain what's on it, or drop in an image and ask about that. Or when my stock drops, I can just type "why did my INTC stock go down?" without thinking, and it searches the web and comes back with an answer. In 0.7.3 you can also talk to it, and it answers out loud. It doesn't need frontier-level intelligence, but it makes my day noticeably better. If your work looks anything like mine, I think you'll find it useful too. To try it, turn it on in the macOS app under App > Ask AI. Option-Space opens the question bar, and Option-Shift-Space starts a voice conversation. There's also a lot more in this release on the performance side: faster DeepSeek V4.1 prefill on M5, much lower memory when many long conversations run at once, faster batched decode on oQ models, and more. The release notes have all the details: github.com/jundot/omlx/releases/tag/v0.7.3 Thank you to everyone who supports oMLX. Something huge happens in the AI world every single day, but I hope you also get to enjoy some peaceful time in your own life.
oMLX 0.7.3 is out! github.com/jundot/omlx/releases/tag/v0.7.3 Ask AI, the question bar for your local models in any app, now talks back. Press Option-Shift-Space, speak, and your model answers out loud with captions. It can also search the web, read links and look at your screen. Ask AI voice and tools - Voice runs on LFM3-Omni, LFM3-Audio and Qwen3-Omni. With LFM3-Omni 2B on an M3 Ultra, the first audio comes about 100 ms after you stop talking. - Cut in while it's speaking, or share your screen in the middle of a conversation. - Web search, link reading and screen capture work in both text and voice. - Prompt placeholders like {date}, {app} and {clipboard}. Faster DeepSeek V4.1 - INT8 Activation Prefill on M5 Ultra: 64K prefill 2,882 -> 3,507 tok/s (+22%), 195K 2,476 -> 3,200 tok/s (+29%). - 12 concurrent requests decode 56.3 -> 125.7 tok/s (+123%). - Engram prefetch adds another +15% to prefill at 128K. Many requests at once, much less memory - 8 Lightning MTP streams at 81K on Qwen3.8-Flash-Next: peak memory 248 -> 161 GiB (-35%), decode 180.7 -> 241.1 tok/s (+33%). - Batched decode on oQ models: 16 requests 146.9 -> 221.9 tok/s (+51%) on M3 Ultra, 8 requests +74% on M2 Max. - Hybrid Qwen models reuse a shared system prompt across conversations, TTFT 0.89 -> 0.25 s. Also in 0.7.3 - Update downloaded models by fetching only the files that changed. - Better oQ quants, KL drops 9-49% for under 5% more size. - LFM3 text, audio and d1 models, and Unlimited-OCR-2. - Temporary chats in the web UI. Thanks to everyone who sent PRs and tested the rc!
It's only been a day since 0.7.1, but oMLX 0.7.2 is out with changes worth shipping now. github.com/jundot/omlx/releases/tag/v0.7.2 This release brings faster DeepSeek V4.1 on every oQ build, Lightning MTP in distributed inference, and Qwen3.8-Flash-Next GGUFs on Splash 1.4.0. Faster DeepSeek V4.1 on oQ2e to oQ4e - oQ3e with Lightning MTP + CED: decode 49.4 -> 68.2 tok/s (+38%), prefill 700 -> 949 tok/s (+36%). Every quant from oQ2e to oQ4e gets faster, with similar prefill gains. - Prefill holds around 900 tok/s all the way to a 128K prompt. - Measured on M3 Ultra 512GB. I'll run more tests on M5 Ultra 256GB. Lightning MTP in Distributed Inference - DeepSeek V4.1 and GLM-5.3 run Lightning MTP across Macs with both PP and TP. - With TP on two M3 Ultras, both models decode more than 50% faster, even over a TCP ring. I'll share exact numbers after more tests over RDMA. - A pipeline stall is fixed, so DeepSeek V4.1 across two Macs went from about 1 tok/s to about 28 tok/s. More Models on Splash - Qwen3.8-Flash-Next GGUFs run on Splash 1.4.0, and oMLX offers the matching DFlash2 draft. - Split GGUFs download and show up as one model, and mlx-community OptiQ builds now start. Other Improvements - Better prompt cache reuse in Claude Code sessions, on both the built-in engines and Splash. - Reasoning models answer Claude Code's auto-mode classifier before it times out. - DFlash works with ParoQuant Qwen models. - SpecPrefill no longer disturbs other requests decoding next to it, and Gemma 3 gets the right positions in sparse prefill. - Keyboard access to the web UI menus and better contrast in the dark theme. Thanks to everyone who sent PRs and tested 0.7.1 so quickly!
oMLX PR: opt-in mmap weights for reclaimable idle memory (reduced memory pressure) I'm the author of PR #4541, which adds an opt-in per-model 'Mapped weights' setting to oMLX. The PR is open for review; this isn't a released feature. omlx#4541 Instead of copying model weights into separate allocations, it maps eligible safetensors tensors as read-only, file-backed memory. When those pages aren't wired for GPU use, macOS can reclaim clean pages under memory pressure and read them back from the model files later. The goal is to let models that already fit coexist better with other apps. This does not reduce the resident memory needed during active inference or replace MoE expert offload. What I measured on an M2 Max with 32 GiB, using MLX 0.32.4: • Qwen 3.8 27B oQ3e: reported process footprint 15.0 → 1.3 GB, at similar generation speed. The weight pages are still RAM while resident; they are accounted as file cache, so this is not a reduction of total RAM usage to 1.3 GB. • With 10 GB of additional memory allocated, memory pressure stayed Normal rather than Warning, with no new swap in the mapped run. • Ornith 1.5 35B-A3B oQ4e: load + prompt + a full 128-token first response took 15.2 s vs 25.0 s with a cold file cache, and 7.6 s vs 24.8 s with a warm cache. Gate/up fusion was disabled in both modes; decode speed was similar. Checkpoint requirements and limitations: • Tensor data must begin at absolute 8-byte-aligned file offsets for zero-copy mapping. Misaligned tensors are copied; shards with no aligned tensors use the native loader. The settings UI reports mappable coverage. • The Qwen checkpoint was re-saved locally with aligned tensors (padding only; tensor values verified unchanged). The Ornith checkpoint was an oMLX-quantized oQ4e that was already mostly aligned. This PR does not rewrite checkpoints. • oMLX normally builds fused gate/up tensors during loading. That copies the routed expert weights into new allocations, so mapped models skip this fusion. Combining it with mmap would require a compatible checkpoint with tensors already fused and aligned on disk; that path is outside this PR. • Requires MLX ≥ 0.32.4, cannot be combined with MoE expert offload, and loaded checkpoint files must remain unchanged. These are local measurements, not a broad benchmark: the load/pressure comparisons are single runs. The PR includes the test setup and raw data. At equal fusion settings, I don't see a decode-speed improvement from mmap itself. I'd appreciate feedback on the loading approach and whether reclaimable idle memory is useful for your workflow.
oMLX 0.7.1 is out! github.com/jundot/omlx/releases/tag/v0.7.1 This is the official 0.7.1 release. It adds GGUF models through Splash, much faster DeepSeek V4.1, 4-bit / 8-bit KV cache formats, and distributed inference for GLM-5.3 and DeepSeek V4.1. What's new - GGUF models through Splash. The downloader filters repos by MLX or GGUF and lets you pick the files, and Qwen3.8-27B and Qwen3.6-35B-A3B GGUFs run on Splash. - Affine4 and Affine8 KV cache. Full-attention KV is 74% smaller with Affine4 at 32K context. - A fix for GPU hangs when several models run at once. - Distributed inference for more models, GLM-5.3 and DeepSeek V4.1 now run across Macs with both pipeline (PP) and tensor (TP) parallelism. - Claude Desktop support. One toggle maps claude-opus, claude-sonnet and claude-haiku to your local models. - Realtime mic transcription with every STT model. Speedups in this release - DeepSeek V4.1 Flash decode on M3 Ultra: 20.9 -> 34.3 tok/s (+64%) for one request, 19.4 -> 91.8 tok/s (+373%) with 8 concurrent requests. - Qwen3.8-Flash-Next with Lightning MTP on M5 Max: decode holds 87-98 tok/s instead of swinging between 64 and 93. - Qwen3.8-Flash-Next with expert offload on M1 Max: +5-8% decode. Coming from 0.7.0? You also get Muse Spark Open, the Recommended models tab, DFlash in the batched engines, decision models, batched prefill and the improved oQe quantization. Thanks to everyone who sent PRs and tested the dev and rc builds!
oMLX 0.7.1.dev2 is out! github.com/jundot/omlx/releases/tag/v0.7.1.dev2 oMLX can now serve Qwen3.8 and Qwen3.6 through Splash as an external engine. Splash runs the model on its own Metal engine with DFlash2, and the model still shows up in oMLX's model list, API, dashboard, cache stats and benchmarks like any other. This release also helps first-time users find a model that fits their Mac without guesswork, reworks model settings and chat in the web UI and the macOS app, and speeds up GLM-5.3 and DFlash. Splash as an external engine - Turn it on per model in Model Settings > Advanced. oMLX finds the Splash launcher and hands it the model and DFlash2 draft from your model directories, so nothing extra is downloaded. - Memory and context window are fitted to your oMLX memory limit, and an MLX model can run alongside it. - Vision checkpoints take images too. Speedups in this release - GLM-5.3-Flash decode on M5 Ultra: 85.5 -> 95.3 tok/s (+11%) at 64K context. - GLM-5.3-Flash, 4 streams on a shared 32K prompt on M5 Ultra: 81 -> 91 tok/s (+12%). - DFlash with 4 concurrent requests on M3 Ultra: +82% to +157% (MiMo V2.6 Flash, Qwen3-4B, Muse Glimmer 30B). What's new - Find the right model for your Mac. New users no longer have to guess which model to start with. The new Recommended tab shows only oQ models that will actually load on your Mac, with Lightning MTP on by default. - Reworked model settings. The web UI groups them into sections with a side menu, and the macOS app splits them into pages that save as you edit. - A new chat composer. Pick the model, profile and reasoning level right where you type, with a reasoning slider that follows the levels each model accepts. - DFlash now runs inside the batched engines, so it gets continuous batching and the prefix cache. DSpark drafters are accepted 61-85% of the time instead of 16-21%. - Requests survive memory pressure. They are paused and resumed instead of aborted, and cancelled prompts stay cached. - /v1/decisions for any chat model, plus the Decision 3.0 models with image and video input. - YaRN for Qwen3.5-family models past the native 262K context. - An "Allow inference without API key" toggle for LAN setups, and a Mistral Vibe launcher. - oQ for DeepSeek V4.1 at oQ2 to oQ2.7 and for EmbeddingGemma 2, plus MiniCPM-V 4.7 support. - A log table in the macOS app, copy buttons for model names, and Downloaded badges in the model browser. Most of the speedups came from contributors again. Thanks to everyone who sent PRs and tested the dev builds!
🔔 **oMLX Issue Digest** (Oct 8-12, 2026) 5 issues closed in [jundot/omlx](jundot/omlx) --- ### 📋 Issue Summary **#4368** - Partial Block Caching writes tail blocks but never matches them on Qwen3.6-35B-A3B 👤 *nerohwang55-netizen* | ℹ️ *has-pr* Reprocess still ~1795 instead of expected ~37. macOS M4 Pro, oMLX 0.7.0 **#4463** - JSON-schema whitespace cap causes excessive compilation memory 👤 *hujeremy* | ✅ *completed* `max_whitespace_cnt=32` increases peak RSS from 0.47GB to 2.81GB on small schemas. Causes memory guard rejections even with caching disabled. **#4491** - How are the 16 x 2048 KL windows placed for oQe model cards? 👤 *anemll-robot* | ✅ *completed* Request for KL evaluation details and scoring script to reproduce oQe benchmarks. **#4492** - Cache-freshness deferral fails below one paged block (block_size 4096) 👤 *danigame376* | ✅ *completed* Follow-up prompts <4096 tokens re-prefill entire prompt instead of restoring prefix. ~5-23s wasted vs 50ms wait could save. **#4493** - New oQe 4-bit fp16 has shorter usable context than older oQ5e on M5 Max 👤 *alexanderbi1984* | ✅ *completed* Qwen3.8-Flash-Next-oQ4e-fp16-mtp hits ~65k context vs oQ5e's 256k on M5 Max 128GB. Prefill transient memory appears to be the bottleneck. --- ### 📊 Quick Stats - Total: 5 issues | Closed: 5 - High impact: Memory compilation (#4463), Cache deferral (#4492) - All resolved by jundot
How to install and tune oMLX on M5 Max 128 GB --- title: How to install and tune oMLX on M5 Max 128 GB published: true description: Recipe for oMLX 0.6.4 on a Mac M5 Max with 128 GB: install, Claude Code and advanced settings with screenshots (Lightning MTP, VLM MTP and DFlash). tags: macos, llm, ai, apple canonical_url: jacar.es cover_image: media.jacar.es --- oMLX is an LLM inference server built on MLX, the framework Apple shipped in December 2023 for Apple Silicon. It adds continuous batching, two-tier KV cache (RAM + SSD), and an OpenAI- and Anthropic-compatible API. On a Mac M5 Max with 128 GB of unified memory you can hold three or four large models at once with TurboQuant 3.5-bit on KV cache. That is enough to feed chat, agent and IDE in parallel. This guide collects the configuration tested in May 2026 to get the most out of that combination. Quick answers Which oMLX version do I use and how do I install it? Version 0.6.4 (released 29 August 2026, Apache 2.0). The direct route: download the .dmg from GitHub Releases, open it and drag to Applications. Or install via brew tap jundot/omlx https://github.com/jundot/omlx and brew install jundot/omlx/omlx. The first load of the panel at http://localhost:8000/admin prompts for the API key. The full Homebrew route, including the service that starts on its own, is in installing, updating and uninstalling oMLX with Homebrew. What TurboQuant setting makes sense on 128 GB? 3.5-bit. vLLM’s independent analysis published 11 May 2026 shows 3.5-bit matches full-precision quality with about 4x less KV-cache memory. On M5 Max that turns 128k context from "blows past available RAM" into "fits alongside other loaded models." How much faster is it with the advanced 0.6.4 settings? Between 1.19x and 1.88x when generating code, and between 1.07x and 1.47x with 15,000 tokens of context, measured through the API on 15 September 2026. The biggest gain came from Lightning MTP on Qwen3.8-27B oQ4e, from 21.1 to 39.7 tok/s, and it needs a build of the model that keeps the MTP head. The details are in the advanced settings. Which models can I load concurrently on 128 GB? Primary: unsloth/Qwen3.6-35B-A3B-MLX-8bit (37.7 GB, MoE with 3B active). Fast helper: Qwen3-14B-Instruct-mlx-4bit (8 GB). Vision: Qwen2.5-VL-32B-mlx-4bit (18 GB). Embeddings: BGE-M3-mlx (1.2 GB). Reranker: ModernBERT-base-mlx (150 MB). Comfortable sum: ~65 GB with headroom. How do I point Claude Code at the local endpoint? Since 0.5.0 the short way is omlx launch claude, which exports the variables for you. By hand: export ANTHROPIC_BASE_URL=http://127.0.0.1:8000, ANTHROPIC_AUTH_TOKEN=<your_api_key> and the three ANTHROPIC_DEFAULT_*_MODEL variables (Opus/Sonnet/Haiku). Launch with claude --bare to drop the system prompt to ~1,795 tokens. The dashboard’s Claude Code with oMLX section assembles the full command for you. Does this replace Claude Opus 4.7? Not one for one. Claude Code is tuned for Claude’s tool-use format; a non-Claude model behind the endpoint loses reliability in agentic loops. Use this for offline work, sensitive data that should not leave the Mac, or as a fallback when api.anthropic.com is rate-limiting you. Install The easiest path: download the .dmg from GitHub Releases, open it and drag the app to Applications. The server starts in the background with a menu-bar icon. Or install via Homebrew: brew tap jundot/omlx https://github.com/jundot/omlx brew install jundot/omlx/omlx omlx start omlx serve --model-dir ~/models The first hit to http://localhost:8000/admin prompts for the API key. By default oMLX ships with no API key set (empty field), which is fine for local testing. If the instance only listens on 127.0.0.1 this is safe; the moment you expose it to the LAN or plan to share the endpoint, go to Settings → Auth & Info and set a long random string. The section accepts multiple keys at once. !Login screen of the oMLX admin panel at localhost:8000/admin, with the API Key field empty and a "Stay signed in for 30 days" checkbox Server settings for 128 GB Settings → Global Settings holds the full server configuration. The decisions that matter on a 128 GB machine: - Server → Host: Localhost only (127.0.0.1). If you open it to the LAN, put real auth in front first. - Server → Port: 8000 by default. - Resource Management → Memory Limit (Total): Auto. oMLX subtracts what macOS reserves; on 128 GB you end up around 110-114 GB for inference. - Resource Management → Memory Limit (Models Only): Auto. Keeps a percentage for activations, KV cache and auxiliary processes. - Resource Management → Hot Cache Limit: 10%. Intermediate RAM-tier KV cache. With TurboQuant enabled on the models that support it (see below), 10% is the value people running this setup in practice land on for 128 GB. With TurboQuant off and a single model loaded you can drop to Off, but you stop gaining. - Resource Management → Cold Cache Limit (SSD Cache): 10%. Around 80 GB of SSD for cold tokens without saturating it. - Resource Management → Max Concurrent Requests: 16. Comfortable for a single user with an agent, a chat session and an IDE pinging at once. Raise to 32 if you share with a small team. - Resource Management → Idle Timeout: None. Keeps models warm; the first token arrives in well under a second instead of seconds later. - Generation Defaults → Max Context Window: 256000 as the global default. The Qwen3.6 family holds up at long context, and with TurboQuant the effective RAM stretches enough to actually use it. Each model can be capped lower in Model Settings. - Generation Defaults → Max Tokens: 64000. Upper bound per response. Going higher only matters if you plan to generate full books in one shot. - Generation Defaults → Temperature: 1.0 for general use, 0.2-0.5 for code models. - Model Settings → Experimental Features → TurboQuant KV Cache: enable at 3.5-bit on the large dense models. vLLM’s independent study published 11 May 2026 of Google’s TurboQuant shows 3.5-bit matches full-precision quality with roughly 4x less KV-cache memory. On M5 Max that makes 128k context fit on models where FP16 does not. On 0.6.4 it cannot be combined with VLM MTP, and in these tests it did not speed up generation; if speed is what you are after, see the advanced settings. !oMLX Global Settings page: Host set to Localhost only (127.0.0.1), Port 8000, Memory Limit Total Auto (120 GB), Memory Limit Models Only Auto (108 GB), Hot Cache at 9% (12 GB), Cold Cache SSD at 10% (372 GB), Max Concurrent Requests 16, Idle Timeout None, Max Context Window 256000, Max Tokens 64000, Temperature 1.0, Top P 0.95 Model stack for multi-LLM With 128 GB you have room to load one large primary model, a fast helper, a VLM, embeddings and a reranker concurrently. Download them from Models → Downloader by pasting the Hugging Face repo URL. The recommendation holding up in May 2026 comes from people actually running Claude Code on oMLX on M5 Max (see Diego R. Baquero’s gist for the source): Unsloth’s Qwen 3.6 35B-A3B family in MoE with ~3B active parameters per token. On 128 GB the 8-bit fits without breaking a sweat: - Primary (chat + code + reasoning): unsloth/Qwen3.6-35B-A3B-MLX-8bit (~37.7 GB). MoE 35B with 3B active. The all-rounder: long chat, code, agents. On 128 GB it fits alongside everything else without squeezing. - More compressed alternative: unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit (~21.6 GB) if you want two primary models loaded at once. Loses a notch of quality versus 8-bit but leaves room to experiment. - Dense reasoning (when MoE falls short): Mistral-Large-2-123B-Instruct-mlx-4bit (~70 GB) or Llama-3.3-70B-Instruct-mlx-4bit (~40 GB). For deep reasoning in a single pass, where a dense architecture beats a small-activation MoE. - Fast helper: Qwen3-14B-Instruct-mlx-4bit (~8 GB). For cheap tasks in agents, parsing and summaries. - Vision: Qwen2.5-VL-32B-Instruct-mlx-4bit (~18 GB) covers OCR, image description and multimodal reasoning. - Embeddings: BGE-M3-mlx (~1.2 GB), dense + sparse + multi-vector in a single model. - Reranker: ModernBERT-base-mlx (~150 MB) to close the loop on a decent RAG pipeline. Comfortable concurrent load with Qwen3.6 35B-A3B 8-bit as primary + 14B helper + VL-32B + embeddings + reranker: around 65-70 GB. Add Mistral Large 2 123B on top for dense reasoning and you hit ~135 GB nominal. The LRU policy moves idle models to SSD, so the cohabitation works in practice for sessions that do not pin everything at once. Pin the primary model from Model Manager so it never gets evicted. Point Claude Code at the local endpoint oMLX exposes an Anthropic-compatible API at http://127.0.0.1:8000. In current versions the routes live under /v1: POST /v1/messages and POST /v1/messages/count_tokens. Claude Code respects ANTHROPIC_BASE_URL, so pointing the CLI at your Mac is a matter of exporting environment variables. First, turn off the attribution header in Claude Code’s global config so the gateway does not bolt extra noise onto the prompt: Since 0.5.0 there is a shortcut that does all of this for you and did not exist when this recipe was written: omlx launch claude It sets ANTHROPIC_BASE_URL against your server, puts the configured key in ANTHROPIC_AUTH_TOKEN (or omlx if there is none), empties ANTHROPIC_API_KEY to force the base URL to be used, raises API_TIMEOUT_MS to 3,000,000 and disables non-essential traffic. It also sets CLAUDE_CODE_MAX_CONTEXT_TOKENS and CLAUDE_CODE_AUTO_COMPACT_WINDOW to what the engine can actually do, which is what the context-scaling option used to do by hand. It enforces a minimum window of 48,000 tokens and will not start if the chosen model falls short. There are equivalent commands for Codex, OpenCode, OpenClaw, Hermes and Pi, described in the dashboard and command line guide. The manual recipe below still works and is the one to use if you want fine control over each variable. ~/.claude/settings.json: { "env": { "CLAUDE_CODE_ATTRIBUTION_HEADER": "0" } } Then the startup command: export ANTHROPIC_BASE_URL=http://127.0.0.1:8000 export ANTHROPIC_AUTH_TOKEN=<your_api_key> export ANTHROPIC_DEFAULT_OPUS_MODEL=unsloth/Qwen3.6-35B-A3B-MLX-8bit export ANTHROPIC_DEFAULT_SONNET_MODEL=unsloth/Qwen3.6-35B-A3B-MLX-8bit export ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen3-14B-Instruct-mlx-4bit export ANTHROPIC_DEFAULT_MODEL=unsloth/Qwen3.6-35B-A3B-MLX-8bit export API_TIMEOUT_MS=600000 export CLAUDE_CODE_USE_BEDROCK=0 export DISABLE_NONESSENTIAL_TRAFFIC=1 claude --bare The --bare flag skips hooks, LSP, plugin sync and auto-memory, dropping Claude Code’s system prompt to around 1,795 tokens (versus the thousands it uses with everything loaded). For offline work against a local model that is the sensible default: every token in the system prompt is bandwidth your Mac would otherwise burn. Drop the flag when you point back at api.anthropic.com. The oMLX dashboard builds this command for you in the Claude Code with oMLX section: pick Opus, Sonnet and Haiku from three dropdowns and copy the ready-to-paste command. Set Context scaling for Claude Code to 64000: Claude Code requests 200k tokens by default, but a local model behaves better with a 64k ask than a 200k ask it cannot honor. The option scales the reported counts so auto-compact triggers at the target size. !oMLX dashboard with the Claude Code with oMLX section fully configured: Local mode active, Opus mapped to Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-mlx-8bit, Sonnet to Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-8bit, Haiku to Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT-HERETIC-UNCENSORED-MLX-mxfp8, Context scaling for Claude Code enabled with target 200000, plus pre-built commands for Codex, OpenCode and other integrations below A real caveat: Claude Code is tuned to Claude’s tool-use format and response patterns. A non-Claude model behind ANTHROPIC_BASE_URL works for autocomplete and reasoning, but you will see drops in tool-call reliability and in agentic loops. Use this for offline work, sensitive data that should not leave the Mac, or as a fallback when api.anthropic.com is rate-limiting you. It is not a 1:1 substitute for Claude Opus 4.7. SSH from another Mac If oMLX runs on a Mac Studio and you want to use Claude Code from your MacBook, SSH port forwarding is the bridge: ssh -L 8000:localhost:8000 user@mac-studio Once connected, 127.0.0.1:8000 on your laptop points to the Studio’s oMLX. Save this script as claude-local.sh: #!/bin/bash export ANTHROPIC_BASE_URL='http://localhost:8000' export ANTHROPIC_AUTH_TOKEN='' export ANTHROPIC_DEFAULT_OPUS_MODEL='Qwen3.6-35B-A3B-MLX-8bit' export ANTHROPIC_DEFAULT_SONNET_MODEL='Qwen3.6-35B-A3B-MLX-8bit' export ANTHROPIC_DEFAULT_HAIKU_MODEL='Qwen3-14B-MLX-4bit' export ANTHROPIC_DEFAULT_MODEL='Qwen3.6-35B-A3B-MLX-8bit' export API_TIMEOUT_MS=600000 export CLAUDE_CODE_USE_BEDROCK=0 export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude --bare --dangerously-skip-permissions Make it executable with chmod +x claude-local.sh and run it from the SSH session. The default API key is the empty field. If you set one in Settings → Auth & Info, replace '' with it. --bare drops Claude Code’s system prompt to ~1,795 tokens. --dangerously-skip-permissions bypasses interactive permission prompts that don’t make sense in an automated pipeline. Benchmark on real hardware Bench → Performance runs tests on your actual hardware. The panel covers prefill at eight prompt sizes (pp1024, pp4096, pp8192, pp16384, pp32768, pp65536, pp131072, pp200000) and continuous batching at 2x, 4x and 8x concurrency. Results go to the public oMLX leaderboard, and My Submissions shows yours next to other machines on the same profile. On 0.6.4 the upload happens automatically when each run finishes; how to avoid it is in the 0.6.4 benchmarks. Reference numbers measured on M5 Max 40-core with 128 GB on oMLX 0.3.8, in May 2026. Not extrapolated (the omlx.ai leaderboard already has M5 Max submissions, and vLLM’s independent TurboQuant study published 11 May 2026 confirms the figures for the large dense models): - Qwen 3.6 35B-A3B 8-bit (MoE, 3B active): 65-80 tok/s on decode at short context. The model that most changes day-to-day use on this machine. Live benchmarks from this setup: | Test | TTFT | Decode TPS | E2E | Peak Mem | | --- | --- | --- | --- | --- | | pp1024/tg128 | 577 ms | 72.4 tok/s | 2.35s | 38.5 GB | | pp4096/tg128 | 1,293 ms | 83.1 tok/s | 2.83s | 36.2 GB | | pp8192/tg128 | 2,995 ms | 82.1 tok/s | 4.55s | 36.5 GB | | pp32768/tg128 | 13,785 ms | 15.2 tok/s | 22.18s | 40.9 GB | Continuous batching (pp1024 / tg128): | Batch | Decode TPS | Speedup | | --- | --- | --- | | 1x (baseline) | 72.4 tok/s | 1.00x | | 2x | 87.1 tok/s | 1.20x | | 4x | 123.8 tok/s | 1.71x | | 8x | 237.7 tok/s | 3.28x | - Llama 3.3 70B 4-bit: 14-18 tok/s on decode. - Mistral Large 2 123B 4-bit with TurboQuant 3.5-bit on KV cache: 8-11 tok/s on decode. The number that matters at 128k context is peak memory, ~74 GB (FP16 KV blows past 128 GB). dasroot.net’s long-context M5 Max experiment documents the same peaks for the 104B dense model. - gpt-oss-20b MXFP4-Q4 on M5 Max 40c per the public oMLX benchmark submission lands around 100 tok/s on decode. The tables above are our own measurement on 0.3.8, and on 15 September 2026 I re-ran the tests on 0.6.4 on this same machine. With no acceleration at all, this model goes from 15.2 to 77.9 tok/s with a 32k prompt. Speculative decoding adds between 1.07x and 1.88x, depending on the model and the context. The new tables are in the 0.6.4 benchmarks. vLLM measured the TurboQuant KV-cache compression at 4.41x in May 2026. That figure is what separates "I have 128k context but it does not fit" from "I have it and can load two other models alongside it". At 4-bit and 3.5-bit, quality stays close to full precision; at 3-bit you start to feel it in code and long-form reasoning. !oMLX Bench → Performance Benchmark page with Qwen3.6-35B-A3B-MLX-8bit selected, Single Request Test checkboxes (pp1024, pp4096, pp8192, pp32768) and Continuous Batching Test checkboxes (2x, 4x, 8x), Run Benchmark button, and My Submissions link Advanced settings for maximum performance on 0.6.4 On 0.6.4, the biggest speed lever on this M5 Max is speculative decoding, and the right method depends on the model. Measured through the API at temperature 0.6, Qwen3.8-27B oQ4e went from 21.1 to 39.7 tok/s with Lightning MTP. Qwen3.6-35B-A3B 8-bit went from 79.7 to 117.5 tok/s with VLM MTP. Everything in this section was measured on 15 September 2026 on the machine this guide is about, and the method and the full tables are in the 0.6.4 benchmarks section. Which acceleration method to use with each model oMLX 0.6.4 has three speculative decoding methods and allows only one per model: - Lightning MTP: uses the multi-token prediction (MTP) head that ships inside the weights. It only works if the conversion kept the mtp.* tensors. - VLM MTP: uses an external MTP drafter under 2 GB, trained for that base model. It only kicks in when no other request is running. - DFlash: uses a z-lab block-diffusion drafter that proposes up to 16 tokens per pass. It serves requests one at a time and keeps its own prefix cache. In all three the large model verifies every proposed token, so the output comes from the large model, not from the drafter. This is what each combination measured on this M5 Max: | Model | Method | What to download | Size | Decode writing code | Decode with 15,000 tokens of context | | --- | --- | --- | --- | --- | --- | | Qwen3.6-35B-A3B 8-bit | VLM MTP | mlx-community/Qwen3.6-35B-A3B-MTP-bf16 | 1.7 GB | 79.7 → 117.5 tok/s | 84.0 → 90.7 tok/s | | Qwen3.6-35B-A3B oQ4e | Lightning MTP | Jundot/Qwen3.6-35B-A3B-oQ4e-mtp (full model) | 21.6 GB | 100.2 → 119.2 tok/s | 100.2 → 115.5 tok/s | | Qwen3.8-27B oQ4e | Lightning MTP | Jundot/Qwen3.8-27B-oQ4e-mtp (full model) | 17.0 GB | 21.1 → 39.7 tok/s | 21.4 → 27.2 tok/s | | Qwen3.8-27B 4-bit | VLM MTP | mlx-community/Qwen3.8-27B-MTP-bf16 | 0.87 GB | 24.1 → 34.5 tok/s | 21.7 → 23.2 tok/s | | Gemma 4 12B 8-bit | VLM MTP | mlx-community/gemma-4-12B-it-assistant-bf16 | 0.88 GB | 27.4 → 50.5 tok/s | 26.6 → 39.2 tok/s | | gpt-oss-20b | None | – | – | – | – | For Qwen3.6-35B-A3B, the oQ4e build with Lightning MTP was the fastest in both tests and takes about 15 GB less than the 8-bit one. With long context and prose answers every gain shrinks, because the large model accepts fewer draft tokens, but no configuration was slower than its baseline. If a Qwen model’s settings show the warning Config declares MTP layers but the weight files contain neither mtp.* tensors nor native nextn layers, the conversion stripped the MTP head. The default mlx-lm converters do that: it applied to the Qwen3.6-35B-A3B builds from mlx-community and the Qwen3.8-27B builds from lmstudio-community installed on this Mac. You have two ways out: download a build that keeps the head, such as the oQ4e-mtp builds published by the oMLX author, or turn on VLM MTP with an external drafter. Downloading the drafters Drafters download like any other model. In Models → Downloader, paste the Hugging Face repository id and start the download. oMLX recognises drafters by their type and marks them as helpers; with Hide Helper Models on, they also stay out of the API’s model list. !oMLX 0.6.4 Download from HuggingFace form with the repository id mlx-community/Qwen3.6-35B-A3B-MTP-bf16 filled in, the HF Token field empty and the Download button Turning on Lightning MTP With a model that keeps its MTP head, open Settings → Model Settings, click the model’s gear icon and turn on Lightning MTP in the Acceleration block. When you save, oMLX reloads the model. On Qwen the default is 3 draft tokens per cycle, and an adaptive controller adjusts it from each response’s acceptance rate. The oMLX log writes one line per response with that rate: on Qwen3.6-35B-A3B oQ4e it sat around 72-77%, with 2.5 to 2.9 tokens emitted per cycle. With greedy sampling the output matches the model without MTP, apart from tiny numerical differences. With temperature, rejection sampling keeps the large model’s distribution. !Acceleration block of the Jundot/Qwen3.8-27B-oQ4e-mtp model settings in oMLX 0.6.4, with the Lightning MTP switch turned on Turning on VLM MTP with an external drafter In the same Acceleration block, turn on VLM MTP, pick the drafter from the drop-down and save. Three conditions in the 0.6.4 code are worth knowing first: - The model has to load on the vision engine. The multimodal Qwen3.6 and Qwen3.8 builds do so by default, but on a text-only model the setting is silently ignored. - It cannot be combined with TurboQuant or with repetition or presence penalties. The built-in Qwen presets set presence_penalty to 1.5, which turns it off. - It only speeds up a request when nothing else is running. With concurrent requests, oMLX falls back to normal batching. !VLM MTP card in the mlx-community/Qwen3.6-35B-A3B-8bit settings in oMLX 0.6.4, turned on with the Qwen3.6-35B-A3B-MTP-bf16 drafter and the block size left empty, which means 4 tokens You can apply the same change through the admin API, which is handier when you configure more than one model or script it. The first command logs in with your key and stores the cookie; the second updates the model’s settings: curl -c omlx.cookies -X POST http://127.0.0.1:8000/admin/api/login \ -H "Content-Type: application/json" \ -d '{"api_key": "your_api_key_here"}' curl -b omlx.cookies -X PUT \ http://127.0.0.1:8000/admin/api/models/Qwen3.6-35B-A3B-8bit/settings \ -H "Content-Type: application/json" \ -d '{"vlm_mtp_enabled": true, "vlm_mtp_draft_model": "Qwen3.6-35B-A3B-MTP-bf16"}' The model name in the path and the drafter name are the ids shown in the model list. For Lightning MTP the body is {"mtp_enabled": true}. The endpoint rejects any field it does not know, so a typo does not slip through. DFlash only pays off with short prompts DFlash was the fastest method with 1,000-token prompts and the worst with long context. In the built-in benchmark, Qwen3.6-35B-A3B 8-bit rose from 92 to 112 tok/s with a 1k prompt, but dropped from 78 to 17 tok/s with 32k. Qwen3.8-27B 4-bit dropped from 23.5 to 6.3 tok/s with 32k. If you use Claude Code, which sends tens of thousands of tokens of context every turn, leave it off. For short-prompt work such as chat or autocomplete, download the matching z-lab drafter: z-lab/Qwen3.6-35B-A3B-DFlash (0.77 GB), z-lab/Qwen3.8-27B-DFlash2 (3.85 GB) or z-lab/gemma4-12B-it-DFlash (1.46 GB). Turn it on in Acceleration → DFlash and quantise the drafter to 4 bits. On the 27B, the 4-bit drafter gave 48.2 tok/s with a 1k prompt against 36.9 tok/s in bf16, and used 2.5 GiB less. It has two more limits. It serves one request at a time. And although dflash_max_ctx sends long requests to the normal engine, after the first one DFlash stays off until you reload the model. !oMLX 0.6.4 DFlash card with the Qwen3.6-35B-A3B-DFlash drafter selected, drafter quantisation on with 4-bit weights, 16-bit activations and group size 64, and the max context threshold left unlimited Prefill on the Neural Engine: no gain with Lightning MTP Qwen ANE Prompt Processing splits part of the Qwen3.5, 3.6 and 3.8 prefill between Apple’s Neural Engine (ANE) and the GPU. The Tune for this Mac button measures the best split on your machine and saves nothing until you apply the result. On Qwen3.8-27B oQ4e it recommended sending 35% of the MLP channels and 37.5% of the GDN layers to the ANE. With that split, prefill on an 8,193-token prompt went from 452 to 510 tok/s, 12.8% more. That 12.8% did not hold with the full model. With the split applied and Lightning MTP on, the built-in benchmark measured 505 tok/s of prefill with a 4k prompt against 499 without the ANE, and 422 against 471 with 32k. Through the API, the first token with 15,100 tokens of context took 34.8 s, against 35.2 s without the ANE. In exchange the loaded model grew from 16.0 to 20.7 GB and loading went from 2 to 20 s, so on this Mac I left it off. Before tuning it on an M5 Max, turn off Use both ANEs in the model settings. That mode is meant for the M3 Ultra, which joins two chips and has two ANEs. It only speeds up prompts of 2,048 tokens or more; generation stays on the GPU. The project itself documents three costs. It is not bit-exact, because the ANE runs weights requantised to INT8. It relies on private Apple interfaces that a macOS update can break, and it lengthens model load. If you turn it on anyway, also turn on Reuse Compiled ANE Programs under Global Settings → Advanced so it does not recompile on every start. !oMLX 0.6.4 Qwen ANE Prompt Processing card with the split the tuner proposed: 2048-token block, padding from 1816, 0.35 of the MLPs and 0.375 of the GDN layers on the ANE, Use both ANEs off and the Tune for this Mac button The global settings that matter oMLX’s default global configuration already suits a single user, and only four settings change anything measurable: - Kernel memory limit: macOS caps the memory the GPU can wire by default, and the dashboard shows a red warning when that cap is below what oMLX could use. Run sudo sysctl iogpu.wired_limit_mb=124518, which on 128 GB is RAM minus 5%, and restart oMLX, which applies it at start-up. The value is lost when the Mac restarts. - Prefill Priority: set it to Speed. When memory is tight, Max Context shrinks prefill down to 32 tokens per step so the prompt fits, while Speed keeps full speed and rejects what does not fit. - Hide Helper Models: turn it on so drafters stay out of /v1/models and no client picks one by mistake. - Max Concurrent Requests: keep it between 8 and 16, and do not lower it to 1. The scheduler admits no new request while as many cache writes are pending as that limit. Burst Decode can stay on Balanced. Aggressive holds tokens for up to 200 ms before sending them, against 100 ms, so the client sees the first token later. One last warning: clicking Save Settings unloads every loaded model, even if you did not touch the cache, because the dashboard resends every field. !Resource Management section of the oMLX 0.6.4 global settings: Balanced memory guard, hot cache at 9% (12 GB), SSD cache at 10% (372 GB), 16 concurrent requests, Chunked Prefill off, Prefill Priority set to Speed and no idle timeout What does not make it faster Three settings with promising names did not improve single-user speed: - TurboQuant KV: with Lightning MTP on Qwen3.8-27B oQ4e, turning it on at 4 bits left peak memory unchanged with a 32k prompt (24.4 GiB) and lowered decode from 35.4 to 30.3 tok/s. In these hybrid models only the full-attention layers keep a KV cache, so there is little to compress. It saves memory on huge contexts; it does not make generation faster. - SpecPrefill: brings the first token forward by having a small model drop the prompt tokens it scores as unimportant. The large model never sees them, so it loses information by design. - Chunked Prefill: it only cuts time to first token when requests overlap. Benchmarks on 0.6.4 on this M5 Max With 0.6.4 and the settings above, this M5 Max generates between 1.19 and 1.88 times more tokens per second when writing code. With 15,000 tokens of context, the gain sits between 1.07 and 1.47 times. Without any acceleration, Qwen3.6-35B-A3B 8-bit now holds 77.9 tok/s with a 32k prompt, where 0.3.8 dropped to 15.2 tok/s. The tests date from 15 September 2026. How it was measured There are two test sets, because the built-in benchmark shows how speed changes with context but cannot compare with and without acceleration: - Through the API: streaming requests to /v1/chat/completions at temperature 0.6 with reasoning off. There are two tests. In the short one, each configuration answers three coding tasks (Python, TypeScript and Go) twice, with 768-token answers. In the long one, it answers two questions about roughly 15,100 tokens of source code, with 512-token answers. Timing is taken on the client. - Built-in benchmark: started from POST /admin/api/bench/start, three repeats per configuration and the median, prompts of 1,025, 4,097, 16,385 and 32,769 tokens, 256 generated tokens and the code_python corpus. It always samples greedily, which is the best case for speculative decoding. Every request starts with a unique prefix, so none of them benefits from the prefix cache. Tables show the median, with the minimum and maximum in brackets when they spread more than 10% from the median in the API tests, or more than 25% in the built-in benchmark. The tests ran back to back for about three hours, with other applications open. The built-in benchmark records macOS thermal pressure, and 194 of its 242 measurements were taken at the Heavy level, the third of five (Nominal, Moderate, Heavy, Trapping and Sleeping). Only the first gpt-oss batch ran cold. Its prefill with a 16k prompt gave 3,140 tok/s cold and 1,755 tok/s hot. Read the absolute figures as those of a hot Mac. The comparisons with and without each setting were run one after the other, so the relative gains are more reliable than the absolute values. Two details of the 0.6.4 built-in benchmark change how to read it. First, when it finishes it uploads the results to omlx.ai with the chip, the models and an identifier derived from the Mac, and the dashboard has no option to prevent it. With a local model, the only way to keep the results on the Mac is to tick ANE-aligned prompts (+1 token). That is the checkbox I used, which is why the prompts are one token longer. Second, it loads vision models on the text-only engine unless MTP is on, so comparing with and without acceleration mixes two engines. With short prompts, Qwen3.6-35B-A3B oQ4e scored 49 tok/s on that engine and 100 tok/s through the API. That is why the gains in this guide come from the API tests. Speed through the API with coding answers | Model | Configuration | Decode | First token | 768-token answer | | --- | --- | --- | --- | --- | | Qwen3.6-35B-A3B 8-bit | No acceleration | 79.7 tok/s | 0.37 s | 10.0 s | | | VLM MTP | 117.5 (110.5-125.5) tok/s | 0.37 s | 6.9 s | | Qwen3.6-35B-A3B oQ4e-mtp | No acceleration | 100.2 tok/s | 0.34 s | 8.0 s | | | Lightning MTP | 119.2 (108.5-139.1) tok/s | 0.46 s | 6.9 s | | Qwen3.8-27B 4-bit | No acceleration | 24.1 tok/s | 0.79 s | 32.6 s | | | VLM MTP | 34.5 tok/s | 0.65 s | 22.8 s | | Qwen3.8-27B oQ4e-mtp | No acceleration | 21.1 tok/s | 0.82 s | 37.3 s | | | Lightning MTP | 39.7 tok/s | 0.94 s | 20.4 s | | Gemma 4 12B 8-bit | No acceleration | 27.4 tok/s | 0.84 s | 28.9 s | | | VLM MTP | 50.5 (47.5-52.7) tok/s | 0.66 s | 15.9 s | Speed through the API with 15,000 tokens of context | Model | Configuration | Decode | First token | | --- | --- | --- | --- | | Qwen3.6-35B-A3B 8-bit | No acceleration | 84.0 tok/s | 6.20 s | | | VLM MTP | 90.7 tok/s | 5.96 s | | Qwen3.6-35B-A3B oQ4e-mtp | No acceleration | 100.2 tok/s | 5.83 s | | | Lightning MTP | 115.5 tok/s | 6.14 s | | Qwen3.8-27B 4-bit | No acceleration | 21.7 tok/s | 34.5 s | | | VLM MTP | 23.2 tok/s | 32.6 s | | Qwen3.8-27B oQ4e-mtp | No acceleration | 21.4 tok/s | 31.3 s | | | Lightning MTP | 27.2 (24.4-30.1) tok/s | 35.2 s | | Gemma 4 12B 8-bit | No acceleration | 26.6 tok/s | 20.1 s | | | VLM MTP | 39.2 tok/s | 19.3 s | Each row is the median of two 512-token answers over the same code context. With this context the large model accepted fewer draft tokens than when writing code: on Qwen3.8-27B 4-bit, the VLM MTP gain fell from 1.43x to 1.07x. Built-in benchmark by prompt length Generation speed in tok/s by prompt length, with prefill at 4k and MLX peak memory, which includes the weights: | Model and configuration | 1k | 4k | 16k | 32k | Prefill 4k | Peak | | --- | --- | --- | --- | --- | --- | --- | | Qwen3.6-35B-A3B 8-bit, no acceleration | 92.1 | 91.1 | 84.1 | 77.9 | 2,426 | 38.7 GiB | | Qwen3.6-35B-A3B 8-bit, VLM MTP | 118.3 | 106.6 | 100.2 | 94.4 | 3,113 | 41.8 GiB | | Qwen3.6-35B-A3B 8-bit, DFlash (4-bit drafter) | 112.4 | 110.5 | 32.6 (23.3-35.0) | 16.6 | 2,734 | 38.2 GiB | | Qwen3.6-35B-A3B oQ4e-mtp, no acceleration | 49.3 | 46.1 | 45.2 | 42.2 | 2,258 | 23.2 GiB | | Qwen3.6-35B-A3B oQ4e-mtp, Lightning MTP | 77.2 (47.0-101.7) | 85.8 | 87.9 (85.7-116.6) | 82.8 (54.7-99.5) | 2,014 | 24.6 GiB | | Qwen3.6-35B-A3B oQ4e-mtp, DFlash (4-bit drafter) | 142.7 | 86.3 | 39.2 | 17.3 | 3,373 | 22.7 GiB | | Qwen3.8-27B 4-bit, no acceleration | 30.6 | 27.3 | 25.0 | 23.5 | 525 | 21.7 GiB | | Qwen3.8-27B 4-bit, VLM MTP | 39.3 | 36.6 | 36.4 | 30.9 | 619 | 25.4 GiB | | Qwen3.8-27B 4-bit, DFlash2 (4-bit drafter) | 48.2 (38.7-51.1) | 43.7 | 35.8 (22.0-39.0) | 6.3 | 561 | 22.8 GiB | | Qwen3.8-27B 4-bit, DFlash2 (bf16 drafter) | 36.9 | 35.1 | 22.3 | 5.9 | 568 | 25.3 GiB | | Qwen3.8-27B oQ4e-mtp, no acceleration | 27.0 | 26.4 | 24.4 | 21.4 | 545 | 22.2 GiB | | Qwen3.8-27B oQ4e-mtp, Lightning MTP | 48.0 (42.6-58.6) | 37.7 | 41.1 | 35.4 | 499 | 24.4 GiB | | Qwen3.8-27B oQ4e-mtp, Lightning MTP + TurboQuant 4-bit | 42.2 | 37.7 | 39.7 | 30.3 | 547 | 24.4 GiB | | Qwen3.8-27B oQ4e-mtp, Lightning MTP + ANE prefill | 35.6 | 33.1 | 35.8 | 30.9 | 505 | 28.3 GiB | | Qwen3.8-27B oQ4e-mtp, DFlash2 (4-bit drafter) | 41.4 | 33.8 | 18.8 (16.6-21.9) | 5.8 | 458 | 23.3 GiB | | Gemma 4 12B 8-bit, no acceleration | 31.0 | 30.5 | 30.1 | 28.8 | 955 | 14.5 GiB | | Gemma 4 12B 8-bit, VLM MTP | 65.6 | 42.7 | 50.6 | 49.0 | 940 | 15.5 GiB | | Gemma 4 12B 8-bit, DFlash (4-bit drafter) | 92.4 (35.5-158.0) | 34.4 | 53.4 | 51.2 | 997 | 18.1 GiB | | gpt-oss-20b, no acceleration | 120.2 | 106.5 | 100.4 | 78.8 | 3,491 | 12.4 GiB | DFlash figures with a 1k prompt varied widely between repeats: on Gemma 4 12B they were 35.5, 92.4 and 158.0 tok/s. Speed depends on how many draft tokens the large model accepts, and that changes with the text it has to continue. Compared with 0.3.8 The May table and this one are not fully comparable. That one was measured on the Qwen3.6-35B-A3B-MLX-8bit copy with 128 output tokens; this one, on the mlx-community copy with 256. Even so, the long-context improvement is too large to come from those differences: | Qwen3.6-35B-A3B 8-bit, built-in benchmark | 0.3.8 (May) | 0.6.4 (September) | | --- | --- | --- | | Decode with a 1k prompt | 72.4 tok/s | 92.1 tok/s | | Decode with a 4k prompt | 83.1 tok/s | 91.1 tok/s | | Decode with a 32k prompt | 15.2 tok/s | 77.9 tok/s | | First token with a 1k prompt | 577 ms | 298 ms | End-to-end verification A curl call to the OpenAI-compatible API to confirm the server is alive and a model responds: curl http://127.0.0.1:8000/v1/chat/completions \ -H "Authorization: Bearer your_api_key_here" \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-Coder-30B-A3B-Instruct-mlx-8bit", "messages": [{"role": "user", "content": "Hello from oMLX"}] }' And from Claude Code, once you have exported the variables from the previous section: claude --print "Are you running locally?" The reply should come back from local without touching api.anthropic.com. To confirm at the network level, open Activity Monitor → Network, filter by the claude process and check that the outbound connection is against 127.0.0.1:8000. !oMLX Chat view at localhost:8000/admin/chat with the model dropdown open, listing the four chat-completion models available locally (both Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus variants, Qwen3.5-9B-Claude-4.6-HighIQ, and Qwen3.6-40B-Claude-4.6-Opus-Deckard-Thinking); the embeddings and reranker models do not appear here because they are not chat-completion What has changed since 0.3.8 This guide was written against oMLX 0.3.8 in May 2026. The current version is 0.6.4, released on 29 August 2026, and more than a dozen releases in between touch exactly what is tuned here. What follows is the summary of what changed, with the figures attributed to whoever measured them. Lightning MTP speculative decoding This is the change that affects this configuration most. Version 0.5.0, from 10 July 2026, added native depth-k speculative decoding for Qwen3.6-27B, Qwen3.6-35B-A3B, DeepSeek-V4-Flash and GLM-5.2. On an Apple M3 Ultra the project measured Qwen3.6-35B-A3B going from 89.6 to 140.4 tok/s, a 1.57x gain. Qwen3.6-27B went from 35.0 to 55.1 tok/s. On this M5 Max, measured through the API on 15 September 2026, Lightning MTP took Qwen3.8-27B oQ4e from 21.1 to 39.7 tok/s. How to turn it on, and when another method is the better choice, is in the advanced settings. Custom kernels, with a note for M5 The same 0.5.0 added custom compute kernels for DeepSeek V4, Qwen3.5/3.6 and GLM-5.2, with prefill improvements measured on M3 Ultra. They are 45% on DeepSeek-V4-Flash (313.3 to 455.8 tok/s at 64k context), 33% on Qwen3.6-27B and 99% on GLM-5.2 at 32k. What matters for this machine is that dispatch is NAX-aware precisely to avoid regressions on the M5 series, so it is worth confirming the effect rather than assuming it. oQ and oQe quantisation oMLX no longer depends only on the weights you download: it brings its own quantisation scheme. Version 0.5.0 added oQe, an activation-importance calibration pass with expert-coverage tracking for MoE models. In the project’s measurements oQ4e improves average accuracy over oQ4 while staying in the same disk-size class. That is 83.88% against 82.83% on Qwen3.6-35B-A3B, and 73.09% against 70.01% on Qwen3.5-9B. The neural engine and cold starts The 0.6.3 branch worked on the prefill path over Apple’s neural engine. Two of the project’s figures matter day to day. Bank compilation memory dropped from 35.8 GB to 4.7 GB, and an optional persistent compile cache cuts fresh-process startup by 53% to 66%. Version 0.6.4 continued down that road and improved Qwen3.8-Flash-Next prompt processing by 33.5%, with 24.2% less total request time at 32k, again on M3 Ultra. New endpoints The OpenAI- and Anthropic-compatible pair has been joined by /v1/rerank for document reranking, /v1/responses for Codex compatibility and /v1/mcp/tools for Model Context Protocol. With an embedding model and a reranker loaded at once, a whole retrieval pipeline fits on the same server. The full breakdown is in the guide to the API, the key and the port. Sub-key authentication Where there used to be a list of equivalent keys, there are now three credentials with different powers. A main key opens the API and the panel; sub-keys only call the API and cannot log into the panel or change settings; panel session tokens are held in a signed cookie. If you are handing access to more than one application, hand out sub-keys. The setting names The memory configuration in the earlier section still holds as judgement, but the keys are named differently in ~/.omlx/settings.json. The memory ceiling is now memory_guard_tier, with four levels (safe, balanced, aggressive and custom), and the bespoke value goes in memory_guard_custom_ceiling_gb. By default the ceiling is system RAM minus 8 GB. The two cache tiers are hot_cache_max_size and ssd_cache_max_size, and the concurrency limit is max_concurrent_requests. Watch one of them: hot_cache_max_size ships as "0", which disables the hot cache in RAM. I go through it in benchmarks and the memory ceiling. Other pieces that did not exist Profiles save named bundles of settings over a single model, exposed as model:profile and at no extra memory cost. Aliases rename a model as far as the API is concerned. And there is experimental distributed inference, splitting a model’s layers across more than one Mac over SSH according to each machine’s memory, accounting for whether the link is Thunderbolt or 10GbE. The first two are in the dashboard and command line guide. Sources and further reading - Diego R. Baquero, "Running Claude Code with a local LLM": the tested configuration recipe this post’s settings are based on (TurboQuant 3.5-bit, 64k context, attribution off, Qwen 3.6 35B-A3B models). - A First Comprehensive Study of TurboQuant, vLLM blog, 11 May 2026: independent analysis confirming 3.5-bit matches full-precision quality. - omlx.ai: public leaderboard with real submissions from M5 Max and other machines. - Maxing Out M5 Max Context Windows: Memory Fragmentation and TurboQuant Benchmarks (dasroot.net, April 2026): the other detailed experiment on long-context M5 Max with TurboQuant. - Repositories: jundot/omlx, ml-explore/mlx, ml-explore/mlx-lm, huggingface.co, huggingface.co. If you came here wondering what this actually is and how it differs from Apple’s MLX framework, I separate the two in what is oMLX. For a cross-platform alternative outside Apple Silicon, the Ollama local LLM guide covers the model catalogue and OpenAI-compatible API. To wire the local endpoint into an agent, the Anthropic SDK tutorial and the MCP multi-vendor patterns already talk to this same format with no code changes. Related resources - Calculator: which LLM fits your GPU? - Learning path: Local LLMs Frequently asked questions Do I need to set an API key to use oMLX locally only? No. oMLX ships with the API key field empty, and as long as the instance only listens on 127.0.0.1 that is safe for local testing. The moment you expose it to the LAN or share the endpoint, go to Settings → Auth & Info and set a long random string. Current versions also offer sub-keys that can only call the API and cannot log into the panel or change settings, meant for handing access to more than one application. Can I use Claude Code from a MacBook against oMLX running on a Mac Studio? Yes, through SSH port forwarding: ssh -L 8000:localhost:8000 user@mac-studio makes 127.0.0.1:8000 on the laptop point at the Studio's oMLX. Then export the same variables (ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and the ANTHROPIC_DEFAULT_*_MODEL set) in a script such as claude-local.sh, make it executable with chmod +x and run claude --bare from the SSH session. If you set a key in Settings → Auth & Info, replace the empty string with it. Are the benchmarks in this post still valid if I install 0.6.4? The May tables, no: they were measured on 0.3.8, and when I re-ran them on 15 September 2026, 0.6.4 was faster on this same machine. Without acceleration, Qwen3.6-35B-A3B 8-bit goes from 15.2 to 77.9 tok/s with a 32k prompt. Lightning MTP or VLM MTP add between 1.07x and 1.88x, depending on the model and the context. If you measure on your own Mac, compare through the API and not only with Bench → Performance, which runs vision models on a different engine when MTP is off. --- Originally published on jacar.es, where the benchmarks are kept up to date with each oMLX release.
Qwen3.8 vs Ornith 1.5 on a Mac: speed and code review On a Mac Studio M4 Max with 64 GB, Qwen3.8-27B with MTP does not replace the 35B mixture-of-experts model I already run. Measured on 12 October 2026 under oMLX 0.7.0, Qwen generated 55 tokens per second against 90, and took 3.46 seconds to produce its first token on an 800-token prompt against 0.49. On a code review test the two caught the same bugs with reasoning off, and Qwen caught fewer with it on. The number that decides it is time to first token Qwen3.8-27B needs seven times longer than Ornith 1.5 to start answering an 800-token prompt: 3.46 seconds against 0.49, averaged over 8 requests per model. Output speed is closer, 55.1 tokens per second against 90.4. In a chat that gap is a nuisance. A coding agent or a code review sends long prompts many times in a row, and there the wait before the first token is most of the cost. oMLX is an inference server for Apple Silicon Macs that serves MLX models behind an OpenAI-compatible API. I run both models on it, and each one has its own trick for generating faster. Qwen3.8-27B is a dense model: all 27 billion parameters work on every token. It ships with MTP, multi-token prediction, an extra head trained with the model that drafts the next few tokens so the model can verify them in one pass. Ornith 1.5 35B-A3B is a mixture of experts: 35 billion parameters, about 3 billion active per token. I run it with DFlash, where a separate small draft model proposes a block of tokens and the main model checks them. The chart puts both measurements side by side. The first pair of bars is the one I care about. !Bar chart of two measurements on oMLX 0.7.0. Time to first token on an 800-token prompt: Ornith with DFlash 0.49 seconds, Qwen3.8-27B with MTP 3.46 seconds. Output speed: Ornith 90.4 tokens per second, Qwen 55.1. Ornith wins both. I measured with aiperf 0.12.0, one model loaded at a time, the server restarted before each run and no other traffic. This is the command. It works against any OpenAI-compatible server: aiperf profile --url http://localhost:8010 --model <model-id> \ --endpoint-type chat --streaming \ --concurrency 1 --request-count 8 --num-warmup-requests 2 \ --prompt-input-tokens-mean 800 --prompt-input-tokens-stddev 0 \ --output-tokens-mean 400 --output-tokens-stddev 0 \ --extra-inputs '{"temperature":0.6,"chat_template_kwargs":{"enable_thinking":false}}' \ --tokenizer <model-snapshot-dir> --random-seed 7 Upgrading oMLX helped Qwen by about 20%, and it was not enough Moving from oMLX 0.6.4 to 0.7.0 raised Qwen3.8-27B from 57-59 to 71.7 tokens per second on a code prompt and from 46-48 to 55-56 on prose, two runs per cell with 400 output tokens. The oMLX 0.7.0 release notes list faster MTP verify and decode for this model, and the numbers agree. Ornith stayed where it was: 103-121 tokens per second on code, 99-102 on prose. I also tried the obvious knob. MTP drafts up to 4 tokens per cycle by default on this model, and oMLX lets you raise the ceiling. At 6 the output speed was 53.9 tokens per second, at 8 it was 52.4, against 55.1 at the default. The adaptive controller already picks a depth that pays, so I put the setting back. On a code review test the two models tie With reasoning off, Qwen3.8-27B caught 9 of 20 planted bugs and Ornith 1.5 caught 8. With 20 cases, one bug of difference is noise. The test takes 20 real commits from four of my repositories, flips one operator on an added line (< to <=, == to !=, a dropped not) and asks the model to review the diff in a single call, without tools. Each case is paired with the clean diff, and a model only gets credit if it flags the planted line on the mutant and not on the clean version. | Model | Reasoning | Bugs caught | Unreadable answers | Findings per clean diff | Median time | |---|---|---|---|---|---| | Qwen3.8-27B + MTP | off | 9 of 20 (45%) | 4 of 40 | 0.7 | 10 s | | Ornith 1.5 + DFlash | off | 8 of 20 (40%) | 2 of 40 | 1.6 | 3 s | | Qwen3.8-27B + MTP | on | 6 of 20 (30%) | 12 of 40 | 0.3 | 194 s | | Ornith 1.5 + DFlash | on | 8 of 20 (40%) | 13 of 40 | 0.0 | 102 s | The Ornith rows were measured on 2 October 2026 on oMLX 0.6.4, the Qwen rows on 12 October on 0.7.0. With reasoning off, where they differ is false alarms. Qwen reports half as many problems on code that has none, and it takes three times as long per review. Turning reasoning on does not help Qwen. It caught 6 of 20 against 9 without it, and half of its 40 answers used up the whole 12,000-token output limit, which is why 12 of them never reached valid JSON. Ornith overthinks in the same way and loses 13 of 40 answers, but it holds at 8 of 20 and takes about half the time per review. A review that needs three minutes and fails to parse a third of the time is no use to me from either model. The two models do not fit in memory together Qwen3.8-27B takes 16.1 GB once loaded and Ornith 1.5 takes 19.8 GB. On a 64 GB machine that sounds fine, and under oMLX 0.6.4 both did load, with swap climbing to 15.3 of 16 GB. The memory guard in 0.7.0 refuses instead. While Qwen was busy, every request for Ornith got this: POST /v1/responses → 507: Cannot load Ornith-1.5-35B-A3B-MLX-4bit: projected memory 38.24GB would exceed the dynamic memory ceiling 34.51GB When Qwen is idle, oMLX evicts one model to load the other, and the log marks the unload as an immediate abort, so a request in flight on the evicted model is lost. I found out what that looks like under load by accident. I ran the review test on Qwen with reasoning on for 90 minutes while another tool kept sending requests to Ornith. oMLX loaded Ornith, evicted it six seconds later to make room for Qwen's prompt, loaded it again, and eventually evicted Qwen in the middle of an answer. Of Qwen's 40 requests, 19 failed. I had hoped to keep Qwen beside Ornith as a second opinion, and that is off the table. I have to pick one, and on these numbers I pick Ornith. Qwen sometimes cuts its own JSON, and the fix costs half the speed In 5 of 40 review answers Qwen3.8-27B returned {"findings":[] and stopped, with the closing brace missing. It only happened on empty answers of five or six tokens. I reran the five affected diffs three times each under four settings, with MTP on and off and with the presence penalty at 1.5 and 0. The cut answers appeared with MTP off too, and without the penalty, in up to 4 of 15 runs. As far as I can tell the model just ends early now and then. Constrained decoding removes it. oMLX has it built in through xgrammar, but the Homebrew formula leaves it out unless you ask: brew reinstall jundot/omlx/omlx --with-grammar With a JSON schema in the request, 0 of 30 answers were unparseable. The diagram shows what that costs. A grammar and MTP cannot run together, so those requests fall back to plain decoding at about 25 tokens per second. !Flow diagram of a review request to Qwen3.8-27B. Without a JSON schema it uses MTP decoding at about 55 tokens per second and up to 4 of 15 answers are unparseable. With a JSON schema it uses grammar-constrained decoding with MTP switched off at about 25 tokens per second and 0 of 30 answers are unparseable. What is still open Qwen's row with reasoning on mixes two runs. 21 answers come from the morning run described above, with other traffic on the server, and I repeated the 19 failed requests in the afternoon with everything else paused. I counted the morning answers because they were complete, but I did not rerun all 40 in one clean pass. The speed test used 8 requests and one prompt size. I did not test Qwen's vision, and I do not know what a JSON schema costs Ornith with DFlash. I removed Qwen from the proxy that evening and deleted its weights, so none of that will be measured. The next candidate is Muse Glimmer 30B, which is being measured now. For interactive work, where you wait on every answer, Ornith stays. If you are choosing a local model for a Mac, measure time to first token at the prompt length you really send before you look at tokens per second. For unattended jobs that gap matters much less.
oMLX v0.7 Splash PR benchmark on M3U 512 Studio & M4M 128 Studio For those interested. Benchmarks using the oMLX Splash PR Models are Q4. M4M includes comparison with oQ4e MTP equivalent models. The M4M Studio is my desktop and has a background gpu load with other apps etc. The M3U Studio runs as a headless server so the GPU is just for oMLX.