The best Arabic LLM for your enterprise is the one that passes your eval, not the leaderboard
Four serious Arabic models now come out of the Gulf: Falcon-H1-Arabic from Abu Dhabi, ALLaM from Saudi Arabia, Fanar 2 from Qatar and Jais 2 from the UAE. None of them can be picked from a leaderboard, because the Open Arabic LLM Leaderboard measures academic Arabic, not your tickets, your policy documents or your customers' dialect. The two things that do transfer to your organisation are both measurable in about a week: a 200 to 500 item eval set built from your own material, and the tokenization economics that decide what each Arabic conversation really costs in money, latency and GPU memory.
If you are building Arabic-language customer service, a government services assistant or an internal knowledge bot in the GCC, you face a decision that looks like a model choice but is actually two measurements. First: does a regional open-weight model score within your tolerance on your content, with your product names, place names, dialects and policy language? Second: what does an Arabic token actually cost you, in API spend, time to first token, KV-cache footprint and concurrent sessions per GPU? Answer both and the decision makes itself. Skip them and you will either overpay a frontier API for years or ship a model that fails quietly on Gulf dialect.
The field in late 2026
Four model families are credible candidates for enterprise Arabic work today. All of them are open weight unless noted, which means you can run them on your own GPUs and keep data in-country.
| Model | Developer | Sizes | Context | Licence | What the evidence says |
| --- | --- | --- | --- | --- | --- |
| Falcon-H1-Arabic | TII, Abu Dhabi | 3B, 7B, 34B | 128K (3B), 256K (7B, 34B) | Open weights (check the model card for your size) | About 71.5% (7B) and 75.4% (34B) on the Open Arabic LLM Leaderboard, beating 70B-class transformers including Llama-3.3-70B and Qwen2.5-72B. Hybrid Mamba-Transformer architecture, runs on vLLM. |
| ALLaM | SDAIA, now run by HUMAIN, Saudi Arabia | 7B open checkpoint; 34B flagship powers HUMAIN Chat | 4,096 (7B) | Apache 2.0 (7B Instruct preview) | The 7B Instruct preview is trained from scratch on 4T English tokens plus 1.2T mixed Arabic/English, with two published revisions (v1.27.2.25 and v2). ALLaM 34B backs HUMAIN Chat, hosted end to end in the Kingdom and PDPL-compliant. |
| Fanar-2-27B-Instruct | QCRI, Qatar | 27B | 32,768 | Apache 2.0 | Continually pretrained from gemma-3-27b-pt on roughly 166B Arabic and English tokens over 75,000 H100 hours. Explicit MSA plus Gulf, Levantine and Egyptian dialect support, native Arabic reasoning traces, tool calling and hallucination mitigation. |
| Jais 2 | Inception (G42), Cerebras and MBZUAI, UAE | 8B, 70B | 8,192 | Apache 2.0 | Trained from scratch on 600B+ curated Arabic tokens plus 1.6T of web, math and code, with a custom 150,272-entry Arabic-centric vocabulary. Leading results among evaluated open models on OALL2 and AraGen. Note: the hosted API was retired on 21 August 2026; weights remain on Hugging Face. |
Primary sources: the Falcon-H1-Arabic announcement and release blog, the Fanar-2-27B-Instruct model card, the ALLaM-7B-Instruct-preview model card, the HUMAIN Chat launch and the Jais 2 documentation.
The fourth candidate in every real procurement is a frontier API model prompted in Arabic. It belongs in the comparison, because on raw Arabic quality it often wins. The question is by how much, and what that gap costs.
Why the leaderboard cannot pick your model
The Open Arabic LLM Leaderboard is genuinely good work. It runs reproducible evaluations through the EleutherAI lm-evaluation-harness Arabic task set, so a score is a score, not a vendor claim. TII even documents that its own vLLM-based runs land within about one point of the leaderboard's numbers.
But read what it measures: knowledge, reading comprehension and reasoning over web-derived Arabic text, mostly Modern Standard Arabic. Your workload is not that. It is a customer writing "ما أقدر أرسّع الفاتورة من التطبيق" (I can't pay the bill faster from the app, in Gulf dialect), a code-switched complaint ("الـ bundle خلص، كيف أعمل top-up؟"), and questions about a product name that appears nowhere in any pretraining corpus. Leaderboard rank measures none of the things that will make your assistant succeed or embarrass you.
This is not an argument against the leaderboard. It is the screening stage: it tells you which models are worth a week of your team's time. The final stage happens on your data.
What transfers: build a 300-item eval set in three days
The single highest-leverage artefact in an Arabic LLM project is a small, versioned eval set built from your own material. Two hundred items is a floor, five hundred is plenty; beyond that the marginal information per item collapses.
Where the items come from:
1. Real tickets and chat logs (redacted). Take them verbatim, dialect and typos included. Never rewrite them into MSA, because your users will not.
2. Policy and product documents. Turn paragraphs into questions: "What is the fee for an early exit from this plan?"
3. Named entities. Product names, service tiers, districts, mall names, government portals. Models hallucinate most confidently where pretraining data is thinnest, and nothing is thinner than your internal naming scheme.
4. Deliberately hard slices. Aim for roughly 40% formal MSA, 30% Gulf dialect, 30% code-switched Arabic/English, in the proportions your traffic actually shows.
A concrete mix for a 300-item set, drawn from a telecom support case:
- 90 items from the last two years of Arabic chat logs, sampled by intent cluster (billing, top-up, coverage, cancellation), including Emirati and Saudi dialect items
- 60 questions whose answers live only in your policy PDFs
- 60 entity-grounded questions where the correct answer must quote your exact product or tariff name
- 45 dialect and code-switched items, verbatim from logs, never normalised
- 45 adversarial items: questions about services you do not offer, questions that mix a real product name with a wrong assumption, and date and amount arithmetic from a bill
That is one analyst-day to sample, one subject-matter-expert day to write gold answers, one more to review. Gold answers should be short and checkable, not essays.
The scoring sheet and the judge
Score each response on five criteria, 0 to 2 each, so the maximum is 10 per item and a 300-item set yields a 3,000-point scale:
| Criterion | 0 | 1 | 2 |
| --- | --- | --- | --- |
| Correctness | Wrong answer | Partially right or right with wrong entity | Fully right, correct entity and amount |
| Grounding | Invents a policy, product or price | Vague, not traceable to source | Every claim traceable to your documents |
| Dialect handling | Misunderstands dialect or code-switching | Understands but answers in stiff MSA | Registers the register, answers naturally |
| Format and language | Wrong language, broken Arabic | Usable but unnatural | Natural, correctly punctuated Arabic |
| Completeness | Misses a step or an explicit caveat | Usable | Complete, including edge cases |
Two ways to score: a human reviewer (expensive, about 1 to 2 minutes per item, so a full pass on 300 items is a day) or a stronger model as judge. If you use a model as judge, check agreement first: have the human score a 50-item overlap sample and compute percent agreement on the Correctness criterion. Below roughly 85% agreement, your judge is not measuring your rubric, and you must fix the prompt before trusting it at scale.
Here is a copy-paste judge prompt that has the properties that matter: it forces evidence, it forbids generosity, and it outputs a parseable score.
You are evaluating a customer-service assistant's reply in Arabic.
QUESTION (verbatim from a real user, may contain Gulf dialect or
Arabic/English code-switching):
{question}
GOLD ANSWER (written by a domain expert):
{gold_answer}
ASSISTANT'S REPLY:
{reply}
Score each criterion 0, 1 or 2. Be strict: a score of 2 must be
clearly earned, not merely plausible.
1. correctness: 2 = matches the gold answer's facts, entities and
amounts; 1 = partially right; 0 = wrong.
2. grounding: 2 = every claim is traceable to the gold answer or the
supplied policy text; 1 = vague; 0 = invents facts.
3. dialect: 2 = correctly understands the question's register and
replies naturally in it; 1 = understands but replies in stiff MSA;
0 = misunderstands the dialect or code-switching.
4. language: 2 = natural, correct Arabic; 1 = awkward but usable;
0 = broken Arabic or wrong language.
5. completeness: 2 = covers steps and caveats; 1 = usable with gaps;
0 = misses something essential.
Reply ONLY with JSON:
{"correctness": n, "grounding": n, "dialect": n, "language": n,
"completeness": n, "total": n,
"fatal_error": "<one line, or 'none'>"}
Run every candidate model, including the frontier API baseline and its Arabic-prompted variant, on the identical set with the identical retrieval context if you use RAG. Report the mean total, the per-criterion breakdown, and the fatal-error rate. The per-criterion breakdown matters more than the total: a model that loses two points per item entirely on dialect handling might be fixable with prompt work, while one that loses them on grounding needs retrieval surgery, and one that loses them on correctness needs to be dropped.
For dialect specifically, published per-dialect evaluations such as DialectalArabicMMLU show how much scores move between MSA and regional varieties, and TII reports AraDice dialect sub-scores for the Falcon-H1-Arabic family. Your own dialect slice settles the question for your dialect, which is the only one that matters.
The second thing that transfers: tokenization economics
Arabic text costs more tokens than the equivalent English text on English-centric tokenizers, and that premium appears in four places at once: your API bill, time to first token, KV-cache memory and the number of concurrent Arabic sessions a GPU can hold.
The peer-reviewed evidence is blunt. Petrov and colleagues showed that the GPT-4 tokenizer encodes the same content in Standard Arabic with roughly 3.04 times as many tokens as English, and the ChatGPT tokenizer with about 4.40 times as many, with direct consequences for cost, latency and effective context length (arXiv:2305.15425). The original Jais team measured the same pattern at the word level: GPT-2's vocabulary fragments Arabic into about 4.17 tokens per Arabic word against roughly 1.1 per English word, while the bilingual Jais vocabulary achieves about 1.05 (fertility figures summarised here). Newer multilingual vocabularies have narrowed the gap, but a 1.5 to 2 times premium on an Arabic-inefficient tokenizer is still normal, and nothing on your invoice labels it.
Measure it on your own traffic before believing any ratio. Ten lines of Python gives you fertility, tokens per word, on a sample of your real conversations:
from transformers import AutoTokenizer
text = open("sample_arabic_conversations.txt", encoding="utf-8").read()
words = len(text.split())
# Verified repo IDs; add the other candidates from their model cards
for repo in ["QCRI/Fanar-2-27B-Instruct",
"humain-ai/ALLaM-7B-Instruct-preview"]:
tok = AutoTokenizer.from_pretrained(repo)
fertility = len(tok(text).input_ids) / words
print(f"{repo}: {fertility:.2f} tokens per Arabic word")
One caution: studies count words and normalise text differently, so compare ratios within your own sample rather than against published absolutes. Your sample is the number that will hit your bill.
Worked example: what the premium actually costs
Take a 400-word Arabic support answer and compare a model whose tokenizer gives 2.4 tokens per Arabic word against an Arabic-optimised model at 1.5. Same content, same job.
Spend. The answer is 960 tokens on the first tokenizer, 600 on the second, a 1.6 times premium on every request. At an illustrative $5 per million output tokens, one million such answers costs $4,800 versus $3,000, a $1,800 difference before you have written a word of prompt engineering. Multiply by your volume. And note the trap when comparing providers: a model priced 30% lower per token that fragments Arabic 1.6 times more is not 30% cheaper for you, it is roughly 12% more expensive.
Latency. Prefill is compute-bound on prompt length, so time to first token scales with the token count of your system prompt plus retrieved context. Decode is per token, so an answer fragmented into 1.6 times the tokens streams 1.6 times longer at the same hardware speed. Prompt tuning cannot remove this; it is in the vocabulary.
KV cache and concurrency. Per-token KV-cache size for a 7B-class model with 32 layers, 8 grouped-query KV heads and 128-dimension heads at fp16 is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB per token. A 4,000-token Arabic conversation at the efficient tokenizer holds about 500 MB of KV per session; the same content at 1.6 times fertility holds about 800 MB. On a GPU with 60 GB free after model weights, that is roughly 120 concurrent sessions versus 75. Tokenization, not parameter count, decided your concurrency.
Context. A 256K context window (Falcon-H1-Arabic 7B and 34B) holds far more Arabic at a low fertility than a longer window holds at a high one. This is why the Jais 2 team built a custom 150,272-entry Arabic-centric vocabulary and report it makes inference efficient: the efficiency is not marketing, it is arithmetic you can reproduce with the snippet above.
The decision rule
Now the choice reduces to one comparison on one spreadsheet.
1. Run all candidates on your eval set. Suppose the frontier API scores 9.1 out of 10 and Falcon-H1-Arabic 7B scores 8.6. The gap is 0.5 points.
2. Set your tolerance in advance, in the currency of your use case: acceptable fatal-error rate on billing questions, acceptable dialect-mishandling rate. Do not set it after seeing the scores.
3. If the open model is inside your tolerance: run it on your own GPUs. You keep every conversation in-country, you pay a fixed infrastructure cost instead of a token bill that scales with Arabic verbosity, and you are not exposed to an API being retired (Jais 2's hosted endpoint went away in August 2026; its weights did not). A regional open model that scores within tolerance on your data is strictly the better enterprise choice: cheaper per request, in-country data, and no dependency on someone else's pricing page.
4. If the gap exceeds your tolerance: do not pay for it blindly. Quantify the frontier model's advantage in cost per resolved request: with the 1.6 times tokenization premium on an English-centric tokenizer and your ticket volume, what is the API bill for the extra 0.5 points? Then price the alternative: the same eval gap might close with a better Arabic system prompt, stronger retrieval into your policy documents, or three days of fine-tuning the open model on 2,000 of your own tickets. Very often the fine-tune is cheaper in month four than the API premium is in month one.
A practical note on keeping this sustainable: whatever prompts and gold answers survive the eval, version them. The Arabic system prompts that work, the dialect-handling instructions, the few-shot examples, these are assets, and they are exactly the kind of asset that walks out the door in a Markdown file. We keep them in a shared, versioned prompt library (PromptForge), and when we build and run these assistants for clients, from eval set through deployment, that is our AI engineering work, done by our own team here in Dubai.
What to do next
1. This week: pull 300 real items using the mix above. It is three days of one analyst and one domain expert, and it is reusable for every model you will ever evaluate.
2. In parallel: run the fertility snippet on 100 of your real conversations for every shortlisted model, including the frontier API's tokenizer. You now have the two numbers that transfer.
3. Next week: run all candidates through the judge prompt, with a human agreement check on 50 overlapping items. Build the table: eval score, fatal-error rate, tokens per request, cost per resolved request.
4. Then decide with the rule above. If the regional open model is inside tolerance, it wins on data residency and unit economics, and GCC regulators and your security team will thank you. If it is not, you now know exactly what the gap costs per month, which turns a vague preference into a negotiation.
The best Arabic LLM for enterprise is not a fixed answer, because "enterprise" is not a fixed workload. It is whichever of Falcon-H1-Arabic, ALLaM, Fanar 2, Jais 2 or a frontier API passes an eval set built from your own tickets, at a token cost you have measured rather than assumed. Both measurements take a week. The alternative is guessing, and guessing is the most expensive part of any AI project.
---
Originally published on Azrty.
How Jev decision models work (and how to build your own on MLX)
The web is bloated with Jev posts. The thing is a few weeks old, and my feeds are already full of it.. benchmarks, hot takes, threads about decision models being the future of agents. Jev, if you've managed to avoid the hype, is TypeSafe AI's hosted decision API where you send a situation and a few typed questions, it returns the answer with a probability for every option.
Two spoilers, because I like roads with signposts. Classical ML never left the chat, and the newest tool in the box still doesn't answer the oldest question, whether you actually need it in the first place.
---
At some point I stopped skimming news and got really curious about the machinery underneath. How does a model decide instead of write? What is actually different in there?
The short answer turned into a project, and the project turned into a story about building a thing and then discovering, layer by layer (sorry), that most of it already existed. It's called take-my-layer.
---
Init
I'll be honest, I never thought I'd invented something. I knew reading probabilities off logits was a thing and the Jev posts themselves hint at it. But knowing a trick exists and holding it in your own hands are different things, and what I didn't know was how far it stretches. I took a 4-bit Qwen model on a laptop, no training, no labels, questions made up on the spot. That part was worth testing.
So I went scouting before building anything serious. Kind of following my own advice from a post about fog of war as a mental model. The fog doesn't mean the territory is missing, it just means you haven't been there yet. Reading papers is scouting. Building a small proof of concept is scouting. I expected to find maybe one or two adjacent things.
Well.. the territory I found was huuuge :D
---
The trick is from 2021
First stop. Reading answer probabilities instead of generating answer text. This is how the whole evaluation world has scored decoder-only models for years.. it's the loglikelihood default in lm-evaluation-harness. The prompting version has names (verbalizers, pattern-verbalizer pairs, cloze-style prompting). PET (Schick & Schütze, EACL 2021) and "Small Language Models Are Also Few-Shot Learners" (NAACL 2021) are the classics, LM-BFF (ACL 2021) even automates the label-word search.
What I'd been calling "reading the logits" has a standard name, a literature, a codified abstraction, and several thousand citations. The trick predates ChatGPT (obvious for people who are not the first day on the ML street).
---
The bugs are from 2021 too
The naive version has a bug called position bias. Ask "A, B, C or D?" and the model leans toward whichever option is listed first, regardless of the options. I found that myself the fun way when the raw read on my coding data predicted bash for 74 of 120 states. A bash magnet kind of. Its 0.57 accuracy was roughly bash's base rate in the data.
Essentially every bug I hit someone had already hit:
- Calibrate Before Use (Zhao et al., ICML 2021): content-free priors to correct the model's answer skew.
- Order sensitivity of option lists (Pezeshkpour & Hruschka, 2023).
- Fantastically Ordered Prompts (Lu et al., ACL 2022): same disease, few-shot demonstrations.
- PriDe (ICLR 2024): the cheap permutation-plus-prior combination, which is more or less the cheaper cousin of my scheme.
- Batch Calibration (ICLR 2024): the unified view of all of the above.
My own fix (arrived at independently and with more cursing), was to list the options in several rotated orders, ask once per order, average. That's called permutation marginalization, and it is worth +4.7 points of agreement with the reference over the naive single-prompt read (5 answers flip toward the reference, 0 away). All of that was known. Standing on shoulders of giants, etc.
---
Leaving early
The next piece was speed. A full forward pass over a 4B model for a one-word answer is wasteful, and it turns out the decision signal sits pretty early. (Qwen3.5-4B has 32 blocks; layer selection kept picking 16 and 19, before the last full-attention blocks. The interesting states live in the linear-attention trunk.)
The literature here is CALM (NeurIPS 2022), early exit per token by confidence. In my case it's a fixed half-depth exit for one fitted head, the layer-pruning family. Fine again, keep standing on the shoulders!
In MLX land there is exactly one static precedent that I could find: NeoJev, a 27B 4-bit (~15 GB) that keeps 56 of 64 layers with a self-distilled head, 0.727 mean accuracy at 310 ms p50 on an M5 Pro. One detail I genuinely enjoyed is that a plain logit lens at their layer 56 reads 0.663. Same head-over-raw gap I measured on my own side, so kind of proving I'm moving in the right direction.
A practical note for anyone following this route: mlx-lm exposes no intermediate-layer hooks (issue #1053, still open as of date of this blog post), so a hand-rolled block loop is the only way in. Mine matches the stock forward to a max logit difference of about 0.037.
---
A warning from the probing people
Fitting a tiny linear head on hidden states is also a literature: called probing. Hewitt & Liang (2019) and Belinkov's survey (2022) are the standard references, and their criticism is aimed at claims. A probe is weak evidence about what a model "understands". I took that as a warning about wording but not about architecture. A production head is judged by held-out task accuracy on the served artifact (ours reads 0.826 on its holdout), not by what it supposedly reveals. Linear (as the probing people say) is the defensible capacity choice.
---
And finally products
The method hunt was humbling, in that almost everything was already there before. The product hunt was even more humbling.
Jev you've already met. Cloudflare's Clef ships fine-tuned decision models with open Apache-2.0 weights (209 ms median, 38.8 ms for the flash variant). Unsloth has a train-your-own recipe: LoRA plus a Clef-design head, their 0.8B reads 36% on typed-decisions before training and 73% after. Around those there's a whole cluster: Mini-Jev, mlx-decision, laya-mlx. Even the routing people are adjacent, though RouteLLM and friends put a separate model in front of the LLM, whereas I wanted the served model's own distribution to be the classifier.
(!) And then the one that mattered to me: AnyJev, Nokia Applied Research's open method and library (technical report). Essentially I've taken AnyJev's method, deployed on the Mac.
---
What's actually mine
An honest ledger, because I care about this part. The repo's docs have a section literally titled "honest lineage", and it says explicitly that I did not invent the trick. What's original though:
- The missing MLX backend for AnyJev's toolbox (an original implementation of their documented backend contract; they never shipped one).
- The small-memory end: a 4-bit 4B, about 3 GB resident, versus NeoJev's 15.
- Zero-training deployment: the frozen model works the moment it downloads. No dataset, no training run, no labels.
- The observe() loop: while a shadow proxy watches real agent traffic, it quietly banks "what I predicted" next to "what actually happened", and one command refits a head from that bank. As far as I could find, nobody ships fit-on-your-traffic heads.
- And the measurements of course.
None of that is a new mechanism. It's just glue and a scorecard. BTW! I wrote about taste a while ago and there (TLDR) I say that when production becomes abundant, judgment becomes scarce. The judgment here was mostly deciding what not to claim.
---
Benchmarks
The repo holds itself to wording rules that I want to keep in this post too: fidelity claims say "aligned with the reference", never "more correct"; accuracy claims name the arena and the model size. So, with that:
| what was measured | result |
|---|---|
| Agreement with the real Jev API, 300 non-coding questions | 4B 75.7%, 9B 82.0%; Jev agrees with itself 98.7%, which is the ceiling |
| Readout vs greedy decoding of the same model, same 300 questions | +4.7 (4B), +6.3 (9B), McNemar p = 0.003; on long inputs greedy Choice answers collapse (46% vs 85% agreement) |
| BANKING77, 10 intents, gold labels | 87.5% (4B), 91.25% (9B), Jev 95.5%: a trained model wins on its home turf |
| typed-decisions, zero training | 61.6% (4B), 60.4% (9B), vs 73-78% for models fine-tuned on it |
| Latency, long input, 5 questions in one request | 22.2 s down to 3.7 s mean (the state is read once); the coding head answers in ~0.2 s |
The losses are the point, in a way. A trained model beats a zero-training readout whenever labels and a fixed label set exist. Train a classifier! This is for questions and options you cannot enumerate ahead of time, on a machine where the data cannot leave.
Here's one real exchange on my M5:
curl -s localhost:8095/v1/systemone -d '{"state": "Ticket #4821 from Dana (Pro plan, customer for 14 months): \"Since this morning every export to CSV fails with a 502. We run payroll from those exports at 3pm today.\" Status page: all green.",
"questions": {
"route": {"type": "Choice", "instructions": "Which team should take this first?", "options": ["billing", "platform-oncall", "account-management", "docs"]},
"escalate_now": {"type": "Noul", "instructions": "Does this need the on-call engineer right now?"},
"urgency": {"type": "Score", "instructions": "How urgent is this ticket?", "levels": ["low", "medium", "high", "critical"]}}}'
-> {"model": "mlx-community/Qwen3.5-4B-MLX-4bit",
"answers": {
"route": {"answer": "platform-oncall", "confidence": 0.8516, "probabilities": {"billing": 0.0744, "platform-oncall": 0.8516, "account-management": 0.0579, "docs": 0.0161}},
"escalate_now": {"answer": true, "confidence": 0.637, "probabilities": {"Yes": 0.637, "No": 0.363}},
"urgency": {"answer": "critical", "confidence": 0.6651, "probabilities": {"low": 0.0051, "medium": 0.0156, "high": 0.3142, "critical": 0.6651}}}}
> Status: research prototype, and the docs are blunt about it. The Jev-agreement numbers use LLM-authored questions with no human labels. The one fitted head that ships cannot be reproduced from the repo, because its training seed is withheld. If you have a fixed label set and training data, train a classifier instead. All of it is written down in a LIMITATIONS.md.
---
The boring lesson
This field is moving embarrassingly fast.
But the lesson I'd actually keep from these weeks is a boring one: classical ML never left. Every number in the table above says so. Wherever a fixed label set and training data exist, something trained the old way wins. Jev's 95.5% on BANKING77 against my 87.5%, the fine-tuned 2B sitting 16 points above my zero-training readout on typed-decisions. Reading logits doesn't replace training. It covers the corner cases training can't reach e.g. questions you can't enumerate ahead of time, no labels to collect, data that isn't allowed to leave the machine etc.
So before throwing this at a task, the checklist is the one classical ML wrote decades ago. Fixed labels and enough data? Train a classifier, it's still unbeatable there. Data that can't leave the machine, or questions that change per request? That's where reading the model's own probabilities earns its keep. In between, renting a hosted API is probably a fine deal.
---
Oh right, the question
Ah, I actually never answered the "how does a model decide instead of write" part from the beginning. Short version, and experienced ML people will roll their eyes at how obvious it is: the decision seems to form well before the end of the network. My tiny head reads block 16 of 32 and still answers the "what next" question at 0.826 accuracy. AnyJev's own measurements found a middle block to be a better feature space than the final one, since the last blocks are busy converting the answer into token space. And NeoJev keeps 56 of 64 layers, which suggests there is slack even higher up. So roughly: the deciding happens in the first half to two thirds of the layers, and the rest is more like the writing department.
Am I even right here? Probably, for these models, by my measurements. That's as far as I'd push it.
That's it.
Mentions
The best Arabic LLM for your enterprise is the one that passes your eval, not the leaderboard Four serious Arabic models now come out of the Gulf: Falcon-H1-Arabic from Abu Dhabi, ALLaM from Saudi Arabia, Fanar 2 from Qatar and Jais 2 from the UAE. None of them can be picked from a leaderboard, because the Open Arabic LLM Leaderboard measures academic Arabic, not your tickets, your policy documents or your customers' dialect. The two things that do transfer to your organisation are both measurable in about a week: a 200 to 500 item eval set built from your own material, and the tokenization economics that decide what each Arabic conversation really costs in money, latency and GPU memory. If you are building Arabic-language customer service, a government services assistant or an internal knowledge bot in the GCC, you face a decision that looks like a model choice but is actually two measurements. First: does a regional open-weight model score within your tolerance on your content, with your product names, place names, dialects and policy language? Second: what does an Arabic token actually cost you, in API spend, time to first token, KV-cache footprint and concurrent sessions per GPU? Answer both and the decision makes itself. Skip them and you will either overpay a frontier API for years or ship a model that fails quietly on Gulf dialect. The field in late 2026 Four model families are credible candidates for enterprise Arabic work today. All of them are open weight unless noted, which means you can run them on your own GPUs and keep data in-country. | Model | Developer | Sizes | Context | Licence | What the evidence says | | --- | --- | --- | --- | --- | --- | | Falcon-H1-Arabic | TII, Abu Dhabi | 3B, 7B, 34B | 128K (3B), 256K (7B, 34B) | Open weights (check the model card for your size) | About 71.5% (7B) and 75.4% (34B) on the Open Arabic LLM Leaderboard, beating 70B-class transformers including Llama-3.3-70B and Qwen2.5-72B. Hybrid Mamba-Transformer architecture, runs on vLLM. | | ALLaM | SDAIA, now run by HUMAIN, Saudi Arabia | 7B open checkpoint; 34B flagship powers HUMAIN Chat | 4,096 (7B) | Apache 2.0 (7B Instruct preview) | The 7B Instruct preview is trained from scratch on 4T English tokens plus 1.2T mixed Arabic/English, with two published revisions (v1.27.2.25 and v2). ALLaM 34B backs HUMAIN Chat, hosted end to end in the Kingdom and PDPL-compliant. | | Fanar-2-27B-Instruct | QCRI, Qatar | 27B | 32,768 | Apache 2.0 | Continually pretrained from gemma-3-27b-pt on roughly 166B Arabic and English tokens over 75,000 H100 hours. Explicit MSA plus Gulf, Levantine and Egyptian dialect support, native Arabic reasoning traces, tool calling and hallucination mitigation. | | Jais 2 | Inception (G42), Cerebras and MBZUAI, UAE | 8B, 70B | 8,192 | Apache 2.0 | Trained from scratch on 600B+ curated Arabic tokens plus 1.6T of web, math and code, with a custom 150,272-entry Arabic-centric vocabulary. Leading results among evaluated open models on OALL2 and AraGen. Note: the hosted API was retired on 21 August 2026; weights remain on Hugging Face. | Primary sources: the Falcon-H1-Arabic announcement and release blog, the Fanar-2-27B-Instruct model card, the ALLaM-7B-Instruct-preview model card, the HUMAIN Chat launch and the Jais 2 documentation. The fourth candidate in every real procurement is a frontier API model prompted in Arabic. It belongs in the comparison, because on raw Arabic quality it often wins. The question is by how much, and what that gap costs. Why the leaderboard cannot pick your model The Open Arabic LLM Leaderboard is genuinely good work. It runs reproducible evaluations through the EleutherAI lm-evaluation-harness Arabic task set, so a score is a score, not a vendor claim. TII even documents that its own vLLM-based runs land within about one point of the leaderboard's numbers. But read what it measures: knowledge, reading comprehension and reasoning over web-derived Arabic text, mostly Modern Standard Arabic. Your workload is not that. It is a customer writing "ما أقدر أرسّع الفاتورة من التطبيق" (I can't pay the bill faster from the app, in Gulf dialect), a code-switched complaint ("الـ bundle خلص، كيف أعمل top-up؟"), and questions about a product name that appears nowhere in any pretraining corpus. Leaderboard rank measures none of the things that will make your assistant succeed or embarrass you. This is not an argument against the leaderboard. It is the screening stage: it tells you which models are worth a week of your team's time. The final stage happens on your data. What transfers: build a 300-item eval set in three days The single highest-leverage artefact in an Arabic LLM project is a small, versioned eval set built from your own material. Two hundred items is a floor, five hundred is plenty; beyond that the marginal information per item collapses. Where the items come from: 1. Real tickets and chat logs (redacted). Take them verbatim, dialect and typos included. Never rewrite them into MSA, because your users will not. 2. Policy and product documents. Turn paragraphs into questions: "What is the fee for an early exit from this plan?" 3. Named entities. Product names, service tiers, districts, mall names, government portals. Models hallucinate most confidently where pretraining data is thinnest, and nothing is thinner than your internal naming scheme. 4. Deliberately hard slices. Aim for roughly 40% formal MSA, 30% Gulf dialect, 30% code-switched Arabic/English, in the proportions your traffic actually shows. A concrete mix for a 300-item set, drawn from a telecom support case: - 90 items from the last two years of Arabic chat logs, sampled by intent cluster (billing, top-up, coverage, cancellation), including Emirati and Saudi dialect items - 60 questions whose answers live only in your policy PDFs - 60 entity-grounded questions where the correct answer must quote your exact product or tariff name - 45 dialect and code-switched items, verbatim from logs, never normalised - 45 adversarial items: questions about services you do not offer, questions that mix a real product name with a wrong assumption, and date and amount arithmetic from a bill That is one analyst-day to sample, one subject-matter-expert day to write gold answers, one more to review. Gold answers should be short and checkable, not essays. The scoring sheet and the judge Score each response on five criteria, 0 to 2 each, so the maximum is 10 per item and a 300-item set yields a 3,000-point scale: | Criterion | 0 | 1 | 2 | | --- | --- | --- | --- | | Correctness | Wrong answer | Partially right or right with wrong entity | Fully right, correct entity and amount | | Grounding | Invents a policy, product or price | Vague, not traceable to source | Every claim traceable to your documents | | Dialect handling | Misunderstands dialect or code-switching | Understands but answers in stiff MSA | Registers the register, answers naturally | | Format and language | Wrong language, broken Arabic | Usable but unnatural | Natural, correctly punctuated Arabic | | Completeness | Misses a step or an explicit caveat | Usable | Complete, including edge cases | Two ways to score: a human reviewer (expensive, about 1 to 2 minutes per item, so a full pass on 300 items is a day) or a stronger model as judge. If you use a model as judge, check agreement first: have the human score a 50-item overlap sample and compute percent agreement on the Correctness criterion. Below roughly 85% agreement, your judge is not measuring your rubric, and you must fix the prompt before trusting it at scale. Here is a copy-paste judge prompt that has the properties that matter: it forces evidence, it forbids generosity, and it outputs a parseable score. You are evaluating a customer-service assistant's reply in Arabic. QUESTION (verbatim from a real user, may contain Gulf dialect or Arabic/English code-switching): {question} GOLD ANSWER (written by a domain expert): {gold_answer} ASSISTANT'S REPLY: {reply} Score each criterion 0, 1 or 2. Be strict: a score of 2 must be clearly earned, not merely plausible. 1. correctness: 2 = matches the gold answer's facts, entities and amounts; 1 = partially right; 0 = wrong. 2. grounding: 2 = every claim is traceable to the gold answer or the supplied policy text; 1 = vague; 0 = invents facts. 3. dialect: 2 = correctly understands the question's register and replies naturally in it; 1 = understands but replies in stiff MSA; 0 = misunderstands the dialect or code-switching. 4. language: 2 = natural, correct Arabic; 1 = awkward but usable; 0 = broken Arabic or wrong language. 5. completeness: 2 = covers steps and caveats; 1 = usable with gaps; 0 = misses something essential. Reply ONLY with JSON: {"correctness": n, "grounding": n, "dialect": n, "language": n, "completeness": n, "total": n, "fatal_error": "<one line, or 'none'>"} Run every candidate model, including the frontier API baseline and its Arabic-prompted variant, on the identical set with the identical retrieval context if you use RAG. Report the mean total, the per-criterion breakdown, and the fatal-error rate. The per-criterion breakdown matters more than the total: a model that loses two points per item entirely on dialect handling might be fixable with prompt work, while one that loses them on grounding needs retrieval surgery, and one that loses them on correctness needs to be dropped. For dialect specifically, published per-dialect evaluations such as DialectalArabicMMLU show how much scores move between MSA and regional varieties, and TII reports AraDice dialect sub-scores for the Falcon-H1-Arabic family. Your own dialect slice settles the question for your dialect, which is the only one that matters. The second thing that transfers: tokenization economics Arabic text costs more tokens than the equivalent English text on English-centric tokenizers, and that premium appears in four places at once: your API bill, time to first token, KV-cache memory and the number of concurrent Arabic sessions a GPU can hold. The peer-reviewed evidence is blunt. Petrov and colleagues showed that the GPT-4 tokenizer encodes the same content in Standard Arabic with roughly 3.04 times as many tokens as English, and the ChatGPT tokenizer with about 4.40 times as many, with direct consequences for cost, latency and effective context length (arXiv:2305.15425). The original Jais team measured the same pattern at the word level: GPT-2's vocabulary fragments Arabic into about 4.17 tokens per Arabic word against roughly 1.1 per English word, while the bilingual Jais vocabulary achieves about 1.05 (fertility figures summarised here). Newer multilingual vocabularies have narrowed the gap, but a 1.5 to 2 times premium on an Arabic-inefficient tokenizer is still normal, and nothing on your invoice labels it. Measure it on your own traffic before believing any ratio. Ten lines of Python gives you fertility, tokens per word, on a sample of your real conversations: from transformers import AutoTokenizer text = open("sample_arabic_conversations.txt", encoding="utf-8").read() words = len(text.split()) # Verified repo IDs; add the other candidates from their model cards for repo in ["QCRI/Fanar-2-27B-Instruct", "humain-ai/ALLaM-7B-Instruct-preview"]: tok = AutoTokenizer.from_pretrained(repo) fertility = len(tok(text).input_ids) / words print(f"{repo}: {fertility:.2f} tokens per Arabic word") One caution: studies count words and normalise text differently, so compare ratios within your own sample rather than against published absolutes. Your sample is the number that will hit your bill. Worked example: what the premium actually costs Take a 400-word Arabic support answer and compare a model whose tokenizer gives 2.4 tokens per Arabic word against an Arabic-optimised model at 1.5. Same content, same job. Spend. The answer is 960 tokens on the first tokenizer, 600 on the second, a 1.6 times premium on every request. At an illustrative $5 per million output tokens, one million such answers costs $4,800 versus $3,000, a $1,800 difference before you have written a word of prompt engineering. Multiply by your volume. And note the trap when comparing providers: a model priced 30% lower per token that fragments Arabic 1.6 times more is not 30% cheaper for you, it is roughly 12% more expensive. Latency. Prefill is compute-bound on prompt length, so time to first token scales with the token count of your system prompt plus retrieved context. Decode is per token, so an answer fragmented into 1.6 times the tokens streams 1.6 times longer at the same hardware speed. Prompt tuning cannot remove this; it is in the vocabulary. KV cache and concurrency. Per-token KV-cache size for a 7B-class model with 32 layers, 8 grouped-query KV heads and 128-dimension heads at fp16 is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB per token. A 4,000-token Arabic conversation at the efficient tokenizer holds about 500 MB of KV per session; the same content at 1.6 times fertility holds about 800 MB. On a GPU with 60 GB free after model weights, that is roughly 120 concurrent sessions versus 75. Tokenization, not parameter count, decided your concurrency. Context. A 256K context window (Falcon-H1-Arabic 7B and 34B) holds far more Arabic at a low fertility than a longer window holds at a high one. This is why the Jais 2 team built a custom 150,272-entry Arabic-centric vocabulary and report it makes inference efficient: the efficiency is not marketing, it is arithmetic you can reproduce with the snippet above. The decision rule Now the choice reduces to one comparison on one spreadsheet. 1. Run all candidates on your eval set. Suppose the frontier API scores 9.1 out of 10 and Falcon-H1-Arabic 7B scores 8.6. The gap is 0.5 points. 2. Set your tolerance in advance, in the currency of your use case: acceptable fatal-error rate on billing questions, acceptable dialect-mishandling rate. Do not set it after seeing the scores. 3. If the open model is inside your tolerance: run it on your own GPUs. You keep every conversation in-country, you pay a fixed infrastructure cost instead of a token bill that scales with Arabic verbosity, and you are not exposed to an API being retired (Jais 2's hosted endpoint went away in August 2026; its weights did not). A regional open model that scores within tolerance on your data is strictly the better enterprise choice: cheaper per request, in-country data, and no dependency on someone else's pricing page. 4. If the gap exceeds your tolerance: do not pay for it blindly. Quantify the frontier model's advantage in cost per resolved request: with the 1.6 times tokenization premium on an English-centric tokenizer and your ticket volume, what is the API bill for the extra 0.5 points? Then price the alternative: the same eval gap might close with a better Arabic system prompt, stronger retrieval into your policy documents, or three days of fine-tuning the open model on 2,000 of your own tickets. Very often the fine-tune is cheaper in month four than the API premium is in month one. A practical note on keeping this sustainable: whatever prompts and gold answers survive the eval, version them. The Arabic system prompts that work, the dialect-handling instructions, the few-shot examples, these are assets, and they are exactly the kind of asset that walks out the door in a Markdown file. We keep them in a shared, versioned prompt library (PromptForge), and when we build and run these assistants for clients, from eval set through deployment, that is our AI engineering work, done by our own team here in Dubai. What to do next 1. This week: pull 300 real items using the mix above. It is three days of one analyst and one domain expert, and it is reusable for every model you will ever evaluate. 2. In parallel: run the fertility snippet on 100 of your real conversations for every shortlisted model, including the frontier API's tokenizer. You now have the two numbers that transfer. 3. Next week: run all candidates through the judge prompt, with a human agreement check on 50 overlapping items. Build the table: eval score, fatal-error rate, tokens per request, cost per resolved request. 4. Then decide with the rule above. If the regional open model is inside tolerance, it wins on data residency and unit economics, and GCC regulators and your security team will thank you. If it is not, you now know exactly what the gap costs per month, which turns a vague preference into a negotiation. The best Arabic LLM for enterprise is not a fixed answer, because "enterprise" is not a fixed workload. It is whichever of Falcon-H1-Arabic, ALLaM, Fanar 2, Jais 2 or a frontier API passes an eval set built from your own tickets, at a token cost you have measured rather than assumed. Both measurements take a week. The alternative is guessing, and guessing is the most expensive part of any AI project. --- Originally published on Azrty.
Run your LLM evals every night on a GPU that only exists for a minute Most eval suites run when someone remembers to run them. That's usually right before a release, which is the worst time to find out a model regressed two weeks ago. The fix is boring: run the evals every night, on a GPU that exists only while they run, and read the number in the morning. This post sets that up with Runcrate's CLI: one command creates a nightly schedule, each run is a job on a fresh GPU that stops when the eval exits, and a run costs about a cent for the example here. I ran every command below on 2026-10-21 and the outputs are copied from my terminal. The eval I'm using EleutherAI's lm-evaluation-harness on a small model so the run is quick: Qwen/Qwen2.5-0.5B-Instruct on 200 questions of ARC-Easy. Swap in your own model and tasks; the scheduling part doesn't change. The command that runs on the GPU machine: pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && \ uv pip install -q "lm-eval[hf]" torch --torch-backend=cu128 && \ lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct \ --tasks arc_easy --limit 200 --batch_size 16 The first line makes a clean virtualenv with uv, so the eval doesn't fight with whatever Python packages the machine image ships. --torch-backend=cu128 picks the PyTorch build that matches the machine's CUDA 12.8 driver. I tried a plain pip install first and got a PyTorch wheel that refused the driver, so pin this to your driver. Install the CLI and sign in brew install aeonmindai/tap/runcrate runcrate login login opens the browser. You pick a workspace and an environment, and everything the CLI starts runs and bills there. Create the schedule runcrate schedule create --cron '0 2 * * *' --tz Europe/London --name nightly-eval \ --gpu A6000 --max-runtime 30m -- \ 'pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && uv pip install -q "lm-eval[hf]" torch --torch-backend=cu128 && lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct --tasks arc_easy --limit 200 --batch_size 16' What it printed: Schedule created: nightly-eval ID: e18aeb6f-6d15-4f51-804b-65b258ce71e1 Kind: job When: 0 2 * * * (Europe/London) Overlap: skip Enabled: yes Next run: 2026-10-22 02:00 BST GPU: 1x A6000 Max runtime: 30m A few things worth knowing about those flags: - A command after -- makes it a job. At 02:00 London time Runcrate starts a fresh A6000, runs the command, and stops the machine when the command exits. Without a command it would start an instance instead. - --max-runtime is required. If an eval hangs, the job is stopped after 30 minutes. Nothing a schedule starts can run unbounded. - --tz keeps wall-clock time. 0 2 * * * in Europe/London stays at 02:00 local across daylight saving changes. - Overlap: skip is the default. If last night's run is somehow still going, tonight's is skipped instead of stacking a second machine. Don't wait until 2am to test it runcrate schedule run-now e18aeb6f-6d15-4f51-804b-65b258ce71e1 Started job nightly-eval (ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33). Follow it with: runcrate jobs logs -f ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 Run now starts the same job straight away and doesn't move the next scheduled run. I also let a schedule fire on its own while testing: it was due at 02:00 BST and launched at 02:02, which matches the docs (schedules are checked every 2 minutes). Read the result runcrate jobs logs prints the job's output. The end of it: hf ({'pretrained': 'Qwen/Qwen2.5-0.5B-Instruct'}), gen_kwargs: ({}), limit: 200.0, num_fewshot: None, batch_size: 16 | Tasks |Version|Filter|n-shot| Metric | |Value| |Stderr| |--------|------:|------|-----:|--------|---|----:|---|-----:| |arc_easy| 1|none | 0|acc |↑ |0.565|± |0.0351| | | |none | 0|acc_norm|↑ |0.560|± |0.0352| And runcrate jobs status for what it cost: Name: nightly-eval Status: ● succeeded Exit code: 0 GPU: 1x A6000 Max runtime: 30m Runtime: 1m3s Cost: $0.01 ($0.65/hr) Created: 2026-10-21T01:08:32.934+00:00 Started: 2026-10-21T01:12:50.771+00:00 Ended: 2026-10-21T01:13:53.945+00:00 About four minutes went to getting the machine ready and one minute to the eval itself. The max runtime counts from when the command starts, so setup doesn't eat into your 30 minutes. A6000 was $0.65 an hour when I ran this; check the live price before you pick a GPU. runcrate jobs list shows every night's run with its status, exit code and cost, so a week of runs is one command: ID NAME STATUS GPU EXIT RUNTIME AGE COST ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 nightly-eval ● succeeded A6000 0 1m 7m ago $0.01 Making it a real regression check The toy version prints a number. To make it catch regressions: 1. Evaluate your checkpoint, not a hub model. Attach a storage volume to the job and point pretrained= at the checkpoint under /workspace. Volumes are covered in the jobs docs. 2. Fail the job on a bad score. Add --output_path /workspace/evals to lm_eval, then a two-line Python check that exits non-zero when accuracy drops below your threshold. The job ends failed with that exit code, which is what you watch for. 3. Keep secrets out of the command. The command and logs are visible to everyone in the workspace. Put HF_TOKEN and friends in the environment's secrets (set in the dashboard), which every job receives as environment variables. Pause, run, delete runcrate schedule list runcrate schedule pause e18aeb6f-6d15-4f51-804b-65b258ce71e1 runcrate schedule delete e18aeb6f-6d15-4f51-804b-65b258ce71e1 -y Pausing stops future runs and leaves anything already running alone. Or ask your agent If you use Claude Code, Codex or Cursor, the Runcrate MCP server has the same schedule tools, so "every night at 2am London time, run my eval on an A6000 for at most 30 minutes" works as a prompt. Setup for each client is on the MCP setup page. Links - Schedules: runcrate.ai - Jobs: runcrate.ai - lm-evaluation-harness: EleutherAI/lm-evaluation-harness If you set this up for your own model and something gets in the way, tell me. Reply here or write to hello@runcrate.ai. Written with AI assistance. Every command and output above was run on 2026-10-21.
How Jev decision models work (and how to build your own on MLX) The web is bloated with Jev posts. The thing is a few weeks old, and my feeds are already full of it.. benchmarks, hot takes, threads about decision models being the future of agents. Jev, if you've managed to avoid the hype, is TypeSafe AI's hosted decision API where you send a situation and a few typed questions, it returns the answer with a probability for every option. Two spoilers, because I like roads with signposts. Classical ML never left the chat, and the newest tool in the box still doesn't answer the oldest question, whether you actually need it in the first place. --- At some point I stopped skimming news and got really curious about the machinery underneath. How does a model decide instead of write? What is actually different in there? The short answer turned into a project, and the project turned into a story about building a thing and then discovering, layer by layer (sorry), that most of it already existed. It's called take-my-layer. --- Init I'll be honest, I never thought I'd invented something. I knew reading probabilities off logits was a thing and the Jev posts themselves hint at it. But knowing a trick exists and holding it in your own hands are different things, and what I didn't know was how far it stretches. I took a 4-bit Qwen model on a laptop, no training, no labels, questions made up on the spot. That part was worth testing. So I went scouting before building anything serious. Kind of following my own advice from a post about fog of war as a mental model. The fog doesn't mean the territory is missing, it just means you haven't been there yet. Reading papers is scouting. Building a small proof of concept is scouting. I expected to find maybe one or two adjacent things. Well.. the territory I found was huuuge :D --- The trick is from 2021 First stop. Reading answer probabilities instead of generating answer text. This is how the whole evaluation world has scored decoder-only models for years.. it's the loglikelihood default in lm-evaluation-harness. The prompting version has names (verbalizers, pattern-verbalizer pairs, cloze-style prompting). PET (Schick & Schütze, EACL 2021) and "Small Language Models Are Also Few-Shot Learners" (NAACL 2021) are the classics, LM-BFF (ACL 2021) even automates the label-word search. What I'd been calling "reading the logits" has a standard name, a literature, a codified abstraction, and several thousand citations. The trick predates ChatGPT (obvious for people who are not the first day on the ML street). --- The bugs are from 2021 too The naive version has a bug called position bias. Ask "A, B, C or D?" and the model leans toward whichever option is listed first, regardless of the options. I found that myself the fun way when the raw read on my coding data predicted bash for 74 of 120 states. A bash magnet kind of. Its 0.57 accuracy was roughly bash's base rate in the data. Essentially every bug I hit someone had already hit: - Calibrate Before Use (Zhao et al., ICML 2021): content-free priors to correct the model's answer skew. - Order sensitivity of option lists (Pezeshkpour & Hruschka, 2023). - Fantastically Ordered Prompts (Lu et al., ACL 2022): same disease, few-shot demonstrations. - PriDe (ICLR 2024): the cheap permutation-plus-prior combination, which is more or less the cheaper cousin of my scheme. - Batch Calibration (ICLR 2024): the unified view of all of the above. My own fix (arrived at independently and with more cursing), was to list the options in several rotated orders, ask once per order, average. That's called permutation marginalization, and it is worth +4.7 points of agreement with the reference over the naive single-prompt read (5 answers flip toward the reference, 0 away). All of that was known. Standing on shoulders of giants, etc. --- Leaving early The next piece was speed. A full forward pass over a 4B model for a one-word answer is wasteful, and it turns out the decision signal sits pretty early. (Qwen3.5-4B has 32 blocks; layer selection kept picking 16 and 19, before the last full-attention blocks. The interesting states live in the linear-attention trunk.) The literature here is CALM (NeurIPS 2022), early exit per token by confidence. In my case it's a fixed half-depth exit for one fitted head, the layer-pruning family. Fine again, keep standing on the shoulders! In MLX land there is exactly one static precedent that I could find: NeoJev, a 27B 4-bit (~15 GB) that keeps 56 of 64 layers with a self-distilled head, 0.727 mean accuracy at 310 ms p50 on an M5 Pro. One detail I genuinely enjoyed is that a plain logit lens at their layer 56 reads 0.663. Same head-over-raw gap I measured on my own side, so kind of proving I'm moving in the right direction. A practical note for anyone following this route: mlx-lm exposes no intermediate-layer hooks (issue #1053, still open as of date of this blog post), so a hand-rolled block loop is the only way in. Mine matches the stock forward to a max logit difference of about 0.037. --- A warning from the probing people Fitting a tiny linear head on hidden states is also a literature: called probing. Hewitt & Liang (2019) and Belinkov's survey (2022) are the standard references, and their criticism is aimed at claims. A probe is weak evidence about what a model "understands". I took that as a warning about wording but not about architecture. A production head is judged by held-out task accuracy on the served artifact (ours reads 0.826 on its holdout), not by what it supposedly reveals. Linear (as the probing people say) is the defensible capacity choice. --- And finally products The method hunt was humbling, in that almost everything was already there before. The product hunt was even more humbling. Jev you've already met. Cloudflare's Clef ships fine-tuned decision models with open Apache-2.0 weights (209 ms median, 38.8 ms for the flash variant). Unsloth has a train-your-own recipe: LoRA plus a Clef-design head, their 0.8B reads 36% on typed-decisions before training and 73% after. Around those there's a whole cluster: Mini-Jev, mlx-decision, laya-mlx. Even the routing people are adjacent, though RouteLLM and friends put a separate model in front of the LLM, whereas I wanted the served model's own distribution to be the classifier. (!) And then the one that mattered to me: AnyJev, Nokia Applied Research's open method and library (technical report). Essentially I've taken AnyJev's method, deployed on the Mac. --- What's actually mine An honest ledger, because I care about this part. The repo's docs have a section literally titled "honest lineage", and it says explicitly that I did not invent the trick. What's original though: - The missing MLX backend for AnyJev's toolbox (an original implementation of their documented backend contract; they never shipped one). - The small-memory end: a 4-bit 4B, about 3 GB resident, versus NeoJev's 15. - Zero-training deployment: the frozen model works the moment it downloads. No dataset, no training run, no labels. - The observe() loop: while a shadow proxy watches real agent traffic, it quietly banks "what I predicted" next to "what actually happened", and one command refits a head from that bank. As far as I could find, nobody ships fit-on-your-traffic heads. - And the measurements of course. None of that is a new mechanism. It's just glue and a scorecard. BTW! I wrote about taste a while ago and there (TLDR) I say that when production becomes abundant, judgment becomes scarce. The judgment here was mostly deciding what not to claim. --- Benchmarks The repo holds itself to wording rules that I want to keep in this post too: fidelity claims say "aligned with the reference", never "more correct"; accuracy claims name the arena and the model size. So, with that: | what was measured | result | |---|---| | Agreement with the real Jev API, 300 non-coding questions | 4B 75.7%, 9B 82.0%; Jev agrees with itself 98.7%, which is the ceiling | | Readout vs greedy decoding of the same model, same 300 questions | +4.7 (4B), +6.3 (9B), McNemar p = 0.003; on long inputs greedy Choice answers collapse (46% vs 85% agreement) | | BANKING77, 10 intents, gold labels | 87.5% (4B), 91.25% (9B), Jev 95.5%: a trained model wins on its home turf | | typed-decisions, zero training | 61.6% (4B), 60.4% (9B), vs 73-78% for models fine-tuned on it | | Latency, long input, 5 questions in one request | 22.2 s down to 3.7 s mean (the state is read once); the coding head answers in ~0.2 s | The losses are the point, in a way. A trained model beats a zero-training readout whenever labels and a fixed label set exist. Train a classifier! This is for questions and options you cannot enumerate ahead of time, on a machine where the data cannot leave. Here's one real exchange on my M5: curl -s localhost:8095/v1/systemone -d '{"state": "Ticket #4821 from Dana (Pro plan, customer for 14 months): \"Since this morning every export to CSV fails with a 502. We run payroll from those exports at 3pm today.\" Status page: all green.", "questions": { "route": {"type": "Choice", "instructions": "Which team should take this first?", "options": ["billing", "platform-oncall", "account-management", "docs"]}, "escalate_now": {"type": "Noul", "instructions": "Does this need the on-call engineer right now?"}, "urgency": {"type": "Score", "instructions": "How urgent is this ticket?", "levels": ["low", "medium", "high", "critical"]}}}' -> {"model": "mlx-community/Qwen3.5-4B-MLX-4bit", "answers": { "route": {"answer": "platform-oncall", "confidence": 0.8516, "probabilities": {"billing": 0.0744, "platform-oncall": 0.8516, "account-management": 0.0579, "docs": 0.0161}}, "escalate_now": {"answer": true, "confidence": 0.637, "probabilities": {"Yes": 0.637, "No": 0.363}}, "urgency": {"answer": "critical", "confidence": 0.6651, "probabilities": {"low": 0.0051, "medium": 0.0156, "high": 0.3142, "critical": 0.6651}}}} > Status: research prototype, and the docs are blunt about it. The Jev-agreement numbers use LLM-authored questions with no human labels. The one fitted head that ships cannot be reproduced from the repo, because its training seed is withheld. If you have a fixed label set and training data, train a classifier instead. All of it is written down in a LIMITATIONS.md. --- The boring lesson This field is moving embarrassingly fast. But the lesson I'd actually keep from these weeks is a boring one: classical ML never left. Every number in the table above says so. Wherever a fixed label set and training data exist, something trained the old way wins. Jev's 95.5% on BANKING77 against my 87.5%, the fine-tuned 2B sitting 16 points above my zero-training readout on typed-decisions. Reading logits doesn't replace training. It covers the corner cases training can't reach e.g. questions you can't enumerate ahead of time, no labels to collect, data that isn't allowed to leave the machine etc. So before throwing this at a task, the checklist is the one classical ML wrote decades ago. Fixed labels and enough data? Train a classifier, it's still unbeatable there. Data that can't leave the machine, or questions that change per request? That's where reading the model's own probabilities earns its keep. In between, renting a hosted API is probably a fine deal. --- Oh right, the question Ah, I actually never answered the "how does a model decide instead of write" part from the beginning. Short version, and experienced ML people will roll their eyes at how obvious it is: the decision seems to form well before the end of the network. My tiny head reads block 16 of 32 and still answers the "what next" question at 0.826 accuracy. AnyJev's own measurements found a middle block to be a better feature space than the final one, since the last blocks are busy converting the answer into token space. And NeoJev keeps 56 of 64 layers, which suggests there is slack even higher up. So roughly: the deciding happens in the first half to two thirds of the layers, and the rest is more like the writing department. Am I even right here? Probably, for these models, by my measurements. That's as far as I'd push it. That's it.