nobodywho-ooo

They almost certainly knew... huggingface.co > In thinking mode, we have observed that when using the recommended sampling parameters, Qwen3.5-0.8B is more prone to entering thinking loops compared to other Qwen3.5 models There's also an active issue tracking this here: nobodywho#557 Pretty sure that is also why this model was specifically chosen.

Decision models support! A new major version is available for all bindings with breaking changes. Decision models NobodyWho now runs decision models in all our bindings: Python, Godot, Flutter, Kotlin, Swift and React Native/Expo. A decision model does not write text. You give it some input and a few questions, each with the answers you allow. It scores every answer and returns the probabilities, with token usage. It generates zero output tokens. In Python it's Decision and DecisionAsync; the other bindings have the same feature in their own style. There are three kinds of questions: - choice picks one option from a set and gives the probability of each one - noul gives the probability that the answer is yes - score rates something on an ordered scale of 2 to 10 levels With Laya Q8, our reference case (a customer message about being charged twice) routes to billing at 97.5%, gives angry a 0.78 probability of yes, and scores urgency at 1.91 out of 3. The call reads 166 input tokens and writes 0. For now we support Laya GGUF decision models. Fixes worth calling out - Conversation history as rendered to the LLM now matches exactly what the model saw and wrote. Previously some special tokens and whitespace could be added or lost. This can change output, but probably won't do so much. - Flutter: no longer needs to download onnxruntime.so on Android x86_64, it is bundled with the rest of libnobodywho_flutter.so. - A Hugging Face model path without a file name, such as hf://owner/repo, now fails with an error that shows the expected hf://owner/repo/filename.gguf format instead of a raw parser error. - React Native: added the missing proguard-rules.pro, which should allow enabling R8 on Android. Breaking changes Several changes to how tool calls are read from a model's output: - With tools, a chat now fails to start if the model's vocabulary doesn't have its format's tool call markers as single special tokens. All supported models have them. - A tool call cut off by stopping the generation or by the token limit is now kept as text and never runs. An LFM2 call could run before. - Qwen3.5, Qwen3.6 and FunctionGemma tool calls now pass an argument to the tool with the type its schema gives. Previously anything that parsed as JSON was converted, so a string parameter given 10 received the number 10 instead of "10". - A tool call marker a model writes inside its reasoning is now part of the reasoning, and no longer starts a tool call. - A block of tool calls that can't be read is now streamed as text, not only returned in the response. And one dependency bump: - ONNX Runtime, used for speech-to-text, text-to-speech and voice activity detection, is updated from 1.28 to 1.31. More in the docs: docs.nobodywho.ai Github: nobodywho-ooo/NobodyWho

Run a decision model locally in your React Native app Hey RN devs, We can now run Laya decision model in NobodyWho, our open-source on-device inference library. Unlike the LLMs that we also support, this type of model does not generate text. Instead you can score predefined answers, classify text or answer yes/no. For example, if you are building a personal diary app, you can easily detect the user’s mood, categorize an entry or detect stress, offline after model download, without sending any sensitive data to the cloud. First, you need to load the model: import { Decision } from 'react-native-nobodywho'; const decisionModel = await Decision.fromPath({ modelPath: 'hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf' }); Then you can ask questions/analyze text: const results = await decisionModel.decide({ state: Diary entry: ${diaryEntry}, questions: { topics: { type: 'choice', instructions: 'Choose the main topic of this diary entry', criteria: { work: 'job, colleagues, deadlines, career', relationships: 'family, friends, partner', health: 'sleep, exercise, food, illness', leisure: 'hobbies, travel, going out', }, }, stressed: { type: 'noul', instructions: 'Does the writer sound stressed?', }, mood: { type: 'score', instructions: "How is the writer's overall mood?", criteria: ['bad', 'okay', 'great'], }, }, }); const { stressed, topics, mood } = results.answers; Output example: // stressed { type: noul, noul: 0.00002 } // topics { type: choice, choice: leisure, probabilities: { work: 0.0422, relationships: 0.1518, health: 0.0302, leisure: 0.7756 }, confidence: 0.7008 } // mood { type: score, score: 1.6995, legend: {0: bad, 1: okay, 2: great}, probabilities: {0: 0.0082, 1: 0.2839, 2: 0.7078}, confidence: 0.5493 } A question can have the following types: - noul gives you probability of a yes answer (0 to 1) - choice select the most likely option and returns a probability for each option - score gives you an a weighted average and a probability per option. For example, you could place const x = mood.score; on this axis: ​ [bad - - - - - - - okay - - - x - - - great]  where bad is 0, okay is 1 and great is 2 (given by mood.legend). The Laya model is small enough to run on mid range phones (around 450 MB in its Q8 variant) and supports 100+ languages. For reference, the full answer comes after 0.88 seconds on S25 and 0.45 seconds on iPhone Air (cf. screenshot). Here is a runnable example if you want to experiment with decision models. Links: - NobodyWho inference library - npm - Docs

Run a decision model locally in your Flutter app Hey Flutter devs 👋 We can now run Laya decision model with NobodyWho, our open-source on-device inference library. Unlike the LLMs that we also support, this type of model does not generate text. Instead you can score predefined answers, classify text or answer yes/no. For example, if you are building a personal diary app, you can easily detect the user’s mood, categorize an entry or detect stress, offline after model download, without sending any sensitive data to the cloud. First, you need to load the model: import 'package:nobodywho/nobodywho.dart' as nobodywho; final model = await nobodywho.Decision.fromPath( modelPath: 'hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf' ); Then you can ask questions/analyse text: final result = await model.decide( state: "Diary entry: ${userInput}", questions: { "topics": { "type": "choice", "instructions": "Choose the main topic for this entry", "criteria": { "work": "job, colleagues, deadlines, career", "health": "sleep, exercise, food, illness", "relationships": "family, friends, partner", "leisure": "hobbies, travel, going out", }, }, "stressed": { "type": "noul", "instructions": "Does the writer sound stressed?", }, "mood": { "type": "score", "instructions": "How is the writer's overall mood?", "criteria": ["bad", "okay", "great"], }, }, ); final answers = (result as Map)["answers"] as Map; final topics = answers["topics"] as Map; final stressed = answers["stressed"] as Map; final mood = answers["mood"] as Map; Output example: // stressed { type: noul, noul: 0.00002 } // topics { type: choice, choice: leisure, probabilities: { work: 0.0422, relationships: 0.1518, health: 0.0302, leisure: 0.7756 }, confidence: 0.7008 } // mood { type: score, score: 1.6995, legend: {0: bad, 1: okay, 2: great}, probabilities: {0: 0.0082, 1: 0.2839, 2: 0.7078}, confidence: 0.5493 } A question can have the following types: - "noul" gives you the probability of a yes answer (0 to 1) - "choice" selects the most likely option and returns a probability for each option - "score" gives you a weighted average and a probability per level. For example, you could place final x = mood["score"]; on this axis:[bad - - - - - - - okay - - - x - - - great]  where bad is 0, okay is 1 and great is 2 (given by mood["legend"]). The Laya model is small enough to run on mid-range/recent phones (around 450 MB in its Q8 variant) and supports 100+ languages. For reference, the full answer comes after 2.2 seconds on S25 and 0.49 seconds on iPhone Air. Here is a runnable example if you want to experiment with decision models. Links: - NobodyWho inference library - pub.dev - Docs

大模型不一定要联网跑：NobodyWho 是 Rust 写的本地推理引擎，底层 llama.cpp，跑 GGUF 模型、不用 API Key，桌面手机都能离线用。新加的决策模型不生成文本，给备选答案打概率，本地分类、路由很顺手。不过中文输入效果还没验证，中文场景先当实验用，值得跟进。 nobodywho-ooo/nobodywho

449MBの決定モデルで、問い合わせチケットをローカルで振り分ける チャットモデルでテキストを分類しようとすると、だいたい次のような流れになります。プロンプトの最後に「JSONのみで回答してください」と書き、返ってきた文字列をパースし、「はい！カテゴリは以下の通りです：」から始まってしまったときのためにリトライ処理を入れる。これでも動きますが、5つのラベルから1つを選ぶだけにしては周辺コードが多すぎます。 決定モデル（decision model）は別のアプローチを取ります。入力と、受け付ける回答の候補を渡すと、モデルは各候補をスコアリングし、すべての候補について確率を返します。テキストは一切生成しないので、候補にない回答が返ってくることはありません。 この記事では、ローカルで推論を行う Rust 製ライブラリ NobodyWho の Python バインディングを使って、問い合わせチケットの振り分けを実装します。決定モデルは 10 月 29 日のリリース（Python 版 v5.0.0）で追加されました。 :::message 筆者は NobodyWho のコントリビューターです（コアチームのメンバーではありません）。 ::: インストールとモデルの読み込み pip install -U "nobodywho>=5.0.0" from nobodywho import Decision, Model model = Model(model_path="hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf") decisions = Decision(model=model, n_ctx=1024) 使うモデルは Laya です。0.4B の ModernBERT ベースのモデルで、Q8_0 の GGUF ファイルは 449MB、ライセンスは Apache-2.0 です。推論は同じプロセス内で動くので、サーバーの起動も API キーも不要です。ダウンロード済みのファイルを使う場合は、model_path にローカルのパスを渡します。 3つの質問を一度に投げる message = "I was charged twice. Please refund the duplicate payment." questions = { "department": { "type": "choice", "instructions": "Which team should handle this?", "criteria": { "billing": "Payments and refunds", "technical": "Bugs and login problems", }, }, "refund": { "type": "noul", "instructions": "Is a refund requested?", }, "urgency": { "type": "score", "instructions": "How urgent is this?", "criteria": ["can wait", "soon", "urgent"], }, } result = decisions.decide(state={"message": message}, questions=questions) 質問のタイプごとに、返ってくる値の形が異なります。 - choice：候補から1つを選びます。result["answers"]["department"]["choice"] が選ばれた候補で、["probabilities"] に全候補の確率が入っています。 - noul：はい／いいえの質問です。result["answers"]["refund"]["noul"] は「はい」の確率（0〜1）です。 - score：入力を 2〜10 段階の順序付きスケール上に位置づけます。段階は 0 から番号が振られるので、この例で 1.6 なら「soon」と「urgent」の間です。値は確率の加重平均なので、「urgent」そのものの確率を知りたい場合は probabilities["2"] を読みます。 また、result["usage"]["output_tokens"] は常に 0 です。 一番確率の高い答えだけでなく、確率そのものを使う 実際のパイプラインで決定モデルが役に立つのはここです。全候補の確率が得られるので、判断に自信がないケースだけを人間に回せます。 dept = result["answers"]["department"] if dept["confidence"] < 0.6: send_to_human(message) else: route(dept["choice"], message) send_to_human と route は自分のコードに置き換えてください。0.6 はあくまで例なので、自分のデータで調整します。 ドキュメントにも書かれている注意点として、confidence は確率がどれだけ一つの候補に集中しているかを表す値で、高いからといって答えが正しいとは限りません。この閾値を信用する前に、実際のチケット数百件で確かめておくことをおすすめします。 非同期で使う import asyncio from nobodywho import DecisionAsync async_decisions = DecisionAsync(model=model, n_ctx=1024) async def classify(): return await async_decisions.decide(state={"message": message}, questions=questions) result = asyncio.run(classify()) DecisionAsync のコンストラクタ自体は同期的に呼び出します。 知っておきたい制限 - 対応している決定モデルは今のところ Laya だけです。Laya 自身の README によると、型付き判定のベンチマークでの正解率はゼロショットで 0.362、ドメインに合わせてファインチューニングすると 0.766 です。重要な振り分けに使う前に、自分のデータで必ず測ってください。 - 日本語の入力での精度は未検証です。 この記事の例はすべて英語です。日本語のチケットで試した方がいれば、結果をコメントで教えてもらえると助かります。 - 選ぶことしかできません。理由の説明やテキストの抽出が必要なら、生成モデルは引き続き必要です。 - 決定モデルは Python のほか、Swift・Kotlin・Flutter・React Native・Godot のバインディングにも同じリリースで入りました。モバイルアプリ内で同じ判定を動かすこともできます。 ドキュメント：docs.nobodywho.ooo リポジトリ：nobodywho-ooo/nobodywho

Classifying text when you already know the labels: what a decision model is if you've tried text classification with an llm, you know the routine. write a prompt, ask for json, parse what comes back, and handle the times it answers with something that isn't one of your labels. a decision model works the other way round. you give it the text and the possible answers, and it scores each answer. you get back the winner, the probability of every option and a confidence value. it never generates text, so it can't answer outside your list. there are three kinds of question: pick one option, the probability of yes, or a position on a scale like "can wait / soon / urgent". nobodywho added these this week with laya, a 449 MB apache-2.0 model that runs on your own machine, so it's easy to try from python. Repo: nobodywho-ooo/nobodywho Docs: docs.nobodywho.ooo

Should this prompt leave the device? Let a 449 MB model decide first Most apps with an LLM in them make the same decision on every request, and they make it without looking: send it to the cloud. "Fix the typo in this sentence" goes to the same frontier model as "refactor this 2,000-line module", and the user's half-finished email with their bank details in it goes along for the ride. A better default is to look at the request first. Is it easy enough for a small local model? Does it have something in it that shouldn't leave the machine? Only then decide where it goes. The catch has always been that "look at the request first" meant another LLM call, plus a parser for its answer, which defeats the point. NobodyWho's new decision models make that first look cheap. A decision model doesn't write text. You give it the request and a set of answers you've already written, and it returns a probability for each one. It generates no tokens and needs no network, so the request is read on the device before anything is sent anywhere. NobodyWho runs it with Laya, an Apache-2.0 decision model whose Q8_0 GGUF file is 449 MB. Load the model Install NobodyWho with pip install nobodywho, then point a Model at the Laya file on Hugging Face, or at a local GGUF path if you already have it. from nobodywho import Decision, Model model = Model(model_path="hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf") decisions = Decision(model=model, n_ctx=1024) Ask the request three questions The router needs three answers: which model should handle the request, whether it contains anything private, and how hard it is. Each one maps to a question type. A choice question picks one option, a noul question returns the probability that the answer is yes, and a score question places the request on a scale you define, from two to ten levels. prompt = "Can you shorten this email? Hi Sam, my account number is 4471-2290, please move the payment to Friday." questions = { "handler": { "type": "choice", "instructions": "Which model should answer this request?", "criteria": { "local": "Everyday requests a small model handles well, like rewording, shortening or summarising text the user provided", "cloud": "Requests that need deep reasoning, long code or broad knowledge", }, }, "private": { "type": "noul", "instructions": "Does the request contain personal, financial or confidential information?", }, "difficulty": { "type": "score", "instructions": "How hard is this request to answer well?", "criteria": ["trivial", "moderate", "hard", "expert"], }, } result = decisions.decide(state={"message": prompt}, questions=questions) assert result["usage"]["output_tokens"] == 0 The criteria are plain English, and they're the part you'll tune most. "local" describes what your local model is actually good at, not what you hope it's good at. If your local model is bad at summaries, take summaries out of that sentence. Turn three answers into one route Here's the whole router. The order of the checks is the policy. handler = result["answers"]["handler"] private = result["answers"]["private"]["noul"] if private >= 0.5: route = "local" # private stays on the device, whatever else is true elif handler["choice"] == "cloud" and handler["confidence"] >= 0.6: route = "cloud" # clearly hard, nothing private else: route = "local" # unsure: try local first answer = ask_local_llm(prompt) if route == "local" else ask_cloud_llm(prompt) (ask_local_llm and ask_cloud_llm stand in for your own code.) Two things are worth noticing. First, privacy wins. If the model thinks there's a reasonable chance the request contains something personal, it never leaves the machine, even when it looks hard. You decide what "reasonable chance" means: 0.5 is an example, and a stricter app would go lower. Second, "unsure" has somewhere to go. The confidence value runs from 0 to 1 and measures how concentrated the probabilities are. Low confidence on the handler question doesn't mean "cloud". It means the model can't tell, and for most apps the cheaper, more private guess is the right default. If the local answer turns out to be bad, escalate then. Use the score for logging, not for routing The difficulty score is tempting to route on directly, and it's better not to. A score is a weighted average of the levels, so a request the model sees as either "trivial" or "hard" can come out looking "moderate", a level it gave very little weight. The full picture is in the probabilities, and the confidence tells you when the model is split. What the score is good for is your logs. Store it next to the route and the answer the user got, and after a week you can see whether the requests you kept local were really the easy ones. That's also how you tune the criteria and the two thresholds above: against your own traffic, not against a guess. The docs are clear that high confidence doesn't guarantee a correct answer, so check a sample by hand before you trust the router. Running it in your app DecisionAsync gives you the same API with await, so the router can sit in front of a web handler without blocking it. Decision models are also in the Flutter, React Native, Swift, Kotlin and Godot bindings, and that's where this pattern pays off most: on a phone, the router runs on the device, so a request with an account number in it is read locally and never has to be sent anywhere to find out it shouldn't have been. The full Python guide is at docs.nobodywho.ooo, and NobodyWho is open source at nobodywho-ooo/nobodywho. If this kept one prompt on the device, a star on the repo is the easiest way to tell the team to keep going.

Laya decision models now run in NobodyWho: classification with zero output tokens Decision models just landed in NobodyWho, the open-source engine for running models on-device. A decision model scores a set of answers you define instead of generating text, so it fits any classification where you already know the possible answers. You pass in the text and your questions, and you get back a choice, a yes-or-no probability or a score on a scale, along with the probability of each option. It runs Laya, a 449 MB Apache-2.0 model, generates no output tokens, and is available in Python, Flutter, React Native, Swift, Kotlin and Godot. I contribute to NobodyWho but the technology is really interestingg Repo: nobodywho-ooo/nobodywho Docs: docs.nobodywho.ooo

Triage support tickets locally with a 449 MB decision model Every support inbox asks the same three questions of every ticket: who should handle it, does the customer want their money back, and how much is on fire. The usual fix is to ask an LLM, write a parser for whatever it decides to say, and then write a second parser for the day it answers "Billing (probably)" instead of "billing". NobodyWho's new decision model support for Flutter, React Native, Swift, Kotlin and Godot bindings, remove the parser from that story. A decision model scores answers you have already written. You give it the options up front, it returns a probability for each one, and it never generates a single token, so it has no way to invent a fourth department. TypeSafe calls this kind of model a System One model, after the fast, intuitive half of Kahneman's Thinking, Fast and Slow, and describes its own model, Jev, as "unstructured state in, typed probabilistic decisions out". NobodyWho runs the same kind of decision on your own hardware, using Laya, an Apache-2.0 decision model whose Q8_0 GGUF file is 449 MB. Load the model Install NobodyWho with pip install nobodywho, then point a Model at the Laya file on Hugging Face, or at a local GGUF path if you already have it. from nobodywho import Decision, Model model = Model(model_path="hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf") decisions = Decision(model=model, n_ctx=1024) Ask the ticket three questions Each question has an ID, a type and plain-English instructions. A choice question picks one option, a noul question returns the probability that the answer is yes, and a score question places the ticket on a scale you define, from two to ten levels. message = "I was charged twice. Please refund the duplicate payment." questions = { "department": { "type": "choice", "instructions": "Which team should handle this?", "criteria": { "billing": "Payments and refunds", "technical": "Bugs and login problems", }, }, "refund": { "type": "noul", "instructions": "Is a refund requested?", }, "urgency": { "type": "score", "instructions": "How urgent is this?", "criteria": ["can wait", "soon", "urgent"], }, } result = decisions.decide(state={"message": message}, questions=questions) team = result["answers"]["department"]["choice"] probabilities = result["answers"]["department"]["probabilities"] refund_probability = result["answers"]["refund"]["noul"] urgency = result["answers"]["urgency"]["score"] assert result["usage"]["output_tokens"] == 0 That last line is the whole pitch in one assert. The model read the ticket, answered three questions and wrote nothing. What actually comes back NobodyWho's test suite runs a very similar ticket through Laya: "I was charged twice for my order last week and nobody has replied." It asks which of billing, shipping or technical should take it, whether the customer is angry, and how urgent it is on a four-step scale from "can wait" to "right now". These are the reference answers the suite checks against on CPU, rounded: { "route": { "choice": "billing", "probabilities": {"billing": 0.975, "shipping": 0.014, "technical": 0.011}, "confidence": 0.962 }, "angry": {"noul": 0.784}, "urgency": { "score": 1.914, "legend": {"0": "can wait", "1": "this week", "2": "today", "3": "right now"}, "probabilities": {"0": 0.040, "1": 0.458, "2": 0.050, "3": 0.452}, "confidence": 0.006 }, "usage": {"input_tokens": 166, "output_tokens": 0} } The routing answer is as clean as it gets: billing at 97.5%, with a confidence of 0.96. The model also puts the chance that this customer is angry at 78%, which anyone who has been charged twice will find conservative. The urgency answer is where it gets interesting. A score is a weighted average of the levels, and 1.914 rounds neatly to "today", but the model only gives "today" 5%. What it actually believes is that this ticket is either a "this week" problem (45.8%) or a "right now" problem (45.2%), and the confidence of 0.006 is its way of telling you it can't pick. If your code only read the score, it would file the ticket under an answer the model gave a 5% chance. Send the unsure ones to a person Choice and score answers both carry that confidence value, from 0 to 1, which measures how concentrated the probabilities are. That gives you a simple rule for triage: route confident answers automatically and put the rest in front of a human. answer = result["answers"]["department"] queue = answer["choice"] if answer["confidence"] >= 0.6 else "human-review" Applied to the reference output above, the route goes straight to billing at 0.96 and the urgency goes to a person at 0.006, which is exactly the split you want. Treat 0.6 as a starting point to tune against your own tickets, and keep an eye on the queues for the first week, because the docs are clear that high confidence does not guarantee a correct answer. Running it in your app DecisionAsync gives you the same API with await, which suits a web handler that triages tickets as they arrive. Decision models are also in the Flutter, React Native/Expo, Swift, Kotlin and Godot bindings, so the same questions can run inside a mobile or desktop app, and the ticket text never leaves the device. The full Python guide can be found here. If this saved you a parser, a star on the repo is the easiest way to tell the team to keep going ❤️