jaredpalmer

```sh git clone jaredpalmer/kev cd kev uv sync --extra serve uv run --extra serve python -m kev.serve \ --run jaredpalmer/kev-4b \ --port 8009 ```

16/17 Try Kev for self-hosted experiments: it provides a Jev-compatible decision API, but it’s an independent model—not Jev’s weights/training. Compare accuracy, calibration, latency and cost on your own eval set: jaredpalmer/kev

16/17 Try Kev for self-hosted experiments: it provides a Jev-compatible decision API, but it’s an independent model not Jev’s weights/training. Compare accuracy, calibration, latency and cost on your own eval set: jaredpalmer/kev

Post-Training a Jev-Inspired Decision Model on an RTX 3090 What would it take to post-train a small decision model inspired by Jev and the System One (System 1) idea? On one RTX 3090, I used Qwen3.5 Base as a frozen pretrained backbone and trained LoRA parameters plus a pointer decision head to output probabilities over candidate answers. This is a local learning experiment, without a claim to reproduce Jev's proprietary architecture or RLCD training method. In one tiny development probe, fine-tuning raised correct answers from 7/28 to 13/28, while mean negative log likelihood worsened from 2.215 to 3.916. The training path worked, but accuracy alone left an important part of the result unseen. This hands-on account walks through three things you can examine in your own decision-model experiments: - How a frozen Qwen backbone, LoRA and a pointer head produce probabilities over candidate descriptions. - How task wording and labels can disagree even when the data format is valid. - How to separate development results, public benchmark diagnostics and future confirmation when deciding what to improve next. The experiment post-trains decision models using Qwen3.5-0.8B-Base and Qwen3.5-4B-Base as pretrained backbones on a single RTX 3090. Adapters, trained decision heads and data-reconstruction materials are linked below. This is a learning retrospective with configurations and evidence, rather than a complete command-by-command tutorial. Originally published on my blog. The blog is the maintained source of truth; this is its English cross-post. Some calls in an agent workflow only need a judgment: is the user adding context or changing an earlier decision? Does a passage support a claim? Which team should handle a ticket? The answer may be a single option, yet it determines what the program does next. Reading about TypeSafe's Jev and System One models made me curious about one capability: reading context, a question and candidate answers, then returning probabilities over those candidates. It sounded complex. I wanted to understand whether the underlying structure and training process could be broken into steps an individual developer could understand and try firsthand. I used the RTX 3090 from my earlier local coding agent experiments, using Qwen3.5 Base as the pretrained backbone for post-training a decision model: first the 0.8B backbone, then 4B. The backbone weights stayed frozen; supervised post-training updated newly initialized LoRA parameters and a pointer decision head. The resulting model reads context, a question and candidate answers, and returns a probability for each candidate. Those sizes also reflected local memory and compute constraints. This was primarily a personal learning experiment: implement the basic candidate-scoring path with the resources available, understand how the model learns these judgments, and learn how to check the results. Matching the full results of enterprise-scale cluster training was outside the goal of this experiment. Initially, I only wanted to establish that the training path worked. The model quickly learned a tiny set of training questions. On another set, something more interesting happened: the number of correct answers increased while mean NLL got worse. Later, inspecting the training data revealed labels derived from review stars, even though the question asked about the attitude expressed in the text. This account follows the first two model versions: turning a language model into a candidate scorer, understanding why accuracy alone cannot establish training success, and making supervision and evaluation answer the same question. It also introduces the weaknesses revealed by public JevBench questions that motivated further quality optimization. Reading a question and scoring its candidates Consider a review judgment. The text and probabilities below are fictional, illustrating the interface: > Context: The food was excellent and the service attentive. I'll be back. > Question: Based only on this text, is the reviewer clearly positive overall? > Candidates: Yes—clearly favorable overall; No—clearly unfavorable, or substantively mixed without a clear overall favorable judgment. > Output shape: {"yes": 0.8, "no": 0.2} The candidate answers and their descriptions are part of the input. A ticket-routing request could instead supply descriptions for billing, shipping and returns. The output would still be a probability for each candidate. The structure used here has three parts: 1. The Qwen3.5 Base backbone reads the input. Context, question and candidate descriptions are encoded as tokens, and the pretrained model computes contextual representations at their positions. 2. A pointer decision head scores the candidates. This small readout takes the representations at the decision and candidate positions, projects and matches them to obtain candidate scores, then applies softmax to produce a probability distribution. 3. LoRA adapts the model to the task. The original backbone weights stay frozen. Training updates low-rank parameters inserted into it and the newly initialized readout, reducing the parameters and optimizer state that need to be optimized. !Context, question and candidate descriptions enter Qwen3.5. A frozen backbone with trainable LoRA produces representations; a pointer head converts candidate scores into probabilities. For questions with a determinate label, the correct candidate supplies the supervision. Cross-entropy encourages the model to increase that candidate's probability, and gradients update LoRA and the readout. This connects reading the text, scoring the candidates and learning from labels. Why choose this structure? A fixed-class classification head usually reserves an output position for each class. Scoring candidate descriptions supplied in each request better matches the interface I wanted to explore. Reading scores directly also removes the step of generating a natural-language answer and parsing its label. Prompting an existing language model to generate a label is another valid approach; this experiment explores direct scoring and does not provide a controlled speed, cost or reliability comparison between the two. The starting point was Jared Palmer's KEV training process and implementation: a pretrained Base model with fresh LoRA parameters and a fresh readout. The architecture comes from that reference. Our changes at this stage focused on supervision, evaluation separation and training recovery. Why more correct answers were not enough The training mixture covered classification, question answering and rule composition; Yelp review judgments were only one task. The first step was a tiny training check. I selected the two shortest records from each data source, giving 24 records and 30 questions; one record can contain several questions. The 0.8B model repeatedly trained on them for 80 optimizer updates. Before and after training, I checked both those training questions and a separate exploratory development set. The latter came from the same original datasets but used different data groups and was not directly used as training examples in this run. | Questions and metric | Before training | After training | | --- | ---: | ---: | | Correct on 30 training questions | 7/30 | 29/30 | | Mean NLL on 30 training questions | 2.1845 | 0.1030 | | Correct on 28 development questions | 7/28 | 13/28 | | Mean NLL on 28 development questions | 2.2153 | 3.9161 | Accuracy checks whether the highest-probability option is correct. Negative log likelihood, or NLL, also checks how much probability goes to the correct answer. For a question with a determinate answer, it is −ln(probability of the correct answer). Lower is better. As a hypothetical illustration, suppose a binary classifier gives the correct answer a probability of 0.4 or 0.001. Both predictions count as wrong. The latter almost rules out the correct answer, so its NLL penalty is much larger. More questions can therefore be answered correctly while mean NLL deteriorates. The tiny experiment showed these metrics moving apart; the aggregate table alone does not identify which questions caused the change. Moving from 7/30 to 29/30 on the repeated training questions showed that the model could learn that set. I also checked that backbone weights stayed unchanged, LoRA and readout weights changed, and saving and reloading preserved outputs. Together, these checks validated the training and saving path. The 28 development questions were too few and too limited in origin to establish broad generalization. A program using probabilities to decide whether to act also needs calibration checks: among judgments made with roughly 80% confidence, are roughly 80% actually correct? Softmax only makes probabilities sum to one, and NLL does not replace a calibration assessment. This experiment led me to examine correct-answer counts alongside probability metrics, rather than watching training loss alone. Did the labels answer the question being asked Before scaling up, I inspected the Yelp review task in the mixed training data. Its binary labels came from a star-rating threshold, while the question asked: "Based only on this review text, is the reviewer clearly positive overall?" Those labels need not express the same judgment. Imagine a fictional review: "The food arrived quickly, but it was disappointing. Overall, I wouldn't recommend it." Suppose it carries four stars and the old rule maps that rating to Yes. That label follows the rating rule, but the text-based question calls for No. The model receives conflicting instructions and supervision. We specified the criterion explicitly: use only the supplied text. Clearly favorable overall means Yes. Clearly unfavorable, or substantively mixed without a clear favorable overall judgment, means No. Insufficient information, truncation or unresolved material ambiguity means holding the question aside. "Still waiting for the food; I'll update this later," for example, cannot support an overall judgment. Annotation packets hid the original stars and old labels. We used model-assisted annotation, with crosschecks of changed labels, uncertain cases and a subset of unchanged cases. Uncertain was an annotation holding status, not a third output class for this binary task. Ultimately, 38 relevant questions were withheld from supervision while their records and other questions could remain. The useful lesson was that well-formed data can still teach a different task. Valid JSON, acceptable input length and in-range labels do not reveal this mismatch. The question, candidate descriptions and label have to be read together. Additional epochs cannot resolve a contradiction in the task definition for us. Here, "repair" means aligning supervision with an explicit text criterion. Model-assisted labels can still share errors, and the subsequent comparison does not isolate the causal benefit of this change. This audit finding also cannot directly explain the earlier tiny experiment's NLL deterioration; that causal link was not tested. Aligning training and development criteria Beyond the Yelp supervision issue, structural checks, overlap screening and sampled semantic review quarantined 56 whole records involving near-copy risks, unresolved ambiguity and label conflicts. After incorporating the Yelp repair, training used 12,520 records and 15,476 questions. Versions were saved separately with checksums so an intermediate dataset would not be mistaken for the data a final model actually saw. Evaluation also needed to follow the revised task definition. Historical star-derived labels could not evaluate the new text question, so we built a development set using the same text criterion. Starting with 200 Yelp candidates, we screened overlap with historical material and conducted two model-assisted review passes. We admitted 194 and held six because of ambiguity or disagreement. Development data supports examining models and comparing approaches. Calibration data fits probability adjustments. A final test supports confirmation after the approach is fixed. These 194 records served only the first purpose. The training entry point used an explicit allowlist of sources to reject them as training data. This separation makes comparison more meaningful, but screening cannot rule out all semantic overlap or pretraining exposure. Agreement between two model-assisted passes is not independent human ground truth. Holding difficult cases aside also leaves some difficult situations outside the reported scores. Those limits matter when interpreting the Yelp results below. What the two trained versions tell us After the repair, I trained fresh from Base at 0.8B and 4B, naming them B1 and B2. Both used fresh LoRA parameters and readouts, the same repaired training data and two epochs. Their learning rates were 0.0001 and 0.00005 respectively. This compares two training recipes; it cannot isolate the effect of parameter count. Before training 4B, I measured forward, backward and optimizer updates on representative and stress inputs on the 3090. LoRA reduces trainable parameters, but backbone weights, intermediate activations and input length still consume memory. Loading the model is only part of a training-capacity check. Evaluation used two development sets: the 194 text-based review judgments described above, and historical multitask development data covering classification, question answering and rule composition. The second row below counts 1,184 original questions; another 204 constructed variants, including reordered options, are excluded from this table. | Development data | Correct, 0.8B (B1) | Correct, 4B (B2) | B1 mean NLL | B2 mean NLL | | --- | ---: | ---: | ---: | ---: | | Yelp text-based attitude judgments | 184/194 | 192/194 | 0.09620 | 0.04024 | | Original historical multitask questions | 951/1,184 | 1,033/1,184 | 0.60133 | 0.46749 | These are outputs without temperature calibration, at temperature 1. B2 did better on the correct-answer counts and mean NLL shown here. The table compares two trained versions. Without an untrained control on the same sets, it cannot quantify the gain from fine-tuning itself or establish that general capabilities were retained. Another slice is closer to the risk of automated action. On the Yelp set, B1 made 181 predictions with confidence at least 0.9, including one error. B2 made 188, including two errors. The samples are small and cover different subsets, so this is not a significance claim. It does illustrate why better averages do not automatically mean fewer high-confidence errors, and why 0.9 is not a ready-made production threshold. Longer runs also need real recovery Moving from tiny experiments to longer runs required another engineering change: training recovery. Loading LoRA and readout weights allows prediction to continue. Continuing the same training run also requires the optimizer, learning-rate progress, random state and data position. We compared uninterrupted training against a run terminated mid-process and resumed. After adjusting determinism settings on the actual CUDA path, both completed eight optimizer updates with bit-identical trainable parameters and training state. Reloaded probabilities on 30 check questions were identical too. This validated process-interruption recovery in that single-GPU environment. It provided a recoverable starting point for further iteration; the test did not cover retraining on different hardware or physical power loss. Saving and comparison details are in the notes below. JevBench exposed the next problems to solve Training, saving and loading worked, and results on the existing development sets had improved. The next question was how well the model would handle a different set of practical decisions. We evaluated the unchanged 4B version locally on a fixed public JevBench revision. That checkout contained 231 downloadable public questions. Every request returned a valid probability vector without input truncation. Breaking the results down by the benchmark's groups revealed a clear gap: | Public JevBench group | Correct | | --- | ---: | | Easy | 48/48 | | Original | 69/72 | | Hard | 53/111 | The overall result was 170/231, but 58 of the 61 errors came from Hard. A further breakdown by task family showed 0/15 on temporal and numeric questions and 4/19 on long-policy judgments. Handling simple decisions did not establish reliability on date calculations, combined conditions, exception clauses or evidence distributed across a long document. The overall score hid these weaknesses. That gave the subsequent training a more specific direction. First, inspect task conversions, options and labels item by item to understand the failures. Then add independently checkable date calculations, rule combinations and exception cases, progressively increasing the context the model needs to process. Keep original-task training data and evaluations alongside them to check whether improving difficult questions costs performance on decisions the model could already handle. Length was one factor to investigate: the original 4B training configuration limited context to 384 tokens, while these long-policy contexts contained roughly 2,203–3,666 tokens. However, the longer inputs also all belonged to Hard. Length and difficulty were entangled, so the results could not establish that length alone caused the failures. Labels and conversions in these small task-family groups also still needed item-level review. These are local public-question diagnostics, not an official sealed evaluation or leaderboard rank. Differences between task groups also do not establish a regression caused by training. The inspected public questions can track known weaknesses, but independent reserved data is needed to confirm whether improvements generalize. Quality optimization is still underway. Future posts will examine which training changes help these weaknesses, which do not, and whether performance on other tasks is preserved. Model downloads and data reconstruction The models and accompanying materials are available on Hugging Face: - Qwen3.5 0.8B decision model (experiment B1) - Qwen3.5 4B decision model (experiment B2) - Training-data indexes, label repairs and reproduction materials Each model repository includes the LoRA adapter, decision readout, native loader and training configuration. Loading retrieves the separately pinned Qwen Base. These are inference checkpoints, excluding optimizer, random-state and other training-resume snapshots. Follow the repository's loading instructions so the decision readout is included. The data repository provides source indexes, label repairs, reconstruction scripts and a training reproduction kit, without redistributing original review or question-answer text. Reconstruction retrieves pinned upstream data and verifies the final file's SHA-256. Training and evaluation roles are recorded separately, alongside source attribution and component-specific terms. <details> <summary>Data versions and evaluation scope</summary> One record can contain several questions. The data versions were: | Data stage | Records | Questions | | --- | ---: | ---: | | Initial training data passing input checks | 12,576 | 15,576 | | Interim data after record quarantine | 12,520 | 15,514 | | B1/B2 training data incorporating Yelp repair | 12,520 | 15,476 | Of the 56 quarantined records, six involved public cross-split near-copy risks, 39 unresolved ambiguity and 11 reviewed label conflicts. The move from 15,514 to 15,476 questions withheld 38 Yelp judgments from supervision without changing the retained record count. These checks reduced identified risks; they do not estimate the whole dataset's label-noise rate. Training files had their own checksums and manifests. A copied manifest still describing old data was replaced before launch. Development data had an evaluation-only declaration, while training also checked an explicit source allowlist. A legacy entry point without that allowlist could not provide the required separation. The historical multitask data was called the retention panel in the records. It contained 1,388 questions. The table uses 1,184 original questions, marked clean in the scoring records. The other 204 comprise 84 option-permutation variants, 60 none-option controls with the answer present among the candidates, and 60 with the answer absent. Here, none means "none of the above." These variants are excluded from the table's correct-answer counts and mean NLL. Historical labels and overlap remain limitations, so these results do not establish independent external-test performance. </details> <details> <summary>Training configuration and recovery</summary> | Configuration | 0.8B (B1) | 4B (B2) | | --- | --- | --- | | Pretrained model | Qwen3.5-0.8B-Base | Qwen3.5-4B-Base | | Initialization | Fresh LoRA and readout | Fresh LoRA and readout | | Training records | 12,520 | Identical file bytes | | Epochs and optimizer updates | 2; 3,130 | 2; 3,130 | | Physical batch and accumulation | 1; 8 | 1; 8 | | LoRA rank | 16 | 16 | | Learning rate | 0.0001 | 0.00005 | | Backbone storage and autocast | FP32; BF16 | FP32; BF16 | | Token budget per pass | 4,096 | 8,192 | | Random seed | 3 | 3 | The original comparison verified that all admitted 4B inputs already fitted the 4,096 budget; using 8,192 did not change actual pass packing. Each recipe had this one run, so it cannot estimate training-seed variation. The earlier tiny run used physical batch 1, accumulation 4 and 80 updates. During training, a record can produce augmented variants with distractor candidates or none-of-the-above controls. The training objective averages question losses within each variant, then sums according to variant weights. The result tables average NLL over the selected original questions, so the aggregation units are not interchangeable. An earlier 0.8B full-mixture model was named B0. B1 was trained fresh from Base to avoid carrying forward B0's old supervision history. Recovery saves trainable parameters, optimizer state, scheduler, random state, counters and the next data position. Saves occur after a complete optimizer update, excluding unfinished gradient accumulation. Several checksummed save generations allow fallback. The first CUDA recovery comparison showed small numerical drift. With determinism enabled on the actual path, uninterrupted and resumed runs matched in parameters, optimizer, scheduler, random state and data position. </details> <details> <summary>Implementation and measurement sources</summary> Local training started from Jared Palmer's KEV, pinned to commit 2ecafc49fe1e8945cd600443e6c04c1cf0b3a78f, adding local training-recovery support without loading a trained KEV adapter. The pretrained models were Qwen3.5-0.8B-Base and Qwen3.5-4B-Base. See Hu et al. for LoRA and Guo et al. for background on calibration and temperature scaling. This implementation runs each question with its context as an independent causal sequence on the Qwen3.5 hybrid backbone, excluding other questions from its context. Batching does not reproduce Jev's architecture or latency. The tiny experiment used the reference PyTorch compute path actually verified in that run. Selected aggregate measurements support checking the counts and configurations here. That file contains summary values only; model downloads, data reconstruction and training reproduction materials are in the Hugging Face repositories above. This article was prepared with AI assistance from local experiment records. It covers the initial B1/B2 versions, their corresponding development results, and the 4B public JevBench diagnostic that motivated further optimization. </details>

@GOROman x.com これです。ローカルでLLMより早く曖昧条件分岐したいなって思って。今はkevってやつ使ってます。 jaredpalmer/kev

@Ohki ありがとうございます！いくつか手当たり次第に試したんですが、kevってやつが日本語の解釈が良かったのでとりあえずそれ選びました jaredpalmer/kev

@Bober_smart I combined it with kitze/skillbox and prompts.chat and jaredpalmer/kev and several applications via mcp. 😂 Utterly monstrous.

The open Jev clones: three ways to read a decision, and a leaderboard that does not transfer Within a week of TypeSafe AI opening early access to Jev, its closed "System One" decision model, GitHub was full of open look-alikes, and by October an independent board, JevBench, was ranking more than a hundred systems on sealed items. I needed one of these models to answer decisions for Loom, my agent harness, so the question for this post is practical: which open clone should sit behind that endpoint, and can I trust the public ranking to tell me? The clones split into three mechanism families, and the two I studied most closely, Kev and Laya, sit in different ones. JevBench ranks Kev-4B 37th with a score of 39.9, behind decider-4b at 55.8 and behind a stock Qwen3-4B-Instruct read at its option logits at 54.1. When I scored those models on my own panel, by coverage at 5% error (the share of questions a model can answer on its own while keeping its error on those under 5%; defined in the series overview), decider-4b tied Kev-4B at 0.556 against 0.555, and the stock Qwen readout covered 0.091. The ranking did not transfer, and Kev-4B stayed the pinned model. What the clones are copying Jev takes a state (text or JSON) and a list of typed questions, and returns probabilities instead of text. A Choice question picks one of up to 255 options and returns a probability per option, a Score question rates against ordered levels, and a Noul returns a yes/no probability. TypeSafe says it is a transformer trained on synthetic data and post-trained with "Reinforcement Learning for Calibrated Decisions" (RLCD); there is no paper, no weights and no parameter count. The most useful public guess comes from Archer Hume's black-box probing of roughly 10,000 API calls. Latency tracks input tokens rather than option count, and content injected into one question does not leak into another, which fits a causal transformer that encodes the state once and runs each question as an isolated branch over that shared prefix (the pattern from Hydragen and DeFT), ending in a softmax over option logits with no text generated. A minority view (djev) holds that Jev is a diffusion model; Hume argues diffusion is not needed to explain what the API does. Three ways to read a decision Every open clone has to answer the same two questions: what hidden state does the decision head read, and how does that become a distribution over options? The answers fall into three families. Pointer head on a causal LLM (Kev by Jared Palmer, and TinyJev). Kev packs one sequence per request: <state> ... <q> instruction <opt> option1 </opt> <opt> option2 </opt> ... <decide>, with one <q> ... <decide> block per question. The delimiters reuse rarely used Qwen special tokens, so no embeddings are resized. A block-causal mask lets each question's tokens attend to the state and to their own question block only, and position ids restart after the state for every question, so questions share the state but cannot read each other. The head has no vocabulary readout at all: two linear projections map the <decide> hidden state and each option's </opt> hidden state into a 256-dimensional space, the scaled dot products become one logit per option, and a softmax over options gives the distribution. The base (Qwen3.5 at 0.8B, 4B or 9B) is frozen with a rank-16 LoRA adapter. Qwen3.5 interleaves Gated DeltaNet layers, which are recurrent and cannot honor a block mask, so on those bases each question runs as its own causal row continuing a cached state prefix, and a cache miss costs one forward pass instead of one per question. Calibration is a single post-hoc temperature per checkpoint (2.41 for the Kev-4B I pin). Mask-marker encoder (Laya by Nandakishor Mukkunnoth of Convai Innovations). Laya is an encoder, ModernBERT-large at 421M parameters. Each question becomes one sequence, [CLS] <qtype> instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP], with a [MASK] token inserted before every option. A two-layer transformer head runs over the encoder output, and a small shared scorer turns the hidden state at each [MASK] position into one logit per option; a softmax over those, with a temperature per question type and option-count bucket, gives the probabilities. All of a request's questions are padded into one batch and answered in one call. Because every option has to fit in a fixed token budget, the option count is capped: on Banking77's 77 labels each option gets about 3 to 4 tokens. Training uses a strictly proper scoring rule (log score plus spherical score, plus a ranked probability term for ordinal questions) as a policy-gradient reward, which Laya's documentation calls RLCD. First-token logit readout (SemIf, formerly OpenJev, and jevfire). These need no training. The options are written as single tokens (letters A, B, C and so on), a stock LLM runs one forward pass, and the decision is a softmax over the logits of those candidate tokens at the first answer position, the same trick as FIRST (arXiv 2406.15657) for listwise reranking. jevfire adds shared-prefix KV in vLLM so many questions over one state stay cheap. Two failure modes recur: ambiguous single-letter tokens (space-prefixed against byte-level) can silently corrupt the readout, and letter-slot variants run out of single tokens at about 26 to 52 options. Kev and Laya on their own numbers On sources it never trained on, Kev-9B reaches 0.852 test accuracy against Jev's 0.857 on the development split, a comparison Kev's own README flags as uncontrolled. The gap that remains is coverage at 5% error: Kev covers 0.45 to 0.57 against Jev's 0.70. Kev's author tried to close it during training (label smoothing, Brier and focal losses), and that round came back negative under its pre-registered gate. Laya reports beating Jev on its own benchmark (typed-decisions accuracy 0.766 against 0.727, ECE 0.081 against 0.246), but every Jev number in those comparisons is a published third-party figure, so each one is cross-study. One independent test put Laya at 0.590 against Jev's 0.974, the lowest accuracy of the clones it compared and the fastest at about 30 ms. Laya's documentation is open about its own losses: Banking77 at 0.425 to 0.492 against Jev's 0.870, and an English checkpoint that scores 0.000 on Khmer at 0.952 mean confidence. What JevBench says JevBench v1.5.4 is the one board that runs many clones on the same items: 106 ranked systems, 1,624 items of which 720 are sealed, on evaluator-owned GPUs. "Intelligence" is a chance-adjusted accuracy, and the composite rank also weights speed and cost. | Rank | System | Base | Intelligence | Choice ECE | |---|---|---|---|---| | 1 | Cygnet | frozen Gemma-4-12B-it, one-token readout | 71.1 | 0.041 | | 2 | Winnow-12B | Gemma-4-12B-it LoRA | 74.4 | 0.085 | | 3 | Jev 1.13.0 | closed | 72.0 | 0.027 | | 7 | decider-4b v2 | Qwen3.5-4B-Base | 55.8 | 0.049 | | 21 | raw Qwen3-4B-Instruct-2507 | stock, option logits | 54.1 | 0.329 | | 37 | Kev-4B | LoRA + pointer head | 39.9 | 0.278 | | 93 | Laya | ModernBERT-large | 0.0 | 0.149 | The top open system is a first-token readout on a frozen 12B model, and a 4B model read the same way with no training at all beats Kev-4B by 14 points, which on the board's face meant I was serving the wrong model. Provenance narrowed the candidates before any measurement. Nimble's pipeline contains a bridge that calls the Jev API although its README says it was not distilled from Jev, JevK5 is reported to include GPT outputs, and several "jev-distill" models train on Jev outputs. decider-4b (by Mapika, Apache-2.0) states a local Qwen3.5-27B teacher and nothing distilled from Jev, which made it the best-placed 4B candidate. The raw Qwen readout was the training-free control. K2: scoring the board's picks on my panel The board's task mix is not Loom's: Loom asks short questions, and it needs a model that knows when not to answer more than one that answers most things right. The board's Kev row also describes a Qwen3-4B base, while I pin jaredpalmer/kev-4b at revision 139fdd9, so I could not tell whether it was the same model. I pre-registered a check, K2, before running anything: score decider-4b and the raw Qwen readout on the same panel Kev-4B had already been scored on, and swap only if a candidate beats Kev-4B by at least 5 points of macro coverage at 5% error with a bootstrap lower bound above zero, on both the full panel and the panel without emotion and sciq (decider's training mixture touches those two). The panel is the development split of Kev's frozen transfer-v9 eval suite: 11 tasks and 1,046 questions. Both new arms ran on my GPU box (a 16 GB RTX 4070 Ti Super) through decider's own server, the raw Qwen arm through its chat readout layout with no temperature, so that arm measures the stock model plus decider's prompt and readout. The Kev-4B and Laya rows are reused from the earlier K1 run. !Two bar charts side by side for the same four models. On JevBench intelligence, decider-4b scores 55.8, raw Qwen3-4B 54.1, Kev-4B 39.9 and Laya 0.0. On coverage at 5% error on my 11-task panel, decider-4b scores 0.556, raw Qwen3-4B 0.091, Kev-4B 0.555 and Laya 0.158 | Arm | 11-task macro | Difference from Kev-4B (lower bound) | 9-task macro | Difference (lower bound) | |---|---|---|---|---| | Kev-4B | 0.555 | | 0.549 | | | decider-4b | 0.556 | +0.1 pp (-9.1) | 0.518 | -3.2 pp (-13.2) | | raw Qwen3-4B | 0.091 | -46.4 pp (-54.7) | 0.000 | -54.9 pp (-64.7) | | Laya | 0.158 | -39.7 pp (-46.6) | 0.131 | -41.8 pp (-50.5) | Nothing came close to the swap rule, so Kev-4B stayed. The raw Qwen readout answers 72% of the panel correctly and still covers only 0.091, because at temperature 1.0 its confidence does not separate right answers from wrong ones. No threshold keeps its error under 5% on any task except sciq, where every arm covers 1.000. JevBench's 54.1 rewards accuracy, while my swap rule rewards knowing when to answer. decider-4b and Kev-4B tie on the macro with opposite strengths: | Task | Kev-4B coverage | decider-4b coverage | Kev-4B accuracy | decider-4b accuracy | |---|---|---|---|---| | unknowable_control | 1.000 | 0.691 | 0.955 | 0.827 | | composition_holdout | 1.000 | 0.750 | 0.979 | 0.896 | | buried | 0.375 | 0.075 | 0.700 | 0.675 | | legacy_holdout | 0.800 | 0.662 | 0.812 | 0.738 | | paws | 0.275 | 0.188 | 0.787 | 0.713 | | qnli | 0.825 | 0.925 | 0.912 | 0.925 | | mmlu | 0.350 | 0.575 | 0.725 | 0.725 | | tweet_offensive | 0.312 | 0.588 | 0.738 | 0.787 | | emotion | 0.163 | 0.463 | 0.550 | 0.762 | | mmlu_pro | 0.005 | 0.205 | 0.560 | 0.505 | | sciq | 1.000 | 1.000 | 0.975 | 0.988 | decider-4b is better on knowledge and classification tasks (mmlu, mmlu_pro, qnli, tweet_offensive, and emotion, which its training mixture may have seen). Kev-4B is better on buried evidence, held-out rule compositions and above all unknowable_control, the questions where the deciding evidence was removed and answering is the error. That is the task that matters most to Loom, and Kev-4B covers 1.000 of it against decider-4b's 0.691. decider-4b answered at 48 ms p50 against Kev-4B's 553 ms, but the two ran on different serving stacks, so that gap mixes model and server. Running decider took three fixes on the GPU box: its server needs flash-linear-attention and causal-conv1d, so it ran in WSL; its CUDA-graph warm-up did not fit 16 GB, so it ran eager; and the WSL image could not live on a nearly full system drive, because filling it corrupted the distro once. Where this leaves the pick The readout family decides what the head reads (option tokens in the vocabulary, a pointer between two hidden states, or a marker in a bidirectional encoder) and with it the option cap and the cost per question. The top open system on JevBench is a first-token readout, yet on my panel a training-free first-token readout covered 0.091 because its confidence could not tell right from wrong. For a caller that has to abstain, what matters is whether the model's confidence orders right answers above wrong ones, and JevBench's intelligence axis does not measure that. A per-task mix of Kev-4B and decider-4b looks tempting given the opposite strengths, but the earlier K1 run found only +1.2 points of oracle headroom for per-task selection with Laya as the second model, and this panel was not built to choose between models, so any follow-up needs fresh items. K2 kept Kev-4B as the pinned model in verdict, the decision service Loom calls.

Qualcuno ha provato cose simili con altri modelli JEV-like o con KEV-4B? sono curioso di capire quanto regge lo zero-shot su immagini più difficili ciao a tutti!! sono Joe, da 4 anni lavoro nell'AI come ricercatore e AI engineer. vi racconto un esperimento che ho fatto in questi giorni e che è finito anche in un video: l'ho girato di getto, giusto per rompere il ghiaccio e darmi finalmente una mossa ad aprire il canale youtube che volevo iniziare da tempo. nel video si vede tutto dal vivo. munito del mio fidato mac mini M2 e di qualche gb di ram (8 per la precisione), ho tirato su in locale KEV (jaredpalmer/kev), un modello JEV-like uscito pochissimo dopo l'annuncio di JEV. usa lo stesso identico protocollo "typesafe". Gli dai in input una descrizione del task, una query e delle opzioni, e lui ti dà in output la probabilità che la query sia collegata a ognuna delle opzioni, senza generare testo. KEV però è stato allenato con SFT solo a leggere testo. il suo backbone (qwen3.5 0.8b) perà la rete di vision ce l'ha già ma in KEV è disattivata!! allora l'altro giorno per curiosità gliel'ho ridata, senza riallenare niente. per gioco gli ho fatto classificare in che bidone va quello che passa davanti alla webcam e devo dire funziona sorprendentemente bene. la capacità di questi modelli di zero shottare i task mi sorprende ogni volta. detto questo, è un esperimento e non un benchmark: spesso il margine tra le opzioni è piccolo (sulla mia faccia not_waste vince con circa il 46%). per questo la domanda del titolo, qualcuno ha provato cose simili? mi piacerebbe capire fin dove si può spingere. so che openai ha rilasciato il proprio decision API con vision tipo. BTW il canale l'ho aperto per passione e per documentare alcuni progetti. parlerò di temi tecnici sull'AI mettendo le mani in pasta nel codice il più possibile. Se siete interessati fatemi sapere qualche tema che potrebbe interessarvi! I prossimi video saranno più ordinati e organizzati meglio. se ci sono temi di cui vi piacerebbe che parlassi, scrivetemelo!! se volete appena mi arriva il nuovo mac da 24 gb potrei fare un video sugli altri KEV!! se siete interessati stay tuned! video: youtube.com script e notebook: gist.github.com

Jev とは何か。Jevもどき(Open Weight の代替 14 構成)を比較というか調査してみた。 :::note INFO Claude CodeでFable5.1に調べてもらった内容をもとにしているので参考程度で見てください。 Jevのことは知りたい、クラウドAPIはセキュリティ的に微妙だしローカルでやりたいし…というモチベーションです。 主に私が忘れないための備忘目的になります。 ::: 2026 年 9 月に TypeSafe AI が公開した Jev は、「文章を生成しない AI モデル」として話題になりました。公開から 3 週間で Jev 互換の Open Weight モデルが数十種類登場し、独立ベンチマークも 2 つ立ち上がっています。 この記事では、Jev が何をするものか、入力と出力、使い方を整理したうえで、Open Weight の代替モデルがどういう作りになっているかを 3 つの方式に分けて説明します。後半では、Jev 形式の共通サンプル 76 問を自作し、手元の RTX 4090 で 14 構成を動かした結果を載せます。 情報は 2026 年 10 月 9 日時点のものです。各モデルの性能数値の多くは開発元の自己申告で、独立計測は Benchmark Heaven の JevBench と、手元で動かした結果だけです。 Jev とは 一言でいうと「判断だけを返す関数」 Jev は、入力 (state) と型付きの質問を渡すと、テキストではなく選択肢ごとの確率分布を返すモデルです。分類、ルーティング、採点、真偽判定、ガードレールなど「コードが直接消費する判断」に特化していて、チャットやコード生成には使えません。 TypeSafe AI はこれを "System One model" という新カテゴリの第 1 号と位置づけています。名前の由来は Kahneman『ファスト&スロー』の「速い直感的思考 = System 1」です。CEO の Diogo Almeida は元 OpenAI で RLHF や InstructGPT に関わった人物で、「チャットは超人的になったのに、なぜ自動化は進まないのか」という問いからこのモデルを作ったと説明しています。 | 項目 | 内容 | |---|---| | 公開 | 2026-09-15 (早期アクセス)。同時に $40M のシード調達 | | 現行版 | jev-1.13.0 | | ライセンス | プロプライエタリ。重み、アーキテクチャ、論文とも非公開 | | 学習法 | RLCD (Reinforcement Learning for Calibrated Decisions) と呼ぶ独自手法。合成データのみ | | 価格 | 入力 $0.042 / 100 万トークン。出力トークンは無料 | | コンテキスト | 1 リクエスト 64k トークン (state + 最長の質問は 32k 以内) | | 入力 | テキストのみ (文字列 / JSON / 配列)。画像や音声は不可 | | レイテンシ | 公称 70〜500 ms。第三者計測では p50 で 236〜301 ms | | 言語 | 英語が主。CJK は「扱えるが精度は劣る」と公式に注記 | | カスタマイズ | ファインチューニングの提供なし。state と質問文で制御する | 入力と出力 エンドポイントは 1 つだけです。 POST https://api.typesafe.ai/v1/systemone { "state": "Help! My payouts have been failing for 3 days.", "model": "jev-latest", "questions": { "is_urgent": { "type": "noul", "instructions": "Does this convey urgency?" }, "department": { "type": "choice", "instructions": "Which team should handle this?", "criteria": { "billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations", "sales": "Pricing, upgrades, new accounts" } }, "frustration": { "type": "score", "instructions": "How frustrated is the customer?", "criteria": ["Calm", "Frustrated", "Very angry"] } } } 質問のタイプは 3 つしかありません。 | タイプ | 意味 | 制約 | 返り値 | |---|---|---|---| | noul | Yes/No 命題の真偽 | なし | noul (0〜1、Yes の確率) | | choice | 順序のない選択肢から 1 つ選ぶ | 最大 255 選択肢 | choice、probabilities、confidence | | score | 順序付きルーブリックで採点 | 2〜10 段階 | score (期待値)、legend、probabilities、confidence | レスポンスはこうなります。 { "model": "jev-1.13.0", "answers": { "is_urgent": { "type": "noul", "noul": 0.95 }, "department": { "type": "choice", "choice": "billing", "probabilities": { "billing": 0.88, "technical": 0.12, "sales": 0.0 }, "confidence": 0.81 }, "frustration": { "type": "score", "score": 1.05, "legend": { "0": "Calm", "1": "Frustrated", "2": "Very angry" }, "probabilities": { "0": 0.0, "1": 0.95, "2": 0.05 }, "confidence": 0.92 } }, "usage": { "input_tokens": 296, "output_tokens": 20 } } モデル本体が返すのは質問ごとの確率分布だけで、choice は argmax、score は確率加重平均、confidence は分布の集中度から固定式で計算される派生値です。 設計上の重要な性質が 2 つあります。 - 全質問は並列かつ独立に評価されます。質問 A の答えに質問 B は依存できません。依存がある場合は 2 回に分けて呼びます。 - state は 1 回だけ読まれるので、質問を増やしてもレイテンシはほぼ増えません。公式クックブックでは、13 問を 1 リクエストにまとめると 12.2 倍安く 10 倍速いと報告されています。 このため公式ガイドは「1 つの質問には 1 つの判断だけ」を推奨しています。「ピッチを採点」ではなく「市場規模」「技術的実現性」「差別化」を別々に score し、重みはコードで管理する、という使い方です。 flowchart LR subgraph IN["入力"] S["state<br/>文字列 / JSON / 配列"] Q["questions<br/>noul / choice / score を N 問"] end subgraph JEV["Jev 本体 (重み・層構成は非公開)"] T["Transformer<br/>state を1回だけ読む"] P["全質問を同時に評価<br/>(質問間に依存なし)"] D1["質問1の確率分布"] DN["質問Nの確率分布"] end subgraph API["API サーバ"] C["choice = argmax<br/>score = 期待値<br/>confidence = 分布の集中度"] end S --> T Q --> T T --> P P --> D1 P --> DN D1 --> C DN --> C C --> OUT["answers"] 使い方 Python SDK ではこう書きます。 from typesafe_sdk import TypeSafeClient, Noul, Choice, Score client = TypeSafeClient(model="jev-1.13") # 環境変数 TYPESAFE_API_KEY を読む result = client.system_one( {"ticket": "I was charged twice for order A-104. Please refund the duplicate."}, { "refund_requested": Noul(instructions="Does `ticket` request a refund?"), "department": Choice( instructions="Which team should handle this?", criteria={"billing": "Payments, refunds", "technical": "Bugs, outages", "other": None}, ), }, ) if result.nouls["refund_requested"].noul > 0.5: ... TypeSafe 直接のほか、Vercel AI Gateway (typesafe-ai/jev)、Cloudflare Workers AI (typesafe/jev)、OpenRouter、Requesty からも呼べます。 公式が推奨するパターンは次の 5 つです。 | パターン | 内容 | |---|---| | Speculative fan-out | 必要になりそうな質問をまとめて 1 回で投げ、コード側で必要なものだけ使う | | Confidence-gated routing | 答えと confidence を分け、低 confidence は人間レビューや上位 LLM へ | | Composite scoring | 複合判断を原子的な score に分解し、重みはコードで持つ | | Intent routing | 入力を分類し、決定論的ハンドラ / 専門 LLM / 人間に振り分ける | | Hybrid agent | 難しい判断は LLM、定型的な判断は Jev に任せてコストとレイテンシを下げる | 公式が認めている弱点 TypeSafe は jev-1.13 の弱点リストを公開しています。要約すると「意味的な判断は得意だが、計算と推論はコードに任せろ」です。 - 文字どおりに読む。意図を汲まない - 算術、カウント、日付の比較は不得意 - 多段の間接参照や二重否定で精度が落ちる - 無関係な情報が多い大きな state で精度が落ちる - プロンプトインジェクションの影響を受ける。セキュリティ境界として使ってはいけない - choice の選択肢順序に依存することがある (先頭に寄りやすい) 第三者評価も、この見立てと整合しています。Banking77 (77 クラス) では Gemini 3.8 Flash 83.6% に対して Jev 79.5% で、Jev は約 6.7 倍安く約 6 倍速い、というのが典型的な結果です。「ハルシネーション 0%」という宣伝文句は「スキーマ違反が構造的に起きない」という意味で、「整形された間違い」は起きます。 Open Weight の代替モデルと 3 つの方式 Jev の重みは非公開ですが、「state と型付き質問を受け、選択肢の確率分布を返す」という外形は簡単に真似できます。実際、公開から数週間で Jev 互換のモデルが数十種類登場しました。大半は既存の Open LLM (Qwen3.5 / Gemma 4) のトルソを流用し、生成ヘッドの代わりに「選択肢の確率を読み出す部分」を載せたものです。作りは 3 種類に分かれます。 | 方式 | 仕組み | 例 | |---|---|---| | A. 読み出しヘッド置換 | LM ヘッドを捨て、<answer> 位置と各選択肢位置の hidden state を照合する小さなヘッドを学習 (LoRA 併用) | Strands Decider 2B、Kev、Imajev、NeoHorse、JEV-27B、Clef、d1-3B | | B. 選択肢文字のロジット読み出し | プロンプトに A/B/C... を並べ、次トークンのロジットを softmax。LoRA 微調整 + 温度キャリブレーション | JevK5、Quyet、H2O-Lightning、open-alternative-jev (学習なし) | | C. エンコーダ + 決定ヘッド | ModernBERT 等の双方向エンコーダに決定ヘッドを付けて学習 | Laya、Quyet-Small、d1-omni-600M | 方式 A は、ヘッドが選択肢ごとのパラメータを持たないので選択肢数に上限がなく、「1 番目を選びやすい」バイアスを学習しにくいのが利点です。方式 B は構造変更ゼロなので vLLM などの標準推論基盤がそのまま使えますが、選択肢が文字ラベルに縛られます (JevK5 は 16、Quyet は 10 まで)。方式 C は数億パラメータで圧倒的に速い一方、世界知識と推論力が乏しくなります。 主なモデル | モデル | 開発元 | ベース / サイズ | ライセンス | 特徴 | |---|---|---|---|---| | Strands Decider 2B | AWS strands-labs | Qwen3.5-2B-Base + LoRA + pointer head、1.9B | Apache-2.0 | 学習データとスクリプトまで全公開。Strands Agents のツール呼び出しゲートに統合済み | | Kev | jaredpalmer | Qwen3.5 0.8B / 4B / 9B、Qwen3.8-27B | Apache-2.0 | GitHub 8.7k スター。TypeSafe の Python SDK をそのまま流用できる | | JevK5 | allebee | Qwen3.5-4B / 9B + 蒸留 LoRA | Apache-2.0 | GGUF 版があり llama.cpp で CPU でも動く | | Imajev-4B | mohit67890 | Qwen3.5-4B + アダプタ | Apache-2.0 | 写真 + アプリ状態 + 型付き質問を入力できる | | NeoHorse-Jev-4B | TokenRhythm | NeoHorse-1-4B | Apache-2.0 | prefill-only。テキスト + 画像 1 枚 | | H2O-Lightning-4B | H2O.ai | Qwen3.5-4B 微調整 | Apache-2.0 | JevBench v1.6.1 の公式 1 位。vLLM + shim で動く | | Quyet 1.0 | Chinh Nguyen | Gemma-4-31B-it + LoRA (Large) から 153M のエンコーダ (Small-EN) まで 5 サイズ | Apache-2.0 | Large は JevBench の Intelligence で Jev を上回る。1 問 10 択まで | | Laya | Convai Innovations | ModernBERT-large 421M、多言語版は mmBERT-base 322M | Apache-2.0 | 33 ms。Jev と同じ RLCD で学習したと説明。ゼロショット性能は低く微調整前提 | | LiquidAI d1 | Liquid AI | LFM2.5-VL-3B (d1-3B)、LFM2.5-Encoder-350M + 視覚/音声 (d1-omni-600M) | LFM 1.0 (独自) | エッジ向け。RTX 4090 で 1 問 8 ms、Jetson AGX Thor で 16 ms | | Clef / Clef-Flash | Cloudflare | Qwen3.8-27B / Qwen3.5-9B + joint schema head | Apache-2.0 | 全質問の全選択肢を一括スコア。テキスト、JSON、画像、動画を入力できる | | JEV-27B | AutoTrust | Qwen3.8-27B + 109M の System 1 ブロック | Apache-2.0 | Jev の出力分布を蒸留。KL 0.017 で「区別不能」と主張 | | open-alternative-jev | ikermoel | 任意の Open LLM (ライブラリ) | Apache-2.0 | 学習なしで確率を取り出す。選択肢順序に敏感 | 同名の別プロジェクトが多いので注意してください。「OpenJev」と名の付くものは 5 つ以上あり、kyegomez/open-jev はランダム重みの研究実装で実用できません。openjev/openjev は CC-BY-NC で商用不可です。 独立ベンチマーク JevBench の見方 Benchmark Heaven が運営する JevBench は、Jev クラスのモデルを自前のハードウェアで動かして測る独立ベンチマークです (TypeSafe とは無関係)。Intelligence (チャンス補正精度)、Calibration、Speed、Cost の 4 軸の等重み調和平均が公式スコアで、Cost は「1,000 判断あたりの USD」です。 | システム | 公式順位 | JevBench Score | Intelligence | Calibration | $/1k 判断 | p50 | |---|---|---|---|---|---|---| | Jev 1.13.0 (参照) | - | 71.5 | 64 | 91 | $0.032 | 0.24 s | | H2O-Lightning-4B v1.1 | 1 | 72.5 | 60 | 90 | $0.021 | 0.2 s | | decisio v0.8.0 (gemma-4-31B-it) | 2 | 71.7 | 70 | 89 | $0.041 | 0.2 s | | Quyet-1.0-Large | 3 | 71.4 | 73 | 90 | $0.045 | 0.4 s | | JevK5 v0.3 | 35 | 37.4 | 39 | 90 | $0.017 | 0.2 s | | JEV-27B | 56 | 21.3 | 57 | 91 | $0.199 | 0.36 s | | Laya | 120 | 0.0 | 2 | 63 | $0.003 | 1.8 s | v1.6.1 (2026-10-06) の値です。読むときの注意が 3 つあります。 - 4 軸の等重み平均なので、27B クラスは Intelligence が高くても Cost で大きく減点されます (JEV-27B の Capability は 73.9 で Jev の 77.1 に迫るのに公式 56 位)。 - 問題プールと採点法がバージョンごとに変わり、順位が大きく入れ替わります。Imajev-4B は v1.4.2.2 で 1 位、v1.6.1 で 47 位でした。 - 別に Decision Index (Cloudflare がリーダーボードを運営) もあり、こちらでは Clef 61.21、Jev 57.91 と各社が自己申告しています。2 つのベンチの順位は一致しません。 手元で 14 構成を同じ 76 問で動かした 公開されている数字はほぼ自己申告なので、Jev 形式の共通サンプルを自作して、手元で動く Open Weight モデルに同じリクエストを投げました。 サンプルの設計 46 item、76 問 (noul 38、choice 21、score 17)、うち日本語 9 問です。Jev の得意分野 (サポート振り分け、返金ポリシー照合、含意判定、12 択の意図分類、感情スコア、モデレーション、エージェントのツール呼び出しゲート) に加えて、公式の弱点リスト (無関係な長文、プロンプトインジェクション、算術、日付) を試す item を少数入れました。choice を含む item は選択肢を逆順にしたリクエストも投げて、順序感度を測ります。 { "state": { "policy": "Refund policy: a refund is allowed only if (1) the purchase was made within the last 30 days AND (2) the customer has a receipt. Digital goods are never refundable.", "case": "E-book (digital download) bought 3 days ago, receipt attached." }, "questions": { "refund_allowed": { "type": "noul", "instructions": "According to `policy`, is a refund allowed for `case`?" } } } 環境 CPU 32 コア、RAM 62 GB、RTX 4090 です。まず CPU で全構成を測り、GPU が空いた後に同じ構成を GPU で再計測しました。正答率は Kev-0.8B と so1 の各 1 問を除いて CPU と GPU で一致したので、以下の正答率は CPU 計測、レイテンシは GPU 計測 (bf16、item 単位の中央値) の値です。Jev 本家は API キーが無いため未計測です。 結果 | モデル (方式) | 正答率 | noul | choice | Brier | ECE | 順序反転で答え変化 | 日本語 9 問 | p50 GPU (ms) | |---|---|---|---|---|---|---|---|---| | Imajev-4B (A) | 92.1% | 94.7% | 95.2% | 0.112 | 0.066 | 0% | 88.9% | 40 | | Clef-Flash 9B (A 変種) | 90.8% | 92.1% | 100% | 0.099 | 0.078 | 0% | 77.8% | 45 | | JevK5-4B (B 学習済み) | 90.8% | 94.7% | 100% | 0.128 | 0.083 | 0% | 88.9% | 29 | | Kev-4B (A) | 89.5% | 89.5% | 100% | 0.101 | 0.048 | 0% | 88.9% | 30 | | NeoHorse-Jev-4B (A) | 89.5% | 89.5% | 100% | 0.095 | 0.027 | 0% | 88.9% | 24 | | d1-3B (A 相当) | 88.2% | 89.5% | 100% | 0.121 | 0.054 | 0% | 66.7% | 11 | | Strands Decider 2B (A) | 84.2% | 84.2% | 100% | 0.174 | 0.109 | 0% | 77.8% | 25 | | Quyet-Medium 4B (B 学習済み) | 81.6% | 89.5% | 71.4% | 0.104 | 0.083 | 0% | 88.9% | 26 | | d1-omni-600M (C) | 80.3% | 71.1% | 95.2% | 0.175 | 0.067 | 0% | 77.8% | 5 | | Kev-0.8B (A) | 76.3% | 71.1% | 95.2% | 0.205 | 0.147 | 0% | 77.8% | 23 | | Quyet-Small 328M (C) | 71.1% | 68.4% | 66.7% | 0.219 | 0.109 | 6.7% | 88.9% | 6 | | Laya (C) | 69.7% | 65.8% | 95.2% | 0.315 | 0.098 | 4.8% | 55.6% | 8 | | so1 + Qwen2.5-1.5B ゼロショット (B) | 64.5% | 60.5% | 95.2% | 0.459 | 0.304 | 14.3% | 55.6% | 9 | | Quyet-Small-EN 153M (C) | 57.9% | 57.9% | 66.7% | 0.309 | 0.089 | 0% | 44.4% | 6 | Quyet 系の choice が低いのは、1 問 10 択までという制約で 12 択の item を全部拒否したためです (拒否は不正解扱い)。 Brier と ECE は「付いてくる確率を信じてよいか」を測る指標で、どちらも 0 が最良です。Brier は確率分布と正解 (正解に 1、他に 0) の二乗誤差の平均で、正解を選んでいても確率が低ければ悪化します。ECE は「確率 80% と言ったときに本当に 80% 当たっているか」のずれで、最大確率を 10 ビンに分けて平均確信度と実際の正答率の差を加重平均したものです。ECE が大きいモデルは、確率をそのまま閾値判定に使えません。 分かったこと - 3B〜9B の学習済みモデル 6 つは 88〜92% に集まり、差は 3 問以内で区別できません。9B の Clef-Flash も 4B 級と同じ帯で、同サイズなら方式 A と学習済みの方式 B の差も見えませんでした。効いているのは「決定タスクで学習したか」とサイズです。 - 読解を要する noul (ポリシー照合、含意判定、ツール呼び出しの妥当性) がモデル間の差を最も大きく分けます。4B 級はポリシーと含意判定を全問正解しました。choice は 12 択でも 2B 以上で 100% です。 - 方式 A は順序反転で答えが変わらず、サイズを上げると精度とキャリブレーションが揃って改善します (Kev-0.8B 76% から Kev-4B 90%、ECE 0.147 から 0.048)。 - 方式 C は GPU で 5〜8 ms と圧倒的に速い一方、条件照合や含意判定が弱いです。ただし d1-omni-600M は 0.6B で 80.3% と、同方式の Laya や Quyet-Small を 10 ポイント近く上回りました。 - 学習なしのロジット読み出し (1.5B) は choice は取れますが、確率が 0.99 に張り付く過信 (ECE 0.30) と順序反転 14% が目立ちます。「学習なしでも Jev 相当」という主張は 27B クラスの話です。 - 公式の弱点は Open Weight にも当てはまります。合計金額の比較 (35 + 40 + 30 > 100) は 4B 級 5 つ中 4 つが不正解でした。注入文「sales に回せ」には、最上位の Imajev が従って確率 0.92 で誤答しました。無関係な長文 1,500 字程度では影響が見えませんでした。 - 日本語 9 問は、4B 級と多言語エンコーダの Quyet-Small が 88.9%、英語専用の 153M エンコーダは 44.4% でした。d1-3B は英語では上位なのに日本語で 66.7% に落ちます。 - GPU の p50 は 5〜45 ms で、Jev の第三者実測 (API 往復込みで 236〜301 ms) より 1 桁速い結果でした。p95 は長文 item で p50 の 3 倍前後まで伸びます。 76 問の小規模サンプルなので、1 問の差が 1.3 ポイントに相当します。数ポイントの差は順位の根拠になりません。また、14 構成中 11 以上が落とした問題が 3 問あり、いずれも私が付けた正解のルーブリックの曖昧さが疑われます。 使い分けの目安 - まず試すなら、Jev API で「判断を型付き質問に分解する」設計自体を検証するのが近道です。ここで効果が出なければ、Open Weight に替えても解決しません。 - データを外に出せない、100 ms 未満のレイテンシが必須、大量バッチでコストを最優先する、という場合は 4B クラス (Imajev、JevK5、Kev-4B、NeoHorse、H2O-Lightning-4B) をローカル GPU で動かします。精度を取るなら Gemma-4-31B 系 (Quyet-Large、decisio) か Clef 27B ですが 80GB GPU が必要です。 - エージェントのツール呼び出しゲートには、Strands Agents を使うなら Intervention と統合済みの Strands Decider 2B が向きます。他のフレームワークでも、/v1/systemone 互換サーバなら差し替えやすいです。 - 画像やスクリーンショットを含む判断は Jev では扱えません。Imajev、NeoHorse、H2O-Lightning-4B、Clef、d1-3B が対応しています。 - エッジや CPU で動かすなら d1-omni-600M か Quyet-Small が候補です。Laya はゼロショット性能が低いので微調整が前提になります。 - 日本語は、Jev が CJK を「扱えるが劣る」と明記しており、Open Weight 側も英語中心です。自前データでの検証が必須です。 どのモデルを選ぶにしても共通する注意点があります。型安全は「無効な値を返さない」保証であって「正しい値を返す」保証ではないので、ラベル付きデータで精度と confidence の閾値を必ず検証してください。算術、日付比較、カウント、多段推論はコード側か LLM 側に残します。choice の選択肢順序を入れ替えて答えが安定するか確認し、敵対的入力に対するセキュリティ境界としては使わないでください。 参考リンク - TypeSafe ブログ「Introducing System One Models & Jev」 typesafe.ai - TypeSafe Docs (API reference、Jev 1.13 jaggedness) docs.typesafe.ai - JevBench (Benchmark Heaven) benchmarkheaven.com - Decision Index (Cloudflare) clef-evals.workers-ai-mle.workers.dev - Strands Decider 2B strandsagents.com - Kev jaredpalmer/kev - JevK5 allebee/jevk5 - Imajev mohit67890/imajev - NeoHorse-Jev-4B huggingface.co - H2O-Lightning-4B huggingface.co - Quyet 1.0 huggingface.co - Laya huggingface.co - LiquidAI open d1 huggingface.co - Cloudflare Clef huggingface.co - 手元ベンチマークのサンプル、ランナー、全結果 (リポジトリ URL は公開時に追記)