EleutherAI

Stop waiting for these companies to announce opportunities. Start watching their research GitHub organizations. Here are some worth following: Hardware & GPU research github.com/NVlabs github.com/NVIDIA github.com/ROCm github.com/AMD github.com/intel Big Tech research github.com/google-research github.com/google-deepmind github.com/microsoft github.com/facebookresearch github.com/amazon-science AI research labs github.com/openai github.com/anthropics github.com/deepseek-ai github.com/QwenLM github.com/allenai github.com/mistralai github.com/EleutherAI github.com/LAION-AI Cloud & infrastructure github.com/awslabs github.com/GoogleCloudPlatform github.com/cloudflare github.com/vercel-labs github.com/databricks

The best Arabic LLM for your enterprise is the one that passes your eval, not the leaderboard Four serious Arabic models now come out of the Gulf: Falcon-H1-Arabic from Abu Dhabi, ALLaM from Saudi Arabia, Fanar 2 from Qatar and Jais 2 from the UAE. None of them can be picked from a leaderboard, because the Open Arabic LLM Leaderboard measures academic Arabic, not your tickets, your policy documents or your customers' dialect. The two things that do transfer to your organisation are both measurable in about a week: a 200 to 500 item eval set built from your own material, and the tokenization economics that decide what each Arabic conversation really costs in money, latency and GPU memory. If you are building Arabic-language customer service, a government services assistant or an internal knowledge bot in the GCC, you face a decision that looks like a model choice but is actually two measurements. First: does a regional open-weight model score within your tolerance on your content, with your product names, place names, dialects and policy language? Second: what does an Arabic token actually cost you, in API spend, time to first token, KV-cache footprint and concurrent sessions per GPU? Answer both and the decision makes itself. Skip them and you will either overpay a frontier API for years or ship a model that fails quietly on Gulf dialect. The field in late 2026 Four model families are credible candidates for enterprise Arabic work today. All of them are open weight unless noted, which means you can run them on your own GPUs and keep data in-country. | Model | Developer | Sizes | Context | Licence | What the evidence says | | --- | --- | --- | --- | --- | --- | | Falcon-H1-Arabic | TII, Abu Dhabi | 3B, 7B, 34B | 128K (3B), 256K (7B, 34B) | Open weights (check the model card for your size) | About 71.5% (7B) and 75.4% (34B) on the Open Arabic LLM Leaderboard, beating 70B-class transformers including Llama-3.3-70B and Qwen2.5-72B. Hybrid Mamba-Transformer architecture, runs on vLLM. | | ALLaM | SDAIA, now run by HUMAIN, Saudi Arabia | 7B open checkpoint; 34B flagship powers HUMAIN Chat | 4,096 (7B) | Apache 2.0 (7B Instruct preview) | The 7B Instruct preview is trained from scratch on 4T English tokens plus 1.2T mixed Arabic/English, with two published revisions (v1.27.2.25 and v2). ALLaM 34B backs HUMAIN Chat, hosted end to end in the Kingdom and PDPL-compliant. | | Fanar-2-27B-Instruct | QCRI, Qatar | 27B | 32,768 | Apache 2.0 | Continually pretrained from gemma-3-27b-pt on roughly 166B Arabic and English tokens over 75,000 H100 hours. Explicit MSA plus Gulf, Levantine and Egyptian dialect support, native Arabic reasoning traces, tool calling and hallucination mitigation. | | Jais 2 | Inception (G42), Cerebras and MBZUAI, UAE | 8B, 70B | 8,192 | Apache 2.0 | Trained from scratch on 600B+ curated Arabic tokens plus 1.6T of web, math and code, with a custom 150,272-entry Arabic-centric vocabulary. Leading results among evaluated open models on OALL2 and AraGen. Note: the hosted API was retired on 21 August 2026; weights remain on Hugging Face. | Primary sources: the Falcon-H1-Arabic announcement and release blog, the Fanar-2-27B-Instruct model card, the ALLaM-7B-Instruct-preview model card, the HUMAIN Chat launch and the Jais 2 documentation. The fourth candidate in every real procurement is a frontier API model prompted in Arabic. It belongs in the comparison, because on raw Arabic quality it often wins. The question is by how much, and what that gap costs. Why the leaderboard cannot pick your model The Open Arabic LLM Leaderboard is genuinely good work. It runs reproducible evaluations through the EleutherAI lm-evaluation-harness Arabic task set, so a score is a score, not a vendor claim. TII even documents that its own vLLM-based runs land within about one point of the leaderboard's numbers. But read what it measures: knowledge, reading comprehension and reasoning over web-derived Arabic text, mostly Modern Standard Arabic. Your workload is not that. It is a customer writing "ما أقدر أرسّع الفاتورة من التطبيق" (I can't pay the bill faster from the app, in Gulf dialect), a code-switched complaint ("الـ bundle خلص، كيف أعمل top-up؟"), and questions about a product name that appears nowhere in any pretraining corpus. Leaderboard rank measures none of the things that will make your assistant succeed or embarrass you. This is not an argument against the leaderboard. It is the screening stage: it tells you which models are worth a week of your team's time. The final stage happens on your data. What transfers: build a 300-item eval set in three days The single highest-leverage artefact in an Arabic LLM project is a small, versioned eval set built from your own material. Two hundred items is a floor, five hundred is plenty; beyond that the marginal information per item collapses. Where the items come from: 1. Real tickets and chat logs (redacted). Take them verbatim, dialect and typos included. Never rewrite them into MSA, because your users will not. 2. Policy and product documents. Turn paragraphs into questions: "What is the fee for an early exit from this plan?" 3. Named entities. Product names, service tiers, districts, mall names, government portals. Models hallucinate most confidently where pretraining data is thinnest, and nothing is thinner than your internal naming scheme. 4. Deliberately hard slices. Aim for roughly 40% formal MSA, 30% Gulf dialect, 30% code-switched Arabic/English, in the proportions your traffic actually shows. A concrete mix for a 300-item set, drawn from a telecom support case: - 90 items from the last two years of Arabic chat logs, sampled by intent cluster (billing, top-up, coverage, cancellation), including Emirati and Saudi dialect items - 60 questions whose answers live only in your policy PDFs - 60 entity-grounded questions where the correct answer must quote your exact product or tariff name - 45 dialect and code-switched items, verbatim from logs, never normalised - 45 adversarial items: questions about services you do not offer, questions that mix a real product name with a wrong assumption, and date and amount arithmetic from a bill That is one analyst-day to sample, one subject-matter-expert day to write gold answers, one more to review. Gold answers should be short and checkable, not essays. The scoring sheet and the judge Score each response on five criteria, 0 to 2 each, so the maximum is 10 per item and a 300-item set yields a 3,000-point scale: | Criterion | 0 | 1 | 2 | | --- | --- | --- | --- | | Correctness | Wrong answer | Partially right or right with wrong entity | Fully right, correct entity and amount | | Grounding | Invents a policy, product or price | Vague, not traceable to source | Every claim traceable to your documents | | Dialect handling | Misunderstands dialect or code-switching | Understands but answers in stiff MSA | Registers the register, answers naturally | | Format and language | Wrong language, broken Arabic | Usable but unnatural | Natural, correctly punctuated Arabic | | Completeness | Misses a step or an explicit caveat | Usable | Complete, including edge cases | Two ways to score: a human reviewer (expensive, about 1 to 2 minutes per item, so a full pass on 300 items is a day) or a stronger model as judge. If you use a model as judge, check agreement first: have the human score a 50-item overlap sample and compute percent agreement on the Correctness criterion. Below roughly 85% agreement, your judge is not measuring your rubric, and you must fix the prompt before trusting it at scale. Here is a copy-paste judge prompt that has the properties that matter: it forces evidence, it forbids generosity, and it outputs a parseable score. You are evaluating a customer-service assistant's reply in Arabic. QUESTION (verbatim from a real user, may contain Gulf dialect or Arabic/English code-switching): {question} GOLD ANSWER (written by a domain expert): {gold_answer} ASSISTANT'S REPLY: {reply} Score each criterion 0, 1 or 2. Be strict: a score of 2 must be clearly earned, not merely plausible. 1. correctness: 2 = matches the gold answer's facts, entities and amounts; 1 = partially right; 0 = wrong. 2. grounding: 2 = every claim is traceable to the gold answer or the supplied policy text; 1 = vague; 0 = invents facts. 3. dialect: 2 = correctly understands the question's register and replies naturally in it; 1 = understands but replies in stiff MSA; 0 = misunderstands the dialect or code-switching. 4. language: 2 = natural, correct Arabic; 1 = awkward but usable; 0 = broken Arabic or wrong language. 5. completeness: 2 = covers steps and caveats; 1 = usable with gaps; 0 = misses something essential. Reply ONLY with JSON: {"correctness": n, "grounding": n, "dialect": n, "language": n, "completeness": n, "total": n, "fatal_error": "<one line, or 'none'>"} Run every candidate model, including the frontier API baseline and its Arabic-prompted variant, on the identical set with the identical retrieval context if you use RAG. Report the mean total, the per-criterion breakdown, and the fatal-error rate. The per-criterion breakdown matters more than the total: a model that loses two points per item entirely on dialect handling might be fixable with prompt work, while one that loses them on grounding needs retrieval surgery, and one that loses them on correctness needs to be dropped. For dialect specifically, published per-dialect evaluations such as DialectalArabicMMLU show how much scores move between MSA and regional varieties, and TII reports AraDice dialect sub-scores for the Falcon-H1-Arabic family. Your own dialect slice settles the question for your dialect, which is the only one that matters. The second thing that transfers: tokenization economics Arabic text costs more tokens than the equivalent English text on English-centric tokenizers, and that premium appears in four places at once: your API bill, time to first token, KV-cache memory and the number of concurrent Arabic sessions a GPU can hold. The peer-reviewed evidence is blunt. Petrov and colleagues showed that the GPT-4 tokenizer encodes the same content in Standard Arabic with roughly 3.04 times as many tokens as English, and the ChatGPT tokenizer with about 4.40 times as many, with direct consequences for cost, latency and effective context length (arXiv:2305.15425). The original Jais team measured the same pattern at the word level: GPT-2's vocabulary fragments Arabic into about 4.17 tokens per Arabic word against roughly 1.1 per English word, while the bilingual Jais vocabulary achieves about 1.05 (fertility figures summarised here). Newer multilingual vocabularies have narrowed the gap, but a 1.5 to 2 times premium on an Arabic-inefficient tokenizer is still normal, and nothing on your invoice labels it. Measure it on your own traffic before believing any ratio. Ten lines of Python gives you fertility, tokens per word, on a sample of your real conversations: from transformers import AutoTokenizer text = open("sample_arabic_conversations.txt", encoding="utf-8").read() words = len(text.split()) # Verified repo IDs; add the other candidates from their model cards for repo in ["QCRI/Fanar-2-27B-Instruct", "humain-ai/ALLaM-7B-Instruct-preview"]: tok = AutoTokenizer.from_pretrained(repo) fertility = len(tok(text).input_ids) / words print(f"{repo}: {fertility:.2f} tokens per Arabic word") One caution: studies count words and normalise text differently, so compare ratios within your own sample rather than against published absolutes. Your sample is the number that will hit your bill. Worked example: what the premium actually costs Take a 400-word Arabic support answer and compare a model whose tokenizer gives 2.4 tokens per Arabic word against an Arabic-optimised model at 1.5. Same content, same job. Spend. The answer is 960 tokens on the first tokenizer, 600 on the second, a 1.6 times premium on every request. At an illustrative $5 per million output tokens, one million such answers costs $4,800 versus $3,000, a $1,800 difference before you have written a word of prompt engineering. Multiply by your volume. And note the trap when comparing providers: a model priced 30% lower per token that fragments Arabic 1.6 times more is not 30% cheaper for you, it is roughly 12% more expensive. Latency. Prefill is compute-bound on prompt length, so time to first token scales with the token count of your system prompt plus retrieved context. Decode is per token, so an answer fragmented into 1.6 times the tokens streams 1.6 times longer at the same hardware speed. Prompt tuning cannot remove this; it is in the vocabulary. KV cache and concurrency. Per-token KV-cache size for a 7B-class model with 32 layers, 8 grouped-query KV heads and 128-dimension heads at fp16 is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB per token. A 4,000-token Arabic conversation at the efficient tokenizer holds about 500 MB of KV per session; the same content at 1.6 times fertility holds about 800 MB. On a GPU with 60 GB free after model weights, that is roughly 120 concurrent sessions versus 75. Tokenization, not parameter count, decided your concurrency. Context. A 256K context window (Falcon-H1-Arabic 7B and 34B) holds far more Arabic at a low fertility than a longer window holds at a high one. This is why the Jais 2 team built a custom 150,272-entry Arabic-centric vocabulary and report it makes inference efficient: the efficiency is not marketing, it is arithmetic you can reproduce with the snippet above. The decision rule Now the choice reduces to one comparison on one spreadsheet. 1. Run all candidates on your eval set. Suppose the frontier API scores 9.1 out of 10 and Falcon-H1-Arabic 7B scores 8.6. The gap is 0.5 points. 2. Set your tolerance in advance, in the currency of your use case: acceptable fatal-error rate on billing questions, acceptable dialect-mishandling rate. Do not set it after seeing the scores. 3. If the open model is inside your tolerance: run it on your own GPUs. You keep every conversation in-country, you pay a fixed infrastructure cost instead of a token bill that scales with Arabic verbosity, and you are not exposed to an API being retired (Jais 2's hosted endpoint went away in August 2026; its weights did not). A regional open model that scores within tolerance on your data is strictly the better enterprise choice: cheaper per request, in-country data, and no dependency on someone else's pricing page. 4. If the gap exceeds your tolerance: do not pay for it blindly. Quantify the frontier model's advantage in cost per resolved request: with the 1.6 times tokenization premium on an English-centric tokenizer and your ticket volume, what is the API bill for the extra 0.5 points? Then price the alternative: the same eval gap might close with a better Arabic system prompt, stronger retrieval into your policy documents, or three days of fine-tuning the open model on 2,000 of your own tickets. Very often the fine-tune is cheaper in month four than the API premium is in month one. A practical note on keeping this sustainable: whatever prompts and gold answers survive the eval, version them. The Arabic system prompts that work, the dialect-handling instructions, the few-shot examples, these are assets, and they are exactly the kind of asset that walks out the door in a Markdown file. We keep them in a shared, versioned prompt library (PromptForge), and when we build and run these assistants for clients, from eval set through deployment, that is our AI engineering work, done by our own team here in Dubai. What to do next 1. This week: pull 300 real items using the mix above. It is three days of one analyst and one domain expert, and it is reusable for every model you will ever evaluate. 2. In parallel: run the fertility snippet on 100 of your real conversations for every shortlisted model, including the frontier API's tokenizer. You now have the two numbers that transfer. 3. Next week: run all candidates through the judge prompt, with a human agreement check on 50 overlapping items. Build the table: eval score, fatal-error rate, tokens per request, cost per resolved request. 4. Then decide with the rule above. If the regional open model is inside tolerance, it wins on data residency and unit economics, and GCC regulators and your security team will thank you. If it is not, you now know exactly what the gap costs per month, which turns a vague preference into a negotiation. The best Arabic LLM for enterprise is not a fixed answer, because "enterprise" is not a fixed workload. It is whichever of Falcon-H1-Arabic, ALLaM, Fanar 2, Jais 2 or a frontier API passes an eval set built from your own tickets, at a token cost you have measured rather than assumed. Both measurements take a week. The alternative is guessing, and guessing is the most expensive part of any AI project. --- Originally published on Azrty.

【Windows 11】AIでクラシックピアノMIDIを生成｜Ariaローカル環境構築 この記事を書いた理由 Ariaを使ってクラシックピアノのMIDIをAI生成したい 公開されている構築はLinux環境が中心で、Windowsでの構築手順が見つけにくかった クラウドサービスを使わず、手元のPCでローカル実行したい NVIDIA RTX4060 VRAM 8GBで動作確認 　2026年10月26日 前知識 Ariaについて Ariaは、EleutherAI が公開している、ピアノ演奏の MIDI を生成する音楽生成モデルです。 LLaMA 3.2（1B）アーキテクチャをベースとしており、約60,000時間のフィルタリングされたソロピアノ録音をMIDIに書き起こしたデータで学習しています。 このデータは実際の演奏を書き起こしたものなので、実際の演奏に由来するタイミングや強弱などの情報を学習しています。 データセットも公開されています。 上記の通りAriaは既存のデータセットに比べて非常に大規模なものとなっています。 データセット／モデル 公開年 データ規模 MAESTRO 2018〜2019年 約200時間 GiantMIDI-Piano 2022年 約1,237時間 Aria（事前学習データ） 2025年 約60,000時間 ※ データセットとモデルの学習データを比較しているため、厳密に同じ条件での比較ではありません。 得意なことと注意点 Ariaはゼロからの生成よりも既存のピアノMIDIの続きを作ることが得意とされています。 また、入力はピアノ1トラックが最適とされています。 Ariaは入力の質に敏感です。 主に大量の演奏データから音楽のパターンを学習するモデルで、文章生成AIで使われるRLHF（人間のフィードバックによる強化学習）のような指示追従の追加学習は行われていないとGitHub公式にも記載されています。 また、「明るい曲調で」「ソナタ形式で」といったテキストでの指定はできません。 できることはプロンプトの選び方と温度などのサンプリング設定となります。 曲全体の構成を計画して作るわけではなく、次の音を連続して予測するモデルのため、曲全体の構成が常に保証されるわけではありません。（とはいえ通常はほとんど気になりません） Music Transformerとの違い 音楽生成AIの原点はMusic Transformer（2018年）と考えられます。 Music TransformerはYouTubeのピアノ録音を書き起こした約10,000時間で学習されています。 こちらは2020年の記事ですが、ぜひ読んでみてください。 qiita.com Music TransformerはTensorFlow系のライブラリやその他設定が必要であるため、今回は環境構築のしやすさからPyTorch系で実装できるAriaを用いています。 実装（仮想環境） 実行環境 OS：Windows 11 GPU：NVIDIA GeForce RTX 4060（VRAM 8GB） Git Anaconda（もしくはMiniforgeなど Condaが動く環境） 仮想環境はこの後手順を記載しているので事前の用意は不要です。 仮想環境：Conda Python：3.11 PyTorch：2.6.0（CUDA 12.4対応） CUDA Toolkit：12.4（V12.4.131） Transformers：4.57.6 1. アナコンダプロンプトを開く Condaを使用するため、Anaconda Promptを開きます。Miniforgeを使用している場合は、Miniforge Promptを開きます。 見当たらない場合はWindowsのアプリ検索から探してください。 Anaconda、もしくはMiniforgeが入っていない場合は各自入れてください。 他に用途がない場合は軽量のMiniforgeがおすすめです。 2. 仮想環境を作成 今回はpiano-aiという仮想環境を作成します。 Pythonのバージョンは3.11です。 conda create -n piano-ai python=3.11 -y conda activate piano-ai 3. PyTorchとCUDAをインストール 本記事では、動作確認済みのPyTorch（CUDA 12.4対応）を使用します。別のCUDA対応バージョンでも動作する可能性はありますが、本記事では検証していません。 pip install torch --index-url download.pytorch.org GPUの動作確認 python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" 以下のように表示されればOKです。 True NVIDIA GeForce RTX 4060 4. Transformersをインストール "transformers<5"と記載するようにしてください。 最新バージョンの5.x系（5.20.2）ではエラーを確認しています。 pip install "transformers<5" バージョンを確認します。 pip show transformers 今回の環境では 4.57.6 を使用しています。 5. Aria関連ライブラリをインストール Gitをインストールしておく必要があります。 pip install git+EleutherAI/aria-utils これで仮想環境の構築は完了です。 実装（コード、入力MIDIデータ） 作業フォルダの中身 作業フォルダを作ります。（piano-ai フォルダを作成） ここに入力するMIDIと出力されるMIDI、pythonを設置します。 どこでも良いですが、私はユーザーディレクトリ直下に作りました。 フォルダ名も何でも良いです。 C:\Users\ユーザー名\piano-ai 作成したフォルダの中を以下の構造にしてください。 generate_piano.pyのコードは後述します。 piano-ai/ ├── generate_piano.py ├── input_midi/ │ 　└── 入力用.mid ※ └── output_midi/ input_midi...入力するMIDIを入れる output_midi...出力されるMIDIが生成される ※ input_midiの中に入力するMIDIを入れておいてください。ファイル名の指定はありません。 ここに入れるMIDIのデータを前奏として使いその後の続きを生成するため、前述の通り質の良いMIDIデータが理想とされています。 MIDIをお持ちでない方は、公式GitHubのリポジトリにexample-promptsというフォルダがあり、その中にMIDIがありますのでそちらを用いてください。 generate_piano.py（167行）AI加筆ですが内容精査済みです。ご確認なさってください。 """ Aria（ピアノMIDI続き生成）を transformers 経由で動かすスクリプト。 使い方: 1. このファイルの隣にある input_midi フォルダにピアノMIDI（.mid / .midi）を入れる 2. python generate_piano.py 3. 番号で入力MIDIを選び、作る本数を入力 4. 結果は output_midi フォルダに保存される 公式CLI（aria generate）との対応は、下の「設定」のコメントを参照。 """ import time from datetime import datetime from pathlib import Path import torch from huggingface_hub import hf_hub_download from safetensors.torch import load_file from transformers import AutoModelForCausalLM, AutoTokenizer ============================================================ 設定 ============================================================ BASE_DIR = Path(file).resolve().parent INPUT_DIR = BASE_DIR / "input_midi" # 続きを作りたい元のMIDIを置く場所 OUTPUT_DIR = BASE_DIR / "output_midi" # 生成結果の保存先 公式CLIの --checkpoint_path に相当（こちらはHugging Faceから自動取得） MODEL_NAME = "loubb/aria-medium-base" どの重みを使うか。 "base": from_pretrained が標準で読む model.safetensors（事前学習モデル） "gen" : 同じリポジトリ内の model-gen.safetensors（生成品質向けに追加学習したもの）で上書きする WEIGHTS = "gen" 公式CLIの --prompt_duration（秒指定）に相当する部分。 こちらは秒ではなくトークン数で切る。512トークンは、HFページのクイックスタートの値。 PROMPT_TOKENS = 512 公式CLIの --length 2048 に近い長さ。 HFの max_length は「プロンプト込みの総長」なので、プロンプト分を引いた新規トークン数で指定する。 TOTAL_TOKENS = 2048 NEW_TOKENS = TOTAL_TOKENS - PROMPT_TOKENS 公式CLIの --temp 0.98 に相当 TEMPERATURE = 0.98 公式CLIの --min_p 0.035 に相当。 「最も確率の高い候補の3.5%未満の確率しかない候補は捨てる」という足切り。 MIN_P = 0.035 ============================================================ def load_gen_weights(model) -> None: """model-gen.safetensors を読み込み、モデルの重みを gen に差し替える。 キー名がモデルと一致しないと、差し替えが黙って失敗して base のままになる。 それを防ぐため、読み込み結果を必ず表示し、不一致があれば止める。 """ path = hf_hub_download(MODEL_NAME, "model-gen.safetensors") state_dict = load_file(path) result = model.load_state_dict(state_dict, strict=False) print(f"gen重み: {len(state_dict)} 個のテンソルを読み込み") print(f" unexpected_keys（重み側にあるのにモデルに無い）: {len(result.unexpected_keys)} 個 {result.unexpected_keys[:5]}") print(f" missing_keys（モデルにあるのに重み側に無い）: {len(result.missing_keys)} 個 {result.missing_keys[:5]}") if result.unexpected_keys: print("キー名が一致しないため、genの重みは正しく適用されていません。") print("この表示をそのまま共有してください。") raise SystemExit(1) if result.missing_keys: print("注意: missing_keys が lm_head など共有される重みだけなら問題ありません。") def pick_prompt_file() -> Path: """input_midi 内のMIDIを一覧表示して、番号で選ばせる。""" midi_files = sorted( set(INPUT_DIR.glob(".mid")) | set(INPUT_DIR.glob(".midi")) ) if not midi_files: print(f"入力MIDIがありません: {INPUT_DIR}") print("input_midi フォルダにMIDIファイルを入れてください。") raise SystemExit(1) print("\n入力するMIDIを選択してください:") for i, path in enumerate(midi_files, start=1): print(f"{i}. {path.name}") while True: choice = input("\n番号を入力: ").strip() try: index = int(choice) - 1 if 0 <= index < len(midi_files): return midi_files[index] except ValueError: pass print("一覧にある番号を入力してください。") def ask_variations() -> int: """公式CLIの --variations に相当。一度に作る本数を聞く。""" while True: text = input("作る本数（Enterで1）: ").strip() if text == "": return 1 if text.isdigit() and int(text) >= 1: return int(text) print("1以上の整数を入力してください。") def main() -> None: INPUT_DIR.mkdir(exist_ok=True) OUTPUT_DIR.mkdir(exist_ok=True) if not torch.cuda.is_available(): print("CUDAが使えません。PyTorchのCUDA版が入っているか確認してください。") raise SystemExit(1) prompt_path = pick_prompt_file() variations = ask_variations() print(f"\n入力: {prompt_path.name}") print("モデルを読み込んでいます...") fp16で読み込む（8GB VRAM向け） model = AutoModelForCausalLM.from_pretrained( MODEL_NAME, trust_remote_code=True, dtype=torch.float16, ).cuda().eval() tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) if WEIGHTS == "gen": load_gen_weights(model) print(f"使用する重み: {WEIGHTS}") prompt = tokenizer.encode_from_file(str(prompt_path), return_tensors="pt") input_ids = prompt.input_ids[..., :PROMPT_TOKENS].cuda() print(f"プロンプト: {input_ids.shape[-1]} トークン") for n in range(1, variations + 1): print(f"\nMIDIを生成しています... ({n}/{variations})") start = time.time() with torch.no_grad(): output = model.generate( input_ids, max_new_tokens=NEW_TOKENS, do_sample=True, # 確率に従ってランダムに選ぶ（これがないと毎回同じ結果になる） temperature=TEMPERATURE, min_p=MIN_P, top_p=1.0, # 1.0 = 無効。足切りは min_p だけにする top_k=0, # 0 = 無効。HFは何も指定しないと top_k=50 が働くことがあるため明示的に切る use_cache=True, ) timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") output_path = OUTPUT_DIR / f"{prompt_path.stem}_{timestamp}_{n:02d}.mid" tokenizer.decode(output[0].tolist()).to_midi().save(str(output_path)) print(f"生成完了 ({time.time() - start:.0f}秒): {output_path}") if name == "main": main() 入力側のMIDIの名称は自由で、複数配置可能です。 コードを実行した際にどの入力を使うかを選択可能です。 選択した入力で何個出力するかを指定可能です。 モデル等の記載がありますが、基本触らなくて大丈夫です。 トークン数や温度などはコードの中身から変更してください。 これで準備完了です。あとは実行するだけで生成可能です。 モデルや環境はローカルに保存されるため次回からは以下の実行手順を行なうだけで生成が可能です。 実行手順 Anaconda Prompt（Miniforgeの場合はMiniforge Prompt）を開く 仮想環境を有効化する conda activate piano-ai 作業フォルダに移動（それぞれの作業フォルダへ） cd C:\Users\ユーザー名\piano-ai スクリプトを実行 python generate_piano.py 画面の案内に従って、入力するMIDIファイルと生成する本数を選択する。 生成されたMIDIファイルは output_midi フォルダに保存されます。 参考 EleutherAI/aria huggingface.co arxiv.org Music Transformer magenta.tensorflow.org

Run your LLM evals every night on a GPU that only exists for a minute Most eval suites run when someone remembers to run them. That's usually right before a release, which is the worst time to find out a model regressed two weeks ago. The fix is boring: run the evals every night, on a GPU that exists only while they run, and read the number in the morning. This post sets that up with Runcrate's CLI: one command creates a nightly schedule, each run is a job on a fresh GPU that stops when the eval exits, and a run costs about a cent for the example here. I ran every command below on 2026-10-21 and the outputs are copied from my terminal. The eval I'm using EleutherAI's lm-evaluation-harness on a small model so the run is quick: Qwen/Qwen2.5-0.5B-Instruct on 200 questions of ARC-Easy. Swap in your own model and tasks; the scheduling part doesn't change. The command that runs on the GPU machine: pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && \ uv pip install -q "lm-eval[hf]" torch --torch-backend=cu128 && \ lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct \ --tasks arc_easy --limit 200 --batch_size 16 The first line makes a clean virtualenv with uv, so the eval doesn't fight with whatever Python packages the machine image ships. --torch-backend=cu128 picks the PyTorch build that matches the machine's CUDA 12.8 driver. I tried a plain pip install first and got a PyTorch wheel that refused the driver, so pin this to your driver. Install the CLI and sign in brew install aeonmindai/tap/runcrate runcrate login login opens the browser. You pick a workspace and an environment, and everything the CLI starts runs and bills there. Create the schedule runcrate schedule create --cron '0 2 * * *' --tz Europe/London --name nightly-eval \ --gpu A6000 --max-runtime 30m -- \ 'pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && uv pip install -q "lm-eval[hf]" torch --torch-backend=cu128 && lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct --tasks arc_easy --limit 200 --batch_size 16' What it printed: Schedule created: nightly-eval ID: e18aeb6f-6d15-4f51-804b-65b258ce71e1 Kind: job When: 0 2 * * * (Europe/London) Overlap: skip Enabled: yes Next run: 2026-10-22 02:00 BST GPU: 1x A6000 Max runtime: 30m A few things worth knowing about those flags: - A command after -- makes it a job. At 02:00 London time Runcrate starts a fresh A6000, runs the command, and stops the machine when the command exits. Without a command it would start an instance instead. - --max-runtime is required. If an eval hangs, the job is stopped after 30 minutes. Nothing a schedule starts can run unbounded. - --tz keeps wall-clock time. 0 2 * * * in Europe/London stays at 02:00 local across daylight saving changes. - Overlap: skip is the default. If last night's run is somehow still going, tonight's is skipped instead of stacking a second machine. Don't wait until 2am to test it runcrate schedule run-now e18aeb6f-6d15-4f51-804b-65b258ce71e1 Started job nightly-eval (ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33). Follow it with: runcrate jobs logs -f ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 Run now starts the same job straight away and doesn't move the next scheduled run. I also let a schedule fire on its own while testing: it was due at 02:00 BST and launched at 02:02, which matches the docs (schedules are checked every 2 minutes). Read the result runcrate jobs logs prints the job's output. The end of it: hf ({'pretrained': 'Qwen/Qwen2.5-0.5B-Instruct'}), gen_kwargs: ({}), limit: 200.0, num_fewshot: None, batch_size: 16 | Tasks |Version|Filter|n-shot| Metric | |Value| |Stderr| |--------|------:|------|-----:|--------|---|----:|---|-----:| |arc_easy| 1|none | 0|acc |↑ |0.565|± |0.0351| | | |none | 0|acc_norm|↑ |0.560|± |0.0352| And runcrate jobs status for what it cost: Name: nightly-eval Status: ● succeeded Exit code: 0 GPU: 1x A6000 Max runtime: 30m Runtime: 1m3s Cost: $0.01 ($0.65/hr) Created: 2026-10-21T01:08:32.934+00:00 Started: 2026-10-21T01:12:50.771+00:00 Ended: 2026-10-21T01:13:53.945+00:00 About four minutes went to getting the machine ready and one minute to the eval itself. The max runtime counts from when the command starts, so setup doesn't eat into your 30 minutes. A6000 was $0.65 an hour when I ran this; check the live price before you pick a GPU. runcrate jobs list shows every night's run with its status, exit code and cost, so a week of runs is one command: ID NAME STATUS GPU EXIT RUNTIME AGE COST ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 nightly-eval ● succeeded A6000 0 1m 7m ago $0.01 Making it a real regression check The toy version prints a number. To make it catch regressions: 1. Evaluate your checkpoint, not a hub model. Attach a storage volume to the job and point pretrained= at the checkpoint under /workspace. Volumes are covered in the jobs docs. 2. Fail the job on a bad score. Add --output_path /workspace/evals to lm_eval, then a two-line Python check that exits non-zero when accuracy drops below your threshold. The job ends failed with that exit code, which is what you watch for. 3. Keep secrets out of the command. The command and logs are visible to everyone in the workspace. Put HF_TOKEN and friends in the environment's secrets (set in the dashboard), which every job receives as environment variables. Pause, run, delete runcrate schedule list runcrate schedule pause e18aeb6f-6d15-4f51-804b-65b258ce71e1 runcrate schedule delete e18aeb6f-6d15-4f51-804b-65b258ce71e1 -y Pausing stops future runs and leaves anything already running alone. Or ask your agent If you use Claude Code, Codex or Cursor, the Runcrate MCP server has the same schedule tools, so "every night at 2am London time, run my eval on an A6000 for at most 30 minutes" works as a prompt. Setup for each client is on the MCP setup page. Links - Schedules: runcrate.ai - Jobs: runcrate.ai - lm-evaluation-harness: EleutherAI/lm-evaluation-harness If you set this up for your own model and something gets in the way, tell me. Reply here or write to hello@runcrate.ai. Written with AI assistance. Every command and output above was run on 2026-10-21.

How Jev decision models work (and how to build your own on MLX) The web is bloated with Jev posts. The thing is a few weeks old, and my feeds are already full of it.. benchmarks, hot takes, threads about decision models being the future of agents. Jev, if you've managed to avoid the hype, is TypeSafe AI's hosted decision API where you send a situation and a few typed questions, it returns the answer with a probability for every option. Two spoilers, because I like roads with signposts. Classical ML never left the chat, and the newest tool in the box still doesn't answer the oldest question, whether you actually need it in the first place. --- At some point I stopped skimming news and got really curious about the machinery underneath. How does a model decide instead of write? What is actually different in there? The short answer turned into a project, and the project turned into a story about building a thing and then discovering, layer by layer (sorry), that most of it already existed. It's called take-my-layer. --- Init I'll be honest, I never thought I'd invented something. I knew reading probabilities off logits was a thing and the Jev posts themselves hint at it. But knowing a trick exists and holding it in your own hands are different things, and what I didn't know was how far it stretches. I took a 4-bit Qwen model on a laptop, no training, no labels, questions made up on the spot. That part was worth testing. So I went scouting before building anything serious. Kind of following my own advice from a post about fog of war as a mental model. The fog doesn't mean the territory is missing, it just means you haven't been there yet. Reading papers is scouting. Building a small proof of concept is scouting. I expected to find maybe one or two adjacent things. Well.. the territory I found was huuuge :D --- The trick is from 2021 First stop. Reading answer probabilities instead of generating answer text. This is how the whole evaluation world has scored decoder-only models for years.. it's the loglikelihood default in lm-evaluation-harness. The prompting version has names (verbalizers, pattern-verbalizer pairs, cloze-style prompting). PET (Schick & Schütze, EACL 2021) and "Small Language Models Are Also Few-Shot Learners" (NAACL 2021) are the classics, LM-BFF (ACL 2021) even automates the label-word search. What I'd been calling "reading the logits" has a standard name, a literature, a codified abstraction, and several thousand citations. The trick predates ChatGPT (obvious for people who are not the first day on the ML street). --- The bugs are from 2021 too The naive version has a bug called position bias. Ask "A, B, C or D?" and the model leans toward whichever option is listed first, regardless of the options. I found that myself the fun way when the raw read on my coding data predicted bash for 74 of 120 states. A bash magnet kind of. Its 0.57 accuracy was roughly bash's base rate in the data. Essentially every bug I hit someone had already hit: - Calibrate Before Use (Zhao et al., ICML 2021): content-free priors to correct the model's answer skew. - Order sensitivity of option lists (Pezeshkpour & Hruschka, 2023). - Fantastically Ordered Prompts (Lu et al., ACL 2022): same disease, few-shot demonstrations. - PriDe (ICLR 2024): the cheap permutation-plus-prior combination, which is more or less the cheaper cousin of my scheme. - Batch Calibration (ICLR 2024): the unified view of all of the above. My own fix (arrived at independently and with more cursing), was to list the options in several rotated orders, ask once per order, average. That's called permutation marginalization, and it is worth +4.7 points of agreement with the reference over the naive single-prompt read (5 answers flip toward the reference, 0 away). All of that was known. Standing on shoulders of giants, etc. --- Leaving early The next piece was speed. A full forward pass over a 4B model for a one-word answer is wasteful, and it turns out the decision signal sits pretty early. (Qwen3.5-4B has 32 blocks; layer selection kept picking 16 and 19, before the last full-attention blocks. The interesting states live in the linear-attention trunk.) The literature here is CALM (NeurIPS 2022), early exit per token by confidence. In my case it's a fixed half-depth exit for one fitted head, the layer-pruning family. Fine again, keep standing on the shoulders! In MLX land there is exactly one static precedent that I could find: NeoJev, a 27B 4-bit (~15 GB) that keeps 56 of 64 layers with a self-distilled head, 0.727 mean accuracy at 310 ms p50 on an M5 Pro. One detail I genuinely enjoyed is that a plain logit lens at their layer 56 reads 0.663. Same head-over-raw gap I measured on my own side, so kind of proving I'm moving in the right direction. A practical note for anyone following this route: mlx-lm exposes no intermediate-layer hooks (issue #1053, still open as of date of this blog post), so a hand-rolled block loop is the only way in. Mine matches the stock forward to a max logit difference of about 0.037. --- A warning from the probing people Fitting a tiny linear head on hidden states is also a literature: called probing. Hewitt & Liang (2019) and Belinkov's survey (2022) are the standard references, and their criticism is aimed at claims. A probe is weak evidence about what a model "understands". I took that as a warning about wording but not about architecture. A production head is judged by held-out task accuracy on the served artifact (ours reads 0.826 on its holdout), not by what it supposedly reveals. Linear (as the probing people say) is the defensible capacity choice. --- And finally products The method hunt was humbling, in that almost everything was already there before. The product hunt was even more humbling. Jev you've already met. Cloudflare's Clef ships fine-tuned decision models with open Apache-2.0 weights (209 ms median, 38.8 ms for the flash variant). Unsloth has a train-your-own recipe: LoRA plus a Clef-design head, their 0.8B reads 36% on typed-decisions before training and 73% after. Around those there's a whole cluster: Mini-Jev, mlx-decision, laya-mlx. Even the routing people are adjacent, though RouteLLM and friends put a separate model in front of the LLM, whereas I wanted the served model's own distribution to be the classifier. (!) And then the one that mattered to me: AnyJev, Nokia Applied Research's open method and library (technical report). Essentially I've taken AnyJev's method, deployed on the Mac. --- What's actually mine An honest ledger, because I care about this part. The repo's docs have a section literally titled "honest lineage", and it says explicitly that I did not invent the trick. What's original though: - The missing MLX backend for AnyJev's toolbox (an original implementation of their documented backend contract; they never shipped one). - The small-memory end: a 4-bit 4B, about 3 GB resident, versus NeoJev's 15. - Zero-training deployment: the frozen model works the moment it downloads. No dataset, no training run, no labels. - The observe() loop: while a shadow proxy watches real agent traffic, it quietly banks "what I predicted" next to "what actually happened", and one command refits a head from that bank. As far as I could find, nobody ships fit-on-your-traffic heads. - And the measurements of course. None of that is a new mechanism. It's just glue and a scorecard. BTW! I wrote about taste a while ago and there (TLDR) I say that when production becomes abundant, judgment becomes scarce. The judgment here was mostly deciding what not to claim. --- Benchmarks The repo holds itself to wording rules that I want to keep in this post too: fidelity claims say "aligned with the reference", never "more correct"; accuracy claims name the arena and the model size. So, with that: | what was measured | result | |---|---| | Agreement with the real Jev API, 300 non-coding questions | 4B 75.7%, 9B 82.0%; Jev agrees with itself 98.7%, which is the ceiling | | Readout vs greedy decoding of the same model, same 300 questions | +4.7 (4B), +6.3 (9B), McNemar p = 0.003; on long inputs greedy Choice answers collapse (46% vs 85% agreement) | | BANKING77, 10 intents, gold labels | 87.5% (4B), 91.25% (9B), Jev 95.5%: a trained model wins on its home turf | | typed-decisions, zero training | 61.6% (4B), 60.4% (9B), vs 73-78% for models fine-tuned on it | | Latency, long input, 5 questions in one request | 22.2 s down to 3.7 s mean (the state is read once); the coding head answers in ~0.2 s | The losses are the point, in a way. A trained model beats a zero-training readout whenever labels and a fixed label set exist. Train a classifier! This is for questions and options you cannot enumerate ahead of time, on a machine where the data cannot leave. Here's one real exchange on my M5: curl -s localhost:8095/v1/systemone -d '{"state": "Ticket #4821 from Dana (Pro plan, customer for 14 months): \"Since this morning every export to CSV fails with a 502. We run payroll from those exports at 3pm today.\" Status page: all green.", "questions": { "route": {"type": "Choice", "instructions": "Which team should take this first?", "options": ["billing", "platform-oncall", "account-management", "docs"]}, "escalate_now": {"type": "Noul", "instructions": "Does this need the on-call engineer right now?"}, "urgency": {"type": "Score", "instructions": "How urgent is this ticket?", "levels": ["low", "medium", "high", "critical"]}}}' -> {"model": "mlx-community/Qwen3.5-4B-MLX-4bit", "answers": { "route": {"answer": "platform-oncall", "confidence": 0.8516, "probabilities": {"billing": 0.0744, "platform-oncall": 0.8516, "account-management": 0.0579, "docs": 0.0161}}, "escalate_now": {"answer": true, "confidence": 0.637, "probabilities": {"Yes": 0.637, "No": 0.363}}, "urgency": {"answer": "critical", "confidence": 0.6651, "probabilities": {"low": 0.0051, "medium": 0.0156, "high": 0.3142, "critical": 0.6651}}}} > Status: research prototype, and the docs are blunt about it. The Jev-agreement numbers use LLM-authored questions with no human labels. The one fitted head that ships cannot be reproduced from the repo, because its training seed is withheld. If you have a fixed label set and training data, train a classifier instead. All of it is written down in a LIMITATIONS.md. --- The boring lesson This field is moving embarrassingly fast. But the lesson I'd actually keep from these weeks is a boring one: classical ML never left. Every number in the table above says so. Wherever a fixed label set and training data exist, something trained the old way wins. Jev's 95.5% on BANKING77 against my 87.5%, the fine-tuned 2B sitting 16 points above my zero-training readout on typed-decisions. Reading logits doesn't replace training. It covers the corner cases training can't reach e.g. questions you can't enumerate ahead of time, no labels to collect, data that isn't allowed to leave the machine etc. So before throwing this at a task, the checklist is the one classical ML wrote decades ago. Fixed labels and enough data? Train a classifier, it's still unbeatable there. Data that can't leave the machine, or questions that change per request? That's where reading the model's own probabilities earns its keep. In between, renting a hosted API is probably a fine deal. --- Oh right, the question Ah, I actually never answered the "how does a model decide instead of write" part from the beginning. Short version, and experienced ML people will roll their eyes at how obvious it is: the decision seems to form well before the end of the network. My tiny head reads block 16 of 32 and still answers the "what next" question at 0.826 accuracy. AnyJev's own measurements found a middle block to be a better feature space than the final one, since the last blocks are busy converting the answer into token space. And NeoJev keeps 56 of 64 layers, which suggests there is slack even higher up. So roughly: the deciding happens in the first half to two thirds of the layers, and the rest is more like the writing department. Am I even right here? Probably, for these models, by my measurements. That's as far as I'd push it. That's it.