Local Qwen3.8-27B vs Claude Fable 5: It Ties at Coding, but Costs 78%, Not 9%
Qwen3.8-27B, an open-weight model served locally with vLLM, scored 92.4% pass@1 on the 100 most recent hard LiveCodeBench problems. Claude Fable 5, called once, scored 90.4%. Nobody fine-tuned anything. The GVS5H repo gets there by starting fresh Qwen3.8-27B instances over and over and making them coordinate through a handful of files on disk.
That's the result going around, usually with "at 9% of the cost" attached. I read the paper source in the repo. The 27B did match Fable 5. It didn't beat it in any statistical sense, it wasn't cheap, and the 9% figure belongs to a different model. With the harness, the 27B costs about 78% of what Fable 5 does at list prices.
Qwen3.8-27B vs Claude Fable 5 on LiveCodeBench: the actual table
The main table (paper/tab-4new-5pass.tex) covers LCB-100: the 100 most recent problems in the hard split of LiveCodeBench release_v6, with a 128k output cap, reasoning on and five passes per arm. "Manager" means the GVS5H scaffold.
| Model | Single call | With manager | Δ vs single | Δ vs Fable 5 |
|---|---|---|---|---|
| Claude Fable 5 | 90.4 ± 1.4 | not run | | reference |
| Qwen3.8-27B (local, FP8, vLLM) | 66.8 ± 5.2 | 92.4 ± 2.3 | +25.6 | +2.0 |
| Qwen3.8-Flash-Next (local, FP8) | 84.2 ± 1.8 | 93.0 ± 1.2 | +8.8 | +2.6 |
| DeepSeek-V4.1-Flash | 88.4 ± 2.6 | 92.0 ± 1.8 | +3.6 | +1.6 |
| GPT-5.6-Terra | 80.8 ± 1.0 | 88.0 ± 1.2 | +7.2 | -2.4 |
| GPT-5.6-Luna | 70.4 ± 4.6 | 81.2 ± 2.8 | +10.8 | -9.2 |
All five models that ran both arms gain from the manager, and the weaker the single call, the bigger the gain. The authors put it this way: "the smaller the model, the more the scaffold has to add."
The "9% of the cost" figure is for Flash-Next, not the 27B
This is where the reposts get it wrong. The abstract says: "Orchestrated Qwen3.8 Flash Next scores 93.0% against Fable 5's 90.4% at 9% of the cost, while the smaller Qwen3.8-27B reaches 92.4%." The 9% applies to Flash-Next only.
Here's the cost table (fig-cost-tables.tex), priced at list rates per pass over all 100 problems:
| Arm | $/pass | $/solved |
|---|---|---|
| Fable 5 single | $61.11 | $0.68 |
| Qwen3.8-27B single | $18.95 | $0.28 |
| Qwen3.8-27B manager | $47.82 | $0.52 |
| Qwen3.8-Flash-Next manager | $5.76 | $0.062 |
| GPT-5.6-Luna manager | $1.50 | $0.018 |
$5.76 / $61.11 is about 9%. The 27B with the manager comes to $47.82, which is roughly 78% of Fable 5. The paper calls this "a slightly lower price", which is fair. The 27B talks a lot while it reasons: its single call alone emits 7.42M output tokens per pass, against 1.20M for Fable 5.
Those dollar figures are OpenRouter list rates ($0.21 in / $2.55 out per MTok for the 27B). On your own GPU the marginal cost is electricity and waiting time. On HN, badsectoracula reports about 50 tok/s for Qwen 3.8 27B on an RX 7900 XTX (Unsloth Q4 quant, llama.cpp). At that speed the manager arm's 18.6M output tokens per pass take about 100 hours of generation for one run of 100 problems on a single card, assuming no batching. Someone in the thread asked "But in 10x the time?" If you're running this at home, that's the number I'd want answered first.
Does Qwen3.8-27B beat Fable 5? Statistically, no
The +2.0 points don't hold up, and to its credit the paper says so. The p-value for Qwen3.8-27B manager vs Fable 5 single is 0.88 after Holm correction, and the one-sided 95% lower bound on the difference is -0.8 points. You can say it matches Fable 5. You can't say it beats it.
Six caveats, in order of how much they bother me
1. The 27B baseline was hurt by the token cap
The single call was generated at 250k and then replayed at 128k so it matched the manager arm. At 128k, 150 of 500 single calls (30.0%) hit the cap and 35 (7.0%) produced no code at all. At 250k the single call scores 70.0, not 66.8 (tab-qwen-caps.tex). Part of the "+25.6" comes from the manager splitting long reasoning across fresh contexts so nothing gets cut off. That's a real benefit of the scaffold, though anyone reading "+25.6" as "the model got smarter" is reading too much into it.
2. The manager arm gets to run tests and the single call doesn't
The v2 scaffold has a verifier step that runs each candidate against the problem's public stdin tests and feeds the first failure back to the manager. That covers 73 of the 100 problems. On HN, sams99 argued that this adds Python execution to a comparison that was supposed to be zero-tools. I agree, and I'd have liked to see a single call with one test-and-repair round as the baseline.
3. Up to 10 rounds of calls per problem against one call
Both arms ran five passes, so that part is even. Inside each pass, though, the manager arm gets a plan, a brainstorm, up to 10 manager→worker cycles, a finalizer and cut-off summarizers per problem, while Fable 5 gets one call. Fable 5 was run "single-only due to its high cost", so nobody knows what Fable 5 with a manager would score.
4. The Opus comparison uses a different setup
Opus-5 (88 → 95 with the manager) ran on the original scaffold through OpenRouter, for one pass. The original scaffold has 4 cycles instead of 10 and no verifier. The paper does say the 27B reaches "Opus-5's [range] in the OpenRouter-served set", but its own OpenRouter table notes the two sets "are not pooled" because the serving path and scaffold version differ. Treat "27B ≈ Opus" as a loose cross-table comparison.
5. Gains aren't universal
On the OpenRouter set, Qwen3.6-35B with reasoning off went from 35 → 27 at 128k and 28.2 → 27.4 at 16k. Kimi-K3 with reasoning on went 86 → 86. The failure case in the paper shows how this happens: the manager arm's ideation stage found the same optimization the single call had used, talked itself out of it as too error-prone, and committed to an asymptotically slower plan.
6. The authors ran their own benchmark and fixed their own grader
They patched two evaluator issues (non-unique reference answers on four problems, and a sys.stdin mock whose sys.stdin.buffer.readline() returned the first line on every call). Both fixes look reasonable to me. Still, nobody independent has reproduced this. Fable 5 also returned stop_reason=refusal on 6 of 500 problem-passes, which costs it 0–1.2 points.
On contamination: HN commenters called the benchmark old and saturated. The paper picks the 100 most recent hard problems by contest date specifically to limit that, but a model released after those contests could still have seen them.
The ledger pattern you can copy into your own agent loop
None of the above affects the part worth taking home. Every call starts with an empty context and reads state only from these files:
<ws>/task.md the problem statement
<ws>/plan.md the manager's plan (capped at 4,000 chars)
<ws>/notes.md curated findings (capped at 8,000 chars, rewritten, not appended)
<ws>/solution.py the current solution
<ws>/tasks.json debug mirror of the task list
The loop: the manager writes a plan and seed tasks, a worker brainstorms into notes.md, then the manager picks one next task, a fresh worker executes it and rewrites solution.py plus notes.md, and the public tests run. Repeat until the manager says done, it reissues the same task, or 10 rounds pass. Temperatures are 0.3 for the plan, 0.4 for the brainstorm and 0.2 for execution.
To try the authors' harness, here is the command from the README (the example uses an OpenAI model; swap the MULTIAGENT_MODEL provider prefix for your own endpoint):
cd codebase/v2-current
export OPENAI_API_KEY=...
LCB_RELEASE=release_v6 \
ESCALATION_CLOUD_MAX_TOKENS=128000 \
ESCALATION_CLOUD_TIMEOUT=7200 \
MULTIAGENT_MODEL=openai:gpt-5.6-terra \
uv run --no-project --python 3.12 --with 'datasets<4' --with numpy --with anthropic \
python escalation/run_bench.py --engine multiagent --only lcb --lcb 100 --parallel 8
--engine single gives you the baseline. Results go to runs/results.json.
If you already run a local agent, here's what to steal:
- Cap the notes file and make workers rewrite it instead of appending. Unbounded notes are how agent loops rot.
- Give each worker one task plus the ledger, never the full transcript.
- Treat a failing public test as ground truth that overrides the manager's "done".
- Add a no-progress guard. If the manager hands out the same task twice, stop.
- The paper's own suggestion: occasionally spawn a worker that sees only task.md. Inherited notes anchor every worker to an early wrong idea.
My take: if you run a 27B locally, GVS5H is good news. Structure plus a test runner got a mid-size model to frontier-level pass@1 on hard problems. I wouldn't read it as evidence that a 27B is as smart as Fable 5, and I'd stop quoting the 9% figure next to the 27B.
So is a 27B looping through up to 10 manager→worker rounds against one call of Fable 5 a fair fight? Or is that exactly the point, since your local GPU doesn't bill per token? Tell me in the comments, especially if you've run a ledger-style loop on your own hardware and watched it make things worse.
Sources
- GVS5H repo and paper source: slee-persis/GVS5H (paper/paper_latest.tex, tab-4new-5pass.tex, fig-cost-tables.tex, tab-qwen-caps.tex, tab-openrouter.tex)
- Hacker News discussion: news.ycombinator.com
Mentions
That is only the Memdoor version. The original tested against the newest 100 questions: slee-persis/GVS5H
Le papier GVS5H, avec le protocole, les scores et les coûts : arxiv.org Le code des auteurs : slee-persis/GVS5H La reproduction Memdoor, avec les 50 essais : github.com/guregodevo/memdoor/tree/main/examples/… Les notes détaillant les régressions : github.com/guregodevo/memdoor/blob/main/examples/… Quelques précisions : • Le papier utilise les 100 exercices les plus récents de la catégorie « hard » de LiveCodeBench. • Le score de 66,8 % correspond à l’appel unique plafonné à 128k tokens de sortie. Avec un plafond à 250k, il atteint 70,0 %. • Sur les mêmes 10 exercices répétés cinq fois, les résultats des auteurs donnent 48/50, contre 49/50 pour Memdoor. Memdoor parle de résultats comparables, pas d’une amélioration démontrée. L’hébergement diffère : OpenRouter pour Memdoor, contre vLLM en FP8 pour les auteurs. • Dans deux autres essais, l’agent orchestrateur a laissé le travail continuer malgré une solution correcte. Celle-ci a été écrasée, puis une solution correcte a été retrouvée avant la fin.
Пять небольших моделей догнали Claude в кодинге — без дообучения Исследователи показали метод GVS5H: несколько ИИ-агентов решают задачи вместе, координируясь через общие файлы. Результаты на LiveCodeBench: • Qwen3.8-27B: точность с 66,8% до 92,4% • Claude... slee-persis/GVS5H
A shared ledger of plans, notes, and solutions lifted Qwen3.8-27B to 92.4% on 100 hard LiveCodeBench problems, above Fable 5. Persistent context makes coordination part of inference—worth benchmarking separately, though gains may not generalize. slee-persis/GVS5H
AIで儲かってるのはIT業界だけ？ 経済学者の指摘 2026年10月26日（月）のAIニュース完全解説版。図解を見れば本文を読まなくても分かる形にしています。 おはようございます。今日のテーマを一言で言うと、「AIで本当に儲かるのか」なんですよ。アメリカの投資会社Apolloの経済学者が、IT業界の外ではAIの見返りがゼロだと書いて、大きな議論になってます。一方で、安いAIを組み合わせて一流のAIに迫ったという報告や、手元のMacで声を作れる更新も出ました。経営者の皆さんにも、学んでいる皆さんにも、使い方で差がつく話です。 1. IT業界の外では、AIの儲けは「ゼロ」？ 図解（完全解説版） 先に結論を言います。今日いちばん大事な話はこれです。アメリカの大手投資会社Apolloのチーフエコノミスト、トーステン・スロックさんが、IT業界の外では、AIの金銭的な見返りが「ゼロ」だと書いたんですよ。 中身を細かく見ていきます。巨大IT7社、いわゆるMag 7は、社員1人あたりの売上も利益率も伸びている。でも、残りのS&P 493社は横ばい。生産性が上がった兆しも、まだ見えないって出てるんです。しかも普通の会社は、AIの利用料を「成長のための投資」じゃなく「利益を削るコスト」と見始めていて、最新のモデルは要らないし払う気もない、と。この話、Hacker Newsで608ポイント、645コメントと大荒れで、TechCrunchも報じてます。ただし、これは一人の経済学者の見立てなので、確信度は中くらいで読んでください。 じゃあ、AIは意味ないのか。ここで勘違いしないでほしいんですけど、高いオーブンを買っても、メニューもお客さんも変わらなければ、売上は増えないじゃないですか。問われているのは機械じゃなくて、使い方が仕事に根づいてるかなんです。AIは買った日じゃなく、仕事が変わった日から儲かり始めます。 経営者の皆さんは、AIにかけた費用と、戻ってきた時間やお金、つまりROI（かけた費用に対して戻ってくる利益の割合）を、一度数字で並べてみてください。個人で学んでる人は、「AIで何が何分短くなったか」を自分の言葉で言えるようにしておくと、それがそのまま武器になりますよ。僕も非エンジニアですけど、Claude Codeで自分の会社の仕組みを作って毎日回してるのは、まさにそこを見たいからなんです。 出典: apollo.com 2. 安いAIを5体束ねて、一流AIに並んだと主張 図解（完全解説版） これ、めちゃくちゃ面白い話なんですよ。ただし、作った本人たちの自己申告で、外部ではまだ確かめられていません。確信度は中、という前提で聞いてください。 GitHubで公開された「GVS5H」という研究で、公開されているAIのQwen3.8を何体も立ち上げて、共有のファイルに計画とメモと今の答えを書き込みながら、難しいプログラミング問題を分担して解かせたんです。LiveCodeBenchという採点で、最新の難問100問を解かせたら、Qwen3.8-27Bは単体の66.8%から92.4%まで上がった。Claude Fable 5は90.4%です。同じQwenの別の版では、Fable 5の9%の費用で93.0%を出した、と書いてます。Hacker Newsでも180ポイントついて話題になりました。 例えるなら、一人の天才に頼むか、普通の5人がノートを回しながら分担するか、なんです。段取り次第で、5人が天才に並ぶこともある。でもね、論文自身が「変わらない、むしろ悪くなるモデルもある」と正直に書いてるんですよ。ここが大事です。 さっきのApolloの話で、普通の会社は最新モデルに払う気がないって出てたでしょ。高いAIだけが正解じゃない、組み方で差が出る。 経営者の皆さんは、「一番高いのを契約しないと」と焦らなくていい。個人で学んでる人は、モデル選びより、仕事を分けて書き残す段取りを練習してみてください。僕はそっちの方が、長く効く力だと思ってます。 出典: slee-persis/GVS5H 3. Ollama、Macの中で文章を声にできるように 図解（完全解説版） 次は、手元で試せる話です。Ollamaっていう、自分のパソコンの中でAIを動かす道具があるんですけど、10月24日に出た新しい版で、Macで音声を作れるようになったんですよ。文章をそのまま読み上げたり、AIの返事を、作りながら声で流したりできる。さらに、OpenAIと同じ書き方で音声を呼び出せる窓口もつきました。アプリ同士がやりとりする受付みたいなものです。OpenAI向けに作ったアプリから、手元の声を呼びやすくなるってことなんです。 もう一個おまけで、「d3」という判断専用のAIも入りました。文章や画像を渡して、「これは請求の問題？」みたいに、はい・いいえで答えさせたり、選択肢に点数をつけたりできる。大きさは2B、4B、9B、27Bの4種類です。 なぜ大事か。ナレーションって、外のサービスに原稿を送って作ることが多いじゃないですか。それが自分のMacの中で済む。台所に小さな録音スタジオが置かれたようなものです。外に出さずに声を作れる、これが一番の価値です。 ただし、声の質がどこまで仕事に使えるかは、今日の材料ではわからないので、自分の耳で確かめてください。 経営者の皆さんは、動画の仮ナレーションを社内で試せる選択肢が増えました。個人で学んでる人は、Macがあれば今日から触れます。僕は毎朝AIのニュースを自分の仕組みで集めて読んでるんですけど、こういう地味な更新ほど、後から効いてくるんですよね。 出典: github.com/ollama/ollama/releases/tag/v0.40.11 4. Redditの開発者掲示板が、AI製の投稿を禁止 6コマ漫画（案内役: ゼロ） ちょっと荒れてる話からいきますね。Redditに「r/opensource」っていう、誰でも使えて中身も公開されているソフト、オープンソースを紹介し合う掲示板があるんです。そこの管理人たちが土曜の夜に新しい投稿ルールを出して、AIを使ったプロジェクトの投稿を禁止したんですよ。The Vergeによると、その投稿は「あなたのプロジェクトがAIを使っている、AIで作られている、もしくは」という書き出しで始まっていたそうです。しかも対象はAIだけじゃなくて、英語が母語じゃない人や、一人で作ったプロジェクトまで入っている、と報じられてます。Hacker Newsでも104件のコメントがついて議論になりました。 なぜ大事か。作り手の世界で、「AI製は嫌だ」って空気が、ルールの形で表に出てきたってことなんです。手作りパン屋の集まりに、工場のパンを持ち込んだら嫌がられるじゃないですか。気持ちはわかる。でも、英語が苦手な人まで締め出すと、今度は別の反発を生みますよね。 ここで勘違いしないでほしいのは、これは一つの掲示板の決まりだということ。世の中全部がAI禁止になったわけじゃありません。僕はこのニュース、AI嫌いの話というより、伝え方の話だと思ってます。AIを使うこと自体より、どう伝えるかが問われる時代です。 経営者の皆さんは、SNSで「AIで作りました」と出すとき、受け取る側にはこういう人もいる、と知っておくだけで発信の言葉が変わります。個人で学んでる人は、AIに任せた所と自分で考えた所を、正直に言えるようにしておくと、それが信頼になりますよ。 出典: theverge.com 5. AI疑惑で写真の優勝取り消し、粗さが本物の証に 6コマ漫画（案内役: ゼロ） 最後は、写真と映像の話です。Nikonの写真コンテストで優勝した作品が、科学者たちにAIの痕跡を見つけられて、タイトルを取り消されたんですよ。TechSpotが報じて、Hacker Newsでも127ポイントついてます。どこをどう見抜いたのか、細かい経緯までは今日の材料ではつかめていないので、確信度は中で聞いてください。 一方で、正反対の動きもあるんです。BBCの自然番組「Blue Planet III」は、あえて完璧じゃないカットを残すことで、本物だと示そうとしている、とThe Vergeが伝えてます。前のシリーズは、すごくきれいな映像が売りだったのに、ですよ。 これ、面白いですよね。昔は「きれい」が価値だった。でも、AIがいくらでもきれいに作れる時代になると、「ちょっと粗い」が本物の証になる。料理で言うと、形の揃いすぎたコロッケより、手で丸めた不揃いの方が「手作りだな」って伝わるじゃないですか。きれいさより、本物だと示せることが価値になります。 経営者の皆さんは、SNSや広告で、撮影の裏側や生の現場を見せることが、そのまま信頼につながります。個人で学んでる人は、AIで作ったものと自分で撮ったものを、ちゃんと区別して説明できるようにしておくといい。僕は20歳のとき甘い投資話に乗って痛い目を見たので、うまい話ほど裏を取るのが体に染みてるんですけど、見る側も同じ目を持ち始めてるってことなんです。 出典: techspot.com まとめ 今日の話を一つにまとめると、AIは買うだけでは儲からない、ってことなんです。短期で見たら、高いAIを入れるのが一番早く見える。でも長期で伸びるのは、仕事の段取りにAIを根づかせた会社と人です。短期の「導入した」が、長期の「儲かった」に変わるかどうかは、使い方で決まります。今日から一つ、数字で記録してみましょう。 この記事は、公式ブログ・研究者の発信・掲示板（Hacker News など）・X の一次情報を毎朝集めて、非エンジニアにも分かる形にまとめています。 今日読んだ全件リストは別途保管しています。
5つのQwen3.8-27Bモデルが、GVS5H Harnessを使用してFable 5に匹敵する性能を達成 slee-persis/GVS5H 要約:GVS5Hは、複数のAIモデルが計画や情報を共有して協調する手法。追加学習なしでコード生成精度を向上させ、Qwen3.8-27Bの正答率を66.8%から92.4%に改善。高性能な商用AIに匹敵する精度を低コストで実現したと報告している。
Qwen3.8-27B×5のmanager-worker構成がLiveCodeBench hard 100問でClaude Fable 5並みと主張。評価器のstdinモック不具合を直して全実行を再採点し、1/5の価格か単一GPUでFable 5級を狙う / GVS5H: Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard slee-persis/GVS5H
Local Qwen3.8-27B vs Claude Fable 5: It Ties at Coding, but Costs 78%, Not 9% Qwen3.8-27B, an open-weight model served locally with vLLM, scored 92.4% pass@1 on the 100 most recent hard LiveCodeBench problems. Claude Fable 5, called once, scored 90.4%. Nobody fine-tuned anything. The GVS5H repo gets there by starting fresh Qwen3.8-27B instances over and over and making them coordinate through a handful of files on disk. That's the result going around, usually with "at 9% of the cost" attached. I read the paper source in the repo. The 27B did match Fable 5. It didn't beat it in any statistical sense, it wasn't cheap, and the 9% figure belongs to a different model. With the harness, the 27B costs about 78% of what Fable 5 does at list prices. Qwen3.8-27B vs Claude Fable 5 on LiveCodeBench: the actual table The main table (paper/tab-4new-5pass.tex) covers LCB-100: the 100 most recent problems in the hard split of LiveCodeBench release_v6, with a 128k output cap, reasoning on and five passes per arm. "Manager" means the GVS5H scaffold. | Model | Single call | With manager | Δ vs single | Δ vs Fable 5 | |---|---|---|---|---| | Claude Fable 5 | 90.4 ± 1.4 | not run | | reference | | Qwen3.8-27B (local, FP8, vLLM) | 66.8 ± 5.2 | 92.4 ± 2.3 | +25.6 | +2.0 | | Qwen3.8-Flash-Next (local, FP8) | 84.2 ± 1.8 | 93.0 ± 1.2 | +8.8 | +2.6 | | DeepSeek-V4.1-Flash | 88.4 ± 2.6 | 92.0 ± 1.8 | +3.6 | +1.6 | | GPT-5.6-Terra | 80.8 ± 1.0 | 88.0 ± 1.2 | +7.2 | -2.4 | | GPT-5.6-Luna | 70.4 ± 4.6 | 81.2 ± 2.8 | +10.8 | -9.2 | All five models that ran both arms gain from the manager, and the weaker the single call, the bigger the gain. The authors put it this way: "the smaller the model, the more the scaffold has to add." The "9% of the cost" figure is for Flash-Next, not the 27B This is where the reposts get it wrong. The abstract says: "Orchestrated Qwen3.8 Flash Next scores 93.0% against Fable 5's 90.4% at 9% of the cost, while the smaller Qwen3.8-27B reaches 92.4%." The 9% applies to Flash-Next only. Here's the cost table (fig-cost-tables.tex), priced at list rates per pass over all 100 problems: | Arm | $/pass | $/solved | |---|---|---| | Fable 5 single | $61.11 | $0.68 | | Qwen3.8-27B single | $18.95 | $0.28 | | Qwen3.8-27B manager | $47.82 | $0.52 | | Qwen3.8-Flash-Next manager | $5.76 | $0.062 | | GPT-5.6-Luna manager | $1.50 | $0.018 | $5.76 / $61.11 is about 9%. The 27B with the manager comes to $47.82, which is roughly 78% of Fable 5. The paper calls this "a slightly lower price", which is fair. The 27B talks a lot while it reasons: its single call alone emits 7.42M output tokens per pass, against 1.20M for Fable 5. Those dollar figures are OpenRouter list rates ($0.21 in / $2.55 out per MTok for the 27B). On your own GPU the marginal cost is electricity and waiting time. On HN, badsectoracula reports about 50 tok/s for Qwen 3.8 27B on an RX 7900 XTX (Unsloth Q4 quant, llama.cpp). At that speed the manager arm's 18.6M output tokens per pass take about 100 hours of generation for one run of 100 problems on a single card, assuming no batching. Someone in the thread asked "But in 10x the time?" If you're running this at home, that's the number I'd want answered first. Does Qwen3.8-27B beat Fable 5? Statistically, no The +2.0 points don't hold up, and to its credit the paper says so. The p-value for Qwen3.8-27B manager vs Fable 5 single is 0.88 after Holm correction, and the one-sided 95% lower bound on the difference is -0.8 points. You can say it matches Fable 5. You can't say it beats it. Six caveats, in order of how much they bother me 1. The 27B baseline was hurt by the token cap The single call was generated at 250k and then replayed at 128k so it matched the manager arm. At 128k, 150 of 500 single calls (30.0%) hit the cap and 35 (7.0%) produced no code at all. At 250k the single call scores 70.0, not 66.8 (tab-qwen-caps.tex). Part of the "+25.6" comes from the manager splitting long reasoning across fresh contexts so nothing gets cut off. That's a real benefit of the scaffold, though anyone reading "+25.6" as "the model got smarter" is reading too much into it. 2. The manager arm gets to run tests and the single call doesn't The v2 scaffold has a verifier step that runs each candidate against the problem's public stdin tests and feeds the first failure back to the manager. That covers 73 of the 100 problems. On HN, sams99 argued that this adds Python execution to a comparison that was supposed to be zero-tools. I agree, and I'd have liked to see a single call with one test-and-repair round as the baseline. 3. Up to 10 rounds of calls per problem against one call Both arms ran five passes, so that part is even. Inside each pass, though, the manager arm gets a plan, a brainstorm, up to 10 manager→worker cycles, a finalizer and cut-off summarizers per problem, while Fable 5 gets one call. Fable 5 was run "single-only due to its high cost", so nobody knows what Fable 5 with a manager would score. 4. The Opus comparison uses a different setup Opus-5 (88 → 95 with the manager) ran on the original scaffold through OpenRouter, for one pass. The original scaffold has 4 cycles instead of 10 and no verifier. The paper does say the 27B reaches "Opus-5's [range] in the OpenRouter-served set", but its own OpenRouter table notes the two sets "are not pooled" because the serving path and scaffold version differ. Treat "27B ≈ Opus" as a loose cross-table comparison. 5. Gains aren't universal On the OpenRouter set, Qwen3.6-35B with reasoning off went from 35 → 27 at 128k and 28.2 → 27.4 at 16k. Kimi-K3 with reasoning on went 86 → 86. The failure case in the paper shows how this happens: the manager arm's ideation stage found the same optimization the single call had used, talked itself out of it as too error-prone, and committed to an asymptotically slower plan. 6. The authors ran their own benchmark and fixed their own grader They patched two evaluator issues (non-unique reference answers on four problems, and a sys.stdin mock whose sys.stdin.buffer.readline() returned the first line on every call). Both fixes look reasonable to me. Still, nobody independent has reproduced this. Fable 5 also returned stop_reason=refusal on 6 of 500 problem-passes, which costs it 0–1.2 points. On contamination: HN commenters called the benchmark old and saturated. The paper picks the 100 most recent hard problems by contest date specifically to limit that, but a model released after those contests could still have seen them. The ledger pattern you can copy into your own agent loop None of the above affects the part worth taking home. Every call starts with an empty context and reads state only from these files: <ws>/task.md the problem statement <ws>/plan.md the manager's plan (capped at 4,000 chars) <ws>/notes.md curated findings (capped at 8,000 chars, rewritten, not appended) <ws>/solution.py the current solution <ws>/tasks.json debug mirror of the task list The loop: the manager writes a plan and seed tasks, a worker brainstorms into notes.md, then the manager picks one next task, a fresh worker executes it and rewrites solution.py plus notes.md, and the public tests run. Repeat until the manager says done, it reissues the same task, or 10 rounds pass. Temperatures are 0.3 for the plan, 0.4 for the brainstorm and 0.2 for execution. To try the authors' harness, here is the command from the README (the example uses an OpenAI model; swap the MULTIAGENT_MODEL provider prefix for your own endpoint): cd codebase/v2-current export OPENAI_API_KEY=... LCB_RELEASE=release_v6 \ ESCALATION_CLOUD_MAX_TOKENS=128000 \ ESCALATION_CLOUD_TIMEOUT=7200 \ MULTIAGENT_MODEL=openai:gpt-5.6-terra \ uv run --no-project --python 3.12 --with 'datasets<4' --with numpy --with anthropic \ python escalation/run_bench.py --engine multiagent --only lcb --lcb 100 --parallel 8 --engine single gives you the baseline. Results go to runs/results.json. If you already run a local agent, here's what to steal: - Cap the notes file and make workers rewrite it instead of appending. Unbounded notes are how agent loops rot. - Give each worker one task plus the ledger, never the full transcript. - Treat a failing public test as ground truth that overrides the manager's "done". - Add a no-progress guard. If the manager hands out the same task twice, stop. - The paper's own suggestion: occasionally spawn a worker that sees only task.md. Inherited notes anchor every worker to an early wrong idea. My take: if you run a 27B locally, GVS5H is good news. Structure plus a test runner got a mid-size model to frontier-level pass@1 on hard problems. I wouldn't read it as evidence that a 27B is as smart as Fable 5, and I'd stop quoting the 9% figure next to the 27B. So is a 27B looping through up to 10 manager→worker rounds against one call of Fable 5 a fair fight? Or is that exactly the point, since your local GPU doesn't bill per token? Tell me in the comments, especially if you've run a ledger-style loop on your own hardware and watched it make things worse. Sources - GVS5H repo and paper source: slee-persis/GVS5H (paper/paper_latest.tex, tab-4new-5pass.tex, fig-cost-tables.tex, tab-qwen-caps.tex, tab-openrouter.tex) - Hacker News discussion: news.ycombinator.com
@fasterthanli.me did you see this? slee-persis/GVS5H
Qwen3.8-27Bを5台束ね、共有ファイルで計画とメモを共有させる学習なしの協調手法が、LiveCodeBench難問で92.4%に到達。単体は66.8%なので、小型モデルは協調の設計でまだ伸びる。モデル選びより工夫が割に合う。 slee-persis/GVS5H