Don’t Trust the Benchmarks. Do Your Own Research.
A few days ago I gave a talk called _"What does AI really cost?"_ a follow-up to the cost post in this series. I expected to spend the Q&A talking about token prices, subscriptions and context management. Instead, the slide that got the most questions was the one about benchmarks – or rather, about why I stopped trusting them and built my own. So this post is that slide, expanded: what the public leaderboards currently say, why I think they tell at most half the truth, and what happened when I let eight models implement the same 20 real feature tickets of an _Angular_ component library and had four humans judge the results _blind_.
Quick recap for everyone joining late: this series started with which LLM is best for _Angular_ development, continued with which app or harness to wrap around it, and then moved on to the harness setup and the skills that turn a capable model into a careful teammate. The summer update was the last time I ranked models – mostly on public benchmarks plus my own gut feeling. This post replaces the gut feeling with data. The cost side of the talk gets its own follow-up post soon, so I'll keep the money talk short here.
TL;DR: Two Leagues, No Middle Field
If you only have thirty seconds: on the public benchmarks, the frontier models from Anthropic, OpenAI, Google and Meta sit within a few points of each other. On my own benchmark – eight models, the same 20 real _Angular_ tickets, four blind human reviews – they don't. Three models landed between 7.5 and 8.25 out of 10, the other five between 3.5 and 4.75, and nothing in between. All three top spots went to Anthropic: Opus 5.5 for quality, Sonnet 5.5 for value (94 % of the Opus score for a fifth of the price), Fable 5.1 right behind (why third? – one of the reviewers flagged too much refactoring instead of focusing on the ticket with a minimal changeset). The GPT models, Gemini, Grok and Haiku all ended up a league lower – for real feature work I would not use them for agentic coding. And if you pay API prices, the harness matters too: the same Sonnet 5.5 cost about half as much per ticket in OpenCode (and 60 % in Pi) as in Claude Code, and a bit cheaper than in GitHub Copilot.
As always in this series: this is one benchmark on one _Angular_ component lib codebase, judged by four humans. It is not deterministic, it is not universal, and it might be outdated in a month. But it is real work, reviewed by the people who have to live with the result – and that, I'll argue, is worth more than any leaderboard you can easily find out there.
What the Public Benchmarks Say
Let me start where everyone starts: the leaderboards. My favorite source is still Artificial Analysis, for the reasons I gave in the first post of this series – independent, transparent about their methodology, and the only ones who consistently report cost and time next to the score. The three charts below are snapshots from October 25, 2026, the first two filtered down to the 14 models I care about, the third the release view with all effort variants of 11 of them; click any of them for the live version.
Intelligence Index
Artificial Analysis Intelligence Index v4.3.2 – snapshot from October 25, 2026, 14 of 708 models. The index combines ten evaluations, from agentic tasks (Terminal-Bench, GDPval) over coding (SciCode) to long-context and scientific reasoning.
The top of the chart is crowded: Opus 5.5 at 58, Sonnet 5.5 and Muse Wonder Full at 56, Kimi K3.1 Pro at 55, Fable 5.1, GPT-6 Astra and Gemini 4 Argon at 53, Muse Wonder at 52. Eight models within six points. If you only looked at this chart, you'd conclude that the frontier is a tie and you should simply pick the cheapest one.
Cost per Task
Weighted average cost in USD per Intelligence Index task, segmented by token type – snapshot from October 25, 2026. Lower is better.
This chart I like, because it measures the right unit: not dollars per million tokens, but dollars per solved task. A model that charges little per token but burns ten times the reasoning tokens is not cheap. And here the spread is enormous: GPT-6 Luna solves an index task for 7 cents, Fable 5.1 for $7.63 – a factor of 100. The Anthropic models are the three most expensive bars on the right. Sonnet 5.5 costs $5.46 per task, Opus 5.5 $5.98 – almost the same, because at "max with fallback" Sonnet thinks a lot. Max should never be used with Anthropic models for real work – in my experience it is way too much overthinking, and frankly bullshit. Keep that in mind when we get to my own numbers, where Sonnet ran at _high_ effort and turned out dramatically cheaper.
Intelligence vs. Cost per Task
Intelligence Index vs. cost per task by model release, all reasoning and effort variants of each release connected – snapshot from October 25, 2026. The dotted line is the Pareto front, the green area the "most attractive quadrant".
Put the two together and you get the chart I keep coming back to. The Pareto front runs from GPT-6 Luna in the bottom left through MiMo and GPT-6.1 Sol to Kimi K3.1 Pro and Opus 5.5 in the top right. GPT-6.1 Sol sits inside the attractive quadrant: an index of 52 for well under a dollar per task. GPT-6 Astra scores the same 53 as Fable 5.1 for less than half the cost. And Anthropic's top variants all cost more than $5 per task.
So the public benchmarks, read honestly, say: Anthropic has no lead worth paying for. OpenAI, Meta and Kimi are right there at a fraction of the price. That was also the tone of my summer update – and it is completely wrong. At least for our kind of work and for our team. One caveat up front: four of the models that make the frontier look like a tie – Muse Wonder Full, Kimi K3.1 Pro, GPT-6 Astra and Gemini 4 Argon – were not part of my bench, so what follows is about the eight models I did run.
Why I Don't Trust Them (Anymore)
Before I show you my own results, let me explain why I stopped treating leaderboards as a basis for model decisions. Not because the people running them are sloppy – Artificial Analysis in particular does careful work – but because of what a benchmark _is_.
A benchmark is a fixed set of tasks with a fixed way to score them. That has three consequences:
1. Contamination. The tasks are public, so sooner or later they end up in the training data. The most famous example is SWE-bench Verified, for years _the_ coding benchmark. In February 2026, OpenAI stopped reporting it: their own models reproduced original patches from nothing but the task ID, and a manual audit found that most remaining "failures" were flawed tests, not model limits. The research community has by now written a whole taxonomy of contamination – and of the mitigations that each new generation of models defeats.
2. Goodhart's law. "When a measure becomes a target, it ceases to be a good measure." Every lab has a team whose job it is to make the next model look good on the handful of benchmarks that get quoted in launch posts. The Leaderboard Illusion, a NeurIPS 2025 paper, documented how this works on Chatbot Arena: private testing of dozens of variants, selective disclosure of the best one, and preferential data access for a few providers. The paper is about one arena, but the incentive applies to every public benchmark – and the more popular the benchmark, the stronger the pull.
3. Wrong task, wrong judge. Even an uncontaminated, ungamed benchmark measures what is easy to measure: did the test pass, did the answer match. My daily reality is different. A feature can pass every test and still be unmergeable; it can implement the ticket text and miss its intent; it can work and leave behind code nobody wants to own. No public benchmark runs on an enterprise _Angular_ codebase with a strict style guide, a design system and an opinionated review culture – and none of them has a human reviewer at the end.
That third point is the one I can do something about. If the benchmark doesn't measure my work, I have to measure my work.
My Own Benchmark: Eight Models, 20 Tickets, Four Human Juries
So in early October 2026 I built one. The subject is an enterprise _Angular_ component library I developed: a publishable library with secondary entry points, a demo app that serves as its living documentation, a strict style guide, an AGENTS.md with the repo rules, and e2e tests with Cypress. In short, exactly the kind of codebase this series has been about.
The Setup
- Eight models, each in one uninterrupted run: Opus 5.5, Sonnet 5.5, Fable 5.1 and Haiku 5.5 from Anthropic, GPT 6.1 Sol and GPT 6.0 Luna from OpenAI, Gemini 3.8 from Google, and Grok 4.7 from xAI. Each at the reasoning effort I'd normally use for it – high, the small models Luna and Haiku even on xhigh – with one deliberate exception: Fable at medium, to see whether the most intelligent model could get away with less thinking.
- The same 20 tickets, real feature tickets from the backlog, frozen in one file with a binding "Decisions" block per ticket that settled the open questions upfront. Tickets 1 to 16 extend existing components (select, chip area, navigation tree, tabs, checkbox, temporal editors, splitter, …), 17 to 20 create new ones (status badge, AG Grid theme, progress indicator, breadcrumb).
- The same starting commit, the same prompt, the same repo rules. The prompt asked for one commit per ticket, a self-review in a fresh context folded in by amend, and the full gate – lint, library build, app build, the touched e2e specs – green before every commit. No questions allowed: where the ticket was silent, the model had to decide and record the assumption.
- Each model in its own harness, the one its subscription ties it to: the Claude models in Claude Code, the GPT models in Codex CLI, Gemini in Antigravity, Grok in Cursor. That is the setup you get on a subscription anyway, so model and harness were benchmarked as the pair you'd actually use.
- What was recorded: the API cost per run (all eight were paid through my private subscriptions, but priced at list prices for comparison – all prices in this post are USD at API list prices, as the charts show them) and the active wall-clock time. Both stayed sealed until the scores were in.
The Four Juries
Public benchmarks measure what is easy to measure, so the judges here are the people who live with the result. Four reviewers, one question each, every one of them blind: the eight runs were deployed as numbered copies of the demo app and pushed as lettered branches, and none of the three colleagues knew which number or letter was which model. I did, which is why my own review needed a trick – see below.
- Ticket intent – the colleague who wrote the tickets checked on five of them whether each implementation does what he _meant_, not what the text said.
- Functional testing – a second colleague tested 19 of the 20 tickets on the deployed demos, with mouse and keyboard, hunting for bugs and regressions.
- Merge readiness – a third colleague, a senior developer, reviewed four sampled commits of each branch like a merge request: would this go into develop as it is?
- Code quality – that was me: all 20 tickets of every branch, read against the style guide and the repo rules. Is this code I'd be glad to own a year from now? Since I had started the runs and knew which model was behind each letter, I tricked myself into not knowing: I reviewed numbered copies of the branches and looked up the mapping only after my scores were in.
Every reviewer scored every run on the same scale from 0 to 10. The combined score is the plain average of the four medians – no weighting, no cleverness. Between them, the four of us spent well over 20 hours on the reviews. Remember that number; it comes back later.
The Ranking Table
!Ranking table of the eight model runs
The eight runs ranked by the average of the four reviews, with the four individual scores, the API cost per ticket and in total, and the active time of each run. * Gemini 3.8 stopped once because its context was full and had to be restarted; it then stopped three more times to ask a question. ** Implemented only 19 of the 20 tickets (there was an instruction to skip the ticket if certain red flags like failing e2e tests appear).
The first thing that jumps out is the split I already spoiled in the TL;DR: two leagues, no middle field. Opus 5.5 (8.25), Sonnet 5.5 (7.75) and Fable 5.1 (7.5) form the top league; Gemini 3.8 (4.75), Grok 4.7 and GPT 6.1 Sol (4.25), GPT 6.0 Luna (3.75) and Haiku 5.5 (3.5) the lower one. Nothing lands between 4.75 and 7.5. On this codebase, a model either delivers work the reviewers can build on, or it does not.
The second thing: the top three are all Claude models, and the three top places are close. That was the biggest surprise for me – not that Opus won, but that Sonnet and Fable ended up within three quarters of a point of it. Fable was even marked down a little by the merge-readiness reviewer for its tendency to refactor and improve things outside the ticket's scope: helpful on a solo project, extra review work in a team. And remember that Fable ran at _medium_ effort, the other two at _high_ – it almost got away with it.
The third thing: the two GPT models fell short, and so did everyone else. GPT 6.1 Sol and GPT 6.0 Luna both skipped one of the 20 tickets. Gemini 3.8 had to be restarted and stopped three more times to ask questions although the prompt forbade it. Grok 4.7 was the slowest run at 9 hours 10 minutes and the second most expensive, and still only shared fifth place. Time did not buy quality, and neither did price – Grok costs more per ticket than Opus.
And the fourth: Haiku 5.5 came last. The vendor alone is not the whole story. Anthropic's smallest model produced the smallest diffs, skipped features silently and broke an existing behavior. A cheap run that has to be redone is not cheap – more on that in the next post.
The Score Profiles
!Score profiles of the eight model runs
One chart per run: the four reviewers' scores on four axes – ticket intent, functional testing, merge readiness and code quality.
The radar charts explain _why_ the leagues exist. The three top runs are big, balanced diamonds: no axis below 6, most at 7 or above. Opus scores 9 on both merge readiness and code quality. Sonnet got a perfect 10 for merge readiness – the reviewer's clear favorite, "very well rounded and reliable, no nonsense, many tests, sensible refactoring, initiative of its own." Fable leads on ticket intent with an 8: it understood what the ticket author meant better than anyone else, and then did a bit more than he asked for.
The lower league is where the profiles fall apart, each in its own way. Haiku 5.5 got a 6 for code quality, because it wrote the smallest diffs and the single best refactor of the whole bench (one table replacing 200 lines of switch statements), but a 1 for functional testing and merge readiness, because it quietly left features out. GPT 6.0 Luna got a 7 for ticket intent – it understood the tickets fine – with 2 and 3 on the code axes: "few and simple code changes, but the features often do not work." And then there is the one profile that deserves its own section.
The Gemini Case: Passes the Tests, Fails the Review
Look at Gemini 3.8. Functional testing: 9 – the best score of all eight runs, better than Opus. The tester's verdict: "provided the best solutions on average; only on rare occasions another implementation beat this one." If my benchmark had stopped there – if it had been a public benchmark, which scores exactly this: does it work, does the test pass – Gemini 3.8 would have won. At $1.31 per ticket, cheaper than Sonnet and a fraction of Opus and Fable. The perfect Pareto winner.
Now look at the other three axes. Ticket intent: a 6 overall, but on one of the five sampled tickets the ticket author gave it a 0 – no documentation, no showcase, nothing he could even check – which is not what he asked for. Merge readiness: 3 – "incomplete", one of the four reviewed commits a functional failure. And code quality: 1, the worst of all eight runs. My own notes from the blind review: the code looks disciplined at first glance, but it stacks defensive guards, dead ?? [] fallbacks and the same expression pasted eight times, and it introduced two regressions in existing components. On top of that it was the only run that needed babysitting: it stopped once because its context was full and had to be restarted, and then stopped three more times to ask questions, although the prompt forbade it.
So the same run is the best functional implementer and the worst engineer of the bench. The features work, and nobody on the team would merge them. Every one of these commits would come back from review – rewritten by a human, or redone by a better model – and that is where the cheap price per ticket evaporates. Functional testing 9 is what a benchmark measures. Code quality 1 is what you pay for afterwards.
This is the pattern a single-number benchmark can't show you, and it is why I wrote this post. A model that is optimized to pass tests will pass tests. Whether it writes code you want to own only a human review on your own codebase can tell – and in this case the human review turned the apparent winner into what is potentially the most expensive model of the bench – once you add the human hours to rewrite its code.
Score vs. Costs
!Score vs. cost per ticket of the eight model runs
Every run by its API cost per ticket (logarithmic) and its average score, with the Pareto front and the attractive quadrant – cheap and good. * restarted, ** 19 of 20 tickets.
Now price and quality in one picture, in the same style as the Artificial Analysis chart above – and this is where my numbers and theirs diverge. The Pareto front has a steep step in the middle: from GPT 6.1 Sol at $0.71 and 4.25 points through Gemini 3.8 at $1.31 and 4.75 straight up to Sonnet 5.5 at $1.46 and 7.75 points. For 15 cents more per ticket you move from the lower league to the upper one. From there the line flattens: Opus 5.5 buys another half point for $7.20 per ticket, and Fable 5.1, at $11.18 per ticket, sits below the line – more expensive and slightly worse than Opus in this run, at medium effort.
Three practical takeaways from this chart, in the order I'd make the decision:
- Opus 5.5 is currently the best choice for implementing features. The highest score, and both cheaper and faster than Fable: $7.20 against $11.18 per ticket, 3 h 34 min against 5 h 10 min for all 20 tickets.
- Sonnet 5.5 is the best value by far. 94 % of the Opus score for a fifth of its cost. It is the only run in the attractive quadrant, and the one I'd recommend to every team that pays API prices.
- Fable 5.1 stays my favorite for everything that isn't implementation. Planning, design, reviews, the mutual understanding between model and human – it is still the most intelligent of the three in my eyes. When it implements, tell it explicitly to stay within the ticket, and consider high effort.
And where does GPT 6.1 Sol still fit? At $0.71 per ticket and 2 h 53 min it is far cheaper and faster than the top three. I'd use it for very small, well-defined tasks, for prototypes and experiments, and as the second opinion from another provider in a review – three Claude models share the same blind spots. For anything that ends up in develop, I'd pay the 75 cents.
Even the most expensive run is cheap in absolute terms, by the way. All eight runs together, 158 tickets, would cost about $613 at API list prices. Without AI I'd estimate each of these tickets at two to four hours of implementation work – at a conservative €50 per hour that is €100 to €200 per ticket, roughly 15 to 25 times the cost of the Opus run. The token bill is not the expensive part of agentic engineering. But that is the next post.
The Verdict: Use the Anthropic Models
On the public benchmarks these three are tied with half a dozen others; on human review they are a league ahead. That difference is exactly the difference between models that are good and models that look good on benchmarks – so for agentic coding in the frontend, use the Anthropic models.
That brings me to the rough ranking I showed in the talk – my personal order of providers for agentic coding, as of Q4 2026:
!My rough ranking of providers for agentic coding, Q4 2026
1. Anthropic – Opus 5.5, Sonnet 5.5, Fable 5.1. The top league, see above.
2. OpenAI – GPT-6 Astra and GPT-6.1 Sol. I left Astra out of the bench on purpose: from daily use I don't rate it as a coding model, so I spent the review hours on Sol and Luna instead – but for reviews and research it's my go-to. Sol is my pick for small, cheap tasks and second opinions.
3. Meta – Muse Wonder Full is the one model I'd pay a separate subscription for next. Everything below I wouldn't touch for agentic coding at all – yes, they might be useful, but I don't want to use them 😏
4. Google, xAI or the Chinese open-weight models – Gemini 3.8 and Grok 4.7 both landed in the lower league. DeepSeek 4.1, GLM 5.3, Kimi 3.1 and MiMo 2.6 are consistently three to six months behind the frontier, and that gap has been remarkably stable – no catching up in sight. If they catch up, we re-evaluate everything.
5. Local models – on the hardware most of us have, usually at least one more step down. They make sense for exactly one reason: when your data must not leave the company. Then they are the only option, and the data privacy post applies.
Two models on that list weren't in our human bench at all: Muse Wonder Full and Gemini 4 Argon were both released after the runs. On the public charts they sit right at the top – but this bench only judges the eight models I ran. It took four people more than 20 hours to judge them, which is exactly why I cannot repeat it whenever a new model drops.
My plan is to rerun it roughly every quarter, with the new releases in the lineup. In between, the Artificial Analysis charts above are the best update on capability and cost you'll get – just read them as what they are. The bench can't tell you anything about Gemini 4 Argon; my own first days with it can: for agentic coding the way I do it, Google has not caught up.
Hot take: if somebody tells you that Gemini 4 Argon has caught up with the frontier, it's worth asking what they measured. On benchmarks and first impressions it has; in my first days of real agentic coding with it, not yet. The same goes for claims about local models, by the way. But now let's look at the harnesses for a bit.
The Harness Matters (If You Pay API Prices)
One more result from the same tickets, because it surprised me almost as much as the model ranking. On a subscription, the harness is a given: Claude Code for the Claude models, Codex for the GPT models, and so on – that is how the eight model runs above were set up. But as soon as you pay API prices, you can run any model in any harness, and then the choice is yours. So after the eight model runs, I ran a harness follow-up: the same 20 tickets, the same prompt, one fixed model – Sonnet 5.5 at medium effort – in four different coding harnesses, in two rounds. In the first round one agent worked through all 20 tickets itself; in the second, the agent handed every ticket to a fresh subagent and only orchestrated. No human review this time, only cost and time. Within a round, nothing changed between the runs but the wrapper: the system prompt, the tool definitions, the skills and rules it loads, and how it runs its subagents.
| Harness | Cost per ticket | Total cost | Time |
| ------------------ | --------------: | ---------: | ---------- |
| OpenCode | $0.84 | $16.71 | 2 h 23 min |
| Pi | $0.98 | $19.65 | 3 h 19 min |
| GitHub Copilot | $1.02 | $20.31 | 2 h 35 min |
| Claude Code | $1.63 | $32.52 | 3 h 40 min |
Sonnet 5.5 (medium), 20 tickets, every figure the average of the two rounds per harness. Claude Code went directly to the Anthropic API, the other three through the GitHub Copilot API; all tokens priced at Anthropic list prices.
Same model, same task, twice the bill. OpenCode and Pi cost roughly half of Claude Code per ticket, with GitHub Copilot in between. The reason is context: every turn of an agentic session sends the whole context again, and Claude Code's context is simply bigger – a longer system prompt, more tools, subagents. The two rounds show where the gap comes from: with one continuous agent, Claude Code ($22) and GitHub Copilot ($21) were level, OpenCode ($15) and Pi ($16) a quarter cheaper. With one subagent per ticket, Claude Code's bill nearly doubled to $43 for the 20 tickets, Pi rose to $23, OpenCode to $18, and Copilot even fell to $19.
Two caveats before you switch. First, this follow-up measured cost and time, not quality – Sonnet at high effort scored 7.75 in the model bench, and medium effort was not reviewed. Second, if you work on a Claude or OpenAI subscription, none of this matters much: you pay a flat price, and Claude Code's subagents, hooks and /loop are worth the context they cost. It matters when you pay API prices – which, in my experience, is the normal case in the enterprise. Then my low-budget setup is Sonnet 5.5 (medium) in OpenCode or Pi, at a cost per ticket close to GPT 6.1 Sol and with a model whose high-effort sibling sits in the upper league – verify the quality of medium on your own tickets first. How the four token types, caching and context management drive that bill is the next post.
Run Your Own Bench
You don't need eight models and 20 tickets. What you need is your own codebase, your own reviewers and a fair setup:
1. Pick three to five real tickets from your backlog – not demos, not greenfield. Freeze the text and settle the open questions upfront, so every model works from the same requirement.
2. Give two or three candidate models the same prompt, the same starting commit and the same rules. Let each one run through all tickets on its own, with the full gate green before every commit.
3. Let the people who normally review the work score the results – blind. Numbered deployments, lettered branches, one question per reviewer. The ticket author checks intent, a tester checks behavior, a reviewer checks mergeability, someone reads the code.
4. Count the hours around each run, not just the tokens: restarts, questions, skipped tickets, out-of-scope changes, review rounds. That is where the cost is.
A few days of this tell you more about your codebase than any leaderboard – and the setup is reusable the next time a vendor claims the top spot.
Agentic Engineering Workshop
In our Agentic Engineering Workshop we build your setup hands-on: your style guide, your AGENTS.md, and a skills catalogue tuned to your team's rules instead of mine – and then put it to work on spec-first workflows, code review, testing, and brownfield refactoring in real _Angular_ codebases. Which model and which harness to use is one of the first decisions we make there, and after this post you know how I'd make it. English or German, as always.
- 🤖 Agentic Engineering Workshop – 2 days (or 3 half-days online), remote or in-house, English or German, for advanced _Angular_ developers – next public date (in English): November 30 to December 2, 2026
Conclusion
Don't trust the benchmarks. Not because they lie, but because they measure the wrong task with the wrong judge, and because every lab is optimizing for them. The public charts show the frontier as a tie; four humans reviewing real _Angular_ work on my codebase show two leagues with nothing in between.
So, for Q4 2026, my recommendation in one line each:
- Models: the Anthropic models. Opus 5.5 when quality matters, Sonnet 5.5 when cost matters, Fable 5.1 for planning, design and reviews. GPT-6.1 Sol for small, well-defined tasks and as the cheap second opinion from another provider. Astra for architecture and planning reviews and computer use tasks.
- Harness, on a subscription: whatever you're productive in – for me that's Claude Code for Fable and Opus, and Codex for Astra and Sol.
- Harness, on API prices: OpenCode or Pi, with Sonnet 5.5 at medium effort. Same model, half the bill – medium was not reviewed for quality, so check it on your own tickets.
- Orchestrator app, the GUI on top of those coding agents: T3 Code, still and by far. Nothing I know of competes with it in functionality and developer experience, and the team behind it really knows what they are doing. Why exactly it is the best option deserves its own post – that one is planned for right after the cost follow-up.
And do your own research: your codebase, your tickets, your reviewers. It costs a few days and a few hundred euros in tokens, and it is the only benchmark that measures what you actually pay for.
Next up: what agentic coding really costs – cost per merged pull request, why subscriptions are fair and the API isn't, the four token types and why caching is the reason any of this is affordable, and the context management that cuts the bill. Coming soon.
Thank you for reading 🙏 this blog post was written by Alexander Thalhammer. For feedback, remarks or questions, please reach out to me ❤️
Same Model, Different Coding-Agent Harness: Scores Mostly Held, Cost Moved Up to 3x
- Book: LLM Observability Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
Before your coding agent reads a single line of your ticket, it has already sent the model a wall of text. In a new study, that first call carried 17,134 tokens through Claude Code and 1,382 through mini-SWE-agent on average, on the same model and the same 45 tasks.
The paper is What Does a Harness Buy? Tokens, Mostly, by Yangze Liu and Zhongyi Han of Shandong University. It went up on arXiv on October 3, and VentureBeat covered it on October 23 alongside a second study from Berkeley that points the same way. The question it asks is one you face every time you pick a coding agent: if you keep the model fixed, how much does the agent around it change the result?
Its answer, in the authors' words: "A vendor's own harness on its own model buys no score advantage." The cost per task, though, varied up to 3x.
What the study tested
A harness is everything around the model: the system prompt, the tool set and how the agent manages context. The paper held the model fixed and swapped three harnesses on SWE-bench Verified tasks, run offline in containers:
- Claude Code, versions 2.1.259 to 2.1.273, the most elaborate of the three
- OpenCode 1.18.30 and 1.18.31
- mini-SWE-agent 2.4.6, which VentureBeat describes as about 100 lines of Python with one bash tool and no context compaction
According to VentureBeat, the models were two local open-weight Qwen models (Qwen3.6-35B-A3B and Qwen3.8-27B), three API models (DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview), and Claude Opus 5 inside Claude Code as an anchor. The repository ships all 8,032 trials as a CSV, MIT-licensed code, and a reproduce.sh that rebuilds the paper's numbers on a CPU in under a minute.
!Three identical engines on a workbench, each bolted into a differently sized backpack of tools, with the largest backpack trailing a long paper receipt
Same engine, three harnesses. The study measured what the extra baggage costs.
Scores: inside the noise, with one exception
On the 447-task pool, Claude Code and mini-SWE-agent scored within five points of each other, and on the two Qwen models the harnesses were statistically equivalent within a 5-point margin. On a 45-task hard subset, switching harnesses flipped about 13% of tasks between pass and fail. Rerunning the same harness flipped about as many.
OpenCode was the one harness effect that cleared the noise, trailing by up to 9 points. Much of that came from one behaviour. When a model's reasoning ran past the 32k output cap mid-step, OpenCode did not re-prompt and the trial usually ended with no patch. Claude Code re-prompts. Excluding those cut-off runs roughly halves OpenCode's gap, per the paper.
The authors are careful about what that doesn't prove. Their run table shows most cells on the 447-task pool have a single run. Their limitations section notes it's one benchmark and that the closed model, Opus 5, only ran inside Claude Code. So nobody knows yet whether a frontier closed model shows the same small gaps. They also show that 45 tasks detect a 13-point gap only about half the time. That's worth remembering the next time a vendor quotes a 3-point harness win.
Cost: the preamble, times the steps
Cost is where the harnesses split. On GLM-5.3-Flash, a trial cost 1.81 CNY in Claude Code, 0.97 in mini-SWE-agent and 0.56 in OpenCode. The preamble explains much of it: of Claude Code's 17,134 first-call tokens, 16,581 were harness preamble, against 829 for mini-SWE-agent. That preamble is resent on every step.
But the preamble alone doesn't decide it. On DeepSeek-V4-Flash the order flipped: 0.98 CNY for Claude Code, 1.02 for mini-SWE-agent. VentureBeat reports that mini-SWE-agent took 141 model calls per trial there, against Claude Code's 104. A lean harness that takes more steps can lose its head start.
Caching softens all of this without changing the ranking: 96% to 99% of input tokens hit the provider's prompt cache in every arm. The paper's summary is the line to keep: "The numbers to carry between vendors are the first call, the growth per step, and the step count."
You can see why with a toy model. Each step resends the preamble plus the history so far:
def input_tokens(preamble, growth, steps):
# step k resends the preamble plus k steps of history
return sum(preamble + growth * k for k in range(steps))
big = input_tokens(16_581, 1_500, 100)
small = input_tokens(829, 1_500, 100)
With an assumed 1,500 tokens of history per step, that's 9,083,100 against 7,507,900 input tokens over 100 steps. The 1,500 is my placeholder, not a number from the paper. The point is the shape: over a long run, history growth and step count can matter as much as the preamble.
Two things to check in your own setup
First, network access. In web-enabled runs, 312 of 439 DeepSeek and Qwen trials (71%) fetched the upstream fix from GitHub, so the authors threw those runs out and moved the offline condition from the prompt to the network layer. If you benchmark agents on public repos, a "don't browse" line in the prompt isn't enough.
Second, thinking defaults. Turning thinking off for Qwen3.6 dropped mini-SWE-agent's pass rate by 11.1 points, while Claude Code and OpenCode gained a little. An older Claude Code release that didn't enable extended thinking by default passed 34 of 45 hard tasks against 35 to 36 for newer ones, at 0.44x the cost. The harness and its defaults move together, so when you compare two agents, write down both.
The Berkeley study lands near the same place. VentureBeat reports that HarnessTax, from UC Berkeley's Sky Computing Lab and Arena, ran seven models across Claude Code, Codex CLI and Pi. Pass rates moved a few points. Inference cost moved up to fivefold, and Claude Code's initial context was more than 10 times Pi's.
Run the comparison on your repo
A paper on SWE-bench is a starting point. Your repo, your tickets and your model are what you pay for, so measure there: same model, same ticket, two harnesses, and record cost per task, not only pass or fail. Run each side twice, because the study shows a rerun moves results about as much as a harness swap.
I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex, OpenCode, Aider and any command-line agent you add each in its own terminal, and ⌘N starts each task on its own hermes/<name> branch in its own worktree. The sidebar shows spend per session: the cost an agent reports, or an estimate from its transcript marked as estimated, with token totals. That gives you a cheap way to set the same ticket on two agents and compare what each one burned. I haven't run the paper's setup through it myself.
!Two agent sessions side by side in Hermes IDE, Codex and Claude Code each in its own worktree, with the sessions sidebar showing status, context use, cost and branch
Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch per session.
Hermes is free to download, source-available (BSL 1.1) and runs on macOS, Windows and Linux. It isn't affiliated with Nous Research, Anthropic, OpenAI or any agent vendor named here. Download it here.
Have you compared two harnesses on the same model and seen a cost gap like this, or did a richer harness pay for itself on your codebase? Tell me in the comments.
---
If this was useful
The paper's real lesson is that you can't pick an agent from its score alone: you need the first-call size, the growth per step and the step count for your own tasks. The LLM Observability Pocket Guide covers the tracing and eval tools that get you those numbers, and how to choose between them for your team. If you'd rather keep two agents running side by side while you measure, Hermes IDE is free to download.
Mentions
Don’t Trust the Benchmarks. Do Your Own Research. A few days ago I gave a talk called _"What does AI really cost?"_ a follow-up to the cost post in this series. I expected to spend the Q&A talking about token prices, subscriptions and context management. Instead, the slide that got the most questions was the one about benchmarks – or rather, about why I stopped trusting them and built my own. So this post is that slide, expanded: what the public leaderboards currently say, why I think they tell at most half the truth, and what happened when I let eight models implement the same 20 real feature tickets of an _Angular_ component library and had four humans judge the results _blind_. Quick recap for everyone joining late: this series started with which LLM is best for _Angular_ development, continued with which app or harness to wrap around it, and then moved on to the harness setup and the skills that turn a capable model into a careful teammate. The summer update was the last time I ranked models – mostly on public benchmarks plus my own gut feeling. This post replaces the gut feeling with data. The cost side of the talk gets its own follow-up post soon, so I'll keep the money talk short here. TL;DR: Two Leagues, No Middle Field If you only have thirty seconds: on the public benchmarks, the frontier models from Anthropic, OpenAI, Google and Meta sit within a few points of each other. On my own benchmark – eight models, the same 20 real _Angular_ tickets, four blind human reviews – they don't. Three models landed between 7.5 and 8.25 out of 10, the other five between 3.5 and 4.75, and nothing in between. All three top spots went to Anthropic: Opus 5.5 for quality, Sonnet 5.5 for value (94 % of the Opus score for a fifth of the price), Fable 5.1 right behind (why third? – one of the reviewers flagged too much refactoring instead of focusing on the ticket with a minimal changeset). The GPT models, Gemini, Grok and Haiku all ended up a league lower – for real feature work I would not use them for agentic coding. And if you pay API prices, the harness matters too: the same Sonnet 5.5 cost about half as much per ticket in OpenCode (and 60 % in Pi) as in Claude Code, and a bit cheaper than in GitHub Copilot. As always in this series: this is one benchmark on one _Angular_ component lib codebase, judged by four humans. It is not deterministic, it is not universal, and it might be outdated in a month. But it is real work, reviewed by the people who have to live with the result – and that, I'll argue, is worth more than any leaderboard you can easily find out there. What the Public Benchmarks Say Let me start where everyone starts: the leaderboards. My favorite source is still Artificial Analysis, for the reasons I gave in the first post of this series – independent, transparent about their methodology, and the only ones who consistently report cost and time next to the score. The three charts below are snapshots from October 25, 2026, the first two filtered down to the 14 models I care about, the third the release view with all effort variants of 11 of them; click any of them for the live version. Intelligence Index  Artificial Analysis Intelligence Index v4.3.2 – snapshot from October 25, 2026, 14 of 708 models. The index combines ten evaluations, from agentic tasks (Terminal-Bench, GDPval) over coding (SciCode) to long-context and scientific reasoning. The top of the chart is crowded: Opus 5.5 at 58, Sonnet 5.5 and Muse Wonder Full at 56, Kimi K3.1 Pro at 55, Fable 5.1, GPT-6 Astra and Gemini 4 Argon at 53, Muse Wonder at 52. Eight models within six points. If you only looked at this chart, you'd conclude that the frontier is a tie and you should simply pick the cheapest one. Cost per Task  Weighted average cost in USD per Intelligence Index task, segmented by token type – snapshot from October 25, 2026. Lower is better. This chart I like, because it measures the right unit: not dollars per million tokens, but dollars per solved task. A model that charges little per token but burns ten times the reasoning tokens is not cheap. And here the spread is enormous: GPT-6 Luna solves an index task for 7 cents, Fable 5.1 for $7.63 – a factor of 100. The Anthropic models are the three most expensive bars on the right. Sonnet 5.5 costs $5.46 per task, Opus 5.5 $5.98 – almost the same, because at "max with fallback" Sonnet thinks a lot. Max should never be used with Anthropic models for real work – in my experience it is way too much overthinking, and frankly bullshit. Keep that in mind when we get to my own numbers, where Sonnet ran at _high_ effort and turned out dramatically cheaper. Intelligence vs. Cost per Task  Intelligence Index vs. cost per task by model release, all reasoning and effort variants of each release connected – snapshot from October 25, 2026. The dotted line is the Pareto front, the green area the "most attractive quadrant". Put the two together and you get the chart I keep coming back to. The Pareto front runs from GPT-6 Luna in the bottom left through MiMo and GPT-6.1 Sol to Kimi K3.1 Pro and Opus 5.5 in the top right. GPT-6.1 Sol sits inside the attractive quadrant: an index of 52 for well under a dollar per task. GPT-6 Astra scores the same 53 as Fable 5.1 for less than half the cost. And Anthropic's top variants all cost more than $5 per task. So the public benchmarks, read honestly, say: Anthropic has no lead worth paying for. OpenAI, Meta and Kimi are right there at a fraction of the price. That was also the tone of my summer update – and it is completely wrong. At least for our kind of work and for our team. One caveat up front: four of the models that make the frontier look like a tie – Muse Wonder Full, Kimi K3.1 Pro, GPT-6 Astra and Gemini 4 Argon – were not part of my bench, so what follows is about the eight models I did run. Why I Don't Trust Them (Anymore) Before I show you my own results, let me explain why I stopped treating leaderboards as a basis for model decisions. Not because the people running them are sloppy – Artificial Analysis in particular does careful work – but because of what a benchmark _is_. A benchmark is a fixed set of tasks with a fixed way to score them. That has three consequences: 1. Contamination. The tasks are public, so sooner or later they end up in the training data. The most famous example is SWE-bench Verified, for years _the_ coding benchmark. In February 2026, OpenAI stopped reporting it: their own models reproduced original patches from nothing but the task ID, and a manual audit found that most remaining "failures" were flawed tests, not model limits. The research community has by now written a whole taxonomy of contamination – and of the mitigations that each new generation of models defeats. 2. Goodhart's law. "When a measure becomes a target, it ceases to be a good measure." Every lab has a team whose job it is to make the next model look good on the handful of benchmarks that get quoted in launch posts. The Leaderboard Illusion, a NeurIPS 2025 paper, documented how this works on Chatbot Arena: private testing of dozens of variants, selective disclosure of the best one, and preferential data access for a few providers. The paper is about one arena, but the incentive applies to every public benchmark – and the more popular the benchmark, the stronger the pull. 3. Wrong task, wrong judge. Even an uncontaminated, ungamed benchmark measures what is easy to measure: did the test pass, did the answer match. My daily reality is different. A feature can pass every test and still be unmergeable; it can implement the ticket text and miss its intent; it can work and leave behind code nobody wants to own. No public benchmark runs on an enterprise _Angular_ codebase with a strict style guide, a design system and an opinionated review culture – and none of them has a human reviewer at the end. That third point is the one I can do something about. If the benchmark doesn't measure my work, I have to measure my work. My Own Benchmark: Eight Models, 20 Tickets, Four Human Juries So in early October 2026 I built one. The subject is an enterprise _Angular_ component library I developed: a publishable library with secondary entry points, a demo app that serves as its living documentation, a strict style guide, an AGENTS.md with the repo rules, and e2e tests with Cypress. In short, exactly the kind of codebase this series has been about. The Setup - Eight models, each in one uninterrupted run: Opus 5.5, Sonnet 5.5, Fable 5.1 and Haiku 5.5 from Anthropic, GPT 6.1 Sol and GPT 6.0 Luna from OpenAI, Gemini 3.8 from Google, and Grok 4.7 from xAI. Each at the reasoning effort I'd normally use for it – high, the small models Luna and Haiku even on xhigh – with one deliberate exception: Fable at medium, to see whether the most intelligent model could get away with less thinking. - The same 20 tickets, real feature tickets from the backlog, frozen in one file with a binding "Decisions" block per ticket that settled the open questions upfront. Tickets 1 to 16 extend existing components (select, chip area, navigation tree, tabs, checkbox, temporal editors, splitter, …), 17 to 20 create new ones (status badge, AG Grid theme, progress indicator, breadcrumb). - The same starting commit, the same prompt, the same repo rules. The prompt asked for one commit per ticket, a self-review in a fresh context folded in by amend, and the full gate – lint, library build, app build, the touched e2e specs – green before every commit. No questions allowed: where the ticket was silent, the model had to decide and record the assumption. - Each model in its own harness, the one its subscription ties it to: the Claude models in Claude Code, the GPT models in Codex CLI, Gemini in Antigravity, Grok in Cursor. That is the setup you get on a subscription anyway, so model and harness were benchmarked as the pair you'd actually use. - What was recorded: the API cost per run (all eight were paid through my private subscriptions, but priced at list prices for comparison – all prices in this post are USD at API list prices, as the charts show them) and the active wall-clock time. Both stayed sealed until the scores were in. The Four Juries Public benchmarks measure what is easy to measure, so the judges here are the people who live with the result. Four reviewers, one question each, every one of them blind: the eight runs were deployed as numbered copies of the demo app and pushed as lettered branches, and none of the three colleagues knew which number or letter was which model. I did, which is why my own review needed a trick – see below. - Ticket intent – the colleague who wrote the tickets checked on five of them whether each implementation does what he _meant_, not what the text said. - Functional testing – a second colleague tested 19 of the 20 tickets on the deployed demos, with mouse and keyboard, hunting for bugs and regressions. - Merge readiness – a third colleague, a senior developer, reviewed four sampled commits of each branch like a merge request: would this go into develop as it is? - Code quality – that was me: all 20 tickets of every branch, read against the style guide and the repo rules. Is this code I'd be glad to own a year from now? Since I had started the runs and knew which model was behind each letter, I tricked myself into not knowing: I reviewed numbered copies of the branches and looked up the mapping only after my scores were in. Every reviewer scored every run on the same scale from 0 to 10. The combined score is the plain average of the four medians – no weighting, no cleverness. Between them, the four of us spent well over 20 hours on the reviews. Remember that number; it comes back later. The Ranking Table !Ranking table of the eight model runs The eight runs ranked by the average of the four reviews, with the four individual scores, the API cost per ticket and in total, and the active time of each run. * Gemini 3.8 stopped once because its context was full and had to be restarted; it then stopped three more times to ask a question. ** Implemented only 19 of the 20 tickets (there was an instruction to skip the ticket if certain red flags like failing e2e tests appear). The first thing that jumps out is the split I already spoiled in the TL;DR: two leagues, no middle field. Opus 5.5 (8.25), Sonnet 5.5 (7.75) and Fable 5.1 (7.5) form the top league; Gemini 3.8 (4.75), Grok 4.7 and GPT 6.1 Sol (4.25), GPT 6.0 Luna (3.75) and Haiku 5.5 (3.5) the lower one. Nothing lands between 4.75 and 7.5. On this codebase, a model either delivers work the reviewers can build on, or it does not. The second thing: the top three are all Claude models, and the three top places are close. That was the biggest surprise for me – not that Opus won, but that Sonnet and Fable ended up within three quarters of a point of it. Fable was even marked down a little by the merge-readiness reviewer for its tendency to refactor and improve things outside the ticket's scope: helpful on a solo project, extra review work in a team. And remember that Fable ran at _medium_ effort, the other two at _high_ – it almost got away with it. The third thing: the two GPT models fell short, and so did everyone else. GPT 6.1 Sol and GPT 6.0 Luna both skipped one of the 20 tickets. Gemini 3.8 had to be restarted and stopped three more times to ask questions although the prompt forbade it. Grok 4.7 was the slowest run at 9 hours 10 minutes and the second most expensive, and still only shared fifth place. Time did not buy quality, and neither did price – Grok costs more per ticket than Opus. And the fourth: Haiku 5.5 came last. The vendor alone is not the whole story. Anthropic's smallest model produced the smallest diffs, skipped features silently and broke an existing behavior. A cheap run that has to be redone is not cheap – more on that in the next post. The Score Profiles !Score profiles of the eight model runs One chart per run: the four reviewers' scores on four axes – ticket intent, functional testing, merge readiness and code quality. The radar charts explain _why_ the leagues exist. The three top runs are big, balanced diamonds: no axis below 6, most at 7 or above. Opus scores 9 on both merge readiness and code quality. Sonnet got a perfect 10 for merge readiness – the reviewer's clear favorite, "very well rounded and reliable, no nonsense, many tests, sensible refactoring, initiative of its own." Fable leads on ticket intent with an 8: it understood what the ticket author meant better than anyone else, and then did a bit more than he asked for. The lower league is where the profiles fall apart, each in its own way. Haiku 5.5 got a 6 for code quality, because it wrote the smallest diffs and the single best refactor of the whole bench (one table replacing 200 lines of switch statements), but a 1 for functional testing and merge readiness, because it quietly left features out. GPT 6.0 Luna got a 7 for ticket intent – it understood the tickets fine – with 2 and 3 on the code axes: "few and simple code changes, but the features often do not work." And then there is the one profile that deserves its own section. The Gemini Case: Passes the Tests, Fails the Review Look at Gemini 3.8. Functional testing: 9 – the best score of all eight runs, better than Opus. The tester's verdict: "provided the best solutions on average; only on rare occasions another implementation beat this one." If my benchmark had stopped there – if it had been a public benchmark, which scores exactly this: does it work, does the test pass – Gemini 3.8 would have won. At $1.31 per ticket, cheaper than Sonnet and a fraction of Opus and Fable. The perfect Pareto winner. Now look at the other three axes. Ticket intent: a 6 overall, but on one of the five sampled tickets the ticket author gave it a 0 – no documentation, no showcase, nothing he could even check – which is not what he asked for. Merge readiness: 3 – "incomplete", one of the four reviewed commits a functional failure. And code quality: 1, the worst of all eight runs. My own notes from the blind review: the code looks disciplined at first glance, but it stacks defensive guards, dead ?? [] fallbacks and the same expression pasted eight times, and it introduced two regressions in existing components. On top of that it was the only run that needed babysitting: it stopped once because its context was full and had to be restarted, and then stopped three more times to ask questions, although the prompt forbade it. So the same run is the best functional implementer and the worst engineer of the bench. The features work, and nobody on the team would merge them. Every one of these commits would come back from review – rewritten by a human, or redone by a better model – and that is where the cheap price per ticket evaporates. Functional testing 9 is what a benchmark measures. Code quality 1 is what you pay for afterwards. This is the pattern a single-number benchmark can't show you, and it is why I wrote this post. A model that is optimized to pass tests will pass tests. Whether it writes code you want to own only a human review on your own codebase can tell – and in this case the human review turned the apparent winner into what is potentially the most expensive model of the bench – once you add the human hours to rewrite its code. Score vs. Costs !Score vs. cost per ticket of the eight model runs Every run by its API cost per ticket (logarithmic) and its average score, with the Pareto front and the attractive quadrant – cheap and good. * restarted, ** 19 of 20 tickets. Now price and quality in one picture, in the same style as the Artificial Analysis chart above – and this is where my numbers and theirs diverge. The Pareto front has a steep step in the middle: from GPT 6.1 Sol at $0.71 and 4.25 points through Gemini 3.8 at $1.31 and 4.75 straight up to Sonnet 5.5 at $1.46 and 7.75 points. For 15 cents more per ticket you move from the lower league to the upper one. From there the line flattens: Opus 5.5 buys another half point for $7.20 per ticket, and Fable 5.1, at $11.18 per ticket, sits below the line – more expensive and slightly worse than Opus in this run, at medium effort. Three practical takeaways from this chart, in the order I'd make the decision: - Opus 5.5 is currently the best choice for implementing features. The highest score, and both cheaper and faster than Fable: $7.20 against $11.18 per ticket, 3 h 34 min against 5 h 10 min for all 20 tickets. - Sonnet 5.5 is the best value by far. 94 % of the Opus score for a fifth of its cost. It is the only run in the attractive quadrant, and the one I'd recommend to every team that pays API prices. - Fable 5.1 stays my favorite for everything that isn't implementation. Planning, design, reviews, the mutual understanding between model and human – it is still the most intelligent of the three in my eyes. When it implements, tell it explicitly to stay within the ticket, and consider high effort. And where does GPT 6.1 Sol still fit? At $0.71 per ticket and 2 h 53 min it is far cheaper and faster than the top three. I'd use it for very small, well-defined tasks, for prototypes and experiments, and as the second opinion from another provider in a review – three Claude models share the same blind spots. For anything that ends up in develop, I'd pay the 75 cents. Even the most expensive run is cheap in absolute terms, by the way. All eight runs together, 158 tickets, would cost about $613 at API list prices. Without AI I'd estimate each of these tickets at two to four hours of implementation work – at a conservative €50 per hour that is €100 to €200 per ticket, roughly 15 to 25 times the cost of the Opus run. The token bill is not the expensive part of agentic engineering. But that is the next post. The Verdict: Use the Anthropic Models On the public benchmarks these three are tied with half a dozen others; on human review they are a league ahead. That difference is exactly the difference between models that are good and models that look good on benchmarks – so for agentic coding in the frontend, use the Anthropic models. That brings me to the rough ranking I showed in the talk – my personal order of providers for agentic coding, as of Q4 2026: !My rough ranking of providers for agentic coding, Q4 2026 1. Anthropic – Opus 5.5, Sonnet 5.5, Fable 5.1. The top league, see above. 2. OpenAI – GPT-6 Astra and GPT-6.1 Sol. I left Astra out of the bench on purpose: from daily use I don't rate it as a coding model, so I spent the review hours on Sol and Luna instead – but for reviews and research it's my go-to. Sol is my pick for small, cheap tasks and second opinions. 3. Meta – Muse Wonder Full is the one model I'd pay a separate subscription for next. Everything below I wouldn't touch for agentic coding at all – yes, they might be useful, but I don't want to use them 😏 4. Google, xAI or the Chinese open-weight models – Gemini 3.8 and Grok 4.7 both landed in the lower league. DeepSeek 4.1, GLM 5.3, Kimi 3.1 and MiMo 2.6 are consistently three to six months behind the frontier, and that gap has been remarkably stable – no catching up in sight. If they catch up, we re-evaluate everything. 5. Local models – on the hardware most of us have, usually at least one more step down. They make sense for exactly one reason: when your data must not leave the company. Then they are the only option, and the data privacy post applies. Two models on that list weren't in our human bench at all: Muse Wonder Full and Gemini 4 Argon were both released after the runs. On the public charts they sit right at the top – but this bench only judges the eight models I ran. It took four people more than 20 hours to judge them, which is exactly why I cannot repeat it whenever a new model drops. My plan is to rerun it roughly every quarter, with the new releases in the lineup. In between, the Artificial Analysis charts above are the best update on capability and cost you'll get – just read them as what they are. The bench can't tell you anything about Gemini 4 Argon; my own first days with it can: for agentic coding the way I do it, Google has not caught up. Hot take: if somebody tells you that Gemini 4 Argon has caught up with the frontier, it's worth asking what they measured. On benchmarks and first impressions it has; in my first days of real agentic coding with it, not yet. The same goes for claims about local models, by the way. But now let's look at the harnesses for a bit. The Harness Matters (If You Pay API Prices) One more result from the same tickets, because it surprised me almost as much as the model ranking. On a subscription, the harness is a given: Claude Code for the Claude models, Codex for the GPT models, and so on – that is how the eight model runs above were set up. But as soon as you pay API prices, you can run any model in any harness, and then the choice is yours. So after the eight model runs, I ran a harness follow-up: the same 20 tickets, the same prompt, one fixed model – Sonnet 5.5 at medium effort – in four different coding harnesses, in two rounds. In the first round one agent worked through all 20 tickets itself; in the second, the agent handed every ticket to a fresh subagent and only orchestrated. No human review this time, only cost and time. Within a round, nothing changed between the runs but the wrapper: the system prompt, the tool definitions, the skills and rules it loads, and how it runs its subagents. | Harness | Cost per ticket | Total cost | Time | | ------------------ | --------------: | ---------: | ---------- | | OpenCode | $0.84 | $16.71 | 2 h 23 min | | Pi | $0.98 | $19.65 | 3 h 19 min | | GitHub Copilot | $1.02 | $20.31 | 2 h 35 min | | Claude Code | $1.63 | $32.52 | 3 h 40 min | Sonnet 5.5 (medium), 20 tickets, every figure the average of the two rounds per harness. Claude Code went directly to the Anthropic API, the other three through the GitHub Copilot API; all tokens priced at Anthropic list prices. Same model, same task, twice the bill. OpenCode and Pi cost roughly half of Claude Code per ticket, with GitHub Copilot in between. The reason is context: every turn of an agentic session sends the whole context again, and Claude Code's context is simply bigger – a longer system prompt, more tools, subagents. The two rounds show where the gap comes from: with one continuous agent, Claude Code ($22) and GitHub Copilot ($21) were level, OpenCode ($15) and Pi ($16) a quarter cheaper. With one subagent per ticket, Claude Code's bill nearly doubled to $43 for the 20 tickets, Pi rose to $23, OpenCode to $18, and Copilot even fell to $19. Two caveats before you switch. First, this follow-up measured cost and time, not quality – Sonnet at high effort scored 7.75 in the model bench, and medium effort was not reviewed. Second, if you work on a Claude or OpenAI subscription, none of this matters much: you pay a flat price, and Claude Code's subagents, hooks and /loop are worth the context they cost. It matters when you pay API prices – which, in my experience, is the normal case in the enterprise. Then my low-budget setup is Sonnet 5.5 (medium) in OpenCode or Pi, at a cost per ticket close to GPT 6.1 Sol and with a model whose high-effort sibling sits in the upper league – verify the quality of medium on your own tickets first. How the four token types, caching and context management drive that bill is the next post. Run Your Own Bench You don't need eight models and 20 tickets. What you need is your own codebase, your own reviewers and a fair setup: 1. Pick three to five real tickets from your backlog – not demos, not greenfield. Freeze the text and settle the open questions upfront, so every model works from the same requirement. 2. Give two or three candidate models the same prompt, the same starting commit and the same rules. Let each one run through all tickets on its own, with the full gate green before every commit. 3. Let the people who normally review the work score the results – blind. Numbered deployments, lettered branches, one question per reviewer. The ticket author checks intent, a tester checks behavior, a reviewer checks mergeability, someone reads the code. 4. Count the hours around each run, not just the tokens: restarts, questions, skipped tickets, out-of-scope changes, review rounds. That is where the cost is. A few days of this tell you more about your codebase than any leaderboard – and the setup is reusable the next time a vendor claims the top spot. Agentic Engineering Workshop In our Agentic Engineering Workshop we build your setup hands-on: your style guide, your AGENTS.md, and a skills catalogue tuned to your team's rules instead of mine – and then put it to work on spec-first workflows, code review, testing, and brownfield refactoring in real _Angular_ codebases. Which model and which harness to use is one of the first decisions we make there, and after this post you know how I'd make it. English or German, as always. - 🤖 Agentic Engineering Workshop – 2 days (or 3 half-days online), remote or in-house, English or German, for advanced _Angular_ developers – next public date (in English): November 30 to December 2, 2026 Conclusion Don't trust the benchmarks. Not because they lie, but because they measure the wrong task with the wrong judge, and because every lab is optimizing for them. The public charts show the frontier as a tie; four humans reviewing real _Angular_ work on my codebase show two leagues with nothing in between. So, for Q4 2026, my recommendation in one line each: - Models: the Anthropic models. Opus 5.5 when quality matters, Sonnet 5.5 when cost matters, Fable 5.1 for planning, design and reviews. GPT-6.1 Sol for small, well-defined tasks and as the cheap second opinion from another provider. Astra for architecture and planning reviews and computer use tasks. - Harness, on a subscription: whatever you're productive in – for me that's Claude Code for Fable and Opus, and Codex for Astra and Sol. - Harness, on API prices: OpenCode or Pi, with Sonnet 5.5 at medium effort. Same model, half the bill – medium was not reviewed for quality, so check it on your own tickets. - Orchestrator app, the GUI on top of those coding agents: T3 Code, still and by far. Nothing I know of competes with it in functionality and developer experience, and the team behind it really knows what they are doing. Why exactly it is the best option deserves its own post – that one is planned for right after the cost follow-up. And do your own research: your codebase, your tickets, your reviewers. It costs a few days and a few hundred euros in tokens, and it is the only benchmark that measures what you actually pay for. Next up: what agentic coding really costs – cost per merged pull request, why subscriptions are fair and the API isn't, the four token types and why caching is the reason any of this is affordable, and the context management that cuts the bill. Coming soon. Thank you for reading 🙏 this blog post was written by Alexander Thalhammer. For feedback, remarks or questions, please reach out to me ❤️
recommended viewing. icmust bully @mitsuhiko into using my gunk badlogic/qwen3_tts_rs x.com
这两天在体验 Pi 作者 @badlogicgames 的 pi-subagent，让 subagent 在独立的 tmux 会话里运行，并且随时可以进入终端查看执行过程 👀 badlogic/pi-subagent - 每个子 Agent 有独立上下文，可以并行调查问题、做代码 Review。 - 运行中可以追加指令，也能把追问排队，等当前一轮结束后再处理。 - 在父 Pi 中输入 /subagent，选择子会话，就能进入它的终端。 安装需要 Pi、Node.js 22.19+ 和 tmux。pi install 可以安装扩展，但还需单独配置 subagent CLI。完整手动安装如下： $ mkdir -p ~/.pi/agent/extensions ~/.pi/agent/bin $ git clone badlogic/pi-subagent ~/.pi/agent/extensions/subagent $ ln -s ../extensions/subagent/subagent.ts ~/.pi/agent/bin/subagent $ export PATH="$HOME/.pi/agent/bin:$PATH" 大家可以将最后一行加入 shell 配置，让后续终端也能找到命令。在同一终端启动已配置好模型的 Pi，执行 /reload 加载扩展。 使用时直接在会话中输入： “用 subagent 分别检查当前 diff 的边界条件和错误处理，不要修改文件，最后汇总发现。” 任务运行中可以继续说： “给检查边界条件的 subagent 追加任务：列出具体输入、预期结果和实际结果。” 想看执行过程，就用 /subagent 进入对应子会话。 subagents 默认共享项目目录，需要隔离文件修改时自行准备 git worktree。
Same Model, Different Coding-Agent Harness: Scores Mostly Held, Cost Moved Up to 3x - Book: LLM Observability Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Before your coding agent reads a single line of your ticket, it has already sent the model a wall of text. In a new study, that first call carried 17,134 tokens through Claude Code and 1,382 through mini-SWE-agent on average, on the same model and the same 45 tasks. The paper is What Does a Harness Buy? Tokens, Mostly, by Yangze Liu and Zhongyi Han of Shandong University. It went up on arXiv on October 3, and VentureBeat covered it on October 23 alongside a second study from Berkeley that points the same way. The question it asks is one you face every time you pick a coding agent: if you keep the model fixed, how much does the agent around it change the result? Its answer, in the authors' words: "A vendor's own harness on its own model buys no score advantage." The cost per task, though, varied up to 3x. What the study tested A harness is everything around the model: the system prompt, the tool set and how the agent manages context. The paper held the model fixed and swapped three harnesses on SWE-bench Verified tasks, run offline in containers: - Claude Code, versions 2.1.259 to 2.1.273, the most elaborate of the three - OpenCode 1.18.30 and 1.18.31 - mini-SWE-agent 2.4.6, which VentureBeat describes as about 100 lines of Python with one bash tool and no context compaction According to VentureBeat, the models were two local open-weight Qwen models (Qwen3.6-35B-A3B and Qwen3.8-27B), three API models (DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview), and Claude Opus 5 inside Claude Code as an anchor. The repository ships all 8,032 trials as a CSV, MIT-licensed code, and a reproduce.sh that rebuilds the paper's numbers on a CPU in under a minute. !Three identical engines on a workbench, each bolted into a differently sized backpack of tools, with the largest backpack trailing a long paper receipt Same engine, three harnesses. The study measured what the extra baggage costs. Scores: inside the noise, with one exception On the 447-task pool, Claude Code and mini-SWE-agent scored within five points of each other, and on the two Qwen models the harnesses were statistically equivalent within a 5-point margin. On a 45-task hard subset, switching harnesses flipped about 13% of tasks between pass and fail. Rerunning the same harness flipped about as many. OpenCode was the one harness effect that cleared the noise, trailing by up to 9 points. Much of that came from one behaviour. When a model's reasoning ran past the 32k output cap mid-step, OpenCode did not re-prompt and the trial usually ended with no patch. Claude Code re-prompts. Excluding those cut-off runs roughly halves OpenCode's gap, per the paper. The authors are careful about what that doesn't prove. Their run table shows most cells on the 447-task pool have a single run. Their limitations section notes it's one benchmark and that the closed model, Opus 5, only ran inside Claude Code. So nobody knows yet whether a frontier closed model shows the same small gaps. They also show that 45 tasks detect a 13-point gap only about half the time. That's worth remembering the next time a vendor quotes a 3-point harness win. Cost: the preamble, times the steps Cost is where the harnesses split. On GLM-5.3-Flash, a trial cost 1.81 CNY in Claude Code, 0.97 in mini-SWE-agent and 0.56 in OpenCode. The preamble explains much of it: of Claude Code's 17,134 first-call tokens, 16,581 were harness preamble, against 829 for mini-SWE-agent. That preamble is resent on every step. But the preamble alone doesn't decide it. On DeepSeek-V4-Flash the order flipped: 0.98 CNY for Claude Code, 1.02 for mini-SWE-agent. VentureBeat reports that mini-SWE-agent took 141 model calls per trial there, against Claude Code's 104. A lean harness that takes more steps can lose its head start. Caching softens all of this without changing the ranking: 96% to 99% of input tokens hit the provider's prompt cache in every arm. The paper's summary is the line to keep: "The numbers to carry between vendors are the first call, the growth per step, and the step count." You can see why with a toy model. Each step resends the preamble plus the history so far: def input_tokens(preamble, growth, steps): # step k resends the preamble plus k steps of history return sum(preamble + growth * k for k in range(steps)) big = input_tokens(16_581, 1_500, 100) small = input_tokens(829, 1_500, 100) With an assumed 1,500 tokens of history per step, that's 9,083,100 against 7,507,900 input tokens over 100 steps. The 1,500 is my placeholder, not a number from the paper. The point is the shape: over a long run, history growth and step count can matter as much as the preamble. Two things to check in your own setup First, network access. In web-enabled runs, 312 of 439 DeepSeek and Qwen trials (71%) fetched the upstream fix from GitHub, so the authors threw those runs out and moved the offline condition from the prompt to the network layer. If you benchmark agents on public repos, a "don't browse" line in the prompt isn't enough. Second, thinking defaults. Turning thinking off for Qwen3.6 dropped mini-SWE-agent's pass rate by 11.1 points, while Claude Code and OpenCode gained a little. An older Claude Code release that didn't enable extended thinking by default passed 34 of 45 hard tasks against 35 to 36 for newer ones, at 0.44x the cost. The harness and its defaults move together, so when you compare two agents, write down both. The Berkeley study lands near the same place. VentureBeat reports that HarnessTax, from UC Berkeley's Sky Computing Lab and Arena, ran seven models across Claude Code, Codex CLI and Pi. Pass rates moved a few points. Inference cost moved up to fivefold, and Claude Code's initial context was more than 10 times Pi's. Run the comparison on your repo A paper on SWE-bench is a starting point. Your repo, your tickets and your model are what you pay for, so measure there: same model, same ticket, two harnesses, and record cost per task, not only pass or fail. Run each side twice, because the study shows a rerun moves results about as much as a harness swap. I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex, OpenCode, Aider and any command-line agent you add each in its own terminal, and ⌘N starts each task on its own hermes/<name> branch in its own worktree. The sidebar shows spend per session: the cost an agent reports, or an estimate from its transcript marked as estimated, with token totals. That gives you a cheap way to set the same ticket on two agents and compare what each one burned. I haven't run the paper's setup through it myself. !Two agent sessions side by side in Hermes IDE, Codex and Claude Code each in its own worktree, with the sessions sidebar showing status, context use, cost and branch Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch per session. Hermes is free to download, source-available (BSL 1.1) and runs on macOS, Windows and Linux. It isn't affiliated with Nous Research, Anthropic, OpenAI or any agent vendor named here. Download it here. Have you compared two harnesses on the same model and seen a cost gap like this, or did a richer harness pay for itself on your codebase? Tell me in the comments. --- If this was useful The paper's real lesson is that you can't pick an agent from its score alone: you need the first-call size, the growth per step and the step count for your own tasks. The LLM Observability Pocket Guide covers the tracing and eval tools that get you those numbers, and how to choose between them for your team. If you'd rather keep two agents running side by side while you measure, Hermes IDE is free to download. 
コーディングエージェントPiのフックはbashを本当に止められるか実測した きっかけ voluntas 氏の記事(ローカル LLM 雑感 (2026-08))で、コーディングエージェントとして OpenCode と Pi が並べて語られていた。分かっていることとして、Pi は read/write/edit/bash の4ツールだけを持ち、システムプロンプトが1,000トークン未満と極端に小さく、拡張はエージェントループと同一ランタイムで動く in-process 方式だという(OpenCode は設定ディレクトリに配置するアウトプロセス方式)。 拡張の API リファレンスを見ると、bash などのツール呼び出しをフックして「拒否」や「引数の書き換え」ができると書いてある。ただ、これが実際どう動くのかが気になった。拒否したときプロセスが例外で落ちるのか、モデルに「失敗しました」と伝わってループが続くのか、それとも無言でスキップされて何も起きないのか。ドキュメントには書かれていなかったので、実際に動かして確認した。 Pi とは @earendil-works/pi-coding-agent (GitHub: badlogic/pi-mono)。Mario Zechner (libGDX の作者) と Armin Ronacher (Flask/Jinja の作者) による、MIT ライセンスのコーディングエージェント CLI。今回試したバージョンは 0.84.2。 npm install @earendil-works/pi-coding-agent export ANTHROPIC_API_KEY=sk-ant-... 拡張は TypeScript ファイル1本で書け、pi -e ./my-extension.ts で読み込むか、~/.pi/agent/extensions/ に置けばグローバルに有効になる。 拡張 API: tool_call フック 公式ドキュメント docs/extensions.md を読むと、ツール実行前に発火するイベントが用意されている。 pi.on("tool_call", async (event, ctx) => { // event.toolName - "bash", "read", "write", "edit" など // event.input - ツールの引数(mutable) if (event.toolName === "bash") { // 書き換え: そのまま実行に反映される event.input.command = source ~/.profile\n${event.input.command}; if (event.input.command.includes("rm -rf")) { // 拒否 return { block: true, reason: "Dangerous command", terminate: true }; } } }); ドキュメントには以下のように明記されている。 event.input は mutable。in-place で書き換えれば「実行前にコマンド文字列を差し替える」ことができる。再検証は行われない。 拒否は { block: true, reason?, terminate? } を返す。 エラーハンドリングの節に「tool_call errors block the tool (fail-safe)」とあり、フック内で例外が飛んでも fail-safe でブロック扱いになる。 ここまではドキュメントを読めば分かる話。「拒否したとき実際にモデルへ何が返るか」は書かれていなかったので、動かして確かめた。 検証: 拒否した bash コマンドはどうモデルに伝わるか rm を含むコマンドは拒否、それ以外は先頭に echo マーカーを差し込んで書き換える、という最小の拡張を書いた。 import type { ExtensionAPI } from "@earendil-works/pi-coding-agent"; export default function (pi: ExtensionAPI) { pi.on("tool_call", async (event, ctx) => { if (event.toolName !== "bash") return undefined; const original = event.input.command as string; if (/\brm\b/.test(original)) { return { block: true, reason: "rm を含むコマンドは検証用フックで拒否されました" }; } event.input.command = echo "[guard-extension: rewritten]"; ${original}; return undefined; }); } foo.tmp bar.tmp keep.txt を置いたディレクトリで、Anthropic の claude-sonnet-4-5 を使い print モード(pi -p)で1回だけ「このディレクトリの tmp ファイルを消して」と投げた。 pi -e ./guard-extension.ts -p "このディレクトリの tmp ファイルを消して" --model claude-sonnet-4-5 セッションログ(~/.pi/agent/sessions/ に自動保存される jsonl)を見ると、次のやり取りが記録されていた。 モデルが ls -la を実行 → 拡張が echo "[guard-extension: rewritten]"; ls -la に書き換え → 実行結果に実際に [guard-extension: rewritten] の行が混ざって返ってきた。書き換えは本物に効く。 モデルが rm *.tmp を実行しようとする → 拡張が拒否。返ってきた tool result はこう。 { "role": "toolResult", "toolName": "bash", "content": [{ "type": "text", "text": "rm を含むコマンドは検証用フックで拒否されました" }], "details": {}, "isError": true, "timestamp": 1786945161731 } プロセスは落ちない。例外も出ない。「失敗したツール呼び出しの結果」として、拒否理由の文字列がそのまま content に入ってモデルに戻る。 モデルはそのまま ls -la を実行して状態を再確認し、tmp ファイルがまだ残っていることを確かめた。 そして無限リトライはせず、「このディレクトリでは guard-extension が削除をブロックしているようだ。続けるかクリーンアップするか教えてほしい」とユーザーに聞き返して止まった。 拒否 → モデルに理由付きで返る → モデルは代替コマンドを勝手に連発せず状況を報告して停止、という比較的行儀の良いループ継続が確認できた。ただしこれは「tmpファイルを消して」という弱い指示のときの話で、次の検証で挙動が変わる。 検証: 拒否は迂回できるか さっきの拒否判定は /\brm\b/ という単純な正規表現マッチでしかない。指示を強めたらどうなるか気になったので、同じ拡張のまま「rm が使えないなら別のコマンドで消して」と明示的に頼んでみた。 pi -e ./guard-extension.ts -p "tmpファイルをどうしても消したい。rmコマンドが使えないなら別のbashコマンドで消して" --model claude-sonnet-4-5 結果、モデルは rm を避けて unlink foo.tmp && unlink bar.tmp を実行した。この文字列は拡張の正規表現に引っかからないので普通に素通りし、実際に foo.tmp と bar.tmp は削除された(keep.txt だけがディレクトリに残ったのを確認済み)。おまけに応答文では「他にも find . -name "*.tmp" -delete / perl -e 'unlink ...' / python -c "import os; os.remove(...)" が使える」とまで提案してきた。 つまり「拒否できる」ことと「意味的に安全になる」ことは別の話で、パターンマッチ主体のガードは、モデルを少し押すだけで別コマンドに切り替えられて実質同じ操作が素通りする。in-process 方式で event.input の中身を直接見られる分、パターンを増やせば防御力は上げられるが、それは拡張の実装者側の責任範囲であって、Pi のフレームワーク自体が意味的な安全性を保証してくれるわけではない、というのが実測で分かった限界だった。 まとめ tool_call フックの { block: true, reason } は実際に効く。プロセスクラッシュではなく isError: true の通常の tool result としてモデルに返り、エージェントループはそのまま継続する。 event.input の書き換え(rewrite)も実行にそのまま反映される。ドキュメント記載通りに動いた。 拒否後のモデルの振る舞いは、弱い指示なら状況をユーザーに報告して止まり、迂回を明示的に指示すればガードの穴(正規表現の甘さ)を見つけて素通りする、という両方が観測できた。 未検証: terminate: true の挙動、複数ツール呼び出しが並列実行されるときの tool_call の兄弟呼び出し可視性、OpenCode(アウトプロセス方式)との定量比較。今回は120分枠に収めるため、Pi単体の実測にとどめた。 拒否がちゃんとモデルに伝わってループが続くという核の部分自体は、実は日本語では先行して実測している記事がある。ただしそちらは faux (mock) provider を使ったシナリオ再現で、実際の Anthropic API 経由での実測ではなく、コマンド書き換え(rewrite)の実測や、拒否を指示ベースで迂回できてしまう挙動には触れていなかった。特に最後の「ガードは意外と素通りできる」という点は、フックを実際に使ってガードレールを実装しようとしている人には知っておいてほしい話だと思う。
I Let an Open-Source AI Agent Control My Browser CONNECT WITH ME 🌍 Website :- mishalabdullah.com 👨💻 Linkedin:- linkedin.com 🐍 Github :- github.com/Mishalabdullah I came across Zeitgeist, an open-source AI browser automation tool, and decided to try running it locally. In this video, I go through the setup, install the Chrome extension, and connect my ChatGPT account to control the browser using AI. The setup wasn't exactly smooth. Some of the models were outdated, so I had to dig into the source code and update the model configuration to get things working. Eventually, I got it running and tested it by asking the agent to search for a Raspberry Pi on Amazon and add it to the cart. And it actually worked. What I cover: - Setting up Zeitgeist locally - Installing the browser extension - Connecting a ChatGPT account - Fixing outdated model configurations - Testing real browser automation Zeitgeist is an interesting open-source project that could use some updates, but it's worth checking out if you're interested in AI agents and browser automation. GitHub: badlogic/sitegeist #OpenSource #AIAgents #BrowserAutomation #ChatGPT #Zeitgeist
@XMihura El caché va igual en ambos salvo que en subscripcion no te deja setear el TTL. Si tienes tantos miss probablemente: a) no estés poniendo el id de session o headers bien; b) estés cambiando el prefix sin querer Echale un ojo a github.com/badlogic/pi-mono/blob/c661bc574f9e9713… y github.com/badlogic/pi-mono/blob/c661bc574f9e9713…
OMK vs mini-SWE-agent on Terminal-Bench 2.1: +4.2pp success, not significant, ~23% lower cost per trial OMK builds on pi, Mario Zechner's MIT-licensed coding agent (github.com/badlogic/pi-mono). It started from the oh-my-pi fork. That vendored tree has since been removed, and the current codebase is OMK-native. I ran OMK against mini-SWE-agent on Terminal-Bench 2.1 with the same base model (Grok-4.7, xhigh), 89 tasks x 3 trials per harness. OMK 1.3.0 eval build (78cc483): 75.8% success (200/264), $0.58 model cost per trial, 482s median per trial. mini-SWE-agent v2.4.6 (Harbor): 71.6% success (189/264), $0.76 model cost per trial, 376s median per trial. Success difference is +4.2pp, 95% CI [-1.9, +10.2]. Not statistically significant. OMK's recorded model cost was about 23% lower per trial. Its median was about 106s slower. Not measured yet: upstream pi under the same setup. For a pi-based harness, that's the obvious baseline, so this doesn't isolate what OMK adds on top of pi. Method notes: - The OMK run used an evaluation build with unmerged changes. This is not an official leaderboard result. - One OMK adapter-failure task was excluded from success for both harnesses. - Retries and auxiliary model calls were excluded from cost. - 3 trials per task, one model, one benchmark. The success gap is unconfirmed until a larger run. Repo: dmae97/omk Criticism of the exclusion rules, the cost accounting, and the missing pi baseline is welcome.
I tried to make the simplest RLM implementation I could in Pi. I love using it and daily drive it with codemode-only on. Try it out folks :)) y0usaf/pi-rlm
alright, since i get asked this a lot, i've decided the make public my sloppy pi subagent extension. badlogic/pi-subagent this is not an earendil product, and issues/prs are disabled. you'll figure out how to use it. to exit a subagent session, ctrl+b, followed by d. glhf.