I gave 9 free AI coding models the same real bug. Here is what happened.
---
title: I gave 9 free AI coding models the same real bug. Here is what happened.
published: true
description: Same open source bug, same brief, hidden tests the models never saw. 8 of 9 free models fixed it, the fastest in 84 seconds. The interesting part was how the others went wrong.
tags: ai, opensource, python, llm
cover_image: raw.githubusercontent.com
---
Everyone has an opinion about which AI coding model is best. Most of those opinions come from toy prompts or leaderboards. I wanted to see what happens on a real bug in a real open source repo, with tests the models cannot see.
So I gave the same bug to 9 free models at the same time and graded every result the same way.
The bug
The repo is pydynox, a Python DynamoDB library. Issue #560: the MemoryBackend test helper works as a decorator, but on an async function it exits its context before the coroutine runs. So the test runs against the real backend instead of the in-memory one. It is a small, realistic bug: easy to describe, easy to get subtly wrong.
The setup
- Agent: OpenCode, run headless with opencode run, one model per run.
- Models: 9 free models from the OpenCode catalogue, all started at the same time, each in its own copy of the repo.
- Brief: identical for all 9. Read the issue and the repo's own AI guidelines, make the smallest correct change, add tests that fail without the fix, do not commit.
- Hidden tests: before the run I wrote my own tests for the bug. They fail twice on the original code. The models never saw them. After each run I added them and ran the suite.
- Grading: hidden tests pass, the model's own tests pass, and a diff review for anything outside the task.
The results
!Time to a finished fix for each of the 9 free models. Big Pickle and Pulse Alpha were fastest at 84 seconds. Crouton Alpha gave up after 10 seconds.
8 of 9 models produced a fix that passed the hidden tests. Big Pickle and Pulse Alpha both finished in 84 seconds. DeepSeek V4.1 Flash took 2 minutes 11 seconds. The rest took between 4 and 6.5 minutes. Total model cost: $0.
Speed was not the story, though. The failures were.
How the models went wrong
Crouton Alpha gave up after 10 seconds. No fix, no tests. That is the easy failure, because you notice it.
Nemotron 3.5 Lightning passed, but deleted an existing test. Its fix was correct and the hidden tests passed. It also removed a test that was already in the file. A green test run would never tell you that. Only reading the diff does.
That is the failure I worry about most with coding agents. Not "it did not work", but "it worked, and quietly changed something you did not ask for".
A second test: a refactor in a 4,091-test suite
One bug is a small sample, so I ran a different kind of task. awslabs/stickler #517 asks for two classes to share their state handling through one private base class. The repo has 4,091 tests, so a refactor has nowhere to hide.
I gave it to 5 free models. All 5 kept the full suite green without editing a single test. Big Pickle was fastest again, at 4 minutes 34 seconds.
One model, Flint 40B, also ran ruff --fix across 8 Jupyter notebooks that had nothing to do with the task. Its diff touched 11 files. The others touched 3. Again: the tests passed, and the change was still not one you would merge.
Free versus paid
For comparison I gave the first bug to three paid models: GLM 5.3 Flash, Kimi K3.1 Pro and Qwen 3.8 Max. All three fixed it, with tests that fail without the fix. They were not faster than the best free models.
There was one real quality difference. Only Qwen detected coroutine functions with inspect.iscoroutinefunction. The other two used asyncio.iscoroutinefunction, which is deprecated from Python 3.14. Both pass today's tests. Only one is still right next year. No test suite catches that. A reviewer who knows the language does.
What I took from it
1. For small, well-described bugs, free models are good enough. 8 of 9 got it right. The gap between free and paid was smaller than I expected.
2. Write your own tests, and keep them hidden. A model grading itself with its own tests tells you little. Tests it never saw tell you a lot.
3. Read every diff. The two most worrying results both passed their tests. One deleted a test, one reformatted files outside the task.
4. The human work moved, it did not disappear. Choosing the issue, writing the brief and the hidden tests, and reviewing the diff took longer than the fixes did. That is the job now.
One more lesson from the open source side: by the time I had a fix for one issue, a human contributor had claimed it 6 minutes earlier. Check the issue again right before you post.
I now use the same workflow for real contributions. Each pull request is reviewed and tested by me and labelled as AI-assisted, which is what the maintainers ask for.
Caveats: one run per model, on two tasks. Free models in this catalogue change often, so treat the names as a snapshot from November 2026, not a ranking.
Mentions
I gave 9 free AI coding models the same real bug. Here is what happened. --- title: I gave 9 free AI coding models the same real bug. Here is what happened. published: true description: Same open source bug, same brief, hidden tests the models never saw. 8 of 9 free models fixed it, the fastest in 84 seconds. The interesting part was how the others went wrong. tags: ai, opensource, python, llm cover_image: raw.githubusercontent.com --- Everyone has an opinion about which AI coding model is best. Most of those opinions come from toy prompts or leaderboards. I wanted to see what happens on a real bug in a real open source repo, with tests the models cannot see. So I gave the same bug to 9 free models at the same time and graded every result the same way. The bug The repo is pydynox, a Python DynamoDB library. Issue #560: the MemoryBackend test helper works as a decorator, but on an async function it exits its context before the coroutine runs. So the test runs against the real backend instead of the in-memory one. It is a small, realistic bug: easy to describe, easy to get subtly wrong. The setup - Agent: OpenCode, run headless with opencode run, one model per run. - Models: 9 free models from the OpenCode catalogue, all started at the same time, each in its own copy of the repo. - Brief: identical for all 9. Read the issue and the repo's own AI guidelines, make the smallest correct change, add tests that fail without the fix, do not commit. - Hidden tests: before the run I wrote my own tests for the bug. They fail twice on the original code. The models never saw them. After each run I added them and ran the suite. - Grading: hidden tests pass, the model's own tests pass, and a diff review for anything outside the task. The results !Time to a finished fix for each of the 9 free models. Big Pickle and Pulse Alpha were fastest at 84 seconds. Crouton Alpha gave up after 10 seconds. 8 of 9 models produced a fix that passed the hidden tests. Big Pickle and Pulse Alpha both finished in 84 seconds. DeepSeek V4.1 Flash took 2 minutes 11 seconds. The rest took between 4 and 6.5 minutes. Total model cost: $0. Speed was not the story, though. The failures were. How the models went wrong Crouton Alpha gave up after 10 seconds. No fix, no tests. That is the easy failure, because you notice it. Nemotron 3.5 Lightning passed, but deleted an existing test. Its fix was correct and the hidden tests passed. It also removed a test that was already in the file. A green test run would never tell you that. Only reading the diff does. That is the failure I worry about most with coding agents. Not "it did not work", but "it worked, and quietly changed something you did not ask for". A second test: a refactor in a 4,091-test suite One bug is a small sample, so I ran a different kind of task. awslabs/stickler #517 asks for two classes to share their state handling through one private base class. The repo has 4,091 tests, so a refactor has nowhere to hide. I gave it to 5 free models. All 5 kept the full suite green without editing a single test. Big Pickle was fastest again, at 4 minutes 34 seconds. One model, Flint 40B, also ran ruff --fix across 8 Jupyter notebooks that had nothing to do with the task. Its diff touched 11 files. The others touched 3. Again: the tests passed, and the change was still not one you would merge. Free versus paid For comparison I gave the first bug to three paid models: GLM 5.3 Flash, Kimi K3.1 Pro and Qwen 3.8 Max. All three fixed it, with tests that fail without the fix. They were not faster than the best free models. There was one real quality difference. Only Qwen detected coroutine functions with inspect.iscoroutinefunction. The other two used asyncio.iscoroutinefunction, which is deprecated from Python 3.14. Both pass today's tests. Only one is still right next year. No test suite catches that. A reviewer who knows the language does. What I took from it 1. For small, well-described bugs, free models are good enough. 8 of 9 got it right. The gap between free and paid was smaller than I expected. 2. Write your own tests, and keep them hidden. A model grading itself with its own tests tells you little. Tests it never saw tell you a lot. 3. Read every diff. The two most worrying results both passed their tests. One deleted a test, one reformatted files outside the task. 4. The human work moved, it did not disappear. Choosing the issue, writing the brief and the hidden tests, and reviewing the diff took longer than the fixes did. That is the job now. One more lesson from the open source side: by the time I had a fix for one issue, a human contributor had claimed it 6 minutes earlier. Check the issue again right before you post. I now use the same workflow for real contributions. Each pull request is reviewed and tested by me and labelled as AI-assisted, which is what the maintainers ask for. Caveats: one run per model, on two tasks. Free models in this catalogue change often, so treat the names as a snapshot from November 2026, not a ranking.