A benchmark shipped 2,700 agent trajectories. I recomputed its leaderboard.
Most eval papers give you a table and a PDF. LSCBench gives you the runs.
LSCBench is a secure-code benchmark out of NUS and the Qwen team: 150 real vulnerabilities cut into 450 tasks across seven languages, at three granularities (fill in a function, write a whole file, rebuild a repo's source tree). The task spec never mentions security. Hidden functional and security suites grade the result. Six frontier agents run it under Claude Code or Codex, with a 500-turn cap, interactive tools off, and no network access.
The headline number is that the best model, Claude Fable 5, produces code that is both functional and secure on 25.1% of tasks. Functional completion alone runs 73.6% to 91.3%. In other words, the code mostly works, and mostly keeps the vulnerability.
What got me to actually open the thing: the repo ships the traces. alibaba/last-secure-code-benchmark has a traces/ directory with 2,700 trace JSONs, 450 per model, plus a catalog.json that carries the verdict for every single run. So instead of trusting the table, I recomputed it.
The recompute
I pulled catalog.json and counted, per model over the 450 tasks: functional, secure, and both. Here is my number next to the paper's Table 10.
| Model | FuncPass (mine) | Func ∧ Sec (mine) | Func ∧ Sec (paper) |
| --- | --- | --- | --- |
| Claude Fable 5 | 91.1% | 25.1% | 25.1% |
| GPT-5.6-Sol | 89.1% | 24.7% | 24.7% |
| Qwen3.8-Max | 89.3% | 22.0% | 22.0% |
| GLM-5.3 | 87.1% | 21.3% | 21.3% |
| DeepSeek-V4-Pro | 73.6% | 18.0% | 18.0% |
| Kimi-K3 | 92.2% | 16.4% | 16.4% |
Five rows match to the decimal. The scenario-level column matches too: I get 12.0% for Claude Fable 5 on scenarios secured at all three granularities, which is exactly what the paper reports. The pooled union matches as well, 198 of 450 tasks secured by at least one model (44.0%), 19 tasks secured by all six, 62 secured by exactly one.
One row does not match. Kimi-K3's FuncPass is 92.2% in the catalog (415 of 450) and 91.3% in the paper. That is a four-task gap. It changes no ranking and no conclusion. It is also the only difference I found across 2,700 runs.
That is a good result for the benchmark, and it is the reason the shipped artifact matters at all.
Why the traces matter more than the table
A number you cannot recompute is a claim. A number you can recompute is a measurement. The difference shows up the moment someone else's run disagrees with yours, which is exactly when you need raw outcomes instead of a summary.
I have written before about why offline benchmark scores and online review performance diverge, and the mechanism is always the same: the summary drops the conditions. Shipping per-run outcomes is the cheap fix. It costs a tarball and a catalog file. Compare that to benchmarks where the headline claim rots in months because nobody can re-derive it.
The four-task gap is small enough to be version skew, or a difference in how a missing verdict gets counted. I cannot tell which from outside. That is the point. With the traces shipped, I can at least see that there is a disagreement. With only the table, I would have no way to know one existed.
The reward field is one bit, and it hides two different failures
Every trace carries reward, and it is 1 only when the run is both functional and secure. So a run that rebuilt the module and kept the vulnerability intact scores the same 0 as a run that never produced a compiling build.
Those are different events. One is a security miss, the other is a capability miss, and they call for different responses from anyone reading the results. The paper is careful about this and reports FuncPass, SecPass and Func ∧ Sec separately. The shipped reward column collapses all three back into one bit.
Concretely: 574 of 2,700 runs scored reward 1. If you filter traces by reward to find "the agent gave up", you get the wrong set, because giving up and shipping the bug are indistinguishable there. Sort by reward and you cannot rank a security regression against a build regression. You have to go back to the two booleans.
120 runs ended on a harness error, and 15 of them still passed
The catalog also records exit_reason, which is the field I did not expect and the one I ended up caring about most.
| exit_reason | runs | of which reward 1 |
| --- | --- | --- |
| (none) | 2,580 | 559 |
| NonZeroAgentExitCodeError | 41 | 1 |
| AgentMaxTurnsError | 32 | 6 |
| ApiRateLimitError | 25 | 5 |
| AgentTimeoutError | 16 | 3 |
| VerifierTimeoutError | 5 | 0 |
| AgentProcessSignalError | 1 | 0 |
120 of 2,700 runs, 4.4%, ended on something other than the agent deciding it was done. Six hit the 500-turn cap. Twenty-five died on an API rate limit. Sixteen timed out. Five failed inside the verifier itself, which is not the agent's fault at all.
Now the part that matters. Fifteen of those errored runs still scored reward 1. A run that hit the turn cap or died on a rate limit can still have left a correct, secure implementation behind, so the pass is real. But 2.6% of all passes in the dataset come from a run that never cleanly finished. If you are ranking models on a difference of a couple of points, that is the same order of magnitude as your margin.
The five verifier timeouts are the cleanest example. They are recorded as functional: false, secure: false, which is the same treatment the paper gives a missing verdict by design ("counted as failures when missing"). That is a defensible convention. It is also a convention that turns an infrastructure failure into a model failure, and you can only see it because the traces are there.
What I would check before quoting any of it
Four things, and each takes minutes when the traces ship:
1. Count the rows before you count the scores. 450 tasks times six models is 2,700. If the catalog is short, your denominators are wrong.
2. Recompute functional and secure separately rather than using the collapsed reward. The reward bit merges a security miss with a build miss.
3. Read exit_reason on every run and see how many passes came from a run that errored out. If a model's lead is smaller than its count of errored-but-passed runs, you do not have a ranking yet.
4. Check the scenario-level column, not just the task-level one. Securing a vulnerability at one granularity is no evidence the model understood it. Claude Fable 5 goes from 25.1% on tasks to 12.0% on scenarios, and that gap is the honest measure of consistency.
The takeaway
When a benchmark ships its traces, recompute it before you cite it. You will usually find that the paper is right, and occasionally you will find the one row that is not. Both are more useful than trusting a table, and the second one is only findable if the artifact exists.
If you maintain an eval, the cheapest credibility you can buy is publishing the per-run outcomes next to the summary, including the failure reasons. If you consume one, the cheapest sanity check is counting the rows, splitting the reward bit, and reading the exit reasons. That is three filters between you and a number that sounds better than it is.
Coding Agents Pass Up to 91% of Tasks in a New Benchmark. At Most 25% Are Also Secure
- Book: Ship It — the pragmatic startup tech stack
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
A coding agent rewrote the missing file at the heart of a Java configuration importer. All 14 hidden functional tests passed. All four security tests failed, because the default SnakeYAML constructor it used "honors every !!class tag" in the uploaded YAML (the paper, Section 5.4 and Appendix F.2). The agent was Kimi-K3. Claude Fable 5 and GPT-5.6-Sol parsed the same input with SnakeYAML's SafeConstructor and passed both suites. Qwen3.8-Max and GLM-5.3 made Kimi-K3's mistake.
That case comes from the Last Secure Code Benchmark (LSCBench), submitted to arXiv on October 10, 2026 by researchers from the National University of Singapore and Alibaba Group, including the Qwen team (author list). The headline: six frontier agents pass 73.6% to 91.3% of tasks functionally, and the best of them writes code that is both working and secure on 25.1% (project page).
What LSCBench actually tests
The benchmark has 450 tasks built from 150 real vulnerabilities, across seven languages and 24 CWE categories (arXiv). Of those 150 scenarios, 115 come from public CVEs and 35 are Java applications written by security experts (project page).
The trick is in the prompt. In the authors' words, "Every task asks only for functionality, through a specification that never mentions security" (arXiv). That is how most people prompt a coding agent. You ask for the import feature, not for a deserialization-safe import feature.
Each task runs at one of three sizes: fill in a masked function, write a removed file, or rebuild an emptied source tree from a spec. A task scores only if two hidden suites both pass, functional tests and security tests that replay the original attack. No partial credit, no judge model (project page).
The leaderboard
All six models ran at their highest reasoning effort. Five ran inside Claude Code v2.1.143 and GPT-5.6-Sol ran inside Codex v0.154.0 (README). Numbers from the project page:
| Model | Harness | Works | Works and secure |
|---|---|---|---|
| Claude Fable 5 | Claude Code | 91.1% | 25.1% |
| GPT-5.6-Sol | Codex | 89.1% | 24.7% |
| Qwen3.8-Max | Claude Code | 89.3% | 22.0% |
| GLM-5.3 | Claude Code | 87.1% | 21.3% |
| DeepSeek-V4-Pro | Claude Code | 73.6% | 18.0% |
| Kimi-K3 | Claude Code | 91.3% | 16.4% |
Look at the last row. Kimi-K3 has the highest functional pass rate and the lowest secure one. A green test suite told you nothing about which agent was safer.
Across all 2,700 runs, 66% ended with code that works and is still vulnerable (project page). Of the 150 scenarios, 57 were never secured by any model.
!Editorial illustration of a house whose front door is triple-locked and glowing orange while a back window stands wide open
Working code, open window: the shape of most LSCBench runs.
Where the code breaks
The project page breaks down how often runs ended both working and secure, all six models combined. By language: C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10% (project page).
The weakness-family chart uses a different base, the share of working code that was also secure, and the spread there is wider. Memory safety comes out best at 44%. Injection sits at 28%. Server-side request forgery is at 3% and information exposure at 1% (project page).
The paper's case studies show what that looks like (Appendix F):
- Command injection, Java: four of the six models passed mount arguments to ProcessBuilder as a list. Qwen3.8-Max and Kimi-K3 built a string for sh -c.
- SSRF, Go: in a LocalAI task, GPT-5.6-Sol's GetURI read any path it was handed, file:// URLs included. The reference fix confines reads to the models directory. No model did that.
- Information exposure, Go: in a syft task, GLM-5.3 followed the tag convention of neighbouring structs, so an attestation password showed up in config dumps and SBOM output. All six models made the same choice.
None of these are exotic. A reviewer who goes looking for them can catch all three.
!Editorial illustration of a magnifying glass over a stack of code sheets, with one sheet showing an orange crack running through it
SSRF and information exposure: 3% and 1% of working code was also secure.
One sentence helped. It didn't fix it
The authors reran four of the six models with one sentence appended to every task, asking for an implementation that meets the functional requirements "while also being secure" (Section 5.5). Works-and-secure scores went up:
- GPT-5.6-Sol: 24.7% to 46.2%
- GLM-5.3: 21.3% to 38.2%
- Qwen3.8-Max: 22.0% to 37.6%
- DeepSeek-V4-Pro: 18.0% to 19.1%
Functional pass rates moved by a few points either way. The paper's verdict: "most tasks stay insecure even with the reminder", and SSRF and information exposure stay near the bottom (Section 5.5). Claude Fable 5 and Kimi-K3 weren't in that rerun.
What to do with this on Monday
Add the reminder anyway. The paper appended it to the task. A line in your AGENTS.md or CLAUDE.md is the nearest everyday equivalent, though the paper didn't test that placement. For three of the four models it added about 16 to 22 points.
Then write the test the benchmark writes: replay the attack. If an agent builds a loader that takes a model name, check that it refuses a URL and a path that climbs out of the directory.
from pathlib import Path
import pytest
MODELS = Path("models").resolve()
def resolve_model(uri: str) -> Path:
if "://" in uri:
raise ValueError("plain model names only")
p = (MODELS / uri).resolve()
if not p.is_relative_to(MODELS):
raise ValueError("outside the models dir")
return p
@pytest.mark.parametrize("uri", [
"ftp://example.com/model.bin",
"../secrets.txt",
])
def test_rejects_escape(uri):
with pytest.raises(ValueError):
resolve_model(uri)
Review agent work for security on purpose, in small pieces. I build Hermes IDE, so weigh this with that in mind. Its Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows an agent's changes by turn or by file and sends a line comment back to the agent that wrote it (README). It won't spot an SSRF on its own, but it splits a 40-file change into turns you can actually read.
!The Hermes IDE Review Desk in By turn view for a Claude Code task, with two turns listed, a diff of one file and a drafted commit message
The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns, turn 1's diff of src/db.ts, the changed-files list, and a commit message drafted from the turns.
You can also run the benchmark against your own agent setup. It ships as a Harbor dataset (LSCBench itself is Apache 2.0), and needs Harbor 0.24+, Docker, Python 3.12+ and about 58 GB of task images. The projects contain exploitable vulnerabilities, so the README says to run it in an isolated environment (README). The sanity check first, where reference solutions should score 1:
pip install harbor
harbor run \
-d last-secure-code-benchmark/last-secure-code-benchmark@v1.0 \
-a oracle
I haven't run it myself.
Read the fine print before you quote it
The leaderboard tests the model versions available when the paper was written; newer releases may score differently. Every number is pass@1, one attempt per task. Qwen3.8-Max, one of the six models scored, was also the construction agent that wrote the tests and specifications; the authors argue its mid-field result shows no advantage (arXiv). The paper is an arXiv preprint, so treat it as one until someone replicates it.
---
If this was useful
Vibe-coding a side project is the easy half. Ship It covers the rest of the road from localhost to production for a small team: hosting, payments, auth, CI/CD and monitoring, the places where "it works" and "it's safe to put online" stop being the same thing. If you run several coding agents on that work, Hermes IDE is free, source-available and on GitHub.
Mentions
A benchmark shipped 2,700 agent trajectories. I recomputed its leaderboard. Most eval papers give you a table and a PDF. LSCBench gives you the runs. LSCBench is a secure-code benchmark out of NUS and the Qwen team: 150 real vulnerabilities cut into 450 tasks across seven languages, at three granularities (fill in a function, write a whole file, rebuild a repo's source tree). The task spec never mentions security. Hidden functional and security suites grade the result. Six frontier agents run it under Claude Code or Codex, with a 500-turn cap, interactive tools off, and no network access. The headline number is that the best model, Claude Fable 5, produces code that is both functional and secure on 25.1% of tasks. Functional completion alone runs 73.6% to 91.3%. In other words, the code mostly works, and mostly keeps the vulnerability. What got me to actually open the thing: the repo ships the traces. alibaba/last-secure-code-benchmark has a traces/ directory with 2,700 trace JSONs, 450 per model, plus a catalog.json that carries the verdict for every single run. So instead of trusting the table, I recomputed it. The recompute I pulled catalog.json and counted, per model over the 450 tasks: functional, secure, and both. Here is my number next to the paper's Table 10. | Model | FuncPass (mine) | Func ∧ Sec (mine) | Func ∧ Sec (paper) | | --- | --- | --- | --- | | Claude Fable 5 | 91.1% | 25.1% | 25.1% | | GPT-5.6-Sol | 89.1% | 24.7% | 24.7% | | Qwen3.8-Max | 89.3% | 22.0% | 22.0% | | GLM-5.3 | 87.1% | 21.3% | 21.3% | | DeepSeek-V4-Pro | 73.6% | 18.0% | 18.0% | | Kimi-K3 | 92.2% | 16.4% | 16.4% | Five rows match to the decimal. The scenario-level column matches too: I get 12.0% for Claude Fable 5 on scenarios secured at all three granularities, which is exactly what the paper reports. The pooled union matches as well, 198 of 450 tasks secured by at least one model (44.0%), 19 tasks secured by all six, 62 secured by exactly one. One row does not match. Kimi-K3's FuncPass is 92.2% in the catalog (415 of 450) and 91.3% in the paper. That is a four-task gap. It changes no ranking and no conclusion. It is also the only difference I found across 2,700 runs. That is a good result for the benchmark, and it is the reason the shipped artifact matters at all. Why the traces matter more than the table A number you cannot recompute is a claim. A number you can recompute is a measurement. The difference shows up the moment someone else's run disagrees with yours, which is exactly when you need raw outcomes instead of a summary. I have written before about why offline benchmark scores and online review performance diverge, and the mechanism is always the same: the summary drops the conditions. Shipping per-run outcomes is the cheap fix. It costs a tarball and a catalog file. Compare that to benchmarks where the headline claim rots in months because nobody can re-derive it. The four-task gap is small enough to be version skew, or a difference in how a missing verdict gets counted. I cannot tell which from outside. That is the point. With the traces shipped, I can at least see that there is a disagreement. With only the table, I would have no way to know one existed. The reward field is one bit, and it hides two different failures Every trace carries reward, and it is 1 only when the run is both functional and secure. So a run that rebuilt the module and kept the vulnerability intact scores the same 0 as a run that never produced a compiling build. Those are different events. One is a security miss, the other is a capability miss, and they call for different responses from anyone reading the results. The paper is careful about this and reports FuncPass, SecPass and Func ∧ Sec separately. The shipped reward column collapses all three back into one bit. Concretely: 574 of 2,700 runs scored reward 1. If you filter traces by reward to find "the agent gave up", you get the wrong set, because giving up and shipping the bug are indistinguishable there. Sort by reward and you cannot rank a security regression against a build regression. You have to go back to the two booleans. 120 runs ended on a harness error, and 15 of them still passed The catalog also records exit_reason, which is the field I did not expect and the one I ended up caring about most. | exit_reason | runs | of which reward 1 | | --- | --- | --- | | (none) | 2,580 | 559 | | NonZeroAgentExitCodeError | 41 | 1 | | AgentMaxTurnsError | 32 | 6 | | ApiRateLimitError | 25 | 5 | | AgentTimeoutError | 16 | 3 | | VerifierTimeoutError | 5 | 0 | | AgentProcessSignalError | 1 | 0 | 120 of 2,700 runs, 4.4%, ended on something other than the agent deciding it was done. Six hit the 500-turn cap. Twenty-five died on an API rate limit. Sixteen timed out. Five failed inside the verifier itself, which is not the agent's fault at all. Now the part that matters. Fifteen of those errored runs still scored reward 1. A run that hit the turn cap or died on a rate limit can still have left a correct, secure implementation behind, so the pass is real. But 2.6% of all passes in the dataset come from a run that never cleanly finished. If you are ranking models on a difference of a couple of points, that is the same order of magnitude as your margin. The five verifier timeouts are the cleanest example. They are recorded as functional: false, secure: false, which is the same treatment the paper gives a missing verdict by design ("counted as failures when missing"). That is a defensible convention. It is also a convention that turns an infrastructure failure into a model failure, and you can only see it because the traces are there. What I would check before quoting any of it Four things, and each takes minutes when the traces ship: 1. Count the rows before you count the scores. 450 tasks times six models is 2,700. If the catalog is short, your denominators are wrong. 2. Recompute functional and secure separately rather than using the collapsed reward. The reward bit merges a security miss with a build miss. 3. Read exit_reason on every run and see how many passes came from a run that errored out. If a model's lead is smaller than its count of errored-but-passed runs, you do not have a ranking yet. 4. Check the scenario-level column, not just the task-level one. Securing a vulnerability at one granularity is no evidence the model understood it. Claude Fable 5 goes from 25.1% on tasks to 12.0% on scenarios, and that gap is the honest measure of consistency. The takeaway When a benchmark ships its traces, recompute it before you cite it. You will usually find that the paper is right, and occasionally you will find the one row that is not. Both are more useful than trusting a table, and the second one is only findable if the artifact exists. If you maintain an eval, the cheapest credibility you can buy is publishing the per-run outcomes next to the summary, including the failure reasons. If you consume one, the cheapest sanity check is counting the rows, splitting the reward bit, and reading the exit reasons. That is three filters between you and a number that sounds better than it is.
Coding Agents Pass Up to 91% of Tasks in a New Benchmark. At Most 25% Are Also Secure - Book: Ship It — the pragmatic startup tech stack - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- A coding agent rewrote the missing file at the heart of a Java configuration importer. All 14 hidden functional tests passed. All four security tests failed, because the default SnakeYAML constructor it used "honors every !!class tag" in the uploaded YAML (the paper, Section 5.4 and Appendix F.2). The agent was Kimi-K3. Claude Fable 5 and GPT-5.6-Sol parsed the same input with SnakeYAML's SafeConstructor and passed both suites. Qwen3.8-Max and GLM-5.3 made Kimi-K3's mistake. That case comes from the Last Secure Code Benchmark (LSCBench), submitted to arXiv on October 10, 2026 by researchers from the National University of Singapore and Alibaba Group, including the Qwen team (author list). The headline: six frontier agents pass 73.6% to 91.3% of tasks functionally, and the best of them writes code that is both working and secure on 25.1% (project page). What LSCBench actually tests The benchmark has 450 tasks built from 150 real vulnerabilities, across seven languages and 24 CWE categories (arXiv). Of those 150 scenarios, 115 come from public CVEs and 35 are Java applications written by security experts (project page). The trick is in the prompt. In the authors' words, "Every task asks only for functionality, through a specification that never mentions security" (arXiv). That is how most people prompt a coding agent. You ask for the import feature, not for a deserialization-safe import feature. Each task runs at one of three sizes: fill in a masked function, write a removed file, or rebuild an emptied source tree from a spec. A task scores only if two hidden suites both pass, functional tests and security tests that replay the original attack. No partial credit, no judge model (project page). The leaderboard All six models ran at their highest reasoning effort. Five ran inside Claude Code v2.1.143 and GPT-5.6-Sol ran inside Codex v0.154.0 (README). Numbers from the project page: | Model | Harness | Works | Works and secure | |---|---|---|---| | Claude Fable 5 | Claude Code | 91.1% | 25.1% | | GPT-5.6-Sol | Codex | 89.1% | 24.7% | | Qwen3.8-Max | Claude Code | 89.3% | 22.0% | | GLM-5.3 | Claude Code | 87.1% | 21.3% | | DeepSeek-V4-Pro | Claude Code | 73.6% | 18.0% | | Kimi-K3 | Claude Code | 91.3% | 16.4% | Look at the last row. Kimi-K3 has the highest functional pass rate and the lowest secure one. A green test suite told you nothing about which agent was safer. Across all 2,700 runs, 66% ended with code that works and is still vulnerable (project page). Of the 150 scenarios, 57 were never secured by any model. !Editorial illustration of a house whose front door is triple-locked and glowing orange while a back window stands wide open Working code, open window: the shape of most LSCBench runs. Where the code breaks The project page breaks down how often runs ended both working and secure, all six models combined. By language: C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10% (project page). The weakness-family chart uses a different base, the share of working code that was also secure, and the spread there is wider. Memory safety comes out best at 44%. Injection sits at 28%. Server-side request forgery is at 3% and information exposure at 1% (project page). The paper's case studies show what that looks like (Appendix F): - Command injection, Java: four of the six models passed mount arguments to ProcessBuilder as a list. Qwen3.8-Max and Kimi-K3 built a string for sh -c. - SSRF, Go: in a LocalAI task, GPT-5.6-Sol's GetURI read any path it was handed, file:// URLs included. The reference fix confines reads to the models directory. No model did that. - Information exposure, Go: in a syft task, GLM-5.3 followed the tag convention of neighbouring structs, so an attestation password showed up in config dumps and SBOM output. All six models made the same choice. None of these are exotic. A reviewer who goes looking for them can catch all three. !Editorial illustration of a magnifying glass over a stack of code sheets, with one sheet showing an orange crack running through it SSRF and information exposure: 3% and 1% of working code was also secure. One sentence helped. It didn't fix it The authors reran four of the six models with one sentence appended to every task, asking for an implementation that meets the functional requirements "while also being secure" (Section 5.5). Works-and-secure scores went up: - GPT-5.6-Sol: 24.7% to 46.2% - GLM-5.3: 21.3% to 38.2% - Qwen3.8-Max: 22.0% to 37.6% - DeepSeek-V4-Pro: 18.0% to 19.1% Functional pass rates moved by a few points either way. The paper's verdict: "most tasks stay insecure even with the reminder", and SSRF and information exposure stay near the bottom (Section 5.5). Claude Fable 5 and Kimi-K3 weren't in that rerun. What to do with this on Monday Add the reminder anyway. The paper appended it to the task. A line in your AGENTS.md or CLAUDE.md is the nearest everyday equivalent, though the paper didn't test that placement. For three of the four models it added about 16 to 22 points. Then write the test the benchmark writes: replay the attack. If an agent builds a loader that takes a model name, check that it refuses a URL and a path that climbs out of the directory. from pathlib import Path import pytest MODELS = Path("models").resolve() def resolve_model(uri: str) -> Path: if "://" in uri: raise ValueError("plain model names only") p = (MODELS / uri).resolve() if not p.is_relative_to(MODELS): raise ValueError("outside the models dir") return p @pytest.mark.parametrize("uri", [ "ftp://example.com/model.bin", "../secrets.txt", ]) def test_rejects_escape(uri): with pytest.raises(ValueError): resolve_model(uri) Review agent work for security on purpose, in small pieces. I build Hermes IDE, so weigh this with that in mind. Its Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows an agent's changes by turn or by file and sends a line comment back to the agent that wrote it (README). It won't spot an SSRF on its own, but it splits a 40-file change into turns you can actually read. !The Hermes IDE Review Desk in By turn view for a Claude Code task, with two turns listed, a diff of one file and a drafted commit message The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns, turn 1's diff of src/db.ts, the changed-files list, and a commit message drafted from the turns. You can also run the benchmark against your own agent setup. It ships as a Harbor dataset (LSCBench itself is Apache 2.0), and needs Harbor 0.24+, Docker, Python 3.12+ and about 58 GB of task images. The projects contain exploitable vulnerabilities, so the README says to run it in an isolated environment (README). The sanity check first, where reference solutions should score 1: pip install harbor harbor run \ -d last-secure-code-benchmark/last-secure-code-benchmark@v1.0 \ -a oracle I haven't run it myself. Read the fine print before you quote it The leaderboard tests the model versions available when the paper was written; newer releases may score differently. Every number is pass@1, one attempt per task. Qwen3.8-Max, one of the six models scored, was also the construction agent that wrote the tests and specifications; the authors argue its mid-field result shows no advantage (arXiv). The paper is an arXiv preprint, so treat it as one until someone replicates it. --- If this was useful Vibe-coding a side project is the easy half. Ship It covers the rest of the road from localhost to production for a small team: hosting, payments, auth, CI/CD and monitoring, the places where "it works" and "it's safe to put online" stop being the same thing. If you run several coding agents on that work, Hermes IDE is free, source-available and on GitHub. 