How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark
arxiv.org [𝚌𝚜.𝙲𝚁 𝚌𝚜.𝙰𝙸 𝚌𝚜.𝚂𝙴]
💬Code: snyk-labs/snyk-vulnbench t.co
9 views
How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
We ask what AI coding agents and AI security harnesses find that deterministic static application security testing (SAST) does not, and what they miss that SAST catches. VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications in Go, JavaScript/TypeScript, Python and Java; 62 flow across more than one file. A finding is credited only if it locates the vulnerability's attacker-controlled source and its dangerous sink, each within two lines. We evaluate nine productized configurations under one blind protocol: the Snyk Code static analyzer, Claude Code's /security-review with three model and effort settings (one with the Snyk MCP server), Codex Security with GPT-5.6 Luna, Terra and Sol, and the DeepSec scanner with two model backends. Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart. Neither engine was sufficient alone: together they reached 101 of 108, the ceiling of all seven comparable configurations combined, and the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report. What an AI engine misses depends on its harness as much as on its model: without Claude Code's built-in review policy, the same model matched 16 instead of 9 of the 20 vulnerabilities in the classes that policy excludes or restricts (supplementary run on 19 of 20 applications); Codex Security with GPT-5.6 Sol located 80 vulnerabilities at file level but 34 at line level; DeepSec's model-free file selection kept a file from only 56 of the 108 flows. Coverage and cost were only loosely related. The harness, fixtures, reference sets and all run data are public.
Comments: 44 pages, 21 figures, 7 tables. Benchmark harness, reference sets and all run data: snyk-labs/snyk-vulnbench ; companion site: vulnbench.com
I've put together a fun experiment of Snyk scan + Jev classifier model to filter out noise and focus on remediation advice snyk-labs/vulnwash-cli
Would love for you to try it and share feedback
223 views
I've put together a fun experiment of Snyk scan + Jev classifier model to filter out noise and focus on remediation advice snyk-labs/vulnwash-cli
Would love for you to try it and share feedback
2 likes
VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green
Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal, Luke Tu
arxiv.org [𝚌𝚜.𝚂𝙴 𝚌𝚜.𝙲𝚁]
💬Code: snyk-labs/snyk-vulnbump t.co
22 views
VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green
Software composition analysis (SCA) tools report far more vulnerable dependencies than teams fix: in Snyk Open Source production data, 94% of vulnerability findings have a known fixed version, yet only about 10% are resolved. Agentic coding tools could close this gap, but no benchmark evaluates them on vulnerability-driven upgrades in real, runnable projects across ecosystems, verified independently of the agent. We present VulnBump: 307 upgrade tasks covering 2,066 fixable vulnerabilities in 43 repositories across six package managers, matched to Snyk Open Source production vulnerability distributions. An upgrade passes only if, in a fresh container, dependencies resolve, a rescan shows the targeted vulnerabilities fixed, the project builds, and every previously passing test still passes. A non-agentic baseline that only applies Snyk's recommended version passes 46.9% of upgrades; with the same model (Claude Sonnet 5.5), Snyk's Evo Remediation Agent passes 90.2%, Claude Code + Snyk scan 77.5% and Claude Code unaided 59.6%. The agents fail in opposite ways: Claude Code mostly leaves the vulnerability in place, whereas most of Evo Remediation Agent's failures break the build or tests. An SCA scanner is a key tool for agents: Snyk's guidance raises Claude Code's PASS rate by 17.9 percentage points; without it, Claude Code often settles on a still-vulnerable version. Yet knowing the target is not enough: given the same Snyk findings, Evo Remediation Agent's purpose-built harness passes 12.7 points more than Claude Code + Snyk scan. Major framework migrations remain hard for all harnesses. Because the agents keep projects working at similar rates, evaluating remediation quality in dependency upgrades requires verifying the fix, the build and the tests together.
Comments: 22 pages, 5 figures. Code and data: snyk-labs/snyk-vulnbump
@ptr_steve Better yet, you can just point your agent to my whole setup haha
I named it VulnWash, it's here: snyk-labs/vulnwash-cli
Still rough edges, and not a published CLI yet to npm so kinda at a mid ideation / implementation phase, but it does work if you provide it a Snyk auth token/CLI and a TypeSafe AI Jev token
If you have questions or ideas let me know please!
9 views
@shlomiatar thanks! it's coming along snyk-labs/vulnwash-cli
8 views
Mentions
How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0 Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark arxiv.org [𝚌𝚜.𝙲𝚁 𝚌𝚜.𝙰𝙸 𝚌𝚜.𝚂𝙴] 💬Code: snyk-labs/snyk-vulnbench t.co
How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0 We ask what AI coding agents and AI security harnesses find that deterministic static application security testing (SAST) does not, and what they miss that SAST catches. VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications in Go, JavaScript/TypeScript, Python and Java; 62 flow across more than one file. A finding is credited only if it locates the vulnerability's attacker-controlled source and its dangerous sink, each within two lines. We evaluate nine productized configurations under one blind protocol: the Snyk Code static analyzer, Claude Code's /security-review with three model and effort settings (one with the Snyk MCP server), Codex Security with GPT-5.6 Luna, Terra and Sol, and the DeepSec scanner with two model backends. Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart. Neither engine was sufficient alone: together they reached 101 of 108, the ceiling of all seven comparable configurations combined, and the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report. What an AI engine misses depends on its harness as much as on its model: without Claude Code's built-in review policy, the same model matched 16 instead of 9 of the 20 vulnerabilities in the classes that policy excludes or restricts (supplementary run on 19 of 20 applications); Codex Security with GPT-5.6 Sol located 80 vulnerabilities at file level but 34 at line level; DeepSec's model-free file selection kept a file from only 56 of the 108 flows. Coverage and cost were only loosely related. The harness, fixtures, reference sets and all run data are public. Comments: 44 pages, 21 figures, 7 tables. Benchmark harness, reference sets and all run data: snyk-labs/snyk-vulnbench ; companion site: vulnbench.com
I've put together a fun experiment of Snyk scan + Jev classifier model to filter out noise and focus on remediation advice snyk-labs/vulnwash-cli Would love for you to try it and share feedback
I've put together a fun experiment of Snyk scan + Jev classifier model to filter out noise and focus on remediation advice snyk-labs/vulnwash-cli Would love for you to try it and share feedback
VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal, Luke Tu arxiv.org [𝚌𝚜.𝚂𝙴 𝚌𝚜.𝙲𝚁] 💬Code: snyk-labs/snyk-vulnbump t.co
VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green Software composition analysis (SCA) tools report far more vulnerable dependencies than teams fix: in Snyk Open Source production data, 94% of vulnerability findings have a known fixed version, yet only about 10% are resolved. Agentic coding tools could close this gap, but no benchmark evaluates them on vulnerability-driven upgrades in real, runnable projects across ecosystems, verified independently of the agent. We present VulnBump: 307 upgrade tasks covering 2,066 fixable vulnerabilities in 43 repositories across six package managers, matched to Snyk Open Source production vulnerability distributions. An upgrade passes only if, in a fresh container, dependencies resolve, a rescan shows the targeted vulnerabilities fixed, the project builds, and every previously passing test still passes. A non-agentic baseline that only applies Snyk's recommended version passes 46.9% of upgrades; with the same model (Claude Sonnet 5.5), Snyk's Evo Remediation Agent passes 90.2%, Claude Code + Snyk scan 77.5% and Claude Code unaided 59.6%. The agents fail in opposite ways: Claude Code mostly leaves the vulnerability in place, whereas most of Evo Remediation Agent's failures break the build or tests. An SCA scanner is a key tool for agents: Snyk's guidance raises Claude Code's PASS rate by 17.9 percentage points; without it, Claude Code often settles on a still-vulnerable version. Yet knowing the target is not enough: given the same Snyk findings, Evo Remediation Agent's purpose-built harness passes 12.7 points more than Claude Code + Snyk scan. Major framework migrations remain hard for all harnesses. Because the agents keep projects working at similar rates, evaluating remediation quality in dependency upgrades requires verifying the fix, the build and the tests together. Comments: 22 pages, 5 figures. Code and data: snyk-labs/snyk-vulnbump
@ptr_steve Better yet, you can just point your agent to my whole setup haha I named it VulnWash, it's here: snyk-labs/vulnwash-cli Still rough edges, and not a published CLI yet to npm so kinda at a mid ideation / implementation phase, but it does work if you provide it a Snyk auth token/CLI and a TypeSafe AI Jev token If you have questions or ideas let me know please!
@shlomiatar thanks! it's coming along snyk-labs/vulnwash-cli