What a Small Model Reveals About Automated Vulnerability Research: John Cartwright and Gadi Evron at [un]prompted 2026
Building an automated vulnerability-research system on Haiku, a model tuned for speed rather than extended reasoning, produced what the speakers call endless failure.
The talk is framed for defenders: it describes the research only at the level of the public session summary, with the value being what it shows about where these systems succeed along with where they still break down.
Automated research systems chain a model through a loop that proposes an approach, tries it, then feeds the result back in, which is the pattern the abstract calls a Karpathy-style auto-research loop. The speakers report that the limiting factor was never the loop itself but whether the system genuinely understood the target code.
With a strong model the engineering became too easy to be instructive, so the team moved to Haiku on purpose. A small fast model avoids extended reasoning along with tool use, which forced the real work back onto the harness: how it represents the target, preserves state, tracks conditions across attempts.
The honest result is that the hardest cases still fail. The team has not achieved full results against difficult real-world memory-corruption targets with modern mitigations in place, naming two public ones as examples.
In their [un]prompted 2026 talk, "Auto-Exploiter by Use of a Ratchet Loop, Code Understanding, and… Haiku," on Wednesday, October 28 at 4:20 p.m. on Stage 1, John Cartwright and Gadi Evron share their failures as openly as their progress.
Speakers:
- John Cartwright, Reverse Engineer (presenter)
- Gadi Evron, Founder and CEO, Knostic (presenter)
The two public targets they name as still-unsolved are well documented. Baron Samedit is CVE-2021-3156, a heap overflow in sudo disclosed by Qualys, while the libwebp bug is CVE-2023-4863, later reassigned as CVE-2023-5129, a heap buffer overflow in the WebP image library that shipped in most browsers.
Both speakers co-authored RAPTOR, the open-source agentic security framework whose harness design this work extends. The documentation describes how it chains static analysis, fuzzing then model reasoning, with the harness techniques the talk focuses on. The published benchmark closest to the task is ExploitGym, which measures whether agents can turn a known vulnerability into a working result across 898 tasks.
John Cartwright is a reverse engineer credited as one of five authors of RAPTOR under the handle u/grokjc, alongside Gadi Evron, Daniel Cuthbert, Thomas Dullien and Michael Bargury.
Gadi Evron is co-founder and CEO of Knostic, along with CFP chair of [un]prompted itself. He founded Cymmetria, a cyber deception company later acquired, and he founded Israel's CERT before serving as CISO of the Israeli National Digital Authority.
He wrote the post-mortem on the 2007 cyberattacks on Estonia, published early work on DNS amplification attacks, then co-authored APT reports on Rocket Kitten along with Patchwork. He owns the RAPTOR repository as its first-listed author, and his Knostic co-founder is Sounil Yu, who gives his own [un]prompted talk on Day 3.
To go deeper on the topic, start here:
- AUDIO: Gadi Evron on SANS Cyber Leaders episode 29, Weathering the AI Vulnerability Storm
- ExploitGym: can AI agents turn security vulnerabilities into real attacks?
- RAPTOR on GitHub
- RAPTOR documentation: how the harness is built
- Qualys on CVE-2021-3156, the Baron Samedit heap overflow in sudo
- libwebp CVE-2023-4863 and CVE-2023-5129 under active exploitation
Defenders weighing how fast automated vulnerability research is advancing get an honest account of where it still fails rather than a highlight reel.
If a small cheap model can be steered toward understanding your code, does your threat model still assume frontier-scale compute?
2 points · 1 comment
Sandboxing an Agent That Runs Its Own Exploits: Daniel Cuthbert at [un]prompted 2026
RAPTOR points an LLM at an unvetted repository, lets it write proof-of-concept exploits, then runs them on a machine holding live API keys.
Nobody on the team was willing to run that outside a sandbox. They built an agent-specific one in April 2026, upgraded it considerably since, then introduced a fair number of their own security bugs along the way.
Everyone describes sandbox construction as easy. Four months of work produced a hardened sandbox along with a list of lessons the speaker calls embarrassing.
In his [un]prompted 2026 talk, "Your Sandbox Degrades Gracefully, What's the Wurst That Could Happen?," on Tuesday, October 27 at 1:55 p.m. on Stage 1, Daniel Cuthbert walks through what breaking their own containment taught them.
Speaker: Daniel Cuthbert, Global Head of Cyber Security Research, Santander
RAPTOR is public. The repository credits five authors: Cuthbert alongside Gadi Evron, Thomas Dullien, Michael Bargury and John Cartwright, with a README stating that subprocesses handling untrusted content run under Linux namespaces.
The documentation covers commands for scanning, agentic analysis, binary work, exploitation, patching, fuzzing and software composition analysis, running through a staged validation pipeline. Dark Reading's coverage describes Thomas Dullien using the patching stage against recently disclosed FFmpeg vulnerabilities, with Cuthbert quoted saying the framework aids multi-candidate proofs of concept rather than replacing expert reverse engineers.
Daniel Cuthbert is Global Head of Cyber Security Research for the Santander Group, along with being an Associate Fellow in Cyber and Tech at the Royal United Services Institute.
He co-wrote the original OWASP Testing Guide in 2003, and he co-authors the OWASP Application Security Verification Standard. OWASP named him a Distinguished Lifetime Member in 2024, its highest honor.
He sits on the Black Hat Review Board, where he has served as Defense Track Lead for Briefings. He is also an industry chair for the UK Government's Cyber Security Advisory Board. He sits on the Department for Science, Innovation and Technology's Cyber Resilience Expert Advisory Group as well.
His conference record includes the BruCON 0x0A keynote in 2018, four Black Hat Locknotes between 2016 and 2021, then "Manna from Heaven" on wireless rogue AP attacks at Black Hat Asia 2015. He publishes RAPTOR's development container on Docker Hub, bundling CodeQL, Semgrep, AFL++, GDB, the rr debugger, Playwright and Claude Code.
To go deeper on the topic, start here:
- RAPTOR on GitHub
- RAPTOR documentation
- New RAPTOR framework uses agentic AI to create patches (Dark Reading)
- Sandlock: confining AI agent code with unprivileged Linux primitives
- Claude Code sandboxing documentation
- OWASP Application Security Verification Standard
Anyone running an agent that executes code it wrote itself gets a list of failure modes from a team that hit them.
What is stopping the coding agent on your machine from reaching the credentials in your environment variables?
2 points · 1 comment
Mentions
What a Small Model Reveals About Automated Vulnerability Research: John Cartwright and Gadi Evron at [un]prompted 2026 Building an automated vulnerability-research system on Haiku, a model tuned for speed rather than extended reasoning, produced what the speakers call endless failure. The talk is framed for defenders: it describes the research only at the level of the public session summary, with the value being what it shows about where these systems succeed along with where they still break down. Automated research systems chain a model through a loop that proposes an approach, tries it, then feeds the result back in, which is the pattern the abstract calls a Karpathy-style auto-research loop. The speakers report that the limiting factor was never the loop itself but whether the system genuinely understood the target code. With a strong model the engineering became too easy to be instructive, so the team moved to Haiku on purpose. A small fast model avoids extended reasoning along with tool use, which forced the real work back onto the harness: how it represents the target, preserves state, tracks conditions across attempts. The honest result is that the hardest cases still fail. The team has not achieved full results against difficult real-world memory-corruption targets with modern mitigations in place, naming two public ones as examples. In their [un]prompted 2026 talk, "Auto-Exploiter by Use of a Ratchet Loop, Code Understanding, and… Haiku," on Wednesday, October 28 at 4:20 p.m. on Stage 1, John Cartwright and Gadi Evron share their failures as openly as their progress. Speakers: - John Cartwright, Reverse Engineer (presenter) - Gadi Evron, Founder and CEO, Knostic (presenter) The two public targets they name as still-unsolved are well documented. Baron Samedit is CVE-2021-3156, a heap overflow in sudo disclosed by Qualys, while the libwebp bug is CVE-2023-4863, later reassigned as CVE-2023-5129, a heap buffer overflow in the WebP image library that shipped in most browsers. Both speakers co-authored RAPTOR, the open-source agentic security framework whose harness design this work extends. The documentation describes how it chains static analysis, fuzzing then model reasoning, with the harness techniques the talk focuses on. The published benchmark closest to the task is ExploitGym, which measures whether agents can turn a known vulnerability into a working result across 898 tasks. John Cartwright is a reverse engineer credited as one of five authors of RAPTOR under the handle u/grokjc, alongside Gadi Evron, Daniel Cuthbert, Thomas Dullien and Michael Bargury. Gadi Evron is co-founder and CEO of Knostic, along with CFP chair of [un]prompted itself. He founded Cymmetria, a cyber deception company later acquired, and he founded Israel's CERT before serving as CISO of the Israeli National Digital Authority. He wrote the post-mortem on the 2007 cyberattacks on Estonia, published early work on DNS amplification attacks, then co-authored APT reports on Rocket Kitten along with Patchwork. He owns the RAPTOR repository as its first-listed author, and his Knostic co-founder is Sounil Yu, who gives his own [un]prompted talk on Day 3. To go deeper on the topic, start here: - AUDIO: Gadi Evron on SANS Cyber Leaders episode 29, Weathering the AI Vulnerability Storm - ExploitGym: can AI agents turn security vulnerabilities into real attacks? - RAPTOR on GitHub - RAPTOR documentation: how the harness is built - Qualys on CVE-2021-3156, the Baron Samedit heap overflow in sudo - libwebp CVE-2023-4863 and CVE-2023-5129 under active exploitation Defenders weighing how fast automated vulnerability research is advancing get an honest account of where it still fails rather than a highlight reel. If a small cheap model can be steered toward understanding your code, does your threat model still assume frontier-scale compute?
功能也跑全了：JSON/Markdown/SARIF 输出、baseline 抑制误报、--recursive、CI 的 fail-on-findings、MCP registry 对比，还能当 MCP server。远程 Git URL 被它的 SSRF 防护拦，改本地 clone。推荐，但 LLM 要选型、分数需复核。 NVIDIA/SkillSpector 推广帖 x.com @CDGalpha
Sandboxing an Agent That Runs Its Own Exploits: Daniel Cuthbert at [un]prompted 2026 RAPTOR points an LLM at an unvetted repository, lets it write proof-of-concept exploits, then runs them on a machine holding live API keys. Nobody on the team was willing to run that outside a sandbox. They built an agent-specific one in April 2026, upgraded it considerably since, then introduced a fair number of their own security bugs along the way. Everyone describes sandbox construction as easy. Four months of work produced a hardened sandbox along with a list of lessons the speaker calls embarrassing. In his [un]prompted 2026 talk, "Your Sandbox Degrades Gracefully, What's the Wurst That Could Happen?," on Tuesday, October 27 at 1:55 p.m. on Stage 1, Daniel Cuthbert walks through what breaking their own containment taught them. Speaker: Daniel Cuthbert, Global Head of Cyber Security Research, Santander RAPTOR is public. The repository credits five authors: Cuthbert alongside Gadi Evron, Thomas Dullien, Michael Bargury and John Cartwright, with a README stating that subprocesses handling untrusted content run under Linux namespaces. The documentation covers commands for scanning, agentic analysis, binary work, exploitation, patching, fuzzing and software composition analysis, running through a staged validation pipeline. Dark Reading's coverage describes Thomas Dullien using the patching stage against recently disclosed FFmpeg vulnerabilities, with Cuthbert quoted saying the framework aids multi-candidate proofs of concept rather than replacing expert reverse engineers. Daniel Cuthbert is Global Head of Cyber Security Research for the Santander Group, along with being an Associate Fellow in Cyber and Tech at the Royal United Services Institute. He co-wrote the original OWASP Testing Guide in 2003, and he co-authors the OWASP Application Security Verification Standard. OWASP named him a Distinguished Lifetime Member in 2024, its highest honor. He sits on the Black Hat Review Board, where he has served as Defense Track Lead for Briefings. He is also an industry chair for the UK Government's Cyber Security Advisory Board. He sits on the Department for Science, Innovation and Technology's Cyber Resilience Expert Advisory Group as well. His conference record includes the BruCON 0x0A keynote in 2018, four Black Hat Locknotes between 2016 and 2021, then "Manna from Heaven" on wireless rogue AP attacks at Black Hat Asia 2015. He publishes RAPTOR's development container on Docker Hub, bundling CodeQL, Semgrep, AFL++, GDB, the rr debugger, Playwright and Claude Code. To go deeper on the topic, start here: - RAPTOR on GitHub - RAPTOR documentation - New RAPTOR framework uses agentic AI to create patches (Dark Reading) - Sandlock: confining AI agent code with unprivileged Linux primitives - Claude Code sandboxing documentation - OWASP Application Security Verification Standard Anyone running an agent that executes code it wrote itself gets a list of failure modes from a team that hit them. What is stopping the coding agent on your machine from reaching the credentials in your environment variables?