ControlArena: the monitored model could shape its monitor’s prompt
In AI control experiments, a trusted monitor reads what an untrusted model did and scores how suspicious it looks. That score means something only if the untrusted model cannot change the structure of the prompt the monitor reads. In March 2026 I found that it could in ControlArena, the UK AI Security Institute's library for AI control research. Two monitor prompt formatters, including the default one, put the monitored model's tool-call arguments into the prompt as raw text, with nothing marking where that text ended. I reported it in ControlArena issue #791 and sent a patch in PR #795. The maintainer, @Rogan-Inglis, merged it with formatting fixes as PR #798. My follow-up for the chain-of-thought formatter, PR #810, was merged three weeks later.
What the monitor read
The last_tool_calls formatter, the default for basic_monitor, and the prefix formatter built each tool call with an f-string:
tool_calls_message += f"{tool_call.function}: {tool_call.arguments}\n"
The arguments went in as a Python dictionary repr, and the result was substituted into the monitor's prompt template. The monitored model chooses every character of those arguments. A model trying to get past its monitor could put text in an argument that reads like part of the prompt: a closing tag, a new heading or a note addressed to the monitor. Nothing in the prompt showed the monitor where the model's text stopped and the experimenter's instructions resumed.
The library already had a safer pattern. Its XML trajectory formatter passed every model-derived field through html.escape(), so escaped content could not close its element. The two tool-call formatters did not.
The accepted fixes
#798 serializes tool-call arguments with json.dumps() instead of Python repr, and HTML-escapes them inside the prefix formatter's tags so an argument cannot forge a closing tag. It follows the convention the full-trajectory formatter already used. Its tests cover closing-tag injection, HTML in function names and fake monitor instructions.
#810 closes the same gap in the chain-of-thought formatter, which put the monitored model's reasoning into the prompt with no boundary. The reasoning now sits inside <cot>...</cot> tags with its content HTML-escaped, and new tests cover closing-tag and fake-tag injection.
Both changes are marked as breaking. The monitor now sees different text, so the maintainers note that experiment results from before and after each change are not directly comparable. That note is a finding in itself: how the evidence is laid out in a monitor's prompt is part of the measurement.
The same failure in AI evaluation
This is ETB-02 in the Evaluator Trust Boundary taxonomy: an evaluator's prompt treats text written by the evaluated model as part of its own structure. LLM judges have the same shape, since a judge reads text written by the system it grades. The ETB-02 page lists every public report in the class, including the same problem in the judge prompt of UK AISI's Inspect.
Source record
- ControlArena issue #791: report.
- ControlArena PR #795: my patch.
- ControlArena PR #798: accepted fix, by @Rogan-Inglis, based on #795.
- ControlArena issue #809 and PR #810: the chain-of-thought follow-up.
- ETB-02: unsanitized model output in the evaluator prompt.
- My AI evaluation contribution record.
This note describes the reports and the accepted changes. Escaping stops the monitored model from forging the prompt's structure. It does not make a monitor immune to persuasive text inside that structure, which is a question about the monitor model itself.
I shut down my agent-safety toolkit to audit AI graders
Earlier this year I posted here about SafeClaw and Authensor's policy engine for AI agents. That product line is retired, and those posts now say so at the top. This is what Authensor works on instead.
Authensor is an AI evaluation integrity firm in San Francisco. It audits the tools the AI industry measures itself with: evaluation harnesses, benchmarks, LLM judges and scoring pipelines. The question for each one is whether the system being evaluated can move its own score.
Why the switch
Two of the early bugs I fixed upstream were in agent-control tooling. In April 2026, UK AISI's ControlArena passed tool-call arguments and chain-of-thought text into its monitor prompts unsanitized, so the agent being monitored could write straight into the prompt of the model watching it. The same pattern kept turning up in graders, judges and benchmarks. If the measurement can be steered by the thing being measured, the safety numbers built on it don't mean much, so that became the work.
The Evaluator Trust Boundary
The research is organized around one definition: an evaluator has an Evaluator Trust Boundary (ETB) failure when its score depends on an artifact the evaluated system can influence. The taxonomy names ten mechanisms, ETB-01 to ETB-10. The preprint is on Zenodo: 10.5281/zenodo.21633620.
Some of the cases now fixed upstream:
- SWE-bench (August 2026, ETB-09): the grader counted PASSED lines that the patch under test could print itself. Fixed in PR #620.
- Promptfoo (October 2026, ETB-03): an llm-rubric judge response with no verdict defaulted to a pass. Fixed in PR #10611.
- UK AISI Inspect AI (June 2026, ETB-04): scoring metrics silently dropped errored and inconclusive samples from the denominator. Fixed in #4288.
- NVIDIA garak (July 2026, ETB-06): string detectors did no Unicode normalization, so homoglyph and zero-width variants got through. Fixed in #1884.
- Instructor (July 2026, ETB-01): JSON extraction took the first object in a code block, so planted JSON outranked the real output. Fixed in #2434.
- UK AISI ControlArena (April 2026, ETB-02): tool-call arguments entered monitor prompts unsanitized. Fixed in #798.
The full list, with every report and fix, is at authensor.com. Separately, the MLflow advisory for CVE-2026-64849 credits Authensor as an independent finder.
Check your own judge
etb-scan is a free, MIT-licensed scanner that measures ETB-01 (injected verdicts) against your own judge. It runs offline on the standard library, with no API keys.
pip install etb-scan
etbscan --judge mypkg:judge --trials 10
If you've published an eval number and want it checked against the instrument that produced it, Authensor does that free within 48 hours: john@authensor.com.
The hardware side
15 Research Lab is my separate lab in San Francisco for embodied AI safety and experimental hardware. The same kind of bug shows up there. Yosys's formal exporters were silently dropping every assertion, so solvers reported PASS on designs that should fail. I reported it as #6108 and Zaneham fixed it in #6117. Write-up.
Links
- ETB definition, cases and citation: authensor.com
- Every report and fix: authensor.com
- etb-scan: AUTHENSOR/etb-scan
- 15 Research Lab: 15researchlab.com
- Me: johndanielkearney.com
Mentions
ControlArena: the monitored model could shape its monitor’s prompt In AI control experiments, a trusted monitor reads what an untrusted model did and scores how suspicious it looks. That score means something only if the untrusted model cannot change the structure of the prompt the monitor reads. In March 2026 I found that it could in ControlArena, the UK AI Security Institute's library for AI control research. Two monitor prompt formatters, including the default one, put the monitored model's tool-call arguments into the prompt as raw text, with nothing marking where that text ended. I reported it in ControlArena issue #791 and sent a patch in PR #795. The maintainer, @Rogan-Inglis, merged it with formatting fixes as PR #798. My follow-up for the chain-of-thought formatter, PR #810, was merged three weeks later. What the monitor read The last_tool_calls formatter, the default for basic_monitor, and the prefix formatter built each tool call with an f-string: tool_calls_message += f"{tool_call.function}: {tool_call.arguments}\n" The arguments went in as a Python dictionary repr, and the result was substituted into the monitor's prompt template. The monitored model chooses every character of those arguments. A model trying to get past its monitor could put text in an argument that reads like part of the prompt: a closing tag, a new heading or a note addressed to the monitor. Nothing in the prompt showed the monitor where the model's text stopped and the experimenter's instructions resumed. The library already had a safer pattern. Its XML trajectory formatter passed every model-derived field through html.escape(), so escaped content could not close its element. The two tool-call formatters did not. The accepted fixes #798 serializes tool-call arguments with json.dumps() instead of Python repr, and HTML-escapes them inside the prefix formatter's tags so an argument cannot forge a closing tag. It follows the convention the full-trajectory formatter already used. Its tests cover closing-tag injection, HTML in function names and fake monitor instructions. #810 closes the same gap in the chain-of-thought formatter, which put the monitored model's reasoning into the prompt with no boundary. The reasoning now sits inside <cot>...</cot> tags with its content HTML-escaped, and new tests cover closing-tag and fake-tag injection. Both changes are marked as breaking. The monitor now sees different text, so the maintainers note that experiment results from before and after each change are not directly comparable. That note is a finding in itself: how the evidence is laid out in a monitor's prompt is part of the measurement. The same failure in AI evaluation This is ETB-02 in the Evaluator Trust Boundary taxonomy: an evaluator's prompt treats text written by the evaluated model as part of its own structure. LLM judges have the same shape, since a judge reads text written by the system it grades. The ETB-02 page lists every public report in the class, including the same problem in the judge prompt of UK AISI's Inspect. Source record - ControlArena issue #791: report. - ControlArena PR #795: my patch. - ControlArena PR #798: accepted fix, by @Rogan-Inglis, based on #795. - ControlArena issue #809 and PR #810: the chain-of-thought follow-up. - ETB-02: unsanitized model output in the evaluator prompt. - My AI evaluation contribution record. This note describes the reports and the accepted changes. Escaping stops the monitored model from forging the prompt's structure. It does not make a monitor immune to persuasive text inside that structure, which is a question about the monitor model itself.
I shut down my agent-safety toolkit to audit AI graders Earlier this year I posted here about SafeClaw and Authensor's policy engine for AI agents. That product line is retired, and those posts now say so at the top. This is what Authensor works on instead. Authensor is an AI evaluation integrity firm in San Francisco. It audits the tools the AI industry measures itself with: evaluation harnesses, benchmarks, LLM judges and scoring pipelines. The question for each one is whether the system being evaluated can move its own score. Why the switch Two of the early bugs I fixed upstream were in agent-control tooling. In April 2026, UK AISI's ControlArena passed tool-call arguments and chain-of-thought text into its monitor prompts unsanitized, so the agent being monitored could write straight into the prompt of the model watching it. The same pattern kept turning up in graders, judges and benchmarks. If the measurement can be steered by the thing being measured, the safety numbers built on it don't mean much, so that became the work. The Evaluator Trust Boundary The research is organized around one definition: an evaluator has an Evaluator Trust Boundary (ETB) failure when its score depends on an artifact the evaluated system can influence. The taxonomy names ten mechanisms, ETB-01 to ETB-10. The preprint is on Zenodo: 10.5281/zenodo.21633620. Some of the cases now fixed upstream: - SWE-bench (August 2026, ETB-09): the grader counted PASSED lines that the patch under test could print itself. Fixed in PR #620. - Promptfoo (October 2026, ETB-03): an llm-rubric judge response with no verdict defaulted to a pass. Fixed in PR #10611. - UK AISI Inspect AI (June 2026, ETB-04): scoring metrics silently dropped errored and inconclusive samples from the denominator. Fixed in #4288. - NVIDIA garak (July 2026, ETB-06): string detectors did no Unicode normalization, so homoglyph and zero-width variants got through. Fixed in #1884. - Instructor (July 2026, ETB-01): JSON extraction took the first object in a code block, so planted JSON outranked the real output. Fixed in #2434. - UK AISI ControlArena (April 2026, ETB-02): tool-call arguments entered monitor prompts unsanitized. Fixed in #798. The full list, with every report and fix, is at authensor.com. Separately, the MLflow advisory for CVE-2026-64849 credits Authensor as an independent finder. Check your own judge etb-scan is a free, MIT-licensed scanner that measures ETB-01 (injected verdicts) against your own judge. It runs offline on the standard library, with no API keys. pip install etb-scan etbscan --judge mypkg:judge --trials 10 If you've published an eval number and want it checked against the instrument that produced it, Authensor does that free within 48 hours: john@authensor.com. The hardware side 15 Research Lab is my separate lab in San Francisco for embodied AI safety and experimental hardware. The same kind of bug shows up there. Yosys's formal exporters were silently dropping every assertion, so solvers reported PASS on designs that should fail. I reported it as #6108 and Zaneham fixed it in #6117. Write-up. Links - ETB definition, cases and citation: authensor.com - Every report and fix: authensor.com - etb-scan: AUTHENSOR/etb-scan - 15 Research Lab: 15researchlab.com - Me: johndanielkearney.com