The ghost in my tests was a carriage return
This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 4 - Ghost in the Machine.
What I Built
On Windows, a reStructuredText file came back from a loader as one single section. On Linux, the same file split at every heading.
The file used \n line endings. The loader split on PHP_EOL, which is \r\n on Windows. It was looking for a separator that wasn't there.
That is my kind of ghost: something you cannot see in the editor, but that changes what the program does.
An AI coding agent helped produce five fixes for these kinds of bugs in symfony/ai from my Windows machine. Four involved PHP_EOL being used to parse text from somewhere else. The fifth involved a probe that recognized php but not php.exe.
Those bugs inspired poltergeist, a local tool with two parts.
The scanner finds hidden characters. Carriage returns, byte order marks, zero-width spaces, non-breaking spaces, bidirectional overrides, mixed-script words and Unicode tag characters. Some tag characters render invisibly while encoding text, which makes them particularly interesting in files an agent reads.
poltergeist scan exposes those characters. poltergeist exorcise previews changes as a dry-run diff, making the hidden characters visible before anything is removed.
The model investigates platform assumptions. A hard-coded executable name, an implicit encoding or a filename with the wrong capitalization can look perfectly ordinary. The model reads the code and the reported symptom, suggests a category and proposes a small fix.
The scanner works without AI. The diagnostic part uses small, local open-weight models.
Demo
Open the demo page.
The page shows captured command output and benchmark results. It does not run inference in your browser.
From a checkout of the repository, you can generate examples and inspect them:
python scripts/make_examples.py haunted/
python -m poltergeist scan haunted/
python -m poltergeist exorcise haunted/ --eol lf
The last command previews the changes.
Code
GitHub: Yupsecous/poltergeist
The Python tool uses the standard library. Local model inference runs separately through llama-server from llama.cpp.
The project includes 68 tests, including a check for unexpected hidden characters in its own repository.
How I Built It
The model comparison used:
- Gemma 3 1B
- Gemma 3 4B
- Qwen2.5 3B
All three ran as 4-bit GGUF models on a CPU-only desktop with an i7-8700 and 64 GB of RAM.
Each model received code and a symptom. It had to choose one of five categories:
1. Line endings
2. Invisible characters
3. Paths and the file system
4. Executable naming
5. Encoding and locale
It also had to explain the problem and suggest a fix.
There were two prompt versions. The plain prompt contained the code and symptom. The grounded prompt also exposed hidden characters, included scanner findings and supplied up to three entries from a short, hand-written collection of platform facts.
The inference client accepts only loopback addresses and ignores proxy settings. That behavior has a test. I did not disconnect the computer from the network, so I am not claiming a physically isolated setup.
The benchmark that looked too good
The first benchmark contained five cases based on the Symfony fixes and ten synthetic cases.
The platform-facts lookup classified all fifteen correctly without using a model.
That looked impressive until you considered how it was built: the facts had been written with those cases already available. It was measuring how well the lookup matched examples that had shaped it.
The lookup was then frozen in a commit. Ten additional cases were written afterward, using other languages and different wording.
On those ten new cases, the lookup scored 3 out of 10.
That became the more useful comparison.
| Approach | Correct category on 10 new cases | Fix passed keyword rubric | Median seconds per answer |
| --- | --- | --- | --- |
| Platform-facts lookup, no model | 30% | Not scored | Not measured |
| Gemma 3 1B, plain / grounded | 30% / 30% | 10% / 30% | 6.1 / 6.7 |
| Qwen2.5 3B, plain / grounded | 80% / 90% | 30% / 50% | 11.3 / 10.2 |
| Gemma 3 4B, plain / grounded | 40% / 40% | 30% / 30% | 17.5 / 20.1 |
One case is ten percentage points here. These are small exploratory results, not a general ranking of the models.
The proposed fixes were checked with a keyword rubric. They were not executed to prove that they fixed the bug.
What the results actually showed
The largest model did not perform best on these cases.
Qwen2.5 3B classified eight of the ten new cases correctly with the plain prompt and nine with grounding. Gemma 3 4B classified four correctly with either prompt.
That is an observation about this benchmark, not evidence that one model is better at software debugging in general.
Grounding helped some proposed fixes without consistently improving classification.
Gemma 3 1B went from one to three fixes passing the keyword rubric. Qwen went from three to five. Gemma 3 4B stayed at three.
On the original fifteen cases, grounded Qwen and Gemma 4B both reached 100% classification accuracy. But the lookup already scored 100% there. Those results do not establish that the models independently diagnosed the bugs.
A correct category could hide a wrong explanation.
One new case involved two filenames differing only in letter case. The smallest model chose the correct category, but explained it as a path-separator problem.
The other two models blamed line endings and incorrectly claimed that macOS uses CRLF.
The category score did not capture that failure. This is why the raw answers matter alongside the table.
The cases are small and come from one project. The benchmark is a smoke test that helps expose limitations, not proof that these models can safely repair arbitrary code.
The ghosts inside the project itself
The tool encountered the same problems it was supposed to diagnose.
Hidden characters in its own source.
The first self-scan reported 32 findings. Some intended textual escape sequences had been written as actual invisible characters. The code still ran.
Those characters were replaced with visible escapes, and a repository check was added.
An encoding failure in the test tooling.
A mutation-test script crashed when Python on Windows decoded a child process's output as cp1252. The tool checking the tool had its own platform assumption.
A benchmark process-management bug.
PowerShell variable names are case-insensitive. A $server variable overwrote a -Server parameter, disrupting cleanup and leaving a model server running.
That could have caused a later model's requests to reach the earlier server. The script now refuses to start if the port is already occupied.
A scanner that quietly skipped a file.
Windows PowerShell 5.1 saved a results table as UTF-16 with a byte order mark. The repository check caught it, but the scanner had silently skipped the file because it could not decode it.
The scanner now reports text files it could not read.
That last change matters to me: a quiet scan should not be confused with a complete scan.
Why Does Open Innovation Matter?
The material needed to diagnose these bugs can be sensitive: source code, configuration files and internal logs.
Running an open-weight model locally lets the diagnostic workflow use that material without requiring a hosted inference service. The client code enforcing the loopback restriction is available to inspect.
Open models also made comparison practical. The same cases could be run through three different models, with the answers retained for inspection. There was no need to assume the biggest model would be the best fit.
The runs used temperature zero, a fixed seed and a JSON output schema. The 1B model was run twice across all 25 cases and both prompts. All 50 answers matched, including their explanations, fixes and token counts. Timings differed.
Only the smallest model received that repeatability check. It does not establish reproducibility across different machines or llama.cpp builds.
The useful outcome was being able to inspect where the approach worked and where it failed.
My Agent Session
An AI coding agent, Claude Code, wrote most of the code and tests for poltergeist. I ran the benchmark in my own terminal on Windows. The agent also produced the five Symfony AI fixes that inspired the project.
Agent assistance did not prevent mistakes. The hidden characters, PowerShell variable collision and skipped-file behavior all appeared during development.
The project takes untrusted text as input. Its scanner can expose some suspicious characters before an agent reads them, but it is not a complete prompt-injection defense. An instruction does not need invisible characters to be malicious.
The Spirit of It
The carriage return comes from a physical machine moving a print head back across a page. Decades later, that old behavior can still break a modern loader because two systems disagree about what ends a line.
That connection to the past became the project's Halloween theme: invisible remnants and assumptions haunting otherwise ordinary files.
The scanner makes some of them visible. The local model suggests where else to look.
Sometimes it points at the wrong wall. Keeping those failures in the write-up is part of showing what the tool actually does.
Prize Categories
Best Use of Gemma
The project runs Gemma 3 1B and 4B locally through llama.cpp and compares their diagnostic outputs with another open-weight model and a non-model lookup baseline. The results include both the improvements and the cases where Gemma struggled.
2 reactions
GuardedChatBundle: guardrails for public AI chat endpoints on top of Symfony AI
Disclosure: I wrote this.
If you put a Symfony AI agent behind a public, anonymous chat endpoint, the model call is the easy part. The work is everything around it: bots draining your token budget, session handling, retention, and a way to switch the chat off when something goes wrong. I extracted what I built for my own site into a bundle, stejac/guarded-chat-bundle.
What it does:
- a prefilter for obvious prompt-injection phrases, as a plain input processor, so the model is never called for those
- rate limits per client, a message cap per session and a budget stop
- signed session tokens with a lifetime and an idle timeout
- versioned prompts in the database, editable without a deploy
- a two-stage kill switch and scheduled retention for old conversations
- a Stimulus chat widget with a notice that an AI is answering
It complements Symfony AI instead of replacing it. When a guardrail PR for core was closed, the maintainers suggested a standalone package on top of symfony/ai-agent, so this is one take on that. I wrote up a few observations from building it in symfony/ai#1524: ai#1524
One caveat: the phrase filter is no security boundary, it only catches the obvious attempts. Rate limits and budget caps matter more on a public endpoint.
PHP 8.3+, Symfony 7.4 or 8.0, MIT. The Flex recipe is still pending in recipes-contrib, so for now the config files are created by hand (see README).
GitHub: Stefan39/GuardedChatBundle
Feedback welcome, especially on what you would expect from a guardrail layer like this.
2 points
Mentions
The ghost in my tests was a carriage return This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 4 - Ghost in the Machine. What I Built On Windows, a reStructuredText file came back from a loader as one single section. On Linux, the same file split at every heading. The file used \n line endings. The loader split on PHP_EOL, which is \r\n on Windows. It was looking for a separator that wasn't there. That is my kind of ghost: something you cannot see in the editor, but that changes what the program does. An AI coding agent helped produce five fixes for these kinds of bugs in symfony/ai from my Windows machine. Four involved PHP_EOL being used to parse text from somewhere else. The fifth involved a probe that recognized php but not php.exe. Those bugs inspired poltergeist, a local tool with two parts. The scanner finds hidden characters. Carriage returns, byte order marks, zero-width spaces, non-breaking spaces, bidirectional overrides, mixed-script words and Unicode tag characters. Some tag characters render invisibly while encoding text, which makes them particularly interesting in files an agent reads. poltergeist scan exposes those characters. poltergeist exorcise previews changes as a dry-run diff, making the hidden characters visible before anything is removed. The model investigates platform assumptions. A hard-coded executable name, an implicit encoding or a filename with the wrong capitalization can look perfectly ordinary. The model reads the code and the reported symptom, suggests a category and proposes a small fix. The scanner works without AI. The diagnostic part uses small, local open-weight models. Demo Open the demo page. The page shows captured command output and benchmark results. It does not run inference in your browser. From a checkout of the repository, you can generate examples and inspect them: python scripts/make_examples.py haunted/ python -m poltergeist scan haunted/ python -m poltergeist exorcise haunted/ --eol lf The last command previews the changes. Code GitHub: Yupsecous/poltergeist The Python tool uses the standard library. Local model inference runs separately through llama-server from llama.cpp. The project includes 68 tests, including a check for unexpected hidden characters in its own repository. How I Built It The model comparison used: - Gemma 3 1B - Gemma 3 4B - Qwen2.5 3B All three ran as 4-bit GGUF models on a CPU-only desktop with an i7-8700 and 64 GB of RAM. Each model received code and a symptom. It had to choose one of five categories: 1. Line endings 2. Invisible characters 3. Paths and the file system 4. Executable naming 5. Encoding and locale It also had to explain the problem and suggest a fix. There were two prompt versions. The plain prompt contained the code and symptom. The grounded prompt also exposed hidden characters, included scanner findings and supplied up to three entries from a short, hand-written collection of platform facts. The inference client accepts only loopback addresses and ignores proxy settings. That behavior has a test. I did not disconnect the computer from the network, so I am not claiming a physically isolated setup. The benchmark that looked too good The first benchmark contained five cases based on the Symfony fixes and ten synthetic cases. The platform-facts lookup classified all fifteen correctly without using a model. That looked impressive until you considered how it was built: the facts had been written with those cases already available. It was measuring how well the lookup matched examples that had shaped it. The lookup was then frozen in a commit. Ten additional cases were written afterward, using other languages and different wording. On those ten new cases, the lookup scored 3 out of 10. That became the more useful comparison. | Approach | Correct category on 10 new cases | Fix passed keyword rubric | Median seconds per answer | | --- | --- | --- | --- | | Platform-facts lookup, no model | 30% | Not scored | Not measured | | Gemma 3 1B, plain / grounded | 30% / 30% | 10% / 30% | 6.1 / 6.7 | | Qwen2.5 3B, plain / grounded | 80% / 90% | 30% / 50% | 11.3 / 10.2 | | Gemma 3 4B, plain / grounded | 40% / 40% | 30% / 30% | 17.5 / 20.1 | One case is ten percentage points here. These are small exploratory results, not a general ranking of the models. The proposed fixes were checked with a keyword rubric. They were not executed to prove that they fixed the bug. What the results actually showed The largest model did not perform best on these cases. Qwen2.5 3B classified eight of the ten new cases correctly with the plain prompt and nine with grounding. Gemma 3 4B classified four correctly with either prompt. That is an observation about this benchmark, not evidence that one model is better at software debugging in general. Grounding helped some proposed fixes without consistently improving classification. Gemma 3 1B went from one to three fixes passing the keyword rubric. Qwen went from three to five. Gemma 3 4B stayed at three. On the original fifteen cases, grounded Qwen and Gemma 4B both reached 100% classification accuracy. But the lookup already scored 100% there. Those results do not establish that the models independently diagnosed the bugs. A correct category could hide a wrong explanation. One new case involved two filenames differing only in letter case. The smallest model chose the correct category, but explained it as a path-separator problem. The other two models blamed line endings and incorrectly claimed that macOS uses CRLF. The category score did not capture that failure. This is why the raw answers matter alongside the table. The cases are small and come from one project. The benchmark is a smoke test that helps expose limitations, not proof that these models can safely repair arbitrary code. The ghosts inside the project itself The tool encountered the same problems it was supposed to diagnose. Hidden characters in its own source. The first self-scan reported 32 findings. Some intended textual escape sequences had been written as actual invisible characters. The code still ran. Those characters were replaced with visible escapes, and a repository check was added. An encoding failure in the test tooling. A mutation-test script crashed when Python on Windows decoded a child process's output as cp1252. The tool checking the tool had its own platform assumption. A benchmark process-management bug. PowerShell variable names are case-insensitive. A $server variable overwrote a -Server parameter, disrupting cleanup and leaving a model server running. That could have caused a later model's requests to reach the earlier server. The script now refuses to start if the port is already occupied. A scanner that quietly skipped a file. Windows PowerShell 5.1 saved a results table as UTF-16 with a byte order mark. The repository check caught it, but the scanner had silently skipped the file because it could not decode it. The scanner now reports text files it could not read. That last change matters to me: a quiet scan should not be confused with a complete scan. Why Does Open Innovation Matter? The material needed to diagnose these bugs can be sensitive: source code, configuration files and internal logs. Running an open-weight model locally lets the diagnostic workflow use that material without requiring a hosted inference service. The client code enforcing the loopback restriction is available to inspect. Open models also made comparison practical. The same cases could be run through three different models, with the answers retained for inspection. There was no need to assume the biggest model would be the best fit. The runs used temperature zero, a fixed seed and a JSON output schema. The 1B model was run twice across all 25 cases and both prompts. All 50 answers matched, including their explanations, fixes and token counts. Timings differed. Only the smallest model received that repeatability check. It does not establish reproducibility across different machines or llama.cpp builds. The useful outcome was being able to inspect where the approach worked and where it failed. My Agent Session An AI coding agent, Claude Code, wrote most of the code and tests for poltergeist. I ran the benchmark in my own terminal on Windows. The agent also produced the five Symfony AI fixes that inspired the project. Agent assistance did not prevent mistakes. The hidden characters, PowerShell variable collision and skipped-file behavior all appeared during development. The project takes untrusted text as input. Its scanner can expose some suspicious characters before an agent reads them, but it is not a complete prompt-injection defense. An instruction does not need invisible characters to be malicious. The Spirit of It The carriage return comes from a physical machine moving a print head back across a page. Decades later, that old behavior can still break a modern loader because two systems disagree about what ends a line. That connection to the past became the project's Halloween theme: invisible remnants and assumptions haunting otherwise ordinary files. The scanner makes some of them visible. The local model suggests where else to look. Sometimes it points at the wrong wall. Keeping those failures in the write-up is part of showing what the tool actually does. Prize Categories Best Use of Gemma The project runs Gemma 3 1B and 4B locally through llama.cpp and compares their diagnostic outputs with another open-weight model and a non-model lookup baseline. The results include both the improvements and the cases where Gemma struggled.
GuardedChatBundle: guardrails for public AI chat endpoints on top of Symfony AI Disclosure: I wrote this. If you put a Symfony AI agent behind a public, anonymous chat endpoint, the model call is the easy part. The work is everything around it: bots draining your token budget, session handling, retention, and a way to switch the chat off when something goes wrong. I extracted what I built for my own site into a bundle, stejac/guarded-chat-bundle. What it does: - a prefilter for obvious prompt-injection phrases, as a plain input processor, so the model is never called for those - rate limits per client, a message cap per session and a budget stop - signed session tokens with a lifetime and an idle timeout - versioned prompts in the database, editable without a deploy - a two-stage kill switch and scheduled retention for old conversations - a Stimulus chat widget with a notice that an AI is answering It complements Symfony AI instead of replacing it. When a guardrail PR for core was closed, the maintainers suggested a standalone package on top of symfony/ai-agent, so this is one take on that. I wrote up a few observations from building it in symfony/ai#1524: ai#1524 One caveat: the phrase filter is no security boundary, it only catches the obvious attempts. Rate limits and budget caps matter more on a public endpoint. PHP 8.3+, Symfony 7.4 or 8.0, MIT. The Flex recipe is still pending in recipes-contrib, so for now the config files are created by hand (see README). GitHub: Stefan39/GuardedChatBundle Feedback welcome, especially on what you would expect from a guardrail layer like this.