Dark Matter Engine: A Sloth That Hunts for Missing Answers at 2 AM
What I Built
Every spec I have ever read answers the questions its author thought of. The expensive bugs live in the ones they didn't.
Here is a real sentence from a small checkout spec I wrote for this project:
> If the gateway responds with a 500 error, the service retries the charge up to 3 times with exponential backoff.
Reasonable. Probably even tested. Now picture week three of a sprint, when someone asks: what if the gateway charges the card, and the response never arrives? The retry fires. Does the customer pay twice? Nothing in the docs says, because nobody asked.
That missing answer is what I went hunting for. I call it dark matter: you can't see it directly, but you can find the hole by looking at everything around it. The Dark Matter Engine is an offline scan that reads your docs, asks "what could go wrong here?", and reports the questions your documentation never settled. It runs at 2 AM. A local Gemma does the reading. Plain Python checks its work.
TL;DR. The Dark Matter Engine is an offline scan that reads your docs and lists the failure questions they never answer. A local Gemma 3 (4B) model proposes and judges scenarios, and plain Python checks every quote it relies on. A Task Scheduler job runs it at 2 AM. On my nine planted test pairs the default check flagged 6 of 9 omissions and wrongly flagged 2 of 45 documented behaviors; a stricter mode flagged 9 of 9 but wrongly flagged 23. On a real design document it found one genuine unanswered question and produced eight findings I had to throw out. It's a prototype that produces candidates for a human to review, not verdicts. MIT licensed, Python standard library only.
Who gets to be lazier: anyone who reviews specs or design docs and would rather not brainstorm every failure case by hand. Point it at a folder of Markdown or text. For each behavior the docs describe, it proposes a failure scenario, searches the docs for an answer, and reports whether the text actually establishes one.
My demo is two versions of the checkout spec. Version 1 documents retries on a gateway error and says nothing about a lost response. Version 2 adds one section on idempotency keys. For the scenario "the gateway charges the card but the response is lost and the client retries":
- Version 1: Partially answered. Gemma cites only the sentence about the normal payment call and asks the right question: how does the system ensure the order is charged exactly once when retries are involved? The spec never says, and the engine flags it as unresolved.
- Version 2: Answered, with the exact sentence and its line number: the service returns the stored result and never charges the order a second time.
The verdicts are generated from the specification text, not hard-coded for these two versions.
A warning before the screenshots: this is a prototype that produces candidates for a human to review, not verdicts. On a real design document from another project it found one genuine unanswered question and also produced eight findings I had to throw out. The details are below, because they're the most interesting part.
!Dark Matter Engine report for version 1 of the checkout spec: the lost-response scenario is marked Partially answered, with the question to resolve and the cited sentence from spec.md line 4.
!Report for version 2: the same scenario is marked Answered, citing the idempotency-key sentence at spec.md line 16, which says the service never charges the order a second time.
Demo
{% embed youtube.com %}
!Results table from the offline demo run: the lost-response scenario changes from Partially answered in version 1 to Answered in version 2, with benchmark rows below.
To reproduce it yourself:
ollama pull gemma3:4b
python dark_matter.py demo --quick
That scans the checkout pair and runs the benchmark on it (about 5 minutes on my laptop), then writes one results page you can open in a browser. It's quicker than the 8.5-minute scan I mention below because the demo uses pre-written scenarios, while a plain scan first has the model propose scenarios from every section. I also left the full result files from my runs in the repo.
Code
{% embed Emmimal/dark-matter-engine %}
MIT licensed. Python standard library only; Ollama is the one external runtime. The logs, result files and the offline recording are in the repo's evidence/ folder. Model: gemma3:4b, digest 68ceab7e468a.
How I Built It
I used Google's open-weight Gemma 3 (4B), run locally through Ollama, with temperature 0 and a fixed seed. Everything else is standard-library Python.
The steady part: how it avoids inventing gaps
Ask a language model "what's missing?" and it will cheerfully make things up. So the work is split:
1. Gemma proposes scenarios from sections the docs actually contain.
2. Plain lexical search fetches evidence before the model judges anything.
3. Gemma classifies: Answered, Partially answered or Not established. (A Contradicted status exists in the code, but my corpus has no contradiction case, so I haven't tested it.)
4. Python checks that every quote exists in the retrieved text. A quote it can't find makes the finding Unverified, never a gap.
5. An Answered claim gets a second check. A Not established result gets a second search with a hand-written synonym list, then is classified again.
Gemma decides what the text establishes. Python decides whether Gemma is allowed to say so.
The slow part: does it work?
I wrote nine small spec pairs, each with one planted omission. That's 54 scenarios: 9 planted gaps and 45 documented behaviors. The expected answers live in a separate file the scanner never reads. The first two pairs were used to tune the prompts. Three more were run once after tuning, with expected answers written beforehand but not committed before the run. For the last four, I committed the expected answers before running any model on them (commit d11c484). I wrote those four after the worker-crash failure below, so they deliberately include similar traps. I changed no prompt or engine logic after seeing results on them; I only added per-scenario reporting.
| Method | Planted gaps flagged | Documented behaviors wrongly flagged |
|---|---|---|
| Gemma, default check | 6 of 9 | 2 of 45 |
| Gemma, stricter two-step check | 9 of 9 | 23 of 45 |
| No-model word-coverage baseline (threshold 0.5) | 9 of 9 | 32 of 45 |
| Same baseline (threshold 0.4) | 6 of 9 | 19 of 45 |
"Flagged" means Not established or Partially answered. The baseline is a simple rule: how many of the scenario's words appear in the top three retrieved passages. I'm showing its whole sweep, not one hand-picked setting: thresholds 0.6, 0.7 and 0.8 also catch 9 of 9, but flag 37, 42 and 45 of the 45 documented behaviors.
At the same 6 of 9, the baseline wrongly flags 19 of 45 and the default check flags 2. These are two different methods, so this compares outcomes on my test set. It doesn't show that Gemma is better in general, and it says nothing about stronger retrieval than my simple one. What it does show is what the model contributes here: far fewer false alarms.
These are single runs on a small corpus that I partly built while tuning, so read the counts, not percentages. The last four pairs are the stronger evidence, but even they are not an independent evaluation. Two runs on the same machine gave identical results, which shows determinism, not robustness.
What a run costs, on one Windows laptop (I didn't record the hardware):
| Run | Time |
|---|---|
| One real scan, one spec, the model writes its own scenarios | about 8.5 minutes |
| demo --quick, checkout pair, end to end | about 5 minutes |
| 24 benchmark scenarios, default check | about 20 minutes |
| The same 24 scenarios, stricter two-step check | about 35 minutes |
| API fees | $0 |
Where it breaks
My favorite failure. For "a worker process crashes while running a job", the default check let Gemma quote the sentence about retrying a job that raises an error, and call the scenario Answered. The quote was real. The inference wasn't. Retrying after an error says nothing about what happens when the whole process dies.
My citation check can't catch that, because the sentence exists. A yes/no relevance check I added didn't catch it either. The stricter two-step check does, along with two other missed gaps, but it also flags about half of the documented behaviors. So I left the default alone instead of crowning whichever mode looked best on my own data.
What I learned: when a small model checks its own work, how strict you make the check is a product decision. It's a dial between missing gaps and crying wolf, and no setting of it is free.
A real spec I didn't write
To see whether that failure was just a quirk of my own test pairs, I ran the default settings once, with nothing tuned, on a design document from another project: ADR-22 from NATS, "JetStream Publish Retries on No Responders" (Apache-2.0; I did not copy it into my repo). The engine reported 12 scenarios and 0 potential gaps. (Its report counts only "Not established" as a potential gap; the one "Partially answered" finding sat in a separate section.) Then I read every finding against the document. The labels are one person's judgment, and they're in the repo.
- 3 were correct and properly supported.
- 1 was a genuine unanswered question: the ADR says what a publish returns after a timeout or after a last "no responders" error, but not what happens if the last attempt fails with some other error. The engine labeled it Partially answered, not a gap.
- 5 were marked Answered, but the quote they cited didn't support the claim.
- 3 were scenarios that contradict the spec or aren't about it. One said the client "does not attempt to send the message again", the opposite of the ADR's whole point. Another was about a Go timer failing to start.
So one real finding and three correct confirmations, buried among eight that don't hold up. It's the same "real quote, unsupported inference" failure, on a document nobody tuned for, plus the scenario generator inventing hazards. This is one document labeled by one person, not a benchmark.
Other caveats
- My first run on the first two pairs went badly: Gemma labeled one planted omission "Partially answered" and called the other "Answered". Strictly, it found neither. I then added the relevance check, partial-answer reporting and a second metric ("unresolved", meaning Not established or Partially answered), using only those two pairs. That metric was defined after the first run.
- On a plain scan, where the model writes its own scenarios, it invents requirements the spec never states. In my scheduled run, one flagged "gap" was a false alarm. Treat generated scenarios as prompts for review, not findings.
- One export scenario in my own corpus is arguably ambiguous and got flagged in both versions. I left it as written.
- Markdown and text only. No comparison with closed models.
Bugs I hit along the way
- Gemma copied my prompt's placeholder. In an early real run, the report's "question to resolve" was literally the example text from my own prompt. The prompt no longer contains placeholder text, and any question that shares no words with its scenario is replaced by a plain fallback.
- My relevance check was too literal. It rejected the right answer to the crash scenario in version 2 because the spec says "lease expires", not "crash". I loosened it, and the looser check then accepted the wrong quote in version 1. That is how I ended up with the dial described above.
- Windows Task Scheduler caps the command at 261 characters. My first scheduled command was far longer, so the engine now writes a short wrapper script and schedules that. Task Scheduler also refuses to start on battery by default, so I had to change that setting for a 2 AM run to work on a laptop.
Why Does Open Innovation Matter?
- The docs stay on my machine. Specs for systems that don't exist yet are exactly what I wouldn't paste into a hosted API. The engine refuses any model host that isn't localhost. It also patches Python's socket calls so a non-loopback connection or DNS lookup inside its own process raises an error, and a test checks that. That covers the Python process only, not what Ollama does.
- It works offline. I turned Wi-Fi off, used a failing ping google.com to show it was off, and ran the demo. It finished with the same result. The ping only shows the network was down; the completed run is the evidence.
- No API fees.
- The weights are pinned. I recorded the model digest, so the benchmark can be reproduced instead of drifting under me. The model tag is a command-line flag, though I only benchmarked gemma3:4b.
I never compared it with a closed model, so I'm not claiming this beats one on accuracy. My argument is privacy, offline use, cost and control.
The Lazy Payoff
I do not run it. After one setup command, Windows Task Scheduler does.
python dark_matter.py schedule ./docs --at 02:00
On Oct 22 the task fired by itself at 2:00 AM and had written the report and a log line two seconds later. That was fast because the docs hadn't changed, so it reused the saved result. When they do change, a real scan takes its time: about 8.5 minutes for one spec on my laptop. Nobody is waiting, so slow is fine. A sloth would approve.
After setup, the number of things I do each day is zero. As for the payoff: I ran it on my own spec corpus and, once, on a real design document. That run surfaced one genuine unanswered question, and also eight findings I had to read and throw out. So I can't honestly tell you how many hours it saves a team.
What happens when the gateway charges the card and the response disappears? Version 1 of my spec had no answer. Version 2 did, after I added the idempotency behavior.
That is the gap Dark Matter Engine is built to surface. It can't promise to find every missing answer, and it still produces false alarms. But it can read documentation overnight and leave me questions worth reviewing.
The sloth moves slowly. At 2 AM, nobody has to remind it to get started.
Prize Categories
Best Use of Gemma: a local Gemma decides whether each quote establishes the required behavior. Python validates every citation and every output format. I kept those jobs separate on purpose.
1 reaction
Mentions
AI 软件工厂 1.0：当 Agent 开始写代码、部署和运维，我们需要怎样的架构？ 主页 | Web版 | 订阅 | 归档 | Feed GopherDaily 20261030 每日一谚：Keep interfaces small; 1 to 3 methods is ideal. Go技术生态 AI 软件工厂 1.0：当 Agent 开始写代码、部署和运维，我们需要怎样的架构？ AI 编程正在从个人开发工具演变为企业级软件生产基础设施。Agent Baseline 发布的《The Secure Software Factory》参考架构，将 Agent 的运行环境、网络访问、依赖供应链以及身份授权划分为四道安全边界，并提出三种逐步增强的自主运行模式。本文从架构设计、执行链路、Go 项目落地和工程治理等角度，拆解软件工厂的核心机制，并给出一套可以逐步实施的参考方案。 TinyGo 0.44 - 更小、更快、更好 | TinyGo TinyGo 0.44 版本发布，程序更小更快，支持微控制器上的更多多核，支持 Arduino UNO R4，提供新的驱动程序，以及重大的蓝牙更新！ 云原生技术 KubeCon + CloudNativeCon 北美 2026：终端用户 TAB 建议和技巧 大规模零停机 CNI 迁移：从 Canal 到 Cilium 跨越数百个生产集群。在生产环境中迁移 CNI 插件是 Kubernetes 基础设施中最高风险的操作之一。 智能 AI 中的边界问题是平台问题 这并非反对代理的自主性，而是讨论自主性应归属何处。有用的框架是确定性的主干，非确定性的执行挂靠其上。 在问题出现前捕获错误：JiTTesting 的大规模人机协同方法 大多数基于 AI 的代码防错方法使用 LLM 和/或智能 AI 来审查代码并根据先前经验检测故障。它们通过与人审查者相同的审查流程工作，只是速度更快。 优雅与智能呈负相关 有一个著名的冯·诺依曼故事，他被问及如果一只苍蝇在以固定速度相互接近的两名自行车手之间持续飞行，它需要飞行多远。 编码智能体恐怖故事：不存在的 CVE BuaJi 已移至 Docker 沙箱。只读规则现在是环境的属性，而不是脚本中的约定：代码以只读方式挂载，不传递凭据，并且只有一个网络路径是开放的。 Cloudflare 的 eBPF 平台转换第 5 部分：大规模运行 eBPF 的技巧 本系列记录了 Cloudflare 八年来使用 eBPF 的旅程，从专门的 DDoS 工具到其全球网络的可编程骨干。 是内存延迟，笨蛋 当您看到数据库 CPU 使用率达到 100% 时，它实际上在做什么？为了简单起见，我们喜欢假设 CPU 线性地运行代码，但事实上 CPU 是极其复杂的实体。 Postgres 上每秒 100 万个持久工作流 | DBOS 在 Postgres 上构建持久工作流引擎时，我们最常被问到的问题是 Postgres 能否扩展。现在，已经有大量关于此的博客文章。 如何创建自己的去中心化消息传递协议 假设您想创建一个系统，其中每个服务器都可以加入网络并开始与该网络中的其他成员交换某些内容。您会如何处理这样的任务？ 解释 Shirky 原则：为什么团队保留他们解决的问题 团队为什么保留他们解决的问题？ 懒人开发者观察自己代码的指南 | CNCF 本文重点介绍了 CNCF 项目，包括 OpenTelemetry 等。 在中国最大的 Git 平台上扫描 725 万个公共仓库以查找密钥 在中国最大的 Git 平台上扫描了 725 万个公共仓库以查找密钥。 AI 使用自主的 Fable 级模型构建多人游戏世界 我们的代理直接更改实时代码并每小时发布到生产环境，因此它们自己读取每个错误、错误报告和指标。我们约 60% 的工作始于生产错误、Savi 错误报告或日志中的异常内容。 Mistral Large 4 (Le Chonk) 现在已开源 | Mistral 为程序员构建，受防御者信任。 游戏中的代理 注意：我们正在运行一个新的 Minecraft 殖民地，有 25 个 GPT-6.1 Terra 代理。理解多代理动态是 AI 对齐中最紧迫的挑战之一。 LLM 语音 有时当我阅读一篇文章时，我越来越觉得我正在阅读的是 LLM 的输出。 流行工具与项目 gastownhall/beads Beads - A memory upgrade for your coding agent JuliusBrussee/caveman 🪨 why use many token when few token do trick. Viral skill + proxy for coding agents that cuts 65% of tokens by talking like a caveman. docker/compose Define and run multi-container applications with Docker VictoriaMetrics/VictoriaMetrics VictoriaMetrics: fast, cost-effective monitoring solution and time series database agent-substrate/substrate Agent Substrate: the core system keploy/keploy Open-source platform for creating safe, isolated production sandboxes for API, integration, and E2E testing. google/ax Agent Executor (AX) is Google's open agentic orchestration runtime glanceapp/glance A self-hosted dashboard that puts all your feeds in one place tailscale/tailcat like netcat, but over Tailscale's data plane, without Tailscale's control plane nats-io/nats-server High-Performance server for NATS.io, the cloud and edge native messaging system. 编辑:Tony Bai 编辑主页:tonybai.com GopherDaily项目:github.com/bigwhite/gopherdaily Copyright 2019-2024 GopherDaily gastownhall/beads: gastownhall/beads JuliusBrussee/caveman: JuliusBrussee/caveman docker/compose: docker/compose VictoriaMetrics/VictoriaMetrics: VictoriaMetrics/VictoriaMetrics agent-substrate/substrate: agent-substrate/substrate keploy/keploy: keploy/keploy google/ax: google/ax glanceapp/glance: glanceapp/glance tailscale/tailcat: tailscale/tailcat nats-io/nats-server: nats-io/nats-server github.com/bigwhite/gopherdaily: bigwhite/gopherdaily
Dark Matter Engine: A Sloth That Hunts for Missing Answers at 2 AM What I Built Every spec I have ever read answers the questions its author thought of. The expensive bugs live in the ones they didn't. Here is a real sentence from a small checkout spec I wrote for this project: > If the gateway responds with a 500 error, the service retries the charge up to 3 times with exponential backoff. Reasonable. Probably even tested. Now picture week three of a sprint, when someone asks: what if the gateway charges the card, and the response never arrives? The retry fires. Does the customer pay twice? Nothing in the docs says, because nobody asked. That missing answer is what I went hunting for. I call it dark matter: you can't see it directly, but you can find the hole by looking at everything around it. The Dark Matter Engine is an offline scan that reads your docs, asks "what could go wrong here?", and reports the questions your documentation never settled. It runs at 2 AM. A local Gemma does the reading. Plain Python checks its work. TL;DR. The Dark Matter Engine is an offline scan that reads your docs and lists the failure questions they never answer. A local Gemma 3 (4B) model proposes and judges scenarios, and plain Python checks every quote it relies on. A Task Scheduler job runs it at 2 AM. On my nine planted test pairs the default check flagged 6 of 9 omissions and wrongly flagged 2 of 45 documented behaviors; a stricter mode flagged 9 of 9 but wrongly flagged 23. On a real design document it found one genuine unanswered question and produced eight findings I had to throw out. It's a prototype that produces candidates for a human to review, not verdicts. MIT licensed, Python standard library only. Who gets to be lazier: anyone who reviews specs or design docs and would rather not brainstorm every failure case by hand. Point it at a folder of Markdown or text. For each behavior the docs describe, it proposes a failure scenario, searches the docs for an answer, and reports whether the text actually establishes one. My demo is two versions of the checkout spec. Version 1 documents retries on a gateway error and says nothing about a lost response. Version 2 adds one section on idempotency keys. For the scenario "the gateway charges the card but the response is lost and the client retries": - Version 1: Partially answered. Gemma cites only the sentence about the normal payment call and asks the right question: how does the system ensure the order is charged exactly once when retries are involved? The spec never says, and the engine flags it as unresolved. - Version 2: Answered, with the exact sentence and its line number: the service returns the stored result and never charges the order a second time. The verdicts are generated from the specification text, not hard-coded for these two versions. A warning before the screenshots: this is a prototype that produces candidates for a human to review, not verdicts. On a real design document from another project it found one genuine unanswered question and also produced eight findings I had to throw out. The details are below, because they're the most interesting part. !Dark Matter Engine report for version 1 of the checkout spec: the lost-response scenario is marked Partially answered, with the question to resolve and the cited sentence from spec.md line 4. !Report for version 2: the same scenario is marked Answered, citing the idempotency-key sentence at spec.md line 16, which says the service never charges the order a second time. Demo {% embed youtube.com %} !Results table from the offline demo run: the lost-response scenario changes from Partially answered in version 1 to Answered in version 2, with benchmark rows below. To reproduce it yourself: ollama pull gemma3:4b python dark_matter.py demo --quick That scans the checkout pair and runs the benchmark on it (about 5 minutes on my laptop), then writes one results page you can open in a browser. It's quicker than the 8.5-minute scan I mention below because the demo uses pre-written scenarios, while a plain scan first has the model propose scenarios from every section. I also left the full result files from my runs in the repo. Code {% embed Emmimal/dark-matter-engine %} MIT licensed. Python standard library only; Ollama is the one external runtime. The logs, result files and the offline recording are in the repo's evidence/ folder. Model: gemma3:4b, digest 68ceab7e468a. How I Built It I used Google's open-weight Gemma 3 (4B), run locally through Ollama, with temperature 0 and a fixed seed. Everything else is standard-library Python. The steady part: how it avoids inventing gaps Ask a language model "what's missing?" and it will cheerfully make things up. So the work is split: 1. Gemma proposes scenarios from sections the docs actually contain. 2. Plain lexical search fetches evidence before the model judges anything. 3. Gemma classifies: Answered, Partially answered or Not established. (A Contradicted status exists in the code, but my corpus has no contradiction case, so I haven't tested it.) 4. Python checks that every quote exists in the retrieved text. A quote it can't find makes the finding Unverified, never a gap. 5. An Answered claim gets a second check. A Not established result gets a second search with a hand-written synonym list, then is classified again. Gemma decides what the text establishes. Python decides whether Gemma is allowed to say so. The slow part: does it work? I wrote nine small spec pairs, each with one planted omission. That's 54 scenarios: 9 planted gaps and 45 documented behaviors. The expected answers live in a separate file the scanner never reads. The first two pairs were used to tune the prompts. Three more were run once after tuning, with expected answers written beforehand but not committed before the run. For the last four, I committed the expected answers before running any model on them (commit d11c484). I wrote those four after the worker-crash failure below, so they deliberately include similar traps. I changed no prompt or engine logic after seeing results on them; I only added per-scenario reporting. | Method | Planted gaps flagged | Documented behaviors wrongly flagged | |---|---|---| | Gemma, default check | 6 of 9 | 2 of 45 | | Gemma, stricter two-step check | 9 of 9 | 23 of 45 | | No-model word-coverage baseline (threshold 0.5) | 9 of 9 | 32 of 45 | | Same baseline (threshold 0.4) | 6 of 9 | 19 of 45 | "Flagged" means Not established or Partially answered. The baseline is a simple rule: how many of the scenario's words appear in the top three retrieved passages. I'm showing its whole sweep, not one hand-picked setting: thresholds 0.6, 0.7 and 0.8 also catch 9 of 9, but flag 37, 42 and 45 of the 45 documented behaviors. At the same 6 of 9, the baseline wrongly flags 19 of 45 and the default check flags 2. These are two different methods, so this compares outcomes on my test set. It doesn't show that Gemma is better in general, and it says nothing about stronger retrieval than my simple one. What it does show is what the model contributes here: far fewer false alarms. These are single runs on a small corpus that I partly built while tuning, so read the counts, not percentages. The last four pairs are the stronger evidence, but even they are not an independent evaluation. Two runs on the same machine gave identical results, which shows determinism, not robustness. What a run costs, on one Windows laptop (I didn't record the hardware): | Run | Time | |---|---| | One real scan, one spec, the model writes its own scenarios | about 8.5 minutes | | demo --quick, checkout pair, end to end | about 5 minutes | | 24 benchmark scenarios, default check | about 20 minutes | | The same 24 scenarios, stricter two-step check | about 35 minutes | | API fees | $0 | Where it breaks My favorite failure. For "a worker process crashes while running a job", the default check let Gemma quote the sentence about retrying a job that raises an error, and call the scenario Answered. The quote was real. The inference wasn't. Retrying after an error says nothing about what happens when the whole process dies. My citation check can't catch that, because the sentence exists. A yes/no relevance check I added didn't catch it either. The stricter two-step check does, along with two other missed gaps, but it also flags about half of the documented behaviors. So I left the default alone instead of crowning whichever mode looked best on my own data. What I learned: when a small model checks its own work, how strict you make the check is a product decision. It's a dial between missing gaps and crying wolf, and no setting of it is free. A real spec I didn't write To see whether that failure was just a quirk of my own test pairs, I ran the default settings once, with nothing tuned, on a design document from another project: ADR-22 from NATS, "JetStream Publish Retries on No Responders" (Apache-2.0; I did not copy it into my repo). The engine reported 12 scenarios and 0 potential gaps. (Its report counts only "Not established" as a potential gap; the one "Partially answered" finding sat in a separate section.) Then I read every finding against the document. The labels are one person's judgment, and they're in the repo. - 3 were correct and properly supported. - 1 was a genuine unanswered question: the ADR says what a publish returns after a timeout or after a last "no responders" error, but not what happens if the last attempt fails with some other error. The engine labeled it Partially answered, not a gap. - 5 were marked Answered, but the quote they cited didn't support the claim. - 3 were scenarios that contradict the spec or aren't about it. One said the client "does not attempt to send the message again", the opposite of the ADR's whole point. Another was about a Go timer failing to start. So one real finding and three correct confirmations, buried among eight that don't hold up. It's the same "real quote, unsupported inference" failure, on a document nobody tuned for, plus the scenario generator inventing hazards. This is one document labeled by one person, not a benchmark. Other caveats - My first run on the first two pairs went badly: Gemma labeled one planted omission "Partially answered" and called the other "Answered". Strictly, it found neither. I then added the relevance check, partial-answer reporting and a second metric ("unresolved", meaning Not established or Partially answered), using only those two pairs. That metric was defined after the first run. - On a plain scan, where the model writes its own scenarios, it invents requirements the spec never states. In my scheduled run, one flagged "gap" was a false alarm. Treat generated scenarios as prompts for review, not findings. - One export scenario in my own corpus is arguably ambiguous and got flagged in both versions. I left it as written. - Markdown and text only. No comparison with closed models. Bugs I hit along the way - Gemma copied my prompt's placeholder. In an early real run, the report's "question to resolve" was literally the example text from my own prompt. The prompt no longer contains placeholder text, and any question that shares no words with its scenario is replaced by a plain fallback. - My relevance check was too literal. It rejected the right answer to the crash scenario in version 2 because the spec says "lease expires", not "crash". I loosened it, and the looser check then accepted the wrong quote in version 1. That is how I ended up with the dial described above. - Windows Task Scheduler caps the command at 261 characters. My first scheduled command was far longer, so the engine now writes a short wrapper script and schedules that. Task Scheduler also refuses to start on battery by default, so I had to change that setting for a 2 AM run to work on a laptop. Why Does Open Innovation Matter? - The docs stay on my machine. Specs for systems that don't exist yet are exactly what I wouldn't paste into a hosted API. The engine refuses any model host that isn't localhost. It also patches Python's socket calls so a non-loopback connection or DNS lookup inside its own process raises an error, and a test checks that. That covers the Python process only, not what Ollama does. - It works offline. I turned Wi-Fi off, used a failing ping google.com to show it was off, and ran the demo. It finished with the same result. The ping only shows the network was down; the completed run is the evidence. - No API fees. - The weights are pinned. I recorded the model digest, so the benchmark can be reproduced instead of drifting under me. The model tag is a command-line flag, though I only benchmarked gemma3:4b. I never compared it with a closed model, so I'm not claiming this beats one on accuracy. My argument is privacy, offline use, cost and control. The Lazy Payoff I do not run it. After one setup command, Windows Task Scheduler does. python dark_matter.py schedule ./docs --at 02:00 On Oct 22 the task fired by itself at 2:00 AM and had written the report and a log line two seconds later. That was fast because the docs hadn't changed, so it reused the saved result. When they do change, a real scan takes its time: about 8.5 minutes for one spec on my laptop. Nobody is waiting, so slow is fine. A sloth would approve. After setup, the number of things I do each day is zero. As for the payoff: I ran it on my own spec corpus and, once, on a real design document. That run surfaced one genuine unanswered question, and also eight findings I had to read and throw out. So I can't honestly tell you how many hours it saves a team. What happens when the gateway charges the card and the response disappears? Version 1 of my spec had no answer. Version 2 did, after I added the idempotency behavior. That is the gap Dark Matter Engine is built to surface. It can't promise to find every missing answer, and it still produces false alarms. But it can read documentation overnight and leave me questions worth reviewing. The sloth moves slowly. At 2 AM, nobody has to remind it to get started. Prize Categories Best Use of Gemma: a local Gemma decides whether each quote establishes the required behavior. Python validates every citation and every output format. I kept those jobs separate on purpose.