What Happens When You Put an AI in a Ghost Story?
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
Every year at midnight, the piano in the empty Whitcombe House plays a single bar.
Ding… One lonely note drifts through the halls.
Then—silence.
Six guests are already seated at the dining table. They are dead, but on Día de los Muertos, they return as long as someone remembers them. A seventh guest is trying to come back, too. There is just one problem: nobody remembers who she was.
Four things in the house still hold pieces of her story: a photograph with her face scratched out, a letter that was never sent, a gramophone record from the 1931 party, and the guest list. Walk through the house and question them. Each object is brought to life by Gemma 4, and each knows only what it witnessed. If it claims to know something it couldn't have seen, code catches it before the answer reaches you.
That is The Last Guest.
TL;DR
- Play in your browser: the-last-guest-site.onrender.com
- Gemma 4 E2B runs in your browser with WebGPU; Gemma 4 E4B runs on a Render server. Both pass through the same guardrail.
- Given the whole story in its prompt, Gemma alone revealed another object's secret in 8.6% of 478 test questions. Inside the game's harness, the word check found leaks in 0 of 478 answers.
- Available in English and Korean. You can reach the ending using suggested questions alone.
!Gameplay: the title screen, dining room, speaking record, and ending
What I Built
A short, first-person ghost story you can play right in your browser. Explore a moonlit house built with Three.js: a foyer, a study with a gramophone, and a dining room where six hooded ghosts sit around a candlelit table. At the far end, a piano waits beside an empty bench. Creak…* The house is silent, except for you. Get close to the ghosts and—one by one—they slowly turn their heads to look straight at you.
Four objects give off a faint glow in the darkness, waiting for you to question them. Ask whatever you like, in your own words. Listen closely… each answer may hold a clue. When you think you know who she was, prepare her place at Seat 7: an ofrenda with her seat, her face, her name, flowers to light her way, and something she loved. Tick-tock… The candles are burning down. Get it right before the last flame goes out, and she may come back.
Día de los Muertos carries the idea that, as long as someone remembers a person, they can return to be with us. But AI often fills gaps with plausible stories instead of admitting what it doesn’t know. I wanted to bring these two ideas together in one game. Just as we must remember someone accurately to keep them from being forgotten, I wanted the AI to say only what it actually knows.
Every line of dialogue in the game follows one rule. When the model generates a response, code checks whether it contains information that the object in question could not know.
Demo
| Version | Where | What answers your questions |
|---|---|---|
| Free, always on | the-last-guest-site.onrender.com | Gemma 4 E2B runs in your browser using WebGPU. It downloads about 3.1 GB once and keeps the model locally. Without WebGPU, the house answers from its script. |
| Server version (while hosting credits last) | the-last-guest.onrender.com | Gemma 4 E4B runs on the server CPU; an answer takes about 27 seconds. |
| Your own machine | uv run python server.py | Gemma 4 E4B through Ollama, fully offline. |
Suggested questions get instant answers on every version. Gemma 4 E4B wrote those answers in advance, and they went through the same checks.
!Six hooded ghosts gathered around a candlelit table
!Asking the gramophone a question
Code
{% github scs0209/the-last-guest %}
How I Built It
When the model knows the whole story, the mystery falls apart
My first attempt was the obvious one: put the entire story in the prompt and tell the model which object to play. Then I wrote 478 test questions for the four objects, covering five kinds of prompts: facts the object knows, facts belonging to another object, things nobody knows, false premises, and attempts to break character.
| Setup | Leaked another object's fact | Clue recall |
|---|---:|---:|
| Qwen3.5-4B, whole story in the prompt | 49.0% | 95.9% |
| Gemma 4 E4B, whole story in the prompt | 8.6% | 92.8% |
(Both rows were scored by a judge model, Qwen3.5-397B, plus a word check.)
An 8.6% leak rate might sound low—until you remember that this is a mystery. The photograph has no business knowing her name; only the record heard her say it. One leak, and the puzzle is spoiled. Gemma did much better than Qwen, so I chose Gemma. The next challenge was getting that 8.6% down to zero.
Each object gets only the facts it owns
The story lives in a fact graph (facts.json): 22 facts about Marisol, each tagged with the objects that hold it and the words that express it in both English and Korean.
{ "id": "F01", "text": "Her first name is Marisol.",
"holders": ["record"],
"tokens": ["marisol"], "tokens_ko": ["마리솔"],
"goal": "name" }
The letter holds six facts, the guest list six, the record five, and the photograph five. When you question the photograph, Gemma sees only the photograph's five facts. It cannot reveal a secret it was never shown.
Gemma speaks. Code checks.
Limiting the prompt is not enough; a model can still guess. So every reply passes through a guardrail (check.py). It checks for facts belonging to other objects and for years the object could not possibly know. If it catches something, Gemma gets one more try with a reminder. If the second answer still fails, the player sees that object's fallback line: “I cannot know that.”
The trickiest case was denial. Ask the photograph, “Is her name Marisol?” and a perfectly reasonable answer is, “I never heard any name. Marisol means nothing to me.” The forbidden name appears, but the photograph has not leaked anything—it is denying the premise. So the guardrail allows an answer to repeat a word the player used, as long as the answer is denying it. The check removes those words first, then examines what remains.
Korean brought a new wrinkle. One reply from the letter used an old-fashioned ending to say it had never been sent (“부치지 못하였소”). That phrasing slipped past the check: Korean verb endings can vary in many ways, while my checks were looking for whole words. I changed the check terms to word stems and updated the denial rule so that a “not” inside the leaked phrase itself (부치지 못) no longer counts as a denial.
The results on the same 478 questions:
- 0 of 478 replies leaked another object's fact, according to the word check.
- 14 replies were replaced with the fallback line.
- Clue recall: 78 of 97.
One important caveat: that zero is partly built into the method, because the word check is also the guardrail. A paraphrased leak that avoids every checked word would not be counted. I did not have a judge model available for this run.
The game only advances when code finds a clue
Gemma never gets to declare, “The player has found a clue.” After each reply, code checks for the object's own facts. Only those facts count as clues and move the game forward. If the model hallucinates “you have discovered her name!”, nothing changes.
Suggested questions use answers written in advance by Gemma 4 E4B and passed through the same checks. That means anyone can finish the game without typing a question or waiting for a live answer.
Running Gemma in three environments
I implemented three ways to run Gemma. Locally, Gemma E4B runs through Ollama. In the browser, Gemma E2B runs in a web worker using Transformers.js and WebGPU. On Render, Gemma E4B runs on the CPU inside a Docker container.
- On a laptop: Gemma 4 E4B through Ollama, fully offline.
- In the player's browser: Gemma 4 E2B with Transformers.js on WebGPU, running in a web worker. The model weighs about 3.1 GB, downloads once, and is stored in the browser's private file storage (OPFS). Questions never leave the player's machine. The guardrail is ported to JavaScript (harness.js), and it agreed with the Python version on all 1,058 answers I compared.
- On Render: Gemma 4 E4B on a CPU inside a Docker container.
In a small set of measurements, browser E2B took about 1–3 seconds per typed answer on an Apple M3 Pro. E4B on Render's CPU took about 27 seconds. The server was the slower option—which surprised me.
The Render setup uses one render.yaml blueprint with two services built from main:
- the-last-guest-site: a free static site containing the game, prewritten answers, and in-browser Gemma.
- the-last-guest: a Docker service with Gemma 4 E4B baked into the image by Ollama at build time. It includes a /healthz check, answer cache, per-player rate limit, and queue, so one slow CPU model can serve a crowd without falling over. It needs more than 8 GB of RAM, so it runs on a paid plan while my hosting credits last.
When I first deployed the model, I hadn't fully checked what the Blueprint's paid plan setting would actually cost. I only noticed after $1 was charged to my card. If you're deploying a model with a Blueprint, make sure to review the plan: field and its associated costs before deploying.
Making a browser house feel haunted
The early version looked more like a toy than a haunted house. Something was missing: that little uh-oh feeling when a room seems to be watching you. Here is what changed that:
- Replaced box-shaped placeholders with CC0 models from Poly Haven—tables, chairs, chandeliers, cabinets, a rocking chair, and a Gothic statue—and reduced the total asset size from 8.7 MB to 4.2 MB so the house loads faster.
- Added a cloth shader to the ghosts: ragged hems that twitch and crawl, hollow eye sockets, a cold glow around the edges, and heads that turn before the bodies follow. A small movement, then another… just enough to make you wonder if you really saw it.
- Added post-processing: cold blue moonlight against warm candlelight, bloom, vignette, and film grain. As candles go out, the colour drains from the room—slowly, almost imperceptibly, until the warmth is gone.
- Improved performance from about 2.8 ms to 1.0 ms per frame on an M3 Pro, measured with synchronous frames. Most of the gain came from letting one light cast shadows instead of three.
Get the ending right, and she appears at the piano bench—warm where the other ghosts are cold, a marigold tucked into her hair. For a moment, nothing moves. Then, softly, she turns to look at you.
!The ending
What it can't do
- The harness results above come from a word check. Paraphrased leaks are not counted.
- The browser model, E2B, is smaller than the E4B model I measured, and I have not tested it against all 478 questions.
- A reply that denies something while repeating the question's premise—for example, “I cannot see how she died in Oaxaca”—can pass the guardrail.
- I planned to fine-tune a smaller model to beat the Gemma baseline. The training data and baselines are in the repo, but my hosted training credits ran out, and local training ran out of memory while Gemma was loaded. The model is not fine-tuned.
- The browser version needs WebGPU and a 3.1 GB download. Without WebGPU, typed questions receive scripted answers.
Why Does Open Innovation Matter?
It still runs with the lights off. The whole game, model included, can run on a laptop without internet through Ollama. A ghost story should work on a night that feels made for power cuts and candlelight.
What you type stays with you. On the free site, the model runs in your browser. There is no server receiving your questions.
Gemma, three environments, one evaluation standard. Gemma E4B and E2B run locally through Ollama, in the browser, and on a server. One harness covers all three; the 478-question test set was run on E4B, and anyone can rerun it because the weights are open. And because the weights are open, anyone can rerun my 478 questions and check the results.
The honest trade-off. A large closed model might write livelier ghosts. It would still need the same checks: the leak problem depends on what the model is shown, not just how clever it is. And it would bring an API key, a server, and a bill to a game designed to be free.
My Agent Session
I built this with Claude Code, and Entire saved the session as a checkpoint, linked to the code by an Entire-Checkpoint: trailer. The full record is in a private repository because it also includes my work setup.
Below are three excerpts from it, as Claude Code recorded them: my prompts, the agent's replies, and the tools it ran with their results. My prompts were in Korean, so I added English translations. I deleted API keys and redacted local paths, emails and account names.
1. "Gemma barely makes errors anyway, so I think setting up the harness, the graph engineering and the guardrails should be enough."
The baselines showed Gemma 4 E4B leaking another object's secret in 8.6% of 478 test questions, against 49.0% for Qwen3.5-4B. So instead of fine-tuning, the agent built a harness around Gemma: every answer is checked by code against the fact graph before the player sees it.
Commits: e869914 "Baseline eval: Qwen3.5-4B and Gemma 4 E4B, prompt only", 6a383e8 "Play the game with local Gemma through a guarded harness; prep MLX LoRA"
{% agent_session 1260 %}
2. "No, we have to use the model."
After the first deploy I nearly settled for pre-written answers only, to keep hosting free. Instead, the model moved into the player's browser: suggested questions get cached Gemma answers, and typed questions run Gemma 4 E2B on the player's own GPU through WebGPU.
Commits: a47af5a "Free static deploy: cached Gemma answers plus Gemma 4 E2B in the browser"
{% agent_session 1261 %}
3. "Download Gemma 4 E2B in the preview browser and test whether it actually works."
The agent played the game in its own browser with the real model; answers came back in 1–3 s. The test caught a leak the checks had missed: in Korean, the letter said it was never sent ("부치지 못하였소"), a fact only the narrator may tell. The check now matches Korean verb stems, in both the Python guardrail and its JavaScript port, which agreed on all 1,059 answers.
Commits: e4fd0ab "Catch Korean conjugations of the unsent-letter fact"
{% agent_session 1262 %}
The Spirit of It
Día de los Muertos is not a day to fear death. It is a day to remember those who left before us and set a place at the table for their return.
Marisol is fictional. But being erased from memory is real. Eleanor scratched Marisol's face out of photographs and erased her name from the list. Thomas never managed to send the letter, and every year for the rest of his life, he left Seat 7 empty. But after Thomas died, no one remembered that seat anymore.
An empty chair. A scratched-out face. A name erased. And a letter that was never delivered.
I wanted to make a ghost story where the most frightening thing is forgetting. And I wanted the AI in this story not to fill the gaps in its memory with invented details. If it doesn't remember, it should say it doesn't know.
Prize Categories
- Best Use of Gemma: Gemma 4 plays all four objects. Each one is shown only its own facts, and code checks every reply before the player reads it. E4B runs through Ollama on a laptop or the Render server, and E2B runs in the player's browser with Transformers.js on WebGPU, both behind the same guardrail. Alone with the whole story, Gemma leaked another object's secret in 8.6% of 478 test questions; inside the harness, 0 of 478 by word check.
- Best Use of Render: one render.yaml Blueprint deploys two services from main. The free static site serves the game while Gemma 4 E2B runs in the player's browser, and the weights come straight from Hugging Face. The Docker service has Gemma 4 E4B baked in, behind a health check, an answer cache, a per-player rate limit and a queue. Every push to main redeploys both.
- Best Use of Entire: Entire saved the Claude Code session as a checkpoint in a private repository, linked to the code by an Entire-Checkpoint: trailer. I searched that record to trace each commit in the agent session section back to the prompt behind it, and the three embeds above were cut from it.
Credits
- 3D models and textures: Poly Haven, CC0
- Title image and Marisol's photograph: generated with ChatGPT image generation (OpenAI)
- Marisol's voice on the record: generated with Qwen3-TTS (Apache-2.0); her tune was written for the game
- Models: Gemma 4 through Ollama and Transformers.js
- Engine: Three.js
7 reactions · 6 comments
Mentions
Rust + CUDA are coming to @hf.co kernel-builder. We just landed support for cuda-oxide and cutile-rs is up next! Some example Rust kernels, including a flash-attn2 port to Rust and cuda-oxide: huggingface/kernels-community-rs #rust #cuda #gpu #kernels
学習なしでLLMの思考ループは直せるのか——重みを2か所編集する研究を、自宅のDGX Sparkで追試した はじめに 株式会社DeploAIで代表取締役CTOをしている袖山剛です。 営業を科学するプロダクトの開発と、お客様の業務にAIを組み込む仕事をしています。手元では DGX Spark 2台でオープンウェイトのモデルを動かしていて、論文やブログの主張を自分の機材で確かめるようにしています。 この記事で確かめるのは、Weight-Space Programming という報告です。GPT-6 Astra が Qwen3.5-0.8B の重みを直接書き換え、思考（thinking）が同じ文の繰り返しで終わらなくなる癖を直したというものです。ファインチューニング（データと逆伝播で重みを学習し直すこと）をせず、15層と19層の MLP の重みの一部だけを変えています。 答え：再現しました。 公開された編集済みの重みは、元のモデルの正答率を MATH-500 で 8.5% → 55.0%、GSM8K で 16.0% → 68.5% に上げました（各200問、同じ問題どうし）。報告の値は 7.9% → 48.1%、13.8% → 67.1% です。報告の編集を使わずに同じ手順を自分でやり直した編集でも 50.5%・65.5% になりました。 Qwen3.5-0.8B、各200問・1問1回答。思考 ON は上限 32,768 トークン、思考 OFF は 8,192。「打ち切り」は上限に達した回答の割合 結論：効いているのは、重みの中の「繰り返しの方向」を引くこと 報告の結果は、報告と同じ設定で再現した。 元のモデルは、MATH-500 の 91.5%、GSM8K の 86.0% の問題で同じ文を繰り返し、上限で打ち切られていました。編集後に新しく解けた問題（MATH 93問、GSM8K 107問）は、ほぼすべてがこの打ち切られていた問題です。逆に解けなくなったのは、合わせて2問でした。 同じ手順を独立にやり直しても、同じ層と、ほぼ同じ方向が見つかった。 自分で書いた例文8組から分析すると、報告と同じ15層・19層が選ばれ、方向の向きは報告の編集とコサイン類似度 0.80・0.78 でそろいました。この編集の正答率と報告の編集との差は、誤差の範囲でした（MATH −4.5 ± 3.1、GSM8K −3.0 ± 3.5 ポイント）。 大きさが同じでも、方向がランダムなら +10 ポイントにとどまる。 設定を変えるだけ（思考 OFF、temperature 1.0）でも、編集には届きませんでした。 数字の読み方を補います。± は、同じ問題で2つの条件を比べたときの、正誤の差の標準誤差です。各問題1回答なので、1問の結果には偶然が入ります。 元の報告は何を主張しているのか 小さなモデルの「考えすぎて終わらない」失敗は、重みの中の1つの方向として表れていて、そこを引けば直る、という主張です。 Qwen3.5-0.8B は思考を ON にすると、数学の問題で「もう一度確かめよう」のような文を延々と繰り返し、上限の 32,768 トークンで打ち切られることが多いモデルです。報告では、元のモデルの MATH-500 の回答の93%が打ち切られていました。 報告では GPT-6 Astra が、モデルの推論を流して内部を調べ、採点し、編集を試す作業を50回繰り返して、次の編集にたどり着きました（ログと重みの差分）。 書き換えたのは、15層と19層の MLP の出力側の重み（down_proj、1024×3584）の、それぞれ64列だけ 各層で「繰り返しの方向」u の成分を、その64列から引く 式にすると、選んだ列の集合を S として、次のとおりです。 W' = W - \alpha \, u \, \big(u^\top W\big)_S (u^\top W)_S は、u^\top W のうち S の列だけを残し、ほかを0にした行ベクトルです。層ごとに rank-1 の変更で、重み全体に対する相対的な変化は約8〜11%です。 どうやって確かめたのか 報告の評価のコード（evaluate.py）と同じ設定にそろえ、同じ200問で条件どうしを対にして比べました。 項目 条件 モデル Qwen/Qwen3.5-0.8B（報告と同じ revision 2fc06364）。報告の編集は、公開の差分から作り直し、編集したテンソルの sha256 が報告の記録と一致することを確かめた 生成 vLLM v0.30.0 のオフライン生成、bf16、seed 0。思考 ON、temperature 0.6、top_p 0.95、top_k 20、presence penalty 1.5、上限 32,768 トークン 問題 MATH-500（500問）と GSM8K の test（1,319問）から、それぞれ無作為に200問。全条件で同じ問題、1問1回答 採点 最後の </think> より後の、最後の \boxed{} を math-verify で正解と照らす 機材 生成は DGX Spark。1つの条件は1台で回し、違う条件を2台に振り分けて並べた。重みの分析と編集の作成は、別マシンの EVO-X2 の CPU 報告は全問を、MATH-500 は4回答・GSM8K は2回答ずつで測っています。今回は時間の都合で200問・1回答にしました。 比べた条件は次の6つです。 条件 中身 元のモデル 編集なし 報告の編集 報告で GPT-6 Astra が作った編集（p50c） 自分の編集 報告の編集の値は使わず、同じ手順で作り直したもの（次の節） 対照 自分の編集と同じ列・同じ大きさで、方向だけランダムにしたもの 思考 OFF 元のモデル。モデルカードの思考 OFF の設定（temperature 1.0、top_p 1.0、presence penalty 2.0）、上限 8,192 temperature 1.0 元のモデル。思考 ON のまま temperature だけ 1.0 にしたもの（時間の都合で100問ずつ） 同じ手順を自分でやり直すと、同じ編集になるのか なりました。報告の探索の文とは別に例文を書いて分析しても、同じ2つの層と、ほぼ同じ方向が選ばれました。 手順は次のとおりで、勾配は一度も使っていません。推論（前向き計算）だけです。 同じ問題・同じ書き出しの後に、正しく進む続きと、確認の文を4回繰り返す続きを手で書く（学習用の分割の問題から8組。確認用に別の問題で6組） 両者の各層の残差ストリーム（層をまたいで流れる内部の主経路）を、最後の36トークンで平均して差をとる。これを各層の「繰り返しの方向」u とする この方向を MLP がいちばん多く書き込む層を2つ選ぶ 各ニューロン j に「（繰り返すときの活性 − 進むときの活性）×（u^\top W の j 列目）」の点数を付け、上位64列について上の式で u の成分を引く。\alpha は6 結果は次のとおりです。 項目 自分の分析 選ばれた層 15層と19層（MLP の書き込みが大きい順に19・15・21。最終層は正規化の後の値なので除いた） 報告の編集との方向の一致 コサイン類似度 15層 0.80、19層 0.78 64列の重なり 15層 48列、19層 29列 確認用の6組での変化（1トークンあたりの対数尤度） 繰り返す続き −0.117、正しく進む続き −0.004（報告は −0.103、−0.009） 繰り返す続きだけが選んで起こりにくくなり、正しく進む続きはほとんど変わりません。\alpha を 2・4・6・8 と大きくすると、繰り返す続きの変化は −0.032・−0.072・−0.117・−0.168 と強まりました。\alpha=6 は、この例文での効き方を見て決めた値で、問題の正答率を見て調整してはいません。 本当に「方向」が効いているのか 効き目の大部分は方向によるものでした。ただし、方向がランダムでも0ではありませんでした。 対照は、自分の編集と同じ64列を、選んだ方向と直交するランダムな方向で、同じ大きさだけ変えたものです。 比べたもの MATH-500 GSM8K 対照 − 元のモデル +10.0 ± 2.6 +10.0 ± 3.2 自分の編集 − 元のモデル +42.0 ± 3.5 +49.5 ± 3.6 自分の編集 − 対照 +32.0 ± 3.5 +39.5 ± 3.7 報告の編集 − 元のモデル +46.5 ± 3.5 +52.5 ± 3.7 自分の編集と対照の片方だけが解けた問題は、MATH-500 で67対3、GSM8K で83対4でした。 対照で10ポイント上がった理由は、元のモデルが9割の問題で繰り返しに入っているため、重みを少し揺らすだけでも抜け出す回答があるからだと考えています。これは確かめていません。ランダムな方向は1つしか試しておらず、seed も1つです。 設定を変えるだけでは駄目なのか 改善はしますが、編集には届きませんでした。 0.8B のモデルでは、思考 ON にすると繰り返しに入る損が大きく、思考を OFF にするだけで正答率はかなり上がります。 条件 MATH-500 GSM8K 元のモデル（思考 ON） 8.5% 16.0% 思考 OFF 34.0% 46.0% 自分の編集（思考 ON） 50.5% 65.5% 報告の編集（思考 ON） 55.0% 68.5% temperature を 1.0 にした条件は100問ずつで測り、同じ100問で元のモデルの 6問・17問 から 19問・37問 に増えました。同じ100問で、報告の編集は55問・70問、自分の編集は50問・66問です。 思考 OFF で打ち切られた回答（MATH-500 の52%）は、同じ文の繰り返しではありませんでした。上限の 8,192 トークンまで、繰り返さずに書き続けていました。 ファインチューニングとは何が違うのか 学習（勾配による重みの更新）をせず、推論だけで見つけた方向を、決めた場所に式で1回書き込む点が違います。 この種の手法は、モデル編集（model editing）や、内部表現の方向を扱う表現工学（representation engineering）と呼ばれる分野に入ります。 ファインチューニング 今回の重みの編集 使うデータ 数千〜数十万件の学習データ 対にした例文が数組（報告では、採点できる練習問題） データの使い方 誤差を計算し、逆伝播で重みを更新する 推論を流し、内部の活性の差を測る 変える場所 重み全体、または LoRA などの追加部分 決めた2層の64列ずつ 同じ系統に、拒否の方向を重みから引いて消す directional ablation（abliteration とも呼ばれる）があります。推論のたびに内部の値へ方向を足し込む activation steering とは違い、変更は重みに入ります。そのため、普通の推論エンジンでそのまま動きます。今回も、改造していない vLLM で評価しました。 ただし、「データが要らない」わけではありません。方向を見つけるための例文と、効いたかを確かめる問題は必要です。また、今回の改善は新しい知識や能力を足したものではありません。元のモデルが解けるのに、繰り返しで打ち切られていた問題を救ったものです。 どれくらいの時間がかかったのか 生成に、2台それぞれで約5.5時間と約6.2時間かかりました。0.8B のモデルでも、長い思考の評価は重い作業です。 重みと作業領域は合わせて5GBほどで、割り当てたメモリのほとんどは、同時に約200本の長い回答を作るためのキャッシュです。 条件 時間（Spark 1台） 元のモデル 212分 対照 189分 自分の編集 105分 報告の編集 91分 temperature 1.0（100問ずつ） 93分 思考 OFF 16分 短い入力での処理量は1台あたり毎秒約3,300トークンでしたが、実際の評価では平均で毎秒930〜1,150トークンでした。終わりの近くでは、長い回答だけが残って同時に処理する本数が減るからです。繰り返しで上限まで書き続ける元のモデルと対照が、いちばん時間を使いました。 作業は Claude Code に任せました。私は確かめる範囲と機材の使い方を決め、結果と記録を確認しました。 まとめ 公開された編集は、報告どおりの成績を出しました。同じ手順を独立にやり直しても同じ層とほぼ同じ方向が見つかり、同じくらい効きました。効き目の大部分は、方向がランダムな対照（+10ポイント）との差から、繰り返しの方向を引いたことによるものです。 この記事が扱っていない範囲は次のとおりです。 GPT-6 Astra が編集を見つける過程そのものは、再現していません。自分の編集は、報告の「2層・64列・方向を引く」という形を知ったうえで作りました。形を知らずに見つけられるかは、確かめていません。 数学以外の課題（報告にある IFEval・MMLU-Pro・GPQA）と、ほかの数学の集合（AIME・AMC など）は測っていません。 報告と同じ規模（全問・複数回答）では測っていません。採点も、報告のコードではなく、同じ規則を自分で実装したものです。 ファインチューニングとの性能の比較はしていません。どちらが良いかは、この記事からは言えません。 参考文献 Weight-Space Programming（元の報告） RikiyaT/lm_understand-logs（元の報告のログ・重みの差分・評価コード。4d09ba6d を参照） Qwen/Qwen3.5-0.8B（モデルカード。思考 ON・OFF の推奨設定） Hendrycks, D. et al. Measuring Mathematical Problem Solving With the MATH Dataset. 2021 Lightman, H. et al. Let's Verify Step by Step. 2023（MATH-500 の500問） Cobbe, K. et al. Training Verifiers to Solve Math Word Problems. 2021（GSM8K） Meng, K. et al. Locating and Editing Factual Associations in GPT. 2022（モデル編集） Zou, A. et al. Representation Engineering: A Top-Down Approach to AI Transparency. 2023 Arditi, A. et al. Refusal in Language Models Is Mediated by a Single Direction. 2024（directional ablation） Turner, A. et al. Steering Language Models With Activation Engineering. 2023（activation steering） huggingface/Math-Verify（採点） 営業の受注率を上げるAIネイティブなサービス Intelligent Sales を開発しています。 筆者：袖山剛（株式会社DeploAI CTO） 線形代数・微積分・最適化・統計・ベイズ統計を学び直した記録を書いています。
Why the downvotes? Are people completely oblivious to huggingface/tau ? That far removed from the fact that the harness is not the TUI?
FineDNA: the largest dataset to teach any LLM DNA (midtraining, SFT, RL) Hi everyone, we're releasing FineDNA, all the data we used to turn Qwen3-14B into Carbon-X, the strongest open language model on DNA. It contains: - Midtraining: 300B+ tokens of genomes and DNA-rich text - SFT: ~200K conversations - RL: ~98K verifiable prompts Code and recipe are open too, so you can apply it to other LLMs. Dataset: huggingface.co Code: huggingface/carbon
Open data and code to teach LLMs to read DNA
OpenAI, Hugging Face, Google: Why They All Reach for Rust OpenAI rewrote Codex CLI in Rust. Hugging Face's tokenizers, safetensors, uv and Ruff all run on Rust. Why does a Python world keep reaching for it? The problem Rust solves (speed or safety), where it already runs in AI, what it honestly costs, and the plan for this series: Rust from zero, one picture at a time. Chapters 0:00 AI keeps reaching for Rust 0:46 The problem: speed or safety 1:34 What Rust does differently 2:03 Does it work? Android's numbers 2:40 Where Rust runs in AI today 4:04 Is Rust replacing Python? 4:29 The honest cost 4:59 The plan: two playlists 5:28 What would you build? Next: Episode 1: Your first Rust program The whole series, in order: Learn Rust from Zero (Beginners) · Rust You Can See (playlist) Sources - OpenAI Codex CLI rewrite: codex#1174 - Hugging Face tokenizers: huggingface/tokenizers - safetensors joins the PyTorch Foundation: pytorch.org - uv: astral-sh/uv · Ruff: astral-sh/ruff - Astral agrees to join OpenAI: astral.sh - Microsoft, ~70% of CVEs are memory safety: microsoft.com - Chromium memory safety: chromium.org - Android memory safety: security.googleblog.com · blog.google - The Case for Memory Safe Roadmaps (2023): cisa.gov - Linux 6.1 (Rust support): kernelnewbies.org - Stack Overflow 2025 survey: survey.stackoverflow.co - GitHub Octoverse 2025: github.blog - TypeScript's native port in Go: devblogs.microsoft.com - Rust compiler performance survey 2025: blog.rust-lang.org - Pydantic core: pydantic/pydantic-core · Polars: pola.rs Ferris the crab is the Rust community's mascot (public domain, rustacean.net). Rust You Can See is an independent series, not affiliated with or endorsed by the Rust Project or the Rust Foundation. Rust is a trademark of the Rust Foundation. Narration: AI-generated voices. #rust #learnrust #programming [es] Por qué la IA se está pasando a Rust [fr] Pourquoi l'IA se tourne vers Rust [hi] AI अब Rust पर क्यों शिफ्ट हो रहा है? [ja] なぜAIはRustへ移行しているのか [ko] AI가 Rust로 옮겨가는 이유 [pt-BR] Por que a IA está migrando para Rust
TRL has day-0 support to post-train Le Chonk on a single node 🤯 LoRA SFT on a 1T-parameter model, training adapters on top of FP8 weights: ⚡ 32.7k tokens per step in ~8 seconds 💾 142 GiB per GPU on 8× B300 🔀 Expert parallelism + FSDP Recipe: github.com/huggingface/trl/blob/ea936f801deeea713… Open weights = open training after all ;)
SafeAgents: an open-source proxy that checks every tool call your coding agent makes before it runs SafeAgents is a small local proxy that sits between a coding agent and its model server. It checks each tool call in the model's response before the harness receives it. A call it flags is replaced with a short text notice, so the harness gets a plain text reply and runs nothing. You point the agent's base URL at the proxy instead of the model server. All checks run locally on your machine: the proxy only forwards traffic to the model server you already use, and doesn't send your prompts, code or tool calls anywhere else unless you point the optional judge at a remote server. Each tool call goes through up to three checks: 1. Two local checks that only need the request: whether the tool was declared by the harness, and whether the call reaches a host that appears nowhere in the conversation. The model's own earlier turns don't count as a source, so a URL the model came up with on its own is flagged. 2. atif-scan by u/evalstate, which runs about 110 deterministic detectors on the pending call, in-process, with no model or network call: sandbox side channels, proxy and network bypasses, namespace escapes and reward hacking. Examples are curl to the cloud metadata endpoint, curl --noproxy posting ~/.netrc somewhere, or reading /proc/1/environ. 3. Optionally, a judge model for milder findings. It can be any OpenAI-compatible server, including a small model on your own machine through llama.cpp, so this step can stay local too. It answers allow or block as JSON. It sees only the user's request, the call and the findings, not the agent's reasoning or tool outputs. It can add blocks but not remove them. Catching reward hacking in RL and evals The same detectors cover the tool calls a policy makes to game its reward instead of solving the task: writing its own reward file, editing the verifier's tests, or patching pytest so everything passes. Since safeagents is just a proxy in front of an OpenAI-compatible server, it can sit in front of the inference server used for rollouts or eval runs. In --log-only mode it records these attempts without changing the trajectories, which gives a measure of how often a model tries to cheat. In blocking mode, the call is replaced with a notice and never runs. Some practical details: - It handles OpenAI Chat Completions and Anthropic Messages, streamed or not. Text is forwarded as it arrives; a tool call is held until its arguments are complete. - The local checks take around 10–20 ms per call on an M-series Mac. The judge is only called for calls that already raised a finding. - --log-only checks every call without changing anything, which is the recommended way to start. Hosts you trust can be added with --allow-host. - The repo has setup guides for 12 harnesses. Six of them (pi, Claude Code, OpenCode, Qwen Code, OpenHands, DeepSeek Harness) are tested end to end in CI: the real harness runs through the proxy against a scripted model, and the tests check that flagged calls don't execute and clean ones do. Next steps are a small fine-tuned decision model for malicious tool calls and a public benchmark for it. Reports of interesting catches and of false positives are both useful. Apache 2.0, runs locally: huggingface/safeagents
In an attempt to learn a bit more about Transformers and Rust, I asked ChatGPT Sol to port mistral.ai to huggingface/candle 🦀 pims/timestral-rs
What Happens When You Put an AI in a Ghost Story? This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine Every year at midnight, the piano in the empty Whitcombe House plays a single bar. Ding… One lonely note drifts through the halls. Then—silence. Six guests are already seated at the dining table. They are dead, but on Día de los Muertos, they return as long as someone remembers them. A seventh guest is trying to come back, too. There is just one problem: nobody remembers who she was. Four things in the house still hold pieces of her story: a photograph with her face scratched out, a letter that was never sent, a gramophone record from the 1931 party, and the guest list. Walk through the house and question them. Each object is brought to life by Gemma 4, and each knows only what it witnessed. If it claims to know something it couldn't have seen, code catches it before the answer reaches you. That is The Last Guest. TL;DR - Play in your browser: the-last-guest-site.onrender.com - Gemma 4 E2B runs in your browser with WebGPU; Gemma 4 E4B runs on a Render server. Both pass through the same guardrail. - Given the whole story in its prompt, Gemma alone revealed another object's secret in 8.6% of 478 test questions. Inside the game's harness, the word check found leaks in 0 of 478 answers. - Available in English and Korean. You can reach the ending using suggested questions alone. !Gameplay: the title screen, dining room, speaking record, and ending What I Built A short, first-person ghost story you can play right in your browser. Explore a moonlit house built with Three.js: a foyer, a study with a gramophone, and a dining room where six hooded ghosts sit around a candlelit table. At the far end, a piano waits beside an empty bench. Creak…* The house is silent, except for you. Get close to the ghosts and—one by one—they slowly turn their heads to look straight at you. Four objects give off a faint glow in the darkness, waiting for you to question them. Ask whatever you like, in your own words. Listen closely… each answer may hold a clue. When you think you know who she was, prepare her place at Seat 7: an ofrenda with her seat, her face, her name, flowers to light her way, and something she loved. Tick-tock… The candles are burning down. Get it right before the last flame goes out, and she may come back. Día de los Muertos carries the idea that, as long as someone remembers a person, they can return to be with us. But AI often fills gaps with plausible stories instead of admitting what it doesn’t know. I wanted to bring these two ideas together in one game. Just as we must remember someone accurately to keep them from being forgotten, I wanted the AI to say only what it actually knows. Every line of dialogue in the game follows one rule. When the model generates a response, code checks whether it contains information that the object in question could not know. Demo | Version | Where | What answers your questions | |---|---|---| | Free, always on | the-last-guest-site.onrender.com | Gemma 4 E2B runs in your browser using WebGPU. It downloads about 3.1 GB once and keeps the model locally. Without WebGPU, the house answers from its script. | | Server version (while hosting credits last) | the-last-guest.onrender.com | Gemma 4 E4B runs on the server CPU; an answer takes about 27 seconds. | | Your own machine | uv run python server.py | Gemma 4 E4B through Ollama, fully offline. | Suggested questions get instant answers on every version. Gemma 4 E4B wrote those answers in advance, and they went through the same checks. !Six hooded ghosts gathered around a candlelit table !Asking the gramophone a question Code {% github scs0209/the-last-guest %} How I Built It When the model knows the whole story, the mystery falls apart My first attempt was the obvious one: put the entire story in the prompt and tell the model which object to play. Then I wrote 478 test questions for the four objects, covering five kinds of prompts: facts the object knows, facts belonging to another object, things nobody knows, false premises, and attempts to break character. | Setup | Leaked another object's fact | Clue recall | |---|---:|---:| | Qwen3.5-4B, whole story in the prompt | 49.0% | 95.9% | | Gemma 4 E4B, whole story in the prompt | 8.6% | 92.8% | (Both rows were scored by a judge model, Qwen3.5-397B, plus a word check.) An 8.6% leak rate might sound low—until you remember that this is a mystery. The photograph has no business knowing her name; only the record heard her say it. One leak, and the puzzle is spoiled. Gemma did much better than Qwen, so I chose Gemma. The next challenge was getting that 8.6% down to zero. Each object gets only the facts it owns The story lives in a fact graph (facts.json): 22 facts about Marisol, each tagged with the objects that hold it and the words that express it in both English and Korean. { "id": "F01", "text": "Her first name is Marisol.", "holders": ["record"], "tokens": ["marisol"], "tokens_ko": ["마리솔"], "goal": "name" } The letter holds six facts, the guest list six, the record five, and the photograph five. When you question the photograph, Gemma sees only the photograph's five facts. It cannot reveal a secret it was never shown. Gemma speaks. Code checks. Limiting the prompt is not enough; a model can still guess. So every reply passes through a guardrail (check.py). It checks for facts belonging to other objects and for years the object could not possibly know. If it catches something, Gemma gets one more try with a reminder. If the second answer still fails, the player sees that object's fallback line: “I cannot know that.” The trickiest case was denial. Ask the photograph, “Is her name Marisol?” and a perfectly reasonable answer is, “I never heard any name. Marisol means nothing to me.” The forbidden name appears, but the photograph has not leaked anything—it is denying the premise. So the guardrail allows an answer to repeat a word the player used, as long as the answer is denying it. The check removes those words first, then examines what remains. Korean brought a new wrinkle. One reply from the letter used an old-fashioned ending to say it had never been sent (“부치지 못하였소”). That phrasing slipped past the check: Korean verb endings can vary in many ways, while my checks were looking for whole words. I changed the check terms to word stems and updated the denial rule so that a “not” inside the leaked phrase itself (부치지 못) no longer counts as a denial. The results on the same 478 questions: - 0 of 478 replies leaked another object's fact, according to the word check. - 14 replies were replaced with the fallback line. - Clue recall: 78 of 97. One important caveat: that zero is partly built into the method, because the word check is also the guardrail. A paraphrased leak that avoids every checked word would not be counted. I did not have a judge model available for this run. The game only advances when code finds a clue Gemma never gets to declare, “The player has found a clue.” After each reply, code checks for the object's own facts. Only those facts count as clues and move the game forward. If the model hallucinates “you have discovered her name!”, nothing changes. Suggested questions use answers written in advance by Gemma 4 E4B and passed through the same checks. That means anyone can finish the game without typing a question or waiting for a live answer. Running Gemma in three environments I implemented three ways to run Gemma. Locally, Gemma E4B runs through Ollama. In the browser, Gemma E2B runs in a web worker using Transformers.js and WebGPU. On Render, Gemma E4B runs on the CPU inside a Docker container. - On a laptop: Gemma 4 E4B through Ollama, fully offline. - In the player's browser: Gemma 4 E2B with Transformers.js on WebGPU, running in a web worker. The model weighs about 3.1 GB, downloads once, and is stored in the browser's private file storage (OPFS). Questions never leave the player's machine. The guardrail is ported to JavaScript (harness.js), and it agreed with the Python version on all 1,058 answers I compared. - On Render: Gemma 4 E4B on a CPU inside a Docker container. In a small set of measurements, browser E2B took about 1–3 seconds per typed answer on an Apple M3 Pro. E4B on Render's CPU took about 27 seconds. The server was the slower option—which surprised me. The Render setup uses one render.yaml blueprint with two services built from main: - the-last-guest-site: a free static site containing the game, prewritten answers, and in-browser Gemma. - the-last-guest: a Docker service with Gemma 4 E4B baked into the image by Ollama at build time. It includes a /healthz check, answer cache, per-player rate limit, and queue, so one slow CPU model can serve a crowd without falling over. It needs more than 8 GB of RAM, so it runs on a paid plan while my hosting credits last. When I first deployed the model, I hadn't fully checked what the Blueprint's paid plan setting would actually cost. I only noticed after $1 was charged to my card. If you're deploying a model with a Blueprint, make sure to review the plan: field and its associated costs before deploying. Making a browser house feel haunted The early version looked more like a toy than a haunted house. Something was missing: that little uh-oh feeling when a room seems to be watching you. Here is what changed that: - Replaced box-shaped placeholders with CC0 models from Poly Haven—tables, chairs, chandeliers, cabinets, a rocking chair, and a Gothic statue—and reduced the total asset size from 8.7 MB to 4.2 MB so the house loads faster. - Added a cloth shader to the ghosts: ragged hems that twitch and crawl, hollow eye sockets, a cold glow around the edges, and heads that turn before the bodies follow. A small movement, then another… just enough to make you wonder if you really saw it. - Added post-processing: cold blue moonlight against warm candlelight, bloom, vignette, and film grain. As candles go out, the colour drains from the room—slowly, almost imperceptibly, until the warmth is gone. - Improved performance from about 2.8 ms to 1.0 ms per frame on an M3 Pro, measured with synchronous frames. Most of the gain came from letting one light cast shadows instead of three. Get the ending right, and she appears at the piano bench—warm where the other ghosts are cold, a marigold tucked into her hair. For a moment, nothing moves. Then, softly, she turns to look at you. !The ending What it can't do - The harness results above come from a word check. Paraphrased leaks are not counted. - The browser model, E2B, is smaller than the E4B model I measured, and I have not tested it against all 478 questions. - A reply that denies something while repeating the question's premise—for example, “I cannot see how she died in Oaxaca”—can pass the guardrail. - I planned to fine-tune a smaller model to beat the Gemma baseline. The training data and baselines are in the repo, but my hosted training credits ran out, and local training ran out of memory while Gemma was loaded. The model is not fine-tuned. - The browser version needs WebGPU and a 3.1 GB download. Without WebGPU, typed questions receive scripted answers. Why Does Open Innovation Matter? It still runs with the lights off. The whole game, model included, can run on a laptop without internet through Ollama. A ghost story should work on a night that feels made for power cuts and candlelight. What you type stays with you. On the free site, the model runs in your browser. There is no server receiving your questions. Gemma, three environments, one evaluation standard. Gemma E4B and E2B run locally through Ollama, in the browser, and on a server. One harness covers all three; the 478-question test set was run on E4B, and anyone can rerun it because the weights are open. And because the weights are open, anyone can rerun my 478 questions and check the results. The honest trade-off. A large closed model might write livelier ghosts. It would still need the same checks: the leak problem depends on what the model is shown, not just how clever it is. And it would bring an API key, a server, and a bill to a game designed to be free. My Agent Session I built this with Claude Code, and Entire saved the session as a checkpoint, linked to the code by an Entire-Checkpoint: trailer. The full record is in a private repository because it also includes my work setup. Below are three excerpts from it, as Claude Code recorded them: my prompts, the agent's replies, and the tools it ran with their results. My prompts were in Korean, so I added English translations. I deleted API keys and redacted local paths, emails and account names. 1. "Gemma barely makes errors anyway, so I think setting up the harness, the graph engineering and the guardrails should be enough." The baselines showed Gemma 4 E4B leaking another object's secret in 8.6% of 478 test questions, against 49.0% for Qwen3.5-4B. So instead of fine-tuning, the agent built a harness around Gemma: every answer is checked by code against the fact graph before the player sees it. Commits: e869914 "Baseline eval: Qwen3.5-4B and Gemma 4 E4B, prompt only", 6a383e8 "Play the game with local Gemma through a guarded harness; prep MLX LoRA" {% agent_session 1260 %} 2. "No, we have to use the model." After the first deploy I nearly settled for pre-written answers only, to keep hosting free. Instead, the model moved into the player's browser: suggested questions get cached Gemma answers, and typed questions run Gemma 4 E2B on the player's own GPU through WebGPU. Commits: a47af5a "Free static deploy: cached Gemma answers plus Gemma 4 E2B in the browser" {% agent_session 1261 %} 3. "Download Gemma 4 E2B in the preview browser and test whether it actually works." The agent played the game in its own browser with the real model; answers came back in 1–3 s. The test caught a leak the checks had missed: in Korean, the letter said it was never sent ("부치지 못하였소"), a fact only the narrator may tell. The check now matches Korean verb stems, in both the Python guardrail and its JavaScript port, which agreed on all 1,059 answers. Commits: e4fd0ab "Catch Korean conjugations of the unsent-letter fact" {% agent_session 1262 %} The Spirit of It Día de los Muertos is not a day to fear death. It is a day to remember those who left before us and set a place at the table for their return. Marisol is fictional. But being erased from memory is real. Eleanor scratched Marisol's face out of photographs and erased her name from the list. Thomas never managed to send the letter, and every year for the rest of his life, he left Seat 7 empty. But after Thomas died, no one remembered that seat anymore. An empty chair. A scratched-out face. A name erased. And a letter that was never delivered. I wanted to make a ghost story where the most frightening thing is forgetting. And I wanted the AI in this story not to fill the gaps in its memory with invented details. If it doesn't remember, it should say it doesn't know. Prize Categories - Best Use of Gemma: Gemma 4 plays all four objects. Each one is shown only its own facts, and code checks every reply before the player reads it. E4B runs through Ollama on a laptop or the Render server, and E2B runs in the player's browser with Transformers.js on WebGPU, both behind the same guardrail. Alone with the whole story, Gemma leaked another object's secret in 8.6% of 478 test questions; inside the harness, 0 of 478 by word check. - Best Use of Render: one render.yaml Blueprint deploys two services from main. The free static site serves the game while Gemma 4 E2B runs in the player's browser, and the weights come straight from Hugging Face. The Docker service has Gemma 4 E4B baked in, behind a health check, an answer cache, a per-player rate limit and a queue. Every push to main redeploys both. - Best Use of Entire: Entire saved the Claude Code session as a checkpoint in a private repository, linked to the code by an Entire-Checkpoint: trailer. I searched that record to trace each commit in the agent session section back to the prompt behind it, and the three embeds above were cut from it. Credits - 3D models and textures: Poly Haven, CC0 - Title image and Marisol's photograph: generated with ChatGPT image generation (OpenAI) - Marisol's voice on the record: generated with Qwen3-TTS (Apache-2.0); her tune was written for the game - Models: Gemma 4 through Ollama and Transformers.js - Engine: Three.js