An AI agent read a Japanese novel as 208,953 replacement characters
OpenClaw is an open-source AI agent. When a message contains a link, its link-understanding step fetches the page and hands the text to the model, so the agent can answer questions about it. I pointed it at Natsume Sōseki's Kokoro on Aozora Bunko, and the model received 208,953 copies of U+FFFD, the replacement character. The Japanese text was gone.
The bug
Aozora Bunko serves its texts as Shift_JIS. The link fetcher didn't care what the page declared and did this:
const content = new TextDecoder().decode(buffer).trim();
new TextDecoder() with no argument is a UTF-8 decoder. Shift_JIS bytes are not valid UTF-8, so almost every character became �. Any page served in a legacy encoding (Shift_JIS, EUC-JP, GBK, Big5) went through the same thing.
Nothing failed loudly, which is the annoying part. The model got a wall of replacement characters, and the replies came back garbled too.
The decoder was already there
It turned out web_fetch, the agent's general web tool, already had a decoder that does this properly. It checks, in order:
1. A byte order mark (UTF-8, UTF-16LE, UTF-16BE)
2. The charset parameter of the Content-Type header
3. For HTML and XML responses without a header charset: an <?xml encoding="..."> declaration, <meta charset>, or <meta http-equiv="Content-Type">
If the label is one TextDecoder doesn't know, it falls back to UTF-8 instead of throwing.
So the fix is mostly an export:
// src/agents/tools/web-shared.ts
export function decodeResponseBytes(res: Response, bytes: Uint8Array, truncated = false): string {
// src/link-understanding/apply.ts
import { decodeResponseBytes } from "../agents/tools/web-shared.js";
const content = decodeResponseBytes(response, buffer).trim();
Writing a second decoder would have been easy. Keeping two of them in sync would not.
Why the first attempt stalled
An earlier PR for the same bug used a strict decoder. That fixes Shift_JIS pages but breaks pages that declare no charset at all, and plenty of pages don't declare one. Reusing the web_fetch decoder keeps those on UTF-8, exactly where they were before.
The test
The regression test serves a page from a local HTTP server with this header:
content-type: text/plain; charset=Shift_JIS
and the body [147, 250, 150, 123, 140, 234], which is 日本語 ("Japanese") in Shift_JIS. The test expects the agent to see 日本語. On main it fails, with the fix it passes, and nothing else in the suite changes.
Proving it on real pages
Six bytes in a unit test is a start, not proof. I ran the change through the production link-understanding path, with the strict SSRF guard on, against live public pages:
| Page | Replacement characters before | After |
|---|---|---|
| Kokoro on Aozora Bunko | 208,953 | 0 |
| 阿部寛のホームページ | 13 | 0 |
Then the maintainer, obviyus, re-ran it through a built Gateway: a real session, a real chat message, and a model fixture that records exactly what the model receives. On main the Shift_JIS turns came back garbled. With the fix the model got 阿部寛のホームページ, and the UTF-8 control page stayed the same. It merged as openclaw/openclaw#188766.
If your agent fetches web pages
Check how it turns bytes into text. Both new TextDecoder() and Response.text() decode as UTF-8 no matter what the server says, so an agent built on either one has this bug on legacy-encoded pages. You don't need a library to fix it: read the BOM, the Content-Type charset and the document's own declaration, then fall back to UTF-8.
Mentions
An AI agent read a Japanese novel as 208,953 replacement characters OpenClaw is an open-source AI agent. When a message contains a link, its link-understanding step fetches the page and hands the text to the model, so the agent can answer questions about it. I pointed it at Natsume Sōseki's Kokoro on Aozora Bunko, and the model received 208,953 copies of U+FFFD, the replacement character. The Japanese text was gone. The bug Aozora Bunko serves its texts as Shift_JIS. The link fetcher didn't care what the page declared and did this: const content = new TextDecoder().decode(buffer).trim(); new TextDecoder() with no argument is a UTF-8 decoder. Shift_JIS bytes are not valid UTF-8, so almost every character became �. Any page served in a legacy encoding (Shift_JIS, EUC-JP, GBK, Big5) went through the same thing. Nothing failed loudly, which is the annoying part. The model got a wall of replacement characters, and the replies came back garbled too. The decoder was already there It turned out web_fetch, the agent's general web tool, already had a decoder that does this properly. It checks, in order: 1. A byte order mark (UTF-8, UTF-16LE, UTF-16BE) 2. The charset parameter of the Content-Type header 3. For HTML and XML responses without a header charset: an <?xml encoding="..."> declaration, <meta charset>, or <meta http-equiv="Content-Type"> If the label is one TextDecoder doesn't know, it falls back to UTF-8 instead of throwing. So the fix is mostly an export: // src/agents/tools/web-shared.ts export function decodeResponseBytes(res: Response, bytes: Uint8Array, truncated = false): string { // src/link-understanding/apply.ts import { decodeResponseBytes } from "../agents/tools/web-shared.js"; const content = decodeResponseBytes(response, buffer).trim(); Writing a second decoder would have been easy. Keeping two of them in sync would not. Why the first attempt stalled An earlier PR for the same bug used a strict decoder. That fixes Shift_JIS pages but breaks pages that declare no charset at all, and plenty of pages don't declare one. Reusing the web_fetch decoder keeps those on UTF-8, exactly where they were before. The test The regression test serves a page from a local HTTP server with this header: content-type: text/plain; charset=Shift_JIS and the body [147, 250, 150, 123, 140, 234], which is 日本語 ("Japanese") in Shift_JIS. The test expects the agent to see 日本語. On main it fails, with the fix it passes, and nothing else in the suite changes. Proving it on real pages Six bytes in a unit test is a start, not proof. I ran the change through the production link-understanding path, with the strict SSRF guard on, against live public pages: | Page | Replacement characters before | After | |---|---|---| | Kokoro on Aozora Bunko | 208,953 | 0 | | 阿部寛のホームページ | 13 | 0 | Then the maintainer, obviyus, re-ran it through a built Gateway: a real session, a real chat message, and a model fixture that records exactly what the model receives. On main the Shift_JIS turns came back garbled. With the fix the model got 阿部寛のホームページ, and the UTF-8 control page stayed the same. It merged as openclaw/openclaw#188766. If your agent fetches web pages Check how it turns bytes into text. Both new TextDecoder() and Response.text() decode as UTF-8 no matter what the server says, so an agent built on either one has this bug on legacy-encoded pages. You don't need a library to fix it: read the BOM, the Content-Type charset and the document's own declaration, then fall back to UTF-8.