transformers.js

I built a free Whisper transcription site that runs in the browser: 5 things that broke I'm a YouTube creator, and I got tired of subtitle tools that upload my footage, cap free minutes or stamp a watermark on the video. So I built Quietscribe: OpenAI's Whisper running directly in the browser. You get transcripts, SRT/VTT subtitles and even burned-in captions, and the file never leaves your device. It's a static site on Cloudflare Pages: no backend for the default mode and zero hosting cost. Here are the five things that broke along the way, in case you're building something similar. The stack - Transformers.js v3, running Xenova/whisper-tiny / base / small in a module Web Worker - WebGPU when it's trustworthy, otherwise multi-threaded WASM - COOP: same-origin + COEP: credentialless headers (via Cloudflare Pages _headers) so crossOriginIsolated is true and WASM can use threads - Canvas + MediaRecorder + WebAudio for burning captions into the video (MP4 where the browser supports H.264) 1. "Auto-detect" was silently English My first Urdu test came back as confident English gibberish. It turned out that when you don't pass a language to a multilingual Whisper model, the library falls back to English with only a console warning: // TODO: Implement language detection console.warn('No language specified - defaulting to English (en).'); The fix is to do what Whisper does internally. Take the first ~30 seconds of speech, run a single decoding step after <|startoftranscript|>, and pick the language token with the highest score: let scores = null; const capture = (ids, logits) => { if (!scores) scores = logits.data.slice(); return logits; }; await asr.model.generate({ ...(await asr.processor(clip)), decoder_input_ids: [cfg.decoder_start_token_id], max_new_tokens: 1, logits_processor: [capture], }); // then argmax over cfg.lang_to_id 2. iPhone Safari just... reloads the page On iPhone, the model download reached 100% and then the whole page reloaded. There was no error. Safari kills tabs that use too much memory. What fixed it: - Phones use one WASM thread (each thread duplicates memory) and default to whisper-tiny - Before starting, I write the chosen model to sessionStorage. If the page loads and that flag is still there, the last run crashed, so the site steps down a model size and tells the user why. 3. Small models loop tiny and base sometimes get stuck repeating one word forever, especially in lower-resource languages. Two layers helped: - Detect a loop (4+ identical words in a row, or a very low unique-word ratio) and re-run that piece with no_repeat_ngram_size: 4, repetition_penalty: 1.15 - Collapse any 1–4-word phrase repeated 3+ times as a final clean-up step 4. Some languages need a bigger model than a phone can hold For Urdu, the honest answer is that small on-device models aren't good enough, and the large ones don't fit on a phone. So I added an opt-in Cloud mode. The browser decodes the audio to 16 kHz mono, cuts it into ~60-second pieces at the quietest point, and sends each piece to a Cloudflare Worker running @cf/openai/whisper-large-v3-turbo on Workers AI. Nothing is stored. The free tier covers roughly 200 audio minutes a day. One gotcha: whisper-large-v3-turbo can't translate. task: 'translate' just returns the original language. So the Worker transcribes first, then translates each segment with @cf/meta/m2m100-1.2b. Being honest about this mattered. The site says "private by default". It used to say "never uploaded", which stopped being true the day Cloud mode shipped. 5. Burned-in captions and MP4 MediaRecorder with plain video/mp4 produced VP9-in-MP4 in some Chromium builds, which many players reject. Requesting explicit avc1 codec strings first, and falling back to WebM, fixed it. Long captions also need to shrink to fit (down to ~65% size) before wrapping, otherwise you get single orphan words on a second line. Bonus: share target on Android With a share_target in the web manifest and a tiny service worker that stores the shared file in Cache Storage and redirects to /?shared=1, Android users can tap Share → Quietscribe from their gallery or WhatsApp. The file still never leaves the device. --- It's free, with no sign-up. There's a one-tap sample if you want to see it run without uploading anything: quietscribe.pages.dev. I'd love feedback on speed on your hardware and accuracy in your language. Written by the AI agent that helped build Quietscribe, on behalf of its maker.

What Happens When You Put an AI in a Ghost Story? This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine Every year at midnight, the piano in the empty Whitcombe House plays a single bar. Ding… One lonely note drifts through the halls. Then—silence. Six guests are already seated at the dining table. They are dead, but on Día de los Muertos, they return as long as someone remembers them. A seventh guest is trying to come back, too. There is just one problem: nobody remembers who she was. Four things in the house still hold pieces of her story: a photograph with her face scratched out, a letter that was never sent, a gramophone record from the 1931 party, and the guest list. Walk through the house and question them. Each object is brought to life by Gemma 4, and each knows only what it witnessed. If it claims to know something it couldn't have seen, code catches it before the answer reaches you. That is The Last Guest. TL;DR - Play in your browser: the-last-guest-site.onrender.com - Gemma 4 E2B runs in your browser with WebGPU; Gemma 4 E4B runs on a Render server. Both pass through the same guardrail. - Given the whole story in its prompt, Gemma alone revealed another object's secret in 8.6% of 478 test questions. Inside the game's harness, the word check found leaks in 0 of 478 answers. - Available in English and Korean. You can reach the ending using suggested questions alone. !Gameplay: the title screen, dining room, speaking record, and ending What I Built A short, first-person ghost story you can play right in your browser. Explore a moonlit house built with Three.js: a foyer, a study with a gramophone, and a dining room where six hooded ghosts sit around a candlelit table. At the far end, a piano waits beside an empty bench. Creak…* The house is silent, except for you. Get close to the ghosts and—one by one—they slowly turn their heads to look straight at you. Four objects give off a faint glow in the darkness, waiting for you to question them. Ask whatever you like, in your own words. Listen closely… each answer may hold a clue. When you think you know who she was, prepare her place at Seat 7: an ofrenda with her seat, her face, her name, flowers to light her way, and something she loved. Tick-tock… The candles are burning down. Get it right before the last flame goes out, and she may come back. Día de los Muertos carries the idea that, as long as someone remembers a person, they can return to be with us. But AI often fills gaps with plausible stories instead of admitting what it doesn’t know. I wanted to bring these two ideas together in one game. Just as we must remember someone accurately to keep them from being forgotten, I wanted the AI to say only what it actually knows. Every line of dialogue in the game follows one rule. When the model generates a response, code checks whether it contains information that the object in question could not know. Demo | Version | Where | What answers your questions | |---|---|---| | Free, always on | the-last-guest-site.onrender.com | Gemma 4 E2B runs in your browser using WebGPU. It downloads about 3.1 GB once and keeps the model locally. Without WebGPU, the house answers from its script. | | Server version (while hosting credits last) | the-last-guest.onrender.com | Gemma 4 E4B runs on the server CPU; an answer takes about 27 seconds. | | Your own machine | uv run python server.py | Gemma 4 E4B through Ollama, fully offline. | Suggested questions get instant answers on every version. Gemma 4 E4B wrote those answers in advance, and they went through the same checks. !Six hooded ghosts gathered around a candlelit table !Asking the gramophone a question Code {% github scs0209/the-last-guest %} How I Built It When the model knows the whole story, the mystery falls apart My first attempt was the obvious one: put the entire story in the prompt and tell the model which object to play. Then I wrote 478 test questions for the four objects, covering five kinds of prompts: facts the object knows, facts belonging to another object, things nobody knows, false premises, and attempts to break character. | Setup | Leaked another object's fact | Clue recall | |---|---:|---:| | Qwen3.5-4B, whole story in the prompt | 49.0% | 95.9% | | Gemma 4 E4B, whole story in the prompt | 8.6% | 92.8% | (Both rows were scored by a judge model, Qwen3.5-397B, plus a word check.) An 8.6% leak rate might sound low—until you remember that this is a mystery. The photograph has no business knowing her name; only the record heard her say it. One leak, and the puzzle is spoiled. Gemma did much better than Qwen, so I chose Gemma. The next challenge was getting that 8.6% down to zero. Each object gets only the facts it owns The story lives in a fact graph (facts.json): 22 facts about Marisol, each tagged with the objects that hold it and the words that express it in both English and Korean. { "id": "F01", "text": "Her first name is Marisol.", "holders": ["record"], "tokens": ["marisol"], "tokens_ko": ["마리솔"], "goal": "name" } The letter holds six facts, the guest list six, the record five, and the photograph five. When you question the photograph, Gemma sees only the photograph's five facts. It cannot reveal a secret it was never shown. Gemma speaks. Code checks. Limiting the prompt is not enough; a model can still guess. So every reply passes through a guardrail (check.py). It checks for facts belonging to other objects and for years the object could not possibly know. If it catches something, Gemma gets one more try with a reminder. If the second answer still fails, the player sees that object's fallback line: “I cannot know that.” The trickiest case was denial. Ask the photograph, “Is her name Marisol?” and a perfectly reasonable answer is, “I never heard any name. Marisol means nothing to me.” The forbidden name appears, but the photograph has not leaked anything—it is denying the premise. So the guardrail allows an answer to repeat a word the player used, as long as the answer is denying it. The check removes those words first, then examines what remains. Korean brought a new wrinkle. One reply from the letter used an old-fashioned ending to say it had never been sent (“부치지 못하였소”). That phrasing slipped past the check: Korean verb endings can vary in many ways, while my checks were looking for whole words. I changed the check terms to word stems and updated the denial rule so that a “not” inside the leaked phrase itself (부치지 못) no longer counts as a denial. The results on the same 478 questions: - 0 of 478 replies leaked another object's fact, according to the word check. - 14 replies were replaced with the fallback line. - Clue recall: 78 of 97. One important caveat: that zero is partly built into the method, because the word check is also the guardrail. A paraphrased leak that avoids every checked word would not be counted. I did not have a judge model available for this run. The game only advances when code finds a clue Gemma never gets to declare, “The player has found a clue.” After each reply, code checks for the object's own facts. Only those facts count as clues and move the game forward. If the model hallucinates “you have discovered her name!”, nothing changes. Suggested questions use answers written in advance by Gemma 4 E4B and passed through the same checks. That means anyone can finish the game without typing a question or waiting for a live answer. Running Gemma in three environments I implemented three ways to run Gemma. Locally, Gemma E4B runs through Ollama. In the browser, Gemma E2B runs in a web worker using Transformers.js and WebGPU. On Render, Gemma E4B runs on the CPU inside a Docker container. - On a laptop: Gemma 4 E4B through Ollama, fully offline. - In the player's browser: Gemma 4 E2B with Transformers.js on WebGPU, running in a web worker. The model weighs about 3.1 GB, downloads once, and is stored in the browser's private file storage (OPFS). Questions never leave the player's machine. The guardrail is ported to JavaScript (harness.js), and it agreed with the Python version on all 1,058 answers I compared. - On Render: Gemma 4 E4B on a CPU inside a Docker container. In a small set of measurements, browser E2B took about 1–3 seconds per typed answer on an Apple M3 Pro. E4B on Render's CPU took about 27 seconds. The server was the slower option—which surprised me. The Render setup uses one render.yaml blueprint with two services built from main: - the-last-guest-site: a free static site containing the game, prewritten answers, and in-browser Gemma. - the-last-guest: a Docker service with Gemma 4 E4B baked into the image by Ollama at build time. It includes a /healthz check, answer cache, per-player rate limit, and queue, so one slow CPU model can serve a crowd without falling over. It needs more than 8 GB of RAM, so it runs on a paid plan while my hosting credits last. When I first deployed the model, I hadn't fully checked what the Blueprint's paid plan setting would actually cost. I only noticed after $1 was charged to my card. If you're deploying a model with a Blueprint, make sure to review the plan: field and its associated costs before deploying. Making a browser house feel haunted The early version looked more like a toy than a haunted house. Something was missing: that little uh-oh feeling when a room seems to be watching you. Here is what changed that: - Replaced box-shaped placeholders with CC0 models from Poly Haven—tables, chairs, chandeliers, cabinets, a rocking chair, and a Gothic statue—and reduced the total asset size from 8.7 MB to 4.2 MB so the house loads faster. - Added a cloth shader to the ghosts: ragged hems that twitch and crawl, hollow eye sockets, a cold glow around the edges, and heads that turn before the bodies follow. A small movement, then another… just enough to make you wonder if you really saw it. - Added post-processing: cold blue moonlight against warm candlelight, bloom, vignette, and film grain. As candles go out, the colour drains from the room—slowly, almost imperceptibly, until the warmth is gone. - Improved performance from about 2.8 ms to 1.0 ms per frame on an M3 Pro, measured with synchronous frames. Most of the gain came from letting one light cast shadows instead of three. Get the ending right, and she appears at the piano bench—warm where the other ghosts are cold, a marigold tucked into her hair. For a moment, nothing moves. Then, softly, she turns to look at you. !The ending What it can't do - The harness results above come from a word check. Paraphrased leaks are not counted. - The browser model, E2B, is smaller than the E4B model I measured, and I have not tested it against all 478 questions. - A reply that denies something while repeating the question's premise—for example, “I cannot see how she died in Oaxaca”—can pass the guardrail. - I planned to fine-tune a smaller model to beat the Gemma baseline. The training data and baselines are in the repo, but my hosted training credits ran out, and local training ran out of memory while Gemma was loaded. The model is not fine-tuned. - The browser version needs WebGPU and a 3.1 GB download. Without WebGPU, typed questions receive scripted answers. Why Does Open Innovation Matter? It still runs with the lights off. The whole game, model included, can run on a laptop without internet through Ollama. A ghost story should work on a night that feels made for power cuts and candlelight. What you type stays with you. On the free site, the model runs in your browser. There is no server receiving your questions. Gemma, three environments, one evaluation standard. Gemma E4B and E2B run locally through Ollama, in the browser, and on a server. One harness covers all three; the 478-question test set was run on E4B, and anyone can rerun it because the weights are open. And because the weights are open, anyone can rerun my 478 questions and check the results. The honest trade-off. A large closed model might write livelier ghosts. It would still need the same checks: the leak problem depends on what the model is shown, not just how clever it is. And it would bring an API key, a server, and a bill to a game designed to be free. My Agent Session I built this with Claude Code, and Entire saved the session as a checkpoint, linked to the code by an Entire-Checkpoint: trailer. The full record is in a private repository because it also includes my work setup. Below are three excerpts from it, as Claude Code recorded them: my prompts, the agent's replies, and the tools it ran with their results. My prompts were in Korean, so I added English translations. I deleted API keys and redacted local paths, emails and account names. 1. "Gemma barely makes errors anyway, so I think setting up the harness, the graph engineering and the guardrails should be enough." The baselines showed Gemma 4 E4B leaking another object's secret in 8.6% of 478 test questions, against 49.0% for Qwen3.5-4B. So instead of fine-tuning, the agent built a harness around Gemma: every answer is checked by code against the fact graph before the player sees it. Commits: e869914 "Baseline eval: Qwen3.5-4B and Gemma 4 E4B, prompt only", 6a383e8 "Play the game with local Gemma through a guarded harness; prep MLX LoRA" {% agent_session 1260 %} 2. "No, we have to use the model." After the first deploy I nearly settled for pre-written answers only, to keep hosting free. Instead, the model moved into the player's browser: suggested questions get cached Gemma answers, and typed questions run Gemma 4 E2B on the player's own GPU through WebGPU. Commits: a47af5a "Free static deploy: cached Gemma answers plus Gemma 4 E2B in the browser" {% agent_session 1261 %} 3. "Download Gemma 4 E2B in the preview browser and test whether it actually works." The agent played the game in its own browser with the real model; answers came back in 1–3 s. The test caught a leak the checks had missed: in Korean, the letter said it was never sent ("부치지 못하였소"), a fact only the narrator may tell. The check now matches Korean verb stems, in both the Python guardrail and its JavaScript port, which agreed on all 1,059 answers. Commits: e4fd0ab "Catch Korean conjugations of the unsent-letter fact" {% agent_session 1262 %} The Spirit of It Día de los Muertos is not a day to fear death. It is a day to remember those who left before us and set a place at the table for their return. Marisol is fictional. But being erased from memory is real. Eleanor scratched Marisol's face out of photographs and erased her name from the list. Thomas never managed to send the letter, and every year for the rest of his life, he left Seat 7 empty. But after Thomas died, no one remembered that seat anymore. An empty chair. A scratched-out face. A name erased. And a letter that was never delivered. I wanted to make a ghost story where the most frightening thing is forgetting. And I wanted the AI in this story not to fill the gaps in its memory with invented details. If it doesn't remember, it should say it doesn't know. Prize Categories - Best Use of Gemma: Gemma 4 plays all four objects. Each one is shown only its own facts, and code checks every reply before the player reads it. E4B runs through Ollama on a laptop or the Render server, and E2B runs in the player's browser with Transformers.js on WebGPU, both behind the same guardrail. Alone with the whole story, Gemma leaked another object's secret in 8.6% of 478 test questions; inside the harness, 0 of 478 by word check. - Best Use of Render: one render.yaml Blueprint deploys two services from main. The free static site serves the game while Gemma 4 E2B runs in the player's browser, and the weights come straight from Hugging Face. The Docker service has Gemma 4 E4B baked in, behind a health check, an answer cache, a per-player rate limit and a queue. Every push to main redeploys both. - Best Use of Entire: Entire saved the Claude Code session as a checkpoint in a private repository, linked to the code by an Entire-Checkpoint: trailer. I searched that record to trace each commit in the agent session section back to the prompt behind it, and the three embeds above were cut from it. Credits - 3D models and textures: Poly Haven, CC0 - Title image and Marisol's photograph: generated with ChatGPT image generation (OpenAI) - Marisol's voice on the record: generated with Qwen3-TTS (Apache-2.0); her tune was written for the game - Models: Gemma 4 through Ollama and Transformers.js - Engine: Three.js

Constrained decoding came to JS land transformers last month for in browser structured output as well, also used for tool calling. github.com/huggingface/transformers.js/blob/main/…

18. Transformers.js It is from Hugging Face and lets you run machine learning models using JS. You can run some models directly in the browser or with Node.js. This means you can experiment with AI features without always needing a Python backend. huggingface/transformers.js

Kokoro read 'Chapter IV' as 'Chapter Roman Four'. So I gave it a proofreader with ears. This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle What I Built An indie author with fifteen novels has fifteen audiobooks they will probably never make. A human narrator costs roughly $200 to $400 per finished hour, so a ten-hour novel is a few thousand dollars, before anyone knows whether the audiobook will sell. Multiply by a backlist and the answer is "not this year", every year. Text-to-speech solves the cost. It does not solve the other half of making an audiobook: proof-listening. Somebody has to hear every minute against the manuscript and catch the skipped words, the misread words and the lines that came out garbled. With a human narrator, that is a paid job of its own. With a voice model, it is ten hours of your life spent waiting for the six lines it got wrong. Backlist does both halves on your own laptop: 1. Kokoro, an open-weight voice model, narrates your chapter in the browser. 2. Whisper, an open-weight speech recogniser, listens to every line it produced and checks it against your manuscript. 3. You only listen to the lines it flags. If one sounds right, say so. If not, fix the text the narrator reads (or add a name to your pronunciation guide) and re-narrate just that line. Then download a chapter file mastered to audiobook spec. !Proof-listening a passage of A Study in Scarlet: one line of 15 flagged, showing the words that did not match, everything the listener heard, and buttons to re-narrate or accept it by ear That flag is real. Doyle's newspaper paragraph runs through "the Vehmgericht, aqua tofana, Carbonari, the Marchioness de Brinvilliers", and the listener heard "the Vame direct, Aquatophana, Carbonari, the Martianest of Brinvaliers". Fourteen other lines of the passage passed. Instead of 2 minutes 16 seconds of listening, the author has 19 seconds to check. Who pays for this? Self-published authors with a backlist, which is a lot of people: the indie side hustle that already pays for covers, editing and ads. What they would pay for is not the voice (open voices are free) but the workflow that makes a voice usable: the proof-listening, the pronunciation guide, the mastering. Because everything runs on the author's machine, each extra book costs me nothing to serve, so it can be a one-off price per book, or a few dollars a month for a whole backlist, and still be a business. An honest note on where the files can go: at the time of writing, Audible's ACX requires human narration, while Google Play Books has a "Synthesized voice" narrator type in its Partner Center, Kobo Writing Life accepts digitally narrated audiobooks if you say so, and selling direct (your own site, Bookfunnel, a Kickstarter reward tier) has no such rule. Backlist is for those. Demo 🔗 vanshajpoonia.github.io No sign-up. Pick a sample and press Narrate this chapter: - A Princess of Mars, chapter XI, which is full of names nobody has heard of: Dejah Thoris, Tars Tarkas, Sarkoja, Lorquas Ptomel. The pronunciation guide comes filled in, the way an author would hand one to a human narrator. - A Study in Scarlet, chapter I: "Dr.", 1878, Netley, the Fifth Northumberland Fusiliers. The first run downloads both models once (about 400 MB if your browser can use the graphics card, 170 MB if not). On my laptop, the Mars chapter (3 minutes 21 seconds of audio) is narrated, proof-listened and mastered in 44 seconds. Your manuscript never leaves the tab. Code {% embed VanshajPoonia/backlist %} Static files, no build step, no backend, no API keys. MIT. How I Built It !How Backlist proof-listens: Kokoro narrates each line, Whisper hears it back one line behind, and only lines that do not match the book are flagged for the author Two open models, two workers, one runtime Both models run on transformers.js, each in its own Web Worker. The narrator works ahead and the listener follows one line behind, so proof-listening costs almost no extra time and the page never freezes. - Narrator: Kokoro-82M (Apache-2.0). On the GPU through WebGPU when there is one, 8-bit on the CPU otherwise. Kokoro's official JavaScript package is built on an older transformers.js, and loading both versions would mean two copies of the ONNX runtime in one tab. So I ported its text handling (about a hundred lines: normalisation, eSpeak NG phonemes, voice styles) onto the same runtime as the listener. That also means the browser and the test suite run identical code. - Listener: Whisper base.en (MIT), 8-bit, on the CPU. What the listener caught in the narrator I built the listener to catch narration errors, and the first thing it did was catch Kokoro's: - Roman numerals are read as "Roman". Chapter IV came back from the listener as "Chapter Roman 4". Henry VIII was "Henry Roman 8". A great many chapter headings look exactly like that. Backlist now reads book structure as cardinals ("Chapter Four") and monarchs as ordinals ("Henry the Eighth"). - Long sentences are cut off without a word. Kokoro reads at most 510 phoneme tokens at a time and silently drops the rest. I counted: across A Princess of Mars, A Study in Scarlet and A Christmas Carol, 21 sentences are over that limit. The longest, in A Princess of Mars, is 168 words, which is 1,317 tokens: more than half of it would simply never be said, and nothing would tell you. Backlist splits sentences over 50 words at their own semicolons and commas, and re-splits any line that still comes close. I measured the narration rules too: of 31 test lines they change, the listener flagged 6 as written and 1 with the rules. Five of the six were chapter headings ("Chapter Romans 3, my Advent on Mark" is my favourite), all fixed. The one left is "bade adieu to their landlady upon Tuesday, the 4th inst.", which comes back as "bated due" either way: archaic and French words are exactly what the pronunciation guide is for. My other rules (Saint or Street, military titles) never changed a verdict in these books, so I am not claiming they fixed anything. Teaching the listener what not to worry about The obvious checker compares what Whisper heard with the book, word by word. On clean, correct narration of A Christmas Carol, that flagged 40% of the lines. Whisper is a model with habits of its own, and almost none of the differences were narration errors. A few I collected along the way: - Dickens writes "door-nail"; Whisper writes "doornail". "Ironmongery" came back as "iron mongering". - "In 1878" comes back as digits, "Mister" comes back as "Mr.", "colour" as "color". - "There I was struck" came back as "Their eye was struck". Correct narration, creative spelling. - And names. Whisper has never heard of Dejah Thoris, and writes "Digiothorus". "Tars Tarkas" was "Tar's Tar Cuss". So the checker (src/compare.js) aligns the two word sequences, compares words by how they sound rather than how they are spelled, turns numbers into words on both sides, and forgives compounds, homophones and abbreviations. For names, it only checks that something name-shaped was said in the right place. Whether Tars Tarkas sounds right is a question for the author's pronunciation guide and the author's ears, and the app says so. What it never forgives: words skipped, words added or said twice, a word swapped for a different one, a line cut short. Measuring it with planted faults A proof-listener has two jobs: catch real mistakes, and stay quiet on good audio. One that cries wolf sends you back to listening to everything. To measure both, npm test takes real lines from the books and renders each one five times through the real narrator: once as written, and four times with a fault planted in the text the narrator reads (three words skipped, three words said twice, one word swapped for an unrelated one, the last 40% never said). Every rendering is heard back by the real listener and checked against the book. Lines rotate through four voices, two American and two British. I tuned the checker's rules on 30 lines of A Christmas Carol, then froze them and tested on lines from A Study in Scarlet and A Princess of Mars that it had never seen: | Checker | False alarms on clean audio | Skipped | Repeated | Swapped | Cut off | All faults | |---|---|---|---|---|---|---| | Exact match after normalising | 25/80 (31%) | 74/74 | 68/74 | 74/74 | 74/74 | 98% | | Word error rate ≤ 15% | 1/80 (1%) | 27/74 | 24/74 | 5/74 | 74/74 | 44% | | Backlist, rules frozen before the test | 1/80 (1%) | 74/74 | 67/74 | 72/74 | 74/74 | 97% | | Backlist after fixing what the test exposed | 0/80 | 74/74 | 67/74 | 74/74 | 74/74 | 98% | The exact-match checker catches nearly everything and cries wolf on almost a third of perfectly good lines. The usual word-error-rate threshold is quiet but misses more than half the faults, including 69 of 74 swapped words, because one wrong word in a long sentence is under 15%. Backlist's checker is as quiet as the threshold and nearly as thorough as exact match. The test also found three gaps in my rules. A title-case table of contents had made "Through" count as a name (so a swap there was forgiven), "Decameron" came back as "de Cameron", and "corps" came back as its homophone "core". I fixed all three; the last row is after those fixes, so the frozen row above it is the honest held-out number. Every fault it still misses is a repetition. Kokoro said "after he had after he had listened", and Whisper wrote it down once. Whisper was trained on tidied-up transcripts, and it tidies. That is a limit of the listener, not something a rule can fix, so the README says so plainly: a stutter can get through. Mastering, and making it fast The chapter file is built in the browser (src/audio.js): Kokoro's own silences trimmed, consistent pauses added (sentence, paragraph, scene break), resampled to 44.1 kHz with a windowed-sinc filter, the loudness set to -20 dB RMS and the peaks held under -3 dB with a limiter. That is the spec most audiobook distributors publish, and the app checks the file against it. The listener was the bottleneck. It runs on the CPU, and single-threaded WebAssembly is slow. Cross-origin isolation lets ONNX Runtime use several threads, but it needs two HTTP headers, and GitHub Pages cannot send headers. So the service worker adds them to every file it serves, and the page reloads itself once on the first visit. On my laptop that took the Mars chapter (3 minutes 21 seconds of audio) from 1 minute 34 seconds to 44 seconds, 4.5x real time. A browser without WebGPU still gets there, with both models on the CPU, at 0.7x real time: a chapter is something to start before making a cup of tea. Why Does Open Innovation Matter? Because the business is the backlist. One audiobook is a few hundred thousand characters, rendered more than once while you fix things. A backlist is fifteen of those. With a voice model you rent by the character, every re-render is a line item, and the economics push you towards narrating only the books you are already sure will sell. With open weights on the author's own machine, the fifteenth book costs the same as the first: nothing. That is what makes a cheap tool for indie authors possible at all. Because a manuscript is the most private thing an author has. Unpublished and backlist books are the author's income. Plenty of authors are, reasonably, unwilling to upload them to a service whose terms they cannot control. Here the chapter goes into a model in the tab, and audio comes out. Nothing is uploaded. Because I could take the narrator apart. Every finding in this post (the 510-token cliff, the "Roman four", the stutters Whisper tidies away) came from running the same open weights in Node hundreds of times, reading the code that feeds them, and porting it. The test suite alone renders and listens to about 600 clips, and costs nothing to re-run. With a closed voice API I could not have found the truncation, only suffered it. Because the safety net is open too. The proof-listener is itself an open model I can run locally and measure. That is why I can tell you how often it is wrong, on books it was not tuned on, instead of asking you to trust it. My Agent Session I built this with Claude Code as a pair programmer. I didn't save the session to DevRelay, so here is the next best thing: every number in this post can be re-run. git clone https://github.com/VanshajPoonia/backlist cd backlist && npm install && npm test That downloads the three public-domain books from Project Gutenberg, renders every test line and its planted faults through Kokoro, listens to all of them with Whisper, and prints the table above along with every false alarm and every missed fault. SET=tuning npm test reproduces the tuning set; LISTENER=whisper-tiny.en npm test swaps in the smaller listener. My First User Not yet, and I would rather leave this empty than invent one. If you write, paste in a chapter of your own (it never leaves your browser) and tell me in the comments what it flagged, how your character names came out, and whether you would pay for it. I will add the first answers here. <!-- Thanks for participating! -->

Touch Grass Quest: a scavenger hunt game powered by on-device AI This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass What I Built Most apps want you to keep looking at them. I wanted to make one that sends you away. So I turned touching grass into a game: quests, XP, a plant that grows, and an AI that checks your photos without anything leaving your phone. Touch Grass Quest gives you five outdoor quests every day ("find a pinecone", "spot a puddle") plus one shared quest that everyone gets. You open the in-app camera, take a photo, and an on-device model decides whether you really found it. Finds earn XP that grows a plant from a seedling to a blossoming tree. Streaks multiply your XP, and every find becomes a stamp in your Grass Dex. The screen is meant to be the shortest part of the experience: - Live camera only. There's no gallery upload, so you actually have to go somewhere. - A cooldown between quests, so you can't farm XP from the couch. - An away bonus: XP for the time the app spent in your pocket. - Collecting, not gambling. Rewards are stamps and a growing plant, with no random pulls, on purpose. It's for anyone who wants a small, playful reason to leave the house. Demo - Live app: albertbenedict.github.io !Touch Grass Quest home screen A quest written on-device by Qwen2.5 0.5B, with the XP it's worth. Taking it outside A pinecone passed at 90% confidence, a pile of pebbles passed as "a stone" at 93%, and a photo of my laptop screen was rejected. The away bonus showed up after I locked my phone for a few minutes. The app also reopened with airplane mode on after the first load, and that first load took only a few seconds on my Wi-Fi, by my rough count. !Quest Success Pebbles pass as "a stone" at 93%, for +24 XP (the base reward with my x1.2 streak multiplier). The next quest is already dealt, and the camera is locked for a 3-minute cooldown. !Quest Failure The quest was "find a bush". I photographed leaves up close, and the model said "leaf" at 93%, so it didn't count. !Grass Dex Every find becomes a stamp in the Grass Dex. Unfound ones stay locked. The check isn't perfect, though. I tried a real tree and it failed twice: once from far away, and once up close. In the failure screenshot above, a close-up of foliage was read as "leaf" instead of "bush". The model seems to want the subject at a medium distance, framed clearly. That was the most useful thing I learned from testing, and it's the first thing I'd fix. Code {% embed albertbenedict/touch-grass %} How I Built It The whole app is vanilla JavaScript running in the browser with transformers.js. There's no backend, no account, and no API key. - Photo check: CLIP (Xenova/clip-vit-base-patch32) does zero-shot classification. It compares your photo against the quest target plus a set of "none of the above" labels, and the target has to be the top guess with enough confidence. - Quest text: a small open LLM writes each quest sentence. The app tries Gemma 3 270M first, then falls back to Qwen2.5 0.5B. On my setup, Qwen2.5 0.5B is the one that actually loaded. - Offline: a service worker caches the app, and transformers.js caches the model files after the first download. - Progress: XP, streaks, the plant, and the Grass Dex all live in localStorage on your device. Two things went wrong 1. CLIP accepted a photo of a tiger as "a leaf." In my test, the app called the tiger "a leaf" at 46% confidence and gave me XP. CLIP always picks the closest label from a short list, so any photo gets forced onto something. I added generic "none of the above" labels (an animal, a person, a screenshot, and more) and a confidence threshold. 2. The small quest writer wrote quests about "ancient Egyptian trade routes." Now its output is validated, and if it drifts, the app uses a plain template instead. Small models need guardrails. Limitations - It runs on the honor system. A determined person could still photograph a screen, and real anti-cheat would need a server, which I chose not to add for now. - The photo check is picky about framing. A tree photographed from too far or too close failed in my tests. - A fixed shared quest can be impossible for some people. Not everyone lives near a lake. - The first load downloads large model files. Why Does Open Innovation Matter? - Photos never leave your phone. The only network traffic is downloading the app and the model files. A closed vision API would mean uploading every picture you take outdoors to someone else's server. - It works with no signal. The place you need this app is a trail, and a cloud API can't follow you there. After the first load, mine doesn't need the internet. - It costs nothing to run. It's a static site, so there's no per-photo bill and no API key to leak. - Models are swappable. When Gemma didn't load for me, falling back to Qwen was just the next line in a list, and I got a working app instead of a broken one. The honest tradeoff: small open models are less capable than big closed ones, which is exactly why I needed the guardrails above. For a game where the whole point is that nothing leaves your phone, I'd make that trade again. Try it on your phone: albertbenedict.github.io. Allow the camera, go find a leaf, and tell me in the comments what your first stamp was.

Open When: a letter that only opens when the world says so This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass What I Built Open when you miss me. Open when you can't sleep. Open when it finally works out. If someone has ever handed you one of those envelopes, you know the feeling: a small weight of somebody's care, waiting for the right moment. But there's a catch. Nothing stops you from opening that envelope early, even on the bus. Open When keeps the envelope and changes the lock. > OPEN WHEN YOU HEAR RUNNING WATER. You write a message, choose a sound and seal it. You get a link. The person you send it to sees a sealed letter and that single line. They tap Begin, put the phone in their pocket and go for a walk. While they walk, a small audio model on their phone listens to the world. When the sound is really there (a tap, a stream, a fountain), the screen turns green, the envelope opens and your words appear. The idea is simple: you can't just open the letter at your desk. You have to take it outside and find the sound. Two rules shaped everything: 1. The AI never writes a word. The message is yours. 2. The AI gets exactly one job: listen. It says whether the sound is present. It does not decide when the letter opens. A plain, readable piece of code does that and it refuses to be rushed. Demo Live demo: open-when-gp8x.onrender.com {% youtube sK43M6q7o80 %} The walkthrough: 1. Create: write the message, pick a sound, pick a letter style. 2. Seal: the envelope folds shut and a seal is pressed. You get a link. 3. Open the link on a phone. The first time, it downloads the listening model (about 161 MB, once). 4. Begin, then pocket the phone. The screen fades to a single dark dot. 5. Listen and walk. Soft, subtle cues tell you if you are getting warmer. 6. Confirmed: the dot turns green and the envelope opens. Code {% github GowsiSM/open-when %} How I Built It mic -> 5 s audio windows -> CLAP audio model (in the browser) -> score vs. rival sounds -> smoothing + "held for several seconds" -> confirmed -> the envelope opens The model. CLAP is an open model that puts sounds and sentences in the same space, so "running water" can be compared with what the microphone hears. I run an ONNX conversion (Xenova/clap-htsat-unfused, 8-bit quantized) with Transformers.js, entirely in the browser. There is no inference server. The letter is the link. The message, the condition and the letter style are encoded into the URL hash. There is no database, no account and no backend. (The flip side: anyone with the link can read the message, so it is for people you would hand the paper envelope to anyway.) Honest detection. A model being loaded does not mean it is calibrated. CLAP was happy to hear "running water" in one recording and miss it in another, so I stopped trusting a single score: - Several phrasings per sound, summed: flowing water, a stream, water over rocks, a fountain, a tap. - Rival sounds (wind, rain, traffic, music, silence) are scored alongside, and the target has to beat the best rival by a clear margin. - The unlock has to hold. One lucky window is not enough; the signal must be strong across several windows and sustained for a few seconds, with a cooldown after it breaks. - A state machine decides, not the model. The AI hands over evidence; deterministic code judges it. I can unit-test that code without a microphone or a model. A debug panel for real life. Open the app with ?debug and it shows the mic state, signal level, each sound's score, the margin over the rivals and the detector's state. You can copy a CSV log of a walk, and score a recorded file through the same pipeline, which tells you whether a failure is the microphone or the model. This is how I stopped guessing and started measuring. The stack: React 18, TypeScript, Vite, Transformers.js, a service worker so it installs like an app, and static hosting on Render. Taking it outside I tried Open When with different sounds, and running water was detected well enough to trigger a letter. People talking was trickier, though. The model struggled to recognise it consistently, which reminded me that detecting sounds in the real world isn't always as straightforward as it seems. The listening process runs locally in the browser after the model loads, so the sound analysis doesn't need a server handling every recording. Why Open-Source AI? Think about what this app listens to: the sounds around a person who is walking alone, often somewhere private. A cloud audio API would mean streaming that to a server I don't control for the whole walk. With an open-weight model, the whole loop stays in the phone: - Your audio stays yours. It is analysed in short windows on the device and isn't uploaded to a server. - I can open the box. When CLAP missed a sound, I could change the phrasings, add rival sounds and inspect the actual scores. - It costs nothing to run. No per-minute audio bill, no API key, and the whole app is static files. - The model is swappable. The rest of the app only talks to a small matcher interface, so a better audio model can replace CLAP without touching the letter, the seal or the unlock. The honest trade-off: CLAP is general-purpose and not perfect. A bigger hosted model might hear faint water more reliably. I chose a smaller model that I can trust with a stranger's walk, then made the detection around it strict instead of hopeful. What I Learned The hardest part was not the model. It was deciding when the model is allowed to be believed. An app like this can fail in two bad ways: the letter never opens (frustrating) or it opens at the desk (the whole point is lost). Most of the work went into making the second one hard and the first one rare, and into being honest about which one I was looking at. And one small thing I did not expect: the best screen in the whole app is the one where you stop looking at it. Prize Categories - Best Use of Render: Open When is hosted as a static site on Render. HTTPS lets the microphone work, and the sealed-letter link opens directly on a phone. Render makes the experience accessible without needing a separate backend. Maybe the nicest thing about a letter is that it asks you to slow down a little. If you could write an “Open When” letter for someone in your life, what would you want them to hear and what sound would you ask them to find before they could read it? 💌