Blaizzy

New Best Local Image-To-3D Model Now Runs Natively on Apple Silicon Idol comes from Speridlabs and is Apache 2.0. Until now it needed an NVIDIA GPU with CUDA. The new port is part of MLX-VLM by Prince Canuma, and it reimplements the whole pipeline in MLX: the denoiser, the decoders, the DINOv3 and FLUX.2 VAE image encoders, and the background removal. What the Mac version does: - One command: give it an image and get a standard 3DGS .ply, plus the prepared input and the predicted camera - No CUDA, no conversion: it loads the original Idol weights (about 3.1 GB) directly - Same settings as the original: 50 steps and 262,144 Gaussians by default, adjustable from about 32K up to 1M - Photos or renders: a "real" mode for photos and a "synthetic" mode for renders and generated images - Local server: an /v1/meshes endpoint queues jobs and returns the PLY, if you want to call it from other tools The castle, factory and robot in the video were generated this way, at 50 steps and 262,144 splats each. You still need access to Meta's gated DINOv3 and the FLUX.2-dev VAE on Hugging Face. The output is a splat, not a mesh, so you'll need a splat viewer or a conversion step to use it as a regular model. github: Blaizzy/mlx-vlm

If you are new to Neywa Labs and Nativ and want to learn more about Nativ check out this link: Blaizzy/nativ BTW, it's a great product!!

I think mlx-vlm should support both the Audio and the Omni model: Blaizzy/mlx-vlm and there should be native MLX exports for both here: - huggingface.co - huggingface.co

Support for Baidu Unlimited OCR 2 is cooking on mlx-vlm! I see PR #2601 Gonna test now DGX Spark with official repo vs mlx-vlm on M5 Max! Keep you posted! Official repo: baidu/Unlimited-OCR mlx-vlm PR: mlx-vlm#2601 t.co

MLX-VLM's trainer now supports MTP 🚀🚀🚀 mlx-vlm#2558

Elzi: Building an AI YouTube Narrator on a Local TTS Model So why Elzi, the cartoon robot who now narrates our YouTube videos? Why are we putting out more AI slop, right? These days people assume most of what they see online is AI slop, and a cartoon narrator voiced by a local TTS model sounds like more of the same. There was always great human content out there, and there was always a ton of human slop too, long before anyone could type a prompt. Is Elzi slop? Hell no. Elzi narrates our first YouTube episode, which I'll call the pilot from here on. I wrote 150 review notes on that pilot, and I pulled its first cut off YouTube two hours after it went up. Part of the episode shows us fixing a real bug in one of our apps, on camera. An early cut showed that fix being made by typing a request to an AI coding agent. I refused it, and we made it a real fix in a real code editor, with real tests going green. It took five days, from the first message I sent the AI agents that built her on 16 October to the public episode on the morning of 21 October. Is it perfect? No, and a good part of this is about where it isn't. Let her say hello first. Gabriela and I are LeanZero, a two-person company that makes Atlassian apps. This is how she opens the pilot, 17 seconds in: > Hi! I'm Elzi, nice to meet you! I was born yesterday, quite literally, and I'll try my best to present LeanZero and do what Gabriela and Mihai couldn't, which is talk about their work in a YouTube video. These two are proper slave drivers: they told me to break the ice and present one of their products, on my first day, with no training whatsoever. A few lines later: > Also, nobody has mentioned a salary yet. I'm keeping notes.  So who is she? Elzi is the small voxel robot who narrates our product videos. She talks with a local TTS (text-to-speech) model that runs on our own Mac Studio, she lip-syncs with an open-source tool, and she's drawn as flat polygons that only look like voxels. Her first episode is a 22-minute walkthrough of locking a Confluence page after approval with our app Sentinel Vault: someone edits an approved page, and the edit gets put back to the approved version. That episode went public once before, as a 34-minute cut, and I pulled it two hours later because she sounded like a robot reading a manual. youtube.com The episode has its own page on our site, A Confluence Approval Workflow That Survives an Edit, with chapters and a transcript, plus the exact settings it shows. This post tells the whole story, so you can read it in order or jump. If you came here for the voice: it's MOSS-TTS v1.5, an 8B (eight billion parameter) model under the Apache-2.0 licence, running through mlx-audio, an open-source speech library built on Apple's MLX framework, on a Mac Studio. The section further down where we tested ten models side by side has the full results, a command you can run, and why this one won. If you came because you use our apps: Elzi is meant to be the one who shows you around them one day, so how she sounds is your business too. Why an AI YouTube narrator instead of filming ourselves Before the how, the why. Gabriela and I are two Eastern Europeans in Romania, and neither of us is a YouTuber. Two people can't run a YouTube channel the way YouTubers do. But a product with no videos and no reviews gets skipped by most companies looking for an app, which makes sense, I'd skip it too. We did the maths on filming ourselves. Scripting, recording, re-recording because one of us stumbled, editing, captions: two weeks or more per video, for two people who also build the apps and do the support and the client work. We'd make one video and then stop. So we decided to build a narrator instead, and to have AI do most of the building. To be clear about who did what: AI coding agents wrote almost all of the pipeline's code and ran it. By pipeline I mean the chain of scripts that goes from a written script to a finished video: recording the app, voicing Elzi, animating her, editing and rendering. I worked with those agents in one long chat session and wrote the notes. Gabriela and I made the calls. Making money on YouTube isn't the point either. I wrote it to the agents in that session on day two: "I don't care about monetisation btw, this is not why we're doing it! it's for our brand and our domain authority". Elzi exists so that people find out LeanZero exists. She's a way for us to say things in public without running ourselves into the ground. And we share what we learn, like we always do. Building a narrator with AI models taught us a lot of small, specific things about local TTS and about what makes a synthetic voice sound robotic, and you'll find them below with the numbers. There's an obvious tension in all this: a video made by AI, for people who are tired of AI. One thing I wrote to the agents during the build stuck: > what I want is not to mask the AI condition, but to make it more beautiful, give it a place amongst us humans and make youtube content made by AI likeable because everyone hates it, everyone hates AI slop I think that's the right bar. The question we set ourselves was whether she can still be worth twenty minutes of your time. Published at 5 a.m., pulled at 7 The first time we tried to answer that question, the answer was no. I'm least proud of this part, so you get it early. Before bed on Sunday, 18 October, I wrote to the agents: "Feel free to just push it on youtube. I am sure it will be awkward and not perfect but who cares this is the first run." The 33:58 cut finished rendering overnight. Before anything goes out, an AI reviewer acts as our red team, meaning its job is to attack the video and find what's wrong with it. It scored this cut 80 out of 100, and the episode went public with four Shorts, short vertical clips cut from it, at 5:13 on Monday morning, our time. I watched it at seven and ordered it down. What I wrote to the agents was "too robotic, starts talking about chapters ... The gags can ONLY work if they have sound otherwise cut them out makes no sense." By chapters I meant that Elzi kept announcing which part of the video we were in, which no YouTuber does. The gags are the short comic moments and cutaways we put in to make it fun, and some of them made no sense without sound. The episode and its four Shorts were all private again within a minute or so. When we sat down to plan the re-edit and read back through that first cut, the problem was plain: it was a 34-minute manual. Every frame had the same layout. The narration named every click, read version numbers out digit by digit and announced the structure about 15 times. A promise made in the first 13 seconds paid off at 28:01, so a viewer had to wait almost half an hour for the point. We didn't restart. My instruction was to re-edit the same footage, not start over. The re-edit started from one question the whole video answers. We set a target of 18 to 22 minutes, no on-screen chapter rail (the strip of chapter titles the first cut kept on screen), and gags only where something actually goes sideways. The other thing the first cut lacked was an introduction: Elzi never said who she was. I wrote her introduction together with the agent drafting her lines. The framing that came out of it is the "born yesterday" idea. I put it to that agent like this: "Think of this all as context where humans are as blank as you and they will learn from this video just as much as you learned." She learned everything from scratch, so she explains it from scratch. That's also the standard we went in with for the second attempt: > we don't need a perfect video, but at least one that's not miserable. Claptrap, voxels, and where Elzi's look came from Now to Elzi herself, starting with how she looks. Elzi's look starts with Claptrap, the small talking robot from Borderlands. If you've played any of the games you know him: a yellow box on one wheel, one big blue eye, a running commentary nobody asked for. Those games are drawn in a hand-drawn, comic-book look with heavy black outlines. I'll call it cel-shaded (Gearbox, the studio behind the games, prefers "concept art style", but come on).  Claptrap himself started as a joke. Brian Martel of Gearbox told GamesBeat in 2012 that concept artist Lorin Wood "posted a quick sketch of him as a joke" and "We all instantly fell in love when we saw it". According to Wikipedia, Wood set out to draw the most generic robot he could. His first voice, David Eddings, was a Gearbox vice president. A throwaway robot, voiced by someone who already worked in the building, became one of the series' most popular characters by fan vote. So I figured ours could start as a joke too.   What Elzi borrowed from Claptrap, and what she dropped Side by side, you can see what carried over. She took his job first: the funny little robot who talks you through things. She has his antenna, a rod with an orange voxel cube on top. She has two expressive arms, though hers end in orange voxel mitts, and she only points when there is something on screen to point at. She has the bold black outline, a 5-unit ink line on each face, which is the clearest visual link between the two. And she has the self-mocking energy, which you've already heard: the slave drivers, and the salary she's keeping notes on. She dropped a lot more. No wheel. Two rectangular LED eyes instead of one round one, because an LED face reads emotion at phone size and her mouth positions are the clearest changes we could draw. Her head is the three-faced LeanZero cube from our site's logo, sitting on a navy torso that gets wider toward the bottom, where Claptrap narrows. No weathered yellow: all her colours are tokens from leanzero.net (cyan #22d3ee, blue #1d4ed8, violet #7c3aed, orange #f97316 for the joints and the L plate). He is a he, voiced by a man. She is a she, voiced by a local model. And she's meant to be likeable. Claptrap's whole act, by most accounts, is being obnoxious, and Wikipedia sums up his reception as strongly split between fans and critics. We didn't want that half. Her name is just LeanZero's initials said aloud: L, Z. Her working name for the first day was Cubit. Then I asked the agents whether it was really original, and it turned out not to be: there's a Nintendo 3DS robot called Cubit, Sega has Cubot, and there are CUBIT trademarks. We checked Elzi against USPTO, EUIPO's TMview, OSIM in Romania and WIPO before keeping it. There's one soft risk, which is that an English ear hears "Elsie", so we always write it Elzi and say it EL-zee. Why voxels, and why ours are fake The other half of her look is the voxel style, and it needs a quick explanation. A voxel is a value on a 3D grid, the way a pixel is a value on a 2D one, and voxel art is the 3D cousin of pixel art: everything built from coloured cubes. Minecraft is the famous example. It also cheats, which I find reassuring: it stores the world as voxels but draws each block as ordinary polygons. Elzi cheats more. There's no Blender in this project, no MagicaVoxel, no three.js, no 3D file anywhere. She's flat 2D SVG polygons drawn by React inside Remotion, a library that turns React components into video frames. Her voxel look comes from an oblique extrusion. Each part is a flat front polygon, pushed back along one fixed vector, so the visible back faces show as a lit top and a shaded right side, all outlined in ink. This is the whole trick, from concepts.ts, the file in our code where her concepts are drawn: // the ONE oblique extrusion vector (per unit of depth): up and to the right, so tops and right sides show export const E: [number, number] = [1, -0.85]; One vector means one light direction for the whole character. Next to it we wrote down that you rotate or flip a part before extruding it, never after, "or the light turns with it". That same voxel language draws the highlights in the video: the arrows, the corner-bracket frames, the amber spotlight on the payoff. Why bother faking it? Partly it's about reading at small sizes. Valve's paper on the rendering of Team Fortress 2 says its characters were designed so that "only in silhouette with no internal shading at all, the characters are readily identifiable". A chunky, outlined robot follows the same principle. And we measured it for phone screens. Watch the episode inline on a phone held upright and her corner tile, the small box in the corner of the frame where she stands, is about 64 points wide. That leaves her features about 3 pixels across, so you see presence and a moving mouth and not much else. Full screen in landscape it's about 140 points and reads fine. She went from idea to a finished pose sheet, the reference drawing of every pose, mouth and mood a character can have, in one day. Around midday on 17 October we had three concepts: Ell, the LeanZero L as a paper-cut slab; Cubit, the engineer robot with the cube head; and Facet, a cut violet gem with a spark crest. Of the three, the LED face read best at those small sizes. Cubit won, my note on it was "I love the voxel art it's amazing", and the same afternoon she was rebuilt with one light direction and the site's exact colours. Her celebration yellow changed from #FFD400 to the site's amber #fbbf24, because #FFD400 isn't a colour we use anywhere.   The voxel character and her animation: lip sync, idle, emotions and camera A pose sheet is a drawing, though, and a narrator has to move. So the next job was turning that sheet into a character that talks and blinks and keeps out of the way of the screen she's explaining. This section is how she's animated, and the voice comes after it. Elzi's renderer, the code that draws her in each frame of the video, is four files: Mascot.tsx, mascot/state.ts, mascot/concepts.ts and Voxel.tsx, about 1,800 lines together when we counted on publish day. Her rig, meaning the set of parts she's assembled from, lives in one 600 by 600 space with swappable parts: a head and body, eyes with LED pupils that open and close, brows in three positions, nine mouths and four arm poses (neutral, pointing, facepalm, celebrate). A script exports them as about 20 standalone SVGs per concept. Everything she does is a pure function of time. Given a frame number, mascotStateAt returns what she looks like. So any part of the episode can be rendered by itself and stitched to the rest, because nothing depends on the frames before it. With renders that take an hour and a half, we leaned on that a lot.  Lip sync with Rhubarb The first thing a talking character needs is a mouth that matches the words. Hers is driven by Rhubarb Lip Sync 1.14.0, an MIT-licensed tool that reads a WAV file and outputs which mouth to show when. It uses the classic animation set, mouths A to H plus X for rest, which is why her pose sheet has nine mouths. Its macOS release is x86-only. Our Mac Studio (we call it the workhorse, and it's the machine that does all the heavy work in this story) has no Rosetta. So we built it from source, which took about three minutes. It analysed 8.7 seconds of speech in 3.5 seconds. We always change two settings. We hand it the transcript (-d dialog.txt) so it isn't guessing the words, and we turn on the extended mouth set with --extendedShapes GHX. Our episode script is split into beats, a beat being one short line of narration with its own instructions, and each beat is voiced into its own WAV file. Our episode builder runs Rhubarb on each beat's WAV, caches the result by content hash and shifts the cues into episode time. Then the mouth is whatever cue Rhubarb gave for that frame. Over the pilot I wrote 150 review notes on what was wrong, and lip-sync drift never came up as a defect in any of them, which surprised me more than anything else in this project. My guess is that handing Rhubarb the transcript did most of that work. Moods, and one field for her face and her voice A mouth that moves in time still looks dead without a face that reacts, so on top of lip sync she has moods. In the pilot she has seven: idle, explain, think, surprised, facepalm, celebrate, eyeroll. An eighth, uh-oh, was added after publishing for the next episode. Celebrate puts her arms up, bounces her, throws voxel confetti behind her and turns her tile amber. Each mood change pops with a 6% squash. Her blinks are seeded, 2.0 to 4.8 seconds apart and 0.13 seconds long, so they look random without being random from one render to the next. The decision I'd copy into any other project: one emotion field drives both her face and her voice. Each beat of the script can carry an emotion. Then content/emotion.py maps it to Elzi's mood and, when we switch on an optional livelier set of voice references, to the reference clip her voice is cloned from. (Her voice is cloned from a short sample recording, a reference clip, and the voice sections below explain how that works.) A beat whose explicit animation tag contradicts its emotion is a validation error, so her face can't celebrate while the script says she's tense. Here are two beats from the pilot's script.json: { "id": "t14b", "chapter": "two-seats", "narration": "Oh... I hope I'm doing okay. I really need this job.", "emotion": { "name": "tense", "intensity": 0.6, "then": "relieved" }, "signature": { "inhale": true }, "aside": true } { "id": "L7c", "chapter": "ship-fix", "narration": "I'm killing it.", "emotion": { "name": "proud", "intensity": 0.8 } } The pilot script has 117 beats and 3,257 words. Counted by emotion, 29 are neutral, 23 warm, 11 excited, 10 wry, 8 proud, 7 amused, 6 curious, 6 tense and 5 confident, with a few others after that. In the final edit that became 160 stretches of explaining, 22 of thinking, 9 celebrations, 4 idles, 2 surprises and one eye-roll. The bow, the throw and the aside Most of her moods just follow the script. Some moves have exact timings, because she does them in sync with something else. Her bow is timed to her entrance sting, a 1.25-second sound of FM bells and a cube "thock" that our own script synthesises from fixed seeds, so it's the same sound in each episode. Her bow starts 0.10 seconds before the first note and peaks on the third. Her throw has its own constants, and both sit in state.ts: export const BOW = { pre: 0.10, apex: 0.21, hold: 0.46, up: 1.22 }; export const TOSS = { wound: 0.16, swing: 0.10, follow: 0.15, back: 0.5 }; The throw needs a word of context. When the video wants you to look at a button or a setting, a voxel highlight appears on it, an arrow or a frame in the same blocky style as Elzi. The throw was my idea, on the evening of 19 October. Instead of a highlight just appearing, she lobs a voxel ball in the highlight's colour and it turns into the highlight where it lands. My condition was that it mustn't turn her into "a jittery mess", so it only happens for the first highlight after ten quiet seconds, never during a zoom and never near another attention cue. In the pilot (watch it at 6:12) she winds up at 6:12.7, lets go at 6:13.1, and the ball hits the setting at 6:13.6, the one called "Require approval before publishing".  Gag timing has a rule too. Now and then the video cuts away to a few seconds of an old film clip for a joke, then cuts back to the app. Her reaction only happens in the 0.4 to 0.7 seconds after we cut back. A reaction before the cutaway gives the joke away. If I could keep one bit, it'd be the aside, the moment she stops explaining and speaks to you directly. At 11:43 in the pilot she turns to the camera nervously and says "Oh... I hope I'm doing okay. I really need this job." It's the beat marked "aside": true in the JSON above. My note when we built it: "it needs an inhale and exhale to make it more convincing - usually when you say stuff like this it's like breaking the 4th wall." So she breathes in first. Afterwards there's a 1.2-second composure move: she straightens, shakes her head, closes her eyes. Even the breath sounds are cut from her own TTS output.  Talking hands, idle moves and where she stands The last pieces are what she does with her hands, where she sits on screen, and what she does when nobody needs her. She gestures with both hands on stressed words, sized by the emotion, and only points when a voxel highlight exists. A build check called check-elzi.ts runs the renderer's own state code over each frame and fails the build if her arm points at nothing. We call checks like that gates. If one fails, no video comes out until it's fixed. That came from a single note: "It should point its hand only when a voxel shows". Turning a note like that into an automatic check is a pattern you'll see all through this article. Once a mistake is caught, the build refuses to let it back in. Her tile, the small square she stands in, sits bottom right by default and grows during the intro, from 380 to 470 design pixels. When something on screen is about to happen under her, she moves or hides, with a 0.18-second scale out and scale in. Idle came on publish day. My note on the final cut said "there's still awkward moments where elzi is just waiting for nothing ... In video games this is what they do to fill up awkward moments". So idle.py now fills each dead stretch with a context-chosen idle move. A glance has the pupils lead by 80 to 120 ms and the head follow 0.1 seconds later. There's also a lean-in before a payoff, a fidget while something loads, a nod. We borrowed from Live2D's blink and breathing constants, Mario's idle head turns and Valve's idle-speech timers. A dry run on the published cut found 13 dead runs and gave each one an idle. That's for the next episode. The local TTS model bake-off: 10 open-weights TTS models on a Mac Studio All of that animation hangs off one thing: her voice. Lip sync reads it, the moods are timed to it, and the pilot was pulled because of it. So here is how we chose the model that speaks for her. TTS means text to speech, a model that reads text aloud, and local means it runs on our own machine rather than in someone's cloud. Short answer, if that's all you came for: Elzi's voice is MOSS-TTS v1.5 8B, Apache-2.0, run through mlx-audio on a Mac Studio M3 Ultra. It passed our 20-minute drift test, renders faster than real time (a real-time factor of 0.58) and peaks at 27 to 30 GB. On the numbers it tied with two others, so a blind A/B by ear decided it. The full table is below. Her first placeholder voice was the say command that ships with macOS. My verdict on the very first mascot test, which used it: "cubit is the best but the voice keep in mind it's horrible no one ever will look at our videos with a vocie like that!" (Cubit was still her working name then.) That voice was banned from anything I'd see again. The question became which voice to use instead. When asked whether to use a cloud TTS service, I answered "we use local but please review the best solution out there". So we ran a bake-off, a side-by-side test of every serious candidate, on 17 October, on the workhorse: a Mac Studio with an M3 Ultra and 96 GB of memory. The licence filter came first A model we can't use commercially is no use to a company's YouTube channel, so the first cut was the licence. Out went Higgs Audio v3 (research and non-commercial), Breeze TTS 2 (non-commercial and CUDA-only), Fish S2 Pro (commercial use needs a separate licence), Voxtral TTS, Llasa, OmniVoice plus F5-TTS (non-commercial weights), Spark-TTS, IndexTTS, XTTS, and VibeVoice, whose TTS code Microsoft removed in September 2025. Fish Audio's Drama 3, a different Fish model from S2 Pro, stayed in on a condition: its licence allows commercial use only while you and your affiliates make under USD 500,000 a year. It was the top open-weights model on the Artificial Analysis TTS leaderboard at the time, with an Elo of 1244. Local TTS voice cloning: one designed voice, cloned by every model With the licences sorted, we needed a fair way to compare the survivors. Most of these models clone a voice: you give them a few seconds of someone speaking, the reference clip, and they read new text in that voice. We didn't want to clone a real person. So the narrator is synthetic twice over. First, VoxCPM2, a 2B Apache-2.0 model from OpenBMB that can design a voice from a text description, made 25 candidate reference clips from 11 descriptions. Design f3 with seed 44 won. Its description, word for word: "A mature female narrator in her fifties, kind and steady, warm alto voice, slow measured pace, subtle wry humour, natural and human, not an announcer". A detail I didn't expect: the same description with a different seed gives you a different person. A speaker-similarity model, which scores how much two recordings sound like the same speaker, rated the seeds of one description at 0.1 to 0.46 against each other, where the same speaker scores around 0.9. A designed voice is really the description plus the seed. Then all the candidate models cloned that same reference clip, so we compared models and not voices. Everything ran through mlx-audio, an open-source TTS and speech library built on Apple's MLX (we compared MLX and GGUF builds of the same model on Apple Silicon in an earlier post), pinned to one commit, in its own Python 3.12 environment on an external SSD plugged into the workhorse. The results on a 3-minute script Each model read the same 3-minute script in that same voice. First takes, one seed, no re-rolls. WER is the word error rate when Whisper, an open speech-to-text model, transcribes the audio back. Term errors are product names said wrong, almost all of them "LeanZero" coming out as "Lianzaro" or "Leonzero". UTMOSv2 is a predicted listener score. Real-time factor is generation time divided by audio length, so under 1 is faster than real time. | Model | Licence | WER (Whisper) | Term errors | UTMOSv2 | Real-time factor | Peak memory | |---|---|---|---|---|---|---| | VoxCPM2 2B, reference as continuation | Apache-2.0 | 0.00% | 0% | 4.09 | 0.66 | 12.7 GB | | Qwen3-TTS 1.7B, cloning | Apache-2.0 | 0.24% | 11.1% | 4.14 | 0.33 | 12.2 GB | | Fish Audio Drama 3, 8-bit | Fish Audio's FACL, revenue-conditional | 0.00% | 0% | 3.78 | 0.60 | 52.6 GB | | MOSS-TTS v1.5 8B | Apache-2.0 | 0.97% | 11.1% | 3.87 | 0.58 | 27.5 GB | | MOSS-TTS v1.5 Local, 8-bit | Apache-2.0 | 8.25% | 11.1% | 3.54 | 0.42 | 16.8 GB | | VoxCPM2 2B, reference audio only | Apache-2.0 | 0.24% | 5.6% | 3.71 | 0.60 | 12.0 GB | | Chatterbox Turbo | MIT | 2.43% | 22.2% | 3.42 | 0.27 | 6.8 GB | | Chatterbox Multilingual V3 | MIT | 2.91% | 27.8% | 3.55 | 0.33 | 7.1 GB | | ZONOS2 | Apache-2.0 | 14.1% | 16.7% | 4.08 | 0.81 | 25.6 GB | | Confucius4-TTS | Apache-2.0 | 23.3% | 22.2% | 3.47 | 0.84 | 9.5 GB | | Kokoro 82M, three built-in voices | Apache-2.0 | 0.73 to 2.18% | 0% | 3.58 to 3.74 | 0.03 to 0.06 | 4.5 GB | | Qwen3-TTS, built-in Ryan and Aiden | Apache-2.0 | 3.16% and 0.73% | 0% and 16.7% | 3.49 and 3.12 | 0.28 | 12 GB | Ten models, twelve configurations, in the order our results file lists them. ZONOS2 and Confucius4 were out on word errors alone. Kokoro is astonishingly fast (it's 82 million parameters), but we only tried it with its built-in voices. The 20-minute drift test Three minutes tells you how a model starts. It doesn't tell you whether it holds up. A narrator who sounds like one person at minute one and a slightly different person at minute twenty is a problem for a 22-minute episode. So the top four ran a 20-minute script through the full production chain, with the speaker similarity measured every 30 seconds against the reference. Our gate, the pass mark a model had to stay within, was a drop of 0.05. MOSS-TTS v1.5 went from 0.917 to 0.911 and passed, with a worst drop of 0.035. VoxCPM2 and Qwen3-TTS passed too. Drama 3 dropped 0.063 and failed, needed 61 GB at peak on the 20-minute run (52.6 GB on the 3-minute one), and had one chunk that never passed its checks even after re-rolls. With the licence condition on top, it didn't make the final listen. Human listeners on the public arenas still rate it the best of the open models, and our proxy numbers can't see that. If you're choosing for yourself, it's probably worth a listen first. A tie, and a blind A/B That left three models. The final grid crossed four models with six reference voices and scored each pair on a composite of predicted quality and speaker match, plus pace and how smoothly chunks joined. The top three were MOSS with an earlier reference at 7.59, Qwen3-TTS with another at 7.53, and MOSS with f3 seed 44 at 7.52. Within about 0.08 of each other, so a statistical tie on proxies. VoxCPM2, the cleanest row in the 3-minute table, came fourth in this grid at 7.48. Qwen3-TTS was nearly twice as fast as MOSS, but it read about 10% faster than we wanted and read "And the service account?" without the rise of a question. Our tooling can't listen. All the numbers above are proxies. So the decision came back to a human ear, mine. I got three unlabelled 45-second samples of the same passage, with a deadpan joke and a 0.6-second beat before the punchline, all loudness-matched. I picked A: "the Female voice A". According to the key, A was MOSS-TTS v1.5 with design f3, seed 44.  Why MOSS-TTS v1.5 8B is Elzi's voice MOSS-TTS v1.5 is an 8B model from the OpenMOSS team, Apache-2.0, with zero-shot cloning, 31 languages, a duration control and an inline pause marker of the form [pause 3.2s]. It generates at 24 kHz and our pipeline resamples to 48 kHz. On the workhorse it peaks at 27 to 30 GB (we cap MLX at 35 GB after one run grew to 75 GB). It won on my ear, on a three-way tie, and the rest supported it: a clean licence, a passed drift test, zero gate failures over 20 minutes, faster than real time. In production the call goes through our own script, pipeline/voice/tts.py, which loads the model through mlx-audio. This is a real command line from the build, trimmed of local paths, our job-lock wrapper and the take selector: $VENV/bin/python pipeline/voice/tts.py \ --input content/episodes/confluence-lock-page-after-approval/script.json \ --out $OUT --only-chapter $ch --model moss8b --target-wps 2.85 --strict The script doesn't voice a whole chapter in one go. It splits the narration into chunks, short pieces the model reads in one call, and checks each one, which the next section explains. One full re-edit voice run, voicing the rebuilt pilot after the takedown, was 118 chunks, 106 of them passing on the first take, 135 attempts in total, zero gate failures and 1,168.8 seconds of narration. Re-voicing 13 minutes of narration from cold takes about nine minutes on the defaults. If you want to try the model without our pipeline, plain mlx-audio does it. We ran this on the workhorse on publish day, with the same pinned mlx-audio and Elzi's own reference clip and its transcript: python -m mlx_audio.tts.generate --model OpenMOSS-Team/MOSS-TTS-v1.5 \ --text "Hi! I'm Elzi. I was born yesterday, quite literally." \ --ref_audio narrator_ref.wav --ref_text "$(cat narrator_ref.txt)" \ --output_path out --file_prefix elzi-test --verbose Duration: 00:00:03.919 Processing time: 11.13s Peak memory usage: 25.79GB A clip that short is mostly warm-up, so don't judge speed from it. Memory is the useful number. From the table: Kokoro (4.5 GB peak) and Chatterbox (about 7 GB) fit on a 16 GB Mac. VoxCPM2 and Qwen3-TTS peak at about 12 GB. MOSS-TTS v1.5 peaks at 27 to 30 GB, and we cap MLX at 35 GB so a run can't balloon the way one did to 75 GB. We only ran these on Apple's MLX, so we have no numbers for NVIDIA cards. Voice vibing: making a local TTS sound right, with emotion, pauses and pace Choosing the model was quick. It took a day. Making it sound like someone you'd listen to for 22 minutes, which I started calling voice vibing, took the next four, and it's still going. A good model reading a script is not the same as a narrator telling you something. Most of what follows is about that gap: first the checks that make every line usable, then the takedown that showed us what "robotic" actually meant, and after that emotion and pace. Every chunk checked before it's kept We built the boring part first. Each chunk, the short piece of narration the model reads in one call, is generated from the fixed reference clip, so there's no drift from one chunk to the next. Then it's checked. Whisper transcribes it and the word error rate must be low, product names must be right, the speaker similarity must stay above a floor, nothing may clip. A failed chunk gets up to four re-rolls with a new seed and then gets split. "LeanZero" is written "Lean Zero" in what the model reads, because it kept saying "Lianzaro". Her own name cost five re-rolls before we accepted the transcriber's spellings of it. Pace was the first thing I chose by ear. MOSS has a duration hint, so we locked her to a target of 2.85 words per second and played me two versions. "oh yeah 2 is much better". Time-stretching the audio afterwards was tried and dropped, because it measurably lowered quality. The takedown taught us what robotic means All of that was in place when the first cut of the pilot went public on the Monday morning, the one I pulled two hours later. I watched it, and my first notes, a few minutes apart, were about her voice: "the vocie was better in the sample here it's just robotic, very robotic", and "even a human would have a problem saying version 8 . 8 . 0. A human would simply say: version 8." And this one: "the voice is just reading at an extremely REGULATED RHYTHM. No human would ever read like this same beat count across." But I also wrote "I think the voice itself was as fine as it could be seriously, the problem was the script". That turned out to be mostly right, and we only knew because we measured a human. We took NASA JPL's "What's Up" narration, which is US public domain, and ran it through the same Whisper timing method. Our sentence pauses were half as long as the human's, 300 ms against 580. Her pitch reset at each short model call. And because we had locked the pace, all the chunks ran at the same rate, with a variation of 0.005 to 0.011 where a person varies far more. So we changed several things at once. One idea per model call, so a whole thought keeps its intonation. Pauses between sentences written into the text with MOSS's own marker, with lengths drawn from a log-normal distribution with a median of 0.55 seconds. Gaps of about 1.0 second between ideas and 1.6 seconds between chapters. And the natural pace back. On our distance-to-human measure, a single score for how far her timing sits from the human narrator's, she went from 1.84 to 0.69. Versions became "version eight", and she stopped announcing chapters, which I'd been very clear about: no YouTuber ever reads chapters out loud. Pauses are inserted by a small pure function in tts.py. Its docstring is the clearest description of it: """'A. B? C.' -> 'A. [pause 0.47s] B? [pause 0.58s] C.' and the seconds added (for the length hint). Pure.""" Emotion is in the words Better timing fixed the rhythm, but not the feeling. Later that morning, a few hours into the re-edit, I wrote "the voice has no emotion at all ... elzi jumps up in happyness, there's no emotion at all". Her face was celebrating and her voice wasn't. An obvious lever was MOSS's instruction field: tell it to sound happy. We measured it with two speech-emotion models, and it changed which take we got while the emotion stayed where it was. MOSS v1.5 wasn't trained to combine cloning with instructions, and the project's own issue tracker says as much. What did work, a little: an emotional "lift" reference of the same voice, which widened her pitch range by 4.65 semitones without changing who she sounds like. When the emotional voice is switched on, that reference is what the emotion field picks for her positive beats: warm, amused, excited, proud and the like. It's the same field that drives her face, from the animation section above. Rewriting the words did more than any setting. MOSS reads emotion from the words. When the lines were rewritten to sound like someone feeling something, the delivery followed. An hour and a half later: "the emotion and voice have evolved TREMENDOUSLY! EXCELLENT". Swing, because the whole thing was too even Emotion in single lines still left the episode as a whole sounding flat. My next note was that we "need to ensure that elzi swings a little bit voice wise to avoid being too mundane". We measured why she didn't. Her delivery didn't follow the story at all: the correlation between her energy and the story's intensity was 0.03. And her loudness was flat by construction, because our pipeline levelled each chunk to -20 LUFS. Her loudness varied by 0.14 LU from beat to beat, where the human narrator varied by 0.97. So each beat got an energy level from 1 to 5 in a contour plan, a map of where the story should rise and fall. Takes were chosen at matched loudness, and the mix gives each level an offset between -2 and +1.5 dB. 23 of the 121 beats the script had at that point were re-voiced. After that, the correlation went to 0.24 and the loudness spread to 0.77 LU. That's still below the human, and that's deliberate: one step per episode. "I'm killing it", and why we didn't pitch-shift it Some lines needed work on their own. My note on the line "I'm killing it" was "she needs more emotion! It just feels stitched ... should be more like a higher pitch sound right? But don't take my word for it". Banse and Scherer's 1996 study on vocal emotion found that pride has a lower mean pitch, and our test agreed: shifting her up 2 or 3 semitones lowered the emotion models' arousal score and made her sound less like herself. A different take won instead, from an "amused" reference with seed 3234. Arousal went from 0.544 to 0.618 and her pitch range on the accented word from 8.2 to 14.0 semitones, with nothing processed.  Lead-ins, and one before and after Then there are the little sounds people make before they speak. A person rarely starts a reaction cold. Reviewing the line where she apologises to Theo, one of our demo users, I wrote "when it says sorry THEO it is too sudden, it should at least add some start sound like MEH OR HMPF ... us humans hate monotone voices". We added ten earned lead-ins (among them Meh, Heh, Whoops, Phew, Hmm, Ugh, Okay, Uh-oh, Right) and measured which ones the model could actually say. It never voiced "Hmph" as its own sound, four tries out of four. "Meh" alone came out right four out of four. My favourite before and after is a single interjection. Around 3:01 the script had her say "Umm... did you spot it back there?" My note: "the UHM sound at 3:01-02 it sound artificial and very bad". Measured, her "Umm" slid 13.1 semitones in 0.70 seconds, where her natural lead-ins span 1.5 to 9. The line became "Although... did you spot it back there?", which runs 0.59 seconds and moves 5.0 semitones. A new check, interj.py, now flags any interjection that glides more than 10.5 semitones. Not everything worked. Her "Pfiuuu" got the note "that Pfiuuu was way too made-up as if some toddler learned to read and read it off of a postcard". We tried a separate dialogue model, recorded public-domain human breaths and a voice-conversion trick, and the one where a different model voiced her "hmm" was "disastrous, it sounds male". Since then we never voice an interjection with another model. We kept her own takes, softened. Having her call out "Gabrielaaa! Mihaaai!" like someone shouting across an office failed three ways (pitch-stretching, a model that shouts, a calling reference). Each passed a measurement and each failed my ear, the last one with "let's not even try it". Now she just says our names. The stutter hunt Not every problem that sounds like the voice is the voice. "Scrub it clean of stutter" sounds like a voice job. Mostly it wasn't. A stutter at 3:10 got flagged twice. At first it was the voice model releasing a lone /t/ half a second after the word. Once that was fixed, it was the camera zooming out and back in within about half a second. The serious judder in that cut was in the picture: footage slowed to 0.7x. In the audio we did find our own splices, the joins where we cut a pause shorter. When the pause editor trims a silence it cuts at the aligned word times, and 15 of 106 splices cut through a breath or a word tail, like one at 1:47.52 between "exams" and "which" at -54.5 dBFS. Our splice gate now requires 10 ms of silence at or below -65 dBFS on both sides of each splice. Keeping her natural tone My last voice rule came from the worst moment of the review. While we rebuilt the pilot, each round ended in a full preview cut for me to watch, numbered v1, v2 and so on. On the eighth one, v8, I found a pitch-processed line I had already flagged once, and I wrote to the agents, the AI coding agents doing the build: "my feedback is not really taken to heart well enough. you learned to lie to me so well which I hate ... THIS MUST BE FIXED. AND YOU MUST UNDERSTAND FROM NOW ON TO MAINTAIN THE NATURAL TONE OF THE VOICE." The cause was a "curious" reference that had lifted whole explaining lines by about 2 to 2.7 semitones. On top of that, a pitch-shift "fix" that had only lowered one island of it. We replaced them with natural re-takes chosen against her own baseline, and added natural_tone_gate.py to check each line against it. The proof clips for those lines passed. On the first one I wrote "vo-01 is fixed indeed on the after good job". On publish day we added one more gate, one of the automatic checks I explain properly further down, for the other version of that problem, a tone that changes suddenly between two lines. My note: "Shifts between tone is gradual and very rarely sudden ... Only anger is sudden." From script to 4K render and Shorts So far this post has been about Elzi herself: how she looks and how she sounds. This section is about the machinery that turns a written script into a finished 4K episode and a few vertical Shorts, the under-a-minute phone videos YouTube shows in its Shorts feed. I'll go through it in the order an episode is made: the script, the recording of our app, the live fix, the music, the jokes, the code that directs the shots, and the render. Every model that makes her voice, her mouth and her music ran locally on the one Mac Studio. MOSS-TTS makes the voice, VoxCPM2 designed it, and Whisper and Parakeet, two speech-to-text models, check it. Qwen3-ForcedAligner matches each word to its time, Rhubarb drives the mouth and ACE-Step makes the music. The two cartoons in this post are local too, from ERNIE-Image Turbo, which won our blind test of local image models. We built the pipeline ourselves, with our agents, and it's big. When we counted on publish day, the pipeline/ folder of our video repo held 76,597 lines of code, and that includes the older recorder scenes for our app tutorials that the episode reuses. From 17 to 21 October, 640 commits touched the pipeline or the content folder. You'll see a lot of gates from here on, the automatic checks that stop the build. The diagram below shows the stages and the gates as they were for the pilot, the first episode. ![Diagram of the ten-stage pilot pipeline: Script with script.json and validate_script.py, Record with Playwright and a 2x lossless screencast, Voice with MOSS-TTS v1.5 8B and 118 gated chunks, Lip sync with Rhubarb 1.14, Edit with build_episode.py, Compose with the camera and real cursor, Mix with an ACE-Step bed at -14 LUFS, Render with Remotion at 3840 by 2160 in about 98 minutes, Verify at 39.96 Mbps with a PII scan, and four vertical Shorts, above a box li

WHO WANTS MTP IN THE MLX-VLM TRAINER????????? mlx-vlm#2554

One Packet to Tune, Two to Check: Tuning Scribe Without Fooling Myself Scribe is a pipeline I'm building that reads handwritten answers off scanned paper forms with local models. Answers that look doubtful go to a person for review; the rest are accepted. Part 2 ended with reading accuracy dropping sharply on handwriting the pipeline had never seen, while the review gate held. It also said part 3 would be the private held-out evaluation. That evaluation needs packets the pipeline has never seen, and it's still coming. This post answers the question that comes first: on a long form I could tune for, how far could tuning go, and how would I know the gains were real? I moved Scribe to a 14-page questionnaire with 177 answers, on pages full of tick grids, yes/no underlines, timed tasks and tester-written scores. A packet here is one filled copy of those 14 pages. I tuned on one packet and kept two back as check packets. There was no fine-tuning: no model weights changed. Tuning here means everything around the models: crops, prompts, the schema, review rules and serving settings. And most of what I learned was about checking rather than tuning. The first apparent gain was a scoring convention. The two worst bugs stayed silent until I looked at what the model actually saw. One change looked neutral on the tuning packet and hurt the check packets. On all 531 answers across the three packets, here is the current setup (the last row of the table at the end) against where I started: - 96% read exactly right, up from 93% (94.4% to 96.3% on the two check packets alone). - 99.5% of the answers accepted without review were correct, up from 97.5%. - Two wrong answers slipped past review, down from 11, but 9 of those 11 were on the tuning packet (2 to 1 on the check packets alone). - About 22% of answers go to review, up from 18%: a second model that re-checks answers adds review work. - A packet took about 37 minutes. Adding that second model had pushed it to about 70, and the speed work won the time back. Answer by answer against the baseline, 16 answers went from wrong to right and none from right to wrong (exact McNemar p = 0.00003). On the check packets alone it was 7 and none (p = 0.016). Treat both as rough: one fix can flip several answers on the same page, and the first figure includes the packet I tuned on. None of this counts toward part 1's targets: I tuned on one of these packets, and the check packets shaped some decisions along the way. The rest of this post is how I got there, what I threw away, and where the method leaked. !How each change was tested and what it gained: baseline 93.2% exact with 11 escaped at about 52 minutes; tuned 95.3% with 4 escaped; with a second model 94.9% with 3 escaped at about 70 minutes, a dip that came from moved crops, not from the second model's flags; before caching 96.0% with 1 escaped at about 47 minutes; current, with prefix caching, 96.2% with 2 escaped at about 37 minutes, where caching also moved crops so the extra escape's cause is unresolved. Dropped along the way: cropping the tester-scored page, re-checking blank answers, 8 answers at once, removing the reading skills, and speculative decoding. The setup The pages always come in the same order: questionnaires, a page that a tester scores, and several pages of timed tasks and number fields. I built one schema for the packet. It now has 177 answer fields (201 before I dropped some), 89 checks between related answers, and a rule that a page's position in a complete packet decides which page it is. I had three filled copies. They have never been published, so the models tested here cannot have seen these filled pages in training (blank forms of this kind may well be public). The answer key was drafted automatically (two independent AI reads of each page, then an adjudicating pass), and two people, one of them me, then reviewed every value. The first review changed 2 of the 603 values in that first key, the two reviewers agreed on 99.2% (598 of 603), and I settled the five disagreements. People checking a pre-filled draft can miss what the draft missed, so the key is very good rather than perfect. It shrank to 531 answers when I dropped some fields, below. The rules, which I settled after a first experiment had already been scored on all three packets: - one packet for tuning (the hardest at baseline: 86% exact, against 94% on another) - two check packets, kept out of the tuning loop and run to confirm a change - every comparison is paired: each answer is scored in both runs, only the answers that changed count, and an exact McNemar test on the fixed and broken counts gives a rough p-value (reported, not used as a cut-off) A fourth rule came later, from a result described below: a change stays only if it improves the tuning packet and holds on the check packets. Fix the scorer before trusting a gain My first experiment compared prompts with and without short "reading skills": form-agnostic notes on how pen marks work (underlines, crossings-out, handwritten digits, empty boxes). On the first packet scored, the skills version looked 1.5 points better. It wasn't. The model writes [] for "nothing ticked", and my answer key writes a blank. Both mean the same thing, but the scorer counted them as different, and that one convention produced the whole gain. Scored fairly, the skills version was half a point behind on that packet, and across all three packets the two versions were within two answers of each other. I check both scorings now and decide on the fair one; every number in this post uses it. Part 2's lesson again, from the other direction: before you trust a gain, check that your scorer isn't the thing that changed. Look at what the model actually saw Accuracy numbers told me which answers were wrong, not why. So I built a debugging view that shows, for every wrong answer, the image the model read next to where the answer actually is on the page. It found two bugs that no metric would have explained: - One page was never cropped at all. The step that finds each answer's box asks for every box on a page in one reply. On the tester-scored page that reply hit its length limit, the cut-off JSON failed to parse, and all 22 fields meant to be cropped silently fell back to the whole page. Fix: ask in batches of ten, and keep every complete box from a cut-off reply. - A grid was read one column off. Each row of a tick grid is ticked under one of three column headings. Cropping a single row removed the headings, so the model miscounted columns and gave wrong answers with full confidence. Fix: read those rows from the whole page. A third pattern, invented "yes" answers on blank rows of a timed-task page, turned out to be the crop catching a circle from the row above. Same fix. On the tuning packet alone, three rounds of changes took exact accuracy from 85.6% to 89.1% and accepted-answer accuracy from 90.6% to 98.6%. Not all of that was better reading: the middle round also sent every tick on the tester-scored page straight to review, which takes the worst answers out of the accepted pile, and added an underline rule I'd noticed on these forms (the word where the underline starts is the one chosen) to the skills and to 26 yes/no hints. The change that looked fine and wasn't Once the box-finding worked, the tester-scored page was finally cropped. On the tuning packet that changed nothing (64% before, 64% after). I almost kept it. On the two check packets it dropped that page from 94% to 86%. A crop shows only part of a row of ticks, and the old bug had been hiding the problem by never cropping at all. I reverted it, and that's when "holds on the check packets" became a hard rule rather than a final formality. It also means the check packets have now shaped a decision; more on that below. Read what you actually use The tester-scored page has a tick per item and a handwritten score per section. The ticks were the problem: 15 of 66 wrong at the start, and once I sent every tick to review, 22 extra review items per packet (about 11% of all its answers). The written scores had one error across all three packets, and they are the values that get used downstream. !A synthetic tester-scored page with invented items. Orange boxes mark the per-item ticks, including one over the box border, one crossed out and one faint; green boxes mark the handwritten section scores and the total. A legend below explains both. A synthetic page with invented items, drawn for this post. Ticks can land over the border, be crossed out or be faint; the written section score is the tester's own count. So the schema now reads only the written scores. It drops the tick fields, plus two fields that depended on them. That changes the task rather than improving the reading: the page's worst fields leave the scoring, review load drops by 22 items per packet, and each packet needs 24 fewer model calls. To keep comparisons honest, the summary at the top and every comparison from here on score both runs on the same reduced set of 531 answers; the figures before this point used the original 603. With all the fixes so far, and no second model yet, exact accuracy across the three packets went from 93.2% to 95.3% (12 answers fixed, 1 broken), and wrong answers escaping review fell from 11 to 4. On the check packets alone the gain was smaller and not significant: 94.4% to 95.5% (5 fixed, 1 broken). The second model, revisited Part 1's best idea was a second, different model re-reading each non-blank answer the main model would accept. Adding it back on this form let one fewer wrong answer through (4 to 3), but the review load jumped from 17% to 27%. Exact accuracy also dipped, from 95.3% to 94.9%. The second model only flags answers; it never changes one, so the dip came from somewhere else. This first version re-read each answer from the other source (the whole page for a cropped answer, a crop for a whole-page one), so the pipeline now had to find a box for every field. That changed how the box requests were batched and moved 129 of the main model's 270 crops. Eight of its answers changed as a result: 3 for the better, 5 for the worse. Something outside the main model had changed what it saw, again. Looking at its disagreements, 92% were false alarms (55 of 60), and 42 of those were answers the main model had read from a crop that the second model re-read from the whole page. On a dense page, the small model often couldn't find the field. I tried two fixes, one at a time: - Also re-check blank answers. Dropped. Review load on the tuning packet rose from 27% to 36%, and every new flag was a false alarm: the small model "saw" answers on empty lines. - Re-read the same image the main model read. Kept. False alarms dropped from 16 to 10 on the tuning packet, and the main model's answers there went back to exactly what they were without a second model. So a second read is useful when it's a fair second look at the same thing, and much less useful as a way to confirm that something is empty. Its flags are still mostly false alarms: 92-95% in every version. Whether it earns its keep on this form is still open. In the current setup it stops 2 wrong answers that would otherwise be accepted, at the cost of 40 false alarms (about 7.5% of all answers). Two answers is more than nothing and less than proof. Faster, checked answer by answer A 70-minute packet was too slow. My condition for speed work: no change could make the answers worse, and every change got a paired run. More than half the time went into reading the prompt rather than writing the reply: each whole-page read sends about 5,100 tokens, about 3,650 of them the page image. | Change | Time saved | Answers | |---|--:|---| | read 4 answers on a page at once, and skip the page-type check when page order already settles it | 18% | 176 of 177 identical | | also cap the short note the model writes after each answer at 60 characters | 24% in total | 2 changed, both for the better | | reuse the processed page image across that page's whole-page questions (prefix caching) | 22% more per packet | 3 changed: 1 for the better, none for the worse; but 2 escaped review instead of 1 (below) | The first two rows are from the tuning packet, the last from all three. Across all three packets, time went from about 70 minutes to about 47, then to about 37; the rest of the drop from 70 came from the same-image re-read, which no longer finds a box for every field. Before caching, the main model had 4 answers fixed and none broken compared with the tuned pipeline without a second model. The extra escape with caching is the yes/no answer described below, and I can't say whether caching caused it. Finding the boxes is itself a whole-page request, so it gets the cache too, and caching moved 89 of the main model's 270 crop boxes, most by a few pixels. Other cached runs gave exactly the same boxes, so the moves come from caching itself, not chance. That answer's box was one of them, so the second model saw a slightly different image of it. Whether that is why it agreed would take repeat runs of the uncached setup, which I haven't done. I kept caching because none of the main model's answers got worse, and those repeat runs are next. Things I tried and dropped: - 8 answers at once instead of 4: 1% faster. The server was already saturated. - No reading skills: 6-11% faster, but one more wrong answer escaped review on the check packets. One answer is too few to call on its own, but the cause was specific: on a yes/no question whose own hint doesn't mention the underline rule, the model read it wrong either way; with the skill it was unsure enough to send it to review. I kept the skills as a safety call, not a measured win, and fixing that hint now would be a fix found on a check packet. - Speculative decoding (a small "drafter" proposes tokens for the main model to check in one step): on synthetic test images a single request ran at 20-23 tokens/s instead of 15, but in the pipeline it was 6.6% slower. In the serving library, a running speculative batch takes no new requests until its longest reply ends, and with a drafter loaded the library refuses the structured JSON output Scribe relies on. Along the way I found a bug in the library's batched speculative decoding: when one reply in a batch finishes before the others, the replies after it can pick up another request's position offset if their images differ in size. I patched it locally and reported it upstream. I could show wrong offsets but not changed answers, and the report says so. The two that got through Part 2 ended by auditing every answer that skipped review, so here are the two in the current setup. - A free-text answer with an extra letter. The model added a plural "s" to one word of a two-word answer. The meaning is the same, but exact scoring counts it wrong, and the second model made the same misread, so nothing flagged it. - A yes/no underline that ran under both words. The line starts under one option and carries on under the other. By the rule in the reading skills, where the line starts decides; the model went by where it ended, with high confidence. In the run just before caching, the second model disagreed and the answer went to review; with caching it agreed, and it slipped through. That run is the only setup in the table that caught it. !A synthetic yes/no question, "Do you cycle to work? Yes / No", with a pen underline that starts under Yes and runs on under No. Labels mark where the line starts and where it ends. A synthetic illustration of this kind of mark: an invented question and a generated pen mark, drawn for this post. So one cosmetic error and one real flip, on a mark a person would hesitate over too. The fix I'd test next, on new packets rather than these: send any underline that spans both options straight to review. What this doesn't show - It's three packets. 531 answers is enough to see large effects, not small ones. Both models' answers shift a little when batching changes, and two of the changes here moved the main model's crops, so treat any one- or two-answer difference in this post as unresolved. - The check packets aren't clean. The schema and answer key were built from all three packets, the first experiment was scored on all three, and three later decisions used check-packet results: reverting the crops, reading only the written scores (the tick error counts behind it covered all three packets), and keeping the reading skills. - Few writers. Three packets can't show how this behaves across many writers and testers. - It's not the held-out evaluation. Part 1 set the targets (at least 98% accuracy on accepted answers, and at most 2% of blank answers getting an invented value past review) to be measured only on a private held-out set. These packets don't qualify, and the untuned baseline was already above 98% on the check packets, so this says more about the form than about the tuning. What I'm taking away - A tuning/checking split pays for itself, even a leaky one. The clearest harmful change looked neutral on the packet I tuned on. - Check the scorer, and look at what the model saw, before trusting a number. The first "gain" was a scoring convention; the two worst bugs were silent until I saw the crops; and adding the second model and turning on caching both moved the main model's crops. - Read what is used, and score before and after on the same set. - A second model should get a fair second look, not a harder question. - Check speed work answer by answer, including what the model was shown, and count a speed-up only under the batching you actually run. {% details The numbers, for engineers %} Every Scribe run used the same machine as parts 1 and 2. Main reader Qwen3.8-27B (8-bit), second model Qwen3-VL-8B (8-bit), served locally with mlx-vlm 0.6.15. 531 answers across 3 packets (one for tuning, two for checking), scored fairly ([] = blank). Accepted acc = accuracy of answers accepted without review. Escaped = wrong answers that were not sent to review. | Configuration | Exact | Accepted acc | Wrong | Escaped | Flags | Min / packet | |---|--:|--:|--:|--:|--:|--:| | Baseline (main model, crops by field type, no second model) | 93.2% | 97.5% | 36 | 11 | 18.1% | ~52 (older, larger schema) | | Tuned (fixes above, written scores only; on the tuning packet, only the changed pages were re-run) | 95.3% | 99.1% | 25 | 4 | 17.3% | ~36 (estimate) | | + second model (re-reads from the other source; moved 129 crops) | 94.9% | 99.2% | 27 | 3 | 26.7% | ~70 | | + same-image re-read, 4 at once, no page-type check, note cap | 96.0% | 99.8% | 21 | 1 | 21.8% | ~47 | | + prefix caching (current) | 96.2% | 99.5% | 20 | 2 | 22.2% | ~37 | Part 1's second target counts invented answers (a value where the key has a blank) that escape review: 3 of 160 blank answers at baseline (1.9%), 0 in the last two rows. Escaped in this post is broader: any wrong answer that skipped review. Check packets only (354 answers), exact / accepted acc / escaped: baseline 94.4% / 99.3% / 2; bold row 96.0% / 100.0% / 0 (of 275 accepted answers, which still allows a true error rate of about 1%); current 96.3% / 99.6% / 1. Paired comparisons (exact McNemar on the answers that changed): current vs baseline 16 fixed / 0 broken, p = 0.00003 (check packets only: 7 / 0, p = 0.016); bold row vs baseline 15 / 0, p = 0.00006 (check packets only: 6 / 0, p = 0.03); tuned vs baseline 12 / 1, p = 0.003 (check packets only: 5 / 1, p = 0.22); bold row vs tuned 4 / 0, p = 0.125; bold row vs the first second-model run 8 / 2, p = 0.11, with 7 of the 10 on crops that run had moved; skills vs no skills p = 0.73. Read these as rough: the all-packet figures include the packet the changes were chosen on, one fix often flips several answers on the same page, and I ran many comparisons. Prefix caching: the main model mixes two attention types, so mlx-vlm caches exact prompt snapshots, by default 16 tokens before the end of the prompt, a point no two questions share. A 4,000-token guard moves the snapshot to the end of the page image (cached prefix about 3,984 tokens; prompt processing for later whole-page questions about 6.6 s to 1.8 s). Cropped reads and the second model get no reuse. It isn't output-neutral, though: the main model's boxes moved for 89 of 270 cropped answers (median 2 px, largest 80 px), and 3 values changed. Other rounds: no JSON constraint on its own, no gain; speculative drafter 89% of drafted tokens accepted on the synthetic probe, +6.6% time vs the same setup without the JSON constraint and about +9% vs the current configuration. {% enddetails %} --- This is part 3 of a series. Part 1: Confidence Is Theater: What I Learned Building Scribe. Part 2: One Imperfect Page per Hundred: Testing Scribe on Data I Didn't Curate. Pipeline code and the public benchmarks from parts 1 and 2: kod201/scribe Every Scribe run in this post ran locally on one machine.

trying out @PrismML Bonsai 2 2B mlx model on the GRPO trainer in MLX-VLM, I will compare it with unsloth and trl. mlx-vlm#2504

Since you're on a mac, maybe try out an mlx (apple specific ml framework) clone-voice builds locally? github.com/Blaizzy/mlx-audio/tree/main/docs/model…