Elzi: Building an AI YouTube Narrator on a Local TTS Model
So why Elzi, the cartoon robot who now narrates our YouTube videos? Why are we putting out more AI slop, right? These days people assume most of what they see online is AI slop, and a cartoon narrator voiced by a local TTS model sounds like more of the same. There was always great human content out there, and there was always a ton of human slop too, long before anyone could type a prompt. Is Elzi slop? Hell no. Elzi narrates our first YouTube episode, which I'll call the pilot from here on. I wrote 150 review notes on that pilot, and I pulled its first cut off YouTube two hours after it went up. Part of the episode shows us fixing a real bug in one of our apps, on camera. An early cut showed that fix being made by typing a request to an AI coding agent. I refused it, and we made it a real fix in a real code editor, with real tests going green. It took five days, from the first message I sent the AI agents that built her on 16 October to the public episode on the morning of 21 October. Is it perfect? No, and a good part of this is about where it isn't.
Let her say hello first. Gabriela and I are LeanZero, a two-person company that makes Atlassian apps. This is how she opens the pilot, 17 seconds in:
> Hi! I'm Elzi, nice to meet you! I was born yesterday, quite literally, and I'll try my best to present LeanZero and do what Gabriela and Mihai couldn't, which is talk about their work in a YouTube video. These two are proper slave drivers: they told me to break the ice and present one of their products, on my first day, with no training whatsoever.
A few lines later:
> Also, nobody has mentioned a salary yet. I'm keeping notes.
So who is she? Elzi is the small voxel robot who narrates our product videos. She talks with a local TTS (text-to-speech) model that runs on our own Mac Studio, she lip-syncs with an open-source tool, and she's drawn as flat polygons that only look like voxels. Her first episode is a 22-minute walkthrough of locking a Confluence page after approval with our app Sentinel Vault: someone edits an approved page, and the edit gets put back to the approved version. That episode went public once before, as a 34-minute cut, and I pulled it two hours later because she sounded like a robot reading a manual.
youtube.com
The episode has its own page on our site, A Confluence Approval Workflow That Survives an Edit, with chapters and a transcript, plus the exact settings it shows.
This post tells the whole story, so you can read it in order or jump. If you came here for the voice: it's MOSS-TTS v1.5, an 8B (eight billion parameter) model under the Apache-2.0 licence, running through mlx-audio, an open-source speech library built on Apple's MLX framework, on a Mac Studio. The section further down where we tested ten models side by side has the full results, a command you can run, and why this one won. If you came because you use our apps: Elzi is meant to be the one who shows you around them one day, so how she sounds is your business too.
Why an AI YouTube narrator instead of filming ourselves
Before the how, the why. Gabriela and I are two Eastern Europeans in Romania, and neither of us is a YouTuber. Two people can't run a YouTube channel the way YouTubers do. But a product with no videos and no reviews gets skipped by most companies looking for an app, which makes sense, I'd skip it too.
We did the maths on filming ourselves. Scripting, recording, re-recording because one of us stumbled, editing, captions: two weeks or more per video, for two people who also build the apps and do the support and the client work. We'd make one video and then stop.
So we decided to build a narrator instead, and to have AI do most of the building. To be clear about who did what: AI coding agents wrote almost all of the pipeline's code and ran it. By pipeline I mean the chain of scripts that goes from a written script to a finished video: recording the app, voicing Elzi, animating her, editing and rendering. I worked with those agents in one long chat session and wrote the notes. Gabriela and I made the calls.
Making money on YouTube isn't the point either. I wrote it to the agents in that session on day two: "I don't care about monetisation btw, this is not why we're doing it! it's for our brand and our domain authority". Elzi exists so that people find out LeanZero exists. She's a way for us to say things in public without running ourselves into the ground.
And we share what we learn, like we always do. Building a narrator with AI models taught us a lot of small, specific things about local TTS and about what makes a synthetic voice sound robotic, and you'll find them below with the numbers.
There's an obvious tension in all this: a video made by AI, for people who are tired of AI. One thing I wrote to the agents during the build stuck:
> what I want is not to mask the AI condition, but to make it more beautiful, give it a place amongst us humans and make youtube content made by AI likeable because everyone hates it, everyone hates AI slop
I think that's the right bar. The question we set ourselves was whether she can still be worth twenty minutes of your time.
Published at 5 a.m., pulled at 7
The first time we tried to answer that question, the answer was no. I'm least proud of this part, so you get it early. Before bed on Sunday, 18 October, I wrote to the agents: "Feel free to just push it on youtube. I am sure it will be awkward and not perfect but who cares this is the first run." The 33:58 cut finished rendering overnight. Before anything goes out, an AI reviewer acts as our red team, meaning its job is to attack the video and find what's wrong with it. It scored this cut 80 out of 100, and the episode went public with four Shorts, short vertical clips cut from it, at 5:13 on Monday morning, our time.
I watched it at seven and ordered it down. What I wrote to the agents was "too robotic, starts talking about chapters ... The gags can ONLY work if they have sound otherwise cut them out makes no sense." By chapters I meant that Elzi kept announcing which part of the video we were in, which no YouTuber does. The gags are the short comic moments and cutaways we put in to make it fun, and some of them made no sense without sound. The episode and its four Shorts were all private again within a minute or so.
When we sat down to plan the re-edit and read back through that first cut, the problem was plain: it was a 34-minute manual. Every frame had the same layout. The narration named every click, read version numbers out digit by digit and announced the structure about 15 times. A promise made in the first 13 seconds paid off at 28:01, so a viewer had to wait almost half an hour for the point. We didn't restart. My instruction was to re-edit the same footage, not start over.
The re-edit started from one question the whole video answers. We set a target of 18 to 22 minutes, no on-screen chapter rail (the strip of chapter titles the first cut kept on screen), and gags only where something actually goes sideways. The other thing the first cut lacked was an introduction: Elzi never said who she was. I wrote her introduction together with the agent drafting her lines. The framing that came out of it is the "born yesterday" idea. I put it to that agent like this: "Think of this all as context where humans are as blank as you and they will learn from this video just as much as you learned." She learned everything from scratch, so she explains it from scratch.
That's also the standard we went in with for the second attempt:
> we don't need a perfect video, but at least one that's not miserable.
Claptrap, voxels, and where Elzi's look came from
Now to Elzi herself, starting with how she looks. Elzi's look starts with Claptrap, the small talking robot from Borderlands. If you've played any of the games you know him: a yellow box on one wheel, one big blue eye, a running commentary nobody asked for. Those games are drawn in a hand-drawn, comic-book look with heavy black outlines. I'll call it cel-shaded (Gearbox, the studio behind the games, prefers "concept art style", but come on).
Claptrap himself started as a joke. Brian Martel of Gearbox told GamesBeat in 2012 that concept artist Lorin Wood "posted a quick sketch of him as a joke" and "We all instantly fell in love when we saw it". According to Wikipedia, Wood set out to draw the most generic robot he could. His first voice, David Eddings, was a Gearbox vice president. A throwaway robot, voiced by someone who already worked in the building, became one of the series' most popular characters by fan vote. So I figured ours could start as a joke too.
What Elzi borrowed from Claptrap, and what she dropped
Side by side, you can see what carried over. She took his job first: the funny little robot who talks you through things. She has his antenna, a rod with an orange voxel cube on top. She has two expressive arms, though hers end in orange voxel mitts, and she only points when there is something on screen to point at. She has the bold black outline, a 5-unit ink line on each face, which is the clearest visual link between the two. And she has the self-mocking energy, which you've already heard: the slave drivers, and the salary she's keeping notes on.
She dropped a lot more. No wheel. Two rectangular LED eyes instead of one round one, because an LED face reads emotion at phone size and her mouth positions are the clearest changes we could draw. Her head is the three-faced LeanZero cube from our site's logo, sitting on a navy torso that gets wider toward the bottom, where Claptrap narrows. No weathered yellow: all her colours are tokens from leanzero.net (cyan #22d3ee, blue #1d4ed8, violet #7c3aed, orange #f97316 for the joints and the L plate). He is a he, voiced by a man. She is a she, voiced by a local model. And she's meant to be likeable. Claptrap's whole act, by most accounts, is being obnoxious, and Wikipedia sums up his reception as strongly split between fans and critics. We didn't want that half.
Her name is just LeanZero's initials said aloud: L, Z. Her working name for the first day was Cubit. Then I asked the agents whether it was really original, and it turned out not to be: there's a Nintendo 3DS robot called Cubit, Sega has Cubot, and there are CUBIT trademarks. We checked Elzi against USPTO, EUIPO's TMview, OSIM in Romania and WIPO before keeping it. There's one soft risk, which is that an English ear hears "Elsie", so we always write it Elzi and say it EL-zee.
Why voxels, and why ours are fake
The other half of her look is the voxel style, and it needs a quick explanation. A voxel is a value on a 3D grid, the way a pixel is a value on a 2D one, and voxel art is the 3D cousin of pixel art: everything built from coloured cubes. Minecraft is the famous example. It also cheats, which I find reassuring: it stores the world as voxels but draws each block as ordinary polygons.
Elzi cheats more. There's no Blender in this project, no MagicaVoxel, no three.js, no 3D file anywhere. She's flat 2D SVG polygons drawn by React inside Remotion, a library that turns React components into video frames. Her voxel look comes from an oblique extrusion. Each part is a flat front polygon, pushed back along one fixed vector, so the visible back faces show as a lit top and a shaded right side, all outlined in ink. This is the whole trick, from concepts.ts, the file in our code where her concepts are drawn:
// the ONE oblique extrusion vector (per unit of depth): up and to the right, so tops and right sides show
export const E: [number, number] = [1, -0.85];
One vector means one light direction for the whole character. Next to it we wrote down that you rotate or flip a part before extruding it, never after, "or the light turns with it". That same voxel language draws the highlights in the video: the arrows, the corner-bracket frames, the amber spotlight on the payoff.
Why bother faking it? Partly it's about reading at small sizes. Valve's paper on the rendering of Team Fortress 2 says its characters were designed so that "only in silhouette with no internal shading at all, the characters are readily identifiable". A chunky, outlined robot follows the same principle. And we measured it for phone screens. Watch the episode inline on a phone held upright and her corner tile, the small box in the corner of the frame where she stands, is about 64 points wide. That leaves her features about 3 pixels across, so you see presence and a moving mouth and not much else. Full screen in landscape it's about 140 points and reads fine.
She went from idea to a finished pose sheet, the reference drawing of every pose, mouth and mood a character can have, in one day. Around midday on 17 October we had three concepts: Ell, the LeanZero L as a paper-cut slab; Cubit, the engineer robot with the cube head; and Facet, a cut violet gem with a spark crest. Of the three, the LED face read best at those small sizes. Cubit won, my note on it was "I love the voxel art it's amazing", and the same afternoon she was rebuilt with one light direction and the site's exact colours. Her celebration yellow changed from #FFD400 to the site's amber #fbbf24, because #FFD400 isn't a colour we use anywhere.
The voxel character and her animation: lip sync, idle, emotions and camera
A pose sheet is a drawing, though, and a narrator has to move. So the next job was turning that sheet into a character that talks and blinks and keeps out of the way of the screen she's explaining. This section is how she's animated, and the voice comes after it.
Elzi's renderer, the code that draws her in each frame of the video, is four files: Mascot.tsx, mascot/state.ts, mascot/concepts.ts and Voxel.tsx, about 1,800 lines together when we counted on publish day. Her rig, meaning the set of parts she's assembled from, lives in one 600 by 600 space with swappable parts: a head and body, eyes with LED pupils that open and close, brows in three positions, nine mouths and four arm poses (neutral, pointing, facepalm, celebrate). A script exports them as about 20 standalone SVGs per concept.
Everything she does is a pure function of time. Given a frame number, mascotStateAt returns what she looks like. So any part of the episode can be rendered by itself and stitched to the rest, because nothing depends on the frames before it. With renders that take an hour and a half, we leaned on that a lot.
Lip sync with Rhubarb
The first thing a talking character needs is a mouth that matches the words. Hers is driven by Rhubarb Lip Sync 1.14.0, an MIT-licensed tool that reads a WAV file and outputs which mouth to show when. It uses the classic animation set, mouths A to H plus X for rest, which is why her pose sheet has nine mouths. Its macOS release is x86-only. Our Mac Studio (we call it the workhorse, and it's the machine that does all the heavy work in this story) has no Rosetta. So we built it from source, which took about three minutes. It analysed 8.7 seconds of speech in 3.5 seconds.
We always change two settings. We hand it the transcript (-d dialog.txt) so it isn't guessing the words, and we turn on the extended mouth set with --extendedShapes GHX. Our episode script is split into beats, a beat being one short line of narration with its own instructions, and each beat is voiced into its own WAV file. Our episode builder runs Rhubarb on each beat's WAV, caches the result by content hash and shifts the cues into episode time. Then the mouth is whatever cue Rhubarb gave for that frame. Over the pilot I wrote 150 review notes on what was wrong, and lip-sync drift never came up as a defect in any of them, which surprised me more than anything else in this project. My guess is that handing Rhubarb the transcript did most of that work.
Moods, and one field for her face and her voice
A mouth that moves in time still looks dead without a face that reacts, so on top of lip sync she has moods. In the pilot she has seven: idle, explain, think, surprised, facepalm, celebrate, eyeroll. An eighth, uh-oh, was added after publishing for the next episode. Celebrate puts her arms up, bounces her, throws voxel confetti behind her and turns her tile amber. Each mood change pops with a 6% squash. Her blinks are seeded, 2.0 to 4.8 seconds apart and 0.13 seconds long, so they look random without being random from one render to the next.
The decision I'd copy into any other project: one emotion field drives both her face and her voice. Each beat of the script can carry an emotion. Then content/emotion.py maps it to Elzi's mood and, when we switch on an optional livelier set of voice references, to the reference clip her voice is cloned from. (Her voice is cloned from a short sample recording, a reference clip, and the voice sections below explain how that works.) A beat whose explicit animation tag contradicts its emotion is a validation error, so her face can't celebrate while the script says she's tense. Here are two beats from the pilot's script.json:
{ "id": "t14b", "chapter": "two-seats",
"narration": "Oh... I hope I'm doing okay. I really need this job.",
"emotion": { "name": "tense", "intensity": 0.6, "then": "relieved" },
"signature": { "inhale": true }, "aside": true }
{ "id": "L7c", "chapter": "ship-fix",
"narration": "I'm killing it.",
"emotion": { "name": "proud", "intensity": 0.8 } }
The pilot script has 117 beats and 3,257 words. Counted by emotion, 29 are neutral, 23 warm, 11 excited, 10 wry, 8 proud, 7 amused, 6 curious, 6 tense and 5 confident, with a few others after that. In the final edit that became 160 stretches of explaining, 22 of thinking, 9 celebrations, 4 idles, 2 surprises and one eye-roll.
The bow, the throw and the aside
Most of her moods just follow the script. Some moves have exact timings, because she does them in sync with something else. Her bow is timed to her entrance sting, a 1.25-second sound of FM bells and a cube "thock" that our own script synthesises from fixed seeds, so it's the same sound in each episode. Her bow starts 0.10 seconds before the first note and peaks on the third. Her throw has its own constants, and both sit in state.ts:
export const BOW = { pre: 0.10, apex: 0.21, hold: 0.46, up: 1.22 };
export const TOSS = { wound: 0.16, swing: 0.10, follow: 0.15, back: 0.5 };
The throw needs a word of context. When the video wants you to look at a button or a setting, a voxel highlight appears on it, an arrow or a frame in the same blocky style as Elzi. The throw was my idea, on the evening of 19 October. Instead of a highlight just appearing, she lobs a voxel ball in the highlight's colour and it turns into the highlight where it lands. My condition was that it mustn't turn her into "a jittery mess", so it only happens for the first highlight after ten quiet seconds, never during a zoom and never near another attention cue. In the pilot (watch it at 6:12) she winds up at 6:12.7, lets go at 6:13.1, and the ball hits the setting at 6:13.6, the one called "Require approval before publishing".
Gag timing has a rule too. Now and then the video cuts away to a few seconds of an old film clip for a joke, then cuts back to the app. Her reaction only happens in the 0.4 to 0.7 seconds after we cut back. A reaction before the cutaway gives the joke away.
If I could keep one bit, it'd be the aside, the moment she stops explaining and speaks to you directly. At 11:43 in the pilot she turns to the camera nervously and says "Oh... I hope I'm doing okay. I really need this job." It's the beat marked "aside": true in the JSON above. My note when we built it: "it needs an inhale and exhale to make it more convincing - usually when you say stuff like this it's like breaking the 4th wall." So she breathes in first. Afterwards there's a 1.2-second composure move: she straightens, shakes her head, closes her eyes. Even the breath sounds are cut from her own TTS output.
Talking hands, idle moves and where she stands
The last pieces are what she does with her hands, where she sits on screen, and what she does when nobody needs her. She gestures with both hands on stressed words, sized by the emotion, and only points when a voxel highlight exists. A build check called check-elzi.ts runs the renderer's own state code over each frame and fails the build if her arm points at nothing. We call checks like that gates. If one fails, no video comes out until it's fixed. That came from a single note: "It should point its hand only when a voxel shows". Turning a note like that into an automatic check is a pattern you'll see all through this article. Once a mistake is caught, the build refuses to let it back in.
Her tile, the small square she stands in, sits bottom right by default and grows during the intro, from 380 to 470 design pixels. When something on screen is about to happen under her, she moves or hides, with a 0.18-second scale out and scale in.
Idle came on publish day. My note on the final cut said "there's still awkward moments where elzi is just waiting for nothing ... In video games this is what they do to fill up awkward moments". So idle.py now fills each dead stretch with a context-chosen idle move. A glance has the pupils lead by 80 to 120 ms and the head follow 0.1 seconds later. There's also a lean-in before a payoff, a fidget while something loads, a nod. We borrowed from Live2D's blink and breathing constants, Mario's idle head turns and Valve's idle-speech timers. A dry run on the published cut found 13 dead runs and gave each one an idle. That's for the next episode.
The local TTS model bake-off: 10 open-weights TTS models on a Mac Studio
All of that animation hangs off one thing: her voice. Lip sync reads it, the moods are timed to it, and the pilot was pulled because of it. So here is how we chose the model that speaks for her. TTS means text to speech, a model that reads text aloud, and local means it runs on our own machine rather than in someone's cloud.
Short answer, if that's all you came for: Elzi's voice is MOSS-TTS v1.5 8B, Apache-2.0, run through mlx-audio on a Mac Studio M3 Ultra. It passed our 20-minute drift test, renders faster than real time (a real-time factor of 0.58) and peaks at 27 to 30 GB. On the numbers it tied with two others, so a blind A/B by ear decided it. The full table is below.
Her first placeholder voice was the say command that ships with macOS. My verdict on the very first mascot test, which used it: "cubit is the best but the voice keep in mind it's horrible no one ever will look at our videos with a vocie like that!" (Cubit was still her working name then.) That voice was banned from anything I'd see again. The question became which voice to use instead. When asked whether to use a cloud TTS service, I answered "we use local but please review the best solution out there". So we ran a bake-off, a side-by-side test of every serious candidate, on 17 October, on the workhorse: a Mac Studio with an M3 Ultra and 96 GB of memory.
The licence filter came first
A model we can't use commercially is no use to a company's YouTube channel, so the first cut was the licence. Out went Higgs Audio v3 (research and non-commercial), Breeze TTS 2 (non-commercial and CUDA-only), Fish S2 Pro (commercial use needs a separate licence), Voxtral TTS, Llasa, OmniVoice plus F5-TTS (non-commercial weights), Spark-TTS, IndexTTS, XTTS, and VibeVoice, whose TTS code Microsoft removed in September 2025. Fish Audio's Drama 3, a different Fish model from S2 Pro, stayed in on a condition: its licence allows commercial use only while you and your affiliates make under USD 500,000 a year. It was the top open-weights model on the Artificial Analysis TTS leaderboard at the time, with an Elo of 1244.
Local TTS voice cloning: one designed voice, cloned by every model
With the licences sorted, we needed a fair way to compare the survivors. Most of these models clone a voice: you give them a few seconds of someone speaking, the reference clip, and they read new text in that voice. We didn't want to clone a real person. So the narrator is synthetic twice over. First, VoxCPM2, a 2B Apache-2.0 model from OpenBMB that can design a voice from a text description, made 25 candidate reference clips from 11 descriptions. Design f3 with seed 44 won. Its description, word for word: "A mature female narrator in her fifties, kind and steady, warm alto voice, slow measured pace, subtle wry humour, natural and human, not an announcer".
A detail I didn't expect: the same description with a different seed gives you a different person. A speaker-similarity model, which scores how much two recordings sound like the same speaker, rated the seeds of one description at 0.1 to 0.46 against each other, where the same speaker scores around 0.9. A designed voice is really the description plus the seed.
Then all the candidate models cloned that same reference clip, so we compared models and not voices. Everything ran through mlx-audio, an open-source TTS and speech library built on Apple's MLX (we compared MLX and GGUF builds of the same model on Apple Silicon in an earlier post), pinned to one commit, in its own Python 3.12 environment on an external SSD plugged into the workhorse.
The results on a 3-minute script
Each model read the same 3-minute script in that same voice. First takes, one seed, no re-rolls. WER is the word error rate when Whisper, an open speech-to-text model, transcribes the audio back. Term errors are product names said wrong, almost all of them "LeanZero" coming out as "Lianzaro" or "Leonzero". UTMOSv2 is a predicted listener score. Real-time factor is generation time divided by audio length, so under 1 is faster than real time.
| Model | Licence | WER (Whisper) | Term errors | UTMOSv2 | Real-time factor | Peak memory |
|---|---|---|---|---|---|---|
| VoxCPM2 2B, reference as continuation | Apache-2.0 | 0.00% | 0% | 4.09 | 0.66 | 12.7 GB |
| Qwen3-TTS 1.7B, cloning | Apache-2.0 | 0.24% | 11.1% | 4.14 | 0.33 | 12.2 GB |
| Fish Audio Drama 3, 8-bit | Fish Audio's FACL, revenue-conditional | 0.00% | 0% | 3.78 | 0.60 | 52.6 GB |
| MOSS-TTS v1.5 8B | Apache-2.0 | 0.97% | 11.1% | 3.87 | 0.58 | 27.5 GB |
| MOSS-TTS v1.5 Local, 8-bit | Apache-2.0 | 8.25% | 11.1% | 3.54 | 0.42 | 16.8 GB |
| VoxCPM2 2B, reference audio only | Apache-2.0 | 0.24% | 5.6% | 3.71 | 0.60 | 12.0 GB |
| Chatterbox Turbo | MIT | 2.43% | 22.2% | 3.42 | 0.27 | 6.8 GB |
| Chatterbox Multilingual V3 | MIT | 2.91% | 27.8% | 3.55 | 0.33 | 7.1 GB |
| ZONOS2 | Apache-2.0 | 14.1% | 16.7% | 4.08 | 0.81 | 25.6 GB |
| Confucius4-TTS | Apache-2.0 | 23.3% | 22.2% | 3.47 | 0.84 | 9.5 GB |
| Kokoro 82M, three built-in voices | Apache-2.0 | 0.73 to 2.18% | 0% | 3.58 to 3.74 | 0.03 to 0.06 | 4.5 GB |
| Qwen3-TTS, built-in Ryan and Aiden | Apache-2.0 | 3.16% and 0.73% | 0% and 16.7% | 3.49 and 3.12 | 0.28 | 12 GB |
Ten models, twelve configurations, in the order our results file lists them. ZONOS2 and Confucius4 were out on word errors alone. Kokoro is astonishingly fast (it's 82 million parameters), but we only tried it with its built-in voices.
The 20-minute drift test
Three minutes tells you how a model starts. It doesn't tell you whether it holds up. A narrator who sounds like one person at minute one and a slightly different person at minute twenty is a problem for a 22-minute episode. So the top four ran a 20-minute script through the full production chain, with the speaker similarity measured every 30 seconds against the reference. Our gate, the pass mark a model had to stay within, was a drop of 0.05.
MOSS-TTS v1.5 went from 0.917 to 0.911 and passed, with a worst drop of 0.035. VoxCPM2 and Qwen3-TTS passed too. Drama 3 dropped 0.063 and failed, needed 61 GB at peak on the 20-minute run (52.6 GB on the 3-minute one), and had one chunk that never passed its checks even after re-rolls. With the licence condition on top, it didn't make the final listen. Human listeners on the public arenas still rate it the best of the open models, and our proxy numbers can't see that. If you're choosing for yourself, it's probably worth a listen first.
A tie, and a blind A/B
That left three models. The final grid crossed four models with six reference voices and scored each pair on a composite of predicted quality and speaker match, plus pace and how smoothly chunks joined. The top three were MOSS with an earlier reference at 7.59, Qwen3-TTS with another at 7.53, and MOSS with f3 seed 44 at 7.52. Within about 0.08 of each other, so a statistical tie on proxies. VoxCPM2, the cleanest row in the 3-minute table, came fourth in this grid at 7.48. Qwen3-TTS was nearly twice as fast as MOSS, but it read about 10% faster than we wanted and read "And the service account?" without the rise of a question.
Our tooling can't listen. All the numbers above are proxies. So the decision came back to a human ear, mine. I got three unlabelled 45-second samples of the same passage, with a deadpan joke and a 0.6-second beat before the punchline, all loudness-matched. I picked A: "the Female voice A". According to the key, A was MOSS-TTS v1.5 with design f3, seed 44.
Why MOSS-TTS v1.5 8B is Elzi's voice
MOSS-TTS v1.5 is an 8B model from the OpenMOSS team, Apache-2.0, with zero-shot cloning, 31 languages, a duration control and an inline pause marker of the form [pause 3.2s]. It generates at 24 kHz and our pipeline resamples to 48 kHz. On the workhorse it peaks at 27 to 30 GB (we cap MLX at 35 GB after one run grew to 75 GB). It won on my ear, on a three-way tie, and the rest supported it: a clean licence, a passed drift test, zero gate failures over 20 minutes, faster than real time.
In production the call goes through our own script, pipeline/voice/tts.py, which loads the model through mlx-audio. This is a real command line from the build, trimmed of local paths, our job-lock wrapper and the take selector:
$VENV/bin/python pipeline/voice/tts.py \
--input content/episodes/confluence-lock-page-after-approval/script.json \
--out $OUT --only-chapter $ch --model moss8b --target-wps 2.85 --strict
The script doesn't voice a whole chapter in one go. It splits the narration into chunks, short pieces the model reads in one call, and checks each one, which the next section explains. One full re-edit voice run, voicing the rebuilt pilot after the takedown, was 118 chunks, 106 of them passing on the first take, 135 attempts in total, zero gate failures and 1,168.8 seconds of narration. Re-voicing 13 minutes of narration from cold takes about nine minutes on the defaults.
If you want to try the model without our pipeline, plain mlx-audio does it. We ran this on the workhorse on publish day, with the same pinned mlx-audio and Elzi's own reference clip and its transcript:
python -m mlx_audio.tts.generate --model OpenMOSS-Team/MOSS-TTS-v1.5 \
--text "Hi! I'm Elzi. I was born yesterday, quite literally." \
--ref_audio narrator_ref.wav --ref_text "$(cat narrator_ref.txt)" \
--output_path out --file_prefix elzi-test --verbose
Duration: 00:00:03.919
Processing time: 11.13s
Peak memory usage: 25.79GB
A clip that short is mostly warm-up, so don't judge speed from it. Memory is the useful number. From the table: Kokoro (4.5 GB peak) and Chatterbox (about 7 GB) fit on a 16 GB Mac. VoxCPM2 and Qwen3-TTS peak at about 12 GB. MOSS-TTS v1.5 peaks at 27 to 30 GB, and we cap MLX at 35 GB so a run can't balloon the way one did to 75 GB. We only ran these on Apple's MLX, so we have no numbers for NVIDIA cards.
Voice vibing: making a local TTS sound right, with emotion, pauses and pace
Choosing the model was quick. It took a day. Making it sound like someone you'd listen to for 22 minutes, which I started calling voice vibing, took the next four, and it's still going. A good model reading a script is not the same as a narrator telling you something. Most of what follows is about that gap: first the checks that make every line usable, then the takedown that showed us what "robotic" actually meant, and after that emotion and pace.
Every chunk checked before it's kept
We built the boring part first. Each chunk, the short piece of narration the model reads in one call, is generated from the fixed reference clip, so there's no drift from one chunk to the next. Then it's checked. Whisper transcribes it and the word error rate must be low, product names must be right, the speaker similarity must stay above a floor, nothing may clip. A failed chunk gets up to four re-rolls with a new seed and then gets split. "LeanZero" is written "Lean Zero" in what the model reads, because it kept saying "Lianzaro". Her own name cost five re-rolls before we accepted the transcriber's spellings of it.
Pace was the first thing I chose by ear. MOSS has a duration hint, so we locked her to a target of 2.85 words per second and played me two versions. "oh yeah 2 is much better". Time-stretching the audio afterwards was tried and dropped, because it measurably lowered quality.
The takedown taught us what robotic means
All of that was in place when the first cut of the pilot went public on the Monday morning, the one I pulled two hours later. I watched it, and my first notes, a few minutes apart, were about her voice: "the vocie was better in the sample here it's just robotic, very robotic", and "even a human would have a problem saying version 8 . 8 . 0. A human would simply say: version 8." And this one: "the voice is just reading at an extremely REGULATED RHYTHM. No human would ever read like this same beat count across."
But I also wrote "I think the voice itself was as fine as it could be seriously, the problem was the script". That turned out to be mostly right, and we only knew because we measured a human. We took NASA JPL's "What's Up" narration, which is US public domain, and ran it through the same Whisper timing method. Our sentence pauses were half as long as the human's, 300 ms against 580. Her pitch reset at each short model call. And because we had locked the pace, all the chunks ran at the same rate, with a variation of 0.005 to 0.011 where a person varies far more.
So we changed several things at once. One idea per model call, so a whole thought keeps its intonation. Pauses between sentences written into the text with MOSS's own marker, with lengths drawn from a log-normal distribution with a median of 0.55 seconds. Gaps of about 1.0 second between ideas and 1.6 seconds between chapters. And the natural pace back. On our distance-to-human measure, a single score for how far her timing sits from the human narrator's, she went from 1.84 to 0.69. Versions became "version eight", and she stopped announcing chapters, which I'd been very clear about: no YouTuber ever reads chapters out loud.
Pauses are inserted by a small pure function in tts.py. Its docstring is the clearest description of it:
"""'A. B? C.' -> 'A. [pause 0.47s] B? [pause 0.58s] C.' and the seconds added (for the length hint). Pure."""
Emotion is in the words
Better timing fixed the rhythm, but not the feeling. Later that morning, a few hours into the re-edit, I wrote "the voice has no emotion at all ... elzi jumps up in happyness, there's no emotion at all". Her face was celebrating and her voice wasn't.
An obvious lever was MOSS's instruction field: tell it to sound happy. We measured it with two speech-emotion models, and it changed which take we got while the emotion stayed where it was. MOSS v1.5 wasn't trained to combine cloning with instructions, and the project's own issue tracker says as much. What did work, a little: an emotional "lift" reference of the same voice, which widened her pitch range by 4.65 semitones without changing who she sounds like. When the emotional voice is switched on, that reference is what the emotion field picks for her positive beats: warm, amused, excited, proud and the like. It's the same field that drives her face, from the animation section above.
Rewriting the words did more than any setting. MOSS reads emotion from the words. When the lines were rewritten to sound like someone feeling something, the delivery followed. An hour and a half later: "the emotion and voice have evolved TREMENDOUSLY! EXCELLENT".
Swing, because the whole thing was too even
Emotion in single lines still left the episode as a whole sounding flat. My next note was that we "need to ensure that elzi swings a little bit voice wise to avoid being too mundane". We measured why she didn't. Her delivery didn't follow the story at all: the correlation between her energy and the story's intensity was 0.03. And her loudness was flat by construction, because our pipeline levelled each chunk to -20 LUFS. Her loudness varied by 0.14 LU from beat to beat, where the human narrator varied by 0.97.
So each beat got an energy level from 1 to 5 in a contour plan, a map of where the story should rise and fall. Takes were chosen at matched loudness, and the mix gives each level an offset between -2 and +1.5 dB. 23 of the 121 beats the script had at that point were re-voiced. After that, the correlation went to 0.24 and the loudness spread to 0.77 LU. That's still below the human, and that's deliberate: one step per episode.
"I'm killing it", and why we didn't pitch-shift it
Some lines needed work on their own. My note on the line "I'm killing it" was "she needs more emotion! It just feels stitched ... should be more like a higher pitch sound right? But don't take my word for it". Banse and Scherer's 1996 study on vocal emotion found that pride has a lower mean pitch, and our test agreed: shifting her up 2 or 3 semitones lowered the emotion models' arousal score and made her sound less like herself. A different take won instead, from an "amused" reference with seed 3234. Arousal went from 0.544 to 0.618 and her pitch range on the accented word from 8.2 to 14.0 semitones, with nothing processed.
Lead-ins, and one before and after
Then there are the little sounds people make before they speak. A person rarely starts a reaction cold. Reviewing the line where she apologises to Theo, one of our demo users, I wrote "when it says sorry THEO it is too sudden, it should at least add some start sound like MEH OR HMPF ... us humans hate monotone voices". We added ten earned lead-ins (among them Meh, Heh, Whoops, Phew, Hmm, Ugh, Okay, Uh-oh, Right) and measured which ones the model could actually say. It never voiced "Hmph" as its own sound, four tries out of four. "Meh" alone came out right four out of four.
My favourite before and after is a single interjection. Around 3:01 the script had her say "Umm... did you spot it back there?" My note: "the UHM sound at 3:01-02 it sound artificial and very bad". Measured, her "Umm" slid 13.1 semitones in 0.70 seconds, where her natural lead-ins span 1.5 to 9. The line became "Although... did you spot it back there?", which runs 0.59 seconds and moves 5.0 semitones. A new check, interj.py, now flags any interjection that glides more than 10.5 semitones.
Not everything worked. Her "Pfiuuu" got the note "that Pfiuuu was way too made-up as if some toddler learned to read and read it off of a postcard". We tried a separate dialogue model, recorded public-domain human breaths and a voice-conversion trick, and the one where a different model voiced her "hmm" was "disastrous, it sounds male". Since then we never voice an interjection with another model. We kept her own takes, softened. Having her call out "Gabrielaaa! Mihaaai!" like someone shouting across an office failed three ways (pitch-stretching, a model that shouts, a calling reference). Each passed a measurement and each failed my ear, the last one with "let's not even try it". Now she just says our names.
The stutter hunt
Not every problem that sounds like the voice is the voice. "Scrub it clean of stutter" sounds like a voice job. Mostly it wasn't. A stutter at 3:10 got flagged twice. At first it was the voice model releasing a lone /t/ half a second after the word. Once that was fixed, it was the camera zooming out and back in within about half a second. The serious judder in that cut was in the picture: footage slowed to 0.7x.
In the audio we did find our own splices, the joins where we cut a pause shorter. When the pause editor trims a silence it cuts at the aligned word times, and 15 of 106 splices cut through a breath or a word tail, like one at 1:47.52 between "exams" and "which" at -54.5 dBFS. Our splice gate now requires 10 ms of silence at or below -65 dBFS on both sides of each splice.
Keeping her natural tone
My last voice rule came from the worst moment of the review. While we rebuilt the pilot, each round ended in a full preview cut for me to watch, numbered v1, v2 and so on. On the eighth one, v8, I found a pitch-processed line I had already flagged once, and I wrote to the agents, the AI coding agents doing the build: "my feedback is not really taken to heart well enough. you learned to lie to me so well which I hate ... THIS MUST BE FIXED. AND YOU MUST UNDERSTAND FROM NOW ON TO MAINTAIN THE NATURAL TONE OF THE VOICE." The cause was a "curious" reference that had lifted whole explaining lines by about 2 to 2.7 semitones. On top of that, a pitch-shift "fix" that had only lowered one island of it. We replaced them with natural re-takes chosen against her own baseline, and added natural_tone_gate.py to check each line against it. The proof clips for those lines passed. On the first one I wrote "vo-01 is fixed indeed on the after good job".
On publish day we added one more gate, one of the automatic checks I explain properly further down, for the other version of that problem, a tone that changes suddenly between two lines. My note: "Shifts between tone is gradual and very rarely sudden ... Only anger is sudden."
From script to 4K render and Shorts
So far this post has been about Elzi herself: how she looks and how she sounds. This section is about the machinery that turns a written script into a finished 4K episode and a few vertical Shorts, the under-a-minute phone videos YouTube shows in its Shorts feed. I'll go through it in the order an episode is made: the script, the recording of our app, the live fix, the music, the jokes, the code that directs the shots, and the render.
Every model that makes her voice, her mouth and her music ran locally on the one Mac Studio. MOSS-TTS makes the voice, VoxCPM2 designed it, and Whisper and Parakeet, two speech-to-text models, check it. Qwen3-ForcedAligner matches each word to its time, Rhubarb drives the mouth and ACE-Step makes the music. The two cartoons in this post are local too, from ERNIE-Image Turbo, which won our blind test of local image models. We built the pipeline ourselves, with our agents, and it's big. When we counted on publish day, the pipeline/ folder of our video repo held 76,597 lines of code, and that includes the older recorder scenes for our app tutorials that the episode reuses. From 17 to 21 October, 640 commits touched the pipeline or the content folder.
You'll see a lot of gates from here on, the automatic checks that stop the build. The diagram below shows the stages and the gates as they were for the pilot, the first episode.
Mentions
the model is ACE-Step 1.5. it's open source under the MIT license and makes full songs with vocals and lyrics, or instrumentals, up to 10 minutes long. ace-step/ACE-Step-1.5
the-craft-of-selfteaching: lộ trình tự học Python từ cú pháp tới TDD | PaperEngineer #shorts xiaolai/the-craft-of-selfteaching: xiaolai/the-craft-of-selfteaching Tác giả: xiaolai (github.com/xiaolai) Giấy phép: repo chưa công bố giấy phép Ảnh chụp màn hình repo dùng để giới thiệu dự án; bản quyền thuộc về tác giả. #github #AI #lapTrinh #opensource Giọng đọc: pnnbao-ump/VieNeu-TTS-v3-Turbo (vieneu 3.8.3, CPU ONNX) (giấy phép Apache-2.0) huggingface.co Nhạc nền: ACE-Step 1.5 (acestep-v15-turbo) (giấy phép MIT) ace-step/ACE-Step-1.5 Theo dõi kênh Kỹ sư giấy để cập nhật kiến thức công nghệ hay mỗi ngày. [en] the-craft-of-selfteaching: One has no future if one couldn't teach themself. One has no future if one couldn't teach themself. Repo: xiaolai/the-craft-of-selfteaching [en-US] the-craft-of-selfteaching: Python Self-Study Roadmap from Syntax to TDD | PaperEngineer #shorts xiaolai/the-craft-of-selfteaching: xiaolai/the-craft-of-selfteaching Author: xiaolai (github.com/xiaolai) License: Unpublished repo Screenshots from the repo are used to showcase the project; copyright belongs to the author. #github #AI #programming #opensource Voiceover: pnnbao-ump/VieNeu-TTS-v3-Turbo (vieneu 3.8.3, CPU ONNX) (Apache-2.0 license) huggingface.co Background music: ACE-Step 1.5 (acestep-v15-turbo) (MIT license) ace-step/ACE-Step-1.5 Follow the Paper Engineer channel for daily updates on useful technology knowledge.
@davidim This is free to run locally but heavy to run on your laptop, if you give free access to it from your platform you'll be a hero ace-step/ACE-Step-1.5
Single agent loop kit: Anthropic khuyên bớt sub-agent | PaperEngineer #shorts Một agent chạy nhanh gần gấp đôi cả đội sub-agent: Anthropic chỉ cách gộp sub-agent và khi nào nên giữ chúng. anthropics/single-agent-loop-kit: anthropics/single-agent-loop-kit Bài gốc của Anthropic: claude.com Tác giả: Anthropic (github.com/anthropics) Giấy phép: Apache-2.0 Ảnh chụp màn hình repo dùng để giới thiệu dự án; bản quyền thuộc về tác giả. #github #AI #lapTrinh #opensource Giọng đọc: pnnbao-ump/VieNeu-TTS-v3-Turbo (vieneu 3.8.3, CPU ONNX) (giấy phép Apache-2.0) huggingface.co Nhạc nền: ACE-Step 1.5 (acestep-v15-turbo) (giấy phép MIT) ace-step/ACE-Step-1.5 Theo dõi kênh Kỹ sư giấy để cập nhật kiến thức công nghệ hay mỗi ngày. [en-US] Single-agent loop kit: Anthropic advises using fewer sub-agents | PaperEngineer #shorts An agent runs nearly twice as fast as an entire team of sub-agents: Anthropic shows how to fold sub-agents and when to keep them. anthropics/single-agent-loop-kit: anthropics/single-agent-loop-kit Anthropic's original article: claude.com Author: Anthropic (github.com/anthropics) License: Apache-2.0 Screenshot of the repo used to showcase the project; copyright belongs to the author. #github #AI #programming #opensource Voiceover: pnnbao-ump/VieNeu-TTS-v3-Turbo (vieneu 3.8.3, CPU ONNX) (Apache-2.0 license) huggingface.co Background music: ACE-Step 1.5 (acestep-v15-turbo) (MIT license) ace-step/ACE-Step-1.5 Follow the Paper Engineer channel for daily updates on useful technology knowledge.
しかし反、というか嫌AIの人の想像する生成AI音楽観 なんで「AIポン出し」に偏るんだろうね？ メーカー側も、もうそんな想定はしていないのに ACE-Step-1.5/docs/ja/Tutorial.md at main · ace-step/ACE-Step-1.5 · GitHub github.com/ace-step/ACE-Step-1.5/blob/main/docs/j… ここで言う「ワンクリック生成」とかもうどこも過去のもので AIユーザーはその次の「象使い」になっている
Claude Code から呼べるローカル音楽生成環境を作った（ACE-Step 1.5 + MCP + Apple Silicon） はじめに 前回の記事では、Claude Code から呼べるローカル画像生成環境を作りました。 ゲームや動画の素材をローカルだけで作りたいので、次は BGM です。 音楽生成モデルの ACE-Step 1.5 を使って、Claude Code に「子ども向けゲームのタイトル画面の BGM を、ループ用に作って」と頼むだけで曲ができる環境を作りました。 Mac だけで動き、生成はすべてローカルです。30 秒の曲が約 9 秒でできます。 実際に作った曲はこちらです（すべて AI 生成、MP3）。 カフェ風のジャズピアノトリオ（26.8 秒） 昼寝シーンのループ曲（22.0 秒、ループ加工版） 短いジングル（8.9 秒） ただ、最初に動かしたときは 30 秒の BGM 1 曲でメモリを最大 45GB 使い、画像生成サーバーと並べたら 64GB の Mac でもスワップが満杯になりました。 原因を 1 つずつ調べて対策したところ、生成中のピークは約 15GB、読み込み後の常駐は約 12GB まで下がりました。音の劣化は聞き分けられない範囲です。 この記事では、メモリを減らすまでの調べ方・原因・対策と、HTTP API と MCP で呼べる音楽生成サービスにまとめたところまでを書きます。 ! 検証環境は MacBook Pro（M2 Max / 統合メモリ 64GB）、MLX 0.30.6、PyTorch 2.10.0 です。 数値はこの環境での実測で、30 秒の BGM（同じプロンプト・seed）を 1 曲生成したときのものです。 モデルの選定 ローカルで動く音楽生成モデルはいくつかありますが、商用利用の可否がモデルごとにかなり違います。 モデル 得意なこと 重みのライセンス ACE-Step 1.5 BGM から歌入りまで。速い MIT（v1 は Apache-2.0） DiffRhythm（初代） 歌詞付きのフル尺の曲 Apache-2.0 Stable Audio Open 効果音や短いループ Stability AI Community License（年商 100 万ドル未満は無償） MusicGen BGM CC-BY-NC 4.0（非商用） ACE-Step は Apple Silicon で MLX を使って動き、速さとライセンスのバランスが良いので採用しました。 ライセンスは変わることがあるので、使う前に各公式ページで確認してください。 最初に普通に動かしたときは、30 秒の BGM が約 26 秒で生成できました。速さは十分です。問題はメモリでした。 まず、メモリを正しく測る 最初につまずいたのは、Python から見えるメモリ（RSS）があてにならないことでした。 psutil の RSS は 1GB 未満なのに、Activity Monitor では 20GB 以上と表示されます。 Mac の GPU（Metal）が確保したメモリは、RSS には出てこないためです。 そこで、Activity Monitor の「メモリ」列と同じ値（phys_footprint）を、macOS の proc_pid_rusage から直接読むことにしました。 footprint.py（抜粋・フィールドの一部を省略） import ctypes, os class RusageInfoV4(ctypes.Structure): _fields_ = [("ri_uuid", ctypes.c_uint8 * 16)] + [(n, ctypes.c_uint64) for n in ( "ri_user_time", "ri_system_time", "ri_pkg_idle_wkups", "ri_interrupt_wkups", "ri_pageins", "ri_wired_size", "ri_resident_size", "ri_phys_footprint", ..., # 省略 "ri_lifetime_max_phys_footprint", ...)] def footprint(): info = RusageInfoV4() ctypes.CDLL("/usr/lib/libproc.dylib").proc_pid_rusage(os.getpid(), 4, ctypes.byref(info)) return info.ri_phys_footprint, info.ri_lifetime_max_phys_footprint 1GB の配列を確保して値が 1.00GB 増えることと、top の表示と一致することを確かめてから使いました。 もう 1 つの教訓は、計測の間隔です。最初に top で数秒おきに測ったときはピークが 35GB に見えましたが、0.1 秒おきに測り直すと 45GB でした。 山が数秒しか続かないので、粗い計測では取りこぼします。 原因 1: 音声復元を一括で、しかも float32 で処理していた 時系列で見ると、メモリの山はすべて最後の音声復元（VAE デコード）の数秒間に集中していました（冒頭のグラフのオレンジの線）。 16.8GB から 42GB まで、右肩上がりに増えて、終わると戻ります。 ACE-Step は分割して復元する仕組み（tiled decode）を持っていますが、分割の大きさはマシンのメモリ量から自動で決まります。 64GB の Mac では 1024 フレームになり、30 秒の曲（750 フレーム）は分割されずに一括で、しかも float32 のまま復元されていました。 効いた対策は 3 つで、組み合わせるのが大事でした。 対策 設定 生成中のピーク なし - 42.1GB 分割を最小に ACESTEP_MLX_VAE_CHUNK=192 34.8GB 半精度に ACESTEP_MLX_VAE_FP16=1 29.8GB 両方＋MLX のキャッシュ上限 1GB 上 2 つ＋mx.set_cache_limit(int(1e9)) 18.9GB MLX は使い終わった領域をキャッシュとしてため込むので、上限を付けないと分割しても思ったほど下がりませんでした。 音への影響も測りました。言語モデルなしで同じ seed なら途中までの計算は同じになるので、音声復元の違いだけを比べられます。 分割サイズ 192 だけなら、出力はビット単位で同一 半精度にすると、元の音との差は SNR 約 52dB、スペクトルの差は平均 0.33dB。聞き分けられる差ではありません 原因 2: PyTorch と MLX が同じモデルを 2 重に持っていた Mac の ACE-Step は、いったん PyTorch でモデルを読み込み、それを MLX 形式に変換して生成します。 ところが変換した後も、PyTorch 側のモデルを解放していません。 PyTorch 側に残っている重みを数えると、こうなっていました。 部品 大きさ 生成中に使うか 音楽本体（DiT の decoder） 6.3GB 使わない（MLX 版を使う） 条件付けのエンコーダ 2.4GB 使う テキストエンコーダ 2.4GB 使う 音声復元（VAE） 0.3GB 使わない（MLX 版を使う） コードを追うと、PyTorch 版の decoder を呼ぶのは歌詞のタイミング解析など一部の機能だけで、BGM の生成では使っていませんでした。 そこで、MLX への変換が済んだら PyTorch 側の decoder と VAE を meta デバイスに移します。形（shape と dtype）だけを残して中身のメモリを捨てるので、型を参照しているコードも壊れません。 dit.model.decoder.to("meta") # MLX 版があるので PyTorch 側の重みは捨てる dit.vae.to("meta") gc.collect() torch.mps.empty_cache() これで重みは減ったのですが、PyTorch が Mac の GPU 側に確保したメモリは 14.2GB のままでした。実際に使っているのは 5.7GB なので、約 8.5GB は「確保して空いたけれど OS に返していない領域」です。 一度 GPU 側に載せてから捨てるので、その跡が残ります。 発想を変えて、PyTorch 側は最初から CPU に読み込むことにしました。重い計算（音楽本体と音声復元）はどのみち MLX が GPU で行うので、PyTorch に残る条件付けの処理は CPU でも十分速く、生成時間はほぼ変わりませんでした。 dit.initialize_service(project_root=ACESTEP_HOME, config_path="acestep-v15-turbo", device="cpu") # MLX は GPU で動く。PyTorch 側だけ CPU に置く 常駐は 18.5GB から 15.2GB に下がりました。 原因 3: bfloat16 の重みを float32 に広げて持っていた 音楽本体のモデルはディスク上で 4.5GB ですが、メモリ上では倍近い大きさになっていました。 配布されている重みは bfloat16 なのに、Mac では float32 に広げて読み込んでいたためです。 MLX 側の音楽本体を bfloat16 に戻してみました。 from mlx.utils import tree_map import mlx.core as mx dit.mlx_decoder.update(tree_map( lambda v: v.astype(mx.bfloat16) if isinstance(v, mx.array) and mx.issubdtype(v.dtype, mx.floating) else v, dit.mlx_decoder.parameters())) mx.eval(dit.mlx_decoder.parameters()) 常駐は 15.2GB から 12.5GB に下がり、出力は戻す前と完全に同じでした（差は 0）。 もともと bfloat16 だった値を float32 に広げ、また bfloat16 に戻しているだけなので、精度は 1 ビットも失われません。float32 は単にメモリを 2 倍使っていただけでした。 結果 対策を 1 つずつ足したときの変化です。 構成 読み込み後（常駐） 生成中のピーク 生成時間 PyTorch の decoder 解放のみ（言語モデルなし） 19.3GB 42.1GB 7.9 秒 ＋音声復元の半精度・小分割・キャッシュ上限 18.5GB 18.9GB 5.0 秒 ＋PyTorch を CPU に 15.2GB 17.9GB 5.3 秒 ＋MLX の音楽本体を bfloat16 に 12.5GB 14.7GB 5.6 秒 ＋言語モデル 0.6B 11.7GB 15.1GB 8.8 秒 PyTorch の decoder を解放しただけの状態（言語モデル 1.7B）でもピークは 45.2GB あったので、約 3 分の 1 になりました。音声復元を軽くした分、生成もむしろ速くなっています。 言語モデルは 0.6B にした ACE-Step には曲の構成や歌詞を考える言語モデルが付いていて、1.7B と 0.6B があります。 言語モデル 常駐 生成中のピーク 生成時間 1.7B 13.9GB 17.4GB 12.2 秒 0.6B 11.7GB 15.1GB 8.8 秒 なし 12.5GB 14.7GB 5.6 秒 同じプロンプト・seed で作った 4 パターン（元の精度・言語モデルなし、軽量設定で言語モデルなし / 0.6B / 1.7B）を聴き比べて、BGM 用途では 0.6B を選びました。 4 パターンの音源（MP3）はリポジトリに置いています。 1. 元の精度・言語モデルなし 2. 軽量設定・言語モデルなし 3. 軽量設定・言語モデル 0.6B 4. 軽量設定・言語モデル 1.7B 1 と 2 は言語モデルを使っていないので同じ曲で、違いは音声復元の精度だけです。3 と 4 は言語モデルが曲の構成を考えるので、同じ seed でも別の曲になります。 「なし」の常駐が 0.6B より大きく出ているのは、メモリの圧縮などによる測定のぶれの範囲です。 1 つ注意点があります。言語モデルを使うと、指定した長さより短く曲が終わり、残りが無音で埋まることがあります。30 秒を指定して、17 秒で終わったこともありました。 後述のサービスでは、末尾の無音を自動で切り、実際の長さを返すようにしています。 残った課題 読み込みの瞬間だけ、約 21GB の山が残ります（冒頭のグラフの青い線の最初の山）。 ACE-Step が内部で読み込み時の型を float32 に決め打ちしていて、外から変えられないためです。数十秒で終わる一時的な山ですが、24GB のマシンだとここでスワップが起きそうです。 ACE-Step 本体に提案できそうな改善点をまとめておきます。 MLX へ変換した後、PyTorch 側の decoder と VAE の重みを解放する Mac で bfloat16 の重みを float32 に広げない メモリの多い Mac で音声復元の分割サイズが大きくなりすぎないようにする 読み込み時に全体を float32 で展開しない 音楽生成サービスにする 軽くなったので、画像生成と同じ作りの「音楽生成サービス」にまとめました。 コードは local-musicgen で公開しています。 冒頭のサンプル曲と聴き比べ用の音源は samples/ にまとめています（すべて AI 生成、CC0）。 入口 内容 POST /generate 曲を生成して WAV を保存し、seed などのメタ情報を返す GET /health 読み込み状態とメモリ POST /load / POST /unload モデルの先読みと解放 CLI python3 gen_music.py "cozy jazz piano trio ..." --loop -o cafe.wav MCP generate_music などを Claude Code から呼べる 工夫したのは 2 点です。 使わなければ自動で解放する。 モデルは最初の生成で読み込み、一定時間（既定 5 分）使われなければ解放します。 実測では、解放でサーバーのメモリが 11.9GB から 0.5GB に戻りました。画像生成など別の重い処理と、順番にメモリを使い回せます。 ゲーム用のループ加工。 loop=true を付けると、生成した曲をループ再生しやすく加工します。 先頭の無音と、末尾のフェードアウト・無音を切る 終点を「冒頭と同じくらいの音量の位置」から選ぶ 終端の 1.5 秒を始端に等パワーでクロスフェードする 最初は 2 をやっておらず、フェードアウトの途中で切っていたため、継ぎ目の前後で音量が 8dB も違い、ループするたびに音量が跳ねていました。 終点の選び方を変えて、継ぎ目の音量差は 1.2dB まで縮まりました。クロスフェードの作り上、継ぎ目でつながる 2 サンプルは元の曲で隣り合っていたサンプルなので、波形の段差も出ません。 MCP で登録すると、Claude に「子ども向けゲームのタイトル画面の BGM を、ループ用に作って」と頼むだけで WAV ができあがります。 初回はモデルの読み込みに 40 秒ほどかかり、その後は 30 秒の曲が約 9 秒です。 まとめ ACE-Step 1.5 を Mac で動かしたときのメモリを、ピーク 45GB から 15GB 前後まで減らしました。効いたのは次の 3 つです。 音声復元を小さく分割し、半精度にし、MLX のキャッシュに上限を付ける PyTorch 側は最初から CPU に置き、MLX に変換した部品は PyTorch 側を捨てる bfloat16 で配布されている重みを float32 に広げない どれも ACE-Step 本体を書き換えず、呼び出す側の数行と環境変数で済みます。 そして一番大事だったのは、RSS ではなく phys_footprint を、0.1 秒おきに測ることでした。正しく測れれば、山がどこにあるかはすぐ見えます。 画像・音楽・音声のローカル生成がそろったので、次はこれらをつないで、短い動画をローカルだけで作るところを書く予定です。
@heyandras ACE-Step 1.5 2B Turbo model should run on it, if you want to create music: ace-step/ACE-Step-1.5
12. ACE-Step 1.5 ace-step/ACE-Step-1.5 生成游戏音乐，RTX 3090 上一首歌不到 10 秒。
Elzi: Building an AI YouTube Narrator on a Local TTS Model So why Elzi, the cartoon robot who now narrates our YouTube videos? Why are we putting out more AI slop, right? These days people assume most of what they see online is AI slop, and a cartoon narrator voiced by a local TTS model sounds like more of the same. There was always great human content out there, and there was always a ton of human slop too, long before anyone could type a prompt. Is Elzi slop? Hell no. Elzi narrates our first YouTube episode, which I'll call the pilot from here on. I wrote 150 review notes on that pilot, and I pulled its first cut off YouTube two hours after it went up. Part of the episode shows us fixing a real bug in one of our apps, on camera. An early cut showed that fix being made by typing a request to an AI coding agent. I refused it, and we made it a real fix in a real code editor, with real tests going green. It took five days, from the first message I sent the AI agents that built her on 16 October to the public episode on the morning of 21 October. Is it perfect? No, and a good part of this is about where it isn't. Let her say hello first. Gabriela and I are LeanZero, a two-person company that makes Atlassian apps. This is how she opens the pilot, 17 seconds in: > Hi! I'm Elzi, nice to meet you! I was born yesterday, quite literally, and I'll try my best to present LeanZero and do what Gabriela and Mihai couldn't, which is talk about their work in a YouTube video. These two are proper slave drivers: they told me to break the ice and present one of their products, on my first day, with no training whatsoever. A few lines later: > Also, nobody has mentioned a salary yet. I'm keeping notes.  So who is she? Elzi is the small voxel robot who narrates our product videos. She talks with a local TTS (text-to-speech) model that runs on our own Mac Studio, she lip-syncs with an open-source tool, and she's drawn as flat polygons that only look like voxels. Her first episode is a 22-minute walkthrough of locking a Confluence page after approval with our app Sentinel Vault: someone edits an approved page, and the edit gets put back to the approved version. That episode went public once before, as a 34-minute cut, and I pulled it two hours later because she sounded like a robot reading a manual. youtube.com The episode has its own page on our site, A Confluence Approval Workflow That Survives an Edit, with chapters and a transcript, plus the exact settings it shows. This post tells the whole story, so you can read it in order or jump. If you came here for the voice: it's MOSS-TTS v1.5, an 8B (eight billion parameter) model under the Apache-2.0 licence, running through mlx-audio, an open-source speech library built on Apple's MLX framework, on a Mac Studio. The section further down where we tested ten models side by side has the full results, a command you can run, and why this one won. If you came because you use our apps: Elzi is meant to be the one who shows you around them one day, so how she sounds is your business too. Why an AI YouTube narrator instead of filming ourselves Before the how, the why. Gabriela and I are two Eastern Europeans in Romania, and neither of us is a YouTuber. Two people can't run a YouTube channel the way YouTubers do. But a product with no videos and no reviews gets skipped by most companies looking for an app, which makes sense, I'd skip it too. We did the maths on filming ourselves. Scripting, recording, re-recording because one of us stumbled, editing, captions: two weeks or more per video, for two people who also build the apps and do the support and the client work. We'd make one video and then stop. So we decided to build a narrator instead, and to have AI do most of the building. To be clear about who did what: AI coding agents wrote almost all of the pipeline's code and ran it. By pipeline I mean the chain of scripts that goes from a written script to a finished video: recording the app, voicing Elzi, animating her, editing and rendering. I worked with those agents in one long chat session and wrote the notes. Gabriela and I made the calls. Making money on YouTube isn't the point either. I wrote it to the agents in that session on day two: "I don't care about monetisation btw, this is not why we're doing it! it's for our brand and our domain authority". Elzi exists so that people find out LeanZero exists. She's a way for us to say things in public without running ourselves into the ground. And we share what we learn, like we always do. Building a narrator with AI models taught us a lot of small, specific things about local TTS and about what makes a synthetic voice sound robotic, and you'll find them below with the numbers. There's an obvious tension in all this: a video made by AI, for people who are tired of AI. One thing I wrote to the agents during the build stuck: > what I want is not to mask the AI condition, but to make it more beautiful, give it a place amongst us humans and make youtube content made by AI likeable because everyone hates it, everyone hates AI slop I think that's the right bar. The question we set ourselves was whether she can still be worth twenty minutes of your time. Published at 5 a.m., pulled at 7 The first time we tried to answer that question, the answer was no. I'm least proud of this part, so you get it early. Before bed on Sunday, 18 October, I wrote to the agents: "Feel free to just push it on youtube. I am sure it will be awkward and not perfect but who cares this is the first run." The 33:58 cut finished rendering overnight. Before anything goes out, an AI reviewer acts as our red team, meaning its job is to attack the video and find what's wrong with it. It scored this cut 80 out of 100, and the episode went public with four Shorts, short vertical clips cut from it, at 5:13 on Monday morning, our time. I watched it at seven and ordered it down. What I wrote to the agents was "too robotic, starts talking about chapters ... The gags can ONLY work if they have sound otherwise cut them out makes no sense." By chapters I meant that Elzi kept announcing which part of the video we were in, which no YouTuber does. The gags are the short comic moments and cutaways we put in to make it fun, and some of them made no sense without sound. The episode and its four Shorts were all private again within a minute or so. When we sat down to plan the re-edit and read back through that first cut, the problem was plain: it was a 34-minute manual. Every frame had the same layout. The narration named every click, read version numbers out digit by digit and announced the structure about 15 times. A promise made in the first 13 seconds paid off at 28:01, so a viewer had to wait almost half an hour for the point. We didn't restart. My instruction was to re-edit the same footage, not start over. The re-edit started from one question the whole video answers. We set a target of 18 to 22 minutes, no on-screen chapter rail (the strip of chapter titles the first cut kept on screen), and gags only where something actually goes sideways. The other thing the first cut lacked was an introduction: Elzi never said who she was. I wrote her introduction together with the agent drafting her lines. The framing that came out of it is the "born yesterday" idea. I put it to that agent like this: "Think of this all as context where humans are as blank as you and they will learn from this video just as much as you learned." She learned everything from scratch, so she explains it from scratch. That's also the standard we went in with for the second attempt: > we don't need a perfect video, but at least one that's not miserable. Claptrap, voxels, and where Elzi's look came from Now to Elzi herself, starting with how she looks. Elzi's look starts with Claptrap, the small talking robot from Borderlands. If you've played any of the games you know him: a yellow box on one wheel, one big blue eye, a running commentary nobody asked for. Those games are drawn in a hand-drawn, comic-book look with heavy black outlines. I'll call it cel-shaded (Gearbox, the studio behind the games, prefers "concept art style", but come on).  Claptrap himself started as a joke. Brian Martel of Gearbox told GamesBeat in 2012 that concept artist Lorin Wood "posted a quick sketch of him as a joke" and "We all instantly fell in love when we saw it". According to Wikipedia, Wood set out to draw the most generic robot he could. His first voice, David Eddings, was a Gearbox vice president. A throwaway robot, voiced by someone who already worked in the building, became one of the series' most popular characters by fan vote. So I figured ours could start as a joke too.   What Elzi borrowed from Claptrap, and what she dropped Side by side, you can see what carried over. She took his job first: the funny little robot who talks you through things. She has his antenna, a rod with an orange voxel cube on top. She has two expressive arms, though hers end in orange voxel mitts, and she only points when there is something on screen to point at. She has the bold black outline, a 5-unit ink line on each face, which is the clearest visual link between the two. And she has the self-mocking energy, which you've already heard: the slave drivers, and the salary she's keeping notes on. She dropped a lot more. No wheel. Two rectangular LED eyes instead of one round one, because an LED face reads emotion at phone size and her mouth positions are the clearest changes we could draw. Her head is the three-faced LeanZero cube from our site's logo, sitting on a navy torso that gets wider toward the bottom, where Claptrap narrows. No weathered yellow: all her colours are tokens from leanzero.net (cyan #22d3ee, blue #1d4ed8, violet #7c3aed, orange #f97316 for the joints and the L plate). He is a he, voiced by a man. She is a she, voiced by a local model. And she's meant to be likeable. Claptrap's whole act, by most accounts, is being obnoxious, and Wikipedia sums up his reception as strongly split between fans and critics. We didn't want that half. Her name is just LeanZero's initials said aloud: L, Z. Her working name for the first day was Cubit. Then I asked the agents whether it was really original, and it turned out not to be: there's a Nintendo 3DS robot called Cubit, Sega has Cubot, and there are CUBIT trademarks. We checked Elzi against USPTO, EUIPO's TMview, OSIM in Romania and WIPO before keeping it. There's one soft risk, which is that an English ear hears "Elsie", so we always write it Elzi and say it EL-zee. Why voxels, and why ours are fake The other half of her look is the voxel style, and it needs a quick explanation. A voxel is a value on a 3D grid, the way a pixel is a value on a 2D one, and voxel art is the 3D cousin of pixel art: everything built from coloured cubes. Minecraft is the famous example. It also cheats, which I find reassuring: it stores the world as voxels but draws each block as ordinary polygons. Elzi cheats more. There's no Blender in this project, no MagicaVoxel, no three.js, no 3D file anywhere. She's flat 2D SVG polygons drawn by React inside Remotion, a library that turns React components into video frames. Her voxel look comes from an oblique extrusion. Each part is a flat front polygon, pushed back along one fixed vector, so the visible back faces show as a lit top and a shaded right side, all outlined in ink. This is the whole trick, from concepts.ts, the file in our code where her concepts are drawn: // the ONE oblique extrusion vector (per unit of depth): up and to the right, so tops and right sides show export const E: [number, number] = [1, -0.85]; One vector means one light direction for the whole character. Next to it we wrote down that you rotate or flip a part before extruding it, never after, "or the light turns with it". That same voxel language draws the highlights in the video: the arrows, the corner-bracket frames, the amber spotlight on the payoff. Why bother faking it? Partly it's about reading at small sizes. Valve's paper on the rendering of Team Fortress 2 says its characters were designed so that "only in silhouette with no internal shading at all, the characters are readily identifiable". A chunky, outlined robot follows the same principle. And we measured it for phone screens. Watch the episode inline on a phone held upright and her corner tile, the small box in the corner of the frame where she stands, is about 64 points wide. That leaves her features about 3 pixels across, so you see presence and a moving mouth and not much else. Full screen in landscape it's about 140 points and reads fine. She went from idea to a finished pose sheet, the reference drawing of every pose, mouth and mood a character can have, in one day. Around midday on 17 October we had three concepts: Ell, the LeanZero L as a paper-cut slab; Cubit, the engineer robot with the cube head; and Facet, a cut violet gem with a spark crest. Of the three, the LED face read best at those small sizes. Cubit won, my note on it was "I love the voxel art it's amazing", and the same afternoon she was rebuilt with one light direction and the site's exact colours. Her celebration yellow changed from #FFD400 to the site's amber #fbbf24, because #FFD400 isn't a colour we use anywhere.   The voxel character and her animation: lip sync, idle, emotions and camera A pose sheet is a drawing, though, and a narrator has to move. So the next job was turning that sheet into a character that talks and blinks and keeps out of the way of the screen she's explaining. This section is how she's animated, and the voice comes after it. Elzi's renderer, the code that draws her in each frame of the video, is four files: Mascot.tsx, mascot/state.ts, mascot/concepts.ts and Voxel.tsx, about 1,800 lines together when we counted on publish day. Her rig, meaning the set of parts she's assembled from, lives in one 600 by 600 space with swappable parts: a head and body, eyes with LED pupils that open and close, brows in three positions, nine mouths and four arm poses (neutral, pointing, facepalm, celebrate). A script exports them as about 20 standalone SVGs per concept. Everything she does is a pure function of time. Given a frame number, mascotStateAt returns what she looks like. So any part of the episode can be rendered by itself and stitched to the rest, because nothing depends on the frames before it. With renders that take an hour and a half, we leaned on that a lot.  Lip sync with Rhubarb The first thing a talking character needs is a mouth that matches the words. Hers is driven by Rhubarb Lip Sync 1.14.0, an MIT-licensed tool that reads a WAV file and outputs which mouth to show when. It uses the classic animation set, mouths A to H plus X for rest, which is why her pose sheet has nine mouths. Its macOS release is x86-only. Our Mac Studio (we call it the workhorse, and it's the machine that does all the heavy work in this story) has no Rosetta. So we built it from source, which took about three minutes. It analysed 8.7 seconds of speech in 3.5 seconds. We always change two settings. We hand it the transcript (-d dialog.txt) so it isn't guessing the words, and we turn on the extended mouth set with --extendedShapes GHX. Our episode script is split into beats, a beat being one short line of narration with its own instructions, and each beat is voiced into its own WAV file. Our episode builder runs Rhubarb on each beat's WAV, caches the result by content hash and shifts the cues into episode time. Then the mouth is whatever cue Rhubarb gave for that frame. Over the pilot I wrote 150 review notes on what was wrong, and lip-sync drift never came up as a defect in any of them, which surprised me more than anything else in this project. My guess is that handing Rhubarb the transcript did most of that work. Moods, and one field for her face and her voice A mouth that moves in time still looks dead without a face that reacts, so on top of lip sync she has moods. In the pilot she has seven: idle, explain, think, surprised, facepalm, celebrate, eyeroll. An eighth, uh-oh, was added after publishing for the next episode. Celebrate puts her arms up, bounces her, throws voxel confetti behind her and turns her tile amber. Each mood change pops with a 6% squash. Her blinks are seeded, 2.0 to 4.8 seconds apart and 0.13 seconds long, so they look random without being random from one render to the next. The decision I'd copy into any other project: one emotion field drives both her face and her voice. Each beat of the script can carry an emotion. Then content/emotion.py maps it to Elzi's mood and, when we switch on an optional livelier set of voice references, to the reference clip her voice is cloned from. (Her voice is cloned from a short sample recording, a reference clip, and the voice sections below explain how that works.) A beat whose explicit animation tag contradicts its emotion is a validation error, so her face can't celebrate while the script says she's tense. Here are two beats from the pilot's script.json: { "id": "t14b", "chapter": "two-seats", "narration": "Oh... I hope I'm doing okay. I really need this job.", "emotion": { "name": "tense", "intensity": 0.6, "then": "relieved" }, "signature": { "inhale": true }, "aside": true } { "id": "L7c", "chapter": "ship-fix", "narration": "I'm killing it.", "emotion": { "name": "proud", "intensity": 0.8 } } The pilot script has 117 beats and 3,257 words. Counted by emotion, 29 are neutral, 23 warm, 11 excited, 10 wry, 8 proud, 7 amused, 6 curious, 6 tense and 5 confident, with a few others after that. In the final edit that became 160 stretches of explaining, 22 of thinking, 9 celebrations, 4 idles, 2 surprises and one eye-roll. The bow, the throw and the aside Most of her moods just follow the script. Some moves have exact timings, because she does them in sync with something else. Her bow is timed to her entrance sting, a 1.25-second sound of FM bells and a cube "thock" that our own script synthesises from fixed seeds, so it's the same sound in each episode. Her bow starts 0.10 seconds before the first note and peaks on the third. Her throw has its own constants, and both sit in state.ts: export const BOW = { pre: 0.10, apex: 0.21, hold: 0.46, up: 1.22 }; export const TOSS = { wound: 0.16, swing: 0.10, follow: 0.15, back: 0.5 }; The throw needs a word of context. When the video wants you to look at a button or a setting, a voxel highlight appears on it, an arrow or a frame in the same blocky style as Elzi. The throw was my idea, on the evening of 19 October. Instead of a highlight just appearing, she lobs a voxel ball in the highlight's colour and it turns into the highlight where it lands. My condition was that it mustn't turn her into "a jittery mess", so it only happens for the first highlight after ten quiet seconds, never during a zoom and never near another attention cue. In the pilot (watch it at 6:12) she winds up at 6:12.7, lets go at 6:13.1, and the ball hits the setting at 6:13.6, the one called "Require approval before publishing".  Gag timing has a rule too. Now and then the video cuts away to a few seconds of an old film clip for a joke, then cuts back to the app. Her reaction only happens in the 0.4 to 0.7 seconds after we cut back. A reaction before the cutaway gives the joke away. If I could keep one bit, it'd be the aside, the moment she stops explaining and speaks to you directly. At 11:43 in the pilot she turns to the camera nervously and says "Oh... I hope I'm doing okay. I really need this job." It's the beat marked "aside": true in the JSON above. My note when we built it: "it needs an inhale and exhale to make it more convincing - usually when you say stuff like this it's like breaking the 4th wall." So she breathes in first. Afterwards there's a 1.2-second composure move: she straightens, shakes her head, closes her eyes. Even the breath sounds are cut from her own TTS output.  Talking hands, idle moves and where she stands The last pieces are what she does with her hands, where she sits on screen, and what she does when nobody needs her. She gestures with both hands on stressed words, sized by the emotion, and only points when a voxel highlight exists. A build check called check-elzi.ts runs the renderer's own state code over each frame and fails the build if her arm points at nothing. We call checks like that gates. If one fails, no video comes out until it's fixed. That came from a single note: "It should point its hand only when a voxel shows". Turning a note like that into an automatic check is a pattern you'll see all through this article. Once a mistake is caught, the build refuses to let it back in. Her tile, the small square she stands in, sits bottom right by default and grows during the intro, from 380 to 470 design pixels. When something on screen is about to happen under her, she moves or hides, with a 0.18-second scale out and scale in. Idle came on publish day. My note on the final cut said "there's still awkward moments where elzi is just waiting for nothing ... In video games this is what they do to fill up awkward moments". So idle.py now fills each dead stretch with a context-chosen idle move. A glance has the pupils lead by 80 to 120 ms and the head follow 0.1 seconds later. There's also a lean-in before a payoff, a fidget while something loads, a nod. We borrowed from Live2D's blink and breathing constants, Mario's idle head turns and Valve's idle-speech timers. A dry run on the published cut found 13 dead runs and gave each one an idle. That's for the next episode. The local TTS model bake-off: 10 open-weights TTS models on a Mac Studio All of that animation hangs off one thing: her voice. Lip sync reads it, the moods are timed to it, and the pilot was pulled because of it. So here is how we chose the model that speaks for her. TTS means text to speech, a model that reads text aloud, and local means it runs on our own machine rather than in someone's cloud. Short answer, if that's all you came for: Elzi's voice is MOSS-TTS v1.5 8B, Apache-2.0, run through mlx-audio on a Mac Studio M3 Ultra. It passed our 20-minute drift test, renders faster than real time (a real-time factor of 0.58) and peaks at 27 to 30 GB. On the numbers it tied with two others, so a blind A/B by ear decided it. The full table is below. Her first placeholder voice was the say command that ships with macOS. My verdict on the very first mascot test, which used it: "cubit is the best but the voice keep in mind it's horrible no one ever will look at our videos with a vocie like that!" (Cubit was still her working name then.) That voice was banned from anything I'd see again. The question became which voice to use instead. When asked whether to use a cloud TTS service, I answered "we use local but please review the best solution out there". So we ran a bake-off, a side-by-side test of every serious candidate, on 17 October, on the workhorse: a Mac Studio with an M3 Ultra and 96 GB of memory. The licence filter came first A model we can't use commercially is no use to a company's YouTube channel, so the first cut was the licence. Out went Higgs Audio v3 (research and non-commercial), Breeze TTS 2 (non-commercial and CUDA-only), Fish S2 Pro (commercial use needs a separate licence), Voxtral TTS, Llasa, OmniVoice plus F5-TTS (non-commercial weights), Spark-TTS, IndexTTS, XTTS, and VibeVoice, whose TTS code Microsoft removed in September 2025. Fish Audio's Drama 3, a different Fish model from S2 Pro, stayed in on a condition: its licence allows commercial use only while you and your affiliates make under USD 500,000 a year. It was the top open-weights model on the Artificial Analysis TTS leaderboard at the time, with an Elo of 1244. Local TTS voice cloning: one designed voice, cloned by every model With the licences sorted, we needed a fair way to compare the survivors. Most of these models clone a voice: you give them a few seconds of someone speaking, the reference clip, and they read new text in that voice. We didn't want to clone a real person. So the narrator is synthetic twice over. First, VoxCPM2, a 2B Apache-2.0 model from OpenBMB that can design a voice from a text description, made 25 candidate reference clips from 11 descriptions. Design f3 with seed 44 won. Its description, word for word: "A mature female narrator in her fifties, kind and steady, warm alto voice, slow measured pace, subtle wry humour, natural and human, not an announcer". A detail I didn't expect: the same description with a different seed gives you a different person. A speaker-similarity model, which scores how much two recordings sound like the same speaker, rated the seeds of one description at 0.1 to 0.46 against each other, where the same speaker scores around 0.9. A designed voice is really the description plus the seed. Then all the candidate models cloned that same reference clip, so we compared models and not voices. Everything ran through mlx-audio, an open-source TTS and speech library built on Apple's MLX (we compared MLX and GGUF builds of the same model on Apple Silicon in an earlier post), pinned to one commit, in its own Python 3.12 environment on an external SSD plugged into the workhorse. The results on a 3-minute script Each model read the same 3-minute script in that same voice. First takes, one seed, no re-rolls. WER is the word error rate when Whisper, an open speech-to-text model, transcribes the audio back. Term errors are product names said wrong, almost all of them "LeanZero" coming out as "Lianzaro" or "Leonzero". UTMOSv2 is a predicted listener score. Real-time factor is generation time divided by audio length, so under 1 is faster than real time. | Model | Licence | WER (Whisper) | Term errors | UTMOSv2 | Real-time factor | Peak memory | |---|---|---|---|---|---|---| | VoxCPM2 2B, reference as continuation | Apache-2.0 | 0.00% | 0% | 4.09 | 0.66 | 12.7 GB | | Qwen3-TTS 1.7B, cloning | Apache-2.0 | 0.24% | 11.1% | 4.14 | 0.33 | 12.2 GB | | Fish Audio Drama 3, 8-bit | Fish Audio's FACL, revenue-conditional | 0.00% | 0% | 3.78 | 0.60 | 52.6 GB | | MOSS-TTS v1.5 8B | Apache-2.0 | 0.97% | 11.1% | 3.87 | 0.58 | 27.5 GB | | MOSS-TTS v1.5 Local, 8-bit | Apache-2.0 | 8.25% | 11.1% | 3.54 | 0.42 | 16.8 GB | | VoxCPM2 2B, reference audio only | Apache-2.0 | 0.24% | 5.6% | 3.71 | 0.60 | 12.0 GB | | Chatterbox Turbo | MIT | 2.43% | 22.2% | 3.42 | 0.27 | 6.8 GB | | Chatterbox Multilingual V3 | MIT | 2.91% | 27.8% | 3.55 | 0.33 | 7.1 GB | | ZONOS2 | Apache-2.0 | 14.1% | 16.7% | 4.08 | 0.81 | 25.6 GB | | Confucius4-TTS | Apache-2.0 | 23.3% | 22.2% | 3.47 | 0.84 | 9.5 GB | | Kokoro 82M, three built-in voices | Apache-2.0 | 0.73 to 2.18% | 0% | 3.58 to 3.74 | 0.03 to 0.06 | 4.5 GB | | Qwen3-TTS, built-in Ryan and Aiden | Apache-2.0 | 3.16% and 0.73% | 0% and 16.7% | 3.49 and 3.12 | 0.28 | 12 GB | Ten models, twelve configurations, in the order our results file lists them. ZONOS2 and Confucius4 were out on word errors alone. Kokoro is astonishingly fast (it's 82 million parameters), but we only tried it with its built-in voices. The 20-minute drift test Three minutes tells you how a model starts. It doesn't tell you whether it holds up. A narrator who sounds like one person at minute one and a slightly different person at minute twenty is a problem for a 22-minute episode. So the top four ran a 20-minute script through the full production chain, with the speaker similarity measured every 30 seconds against the reference. Our gate, the pass mark a model had to stay within, was a drop of 0.05. MOSS-TTS v1.5 went from 0.917 to 0.911 and passed, with a worst drop of 0.035. VoxCPM2 and Qwen3-TTS passed too. Drama 3 dropped 0.063 and failed, needed 61 GB at peak on the 20-minute run (52.6 GB on the 3-minute one), and had one chunk that never passed its checks even after re-rolls. With the licence condition on top, it didn't make the final listen. Human listeners on the public arenas still rate it the best of the open models, and our proxy numbers can't see that. If you're choosing for yourself, it's probably worth a listen first. A tie, and a blind A/B That left three models. The final grid crossed four models with six reference voices and scored each pair on a composite of predicted quality and speaker match, plus pace and how smoothly chunks joined. The top three were MOSS with an earlier reference at 7.59, Qwen3-TTS with another at 7.53, and MOSS with f3 seed 44 at 7.52. Within about 0.08 of each other, so a statistical tie on proxies. VoxCPM2, the cleanest row in the 3-minute table, came fourth in this grid at 7.48. Qwen3-TTS was nearly twice as fast as MOSS, but it read about 10% faster than we wanted and read "And the service account?" without the rise of a question. Our tooling can't listen. All the numbers above are proxies. So the decision came back to a human ear, mine. I got three unlabelled 45-second samples of the same passage, with a deadpan joke and a 0.6-second beat before the punchline, all loudness-matched. I picked A: "the Female voice A". According to the key, A was MOSS-TTS v1.5 with design f3, seed 44.  Why MOSS-TTS v1.5 8B is Elzi's voice MOSS-TTS v1.5 is an 8B model from the OpenMOSS team, Apache-2.0, with zero-shot cloning, 31 languages, a duration control and an inline pause marker of the form [pause 3.2s]. It generates at 24 kHz and our pipeline resamples to 48 kHz. On the workhorse it peaks at 27 to 30 GB (we cap MLX at 35 GB after one run grew to 75 GB). It won on my ear, on a three-way tie, and the rest supported it: a clean licence, a passed drift test, zero gate failures over 20 minutes, faster than real time. In production the call goes through our own script, pipeline/voice/tts.py, which loads the model through mlx-audio. This is a real command line from the build, trimmed of local paths, our job-lock wrapper and the take selector: $VENV/bin/python pipeline/voice/tts.py \ --input content/episodes/confluence-lock-page-after-approval/script.json \ --out $OUT --only-chapter $ch --model moss8b --target-wps 2.85 --strict The script doesn't voice a whole chapter in one go. It splits the narration into chunks, short pieces the model reads in one call, and checks each one, which the next section explains. One full re-edit voice run, voicing the rebuilt pilot after the takedown, was 118 chunks, 106 of them passing on the first take, 135 attempts in total, zero gate failures and 1,168.8 seconds of narration. Re-voicing 13 minutes of narration from cold takes about nine minutes on the defaults. If you want to try the model without our pipeline, plain mlx-audio does it. We ran this on the workhorse on publish day, with the same pinned mlx-audio and Elzi's own reference clip and its transcript: python -m mlx_audio.tts.generate --model OpenMOSS-Team/MOSS-TTS-v1.5 \ --text "Hi! I'm Elzi. I was born yesterday, quite literally." \ --ref_audio narrator_ref.wav --ref_text "$(cat narrator_ref.txt)" \ --output_path out --file_prefix elzi-test --verbose Duration: 00:00:03.919 Processing time: 11.13s Peak memory usage: 25.79GB A clip that short is mostly warm-up, so don't judge speed from it. Memory is the useful number. From the table: Kokoro (4.5 GB peak) and Chatterbox (about 7 GB) fit on a 16 GB Mac. VoxCPM2 and Qwen3-TTS peak at about 12 GB. MOSS-TTS v1.5 peaks at 27 to 30 GB, and we cap MLX at 35 GB so a run can't balloon the way one did to 75 GB. We only ran these on Apple's MLX, so we have no numbers for NVIDIA cards. Voice vibing: making a local TTS sound right, with emotion, pauses and pace Choosing the model was quick. It took a day. Making it sound like someone you'd listen to for 22 minutes, which I started calling voice vibing, took the next four, and it's still going. A good model reading a script is not the same as a narrator telling you something. Most of what follows is about that gap: first the checks that make every line usable, then the takedown that showed us what "robotic" actually meant, and after that emotion and pace. Every chunk checked before it's kept We built the boring part first. Each chunk, the short piece of narration the model reads in one call, is generated from the fixed reference clip, so there's no drift from one chunk to the next. Then it's checked. Whisper transcribes it and the word error rate must be low, product names must be right, the speaker similarity must stay above a floor, nothing may clip. A failed chunk gets up to four re-rolls with a new seed and then gets split. "LeanZero" is written "Lean Zero" in what the model reads, because it kept saying "Lianzaro". Her own name cost five re-rolls before we accepted the transcriber's spellings of it. Pace was the first thing I chose by ear. MOSS has a duration hint, so we locked her to a target of 2.85 words per second and played me two versions. "oh yeah 2 is much better". Time-stretching the audio afterwards was tried and dropped, because it measurably lowered quality. The takedown taught us what robotic means All of that was in place when the first cut of the pilot went public on the Monday morning, the one I pulled two hours later. I watched it, and my first notes, a few minutes apart, were about her voice: "the vocie was better in the sample here it's just robotic, very robotic", and "even a human would have a problem saying version 8 . 8 . 0. A human would simply say: version 8." And this one: "the voice is just reading at an extremely REGULATED RHYTHM. No human would ever read like this same beat count across." But I also wrote "I think the voice itself was as fine as it could be seriously, the problem was the script". That turned out to be mostly right, and we only knew because we measured a human. We took NASA JPL's "What's Up" narration, which is US public domain, and ran it through the same Whisper timing method. Our sentence pauses were half as long as the human's, 300 ms against 580. Her pitch reset at each short model call. And because we had locked the pace, all the chunks ran at the same rate, with a variation of 0.005 to 0.011 where a person varies far more. So we changed several things at once. One idea per model call, so a whole thought keeps its intonation. Pauses between sentences written into the text with MOSS's own marker, with lengths drawn from a log-normal distribution with a median of 0.55 seconds. Gaps of about 1.0 second between ideas and 1.6 seconds between chapters. And the natural pace back. On our distance-to-human measure, a single score for how far her timing sits from the human narrator's, she went from 1.84 to 0.69. Versions became "version eight", and she stopped announcing chapters, which I'd been very clear about: no YouTuber ever reads chapters out loud. Pauses are inserted by a small pure function in tts.py. Its docstring is the clearest description of it: """'A. B? C.' -> 'A. [pause 0.47s] B? [pause 0.58s] C.' and the seconds added (for the length hint). Pure.""" Emotion is in the words Better timing fixed the rhythm, but not the feeling. Later that morning, a few hours into the re-edit, I wrote "the voice has no emotion at all ... elzi jumps up in happyness, there's no emotion at all". Her face was celebrating and her voice wasn't. An obvious lever was MOSS's instruction field: tell it to sound happy. We measured it with two speech-emotion models, and it changed which take we got while the emotion stayed where it was. MOSS v1.5 wasn't trained to combine cloning with instructions, and the project's own issue tracker says as much. What did work, a little: an emotional "lift" reference of the same voice, which widened her pitch range by 4.65 semitones without changing who she sounds like. When the emotional voice is switched on, that reference is what the emotion field picks for her positive beats: warm, amused, excited, proud and the like. It's the same field that drives her face, from the animation section above. Rewriting the words did more than any setting. MOSS reads emotion from the words. When the lines were rewritten to sound like someone feeling something, the delivery followed. An hour and a half later: "the emotion and voice have evolved TREMENDOUSLY! EXCELLENT". Swing, because the whole thing was too even Emotion in single lines still left the episode as a whole sounding flat. My next note was that we "need to ensure that elzi swings a little bit voice wise to avoid being too mundane". We measured why she didn't. Her delivery didn't follow the story at all: the correlation between her energy and the story's intensity was 0.03. And her loudness was flat by construction, because our pipeline levelled each chunk to -20 LUFS. Her loudness varied by 0.14 LU from beat to beat, where the human narrator varied by 0.97. So each beat got an energy level from 1 to 5 in a contour plan, a map of where the story should rise and fall. Takes were chosen at matched loudness, and the mix gives each level an offset between -2 and +1.5 dB. 23 of the 121 beats the script had at that point were re-voiced. After that, the correlation went to 0.24 and the loudness spread to 0.77 LU. That's still below the human, and that's deliberate: one step per episode. "I'm killing it", and why we didn't pitch-shift it Some lines needed work on their own. My note on the line "I'm killing it" was "she needs more emotion! It just feels stitched ... should be more like a higher pitch sound right? But don't take my word for it". Banse and Scherer's 1996 study on vocal emotion found that pride has a lower mean pitch, and our test agreed: shifting her up 2 or 3 semitones lowered the emotion models' arousal score and made her sound less like herself. A different take won instead, from an "amused" reference with seed 3234. Arousal went from 0.544 to 0.618 and her pitch range on the accented word from 8.2 to 14.0 semitones, with nothing processed.  Lead-ins, and one before and after Then there are the little sounds people make before they speak. A person rarely starts a reaction cold. Reviewing the line where she apologises to Theo, one of our demo users, I wrote "when it says sorry THEO it is too sudden, it should at least add some start sound like MEH OR HMPF ... us humans hate monotone voices". We added ten earned lead-ins (among them Meh, Heh, Whoops, Phew, Hmm, Ugh, Okay, Uh-oh, Right) and measured which ones the model could actually say. It never voiced "Hmph" as its own sound, four tries out of four. "Meh" alone came out right four out of four. My favourite before and after is a single interjection. Around 3:01 the script had her say "Umm... did you spot it back there?" My note: "the UHM sound at 3:01-02 it sound artificial and very bad". Measured, her "Umm" slid 13.1 semitones in 0.70 seconds, where her natural lead-ins span 1.5 to 9. The line became "Although... did you spot it back there?", which runs 0.59 seconds and moves 5.0 semitones. A new check, interj.py, now flags any interjection that glides more than 10.5 semitones. Not everything worked. Her "Pfiuuu" got the note "that Pfiuuu was way too made-up as if some toddler learned to read and read it off of a postcard". We tried a separate dialogue model, recorded public-domain human breaths and a voice-conversion trick, and the one where a different model voiced her "hmm" was "disastrous, it sounds male". Since then we never voice an interjection with another model. We kept her own takes, softened. Having her call out "Gabrielaaa! Mihaaai!" like someone shouting across an office failed three ways (pitch-stretching, a model that shouts, a calling reference). Each passed a measurement and each failed my ear, the last one with "let's not even try it". Now she just says our names. The stutter hunt Not every problem that sounds like the voice is the voice. "Scrub it clean of stutter" sounds like a voice job. Mostly it wasn't. A stutter at 3:10 got flagged twice. At first it was the voice model releasing a lone /t/ half a second after the word. Once that was fixed, it was the camera zooming out and back in within about half a second. The serious judder in that cut was in the picture: footage slowed to 0.7x. In the audio we did find our own splices, the joins where we cut a pause shorter. When the pause editor trims a silence it cuts at the aligned word times, and 15 of 106 splices cut through a breath or a word tail, like one at 1:47.52 between "exams" and "which" at -54.5 dBFS. Our splice gate now requires 10 ms of silence at or below -65 dBFS on both sides of each splice. Keeping her natural tone My last voice rule came from the worst moment of the review. While we rebuilt the pilot, each round ended in a full preview cut for me to watch, numbered v1, v2 and so on. On the eighth one, v8, I found a pitch-processed line I had already flagged once, and I wrote to the agents, the AI coding agents doing the build: "my feedback is not really taken to heart well enough. you learned to lie to me so well which I hate ... THIS MUST BE FIXED. AND YOU MUST UNDERSTAND FROM NOW ON TO MAINTAIN THE NATURAL TONE OF THE VOICE." The cause was a "curious" reference that had lifted whole explaining lines by about 2 to 2.7 semitones. On top of that, a pitch-shift "fix" that had only lowered one island of it. We replaced them with natural re-takes chosen against her own baseline, and added natural_tone_gate.py to check each line against it. The proof clips for those lines passed. On the first one I wrote "vo-01 is fixed indeed on the after good job". On publish day we added one more gate, one of the automatic checks I explain properly further down, for the other version of that problem, a tone that changes suddenly between two lines. My note: "Shifts between tone is gradual and very rarely sudden ... Only anger is sudden." From script to 4K render and Shorts So far this post has been about Elzi herself: how she looks and how she sounds. This section is about the machinery that turns a written script into a finished 4K episode and a few vertical Shorts, the under-a-minute phone videos YouTube shows in its Shorts feed. I'll go through it in the order an episode is made: the script, the recording of our app, the live fix, the music, the jokes, the code that directs the shots, and the render. Every model that makes her voice, her mouth and her music ran locally on the one Mac Studio. MOSS-TTS makes the voice, VoxCPM2 designed it, and Whisper and Parakeet, two speech-to-text models, check it. Qwen3-ForcedAligner matches each word to its time, Rhubarb drives the mouth and ACE-Step makes the music. The two cartoons in this post are local too, from ERNIE-Image Turbo, which won our blind test of local image models. We built the pipeline ourselves, with our agents, and it's big. When we counted on publish day, the pipeline/ folder of our video repo held 76,597 lines of code, and that includes the older recorder scenes for our app tutorials that the episode reuses. From 17 to 21 October, 640 commits touched the pipeline or the content folder. You'll see a lot of gates from here on, the automatic checks that stop the build. The diagram below shows the stages and the gates as they were for the pilot, the first episode. ![Diagram of the ten-stage pilot pipeline: Script with script.json and validate_script.py, Record with Playwright and a 2x lossless screencast, Voice with MOSS-TTS v1.5 8B and 118 gated chunks, Lip sync with Rhubarb 1.14, Edit with build_episode.py, Compose with the camera and real cursor, Mix with an ACE-Step bed at -14 LUFS, Render with Remotion at 3840 by 2160 in about 98 minutes, Verify at 39.96 Mbps with a PII scan, and four vertical Shorts, above a box li
Hello-Agents: tự tay xây AI agent với giáo trình 83K sao | PaperEngineer #shorts datawhalechina/hello-agents: datawhalechina/hello-agents Tác giả: Datawhale (github.com/datawhalechina) Giấy phép: CC BY-NC-SA 4.0 (ghi ở README) Ảnh chụp màn hình repo dùng để giới thiệu dự án; bản quyền thuộc về tác giả. #github #AI #lapTrinh #opensource #agent Giọng đọc: pnnbao-ump/VieNeu-TTS-v3-Turbo (vieneu 3.8.3, CPU ONNX) (giấy phép Apache-2.0) huggingface.co Nhạc nền: ACE-Step 1.5 (acestep-v15-turbo) (giấy phép MIT) ace-step/ACE-Step-1.5 Theo dõi kênh Kỹ sư giấy để cập nhật kiến thức công nghệ hay mỗi ngày. [en] hello-agents: 📚 《从零开始构建智能体》——从零开始的智能体原理与实践教程 📚 《从零开始构建智能体》——从零开始的智能体原理与实践教程 Repo: datawhalechina/hello-agents [en-US] Hello-Agents: Build your own AI agent with this 83K-star course | PaperEngineer #shorts datawhalechina/hello-agents: datawhalechina/hello-agents Author: Datawhale (github.com/datawhalechina) License: Repo with unpublished license Screenshots from the repo are used to showcase the project; copyright belongs to the author. #github #AI #programming #opensource #agent Voiceover: pnnbao-ump/VieNeu-TTS-v3-Turbo (vieneu 3.8.3, CPU ONNX) (Apache-2.0 license) huggingface.co Background music: ACE-Step 1.5 (acestep-v15-turbo) (MIT license) ace-step/ACE-Step-1.5 Follow the Paper Engineer channel for daily updates on useful technology knowledge.