Does a Cloned Voice Stay the Same? Scoring Qwen3-TTS With ECAPA
Written with AI assistance. Every number comes from my own runs; if you spot an error, tell me and I'll fix it.
I had Qwen3-TTS 1.7B read two long narrations, one in Korean and one in English, and scored every sentence chunk against the voice sample using ECAPA speaker embeddings. In these two runs, English held steady and Korean wandered more: the English narration had a median similarity of 0.861 and no chunk below 0.75, while the Korean one had a median of 0.806, with 10 of its 51 chunks under 0.75.
The practical lesson is short. Don't assume a cloned voice stays the same across a long script. Score every chunk against the sample, re-read the low ones, and keep the closest take. Below is how I did that, plus what the numbers can and can't tell you.
Why a cloned voice drifts in long narration
A local text-to-speech (TTS) model rarely reads a 20-minute script in one go. You split it into sentence chunks, generate each one, and stitch the results together. Each chunk is its own generation, so the voice can shift a little from one chunk to the next.
Both narrations used Qwen3-TTS 1.7B with Base-model voice cloning. For each language channel I generated a voice sample once. After that, every chunk was read by cloning that same sample. I wanted to know how closely each chunk actually matched it.
!Chart: Does a Cloned Voice Stay the Same? Scoring Qwen3-TTS With ECAPA
How do you measure whether a TTS voice stays consistent?
I used a speaker embedding. That's a vector a model computes from a speech clip, designed to capture who is talking rather than what's being said. Two clips of the same person should give vectors pointing in roughly the same direction.
To compare two embeddings I used cosine similarity. Higher means more alike, and 1.0 means the vectors point in exactly the same direction. The embedding model was speechbrain's spkrec-ecapa-voxceleb, a pretrained ECAPA speaker recognition model. I ran it on the CPU.
Here's a minimal sketch of the scoring step. It isn't my exact script, just its general structure, using speechbrain's pretrained interface. In older speechbrain versions the import lives under speechbrain.pretrained.
import torch
import torchaudio
from speechbrain.inference.speaker import EncoderClassifier
encoder = EncoderClassifier.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb",
run_opts={"device": "cpu"},
)
def embed(path):
signal, sr = torchaudio.load(path)
if sr != 16000: # the VoxCeleb model expects 16 kHz audio
signal = torchaudio.functional.resample(signal, sr, 16000)
signal = signal.mean(dim=0, keepdim=True) # mono
return encoder.encode_batch(signal).squeeze()
def similarity(path_a, path_b):
a, b = embed(path_a), embed(path_b)
return torch.nn.functional.cosine_similarity(a, b, dim=0).item()
Next comes the retry loop. generate_chunk is a placeholder for the Qwen3-TTS voice-clone call. The usage for cloning from a reference clip is in the Qwen3-TTS repository (QwenLM/Qwen3-TTS), and the details depend on how you've installed it.
THRESHOLD = 0.7
MAX_REREADS = 3
def read_chunk(text, reference_wav):
# reference_wav is the language channel's voice sample:
# the Korean sample for Korean chunks, the English one for English.
best_path, best_score = None, -1.0
for attempt in range(1 + MAX_REREADS):
path = generate_chunk(text, reference_wav) # Qwen3-TTS clone call
score = similarity(path, reference_wav)
if score > best_score:
best_path, best_score = path, score
if score >= THRESHOLD:
break
return best_path, best_score
In plain terms: generate the chunk, score it against the sample, and stop once the score reaches 0.7. If it doesn't, re-read it up to three times and keep whichever take came closest.
The numbers
These scores are for the final chunks that went into each narration, after any re-reads.
| | Korean narration | English narration |
|---|---|---|
| Length | 19.5 min | 23.6 min |
| Sentence chunks | 51 | 108 |
| Lowest similarity | 0.705 | 0.808 |
| Median similarity | 0.806 | 0.861 |
| Highest similarity | 0.897 | 0.908 |
| Chunks below 0.75 | 10 | 0 |
One caution before comparing the columns. Each language was scored against its own voice sample, with a different script and a different number of chunks. So the gap between them isn't proof that the language itself is the cause.
The lowest final Korean chunk was 0.705, just above the 0.7 re-read threshold. I didn't log individual attempts, so I can't tell how many chunks needed re-reads.
In these two runs, English was the tighter of the two. Its worst chunk (0.808) scored higher than the Korean median (0.806). The Korean scores also covered a wider range, from 0.705 to 0.897.
The 0.75 row is just the cutoff I used for counting in this write-up. It's not part of the re-read loop. Ten Korean chunks cleared 0.7 but stayed under 0.75, and no English chunk did.
What these scores don't tell you
There's one narration per language, so one video each. I didn't test why Korean scored lower, and with this sample I can't separate the effect of the language from the script or the voice sample.
I also didn't record scores before re-reading, or how many retries happened, so I can't say how much the loop helped. And I never scored an unrelated speaker, so I don't know where "clearly a different voice" falls on this scale.
The biggest limit is that similarity is only a rough stand-in for "same speaker." It says nothing about pronunciation, intonation, or whether a sentence sounds natural. A chunk could score high and still mispronounce a word.
Generation speed, from a separate test
These figures come from a different record, my project's progress tracker, not from the two narrations above. They cover Korean only, from a single test.
In that test, Qwen3-TTS 1.7B turned 252 Korean characters into 34.6 seconds of audio. Generation took 97 seconds, about 2.8x real time, and used 4.9 GB of VRAM. The tracker marks Korean TTS as done but notes that a listening check is still needed.
I didn't measure how much time the ECAPA scoring and re-reads added to the narrations.
Checklist
What I actually did:
1. Generate the voice sample once per language channel and clone that same file for every chunk.
2. Split the script into sentence-sized chunks.
3. Score each chunk against its channel's sample with spkrec-ecapa-voxceleb on the CPU.
4. Re-read any chunk below 0.7, up to three times, and keep the closest take.
5. Report the lowest, median and highest score, plus how many chunks fall under 0.75.
Suggestions I haven't tested yet:
- Log every attempt's score, not just the final one.
- Use 0.75 as a second line for flagging chunks to review.
- Score a few clips from a different speaker to get a baseline.
- Listen to the low-scoring chunks, since the score won't catch pronunciation problems.
FAQ
How do I check if a cloned TTS voice stays consistent across a long narration?
I scored each sentence chunk against the voice sample using cosine similarity between ECAPA speaker embeddings from speechbrain's spkrec-ecapa-voxceleb, run on the CPU.
What ECAPA similarity scores did Qwen3-TTS 1.7B get?
After re-reads, the English narration (108 chunks) ranged from 0.808 to 0.908 with a median of 0.861. The Korean narration (51 chunks) ranged from 0.705 to 0.897 with a median of 0.806.
What threshold did I use for re-reading a chunk?
Any chunk scoring under 0.7 was re-read up to three times, and I kept the take closest to the voice sample.
Does a high speaker similarity score mean the audio sounds good?
No. Similarity only roughly indicates whether it's the same speaker. It doesn't measure pronunciation or intonation, so you still need to listen.
Does Korean drift more than English in Qwen3-TTS?
In my two runs, Korean scored lower. But each language used a different voice sample and script, with one video each, so this doesn't show a language effect.
How much VRAM does Qwen3-TTS 1.7B use?
In a separate Korean-only test, it used 4.9 GB of VRAM and took 97 seconds to produce 34.6 seconds of audio from 252 characters.
Local speech with llamadart: giving Llama Village a voice
Beyond chat with llamadart · Part 3
Pip’s scarf is missing. I wanted Dash to be able to fly over to Mo and ask about it out loud.
> Dash: “Mo, have you seen Pip’s red scarf?”
> Mo: “Yes, it's in the reeds beside the pond.”
I tried that exchange with local models. For now, it runs in a small command-line program outside Llama Village. The question is a synthetic recording, and Mo gets the scarf’s location in his prompt. The game does not have voice controls yet.
!Llama Village at golden hour
The existing village. I’d like to let the player speak to these characters.
Two ways for Mo to listen
!Two routes from speech to a spoken reply
ASR is speech recognition. TTS is text to speech. Both routes finish with TTS. Concept illustration.
Via text: Qwen3-ASR transcribes the recording, a dialogue model answers, and Qwen3-TTS speaks. Having a transcript helps when “Pip” is misheard.
Direct audio: Qwen2.5-Omni receives the recording and writes an answer. This example has no separate transcript. The GGUF conversion accepts audio but does not generate it, so I still use TTS for Mo’s reply.
For the comparison, both routes use the same 3B dialogue model. In the first route it receives text; in the second it also loads its audio projector. This is one tested multimodal route, not a claim that all multimodal models need separate TTS.
I also tried a smaller pair for the text route: Qwen3.5 0.8B for dialogue and Qwen3-TTS 0.6B Base for speech. Qwen3-ASR 0.6B still handles recognition. The smaller pair is the default below; staged keeps the original 3B/1.7B pair for comparison. Qwen3.5 is used for text here, not audio input.
Try it on your Mac
The complete example is below. You need macOS, Dart 3.10.7 or newer, and about 2.31 GB for the five small-model files, plus space for Dart packages and the native runtime. I tested on an M4 Max with Dart 3.13.5 and llamadart 0.11.0. The commands use macOS say, afconvert, and afplay; they are not a phone or browser demo.
1. Create the project
Run this in a terminal. The Dart program uses pinned ModelSource.huggingFace references. Each engine’s load() uses llamadart’s download manager to fetch missing files and reuse its cache. There is no separate curl script. The first run needs a network connection and takes longer; subsequent generation uses the downloaded models locally.
{% details Copy the setup commands %}
mkdir -p llama-voice-demo/bin
cd llama-voice-demo
cat > pubspec.yaml <<'YAML'
name: llama_voice_demo
publish_to: none
environment:
sdk: ^3.10.7
dependencies:
llamadart: 0.11.0
YAML
dart pub get
{% enddetails %}
2. Add the complete program
Stay in llama-voice-demo and paste this block. It creates bin/voice.dart. The program loads one engine at a time and disposes it before loading the next.
{% details Copy the full Dart program %}
cat > bin/voice.dart <<'DART'
import 'dart:io';
import 'package:llamadart/llamadart.dart';
const params = ModelParams(
contextSize: 4096,
numberOfThreads: 4,
numberOfThreadsBatch: 4,
);
const sources = {
'asr': (
'ggml-org/Qwen3-ASR-0.6B-GGUF',
'928ab958557df9aa2ef1c93e0e83c7ad0933fae2',
),
'omni': (
'ggml-org/Qwen2.5-Omni-3B-GGUF',
'75f1b73b657a50f5092502799457ccb4a4a1f9df',
),
'dialogue': (
'unsloth/Qwen3.5-0.8B-GGUF',
'6ab461498e2023f6e3c1baea90a8f0fe38ab64d0',
),
'tts': (
'ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF',
'ca27d74bc954b73dadab5b71ca265d87fc861a7c',
),
'smallTts': (
'mradermacher/Qwen3-TTS-12Hz-0.6B-Base-GGUF',
'0762b31e9cd99274d49dca4e895bd81863134f08',
),
};
ModelSource model(String family, String file) {
final (repo, revision) = sources[family]!;
return ModelSource.huggingFace(
repoId: repo,
revision: revision,
filePath: file,
);
}
Future<void> main(List<String> args) async {
if (args.isEmpty ||
args.length > 3 ||
(args.length > 1 && !['small', 'staged', 'audio'].contains(args[1]))) {
stderr.writeln(
'Usage: dart run bin/voice.dart question.wav '
'[small|staged|audio] [voice-reference.wav]',
);
exitCode = 64;
return;
}
final question = args[0];
final small = args.length == 1 || args[1] == 'small';
final directAudio = args.length > 1 && args[1] == 'audio';
final reference = args.length == 3 ? args[2] : null;
for (final path in [question, if (reference != null) reference]) {
if (!await File(path).exists()) {
throw ArgumentError('Audio file not found: $path');
}
}
String? transcript;
if (!directAudio) {
final asr = await SpeechToTextEngine.load(
SpeechToTextModel(
model('asr', 'Qwen3-ASR-0.6B-Q8_0.gguf'),
projector: model('asr', 'mmproj-Qwen3-ASR-0.6B-Q8_0.gguf'),
adapter: const Qwen3AsrAdapter(),
),
params: params,
);
try {
final result = await asr.transcribeOnce(
SpeechToTextRequest(
audio: SpeechAudioFileInput(question),
contextPrompt: 'Mo, Pip, Dash, Llama Village',
),
);
transcript = result.text.trim();
if (transcript.isEmpty) throw StateError('No speech recognized');
stdout.writeln('Dash: $transcript');
} finally {
await asr.dispose();
}
}
final llm = await LlamaEngine.load(
LlamaModel(
small
? model('dialogue', 'Qwen3.5-0.8B-Q4_K_M.gguf')
: model('omni', 'Qwen2.5-Omni-3B-Q4_K_M.gguf'),
projector: directAudio
? model('omni', 'mmproj-Qwen2.5-Omni-3B-Q8_0.gguf')
: null,
),
params: params,
);
final reply = StringBuffer();
try {
if (directAudio && !(await llm.capabilities).supportsAudio) {
throw StateError('This model/runtime cannot accept audio');
}
await for (final chunk in llm.create(
[
const LlamaChatMessage.fromText(
role: LlamaChatRole.system,
text:
'You are Mo, a friendly baker llama. Dash is visiting you. '
"Known fact: Pip's red scarf is in the reeds beside the pond. "
'Answer the visitor in one short sentence in English. '
'No stage directions. Do not invent facts you do not know.',
),
if (transcript != null)
LlamaChatMessage.fromText(role: LlamaChatRole.user, text: transcript)
else
LlamaChatMessage.withContent(
role: LlamaChatRole.user,
content: [LlamaAudioContent(path: question)],
),
],
enableThinking: false,
params: const GenerationParams(maxTokens: 100, temp: 0, seed: 42),
)) {
reply.write(chunk.text);
}
} finally {
await llm.dispose();
}
final text = reply.toString().trim();
if (text.isEmpty) throw StateError('The model returned no reply');
stdout.writeln('Mo: $text');
final tts = await TextToSpeechEngine.load(
TextToSpeechModel(
small
? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.Q4_K_M.gguf')
: model('tts', 'Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf'),
projector: small
? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.mmproj-Q8_0.gguf')
: model('tts', 'mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf'),
adapter: const Qwen3TtsAdapter(),
),
params: params,
);
try {
final audio = await tts.synthesizeOnce(
TextToSpeechRequest(
text: text,
language: 'en',
seed: 42,
speakerReference: reference == null
? null
: SpeechAudioFileInput(reference),
),
);
await File('reply.wav').writeAsBytes(audio.toWavBytes());
stdout.writeln('Saved reply.wav');
} finally {
await tts.dispose();
}
}
DART
{% enddetails %}
3. Ask Mo a question
say -o question.aiff "Mo, have you seen Pip's red scarf?"
afconvert -f WAVE -d LEI16@16000 -c 1 question.aiff question.wav
dart run bin/voice.dart question.wav small
afplay reply.wav
This creates a synthetic question so you can try the whole path without a microphone package or a sample download. You can replace question.wav with your own short WAV. The program prints the transcript and answer, then saves reply.wav in the project directory. It does not record a microphone.
The main calls inside the program are:
// ASR: a recording becomes text.
final result = await asr.transcribeOnce(
SpeechToTextRequest(
audio: SpeechAudioFileInput(question),
contextPrompt: 'Mo, Pip, Dash, Llama Village',
),
);
// TTS: the generated answer becomes a WAV file.
final audio = await tts.synthesizeOnce(
TextToSpeechRequest(text: text, language: 'en', seed: 42),
);
await File('reply.wav').writeAsBytes(audio.toWavBytes());
These excerpts sit inside the loaded engines’ try/finally blocks in the full program. Between them, llm.create answers using the transcript and Mo’s supplied fact.
Try the same question as audio
Change the mode to download and load Omni’s audio-capable model and projector, plus the original 1.7B TTS pair:
dart run bin/voice.dart question.wav audio
afplay reply.wav
That route skips ASR and sends LlamaAudioContent(path: question) to the dialogue model. The program checks supportsAudio before asking it to answer. The files used by this route total about 5.13 GB; keeping all three modes cached takes about 7.44 GB.
In the game, Mo should only receive facts he is allowed to reveal. DecisionEngine could choose an allowed intention before dialogue generation. A friendly persona prompt does not enforce those rules.
Make it sound like Mo
!Design a voice once and reuse its reference
A possible production workflow: prepare a voice, save a reference, and reuse it for new dialogue. Concept illustration.
I’d like Mo to sound warm and unhurried. Dash could be brighter and quicker. A short reference clip lets several characters share one TTS model while sounding different. When the player speaks as Dash, they already provide his voice.
The full program accepts an optional third argument. Use small for the 0.8B/0.6B pair, or staged for the original 3B/1.7B pair:
dart run bin/voice.dart question.wav small mo-reference.wav
afplay reply.wav
Use a short, clear recording you have permission to use. For my test I made a synthetic reference with the macOS Daniel voice. Inside TextToSpeechRequest, the program passes it as speakerReference: SpeechAudioFileInput(reference).
Qwen’s voice-design workflow also lets you create a voice during development, then use the resulting clip with a Base model. Players would download the synthesis model and reference, without needing the voice-design model. I have not tested that workflow here.
A reference gives Mo a voice, but the tested API has no instruction such as “say this nervously.” That is not just a missing Dart parameter. Qwen’s 1.7B CustomVoice and VoiceDesign models support instruction control; the Base model used here does not offer that same mode. Exposing it would need a compatible model, native runtime support, and a Dart API. Changing the dialogue prompt only changes the words.
Smaller models, smaller downloads. Faster conversations?
I tested the smaller pair on the same M4 Max (64 GB), with llamadart 0.11.0. Both routes kept Qwen3-ASR 0.6B and used the same synthetic question and Daniel voice reference.
| Text route | Model files | Complete turn, median | Peak process RSS |
| --- | ---: | ---: | ---: |
| Omni 3B + TTS 1.7B | 4.61 GB | 4.27 s | 3.86 GiB |
| Qwen3.5 0.8B + TTS 0.6B | 2.31 GB | 5.83 s | 2.51 GiB |
Half the download, about 35% less peak RSS, but a slower turn in this pilot. The smaller dialogue model gave a longer reply:
> 3B: “Yes, Pip's red scarf is in the reeds beside the pond.”
> 0.8B: “Yes, I saw it! It was tucked inside the reeds by the pond.”
The 0.8B model also ignored the one-sentence instruction. More words mean more speech to generate. These are complete application turns with different replies, so I also tested the components separately:
| Loaded component | Generation time, median | Peak process RSS |
| --- | ---: | ---: |
| Omni 3B dialogue | 171 ms | 2.57 GiB |
| Qwen3.5 0.8B dialogue | 96 ms | 1.08 GiB |
| Qwen3-TTS 1.7B | 1.78 s | 3.85 GiB |
| Qwen3-TTS 0.6B | 1.78 s | 2.48 GiB |
For TTS, both models spoke exactly “Yes, Pip's red scarf is in the reeds beside the pond.” The 1.7B clips lasted 4.16–4.56 seconds; the 0.6B clips lasted 4.80–6.48 seconds. The smaller model’s synthesis times ranged from 1.78 to 2.53 seconds. Smaller did not mean a consistently shorter wait.
Did Mo say the right thing?
Both dialogue models supplied the location and colour correctly when asked. But when I asked “Who put the scarf in the pond?”, both blamed Pip. That fact was never supplied, even with “Do not invent facts you do not know” in the prompt. The smaller model’s “I saw it” also adds an unsupported claim about how Mo knows.
All six measured TTS clips transcribed back to the intended sentence with Qwen3-ASR. That is an intelligibility check, not a listening score for naturalness, emotion, or similarity to Daniel. I would still listen to more lines before choosing a character voice.
I would use the smaller pair for this prototype’s storage budget. I would keep the game’s knowledge rules outside the prompt, and measure real conversations before claiming a latency win.
{% details Test conditions and model files %}
This is a small development-machine pilot, not a device benchmark. Other development workloads were active; the village renderer was not. The complete-turn test alternated the two setups, discarded the first turn for each, then measured three fresh-process turns per setup. Baseline turns ranged from 3.76 to 4.41 seconds; smaller-pair turns ranged from 5.82 to 5.91 seconds. Timing includes model loading and disposal, but excludes process startup, downloads, recording, and playback. Peak RSS was sampled every 100 ms and is the largest observed peak across the three runs. It is process memory, not total GPU or device memory.
Component tests loaded each model in its own process and discarded one warm-up. Dialogue used three questions: the scarf’s location, who put it there, and its location plus colour. TTS repeated the same sentence three times with the same reference. Component times exclude loading and disposal; memory uses the process high-water RSS. Both dialogue models used temperature 0, seed 42, a 100-token limit, thinking disabled, a 4096-token context, and four threads. TTS used seed 42; its waveforms still varied between calls.
The dialogue weights are Q4_K_M; audio projectors are Q8_0. The new files are Unsloth’s Qwen3.5 0.8B conversion and mradermacher’s Qwen3-TTS 0.6B Base conversion. The exact revisions and matching projector filenames are pinned in the program above. A GGUF file made for a different TTS runtime is not automatically interchangeable.
{% enddetails %}
Let the player read while Mo gets ready
This TTS API returns the complete audio before playback. Its progress events are not playable audio chunks. I’d show Mo’s reply as text immediately, then play it when synthesis finishes.
For a first game version, I’d use push-to-talk, stop recording while Mo speaks, and discard an old reply if the player starts another turn. Text should remain available when voice is switched off or a model cannot load. I still need to measure speech alongside rendering before adding this to the game.
For model pairs and platform requirements, see the speech recognition guide and speech synthesis guide. Partial live captions use a different recognition path from this file-based example.
30 reactions · 1 comment
Mentions
Does a Cloned Voice Stay the Same? Scoring Qwen3-TTS With ECAPA Written with AI assistance. Every number comes from my own runs; if you spot an error, tell me and I'll fix it. I had Qwen3-TTS 1.7B read two long narrations, one in Korean and one in English, and scored every sentence chunk against the voice sample using ECAPA speaker embeddings. In these two runs, English held steady and Korean wandered more: the English narration had a median similarity of 0.861 and no chunk below 0.75, while the Korean one had a median of 0.806, with 10 of its 51 chunks under 0.75. The practical lesson is short. Don't assume a cloned voice stays the same across a long script. Score every chunk against the sample, re-read the low ones, and keep the closest take. Below is how I did that, plus what the numbers can and can't tell you. Why a cloned voice drifts in long narration A local text-to-speech (TTS) model rarely reads a 20-minute script in one go. You split it into sentence chunks, generate each one, and stitch the results together. Each chunk is its own generation, so the voice can shift a little from one chunk to the next. Both narrations used Qwen3-TTS 1.7B with Base-model voice cloning. For each language channel I generated a voice sample once. After that, every chunk was read by cloning that same sample. I wanted to know how closely each chunk actually matched it. !Chart: Does a Cloned Voice Stay the Same? Scoring Qwen3-TTS With ECAPA How do you measure whether a TTS voice stays consistent? I used a speaker embedding. That's a vector a model computes from a speech clip, designed to capture who is talking rather than what's being said. Two clips of the same person should give vectors pointing in roughly the same direction. To compare two embeddings I used cosine similarity. Higher means more alike, and 1.0 means the vectors point in exactly the same direction. The embedding model was speechbrain's spkrec-ecapa-voxceleb, a pretrained ECAPA speaker recognition model. I ran it on the CPU. Here's a minimal sketch of the scoring step. It isn't my exact script, just its general structure, using speechbrain's pretrained interface. In older speechbrain versions the import lives under speechbrain.pretrained. import torch import torchaudio from speechbrain.inference.speaker import EncoderClassifier encoder = EncoderClassifier.from_hparams( source="speechbrain/spkrec-ecapa-voxceleb", run_opts={"device": "cpu"}, ) def embed(path): signal, sr = torchaudio.load(path) if sr != 16000: # the VoxCeleb model expects 16 kHz audio signal = torchaudio.functional.resample(signal, sr, 16000) signal = signal.mean(dim=0, keepdim=True) # mono return encoder.encode_batch(signal).squeeze() def similarity(path_a, path_b): a, b = embed(path_a), embed(path_b) return torch.nn.functional.cosine_similarity(a, b, dim=0).item() Next comes the retry loop. generate_chunk is a placeholder for the Qwen3-TTS voice-clone call. The usage for cloning from a reference clip is in the Qwen3-TTS repository (QwenLM/Qwen3-TTS), and the details depend on how you've installed it. THRESHOLD = 0.7 MAX_REREADS = 3 def read_chunk(text, reference_wav): # reference_wav is the language channel's voice sample: # the Korean sample for Korean chunks, the English one for English. best_path, best_score = None, -1.0 for attempt in range(1 + MAX_REREADS): path = generate_chunk(text, reference_wav) # Qwen3-TTS clone call score = similarity(path, reference_wav) if score > best_score: best_path, best_score = path, score if score >= THRESHOLD: break return best_path, best_score In plain terms: generate the chunk, score it against the sample, and stop once the score reaches 0.7. If it doesn't, re-read it up to three times and keep whichever take came closest. The numbers These scores are for the final chunks that went into each narration, after any re-reads. | | Korean narration | English narration | |---|---|---| | Length | 19.5 min | 23.6 min | | Sentence chunks | 51 | 108 | | Lowest similarity | 0.705 | 0.808 | | Median similarity | 0.806 | 0.861 | | Highest similarity | 0.897 | 0.908 | | Chunks below 0.75 | 10 | 0 | One caution before comparing the columns. Each language was scored against its own voice sample, with a different script and a different number of chunks. So the gap between them isn't proof that the language itself is the cause. The lowest final Korean chunk was 0.705, just above the 0.7 re-read threshold. I didn't log individual attempts, so I can't tell how many chunks needed re-reads. In these two runs, English was the tighter of the two. Its worst chunk (0.808) scored higher than the Korean median (0.806). The Korean scores also covered a wider range, from 0.705 to 0.897. The 0.75 row is just the cutoff I used for counting in this write-up. It's not part of the re-read loop. Ten Korean chunks cleared 0.7 but stayed under 0.75, and no English chunk did. What these scores don't tell you There's one narration per language, so one video each. I didn't test why Korean scored lower, and with this sample I can't separate the effect of the language from the script or the voice sample. I also didn't record scores before re-reading, or how many retries happened, so I can't say how much the loop helped. And I never scored an unrelated speaker, so I don't know where "clearly a different voice" falls on this scale. The biggest limit is that similarity is only a rough stand-in for "same speaker." It says nothing about pronunciation, intonation, or whether a sentence sounds natural. A chunk could score high and still mispronounce a word. Generation speed, from a separate test These figures come from a different record, my project's progress tracker, not from the two narrations above. They cover Korean only, from a single test. In that test, Qwen3-TTS 1.7B turned 252 Korean characters into 34.6 seconds of audio. Generation took 97 seconds, about 2.8x real time, and used 4.9 GB of VRAM. The tracker marks Korean TTS as done but notes that a listening check is still needed. I didn't measure how much time the ECAPA scoring and re-reads added to the narrations. Checklist What I actually did: 1. Generate the voice sample once per language channel and clone that same file for every chunk. 2. Split the script into sentence-sized chunks. 3. Score each chunk against its channel's sample with spkrec-ecapa-voxceleb on the CPU. 4. Re-read any chunk below 0.7, up to three times, and keep the closest take. 5. Report the lowest, median and highest score, plus how many chunks fall under 0.75. Suggestions I haven't tested yet: - Log every attempt's score, not just the final one. - Use 0.75 as a second line for flagging chunks to review. - Score a few clips from a different speaker to get a baseline. - Listen to the low-scoring chunks, since the score won't catch pronunciation problems. FAQ How do I check if a cloned TTS voice stays consistent across a long narration? I scored each sentence chunk against the voice sample using cosine similarity between ECAPA speaker embeddings from speechbrain's spkrec-ecapa-voxceleb, run on the CPU. What ECAPA similarity scores did Qwen3-TTS 1.7B get? After re-reads, the English narration (108 chunks) ranged from 0.808 to 0.908 with a median of 0.861. The Korean narration (51 chunks) ranged from 0.705 to 0.897 with a median of 0.806. What threshold did I use for re-reading a chunk? Any chunk scoring under 0.7 was re-read up to three times, and I kept the take closest to the voice sample. Does a high speaker similarity score mean the audio sounds good? No. Similarity only roughly indicates whether it's the same speaker. It doesn't measure pronunciation or intonation, so you still need to listen. Does Korean drift more than English in Qwen3-TTS? In my two runs, Korean scored lower. But each language used a different voice sample and script, with one video each, so this doesn't show a language effect. How much VRAM does Qwen3-TTS 1.7B use? In a separate Korean-only test, it used 4.9 GB of VRAM and took 97 seconds to produce 34.6 seconds of audio from 252 characters.
1/n Qwen3-TTS：克隆声音，支持控制情绪，15 秒生成 22 秒音频 QwenLM/Qwen3-TTS
Local speech with llamadart: giving Llama Village a voice Beyond chat with llamadart · Part 3 Pip’s scarf is missing. I wanted Dash to be able to fly over to Mo and ask about it out loud. > Dash: “Mo, have you seen Pip’s red scarf?” > Mo: “Yes, it's in the reeds beside the pond.” I tried that exchange with local models. For now, it runs in a small command-line program outside Llama Village. The question is a synthetic recording, and Mo gets the scarf’s location in his prompt. The game does not have voice controls yet. !Llama Village at golden hour The existing village. I’d like to let the player speak to these characters. Two ways for Mo to listen !Two routes from speech to a spoken reply ASR is speech recognition. TTS is text to speech. Both routes finish with TTS. Concept illustration. Via text: Qwen3-ASR transcribes the recording, a dialogue model answers, and Qwen3-TTS speaks. Having a transcript helps when “Pip” is misheard. Direct audio: Qwen2.5-Omni receives the recording and writes an answer. This example has no separate transcript. The GGUF conversion accepts audio but does not generate it, so I still use TTS for Mo’s reply. For the comparison, both routes use the same 3B dialogue model. In the first route it receives text; in the second it also loads its audio projector. This is one tested multimodal route, not a claim that all multimodal models need separate TTS. I also tried a smaller pair for the text route: Qwen3.5 0.8B for dialogue and Qwen3-TTS 0.6B Base for speech. Qwen3-ASR 0.6B still handles recognition. The smaller pair is the default below; staged keeps the original 3B/1.7B pair for comparison. Qwen3.5 is used for text here, not audio input. Try it on your Mac The complete example is below. You need macOS, Dart 3.10.7 or newer, and about 2.31 GB for the five small-model files, plus space for Dart packages and the native runtime. I tested on an M4 Max with Dart 3.13.5 and llamadart 0.11.0. The commands use macOS say, afconvert, and afplay; they are not a phone or browser demo. 1. Create the project Run this in a terminal. The Dart program uses pinned ModelSource.huggingFace references. Each engine’s load() uses llamadart’s download manager to fetch missing files and reuse its cache. There is no separate curl script. The first run needs a network connection and takes longer; subsequent generation uses the downloaded models locally. {% details Copy the setup commands %} mkdir -p llama-voice-demo/bin cd llama-voice-demo cat > pubspec.yaml <<'YAML' name: llama_voice_demo publish_to: none environment: sdk: ^3.10.7 dependencies: llamadart: 0.11.0 YAML dart pub get {% enddetails %} 2. Add the complete program Stay in llama-voice-demo and paste this block. It creates bin/voice.dart. The program loads one engine at a time and disposes it before loading the next. {% details Copy the full Dart program %} cat > bin/voice.dart <<'DART' import 'dart:io'; import 'package:llamadart/llamadart.dart'; const params = ModelParams( contextSize: 4096, numberOfThreads: 4, numberOfThreadsBatch: 4, ); const sources = { 'asr': ( 'ggml-org/Qwen3-ASR-0.6B-GGUF', '928ab958557df9aa2ef1c93e0e83c7ad0933fae2', ), 'omni': ( 'ggml-org/Qwen2.5-Omni-3B-GGUF', '75f1b73b657a50f5092502799457ccb4a4a1f9df', ), 'dialogue': ( 'unsloth/Qwen3.5-0.8B-GGUF', '6ab461498e2023f6e3c1baea90a8f0fe38ab64d0', ), 'tts': ( 'ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF', 'ca27d74bc954b73dadab5b71ca265d87fc861a7c', ), 'smallTts': ( 'mradermacher/Qwen3-TTS-12Hz-0.6B-Base-GGUF', '0762b31e9cd99274d49dca4e895bd81863134f08', ), }; ModelSource model(String family, String file) { final (repo, revision) = sources[family]!; return ModelSource.huggingFace( repoId: repo, revision: revision, filePath: file, ); } Future<void> main(List<String> args) async { if (args.isEmpty || args.length > 3 || (args.length > 1 && !['small', 'staged', 'audio'].contains(args[1]))) { stderr.writeln( 'Usage: dart run bin/voice.dart question.wav ' '[small|staged|audio] [voice-reference.wav]', ); exitCode = 64; return; } final question = args[0]; final small = args.length == 1 || args[1] == 'small'; final directAudio = args.length > 1 && args[1] == 'audio'; final reference = args.length == 3 ? args[2] : null; for (final path in [question, if (reference != null) reference]) { if (!await File(path).exists()) { throw ArgumentError('Audio file not found: $path'); } } String? transcript; if (!directAudio) { final asr = await SpeechToTextEngine.load( SpeechToTextModel( model('asr', 'Qwen3-ASR-0.6B-Q8_0.gguf'), projector: model('asr', 'mmproj-Qwen3-ASR-0.6B-Q8_0.gguf'), adapter: const Qwen3AsrAdapter(), ), params: params, ); try { final result = await asr.transcribeOnce( SpeechToTextRequest( audio: SpeechAudioFileInput(question), contextPrompt: 'Mo, Pip, Dash, Llama Village', ), ); transcript = result.text.trim(); if (transcript.isEmpty) throw StateError('No speech recognized'); stdout.writeln('Dash: $transcript'); } finally { await asr.dispose(); } } final llm = await LlamaEngine.load( LlamaModel( small ? model('dialogue', 'Qwen3.5-0.8B-Q4_K_M.gguf') : model('omni', 'Qwen2.5-Omni-3B-Q4_K_M.gguf'), projector: directAudio ? model('omni', 'mmproj-Qwen2.5-Omni-3B-Q8_0.gguf') : null, ), params: params, ); final reply = StringBuffer(); try { if (directAudio && !(await llm.capabilities).supportsAudio) { throw StateError('This model/runtime cannot accept audio'); } await for (final chunk in llm.create( [ const LlamaChatMessage.fromText( role: LlamaChatRole.system, text: 'You are Mo, a friendly baker llama. Dash is visiting you. ' "Known fact: Pip's red scarf is in the reeds beside the pond. " 'Answer the visitor in one short sentence in English. ' 'No stage directions. Do not invent facts you do not know.', ), if (transcript != null) LlamaChatMessage.fromText(role: LlamaChatRole.user, text: transcript) else LlamaChatMessage.withContent( role: LlamaChatRole.user, content: [LlamaAudioContent(path: question)], ), ], enableThinking: false, params: const GenerationParams(maxTokens: 100, temp: 0, seed: 42), )) { reply.write(chunk.text); } } finally { await llm.dispose(); } final text = reply.toString().trim(); if (text.isEmpty) throw StateError('The model returned no reply'); stdout.writeln('Mo: $text'); final tts = await TextToSpeechEngine.load( TextToSpeechModel( small ? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.Q4_K_M.gguf') : model('tts', 'Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf'), projector: small ? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.mmproj-Q8_0.gguf') : model('tts', 'mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf'), adapter: const Qwen3TtsAdapter(), ), params: params, ); try { final audio = await tts.synthesizeOnce( TextToSpeechRequest( text: text, language: 'en', seed: 42, speakerReference: reference == null ? null : SpeechAudioFileInput(reference), ), ); await File('reply.wav').writeAsBytes(audio.toWavBytes()); stdout.writeln('Saved reply.wav'); } finally { await tts.dispose(); } } DART {% enddetails %} 3. Ask Mo a question say -o question.aiff "Mo, have you seen Pip's red scarf?" afconvert -f WAVE -d LEI16@16000 -c 1 question.aiff question.wav dart run bin/voice.dart question.wav small afplay reply.wav This creates a synthetic question so you can try the whole path without a microphone package or a sample download. You can replace question.wav with your own short WAV. The program prints the transcript and answer, then saves reply.wav in the project directory. It does not record a microphone. The main calls inside the program are: // ASR: a recording becomes text. final result = await asr.transcribeOnce( SpeechToTextRequest( audio: SpeechAudioFileInput(question), contextPrompt: 'Mo, Pip, Dash, Llama Village', ), ); // TTS: the generated answer becomes a WAV file. final audio = await tts.synthesizeOnce( TextToSpeechRequest(text: text, language: 'en', seed: 42), ); await File('reply.wav').writeAsBytes(audio.toWavBytes()); These excerpts sit inside the loaded engines’ try/finally blocks in the full program. Between them, llm.create answers using the transcript and Mo’s supplied fact. Try the same question as audio Change the mode to download and load Omni’s audio-capable model and projector, plus the original 1.7B TTS pair: dart run bin/voice.dart question.wav audio afplay reply.wav That route skips ASR and sends LlamaAudioContent(path: question) to the dialogue model. The program checks supportsAudio before asking it to answer. The files used by this route total about 5.13 GB; keeping all three modes cached takes about 7.44 GB. In the game, Mo should only receive facts he is allowed to reveal. DecisionEngine could choose an allowed intention before dialogue generation. A friendly persona prompt does not enforce those rules. Make it sound like Mo !Design a voice once and reuse its reference A possible production workflow: prepare a voice, save a reference, and reuse it for new dialogue. Concept illustration. I’d like Mo to sound warm and unhurried. Dash could be brighter and quicker. A short reference clip lets several characters share one TTS model while sounding different. When the player speaks as Dash, they already provide his voice. The full program accepts an optional third argument. Use small for the 0.8B/0.6B pair, or staged for the original 3B/1.7B pair: dart run bin/voice.dart question.wav small mo-reference.wav afplay reply.wav Use a short, clear recording you have permission to use. For my test I made a synthetic reference with the macOS Daniel voice. Inside TextToSpeechRequest, the program passes it as speakerReference: SpeechAudioFileInput(reference). Qwen’s voice-design workflow also lets you create a voice during development, then use the resulting clip with a Base model. Players would download the synthesis model and reference, without needing the voice-design model. I have not tested that workflow here. A reference gives Mo a voice, but the tested API has no instruction such as “say this nervously.” That is not just a missing Dart parameter. Qwen’s 1.7B CustomVoice and VoiceDesign models support instruction control; the Base model used here does not offer that same mode. Exposing it would need a compatible model, native runtime support, and a Dart API. Changing the dialogue prompt only changes the words. Smaller models, smaller downloads. Faster conversations? I tested the smaller pair on the same M4 Max (64 GB), with llamadart 0.11.0. Both routes kept Qwen3-ASR 0.6B and used the same synthetic question and Daniel voice reference. | Text route | Model files | Complete turn, median | Peak process RSS | | --- | ---: | ---: | ---: | | Omni 3B + TTS 1.7B | 4.61 GB | 4.27 s | 3.86 GiB | | Qwen3.5 0.8B + TTS 0.6B | 2.31 GB | 5.83 s | 2.51 GiB | Half the download, about 35% less peak RSS, but a slower turn in this pilot. The smaller dialogue model gave a longer reply: > 3B: “Yes, Pip's red scarf is in the reeds beside the pond.” > 0.8B: “Yes, I saw it! It was tucked inside the reeds by the pond.” The 0.8B model also ignored the one-sentence instruction. More words mean more speech to generate. These are complete application turns with different replies, so I also tested the components separately: | Loaded component | Generation time, median | Peak process RSS | | --- | ---: | ---: | | Omni 3B dialogue | 171 ms | 2.57 GiB | | Qwen3.5 0.8B dialogue | 96 ms | 1.08 GiB | | Qwen3-TTS 1.7B | 1.78 s | 3.85 GiB | | Qwen3-TTS 0.6B | 1.78 s | 2.48 GiB | For TTS, both models spoke exactly “Yes, Pip's red scarf is in the reeds beside the pond.” The 1.7B clips lasted 4.16–4.56 seconds; the 0.6B clips lasted 4.80–6.48 seconds. The smaller model’s synthesis times ranged from 1.78 to 2.53 seconds. Smaller did not mean a consistently shorter wait. Did Mo say the right thing? Both dialogue models supplied the location and colour correctly when asked. But when I asked “Who put the scarf in the pond?”, both blamed Pip. That fact was never supplied, even with “Do not invent facts you do not know” in the prompt. The smaller model’s “I saw it” also adds an unsupported claim about how Mo knows. All six measured TTS clips transcribed back to the intended sentence with Qwen3-ASR. That is an intelligibility check, not a listening score for naturalness, emotion, or similarity to Daniel. I would still listen to more lines before choosing a character voice. I would use the smaller pair for this prototype’s storage budget. I would keep the game’s knowledge rules outside the prompt, and measure real conversations before claiming a latency win. {% details Test conditions and model files %} This is a small development-machine pilot, not a device benchmark. Other development workloads were active; the village renderer was not. The complete-turn test alternated the two setups, discarded the first turn for each, then measured three fresh-process turns per setup. Baseline turns ranged from 3.76 to 4.41 seconds; smaller-pair turns ranged from 5.82 to 5.91 seconds. Timing includes model loading and disposal, but excludes process startup, downloads, recording, and playback. Peak RSS was sampled every 100 ms and is the largest observed peak across the three runs. It is process memory, not total GPU or device memory. Component tests loaded each model in its own process and discarded one warm-up. Dialogue used three questions: the scarf’s location, who put it there, and its location plus colour. TTS repeated the same sentence three times with the same reference. Component times exclude loading and disposal; memory uses the process high-water RSS. Both dialogue models used temperature 0, seed 42, a 100-token limit, thinking disabled, a 4096-token context, and four threads. TTS used seed 42; its waveforms still varied between calls. The dialogue weights are Q4_K_M; audio projectors are Q8_0. The new files are Unsloth’s Qwen3.5 0.8B conversion and mradermacher’s Qwen3-TTS 0.6B Base conversion. The exact revisions and matching projector filenames are pinned in the program above. A GGUF file made for a different TTS runtime is not automatically interchangeable. {% enddetails %} Let the player read while Mo gets ready This TTS API returns the complete audio before playback. Its progress events are not playable audio chunks. I’d show Mo’s reply as text immediately, then play it when synthesis finishes. For a first game version, I’d use push-to-talk, stop recording while Mo speaks, and discard an old reply if the player starts another turn. Text should remain available when voice is switched off or a model cannot load. I still need to measure speech alongside rendering before adding this to the game. For model pairs and platform requirements, see the speech recognition guide and speech synthesis guide. Partial live captions use a different recognition path from this file-based example.
做视频想复用同一个原创角色声音，可以用Qwen3-TTS：VoiceDesign先按描述生成一段声音，再把音频和对应台词交给Base，建立可复用的声音提示。后续新台词复用它，就不用每次重新提取参考特征。 QwenLM/Qwen3-TTS