silero-vad

Whisper typed its own prompt into my document If you use faster-whisper or openai-whisper with an initial_prompt, try this sometime. Record two seconds of a fan, or a keyboard, or nothing in particular, loud enough to pass whatever loudness check you have. Transcribe it. There is a decent chance you get your prompt back. In YazSes, the offline dictation tool I work on, the built-in prompt is one short sentence: BUILTIN_PROMPT = "The app is called YazSes." It is there so Whisper spells the app's name right. And on a noisy burst with no speech in it, Whisper would sometimes transcribe it as... "The app is called YazSes." Typed straight into whatever window you had open. (#598 has the details.) Why this happens Whisper treats initial_prompt as if it were the text that came just before the audio. My working explanation (a hypothesis, not something I have proven inside the model) is that when the audio carries no words, the easiest thing for the decoder to produce is the text it was just shown. Loud noise is the worst case: a quiet clip gets dropped by the energy gate before Whisper sees it, but a loud one gets through. Why the obvious fix doesn't work Whisper reports a no_speech_prob for each segment. Surely you just drop segments where it is high? I measured it, with the small, base.en and tiny.en models on synthetic noise bursts (white, brown and low-frequency noise and clicks, half a second to three seconds). On the echoes, no_speech_prob ranged from 0.43 to 0.97. On a public-domain speech sample it was 0.02 to 0.27. Two problems with cutting on that: - faster-whisper's default no_speech_threshold is 0.6. Every echo between 0.43 and 0.6 sails straight through the default settings. - The gap between 0.27 and 0.43 comes from one speech sample and some synthetic noise. I would not bet a user's words on a cutoff placed inside a gap that narrow, measured that thinly. And it wasn't only the built-in prompt. A personal vocabulary list came back too: "Ydotool, ydotool, Mohsen, Wayland." Whatever you prime the model with is what it reaches for when there is nothing to hear. The fix: ask a different model The fix is to stop asking Whisper whether there was speech, and ask an independent detector before decoding. faster-whisper already bundles Silero VAD, so it costs no new dependency. Before a burst is decoded, YazSes asks Silero whether it holds any speech. If Silero positively says no, the burst is discarded and logged as discard_reason = "no_speech". !Energy gate, then Silero VAD; only a positive 'no speech' discards the burst The most important design decision is the direction it fails in: def holds_no_speech( audio: Any, sample_rate: int, *, detector: Detector | None = None ) -> bool: """True only when the detector *positively* found no speech. Never raises. ``None`` from the detector (unavailable, wrong sample rate) and any exception both read as "cannot tell", which is ``False``: the burst is decoded. """ try: verdict = (detector or _default_detector)(audio, sample_rate) except Exception: # noqa: BLE001 - a gate must never break dictation return False return verdict is True It fails open. If Silero can't run, can't load, or gets audio at the wrong rate, the burst is decoded exactly as before. Think about the costs: a phantom sentence is annoying and you delete it. Silently swallowing words the user actually said is much worse, because they may not notice until later. A gate that guards against the small error must never cause the big one. Choosing the threshold Silero's default speech threshold is 0.5. I used 0.3, and here is the evidence, including its limits: - I cut 22 single-word bursts from the public-domain JFK speech sample and made 27 seeded noise bursts. - At 0.3: 0 words dropped, 0 noise bursts passed. - At 0.5: 2 quiet words were lost. - With a crude "whisper" proxy (random-phase audio), 3 of 22 words were lost at 0.3. That is synthetic audio and a small set. I have not measured it on real whispered, accented or dysarthric speech, and those are exactly the users a dictation tool should care about most. So the check is skipped in the modes built for them: whispered/low-effort mode, the dysfluency-friendly mode and the quiet sotto-voce channel. It can also be turned off: [accessibility] speech_presence_check = true # the default since v2.43.2 What I took away - Loudness can't tell a person from a fan. An energy gate is a first filter, not a speech detector. - Don't ask the model that is failing whether it is failing. Its own no_speech_prob let echoes through at default settings. A second, independent model that never saw the prompt answers the question more cleanly. - Decide which way a gate should fail before you write it. Then write the test for the failure, not just the happy path. If you run Whisper with a prompt in any app (meeting notes, voice notes, subtitles), check what it returns on two seconds of loud noise. It takes a minute and you may be surprised. --- The fix shipped in YazSes v2.43.2 (release notes). YazSes is offline, open source and Apache-2.0: MSKazemi/yazses. I wrote this with help from an AI assistant for drafting and editing; the bug, the measurements and the design choices are mine, and I checked every fact against the code and release notes.

@Moi_Sidy @benjamincode Pour les silences, tu as Silero VAD, sous MIT : snakers4/silero-vad. Il repère les passages parlés, mais il faut traiter les hésitations à part.

Как мы выбирали шумоподавление для БОЛТУНа Занялись шумоподавлением в БОЛТУНе. Хотелось убрать из разговора клавиатуру, фон и, если получится, чужие голоса рядом с микрофоном. При этом оставить в покое голос самого игрока и не отъесть у игры процессор. Сначала прошлись по основным подходам и инструментам: штатная обработка браузера, пороговые фильтры, SpeexDSP, RNNoise, DeepFilterNet, выделение голоса по образцу. Потом собрали стенды и начали проверять то, что подходило под нашу задачу. Попробовали нейросеть в браузере, вынесли её в нативный процесс, посмотрели на нагрузку от нескольких пользователей. А сейчас занимаемся более простой обработкой пауз. Сразу обозначу, где мы находимся: всё это пока отдельные локальные эксперименты. В опубликованный БОЛТУН их не добавляли. Уже есть результаты, которыми можно поделиться, но проверка с настоящими микрофонами и запущенной игрой ещё впереди. Что посмотрели Слово «шумоподавление» оказалось слишком общим. Гул вентилятора, стук по столу и человек, разговаривающий рядом, требуют разного подхода. Особенно человек: его голос тоже является речью, и обычному детектору голосовой активности всё равно, кого именно мы собирались слушать. Для себя разложили варианты так: Инструмент или подход Для чего нужен До чего дошли у нас Gate и expander Закрывать передачу или ослаблять тихие участки Собрали свой лёгкий вариант и A/B-стенд Silero VAD Находить участки речи; сам звук не очищает Изучили документацию Обработка браузера Запрашивать шумоподавление при захвате микрофона Уже используется в нашей цепочке захвата SpeexDSP Шумоподавление, регулировка усиления и другая предобработка Изучили документацию RNNoise, DeepFilterNet Подавлять шум с помощью нейросетевой обработки Посмотрели устройство и способы подключения VoxLock Выделять нужный голос по его образцу Запустили, замерили, попробовали несколько одновременных входов Кто настраивал микрофон в OBS, наверняка встречал отдельные фильтры gate, expander и noise suppression. Мы тоже стали рассматривать их отдельно. Например, gate может убрать фон, пока человек молчит. Но когда передача открылась, вместе с голосом через неё проходит и остальной звук. Посмотрели и на готовые приложения вроде NVIDIA Broadcast. Там есть удаление шума и комнатного эха с использованием поддерживаемых RTX GPU. Для владельца подходящей видеокарты это отдельный вариант, а нам нужна обработка внутри своего приложения, с учётом веба и Android. В итоге до собственных тестов довели VoxLock и лёгкий пороговый обработчик. Остальные инструменты остались кандидатами. RNNoise против DeepFilterNet мы пока не гоняли, и расставлять их по местам по чужим демо не будем. Попробовали убрать чужой голос С VoxLock собрали отдельную страницу. Сначала записываешь шесть секунд своего голоса в тишине, из них берётся трёхсекундный образец. Затем записываешь или загружаешь звук с фоном. После обработки можно переключать дорожки и слушать разницу. Образец сделали отдельным шагом: нужного говорящего задаём сами, модель не выбирает его из общей записи. Для захвата взяли наш AudioWorklet — 16 кГц, блоки по 20 мс. У браузера запрашивали шумоподавление и эхоподавление, автоматическую регулировку усиления выключили. Здесь есть тонкость с обозначением «до обработки»: при записи микрофона браузер уже мог обработать сигнал. Импортированные файлы шли без этого этапа. Первый прототип использовал FP16-модель через ONNX Runtime Web и WASM. Получили выходную запись, сделали прослушивание и экспорт WAV. Дальше проверили скорость работы по блокам. На Intel Core i5-11400H под Windows 12 секунд звука обрабатывались 25,92 секунды. Размер блока — 80 мс, четыре потока WASM. Во время разговора с такой скоростью мы будем всё сильнее отставать от входящего звука. Из браузера — в нативный процесс Следом попробовали ONNX Runtime Node, CPU и модель INT8 full. Вычисления шли в отдельном локальном процессе, браузер общался с ним через WebSocket на 127.0.0.1. До рабочего сервера БОЛТУНа звук не доходил. По сохранённым прогонам получилось так: Вариант Блок Время на 12 секунд звука RTF WASM, FP16, 4 потока 80 мс 25,92 с 2,16 Native CPU, INT8 full, 2 потока, после настройки 40 мс 7,58 с 0,63 Та же нативная конфигурация, повторный прогон 40 мс 10,51 с 0,88 RTF — время обработки, делённое на длительность звука. Меньше единицы — запись обработана быстрее, чем она звучит. Отдельные блоки при этом всё ещё могут опаздывать. В этой таблице меняется сразу несколько вещей: рантайм, модель, число потоков, размер блока. По ней нельзя сказать, что native сам по себе быстрее WASM во столько-то раз. Это последовательность наших попыток ускорить конкретный прототип. С потоками тоже поэкспериментировали. В предварительной серии переход с двух на четыре сократил время с 7,26 до 6,70 секунды. Оставили два. Ещё ограничили активное ожидание между задачами — время, когда поток продолжает занимать CPU в ожидании следующей работы. Затем подавали тестовую запись через виртуальный микрофон в реальном времени. В 20-секундном и минутном прогонах прерываний из-за нехватки данных для воспроизведения не было. В минутном тесте 95% ответов локального обработчика укладывались в 39,4 мс. К этому ещё добавляются накопление входного блока, сдвиг модели, буфер воспроизведения и работа звукового устройства. Сквозную задержку «микрофон — наушники» этой цифрой мы не измерили. За минуту обработчик потратил около 106 CPU-секунд, то есть в среднем 1,76 CPU-секунды на секунду работы. Память процесса до и после теста — примерно 276 и 266 МиБ. Это только нативный обработчик, без браузера и игры. И в повторном прогоне обработка той же записи заняла уже 10,51 секунды вместо 7,58. Причину разброса пока не нашли. Для нас это повод проверить запас под нагрузкой: у пользователя параллельно будет работать игра, а не только наш стенд. Сервер тоже попробовали Возникла следующая мысль: если не хочется нагружать компьютер игрока, можно перенести модель на сервер. Сначала проверили такую схему на том же ПК. Подняли локальный обработчик и несколько клиентов, которые отправляли звук по расписанию. Генератор не ждал ответа на предыдущий блок: если обработчик тормозил, входящая нагрузка от этого не уменьшалась. Образец голоса и состояние модели у каждого входа были свои. Веса переиспользовались внутри каждого worker. В экономичной конфигурации взяли блоки по 80 мс, до шести workers и по одному вычислительному потоку на worker. На ответ отвели 250 мс с момента готовности блока. Вот сколько блоков уложилось в этот срок за 20 секунд: Непрерывных входов Обработано вовремя 2 100% 5 100% 10 51,6% 40 10,99% Проценты посчитаны от всех отправленных блоков — отвергнутые и опоздавшие тоже учтены. В эти 250 мс не входят накопление блока, интернет, Opus и воспроизведение у собеседника. Это выбранный допуск нагрузочного стенда. Пять входов затем прогнали в течение минуты. Все 3750 блоков обработались вовремя, расход составил 4,61 CPU-секунды на секунду работы, пик памяти — около 796 МиБ. С десятью непрерывными входами эта конфигурация уже не справлялась. Перенос на сервер избавил бы игрока от локальных вычислений, но нам пришлось бы обеспечить ресурс для всех активных потоков. Здесь важно не перепутать входы с участниками комнаты. Сорок человек не обязательно говорят одновременно. В тесте были непрерывные потоки одной открытой записи, сдвинутые относительно друг друга. Проверяли вычислительную нагрузку, а не разговор сорока разных людей. GPU и объединение нескольких клиентов в один вызов модели в этот стенд не добавляли. Пока занялись обработкой попроще После этих замеров тяжёлую изоляцию отложили. Решили отдельно разобраться с поведением микрофона в паузах и на тихих словах. В новом стенде сравниваются существующий WebVoiceEngine БОЛТУНа и экспериментальная пороговая обработка. На вход обоим приходит одна запись после браузерных NS/AEC. Обычный вариант запускаем с усилением 100%, чувствительностью 94 и включённым шумоподавлением. Сохранённые настройки пользователя не читаем, чтобы исходные условия не менялись от запуска к запуску. Начали с калибровки. Две секунды записываем фон, затем ждём нажатия второй кнопки и записываем пять секунд речи. Между этапами можно спокойно подготовиться. Если голос по уровню почти не отличается от фона, просим повторить настройку ближе к микрофону: повышение порога здесь может отрезать тихую речь. Автоматическое увеличение усиления убрали. В обычной ветке при включённом шумоподавлении есть компенсация ×1,18; в экспериментальной её нет. Во время калибровки усиление можно уменьшить при риске перегрузки, но поднимать его вместе с фоном не стали. Сам обработчик устроен так: Для открытия нужны два последовательных блока выше порога — 40 мс. Порог закрытия ниже порога открытия. Это позволяет не закрывать передачу сразу, как только голос стал немного тише. Храним до 100 мс предыдущего звука. После открытия возвращаем этот запас, чтобы сохранить начало слова. После падения уровня есть удержание на 240 мс и плавное ослабление тихих участков. Запас предыдущего звука, pre-roll, сейчас проверяется в офлайн-записи: сохранённые блоки возвращаются на свои места. Как это повлияет на задержку и воспроизведение в живой комнате, нужно будет проверить при подключении к звонку. Громкий чужой голос такой обработчик пропустит. Два одновременно говорящих человека останутся в одной дорожке. Со стуком тоже нет гарантии: короткий всплеск может не открыть передачу, но достаточно громкий или длинный пройдёт. Тихие слова ниже порога всё ещё могут пострадать. Это те случаи, ради которых и нужно прослушивание. Сломали даже настройку микрофона При попытке настроить лёгкий вариант получили «Не удалось уверенно услышать речь». После этой ошибки стенд не давал даже записать микрофон и послушать, что с ним происходит: запись была завязана на обязательную калибровку. Пришлось переделать. Теперь запись доступна сразу, в том числе после неудачной настройки. Добавили название выбранного устройства и индикатор уровня. Можно сначала услышать микрофон, а уже потом разбираться с порогами. Почему именно тот микрофон присылал низкий уровень, пока не установили. Но диагностику хотя бы больше не блокирует сама калибровка. Что получилось на лёгком варианте В A/B-плеере можно менять вариант обработки, не перематывая запись. Для двух обработанных дорожек выравниваем применяемое усиление речи, чтобы разница в громкости не мешала сравнению. В сохранённом браузерном прогоне лёгкий DSP обработал 15 секунд записи за 56,6 мс. Это вычисления над готовой записью, без времени браузерного шумоподавления и без измерения задержки звонка. Напрямую сравнивать эти 56,6 мс с VoxLock бессмысленно: лёгкий вариант управляет уровнем сигнала в паузах, а VoxLock пытается выделить нужный голос из смеси. Автотестами проверили калибровку, короткий всплеск, возврат начала речи из pre-roll, окончания слов, выравнивание усиления и ограничение памяти. В браузере — запись, переключение дорожек, экспорт WAV, отмену и отключение микрофона. Вычислительная часть работает; качество на разных микрофонах предстоит оценить ушами. Пока на этом и остановились. Изоляцию по образцу сохранили отдельным экспериментом. Лёгкую обработку будем проверять на тихой русской речи, с паузами, короткими словами и меняющимся фоном. Потом — во время игры. RNNoise и DeepFilterNet из списка не вычёркиваем. До собственного сравнения с ними ещё не дошли, поэтому утверждать, что выбрали лучший вариант, оснований нет. Для БОЛТУНа сейчас важнее не потерять половину тихой фразы и оставить игре её ресурсы. С шумом продолжим разбираться. Пилим дальше))) Замеры сделаны 23–26 октября 2026 года на локальном ПК с Intel Core i5-11400H под Windows. Браузерные автотесты запускались в Edge headless. Нейросетевые прогоны использовали открытые записи, а потоковые браузерные проверки — виртуальный микрофон. Рабочие комнаты, Android и Tauri/WebView2 в этих тестах не участвовали. Тестовый звук взят из демо VoxLock автора Arbaz Siddiqui, ревизия 08ed37cbb79f449ca6b499bc23fb14138b207bfd. Источник речи — LibriSpeech test-clean, CC BY 4.0, с моделированием комнаты автором демо. Индивидуальные записи пользователей в статье не используются.

Offline speech-to-text for Flutter, with no server and no API key At work I build real-time transcription with cloud speech APIs. They are very good. They also need a network connection, an account, a key, a bill, and a decision about whose servers your users' voices go to. I wanted to know how close a phone could get on its own. So I built transcribe_cpp, a Flutter package that turns speech into text entirely on the device, on iOS and Android. The user talks, your app gets text, and the audio never leaves the phone. It started as an experiment and a way to learn. It is now on pub.dev, MIT licensed, and the code is on GitHub. What it does - Private. No server, no account, no API key. It works in airplane mode. - Hands-free. There is no button to hold. Voice detection decides when each sentence starts and ends. - Live text. Words appear while the person is still talking, and settle in place without flickering. - Fast. On a mid-range phone, the recommended model transcribes speech about 40 times faster than real time. - Survives real life. Phone calls, Siri, headphones plugged in or pulled out, the screen locking, the app going to the background. - Bring your own audio. If your app already records, hand the audio over at any sample rate and skip the microphone handling entirely. Add It flutter pub add transcribe_cpp Using it import 'package:transcribe_cpp/transcribe_cpp.dart'; // Download a model once (90 MB). Later calls return it straight away. final models = await ModelManager.open(); final model = await models.ensure(const ModelId('parakeet-tdt_ctc-110m-Q4_K_M')); // Load it, and listen. final session = await TranscribeSession.open(model); await session.start(); // Show the text as the user speaks. ValueListenableBuilder( valueListenable: session.text, builder: (context, text, _) => Text(text.full), ); await session.stop(); That is the whole thing. Add a microphone permission to Info.plist and the Android manifest, and you have live dictation. If you already capture audio yourself, from a call, a file or your own recorder: await session.transcribeStream( myAudio, format: const AudioFormat(sampleRate: 48000), ); Standing on transcribe.cpp I did not write a speech recognizer. The models run on transcribe.cpp, a C++ engine built on ggml, the library behind llama.cpp and whisper.cpp. It runs a whole catalogue of open speech models, converts them, and publishes their accuracy. transcribe_cpp is the part between that engine and a Flutter app: the microphone, deciding when someone is speaking, splitting speech into sentences, live text, and staying alive through everything a phone throws at a long recording. The model I recommend is NVIDIA's Parakeet 110M: a 90 MB download with a 2.5% word error rate on the LibriSpeech benchmark. Models are downloaded when the app first needs them, so your app stays small; the package itself adds about 6 MB. What turned out to be hard Getting a model to produce text was the easy week. Everything around it took the time. Live text that doesn't flicker. The models I use transcribe a whole sentence at once; they don't stream word by word. To show text while someone is still talking, the package re-reads the unfinished sentence every 350 ms. But the last word or two of each read is a guess, because the model hasn't heard what comes next yet. Show those guesses and the text jitters as words rewrite themselves. So a word only appears once two reads in a row agree on it. The text only grows, and it feels live. Knowing when a sentence ends. With no button, something has to decide when you've finished speaking. Wait too long and the text feels slow; cut too early and you split sentences mid-thought, which costs accuracy. The package runs Silero VAD, a small voice-detection model I ported to plain C, and an endpointer tuned by measuring, not by feel. Most errors came from where sentences were cut, not from the recognition itself. Phones interrupt you. iOS stops your audio for a phone call or Siri and tells you when you can have it back. Headphones change the microphone's sample rate under you. Android doesn't stop the recording at all when a call takes the microphone: it quietly feeds you silence. And iOS refuses GPU work from an app in the background, so when the screen locks, the package moves decoding to the CPU and back to the GPU when the app returns. Each of these needed its own handling, so a long session keeps going without the app babysitting it. Two bugs every Flutter plugin author should know about. Since Flutter 3.29, Dart runs on the platform's main thread, so stopping the microphone froze my UI animations while the audio system shut down. And on iOS, release builds silently removed the C functions Dart calls through FFI, while debug builds hid the problem completely. Both will get their own posts. Try it - Package: pub.dev - Docs: ishangavidusha.github.io - Code and example app: ishangavidusha/transcribe_cpp The example app lets you browse the whole catalogue of 58 models, download a few, and switch between them on your own phone in a few minutes. That beats any number I could give you. It is early. Android runs on the CPU for now, and Bluetooth headset microphones work on iOS but not yet on Android. If you try it, I'd like to hear how it goes: bug reports, questions and ideas are all welcome in GitHub Discussions or the issues. Thanks to the transcribe.cpp authors, the ggml project, the Silero team, NVIDIA and the Moonshine team, whose work this is built on.

Architecting Real-Time Voice AI Pipelines for Phone Calls: Streaming STT, LLM, TTS, and Barge-In Building a voice AI prototype in a browser tab takes an afternoon. Making that same voice agent hold a natural conversation over a live telephone line is a distributed systems engineering problem. The fix is a pipelined architecture where every downstream stage starts working on partial output before the upstream stage finishes. Concretely, that means wiring together five pieces end to end: 1. Bridging narrowband PSTN/SIP telephony streams to full-duplex WebSockets. 2. Combining local audio Voice Activity Detection (VAD) with word-timing endpointing so noisy phone lines do not break turn detection. 3. Streaming LLM tokens into TTS at safe syntactic clause boundaries without clipping prosody or choking on abbreviations. 4. Executing full-stack barge-in by flushing telephony gateway playback buffers and truncating conversation history to what the caller actually heard. 5. Decoupling synchronous webhook and CRM tool calls from the audio loop. In human conversation, turn-taking is remarkably fast. A cross-linguistic study by Stivers et al. (2009) in the Proceedings of the National Academy of Sciences found that across ten languages, the cross-language mean response offset was about 208 milliseconds, while the overall modal (most common) offset was 0 milliseconds, meaning the single most frequent pattern was the next speaker starting at almost the same instant the current speaker finished, not after a gap. Because preparing a single spoken word takes roughly 600 milliseconds before articulation begins (an estimate from picture-naming studies, reviewed by Levinson and Torreira (2015) in Frontiers in Psychology), human listeners do not wait for silence to start thinking; they overlap comprehension with response formulation while the other person is still speaking. In telecommunications network planning, ITU-T Recommendation G.114 recommends that one-way transmission delay for general network planning should not exceed 400 milliseconds, while cautioning that highly interactive tasks can be perceptibly affected by much lower delays than that ceiling. When you place an AI pipeline in the middle of a phone call, every millisecond of silence after the caller stops speaking adds to that turn-taking gap. If your backend waits for Speech-to-Text (STT) to finalize a full paragraph, then waits for a Large Language Model (LLM) to generate a complete answer, and finally waits for Text-to-Speech (TTS) to render a full audio file before streaming bytes back to the carrier gateway, the caller hears dead air and assumes the call dropped. At Rymi, we build voice AI infrastructure that exposes one API for STT, LLM, TTS, and telephony, with voices for English and Hindi, including Hinglish. While our earlier guide on production voice AI agents covered provider trade-offs and failure fallbacks, this deep dive focuses on the protocol and orchestration mechanics behind the five pieces above. --- 1. The Telephony-to-Model Pipeline: What Happens on a Phone Call A browser-based voice demo captures wideband microphone audio (typically 16 kHz or 48 kHz) over WebRTC. A standard public switched telephone network (PSTN) call operates under much stricter physical and protocol constraints: 1. Call Signaling (IETF RFC 3261): Session Initiation Protocol (SIP) negotiates the call session, codecs, and media endpoints between the carrier and your Session Border Controller (SBC) or telephony provider (such as Twilio, Plivo, Vonage, or Telnyx). 2. Media Transport (IETF RFC 3550): Once the call connects, audio frames travel over the Real-time Transport Protocol (RTP), carrying sequence numbers and timestamps so the receiver's jitter buffer can reorder out-of-sequence UDP packets. 3. Narrowband Audio Encoding (ITU-T Recommendation G.711): PSTN audio is typically encoded using G.711 pulse code modulation (PCM) at an 8,000 Hz (8 kHz) sampling rate and 64 kbit/s bitrate, using either mu-law (PCMU, standard in North America and Japan) or A-law (PCMA, standard in India, Europe, and international routes). G.711 narrowband audio is typically limited to roughly 300 to 3,400 Hz, well below the frequency range that carries some high-frequency fricative distinctions (such as "s" vs. "f"), which can make narrowband audio harder for speech recognizers trained primarily on wideband audio. 4. Full-Duplex WebSocket Bridging (IETF RFC 6455): RFC 6455 defines the WebSocket protocol's persistent, bidirectional connection and message framing. Media gateways use this transport to bridge the RTP stream to your orchestrator; the specific chunk size and control-event format, such as 20 ms audio chunks and media/mark/clear events, are defined by the telephony provider's media-streaming product, such as Twilio's Media Streams WebSocket messages. !Flow diagram showing a phone caller's audio passing through a carrier media gateway into an orchestrator with VAD, streaming STT, LLM, and TTS stages connected in sequence End-to-end telephony voice AI pipeline bridging 8 kHz G.711 SIP/RTP phone audio over a full-duplex WebSocket through local Silero VAD, streaming STT, clause-chunked LLM generation, and streaming TTS. In a sequential architecture, total turnaround delay is the sum of the full completion time of every stage: Sequential turn delay = T_Endpointing + T_Full_STT + T_Full_LLM + T_Full_TTS + T_Network In a pipelined streaming architecture, each stage emits incremental frames to the next stage before the upstream task finishes. STT transcribes while the user speaks; the LLM begins generating tokens as soon as the turn endpoint triggers; TTS begins synthesizing audio as soon as the first clause forms; and the carrier gateway plays the first G.711 audio packets while the LLM is still generating the second sentence. Pipelined turn delay = T_Endpointing + T_STT_Finalization + T_LLM_TTFT_First_Clause + T_TTS_TTFB + T_Network --- !Bar chart comparing conversational turn-taking gaps and ITU-T G.114 delay guidance against the stage-by-stage timing of sequential versus pipelined voice AI architectures Conversational and ITU-T G.114 latency benchmarks compared against sequential and pipelined voice AI turn stages, using sourced millisecond thresholds and qualitative streaming labels. 2. Dual-Signal Turn Detection: Combining Audio VAD with Word-Timing Endpointing Determining when a caller has finished speaking is one of the hardest trade-offs in voice engineering: - If your endpointing threshold is too short, the agent interrupts the caller every time they pause to breathe, recall an order number, or switch languages mid-sentence. - If your endpointing threshold is too long, every single turn suffers from an artificial silence penalty before the LLM is even invoked. Relying on a single silence timer fails on real phone calls because phone environments are noisy. Autorickshaw traffic, office chatter, ceiling fans, or carrier line hum can keep an audio-only Voice Activity Detector open indefinitely, preventing the agent from ever responding. Layer 1: Fixed-Window Acoustic VAD On the audio ingest path, lightweight neural voice activity detectors evaluate raw audio frames to distinguish human speech from stationary background noise. For example, Silero VAD (snakers4/silero-vad) processes audio in strict 32-millisecond windows: 256 samples per frame at an 8 kHz telephony sample rate (256 / 8000 = 0.032 s) or 512 samples per frame at 16 kHz (512 / 16000 = 0.032 s). Running an acoustic VAD on incoming frames gives your orchestrator an immediate local signal when speech starts (speech_start) and when acoustic energy drops. Layer 2: Combining endpointing and utterance_end_ms in Streaming STT When streaming audio over WebSockets to a speech recognizer such as Deepgram, one workable design combines two complementary mechanisms documented in the Deepgram Streaming STT specification: 1. VAD-Based Endpointing (endpointing): Deepgram's endpointing parameter monitors the incoming audio waveform for a transition from speech to silence (configurable in milliseconds; while the API default is 10 ms, conversational agents commonly configure 300 to 500 ms so brief mid-sentence pauses do not split a thought). When that silence window elapses, the recognizer locks the segment and emits a transcript response with speech_final: true. 2. Word-Timing Gap Detection (utterance_end_ms + interim_results): Because non-speech background noise on a phone line can prevent audio VAD from detecting silence, Deepgram's utterance_end_ms parameter operates on word timestamps rather than raw audio silence. It requires interim_results=true and measures the time elapsed since the last finalized word. If no new finalized word arrives within the configured window (for example, 1000 ms), it emits an {"type": "UtteranceEnd"} control event over the WebSocket, even if background noise is still present, and even if the caller resumes speaking after a brief pause, so treat it as a fallback signal rather than a guaranteed turn boundary. | Mechanism | Signal Evaluated | Primary Strength | Failure Mode on Phone Calls | Role in Pipeline | | :--- | :--- | :--- | :--- | :--- | | Acoustic VAD (endpointing) | Raw audio waveform energy and silence | Fast detection on quiet phone lines | Loud background noise can keep the stream open (speech_final never fires) | Fast-path turn trigger (one common design) | | Word-Timing Gap (utterance_end_ms) | Time gap since last finalized word in interim_results | Ignores non-speech background noise | Needs a slightly longer timeout window to avoid premature firing | Fallback turn trigger (one common design) | In your orchestrator, accumulate finalized transcript segments as they arrive and use speech_final: true as the signal that the accumulated segments form a complete turn ready to dispatch to the LLM. A single utterance can span multiple is_final: true responses, so dispatching only the segment attached to the speech_final event can clip earlier words from the turn. Use UtteranceEnd as the safety net whenever speech_final has not fired due to line noise. --- 3. Clause-Boundary Token Streaming Without Breaking TTS Prosody Once the turn transcript is dispatched to the LLM via a streaming API (such as Anthropic's Server-Sent Events streaming messages or OpenAI's streaming chat completions), tokens arrive incrementally. A common mistake in early voice AI implementations is splitting the LLM token stream with a naive regular expression like /[.,?!]/ and sending every fragment straight to TTS. In production, naive punctuation splitting causes three problems: 1. Abbreviation and numeric clipping: Splitting on every period breaks sentences in half on honorifics (Dr., Mr., Mrs.), currency and measurements (Rs. 500, $19.99, 3.5%), or initials (U.S., e.g.). 2. Starved TTS prosody: As documented in the ElevenLabs WebSocket Streaming TTS reference, the synthesizer buffers incoming text against a chunk_length_schedule (an array of cumulative character thresholds such as [120, 160, 250, 290]) before generating audio frames unless forced to flush. In this design, sending two-word fragments with forced flushes trades away the forward text context the model uses for pitch contour and word stress, producing more robotic, choppy intonation. 3. Code-mixed punctuation: In Indian phone conversations where callers and agents speak Hinglish (Hindi-English code-mixing) or Devanagari alongside Latin script, sentence boundaries may use the Devanagari danda (।) in addition to standard ASCII punctuation. Implementation: Abbreviation-Safe Clause Buffer Instead of splitting on raw punctuation characters, buffer incoming LLM tokens until both a minimum character threshold is met and a safe syntactic boundary is followed by whitespace: const COMMON_ABBREVIATIONS = new Set([ 'mr', 'mrs', 'ms', 'dr', 'prof', 'sr', 'jr', 'rs', 'st', 'no', 'vs', 'etc', 'eg', 'ie', ]); export interface ClauseChunkerOptions { minClauseChars?: number; softSplitChars?: number; onClause: (clause: string) => void; } export class AbbreviationSafeClauseChunker { private buffer = ''; private readonly minClauseChars: number; private readonly softSplitChars: number; private readonly onClause: (clause: string) => void; constructor(options: ClauseChunkerOptions) { this.minClauseChars = options.minClauseChars ?? 45; this.softSplitChars = options.softSplitChars ?? 110; this.onClause = options.onClause; } public pushToken(token: string, signal?: AbortSignal): void { if (signal?.aborted) return; this.buffer += token; this.extractClauses(signal); } public flush(signal?: AbortSignal): void { if (signal?.aborted) { this.buffer = ''; return; } const remaining = this.buffer.trim(); this.buffer = ''; if (remaining.length > 0) { this.onClause(remaining); } } public reset(): void { this.buffer = ''; } private extractClauses(signal?: AbortSignal): void { // Match terminal punctuation (. ? ! or Devanagari danda ।) or soft clause // punctuation (, ; :) followed by whitespace const boundaryRegex = /([.?!।]|[,;:])(\s+)/g; let match: RegExpExecArray | null; while ((match = boundaryRegex.exec(this.buffer)) !== null) { if (signal?.aborted) { this.buffer = ''; return; } const punct = match[1]; const splitIndex = match.index + punct.length; const candidate = this.buffer.slice(0, splitIndex).trim(); if (candidate.length < this.minClauseChars) { continue; } const isTerminal = /[.?!।]/.test(punct); if (!isTerminal && candidate.length < this.softSplitChars) { // Hold commas/semicolons unless the clause is already long enough continue; } if (punct === '.' && this.isAbbreviationOrDecimal(candidate)) { continue; } this.onClause(candidate); this.buffer = this.buffer.slice(splitIndex + match[2].length); boundaryRegex.lastIndex = 0; } } private isAbbreviationOrDecimal(candidate: string): boolean { // Do not split if the period is preceded by a single letter (e.g. "U.S.") // or a known title/currency abbreviation (e.g. "Dr.", "Rs.") const lastWordMatch = candidate.match(/([A-Za-z]+)\.$/); if (lastWordMatch) { const word = lastWordMatch[1].toLowerCase(); if (word.length === 1 || COMMON_ABBREVIATIONS.has(word)) { return true; } } return false; } } This pattern lets you dispatch the opening sentence to the TTS WebSocket as soon as a complete, prosodically stable thought is ready, while longer sentences automatically split at commas or semicolons once softSplitChars is reached. --- 4. Full-Stack Barge-In: Flushing Media Gateways and Truncating Context In a natural phone call, callers interrupt the agent when they already know the answer, when the agent misunderstood their request, or when they want to skip a long explanation. Most developer tutorials treat interruption handling (barge-in) as calling AbortController.abort() on the active LLM request. On a real phone call, aborting the LLM is only half of the job. Why Backend-Only Cancellation Fails on Phone Calls When your TTS engine synthesizes audio faster than real-time playback, it pushes several seconds of G.711 audio packets into the telephony gateway's outbound buffer. If the caller interrupts and you only cancel the LLM and TTS tasks on your server, the telephony gateway will continue playing the buffered audio for several more seconds before silence occurs. Furthermore, if the LLM had already streamed three sentences before the caller interrupted during sentence one, saving all three sentences in the assistant's conversation history causes state drift: on the next turn, the LLM believes the caller heard facts in sentences two and three that were never actually played over the phone line. The Four-Step Barge-In Protocol Using bidirectional telephony WebSocket messages (such as the media, mark, and clear events specified in Twilio's Media Streams WebSocket documentation), a production orchestrator handles barge-in across four coordinated steps: 1. Tag outbound audio segments with mark checkpoints: Every time you stream synthesized G.711 audio for a clause (clause_0, clause_1, clause_2), send a trailing mark message over the gateway WebSocket. The gateway echoes that mark event back to your server once the associated buffered media has completed playback, but note that sending a clear command can also cause outstanding marks to be returned immediately, so a mark echo alone does not always mean the audio played to completion. 2. Filter passive backchannels from real interruptions: Callers frequently say brief acknowledgments ("uh-huh", "okay", "right", or "haan" and "accha" on Indian phone calls) while listening. If your acoustic VAD mutes playback on every 100 ms sound burst, a polite listener will constantly stall the agent. Require either sustained speech energy across consecutive Silero VAD frames or a multi-word interim STT transcript before firing a full barge-in. 3. Send a clear command to flush the gateway buffer: The instant a true interruption is confirmed, send {"event": "clear", "streamSid": "..."} over the telephony WebSocket while triggering your local AbortController. The gateway immediately purges its outbound audio buffer so the phone speaker goes silent. 4. Truncate assistant history to confirmed playback: When recording the interrupted assistant turn in your LLM conversation context, track each dispatched clause's state as pending, playing, played, or cleared. Include only clauses whose mark checkpoint was echoed back as played before the interruption; do not assume the clause that was actively playing when clear was sent was fully heard, since a clear command can cause the gateway to return a mark for audio it never finished playing. !Timeline diagram showing mark checkpoints on outbound audio, the moment an interruption is confirmed, the gateway clear command, and where conversation history is truncated Full-stack barge-in sequence showing outbound mark tracking, VAD and interim STT interruption confirmation, gateway buffer clearing, LLM/TTS cancellation, and conversation history truncation. type ClauseState = 'pending' | 'playing' | 'played' | 'cleared'; interface ClauseCheckpoint { markId: string; text: string; state: ClauseState; } export class TurnPlaybackTracker { private checkpoints: ClauseCheckpoint[] = []; public registerClause(markId: string, text: string): void { this.checkpoints.push({ markId, text, state: 'pending' }); } public onClauseDispatchedToGateway(markId: string): void { const item = this.checkpoints.find((c) => c.markId === markId); if (item && item.state === 'pending') { item.state = 'playing'; } } public onGatewayMarkEchoed(markId: string): void { const item = this.checkpoints.find((c) => c.markId === markId); // A clear command can echo marks for audio that was never actually // played, so a mark already flagged `cleared` must not flip to `played`. if (item && item.state !== 'cleared') { item.state = 'played'; } } public onGatewayCleared(): void { for (const item of this.checkpoints) { if (item.state === 'pending' || item.state === 'playing') { item.state = 'cleared'; } } } public getSpokenTextOnInterrupt(): string { // Only clauses confirmed `played` are known to have reached the caller; // a clause that was merely `playing` when `clear` fired is not included // because how much of it was actually heard is unknown. return this.checkpoints .filter((c) => c.state === 'played') .map((c) => c.text) .join(' '); } } --- 5. Decoupling Mid-Call Tool and Webhook Execution from the Audio Loop Voice agents on real phone calls rarely exist just to chat. They check appointment slots, verify order status, look up CRM records, or trigger payment links via external webhooks and structured LLM function calling. If your voice orchestrator awaits a synchronous HTTP request to an external CRM or booking API inside the turn loop before emitting any audio, the caller hears silence for the entire duration of the third-party API call. To keep the conversation alive during external I/O: 1. Emit an immediate verbal bridge before awaiting the tool: Configure your system prompt or tool router so that when the model emits a tool call block, the orchestrator immediately dispatches a brief, context-appropriate acknowledgment clause to TTS ("Let me check your booking details right now.") while the HTTP request executes concurrently. 2. Bind tool execution to the turn's AbortSignal: Callers often add extra details while the agent is looking something up ("Actually, check under my other phone number..."). Pass the turn's AbortSignal into fetch() for read-only lookups so stale requests are cancelled cleanly if the caller barges in, and use idempotency keys on state-mutating webhooks so retries never double-book an appointment. 3. Enforce strict timeouts with conversational fallbacks: Never allow an external webhook to hang a phone call indefinitely. Wrap every tool invocation in a hard timeout budget so that if a downstream CRM is slow or unreachable, the orchestrator catches the timeout and instructs the agent to offer an alternative path (such as escalating to a human representative or following up via SMS). --- 6. Architectural Checklist for Production Voice Pipelines Before routing live phone traffic to a voice AI pipeline, verify each stage of your stack against these engineering checks: | Pipeline Layer | Engineering Requirement | Why It Matters on Phone Calls | | :--- | :--- | :--- | | Telephony & Codec | Native 8 kHz G.711 (PCMU / PCMA) handling and STT models tuned for narrowband audio, where available (ITU-T G.711) | Helps limit accuracy loss on narrowband carrier audio and can reduce unnecessary transcoding hops | | Turn Detection | Dual-signal endpointing: acoustic VAD (endpointing) paired with word-timing gap detection (utterance_end_ms) | Prevents noisy phone lines from keeping the turn open forever while respecting natural pauses | | LLM-to-TTS Handoff | Abbreviation-safe clause chunker with minimum character thresholds | Prevents clipped speech on Dr., Rs., and decimals while feeding TTS enough context for natural prosody | | Barge-In & State | Gateway clear message dispatch + mark-based context truncation (Twilio Media Streams) | Stops phone playback immediately on interruption and keeps LLM memory synced with what the caller heard | | Tool & Webhook I/O | Asynchronous tool execution with verbal acknowledgment and hard timeouts | Prevents slow external CRMs or APIs from creating dead air mid-call | | Multilingual & Code-Mixing | Support for mid-call language switching and code-mixed speech (such as Hinglish) | Callers in multilingual markets frequently switch between English and local languages within a single call | --- Building Without Wiring Every Protocol Layer Yourself Putting these pieces together from scratch means maintaining SIP/RTP media bridges, WebSocket state machines, VAD frame buffers, abbreviation-aware token chunkers, gateway clear/mark synchronization, and post-call webhook pipelines across multiple model vendors. If you want to focus on your agent's logic and tools rather than maintaining the underlying voice infrastructure, Rymi provides a unified voice AI platform behind one API for STT, LLM, TTS, and telephony, with voices for English and Hindi, including Hinglish. You can bring your own carrier numbers (Twilio, Plivo, Vonage, or Telnyx) and choose between tuned managed stacks or custom bring-your-own-key stacks. Pricing is straightforward: Managed from $0.05/min, or BYO providers at cost plus $0.02/min runtime. Explore how Rymi compares to other platforms on our comparison page, read the API documentation, or create a free account to test an agent directly from your browser.

correction: Silero outputs a speech probability estimate for each audio chunk, and its threshold applies to that estimate. 'raw decibel gating' above was inaccurate github.com/snakers4/silero-vad/blob/master/src/si…