I built an open-source dictation app in Tauri and Rust. Here are 5 things that broke.
I type a lot less than I used to. I hold two keys, say what I want, let go, and the text shows up wherever my cursor is. Slack, VS Code, a Gmail draft, a terminal. It works with the Wi-Fi off.
That app is Talvio. It's open source (MIT), built with Tauri 2, Rust and React, and it runs on Windows and Linux today. A macOS build is coming.
GitHub: ishangautam7/Talvio
!Talvio's home screen next to the phone app
This post is half "here's what I made" and half "here's what broke". If you build desktop apps with Tauri, the second half is probably the useful part.
Why I built it
I wanted dictation that worked in every app, not just in one website's text box. I also didn't want my voice going to a server every time I wrote a Slack message.
I found Handy by CJ Pais. It's a small, open source, offline speech-to-text app, and it did the core thing well: hold a key, talk, paste. I forked it. The whole local dictation core (the Whisper/Parakeet pipeline, voice activity detection, the recording overlay, the settings architecture) comes from that project, and I still pull fixes from upstream. Go give it a star too.
Then I kept adding the things I wanted on my own machine. That fork grew into Talvio.
What it does
Dictation. Hold Ctrl + Win on Windows (Ctrl + Space on Linux), talk, let go. A small pill at the bottom of the screen shows it's listening, and with a streaming model you see your words as you speak.
!The recording pill: listening, live text, transcribing
Speech recognition runs locally through transcribe.cpp. Whisper, Parakeet, Nemotron and the rest of a 69-model catalog, on your GPU through Vulkan or on the CPU. No account needed. Cloud speech providers are optional and use your own key.
An assistant on another shortcut. Select some text, hold Ctrl + Alt, say "write a polite reply saying I can't make Thursday". It answers out loud, and an Insert button puts the answer back where the text was.
Voice commands. "Play Believer on YouTube." "Turn the volume down." "Email Ram that I'm late." These run in your own apps: Gmail opens with the email written, the song starts in your browser.
!The Commands page
Meeting notes. It records your mic and the system audio as two separate lanes, so it knows which words were yours. No bot joins the call. You get notes at the end.
!A meeting being recorded with two lanes
Smaller things. Optional AI cleanup that removes "um" and follows "at ten, actually no, eleven". A dictionary so names come out spelled right. And an Android app that turns your phone into a wireless mic, keyboard and remote for the PC.
!The phone app
The AI features (assistant, commands, cleanup) use your own API key, a local model through the built-in llama.cpp engine, or $1 of free credit if you sign in. Plain dictation needs none of that.
OK, the lessons.
1. Creating a window from a hotkey thread froze the whole app
The assistant panel opens when you press a global shortcut. The shortcut callback runs on the keyboard hook's thread, not on the main thread.
My first version built the WebView window right there in the callback. It worked most of the time. Sometimes it deadlocked the event loop, and then everything was dead: no overlay, no dictation, windows you couldn't drag.
The fix is boring. Every function that creates, shows or destroys that window queues the work on the main thread:
pub fn enter_conversation_size(app: &AppHandle) {
let app_main = app.clone();
if let Err(e) = app.run_on_main_thread(move || {
apply_panel_geometry(&app_main);
place_call_window(&app_main);
}) {
error!("Could not queue conversation panel sizing: {}", e);
}
}
The functions that actually touch the window are private. The public ones only queue. That had a side benefit I didn't plan for: a create is always ordered behind a destroy, so toggling the feature off and on fast can't leave the app thinking a window exists when it doesn't.
If you call anything window-related from a shortcut handler, a tray handler or a background thread in Tauri, go through run_on_main_thread.
2. Floating windows must never take focus
A dictation app pastes with a synthetic Ctrl+V. That keystroke goes to whatever has keyboard focus at that moment. If your own overlay grabbed focus, the transcript gets pasted into your overlay and disappears.
This happened. The overlay window was bigger than the visible pill (a transparent frame around it), so a click on "empty" space actually landed on my window, activated it, and the next paste went nowhere.
Each platform needs a different answer:
- Windows: the window gets WS_EX_NOACTIVATE, so even a click that lands on it can't make it the foreground window.
- macOS: an NSPanel that never becomes key.
- Linux (Wayland): a GTK layer-shell surface with KeyboardMode::None.
- Plain X11: none of the above works reliably, so the card there stays click-through until the paste is done.
The same rule now applies to every window that pops up on its own: the reminder popup, the "allow this phone?" card, the assistant. Anything that shows up while you are typing somewhere else should not move your cursor.
One more trap: keeping it on top. WS_EX_TOPMOST is a position another app can take from you. I set it once on show and users reported the pill "randomly hiding". Now a small guard re-asserts it every 700 ms while the overlay is up, and logs once when something displaced it. That log line turned a vague bug report into a fixable one.
3. Run the speech engine in a separate process
Local speech recognition on the GPU means running Vulkan compute shaders through whatever driver the user has. Some of those drivers crash. Some hit a GGML_ASSERT and abort. Some just hang.
When the engine ran inside the app, any of those took the whole app down, or left it sitting on "Transcribing…" forever.
Now every call into the speech engine goes to a worker process: the same executable started with --transcribe-worker. main.rs checks for that flag before anything else (before Tauri, logging or single-instance), so the worker is just a small loop reading requests from stdin.
The protocol is plain request/response, one frame per message:
[u32 json length][json][u32 pcm length][f32 pcm samples]
Ten minutes of audio costs about 10 ms to send this way. There's a 1 GiB cap per section so a garbage length can't turn into a huge allocation.
When a worker using the GPU crashes or hangs, the app marks the GPU unavailable for the session and retries once in a worker started with --cpu-only. That flag restricts the engine to CPU backends before any GPU code runs, so a driver that dies during init can't take the fallback down too. The user gets their text, a bit slower, and a log line saying why.
Two details that matter more than they look:
- On Windows the worker goes into a kill-on-close Job Object, and it exits when its stdin closes. A crashed app never leaves a worker eating your GPU memory.
- Esc goes through the engine's abort callback first and only kills the worker after 750 ms. Killing it every time meant reloading the model after every cancelled dictation.
4. My settings file got wiped on shutdown
Early on I used tauri-plugin-store for settings. Then a shutdown reset every setting. Shortcuts, models, everything back to defaults.
Two things combined. The plugin wrote the file in place, so a shutdown at the wrong moment (or Windows Defender holding the file) could leave it empty or half-written. And on the next launch, a file that didn't parse was silently treated as "no settings", so defaults were written over it.
The replacement is small:
pub fn write_atomically(path: &Path, bytes: &[u8]) -> io::Result<()> {
// write to a temp file next to the real one
let mut file = retry_while_held(|| File::create(&temp))?;
file.write_all(bytes)?;
file.sync_all()?;
drop(file);
// then swap it in with one rename
retry_while_held(|| fs::rename(&temp, path))?;
Ok(())
}
retry_while_held retries for a few hundred milliseconds, because on Windows an antivirus, the search indexer or a sync client often has the file open for a moment.
The rest is about what happens when the file is damaged anyway. A broken file is moved aside, not overwritten. Settings come back from a .bak copy that is only refreshed from a launch that read a good file. A read error (as opposed to bad content) never triggers a reset at all. And the user gets a notice instead of silently losing their setup.
If you store anything users care about, don't let "can't parse it" mean "start fresh".
5. You don't need paid code signing to ship auto-updates
I assumed self-update was tied to Apple and Windows code signing certificates. It isn't. Tauri's updater checks a minisign signature made with a key you generate yourself, for free. Code signing is a separate thing that only affects the "unknown publisher" warning on first install.
I had tied them together in CI. The result: from 1.0 to 1.4, no release had a working update path, and nobody noticed because the update UI was also hidden. Fun.
What the pipeline does now:
- Build with the updater key from repo secrets whenever it exists, signed or not.
- Re-sign the Linux AppImage and .deb after they are repacked. The bundler's signatures describe the bytes before repacking.
- Write latest.json once, at the end, from the signatures actually attached to the release, and verify each one against the published file before it goes live.
The app polls releases/latest/download/latest.json. GitHub only serves that from the newest published, non-prerelease release, so drafts and betas never reach anyone.
And back up that private key. If you lose it, installed copies can never update again.
Try it
- Download: GitHub releases (v2.2.0) or talvio.sentramind.dev
- Windows installer isn't code signed yet, so SmartScreen will ask. More info, then Run anyway.
- Linux: AppImage or .deb.
It's a solo project and there are rough edges. If something breaks on your machine, an issue with your GPU and OS helps a lot. If it's useful to you, a star on GitHub helps more people find it.
Repo: ishangautam7/Talvio
I'd also like to know: which of these did you hit in your own Tauri or Electron app? The focus one took me embarrassingly long to understand.
1 reaction
Mentions
今天逛 GitHub 发现一个很不错的开源项目，推荐给大家：Handy，一个完全离线的免费开源语音转文字工具，不用联网就能把录音转成文字，支持多语音模型和语言检测。语音正在成为 Agent 的标配入口，离线可用正好打中隐私和稳定性这个痛点。cjpais/Handy
好家伙，Handy 竟然能离线把语音转成文字。注意是离线！ 再也不用把录音上传到云端转写了，按住快捷键说完就直接粘贴到任意输入框，赶紧冲，不谢。 cjpais/Handy
今日のGitHubトレンド cjpais/Handy Handyは、完全オフラインで動作するオープンソースの音声文字起こしデスクトップアプリです。 ショートカットキーで録音し、Whisperを用いてローカルで処理することで、プライバシーを確保しつつ任意のテキストフィールドに文字起こし結果を直接入力できます。
3) Handy replaces paid dictation apps like Wispr Flow You press a shortcut, speak, and your words appear in any text box. Your voice stays on your computer, and one of its models is built to run on a normal processor. cjpais/Handy
做了个本地音频转写工具：不花一分钱， 76 个模型随取随用， 16 分钟视频 49 秒转完 最近把折腾了一阵子的两个东西都开源了，一条线上的：一个本地跑的语音转写命令行工具，一个长在它上面的 agent 技能（把音视频直接变成一篇能进笔记库的文章）。  为什么不用现成的 - 云 ASR 按分钟计费。一场 2 小时的会议、一周十几个播客，成本很快就不对劲，而且音频要上传到别人服务器。 - whisper.cpp 手搓太烦。编译参数、模型格式、VAD 分块、长音频拼接、中文断句……每换一台机器重来一遍。 - YouTube/B 站 有现成字幕，但只有一部分视频有，而且字幕没标点、断句乱，二次加工成本高。 所以做了两件事：一个把模型管理干净的 CLI，和一个把转写结果整理成笔记的技能。 一、handy-transcribe：Rust 写 CLI ，模型随取随用 它复用了开源桌面应用 [Handy]( cjpais/Handy) 的批处理流水线（解码 → VAD 分块 → transcribe.cpp 推理 → 拼接 → 后处理），去掉录音/UI/剪贴板，只留一个无头 CLI 。 - 完全本地：macOS 走 Metal ，其他平台 CPU 。不上传，不需要账号，不按分钟付费。 - 模型目录内置 76 个、21 个家族：Whisper 、Parakeet 、Canary 、SenseVoice 、Qwen3-ASR 、Voxtral 、Moonshine 、GigaAM… --list-models 一次看全，✓ 标记已缓存。 - 输入随便：本地 mp3/wav/flac/ogg/m4a/aiff ，或 MP4 家族视频容器的音轨；也接受直接的 http(s) 媒体 URL 。 - 输出干净：正文走 stdout （可以直接管道），进度和耗时走 stderr 。--json 给出 backend 、设备、RTF 、识别语言。 cargo install handy-transcribe # 或 cargo binstall handy-transcribe handy-transcribe ./meeting.m4a handy-transcribe ./talk.wav -m Qwen3-ASR-1.7B -l zh handy-transcribe ./podcast.mp3 --json YouTube 之类的网页先用 yt-dlp 抽音轨再喂进来——这是故意的，提取器交给专业项目维护，CLI 只管转写。 实测：16 分钟（ 972 秒）中文技术演讲，49 秒转完。机器是 M1 Pro ，模型 Qwen3-ASR-0.6B Q8_0 ，约 20 倍实时。文本准确到只需要改几个同音字。 二、pi-media-transcribe：转写完，自动变成一篇笔记 光有文本其实没什么用，还得有人分段、写摘要、挑目录、打标签。所以给 [Pi]( pi.dev)（一个开源 coding agent ，可以装技能包）写了个技能，三种模式： - 只转写：出文本（带缓存，重复跑是秒回）。 - --raw 逐字落笔记：不需要任何 API key ，全程离线，零成本。 - 默认模式：LLM 校对错字 → 按主题分 ### 小节 → 写 150~300 字摘要 → 在你的笔记库里挑目录 → 打 1~3 个主题标签 → 写成带 frontmatter 的 Markdown 。 pi install npm:pi-media-transcribe # 然后直接对 agent 说：把这个视频转录一下存到我的笔记里：<链接> 产物大概长这样： --- created_at: 2026/10/31 22:14 tags: - youtube - 自动转录 - RAG - 知识图谱 --- 来源： https://www.youtube.com/watch?v=... ## 核心内容 <150~300 字摘要> ## 正文 ### 传统向量检索的底层天花板 <校对并分段后的正文> 那次 16 分钟的视频，出来是一篇 18.8 KB 的笔记：一段摘要 + 17 个小节。落点在 资料/02-人工智能/AI 智能体/ —— 目录是它自己在现有目录树里挑的。 几个自己比较在意的细节： - 标题语言跟着内容走（中/英/日/韩/俄/德/西/法/葡九种），不是硬编码中文。 - 长音频不会爆上下文：超过阈值自动分块压缩再合并，三小时的播客也能过。 - LLM 失败不丢工作：直接落一份原始转写的笔记，重跑只重试润色那一步。 - 同名笔记不覆盖（自动加序号），路径校验在笔记库之内。 踩过的坑，顺便记一下 - 首次运行会静默下载 850 MB 的默认模型。后来在 SKILL 里加了硬规则：先告诉用户用哪个模型、多大、要不要换。 - 国内拉 HuggingFace 模型经常卡死，支持 HF_ENDPOINT（中文文档里给了镜像示例）。 - LLM 只看得到我给的目录列表，否则它会把笔记往任意路径写；路径还得再校验一次。 - 缓存键一开始用了 stat -f，在 Linux 上语义完全不同（ GNU 的 -f 是查文件系统），换成了「版本号 + 文件大小」。 已知限制（先说清楚） - Windows 没测过。CLI 依赖 CMake + C++ 工具链，macOS/Linux 的 CI 都绿了，Windows 欢迎有人帮忙试。 - 拉丁语系语言的标题回退英文（德/法/西/葡）。正文语言是对的，只是小节标题是英文；可以用 --language/配置指定。 - 润色模式花 token。只要文本不要笔记的话用 --raw，一分钱不花。 - 模型是第三方的，许可证各不相同（ Apache-2.0 / MIT / CC-BY-4.0 / 非商用），商用前看一眼。 想听听用的人怎么说 - 你主要拿它处理什么？会议、课程、播客、还是二次剪辑素材？ - 还需要哪些模型或语言的优先支持？ - 有 Windows 用户愿意帮忙跑一次 CI 之外的真实测试吗？ 两个仓库都是 MIT： - CLI： cyub/handy-transcribe （cargo install handy-transcribe） - 技能： cyub/pi-media-transcribe （pi install npm:pi-media-transcribe）
@SergioGMN TOP!, aunque yo prefiero Handy, es el que ha estado ahí desde el principio apoyando el opensource desde que se empezaron a comercializar los modelos speech to text cjpais/Handy
Game Changer: cjpais/handy Hab damit auf meinem MacBook Air M2 16gb die perfekte lokale Datenschutz sichere Eingabe Methode gefunden... Bye Bye wispr flow
Om det är för Windows så är nog Handy ett bra alternativ. Väldigt populär repo, finns färdig windows-installation. Funkar helt offline. Du kan ställa in så den inte sparar några transkriberingar. Kan numera även transkribera löpande, men har inte testat den funktionen själv. cjpais/handy
Handy: offline voice typing for Windows, macOS and Linux (MIT) Sharing Handy for anyone looking for a free, open-source dictation app. Press a shortcut, speak, and it transcribes into the text field you're using. You download a speech model first, then transcription runs locally. It supports Windows, macOS and Linux, and the code is MIT licensed. cjpais/Handy I'm not affiliated with the project. I'm especially interested in Portuguese dictation with English programming terms mixed in. Has anyone tried it for that? Which model worked best for you?
I built an open-source dictation app in Tauri and Rust. Here are 5 things that broke. I type a lot less than I used to. I hold two keys, say what I want, let go, and the text shows up wherever my cursor is. Slack, VS Code, a Gmail draft, a terminal. It works with the Wi-Fi off. That app is Talvio. It's open source (MIT), built with Tauri 2, Rust and React, and it runs on Windows and Linux today. A macOS build is coming. GitHub: ishangautam7/Talvio !Talvio's home screen next to the phone app This post is half "here's what I made" and half "here's what broke". If you build desktop apps with Tauri, the second half is probably the useful part. Why I built it I wanted dictation that worked in every app, not just in one website's text box. I also didn't want my voice going to a server every time I wrote a Slack message. I found Handy by CJ Pais. It's a small, open source, offline speech-to-text app, and it did the core thing well: hold a key, talk, paste. I forked it. The whole local dictation core (the Whisper/Parakeet pipeline, voice activity detection, the recording overlay, the settings architecture) comes from that project, and I still pull fixes from upstream. Go give it a star too. Then I kept adding the things I wanted on my own machine. That fork grew into Talvio. What it does Dictation. Hold Ctrl + Win on Windows (Ctrl + Space on Linux), talk, let go. A small pill at the bottom of the screen shows it's listening, and with a streaming model you see your words as you speak. !The recording pill: listening, live text, transcribing Speech recognition runs locally through transcribe.cpp. Whisper, Parakeet, Nemotron and the rest of a 69-model catalog, on your GPU through Vulkan or on the CPU. No account needed. Cloud speech providers are optional and use your own key. An assistant on another shortcut. Select some text, hold Ctrl + Alt, say "write a polite reply saying I can't make Thursday". It answers out loud, and an Insert button puts the answer back where the text was. Voice commands. "Play Believer on YouTube." "Turn the volume down." "Email Ram that I'm late." These run in your own apps: Gmail opens with the email written, the song starts in your browser. !The Commands page Meeting notes. It records your mic and the system audio as two separate lanes, so it knows which words were yours. No bot joins the call. You get notes at the end. !A meeting being recorded with two lanes Smaller things. Optional AI cleanup that removes "um" and follows "at ten, actually no, eleven". A dictionary so names come out spelled right. And an Android app that turns your phone into a wireless mic, keyboard and remote for the PC. !The phone app The AI features (assistant, commands, cleanup) use your own API key, a local model through the built-in llama.cpp engine, or $1 of free credit if you sign in. Plain dictation needs none of that. OK, the lessons. 1. Creating a window from a hotkey thread froze the whole app The assistant panel opens when you press a global shortcut. The shortcut callback runs on the keyboard hook's thread, not on the main thread. My first version built the WebView window right there in the callback. It worked most of the time. Sometimes it deadlocked the event loop, and then everything was dead: no overlay, no dictation, windows you couldn't drag. The fix is boring. Every function that creates, shows or destroys that window queues the work on the main thread: pub fn enter_conversation_size(app: &AppHandle) { let app_main = app.clone(); if let Err(e) = app.run_on_main_thread(move || { apply_panel_geometry(&app_main); place_call_window(&app_main); }) { error!("Could not queue conversation panel sizing: {}", e); } } The functions that actually touch the window are private. The public ones only queue. That had a side benefit I didn't plan for: a create is always ordered behind a destroy, so toggling the feature off and on fast can't leave the app thinking a window exists when it doesn't. If you call anything window-related from a shortcut handler, a tray handler or a background thread in Tauri, go through run_on_main_thread. 2. Floating windows must never take focus A dictation app pastes with a synthetic Ctrl+V. That keystroke goes to whatever has keyboard focus at that moment. If your own overlay grabbed focus, the transcript gets pasted into your overlay and disappears. This happened. The overlay window was bigger than the visible pill (a transparent frame around it), so a click on "empty" space actually landed on my window, activated it, and the next paste went nowhere. Each platform needs a different answer: - Windows: the window gets WS_EX_NOACTIVATE, so even a click that lands on it can't make it the foreground window. - macOS: an NSPanel that never becomes key. - Linux (Wayland): a GTK layer-shell surface with KeyboardMode::None. - Plain X11: none of the above works reliably, so the card there stays click-through until the paste is done. The same rule now applies to every window that pops up on its own: the reminder popup, the "allow this phone?" card, the assistant. Anything that shows up while you are typing somewhere else should not move your cursor. One more trap: keeping it on top. WS_EX_TOPMOST is a position another app can take from you. I set it once on show and users reported the pill "randomly hiding". Now a small guard re-asserts it every 700 ms while the overlay is up, and logs once when something displaced it. That log line turned a vague bug report into a fixable one. 3. Run the speech engine in a separate process Local speech recognition on the GPU means running Vulkan compute shaders through whatever driver the user has. Some of those drivers crash. Some hit a GGML_ASSERT and abort. Some just hang. When the engine ran inside the app, any of those took the whole app down, or left it sitting on "Transcribing…" forever. Now every call into the speech engine goes to a worker process: the same executable started with --transcribe-worker. main.rs checks for that flag before anything else (before Tauri, logging or single-instance), so the worker is just a small loop reading requests from stdin. The protocol is plain request/response, one frame per message: [u32 json length][json][u32 pcm length][f32 pcm samples] Ten minutes of audio costs about 10 ms to send this way. There's a 1 GiB cap per section so a garbage length can't turn into a huge allocation. When a worker using the GPU crashes or hangs, the app marks the GPU unavailable for the session and retries once in a worker started with --cpu-only. That flag restricts the engine to CPU backends before any GPU code runs, so a driver that dies during init can't take the fallback down too. The user gets their text, a bit slower, and a log line saying why. Two details that matter more than they look: - On Windows the worker goes into a kill-on-close Job Object, and it exits when its stdin closes. A crashed app never leaves a worker eating your GPU memory. - Esc goes through the engine's abort callback first and only kills the worker after 750 ms. Killing it every time meant reloading the model after every cancelled dictation. 4. My settings file got wiped on shutdown Early on I used tauri-plugin-store for settings. Then a shutdown reset every setting. Shortcuts, models, everything back to defaults. Two things combined. The plugin wrote the file in place, so a shutdown at the wrong moment (or Windows Defender holding the file) could leave it empty or half-written. And on the next launch, a file that didn't parse was silently treated as "no settings", so defaults were written over it. The replacement is small: pub fn write_atomically(path: &Path, bytes: &[u8]) -> io::Result<()> { // write to a temp file next to the real one let mut file = retry_while_held(|| File::create(&temp))?; file.write_all(bytes)?; file.sync_all()?; drop(file); // then swap it in with one rename retry_while_held(|| fs::rename(&temp, path))?; Ok(()) } retry_while_held retries for a few hundred milliseconds, because on Windows an antivirus, the search indexer or a sync client often has the file open for a moment. The rest is about what happens when the file is damaged anyway. A broken file is moved aside, not overwritten. Settings come back from a .bak copy that is only refreshed from a launch that read a good file. A read error (as opposed to bad content) never triggers a reset at all. And the user gets a notice instead of silently losing their setup. If you store anything users care about, don't let "can't parse it" mean "start fresh". 5. You don't need paid code signing to ship auto-updates I assumed self-update was tied to Apple and Windows code signing certificates. It isn't. Tauri's updater checks a minisign signature made with a key you generate yourself, for free. Code signing is a separate thing that only affects the "unknown publisher" warning on first install. I had tied them together in CI. The result: from 1.0 to 1.4, no release had a working update path, and nobody noticed because the update UI was also hidden. Fun. What the pipeline does now: - Build with the updater key from repo secrets whenever it exists, signed or not. - Re-sign the Linux AppImage and .deb after they are repacked. The bundler's signatures describe the bytes before repacking. - Write latest.json once, at the end, from the signatures actually attached to the release, and verify each one against the published file before it goes live. The app polls releases/latest/download/latest.json. GitHub only serves that from the newest published, non-prerelease release, so drafts and betas never reach anyone. And back up that private key. If you lose it, installed copies can never update again. Try it - Download: GitHub releases (v2.2.0) or talvio.sentramind.dev - Windows installer isn't code signed yet, so SmartScreen will ask. More info, then Run anyway. - Linux: AppImage or .deb. It's a solo project and there are rough edges. If something breaks on your machine, an issue with your GPU and OS helps a lot. If it's useful to you, a star on GitHub helps more people find it. Repo: ishangautam7/Talvio I'd also like to know: which of these did you hit in your own Tauri or Electron app? The focus one took me embarrassingly long to understand.