livekit

An 8-vCPU, 32 GB server choked on a 3-person 1080p LiveKit call — the bottleneck was outbound bandwidth I recently added self-hosted video meetings to an internal product using LiveKit on Docker. The VM looked generous on paper: 8 vCPUs, 32 GB RAM, and a fixed 10 Mbps public bandwidth plan. In practice, 1080p worked fine with two people in a room. The moment a third person joined with their camera on, video froze. Here is what happened, why I picked LiveKit **over **Jitsi, and the bandwidth math I should have done before buying a fixed-bandwidth VM. Why LiveKit instead of Jitsi Before deploying, I compared Jitsi + Jibri against LiveKit Server + Egress, specifically looking at protocol design and recording: - Jitsi + Jibri: Jibri records a room by spinning up a headless Chrome instance to render the meeting page and capturing it with ffmpeg. The output looks just like a real meeting, but one Jibri instance only records one meeting at a time—scaling concurrent recordings means running one Jibri per active recorded room on top of the XMPP stack. - LiveKit: Uses WebSocket signaling + WebRTC SFU, separates media routing (livekit-server) from recording/streaming (livekit/egress), and has an official agents framework if you later add voice AI. Note that LiveKit's full-room composite Egress also runs headless Chrome inside its container (requiring SYS_ADMIN capability and roughly 2–6 vCPUs per active recording), and one Egress instance still records one room at a time—so you still size Egress per concurrent recorded room, without carrying the rest of Jitsi's stack. Deployment was smooth (if you stick to official templates) Getting LiveKit Server and Egress running via Docker was straightforward. One takeaway if you use coding agents to help wire up configs: anchor strictly on LiveKit's official config generator (livekit/generate) and sample templates. Letting an LLM improvise WebRTC/TURN/Caddy configs from scratch is a fast way to break UDP media routing (50000-60000/udp, 3478/udp, 7881/tcp) or TLS certificates. Where it actually broke: SFU outbound math LiveKit is a Selective Forwarding Unit (SFU). Every participant uploads their camera stream once to the server, and the server forwards separate copies down to everyone else in the room. If everyone has a 1080p camera on at ~4 Mbps (ignoring simulcast downscaling for a worst-case ceiling), the server's outbound (egress) bandwidth grows as N × (N - 1) × 4 Mbps: | Participants (all cameras on) | Server outbound bandwidth | | --- | --- | | 1 publishing, 1 watching | ~4 Mbps | | 2 people (1-on-1) | ~8 Mbps | | 3 people | ~24 Mbps | | 4 people | ~48 Mbps | | 10 people | ~360 Mbps | On a cloud VM with a 10 Mbps outbound cap: - A 1-on-1 call uses ~8 Mbps—just under the ceiling. - A 3rd person pushes outbound demand to ~24 Mbps—instant packet loss and frozen frames, while the 8 vCPUs and 32 GB RAM sit mostly idle. Before going to production, LiveKit's CLI includes lk load-test to simulate publishers and subscribers against your actual box instead of guessing. Takeaway: bill video servers by metered traffic, not fixed bandwidth For self-hosted video meetings, outbound network cap matters long before CPU or RAM. Traffic for meetings is spiky: heavy while rooms are active, near zero the rest of the day. Paying for a high fixed-Mbps pipe 24/7 is expensive, while a cheap 10 Mbps fixed pipe chokes on the third caller. Using pay-per-GB (metered egress traffic) with a high burst bandwidth ceiling fits SFU workloads much better than a flat low-bandwidth cap. --- I maintain a plain-English checklist of self-hosting requirements, license terms, and real server/cloud costs for LiveKit and 18 other open-source business apps at BuilderStack.

How we deliver voice only to people nearby in a 2D virtual office (SFU subscription permissions, not mixing) In our 2D virtual office, voice reaches only nearby people because the media server refuses to forward it to anyone else. The app server works out who is near whom and gives each speaker a list of allowed listeners. Each speaker's client passes that list to LiveKit, our SFU, as track subscription permissions, so nothing is mixed on the server and a modified listening browser cannot pull a voice it was not given. An SFU (selective forwarding unit) forwards each person's tracks instead of combining them. The stack here is a Python (FastAPI) WebSocket server, a React and Phaser client, and self-hosted LiveKit, which is Apache-2.0 open source. Why not mix audio on the server? A mixer gives you privacy for free. The server decodes every microphone and encodes one mix per listener, so each browser only ever receives its own mix. There is no other voice to steal. The price is one mix per listener. On a map, people rarely share the same set of voices at the same volumes, so nearly every listener needs separate decode, mix, and encode work. An SFU does none of that. It forwards encoded packets, each voice stays a separate track, and the browser sets the volume per track. Each office is one LiveKit room, so walking around only changes subscriptions. The trade-off is that the free privacy boundary is gone. Every browser in the room is one request away from every track. You have to build the boundary yourself. How does the server decide who is near? The client sends only movement. The server owns every position and rejects a move of more than 2 tiles or into a wall. Positions go out on a 100 ms tick, and after each tick with movement the server recomputes near: the set of people whose voice should reach a given listener. Distance is Chebyshev: max(|dx|, |dy|). | Where you are | Your near | Volume | |---|---|---| | Open floor | The 8 closest people within 3 tiles, excluding people in meeting rooms | 1.0 at 0 to 1 tiles, 0.7 at 2, 0.45 at 3 | | Meeting room | Everyone in the same room, no cap | Always 1.0 | | Focus desk | Nobody, and you are in nobody's near | No voice | The server sends near only when it changes. Note the cap: it makes near directional. In a crowd, you can be in my 8 closest while I am not in yours. Version 1: the listener chose The client joined with autoSubscribe: false and subscribed to the people in near (condensed): // Version 1: the listener decides alone for (const pub of participant.trackPublications.values()) { pub.setSubscribed(nearIds.has(memberIdOf(participant))) } It looked right on screen, but it trusted the listener's browser. By default, any published track can be subscribed to by all participants. Calling setSubscribed(true) from the developer tools would let you hear someone at the far end of the map. The access token cannot help: its grants cover the whole room, and proximity changes as people walk. Version 2: the speaker's permission, enforced by the SFU LiveKit lets a publisher restrict who may subscribe to its tracks: room.localParticipant.setTrackSubscriptionPermissions( false, // do not allow everyone [{ participantIdentity: 'u42', allowAll: true }], // allow only these ) Anyone left off the list gets nothing from the SFU, whatever their browser does. The call comes from the speaker's client, but the decision should not. Positions live on the server, so the server computes each list and pushes it down. Server: invert near to get allow allow is the set of people who may receive my voice and video. It is not my near. Because near is directional, copying it would admit people who should not hear me and block people who should. The correct set is everyone whose near contains me, so the server inverts all the near sets: def compute_allow(members, gmap): allow = {m.id: set() for m in members} for listener in members: for speaker in listener.near: if speaker in allow: allow[speaker].add(listener.id) # In a meeting room, add everyone in the same room # so that allow cannot disagree with near meetings = {} for m in members: zone = gmap.zone_by_id(m.zone) if m.zone else None if zone and zone.kind == "meeting": meetings.setdefault(zone.id, []).append(m.id) for ids in meetings.values(): for mid in ids: allow[mid].update(i for i in ids if i != mid) return {mid: frozenset(ids) for mid, ids in allow.items()} The server recomputes allow with near. It sends {"t": "allow", "ids": [...]} only to members whose list changed, plus once right after they join. Client: pass the list to the SFU private applyAllow(ids: string[] | null): void { // null (not received yet, or WebSocket down) means: allow nobody this.room.localParticipant.setTrackSubscriptionPermissions( false, (ids ?? []).map((id) => ({ participantIdentity: `u${id}`, allowAll: true })), ) } allowAll: true covers every track a speaker publishes: microphone, camera, and screen share. This matters because LiveKit's reference notes that track-level grants do not extend to tracks published later. The listening side did not change: near still drives setSubscribed. Subscription says what a listener wants. Permission says what a listener may have. Six pitfalls 1. The default is open Right after joining, the first allow has not arrived, and LiveKit's default lets all participants subscribe. A microphone published in that gap is open to the whole room. So the client sets "allow nobody" before it joins. The SDK sends that on connect, and we send the current list again before publishing the microphone. 2. A dropped WebSocket must revoke everything While the WebSocket is down, the client cannot hear about revocations, so a stale permission could outlive its reason. On disconnect the client discards allow and allows nobody. The next welcome does not restore it; only a fresh allow does. A network blip can cut your voice for a moment, and we accept that. 3. Send allow before near The server sends allow first and near second. A revocation reaches the SFU sooner, and someone walking up is more likely to be permitted before their browser asks to subscribe. 4. An old allow must not land last Sending is async, so another recompute can run during an await. If each pass sent its own snapshot, a stale list could arrive last and restore a revoked permission. So the server computes and assigns with no await in between, then reads the current value at send time: for m in allow_changed: # m.allow is read here, at send time await self.send(m, {"t": "allow", "ids": _id_order(m.allow)}) Whichever pass sends last sends the newest value. 5. Permission can arrive late A listener can ask for a track before the speaker's permission lands. LiveKit connects the subscription on its own once permission arrives. As a safety net, the client waits 600 ms after a grant or a failed subscription, then asks again if it still wants the track, up to 5 times per track (simplified): const stillWanted = this.desired.get(sid) === true && // still in near p.trackPublications.get(sid) === pub && !pub.isSubscribed && pub.permissionStatus === TrackPublication.PermissionStatus.Allowed if (stillWanted) pub.setSubscribed(true) 6. Meeting rooms need one rule for both lists Meeting rooms first inherited the cap of 8, so from the ninth person on, voices could not be heard. We removed the cap there. near and allow now both mean everyone in the room, and nobody outside it. How do we test it? Most of this needs no media server. The server tests use a fake WebSocket that records messages in order: - On the open floor, allow is the inverse of near. - With 8 people packed around a listener, a speaker 3 tiles away falls outside the cap, so that speaker's allow leaves the listener out, but not the reverse. - In a meeting room of 12, everyone reaches everyone and someone just outside the door reaches nobody inside. Whoever walks out drops from the other 11 lists. - A focus desk gives empty lists in both directions. - allow arrives before near, and a volume-only change resends near but not allow. To test pitfall 4, we stall one socket so that a second recompute overtakes the first (condensed): gate = asyncio.Event() class SlowWS(FakeWS): async def send_text(self, text): await super().send_text(text) if len(self.sent) == 1: await gate.wait() # stall after the first message a.ws = SlowWS() first = asyncio.create_task(room.refresh_near()) for _ in range(5): await asyncio.sleep(0) # let it reach the stall b.x = 30 # b walks away await room.refresh_near() # a newer pass overtakes gate.set() await first assert a.ws.of("allow")[-1]["ids"] == [] The client tests stay off the network too. They fake LiveKit's connection and log, in order, what would go to the SFU. They check that "allow nobody" comes before the microphone, that a WebSocket drop revokes everything until a new allow arrives, that the last list is re-sent on every rejoin, and that retries stop at 5 and skip anyone outside near. Finally, with real audio on a local setup, a far-away browser forced a subscription and received 0 bytes. What does this not protect? - A speaker's own voice. The speaker's client applies the permissions, so a modified speaker browser can open its own voice to anyone. It still cannot reach anyone else's. - Walking up to listen. Anyone nearby can hear. The server rejects jumps, so getting close means walking there, visible on the map. - Metadata. Everyone in the room still sees who is present and whose microphone is muted. - The SFU itself. This design controls who receives a track. It does not hide audio from whoever runs the server. What to take from this If you build proximity voice on an SFU, treat subscription as a preference and permission as the boundary. Compute allow on the server by inverting near, not by copying it. And fail closed whenever the client cannot be sure: before the first allow, and while the control channel is down. Sources - Camera & microphone, "Track permissions" section (LiveKit) — docs.livekit.io (retrieved October 31, 2026) - Subscribing to tracks (LiveKit) — docs.livekit.io (retrieved October 31, 2026) - LocalParticipant class reference (LiveKit) — docs.livekit.io (retrieved October 31, 2026) - livekit/livekit repository, Apache License 2.0 (LiveKit) — livekit/livekit (retrieved October 31, 2026) --- We use this in Hotori (hotoriplace.com), a voice-first virtual office that runs in the browser, made in Tokyo by a small team.

I value accurate duration reporting over variable bitrate compression. livekit/agents on GitHub added bitrate and encoding mode controls to RecorderIO. Set constant_bit_rate to True if QuickLook gives misleading duration estimates. Source: github.com/livekit/agents/releases/tag/livekit-ag…

Build an AI Receptionist With Whisper, Claude, Chatterbox and LiveKit: Architecture, Latency Budget and Starter Code This post is AI-assisted. Most "AI receptionist" tutorials stop at a chatbot that talks. A receptionist a dental clinic will actually pay for has to pick up a real phone line, understand messy speech, follow the clinic's rules, book into a real calendar and do all of it fast enough that the caller never notices a gap. This is the architecture we use as a starting point, built from four pieces you can run yourself: | Job | Tool | License | |---|---|---| | Phone line, streaming, turn taking | LiveKit Agents | Apache 2.0 (turn detector model under the LiveKit Model License) | | Speech to text | Whisper | MIT | | Reasoning and tool calls | Claude (Anthropic API) | Paid API | | Text to speech, voice cloning | Chatterbox by Resemble AI | MIT | Why bother Patient Prism's analysis of 11.5 million calls to 8,280 dental locations found 31% of callers hung up before reaching anyone (report). Only 42% of consumers say they leave a voicemail (CallRail survey). Missed calls are lost bookings, which makes "answer every call" an easy thing to sell. The business math people quote is 100 clinics × $300 a month = $30,000 MRR. The engineering below is what makes that number survivable. Architecture PSTN caller │ SIP trunk ▼ LiveKit server (room per call) │ audio frames ▼ Agent worker (Python, LiveKit Agents) ├─ VAD + turn detector → decides "caller finished" ├─ STT adapter → faster-whisper on GPU (streaming chunks) ├─ LLM → Claude with tools: check_slots, book, take_message, transfer └─ TTS adapter → Chatterbox (clinic's consented voice), streamed sentence by sentence │ ▼ Clinic systems: calendar / practice software / CRM / SMS Key design choices: 1. One room per call. LiveKit treats a phone call like any other room participant, so the agent joins the room the SIP call lands in. Scaling is "more worker processes", not a rewrite. 2. Stream everything. Do not wait for the full transcript, the full LLM answer or the full audio file. Start speaking on the first complete sentence. 3. Tools, not prompts, for anything that changes state. The model never "says" it booked an appointment. It calls book_appointment, your code books it, and the result goes back to the model. Latency budget Callers start feeling awkward somewhere past a second of silence. Rough budget from "caller stops talking" to "agent audio starts": | Stage | Target | |---|---| | End of turn detection | 200 to 400 ms | | Final transcript chunk | 100 to 300 ms | | Claude first tokens (short system prompt, cached rules) | 300 to 700 ms | | First sentence of TTS audio | 150 to 400 ms | These are targets to measure against, not guarantees. Measure each stage per call and log it. The biggest wins usually come from shorter prompts, streaming TTS by sentence and keeping the GPU workers warm. Starter code shape LiveKit's starter template (lk agent init my-agent --template agent-starter-python) gives you a working agent with hosted models. The API evolves quickly, so treat the snippet below as the shape of the code and check the current voice AI quickstart for exact signatures. from livekit import agents from livekit.agents import Agent, AgentServer, AgentSession, function_tool from adapters import WhisperSTT, ClaudeLLM, ChatterboxTTS # your thin adapters CLINIC_RULES = """You are the receptionist for Bright Smile Dental. Hours: Mon to Sat, 9am to 6pm. Book cleanings, checkups and consultations only. Never give medical advice. Transfer emergencies, billing disputes and upset callers to a human. Keep answers under two sentences.""" class Receptionist(Agent): def __init__(self): super().__init__(instructions=CLINIC_RULES) @function_tool async def check_slots(self, date: str) -> list[str]: """Return free appointment slots for a date (YYYY-MM-DD).""" return await calendar.free_slots(date) @function_tool async def book_appointment(self, name: str, phone: str, slot: str, reason: str) -> str: """Book a slot and send an SMS confirmation.""" booking = await calendar.book(name, phone, slot, reason) await sms.send(phone, f"Confirmed: {slot} at Bright Smile Dental") return booking.id server = AgentServer() @server.rtc_session(agent_name="receptionist") async def entrypoint(ctx: agents.JobContext): session = AgentSession( stt=WhisperSTT(model="large-v3", device="cuda"), llm=ClaudeLLM(), tts=ChatterboxTTS(voice_ref="voices/clinic_consented.wav"), ) await session.start(room=ctx.room, agent=Receptionist()) await session.generate_reply(instructions="Greet the caller and ask how you can help.") if __name__ == "__main__": agents.cli.run_app(server) The adapters are small classes that implement LiveKit's STT, LLM and TTS interfaces around faster-whisper, the Anthropic SDK and the Chatterbox Python package. Writing them yourself keeps every model swappable. Production checklist - SIP and numbers: connect a SIP trunk to LiveKit and port or forward the clinic's existing number, so patients dial what they already know. - Barge-in: let callers interrupt. Stop TTS immediately when the caller speaks. - Transfer path: a warm transfer to the front desk is a feature, not a failure. - Voice consent: only clone voices you have written permission for. Chatterbox watermarks every output, which helps with provenance. - Observability: store transcript, tool calls and per-stage latency for every call. Your support tickets will thank you. - Data handling: health and personal details need careful storage and retention rules in many countries. Decide this before onboarding the first clinic. Where the money actually is The models are commodities now. The value is in the boring parts: integrations with each niche's software, reliable transfers, and reporting that shows the owner how many calls were saved. Pick one niche, nail those three, and the $30K MRR math stops being a slide. Axionry is an AI product development studio that builds production voice agents. See the AI receptionist build, our voice AI development service, or price your setup with the voice AI cost calculator at axionry.com.

Provides built-in job scheduling, dispatch APIs, and telephony integration. Tired of wiring up speech-to-text and LLMs from scratch? Agents builds realtime voice AI participants that run on your server. livekit/agents

Hi HN, I’m Kajo, co-founder of KugelAudio (YC P26). We build voice models focusing on customer support. One of the main issues our customers are having right now is calls that drift into endless spelling loops and lots of false interruptions, especially in non-English languages. A classic example would be the name “Philipp” or “Philip”. The customer says their name on the phone and the bot either doesn't know about the ambiguity or lets users spell everything letter by letter, which is extremely annoying and prone to error or interruption mid-spelling. This is why we built Kugel Ear. When it isn't sure, it returns other spellings with probabilities, so the bot can ask one targeted question: “Philipp with two P’s?” Kugel Ear also comes with integrated context-aware EOT (end-of-turn) detection so the voicebot knows the user is mid-spelling and doesn't interrupt. We score 1st on LiveKit EOT bench, 1st on AppTek’s call-center dialogues, 1st on calls over phone lines, and 2nd on VoiceCodeBench behind OpenAI, with a median latency of 72 ms from end of speech to final transcript. We don’t win everything: on Earnings22 (long earnings calls) ElevenLabs Scribe beats us (4.0% vs 4.4% WER), and on VoiceCodeBench OpenAI writes code and numbers slightly more accurately (83.1% vs 81.9%). False cut-off rate (pauses taken as end of turn at 300 ms average wait, lower is better, next best = best published model): German: 5.9% (next best: 11.8%) Spanish: 3.8% (next best: 15.5%) French: 5.0% (next best: 12.7%) Italian: 5.8% (next best: 16.9%) Dutch: 4.7% (next best: 12.5%) Portuguese: 5.6% (next best: 19.3%) English: 7.4% (next best: 9.9%) It costs €0.48 per audio hour (about $0.55). You can try it out at kugelaudio.com I’d love to hear where it breaks for you, especially calls where it still cuts someone off or misses a spelling. I’ll be in the thread all day answering questions. Datasets: - Support calls: 280 turns from AppTek’s call-center dialogues, 20 per accent across 14 accents, median WER. huggingface.co - Phone line: the same 280 turns through a simulated G.711 line (8 kHz, mu-law, noise, dropouts). - German names/addresses: 800 synthetic German calls we generated, half through the same phone line. - Codes and numbers: VoiceCodeBench, scored with deterministic exact match instead of its LLM verifier. huggingface.co - Earnings22: Artificial Analysis’ cleaned version. huggingface.co - End of turn: LiveKit’s eot-bench. livekit/eot-bench and huggingface.co

I think duplex auth belongs in the inference layer, not separate vendor keys. livekit/agents on GitHub enabled GPT-Live via LiveKit credentials instead of an OpenAI key. Audit xAI plugin lists to replace Think Fast 2.0 with Grok Voice 3.0. Source: github.com/livekit/agents/releases/tag/livekit-ag…

Built-in turn metrics make voice agent delay directly observable. livekit/agents on GitHub added an e2e_latency metric when you configure local VAD. Read e2e_latency on conversation_item_added to check assistant playback timing. Source: github.com/livekit/agents/releases/tag/livekit-ag…

LiveKit: Realtime infrastructure for voice, video, and AI agents (21.4k stars) What it does All text below is quoted directly from the project README. >LiveKit is an open source platform for building voice, video, and physical AI agents. This repository is the LiveKit server: a scalable, distributed WebRTC SFU that moves realtime audio, video, and data between people, devices, and AI models. The SDKs, agents frameworks, and companion services are linked in the table at the bottom of this page. >LiveKit's server is written in Go, using the awesome Pion WebRTC implementation. Used in production by >LiveKit carries billions of calls a year for companies including Salesforce, Nvidia, Oracle, SAP, Deutsche Telekom, Spotify, Tinder, Coursera, Headspace, Skydio, Retell, Decagon, Cresta, and HeyGen. Features - Scalable, distributed WebRTC SFU (Selective Forwarding Unit) - People, devices, and AI agents join the same room as participants, with agent dispatch to route agents in automatically or on demand - Modern, full-featured SDKs for web, mobile, desktop, embedded, and server - Built for production, supports JWT authentication - Robust networking and connectivity, UDP/TCP/TURN - Easy to deploy: single binary, Docker or Kubernetes - Advanced features including: - speaker detection - simulcast - selective subscription - moderation APIs - end-to-end encryption - SVC codecs (VP9, AV1) - data tracks for low-latency telemetry and teleoperation - telephony over SIP - webhooks - distributed and multi-region Getting Started >Start LiveKit in development mode by running livekit-server --dev. It'll use a placeholder API key/secret pair. >A user connecting to a LiveKit room requires an access token. Access tokens (JWT) encode the user's identity and the room permissions they've been granted. Repository livekit/livekit Repository stats |Metric|Value| |:-|:-| |Stars|21,439| |Forks|2,448| |Last release|v1.13.10 (2026-10-14)| |Languages|Go 99.7%| |Contributors|136| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|claude (80 commits), Copilot (2), cursoragent (1)| |AI co-authored commits|Claude: 80, Copilot: 5, Cursor: 1| Main contributor |Field|Value| |:-|:-| |Name|Raja Subramanian| |GitHub|@boks1971| |Commits to this repo|1,780 (top contributor)| |Followers|63| |Public repos|8| |On GitHub since|2013-12-28|

Connecting STT, LLM, and TTS models over WebRTC, LiveKit Agents lets you build server-side voice AI participants. It includes built-in job scheduling and dispatch APIs to connect users to voice agents. livekit/agents