mattdesl

I replaced FFmpeg.wasm with WebCodecs in my browser video tools. Compression went from 97 s to 5.6 s --- title: I replaced FFmpeg.wasm with WebCodecs in my browser video tools. Compression went from 97 s to 5.6 s published: true description: Moving 93 in-browser video and audio tools from FFmpeg.wasm to WebCodecs, and the five things that quietly made them slow. tags: webdev, javascript, performance, showdev cover_image: dev-to-uploads.s3.us-east-2.amazonaws.com --- I run DashUtils, a site with 392 free file tools: image and PDF converters, a video compressor, an MP3 cutter and so on. The rule for every tool is simple. Nothing gets uploaded; the work happens in your browser. Until last week, every video and audio tool ran on FFmpeg.wasm. It handles every format you can think of, but it has costs: - A 31 MB engine (about 10 MB compressed) has to download before the first job starts. - One CPU thread. The multi-threaded build needs SharedArrayBuffer, which means cross-origin isolation, which fights with third-party embeds. - No hardware acceleration. I tried my own MP4-to-GIF tool and closed the tab before it finished. That was the push I needed. The switch: WebCodecs, through Mediabunny WebCodecs gives JavaScript the browser's own video and audio encoders and decoders, using the GPU where the platform allows. On its own it only handles raw frames, so I used Mediabunny (MPL-2.0) for reading and writing MP4, MOV, WebM, MKV, MP3, WAV, OGG and FLAC. Its core is about 140 KB compressed, and it's only loaded when someone opens a file. The core of a conversion is short: import { ALL_FORMATS, BlobSource, BufferTarget, Conversion, Input, Mp4OutputFormat, Output, QUALITY_MEDIUM } from "mediabunny"; const input = new Input({ source: new BlobSource(file), formats: ALL_FORMATS }); const target = new BufferTarget(); const output = new Output({ format: new Mp4OutputFormat({ fastStart: "in-memory" }), target }); const conversion = await Conversion.init({ input, output, video: { codec: "avc", height: 720, quality: QUALITY_MEDIUM }, audio: { codec: "aac" }, }); // A track we can't encode here gets discarded, not converted. Treat that as "not supported". if (!conversion.isValid || conversion.discardedTracks.some((t) => t.reason !== "discarded_by_user")) { return runWithFFmpeg(file); } await conversion.execute(); const mp4 = new Blob([target.buffer!], { type: "video/mp4" }); FFmpeg.wasm didn't go away. It's the fallback for AVI, FLV and WMV, which browsers can't read, and for the speed and volume filters. Five things that quietly made it slow 1. A silent fallback The first benchmark looked great everywhere except compression, which still took 97 seconds. The reason: Chrome on Linux can't encode AAC audio. Mediabunny discarded the audio track, my check saw that, and the job fell back to FFmpeg without telling anyone. The fix was to check canEncodeAudio("aac") and, only when it's missing, register @mediabunny/aac-encoder, a WebAssembly encoder of about 260 KB compressed. After that, compression took 5.6 seconds. Lesson: log which engine actually ran, not which one you expected. 2. The progress bar was the bottleneck onProgress fires for every frame and every audio chunk. I passed it straight into React state, so converting 30 seconds of MP3 to WAV re-rendered the page about 1,150 times. The conversion itself took 0.43 seconds; the page took 5.1. Throttling progress to ten updates a second brought it to 0.62 seconds. 3. Browser encoders have a bitrate floor To fit a video under Discord's limit, you work backwards: bitrate = size ÷ duration. At 1080p, Chrome's software H.264 encoder simply ignored targets below about 2 Mbps. A "10 MB" export came out at 14.7 MB, even after re-encoding at lower bitrates. Now the compressor picks the resolution from the budget (roughly 0.04 bits per pixel per frame), checks the result, and re-encodes smaller only if it's still over. A 60-second 1080p clip lands at 9.3 MB, scaled to 720p, in 21 seconds. 4. Reading pixels back from the GPU GIFs need raw pixels for colour quantisation. getImageData() on the canvas the decoder drew into cost about 45 ms per frame, because it forces a GPU-to-CPU copy. createImageBitmap(frame, { resizeWidth, resizeHeight }) followed by a willReadFrequently canvas was about three times faster. Moving the quantisation (gifenc) into a pool of workers, one per core, took a 72-frame portrait GIF from 15.2 to 3.5 seconds. 5. Asking for a bitrate turned copies into encodes Trimming a clip should only copy data. But I always passed an audio bitrate, and that made Mediabunny re-encode the audio. Setting the bitrate only when the audio really is being transcoded took a trim from 5.5 seconds to 0.31. Results !Before and after: in-browser video jobs with FFmpeg.wasm vs WebCodecs All measured in the browser on a 2016 laptop (Intel i7-6820HQ, Linux, no hardware video encoder). "Before" is FFmpeg.wasm for compression and my first WebCodecs version for the others. | Job | Before | After | |---|---|---| | Compress a 20 s 1080p clip | 97.1 s | 5.6 s | | 72-frame portrait GIF | 15.2 s | 3.5 s | | Trim a 20 s clip | 5.5 s | 0.31 s | | 30 s MP3 to WAV | 5.1 s | 0.62 s | Opening a file went from "download 31 MB first" to about a tenth of a second. What I'd tell anyone trying this - Detect support per codec, not per API. VideoEncoder existing tells you nothing about AAC or HEVC. - Keep the old engine as a fallback, and make the switch visible in your logs. - Test on your slowest supported setup. Every problem above was invisible on a fast Mac. - Throttle anything that fires per frame before it reaches your UI. If you want to try it, the video compressor and MP4 to GIF are the best demos. I'd love to hear how they run on your machine.

I built a background remover that never sees your photo (and it does GIFs and videos too)! When you’re the only engineer in the family, you automatically become the unpaid IT department, video producer, and resident graphic designer, because to relatives, "computers" is just one big subject (and that includes everything to do with mobile phones too... somehow!). So whenever someone needed a transparent PNG or a quick cutout, it fell on me, and I’d get stuck in the same annoying loop. I'd hunt down a "free" background remover (whatever showed up first on Google), upload the photo, wait, and then get held hostage by a "Buy Credits" button. My only options were settling for a blurry, postage-stamp-sized preview, or making a whole new account if I was desperate enough for the high-res version. Worse, somewhere in that mess, a family photo would get shipped off to a sketchy server I know nothing about. For a product shot, that's annoying. For a passport photo, a signature, or a client's photo, it's a bit more than annoying. So I added a background remover to Fizzdoc, my open-source toolkit where every tool runs inside the browser tab. It now does: - Photos: Remove Background, saved at the photo's own resolution as PNG, JPG or WebP. - Animated GIFs: Remove GIF Background, every frame, same timing, same loop. - Videos: Remove Video Background, every frame, with the sound kept. - WhatsApp stickers: WhatsApp Sticker Maker, a 512 × 512 WebP with a white outline, kept under 100 KB, with words in fun fonts and emoji you drag into place. You can keep the background clear, or put white, any colour, a blur or another photo behind. It's free, there's no sign-up, no watermark, and no limit on how many photos you do. The size limit comes from your device's memory, not from me. And the photo never leaves your device. Not "we delete it after an hour". It just never goes anywhere. Below is how it works, and what broke along the way. The rule: nothing gets uploaded, and you can check !The Remove Background page. The badge under the title says what happens to your photo: 0 bytes uploaded. Fizzdoc is a static site on Cloudflare Pages. There's no backend, so there's nothing that could receive a file. The page also ships a Content Security Policy with connect-src 'self', which means the browser itself blocks the page from talking to any other website. That rule shaped everything below. No CDN for the model, no hosted inference API, no "just this one request". Open DevTools, go to the Network tab, remove the background, and you'll see the model files coming from fizzdoc.com and nothing going out. Picking a model I wanted three things: 1. Good edges on hair and fur, because that's where background removers usually show their seams. 2. Small enough to download once into a browser. 3. A licence that lets anyone use the result, including for work. I went with BiRefNet (MIT licence), the lite variant, using the 512 × 512 ONNX export by studioludens. I store its big weight tensors as 16-bit floats and cast them back to 32-bit when the model loads, which halves the download to about 100 MB without changing the maths it runs. For phones there's a second model, U²-Net small (Apache-2.0), just 4.6 MB. I learned this one the hard way: someone sent me a screenshot of an early version on a 6 GB Android phone, where Chrome killed the tab at 5% ("Aw, Snap!"). A browser tab on a phone gets far less memory than the phone has, and when it runs out there's no error to catch, the tab just dies. So now phones always use the small model, and the page says so ("for the cleanest edges on hair and fur, use a laptop"). If the big model ever crashes a tab on a laptop too, the page remembers it for 30 days and uses the small one there. And for laptops with a graphics chip there's the opposite: a Sharper edges (hair, fur) button. It reruns the photo through a 1024 × 1024 export of the same BiRefNet-lite model (onnx-community, MIT). It's a one-time 135 MB download and needs WebGPU, so it's never offered on phones. If the device can't manage it, you keep the result you already had. Running it: WebGPU when it can, CPU when it can't The model runs with ONNX Runtime Web inside a Web Worker, so the page never freezes. - If the device has a graphics chip with WebGPU, the model runs there. This is the fast path. - If WebGPU isn't there (or the driver says no), the same model runs on the CPU through WebAssembly. - If that fails too, usually because the device ran out of memory, the small U²-Net model takes over. So the chain is GPU, then the same model on CPU, then the small model. Nobody gets an error just because their laptop is old. Two fun problems with "self-host everything" Cloudflare Pages has a 25 MiB limit per file. The model is about 100 MB and ONNX Runtime's WebGPU build is a 27 MB .wasm. So both live on the server as 20 MiB parts: the models are split by a small script when I add them, and the runtime is split at build time. The worker downloads the parts, joins them, and keeps the result in the browser's Cache Storage, so the next time it starts straight from there, even with the internet off. The build also fails if any file it writes goes over 25 MiB, after one deploy failed on it. A runtime that exists in pieces. ONNX Runtime normally fetches its .wasm as one file. Mine exists just as parts on the server, so the worker joins them from Cache Storage and hands the bytes to the runtime directly with ort.env.wasm.wasmBinary. It needs no extra hosting and no change to the security policy. The part I'm proudest of: full resolution from a 512 px model !Top: the page loading the model once, and the options panel. Bottom: the original photo, the cut-out, and a new background in one tap. The model looks at your photo squeezed to 512 × 512 and returns a soft mask. If you stretch that mask back over a 4000 × 3000 photo, the edges look like they were cut with a butter knife. So the mask goes through two classic image-processing filters, both plain arithmetic on typed arrays in the worker: 1. Fast guided filter (He and Sun, 2015). It uses the photo itself as a guide to snap the blurry mask onto the real edges, so hair follows the actual strands. The fit is computed on a smaller copy (up to 1280 px) and then applied to every pixel of the full photo in one pass. 2. Blur-fusion foreground estimation (Forte, 2021). Edge pixels are a mix of your subject and the old background. Put a cut-out on a new colour without fixing that, and you get a faint halo of the old background. This step estimates the subject's true colour at the edges, so the new background looks like it was always there. By default the result is saved at your photo's own size: your own pixels, with a cleaner mask. When the AI gets it almost right It's an AI model, so a busy background or a subject that blends into it can confuse it. For photos there's a small set of touch-ups: - Erase / Restore brush, with Undo, Redo and Zoom for careful work on a phone. - Tap to remove and Tap to keep: one tap picks the whole connected area of a similar colour and removes it, or brings it back. - Shadow and a white Outline, if you want the cut-out to stand out. - Crop: the whole photo, fit to the subject, or a square around the subject for online shop listings. - Hold to see the original, to compare before and after. GIFs: every frame, same rhythm Animated GIFs are sneaky. Many frames aren't full pictures, just small patches drawn over the previous frame, with rules ("disposal methods") about what to clear afterwards. - gifuct-js (MIT) decodes the frames, and Fizzdoc composes each one into a full picture the way a browser would show it. - Each frame goes through the same model (in a faster mode, since there can be dozens). - gifenc (MIT) writes the new GIF, keeping every frame's delay and the loop. One catch: a GIF pixel is either fully see-through or fully solid. So on a clear background the edges are a little sharper than in a PNG. Pick a colour or a photo behind and the soft edges are back. Videos: every frame, sound kept !Remove Video Background: every frame gets a new background, and the sound is kept. For video, Mediabunny (MPL-2.0) reads the file and the browser's own WebCodecs encoder writes the new one. Every frame goes through the model and gets its new background, and the soundtrack is carried over. The output is MP4 (H.264) if the browser can handle it; otherwise, it's WebM (VP9). Before you start, the page shows a preview frame and an estimate of how long the whole clip will take on that device. What does "no background" mean for a video? Truly see-through video would need the encoder library to start helper workers from blob: URLs, which my page's security policy blocks, and most players can't show it anyway. So "None" gives you a green screen, which any video editor removes in one click. I chose to keep the policy strict rather than loosen it for one feature. Stickers: words, emoji, and the last step in WhatsApp !The WhatsApp Sticker Maker turns any photo into a 512 × 512 sticker with a white outline, under 100 KB. The WhatsApp Sticker Maker starts from the same cut-out. It squares it to 512 × 512, adds a white outline, and keeps the WebP under 100 KB by lowering the quality step by step until it fits. On top of that, you can add words in five styles (Funny, Cute, Cozy, BIG, and Serious) and emojis. Each one is a layer you drag, pinch, resize and rotate, on a phone or with a mouse. The fonts are self-hosted like everything else: Bangers, Kalam, Teko and Inter, with Baloo 2 covering Hindi letters. Emoji are drawn with your device's own emoji font. One fun bug on the way: Vite inlines small files as data: URLs, and a few of the font files were small enough. My own CSP (default-src 'self') refused them, and the end-to-end test that fails on any CSP violation caught it before it shipped. Fonts are now always separate files. One limit, though. A web page can't add a sticker to WhatsApp, and a sticker shared from the browser arrives as a normal photo. So instead of a "Share to WhatsApp" button that doesn't do what it says, the page shows WhatsApp's own steps: save the sticker, then in any chat tap Stickers › Create and pick it. On iPhone there's a "Save to Photos" button, because downloads go to Files and WhatsApp picks stickers from Photos. What it's like on low-end phones !The same tool on a phone, in dark mode. Phones use the small 4.6 MB model so the tab doesn't crash. I care about this a lot because many people open Fizzdoc on Android phones. - Phones always get the small model, so it works rather than crashes. Edges are a bit softer than on a laptop, which is a trade I'll take every time. - The worker keeps the page responsive while it thinks. - The model downloads once (the page shows the progress in MB). After that it starts right away, and it keeps working with Wi-Fi and mobile data off, which is also the easiest way to check that nothing is uploaded. - Videos give you a time estimate first: short clips are fine, long ones are a laptop job. Things it doesn't do (yet) - Truly transparent video (see above, green screen instead). - Many photos in one go: the background remover takes one photo at a time. - Several photos in one sticker. That's next. Credits None of this would exist without these people and projects. Thank you: | Project | What it does in Fizzdoc | Licence | |---|---|---| | BiRefNet by Peng Zheng and co-authors | The background model | MIT | | birefnet-lite-512 by studioludens | The 512 px ONNX export | MIT | | BiRefNet_lite-ONNX by onnx-community | The 1024 px export for Sharper edges | MIT | | U²-Net by Xuebin Qin and co-authors | The small model for phones | Apache-2.0 | | ONNX Runtime Web by Microsoft | Runs the models on WebGPU or WebAssembly | MIT | | gifuct-js by Matt Way | Reads animated GIFs | MIT | | gifenc by Matt DesLauriers | Writes animated GIFs | MIT | | Mediabunny by Vanilagy | Reads and writes videos | MPL-2.0 | | Kaiming He and Jian Sun | The (fast) guided filter | Paper | | Marco Forte | Blur-fusion foreground estimation | Paper | The full licence texts are in Fizzdoc's third-party licences. Try it, break it, tell me - Photo: fizzdoc.com - GIF: fizzdoc.com - Video: fizzdoc.com - WhatsApp sticker: fizzdoc.com - Code (Apache-2.0): kingrishabdugar/fizzdoc It's part of a bigger toolkit of 68 free tools (PDF, Word, Excel, PowerPoint, images and audio), all running in your browser, in 16 languages. If something looks off on your device, there's a "Report it" link on every tool. And if you like it, a star on GitHub genuinely makes my day.