tokio-rs

Phone to PC without the cloud: a local HTTPS server, a pinned certificate and mDNS You take a photo on your phone and need it on your laptop, which is right there on the desk. On a Mac with an iPhone, AirDrop does it in two taps. On Windows, or with an Android phone, the photo goes up to someone's cloud, through a chat app or into an email to yourself, and back down again: two metres, by way of a data centre. The phone and the computer are on the same Wi-Fi. They should be able to talk to each other directly, with no account and no server in between. That's the idea behind Dropple. It runs on Mac and Windows, and works with any phone. This post is about how it works, and about the parts that were harder than I expected. The shape of it The desktop app is built with Tauri 2, with a Rust backend. Inside it runs an axum server on the local network: plain HTTP on one port for first contact, and HTTPS (rustls) on another. On the phone there are two ways in: - Any phone's browser. You scan a QR code and the computer serves the whole phone side as a web page. Nothing gets installed, and it works on an iPhone too. - A native Android app, which talks to the same API and stays connected in the background. Files stream both ways. Text sent from the phone goes straight onto the computer's clipboard. Live updates go out to every connected phone as Server-Sent Events, which turned out much simpler than WebSockets for a list that only changes on one side at a time. phone ──HTTPS──▶ axum (Rust, inside the Tauri app) ▲ │ └──── SSE /api/events ┘ "the list changed, fetch it" Trust without a certificate authority HTTPS on a home network is awkward. There's no domain name, so no public certificate. Dropple makes its own certificate authority on first run (with rcgen) and uses it to issue a certificate for the computer's current IP address. When the address changes, it issues a new one. The authority is name-constrained to private addresses and .local, so even if it were trusted somewhere, it couldn't vouch for a real website. A browser would need that authority installed before it trusts the connection, which is a real hurdle on a phone. The Android app gets around it by pinning instead: 1. The QR code carries the SHA-256 fingerprint of the computer's authority. 2. The app's trust manager accepts a connection only if the server's chain contains a certificate with exactly that fingerprint, and the server's own certificate was signed by it. Nothing goes into the phone's trust store, there's nothing for the user to install, and a lookalike server on the same Wi-Fi gets nowhere. I've written up [the trust manager itself](#) separately. Finding the computer again Laptops change IP addresses. They join a different network, the router hands out a new lease, or the laptop wakes up from sleep. The computer announces itself over mDNS as _dropple._tcp, with the start of its fingerprint in a TXT record. When the saved address stops answering, the Android app searches with Android's NSD and reconnects to the computer whose fingerprint matches. Connecting still checks the full fingerprint, so the TXT record only narrows the search; it never grants trust. One Samsung phone turned this into a bug hunt. Its NSD results contained only an IPv6 link-local address (fe80::…%wlan0), which you can't put into an https:// URL. The app sat on "Connecting…" forever. Details in [the NSD post](#). Staying connected on Android, politely Receiving files in the background means a foreground service of type connectedDevice, with an ongoing notification. Two things made that bearable: - Explaining before asking. Before Android shows its notification prompt, a short sheet says what the notifications are for. More people say yes, and nobody is surprised. - An off switch for every annoying part. Users can hide the "Connected" notification's channel, or turn the background connection off entirely. Android also won't let an app in the background read the clipboard, so "send what I just copied" needs a trick. A Quick Settings tile opens an invisible activity, which waits until it has window focus, reads the clipboard, sends it and closes. Write-up [here](#). What I'd tell someone building something similar - Pin, don't install. If you control both ends, pin the certificate rather than asking users to trust a new authority. - Test discovery on more than one brand of phone. NSD results differ between them. - Make reconnecting visible. A spinner with no words looks like a hang; "Looking for your computer…" doesn't. Users forgive waiting when they can see what's being waited for. - Keep the last state. If the computer is out of reach, show the last list you had rather than an empty screen. Sharing in Dropple is free, on Mac, Windows, any phone's browser, and Android through Google Play: scannertechs.com Happy to answer questions about any of this in the comments.

One year of Dioxus fullstack in production: the good, the painful After a year of Dioxus fullstack in production, the Rust code is the most boring part of my business. CI is the most exciting part, and not in a good way: a release build takes 90 minutes on a GitHub-hosted runner. That year started at my previous job and continued on my own products. Back in April I wrote about why I picked Rust for a production micro SaaS. The short version: slower at the start, cheaper to live with later. This post is the bill. What I'm running SeggWat is an in-app feedback tool for SaaS products. Its dashboard is a Dioxus 0.7 fullstack app inside a Cargo workspace: a backend crate with server functions, an auth crate, and about ten UI crates covering insights, feedback, ideas, boards, widgets and settings. Twelve crates depend on dioxus. Everything sits on PostgreSQL. infra.page is built on the same stack. "Fullstack" means one Rust codebase doing three jobs. It renders on the server, runs server functions as the backend, and compiles the client to WebAssembly so the page becomes interactive. The good One language, all the way down Database query, business rule, server function, button: all Rust. One toolchain, one formatter, one set of habits, one place to look when something breaks. That sounds minor until you're a solo founder juggling several products. I don't have a "frontend brain" and a "backend brain" that each need warming up. I open the workspace and keep going. The same types on both sides This is the one I'd miss most. A struct gets defined once. The database layer returns it, the server function sends it, and the component renders it. When I rename a field, cargo check gives me a list of every place that broke, including the UI. There's no OpenAPI spec to regenerate and no TypeScript interface drifting out of sync with the backend. The win isn't writing code faster. It's refactoring without fear, late at night, with nobody to review it. The door to other platforms stays open Dioxus can target desktop and mobile, including iOS, from the same component model. None of my products needs a native app today. But knowing I could ship one without a rewrite mattered when I decided where to put a year of work. Production is boring That's the whole point. Once it compiles, it mostly just runs. A whole class of "undefined is not a function" bugs never reaches users, so the bugs that do get through are logic bugs. That's where I'd rather spend my attention anyway. The "boring in production" argument from April survived contact with real users. The painful Compile times This is the part Rust evangelists tend to skip. The dashboard compiles twice: once for the server and once to wasm32 for the browser. A dozen Dioxus crates plus a macro-heavy UI add up. Locally it's tolerable on a decent machine. CI is a different story, and the comments in my workflow files read like a diary of it: - The WASM build alone takes about 22 minutes on 6 cores. That's a single leaf crate: the dashboard compiled for the browser. - A release image takes about 90 minutes on a GitHub-hosted 2-core runner. On my own VPS runner it's around 40, and that's the good version. - Clippy got OOM-killed. cargo clippy --workspace --all-targets on the 7 GB hosted runner died with exit code 143 and no diagnostics at all. The fix was a 10 GB swapfile plus CARGO_BUILD_JOBS=1 to cut peak memory. My linter needs swap. At some point I stopped building images on every push to main. They now build on release tags or when I trigger them by hand, because 90 billed minutes per push is a silly way to spend money. What I tried - Caching with Swatinem/rust-cache. It helps, but there's a trap. A workflow run can only restore caches from its own ref or the default branch, and GitHub evicts caches unused for 7 days. Once I stopped building on main, release builds kept starting cold. My workaround is almost funny: after a dependency bump, I manually trigger a build on main just to keep the cache warm. - Building and testing in one go, so the tests reuse compiled artifacts instead of compiling the world a second time. - The mold linker. Nice, but linking was never the bottleneck. Compiling was. - A self-hosted runner for the heavy builds. This was by far the biggest win, 90 minutes down to 40. It also means I now maintain a build server. Every one of these helped. None of them changed the underlying math: two compile targets, a big dependency tree and lots of macros means a lot of compiling. "All Rust" comes with an asterisk The business logic is all Rust. The edges aren't. SeggWat's insights page shows charts, and I couldn't find a Rust charting library I'd trust in production. So the charts are Chart.js, loaded from a CDN and wrapped in about 340 lines of JavaScript that Dioxus calls through dioxus-use-js. Styling is Tailwind, so CI also installs Bun and runs the Tailwind CLI before Rust even starts. I'm fine with that. It's pragmatic. But if you're choosing Dioxus specifically to escape JavaScript, know what you're signing up for: Rust, plus some JS glue, plus a JS toolchain in your pipeline. Would I do it again? People ask, "What helps me ship fastest?" The better question is, "What helps me keep shipping without hating my life?" For the backend, the answer is still Rust, and it isn't close. For the frontend, I've changed my mind a bit. If I were building something truly app-like, with heavy client state, offline support or a real mobile app, I'd pick Dioxus again. But most of what I build is dashboards: forms, tables, a few charts and some toggles. For that, I'm paying a 22-minute WASM tax on every release for interactivity I barely use. What's next: Topcoat So my next project is going to use Topcoat, a new fullstack framework from the tokio-rs org, announced in July. It makes the opposite bet from Dioxus. Everything renders on the server, and components are async functions that can query the database directly. For interactivity, you write $(...) expressions in ordinary, type-checked Rust. Topcoat evaluates them on the server for the first render and also translates them to JavaScript, so they re-run instantly in the browser. There's no WASM bundle and no second compile target. #[component] async fn faq(cx: &Cx) -> Result<impl View> { let open = signal(cx, || false); Ok(view! { <button @click=$(|_e| open.set(!open.get()))>"What is Topcoat?"</button> <p :hidden=$(!open.get())>"A full-stack Rust framework."</p> }) } Example from the Topcoat README. On paper, Topcoat keeps what I love about Dioxus (one language, shared types, no API layer) and removes the part that hurts most, the 22-minute WASM build. To be clear, this is a bet, not a result. Topcoat's README calls it early-stage and experimental, with breaking changes expected. I'm giving up the multi-platform option. And I haven't measured a single build yet. I'll write the follow-up with real numbers: compile times, CI minutes, and whatever I end up missing from Dioxus. If my release build drops from 40 minutes to 10, you'll hear about it. If it doesn't, you'll hear about that too.

Topcoat 0.11 is out! The Rust web framework now ships with localizations (Fluent translations, number and date formatting), a topcoat new project generator, popovers, and more. Try it now! github.com/tokio-rs/topcoat/releases/tag/v0.11.0 t.co

Termtastic — new TUI client for Linux, macOS and Windows preview.redd.it Hello, guys! I wanna introduce you Termtastic — my handmade feature-rich Meshtastic® console (TUI) client written in Rust. preview.redd.it Channels screen Chat Nodes tab Traceroute results Multiline message support Emojis Logs tab Settings tab Repo link: acelot/termtastic I wanted to create a cross-platform, minimalist, and aesthetically pleasing console client. I’ve been working on this application for six months. The entire project was built by hand, without the use of AI (aside from looking up reference information). This process gave me a deep understanding of how Meshtastic works under the hood; I’ve thoroughly explored its protocol and behavioral nuances. As the project evolved, I contributed to the official Meshtastic library for Rust, adding new features and making minor fixes. The long-term goal of the project is to achieve feature parity with the official Android app. The project is currently in the alpha-testing stage, so there may be bugs — some of which could even cause the program to crash. That’s why I need your help: please give my application a try — I’m confident many of you will enjoy it! Any feedback is welcome via Issues in Github. Highlights - Vibecode-free fully handwritten code - Dependency-free single-binary application - Cross-platform: Linux, macOS, Windows - Fully asynchronous (Tokio channels-based) - Optimized for small screens (down to 80x24 characters) - Respects the user terminal colors (no fancy opinionated TrueColor schemes) - Keyboard-centric UI navigation - Minimum use of emoji/non-ascii characters in UI - Small memory footprint (~20KB) - Supports multiple connection protocols (TCP, BLE, Serial) - Could discover devices automatically: BLE paired devices, TCP devices via mDNS, Serial devices - Uses standard system directories for storing configuration, database and logs Tech stack |Feature|Library| |:-|:-| |TUI: Framework|ratatui| |TUI: Backend|crossterm| |TUI: Inputs|ratatui-textarea| |TUI: Lists|tui-widget-list| |Interaction with Meshtastic|meshtastic| |Clipboard functionality|arboard| |Bluetooth devices discovery|btleplug| |TCP devices discovery|mdns-sd| |Logging|tracing| |Async/Channels|tokio| |Configuration|confy, etcetera| |Errors|anyhow, thiserror| |Datetime|chrono| |Emoji selector|emojis| |Local DB|rusqlite, rusqlite_migration| Download Latest releases are available here: github.com/acelot/termtastic/releases

Lemme introduce to you my TUI client — Termtastic Hello, guys! I wanna introduce you Termtastic — my handmade feature-rich Meshtastic® console (TUI) client written in Rust. preview.redd.it Repo link: acelot/termtastic I wanted to create a cross-platform, minimalist, and aesthetically pleasing console client. I’ve been working on this application for six months. The entire project was built by hand, without the use of AI (aside from looking up reference information). This process gave me a deep understanding of how Meshtastic works under the hood; I’ve thoroughly explored its protocol and behavioral nuances. As the project evolved, I contributed to the official Meshtastic library for Rust, adding new features and making minor fixes. The long-term goal of the project is to achieve feature parity with the official Android app. The project is currently in the alpha-testing stage, so there may be bugs — some of which could even cause the program to crash. That’s why I need your help: please give my application a try — I’m confident many of you will enjoy it! Any feedback is welcome via Issues in Github. Highlights - Vibecode-free fully handwritten code - Dependency-free single-binary application - Cross-platform: Linux, macOS, Windows - Fully asynchronous (Tokio channels-based) - Optimized for small screens (down to 80x24 characters) - Respects the user terminal colors (no fancy opinionated TrueColor schemes) - Keyboard-centric UI navigation - Minimum use of emoji/non-ascii characters in UI - Small memory footprint (~20KB) - Supports multiple connection protocols (TCP, BLE, Serial) - Could discover devices automatically: BLE paired devices, TCP devices via mDNS, Serial devices - Uses standard system directories for storing configuration, database and logs Tech stack |Feature|Library| |:-|:-| |TUI: Framework|ratatui| |TUI: Backend|crossterm| |TUI: Inputs|ratatui-textarea| |TUI: Lists|tui-widget-list| |Interaction with Meshtastic|meshtastic| |Clipboard functionality|arboard| |Bluetooth devices discovery|btleplug| |TCP devices discovery|mdns-sd| |Logging|tracing| |Async/Channels|tokio| |Configuration|confy, etcetera| |Errors|anyhow, thiserror| |Datetime|chrono| |Emoji selector|emojis| |Local DB|rusqlite, rusqlite_migration| Download Latest releases are available here: github.com/acelot/termtastic/releases

An elapsed timer can still be in use: the Tokio 1.54.1 fix If your application resolved Tokio 1.54.0 and uses time intervals, check the lockfile and move to 1.54.1. The October 21 release reverts a timer ownership change that upstream identified as a data race and use-after-free. The interesting part is the order of operations. A timer can be marked deregistered while the driver still has work to do with the entry's memory. In the failure described in PR #8692, StateCell::fire marked an entry as deregistered before taking its waker. Meanwhile, another thread could poll an interval. After observing the tick as ready, Interval::poll_tick called Sleep::reset_without_timer, which could drop or overwrite that entry while the driver was still accessing it. Tokio 1.54.0 had removed Drop implementations from TimerEntry and the alternative timer. Sleep still had cancellation on drop, but discarding its internal timer during reset did not go through that outer destructor. The lifetime guarantee therefore missed a destruction path. Version 1.54.1 restores cancellation on drop for the timer entries and the runtime handle needed by the traditional timer. Its cancellation path takes the driver lock and clears the entry. The lock provides the synchronization needed before the entry's storage can be reused. That gives a useful review question for intrusive queues and runtime internals: after a completion flag becomes visible, what proves that the producer has made its final access to the object? Removing the object from a logical data structure does not, by itself, answer that question. The destructor needs a synchronization boundary that also covers replacement and reset paths. The patch adds a model test, interval_poll_tick_races_with_fire. One thread polls an interval twice while the timer driver advances time on another thread. This targets the firing/reset interleaving directly. I inspected the upstream patch; I have not independently run the model test or reproduced the race. For an application, the immediate check is small: cargo tree -i tokio If the resolved version is 1.54.0, update the dependency resolution to 1.54.1, inspect the lockfile diff, and run the application's tests. A manifest range alone does not show which version the deployed binary contains. The official changelog dates 1.54.0 to October 19, 2026 and 1.54.1 to October 21. The latter lists this revert as its sole fix. These sources establish the regression in 1.54.0; they do not establish that every interval workload triggers it, or justify assigning a severity or CVE. Sources: tokio#8692 github.com/tokio-rs/tokio/blob/master/tokio/CHANG… github.com/tokio-rs/tokio/releases/tag/tokio-1.54… Disclosure: AI-assisted writing and source review. The race description and test are from upstream; no independent reproduction is claimed.

@heyandras You could do the funniest thing and go full Rust tokio-rs/topcoat

Tokio 1.54.0 is out, bringing two notable additions for low-latency systems: Builder::max_io_events_per_busy_tick and stabilized tokio::runtime::worker_index(). Here is the operational tradeoff and why it matters: (1/3) github.com/tokio-rs/tokio/releases/tag/tokio-1.54…

Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway --- title: "Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway" published: true description: "One Tesla T4 serving Gemma 4 E2B to three paying tenants. A 245-line Rust gateway gives each one a key, its own adapter, a priority and a private cache, and bills them by GPU time from a ledger that matches vLLM's counters to the token." tags: devchallenge, hf26challenge, gemma, rust cover_image: raw.githubusercontent.com --- This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle This article provides a step by step guide to renting out one NVIDIA T4 on a Google Cloud VM to several small customers at once, with Gemma 4 E2B served by vLLM and a small Rust gateway in front. Every log, script and number is committed. xbill9/t4-timeshare One customer sending 4,000-token prompts pushes everyone else's first token from 0.28 s to 29.90 s on a shared T4. vLLM's priority scheduling brings that back to 6.52 s. The gateway adds what turns a shared GPU into a business: keys, a fine-tune per customer that no one else can call, a prefix cache per customer, and a ledger that matched vLLM's own token counters exactly. Billed by GPU time, the long-prompt customer pays 27.2% of the VM for 22.0% of the tokens. --- What I Built A small AI job costs too much either way. A GPU of your own sits idle most of the day. A per-token API bill grows with every customer you add, and every prompt leaves your building. The side hustle is the middle: one person rents one T4, and sells shares of it to a few small businesses at a flat monthly price. Each customer gets: - an API key that works with any OpenAI client - its own fine-tuned adapter on the shared base model, callable by that customer only - a prefix cache that other customers cannot read through response timing - a bill computed from metered tokens, weighted by what each token costs the GPU The operator gets one config file and a ledger. The three demo tenants are picked so each one loads the T4 differently: | Tenant | Job | Shape | | :--- | :--- | :--- | | A | Summarise long daily logs | Few requests, 4,096-token prompts | | B | Triage short support emails | One 512-token request every 2 s | | C | Tag product rows, with its own adapter | One 512-token request every 2 s | --- Demo The gateway listens on port 8080 and speaks the OpenAI chat API. day3/demo.sh, run with bash -v: curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-B" \ -d '{"messages":[{"role":"user","content":"Label this email: my parcel arrived broken"}],"max_tokens":20}' \ | jq -c '{model, usage: .usage.total_tokens}' {"model":"xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4","usage":37} curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-C" \ -d '{"messages":[{"role":"user","content":"Label this email: my parcel arrived broken"}],"max_tokens":20}' \ | jq -c '{model, usage: .usage.total_tokens}' {"model":"rand","usage":26} curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-B" \ -d '{"model":"rand","messages":[{"role":"user","content":"hi"}]}' -w ' %{http_code}\n' {"error":"adapter belongs to another tenant"} 403 curl -s localhost:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}]}' -w ' %{http_code}\n' {"error":"missing or unknown key"} 401 tail -4 ledger.jsonl {"completion_tokens":20,"model":"xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4","prompt_tokens":17,"queue_ms":0.0006799999999999999,"status":200,"tenant":"B","total_ms":355.14126200000004,"ts":1792165075.2638927} {"completion_tokens":9,"model":"rand","prompt_tokens":17,"queue_ms":0.000778,"status":200,"tenant":"C","total_ms":498.805413,"ts":1792165075.7809048} {"refused":"adapter belongs to another tenant","status":403,"tenant":"B","ts":1792165075.7979941} {"refused":"missing or unknown key","status":401,"tenant":null,"ts":1792165075.8125255} The same request from B and C reaches two different models: B gets the base model, C gets its adapter without naming it. B asking for C's adapter is refused, and so is a request with no key. Every one of the four lands in the ledger. --- Code xbill9/t4-timeshare | Path | What | | :--- | :--- | | gateway/src/main.rs | The gateway, 245 lines of Rust (axum, reqwest, tokio) | | gateway/tenants.toml | Keys, adapters, priorities and in-flight caps | | day1/ | Checks on the T4 before building: LoRA, cache salt, streamed usage, priority, prefill cost | | day2/ | The noisy-neighbour test straight to vLLM | | day3/ | The same test through the gateway, isolation checks, the bill and VM pricing | --- How I Built It At This Point You Should Have… - A Google Cloud VM with one NVIDIA T4. This one is an n1-standard-2 in us-west2-b - vLLM 0.29.0 serving xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4, Gemma 4 E2B with Google's quantization-aware 4-bit weights, fp16, at most 8 sequences at once - Rust, for the gateway - A clone: git clone https://github.com/xbill9/t4-timeshare --- Step 1 — Check the Engine Can Do Multi-Tenant Work Four features decide whether one vLLM can serve several paying customers. Each was checked on the T4 before any gateway code was written (day1/RESULTS.md): | Check | Result | | :--- | :--- | | LoRA adapters on the 4-bit weights, Turing GPU | 🟢 A zero adapter returns the base model's output, 40 of 40 tokens. A random adapter differs from token 1. 6 of 6 requests in one mixed batch match their solo output. | | Per-request prefix-cache salt | 🟢 Same salt, repeat: 1,216 of 1,217 tokens from cache. Different salt: 0. | | Token usage on streamed responses | 🟢 Present with stream_options.include_usage, absent without it. The gateway sets it. | | Priority scheduling | 🟢 With --scheduling-policy priority, a high-priority request queued last finished first. | The 6 of 6 is what makes per-customer fine-tunes cheap: requests for different adapters share one batch on one GPU, and each still gets its own model's answer. --- Step 2 — Measure What Hurts on a T4 A customer's request costs the GPU in two parts: reading the prompt (prefill) and writing the answer (decode). day1/prefill_cost.py times prefill alone, one request at a time, 5 per length, with the prefix cache unable to help: prompt ~ 256: median 106.9 ms min 103.4 max 108.7 prompt ~ 512: median 262.0 ms min 258.9 max 267.9 prompt ~ 1024: median 730.8 ms min 717.0 max 737.4 prompt ~ 2048: median 2180.8 ms min 2149.6 max 2223.6 prompt ~ 4096: median 7122.4 ms min 7050.4 max 7363.7 prompt ~ 8192: median 25364.0 ms min 25261.6 max 25697.8 prefill fit over 30 requests: -12.8 ms + 374.32 us x n + 336.9989 ns x n^2; worst error 2.1% decode: median 9.34 ms per output token over 5 pairs (256 tokens minus 1 token, same length prompt) Prefill grows with the square of the prompt. One 4,096-token prompt costs as much GPU time as 767.8 output tokens. On a T4 the customer who hurts the others is the one with long prompts, more than the one with many requests. --- Step 3 — Watch One Customer Stall the Others day2/m1_direct.py runs the three tenants straight against vLLM. B and C each send one 512-token request every 2 seconds and stream 64 tokens back. For 180 seconds, A keeps four 4,096-token summaries in flight. Every prompt starts with a random tag, so nothing comes from cache. First come, first served, every request in the same queue: [quiet] B: 90/90 ok first token median 0.29 s 95th 0.29 s max 0.34 s | total median 0.98 s 95th 0.99 s [flood] A: 16/16 ok first token median 30.82 s 95th 45.37 s max 46.12 s | total median 50.08 s 95th 54.03 s [flood] B: 90/90 ok first token median 29.78 s 95th 44.47 s max 47.81 s | total median 35.07 s 95th 48.91 s A support-email triage that answered in a third of a second now waits half a minute for its first word. B pays for A's prompts. --- Step 4 — Put the Gateway in Front gateway/src/main.rs does six things per request, in order: 1. Key to tenant. Authorization: Bearer <key> looks the tenant up in tenants.toml. No key or an unknown one: 401. 2. Model check. The tenant gets its own adapter by default, or the base model on request. Another tenant's adapter: 403. 3. Priority. The tenant's priority goes into the request body, overwriting whatever the client sent. 4. Cache salt. cache_salt is set to the tenant's name, so cached prompts are shared within a tenant and never across tenants. 5. In-flight cap. The request waits for one of the tenant's slots, a tokio semaphore per tenant. 6. Ledger. When the response ends, or the client disconnects, one JSON line is written with the tenant, the model, the status, vLLM's own prompt and output token counts, the time spent waiting for a slot and the total time. The demo tenants: [[tenant]] name = "A" key = "demo-key-A" max_inflight = 1 priority = 10 [[tenant]] name = "C" key = "demo-key-C" adapter = "rand" max_inflight = 8 priority = 0 A has one slot and runs last. B and C have 8 slots each, the most vLLM runs at once. C's rand adapter is a random rank-8 LoRA from day1/make_adapters.py, standing in for a customer's fine-tune. Streaming responses pass straight through. The gateway watches the stream for the one chunk that carries usage and records it when the stream ends. A request that dies halfway is still in the ledger, and its slot is released at that moment. --- Step 5 — Check Isolation python3 day3/isolation.py PASS no key: 401 PASS unknown key: 401 PASS B asks for C's adapter: 403 PASS B asks for a model that is not served: 404 PASS B gets the base model: xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 PASS C gets its own adapter by default: rand PASS C's answer differs from the base model's: 'Here are a few examples of how a spare G' vs 'Here are three common uses for a spare G' PASS B's repeat is served from B's cache: 2400 hit tokens (first send 0) PASS A sending B's prompt gets none of B's cache: 0 hit tokens 9 of 9 pass The last two lines are the privacy check. When B sends a long prompt twice, the second send is served from cache. A sends the identical prompt and gets nothing from B's cache, so A cannot tell from response time whether B ever sent it. The hit counts come from vLLM's /metrics, not from timing. --- Step 6 — Rerun the Noisy Neighbour Through the Gateway day3/run_m1.sh python3 day3/compare.py run B+C first token median 95th max B+C done A done quiet, no A (direct) 0.28 s 0.29 s 0.37 s 180 0 direct, first come first served 29.90 s 44.49 s 47.81 s 180 16 direct, A at low priority 6.52 s 14.71 s 16.31 s 180 16 gateway: priority + A capped at 1 7.06 s 12.11 s 13.55 s 180 7 Priority does most of the work: B and C's median wait falls from 29.90 s to 6.52 s. Capping A at one request shortens the worst waits, from 14.71 s to 12.11 s at the 95th percentile. A pays for it, finishing 7 summaries in the window against 16. B and C still wait about 7 s, close to one 4,096-token prefill (7.1 s in Step 2). A request that arrives while A's prompt is being read waits for that read to finish. 🔎 Tip: size the cap above what the tenant needs. B sends a request every 2 seconds and each one spends about 12 seconds on the GPU under load, so B needs about 6 slots. With B capped at 4, the same run gave B a median time inside vLLM of 11.61 s and a median wait for a gateway slot of 37.77 s (day3/cap4/). A cap is for the customer you want to slow down. --- Step 7 — Check the Ledger Against vLLM A bill is only as good as its meter. day3/bill.py adds up the ledger and compares it with vLLM's own prompt_tokens_total and generation_tokens_total counters, read before and after the run: ledger vs vLLM counters: prompt 122,442 vs 122,442, output 12,416 vs 12,416; match: True Every token vLLM processed in the run is attributed to a tenant, and no tenant is charged for a token vLLM did not process. --- Step 8 — Send the Bill The VM's on-demand price comes from the Cloud Billing Catalog API, fetched during the build (day3/pricing.us-west2.txt): T4 0.410000 + 2 vCPU x 0.037970 + 7.5 GiB x 0.005089 = 0.5241 USD/hour x 730 hours = 382.60 USD/month bill.py prices each request in GPU seconds, using the prefill fit from Step 2 plus 9.34 ms per output token, and splits the month by each tenant's share. Over the gateway run: tenant requests refused prompt tok output tok token share GPU s GPU share bill USD A 7 0 28,718 896 22.0% 58.7 27.2% 104.22 B 90 0 46,803 5,760 39.0% 78.4 36.4% 139.11 C 90 0 46,921 5,760 39.1% 78.5 36.4% 139.26 A sent 22.0% of the tokens and used 27.2% of the GPU. A flat price per token would undercharge the long-prompt customer and pass the difference on to the other two. Weighting prompt tokens by their measured prefill cost puts it back. --- Compare and Contrast | | Straight to vLLM | vLLM with priority | Through the gateway | | :--- | :--- | :--- | :--- | | B and C, median first token under load | 29.90 s | 🥈 6.52 s | 🥇 7.06 s | | B and C, 95th percentile | 44.49 s | 🥈 14.71 s | 🥇 12.11 s | | Keys per customer | ❌ no | ❌ no | 🟢 yes | | Adapter only its owner can call | ❌ no | ❌ no | 🟢 yes | | Prefix cache per customer | ❌ no | ❌ no | 🟢 yes | | Per-customer bill, checked against vLLM | ❌ no | ❌ no | 🟢 yes | --- So, Which One? Turn on --scheduling-policy priority for any shared vLLM: on these runs it is the single biggest change, and it costs one flag. Selling the GPU takes the gateway, because priority alone still lets any client claim any priority, call any adapter and read any cache. The gateway sets all three from the key. --- Prior Work - The T4 rig: the VM, vLLM 0.29.0 install, Turing patch and the vllm-t4 script come from my earlier gemma4-dev T4 serving work. - gemma-rust: reqwest client patterns for the OpenAI-compatible endpoint. - dev-hacktoberfest (Week 1): every result checked by code that did not produce it. - Open source used: vLLM (Apache 2.0), Gemma 4 E2B (Gemma terms), axum, reqwest, tokio. --- Why Does Open Innovation Matter? Every part of this business needs the weights. - The model runs on hardware the operator controls. That is possible only with downloadable weights, and it is what lets the operator promise a customer that its emails never reach a model vendor. - One base model, one small adapter per customer. Per-customer fine-tunes that share one GPU and one batch need the weights and an engine that serves adapters side by side. vLLM's multi-LoRA does that on a 2018-era T4. - The cost is flat. The operator pays 382.60 USD a month whatever the customers send. A heavy customer costs the others waiting time, which the scheduler controls, and costs the operator nothing extra. - Everything can be checked. Engine, gateway and model are open, so the cache isolation and the bill are tested against vLLM's own counters, and a customer can read the code that bills them. --- Prize Categories Gemma: Gemma 4 E2B, served locally on a T4 with vLLM, with per-customer LoRA adapters on Google's quantization-aware 4-bit weights. --- Summary The goal of this project was to sell shares of one T4 to several small customers without one customer ruining the service for the others. The key to the solution was vLLM's priority scheduling, adapters and cache salt, controlled per customer by a gateway that holds the keys. The results were: - 🟢 A long-prompt customer pushed other customers' first token from 0.28 s to 29.90 s. Priority brought it to 6.52 s; the gateway's cap on that customer brought the 95th percentile from 14.71 s to 12.11 s. - 🟢 9 of 9 isolation checks pass: keys, adapter ownership and a prefix cache per customer. - 🟢 The ledger matched vLLM's token counters exactly: 122,442 prompt and 12,416 output tokens. - 🟢 A per-adapter customer runs in the same batch as base-model customers, at about the same speed. - ⚠️ B and C still wait about one long prefill, around 7 s, while A is active. Chunking long prompts further, or giving A its own hours, are the next steps. - ❌ A's throughput falls from 16 summaries to 7 in the window. That is the price A pays for a cheaper share. Scope: one T4 on one Google Cloud VM, vLLM 0.29.0, one 180-second run per configuration. The two direct runs and the gateway run used separate server starts with the same flags; in the direct runs C used the base model, through the gateway it used its adapter. The token bucket, the per-customer fair queue and a tenant-count sweep from SPEC.md are not built. The VM price excludes disk and network. The strategy for renting out one T4 to several customers with Gemma 4 and vLLM was validated with an incremental step by step approach. --- References - Hacktoberfest Open-Source AI Challenge, Week 2 - t4-timeshare on GitHub - vLLM - Gemma 4 E2B, 4-bit build used here - Google Cloud GPU pricing

RustのWebアプリをCloudflare Workersにデプロイしてみたら、驚くほど簡単で安かった こんにちは。コロンビア大学の博士課程でシステムセキュリティ・AIを研究している Koukyosyumei です。 さて、皆さんはRustでWebアプリケーションを開発するとき、デプロイ先として何を使っていますか？AWS EC2やECS、Fly.io、Renderなど、数えきれないほどの選択肢がありますが、最近はCloudflare Workers もかなり便利なのではないかと感じています。 筆者は普段Rustをメインに開発を行っているのですが、Cloudflare Workersは最近Rustのサポートを急速に拡大させています。例えば、RustをWebAssembly (Wasm) にコンパイルすることで、Rustで書いたWebアプリケーションをそのままエッジ環境で実行できます。そのため、axumやh5iなどのweb フレームワークで通常通り作成したRustのプログラムを、簡単にデプロイすることができます。 この記事では、最近筆者が暇つぶしでo5s というRust製のWebアプリケーションを作った経験をもとに、RustアプリケーションをCloudflare Workersで公開する方法や、料金・使い勝手について紹介します。 Cloudflare Workersとは？ Cloudflare Workers は、Cloudflareが提供するサーバーレスのアプリケーション実行環境です。一般的なVPSや仮想マシンとは異なり、サーバーのセットアップや管理を自分で行う必要がなく、コードをデプロイするだけで、Cloudflareのグローバルネットワーク上でアプリケーションを公開できます。 特に便利だと感じたのは、以下の点です。 - サーバー管理が不要：OSの更新、Webサーバーの設定、プロセス管理などを自分で行う必要がない - 無料枠が大きい：1日10万リクエストまで無料で利用できる - デプロイが簡単：基本的には wrangler deploy だけ - HTTPSが標準で利用可能：workers.dev のサブドメインが提供される - GitHubとの連携：GitHubへのpushをきっかけにビルド・デプロイできる - Rustをサポート：RustコードをWasmにコンパイルして実行できる コンパイルしたバイナリをLinuxサーバーに配置し、リバースプロキシやTLS証明書を設定するなどの作業は、Cloudflare側で自動的にやってくれます。 RustでCloudflare Workersを使う Cloudflareは、Rust向けの公式SDKとして workers-rs を提供しています。これを用いると、Rustのコードは wasm32-unknown-unknown ターゲット向けにコンパイルされ、Cloudflare Workers上で実行されます。 まずは開発環境を準備します。 rustup target add wasm32-unknown-unknown cargo install cargo-generate 次に、公式テンプレートからプロジェクトを作成します。 cargo generate cloudflare/workers-rs 生成されたプロジェクトには、Rustコードのほかに wrangler.toml という設定ファイルが含まれています。WranglerはCloudflare Workersの開発・デプロイ用CLIで、これだけで、Rustで書いたAPIをインターネットに公開できます。 # ローカルで実行 npx wrangler dev # Cloudflareにデプロイ npx wrangler deploy 初回はCloudflareへのログインやサブドメインの設定が必要ですが、一度設定すれば、以降はCLIから簡単に更新できます。詳細は公式のRustガイドにまとまっています。 実例：Rust製Webアプリケーション「o5s」 今回Cloudflare Workersにデプロイしたのは、先週末に暇つぶしで開発してみた o5s というオープンソースプロジェクトです。 - GitHub：h5i-dev/o5s o5sは、多数のAIエージェントが同じコードベースを並行編集することを想定した、Git互換のバージョン管理・コラボレーションシステムです。通常のGitでは、複数のエージェントが同時にコードを変更すると、rebaseやコンフリクト解決がボトルネックになることがあります。そこで、o5sでは、変更を依存関係付きのpatchとして管理し、競合しない変更を順序に依存せず統合できる仕組みを実装しています。 ただし、今回紹介したいのはバージョン管理アルゴリズムではなく、そのWebアプリケーションをどのようにホスティングしたか、という部分です。 アプリケーション構成 o5sのバックエンドはRustで実装されています。Webフレームワークには、Axumベースの h5i-app を使用しています。 h5i-appは、通常のRust Webアプリケーションを構築するためのフレームワークで、Axumのエコシステムを利用しながら、必要に応じてLean 4を使った形式検証も行えることが特徴です。例えば、認可チェックや状態遷移など、アプリケーションの重要なロジックについて、Rustの実装をLean 4へ変換し、性質を数学的に証明できます。o5sでもこの仕組みを使い、パッチの承認条件や組織間のデータ分離などの性質を検証しています。 ローカルとCloudflare Workersでロジックを共有 o5sは、ローカル環境ではAxumベースのHTTPサーバーとして、Cloudflare環境では workers-rs を使ったWorkerとして動作します。 ここで便利なのは、アプリケーションのコアロジックを両方の環境で共有できることです。デプロイ先に応じてHTTPリクエストの受け取り方は異なりますが、ビジネスロジックを二重に実装する必要はありません。 o5sでは、おおむね次のような構成にしています。 Web UI / API Clients | +--------+--------+ | | Local Server Cloudflare (Axum) Workers | | +--------+--------+ | Shared Rust Logic (h5i-app) | PostgreSQL | Lean 4 verified core 実際にはCloudflare側ではHyperdrive経由でPostgreSQLへ接続し、Gitリポジトリの保存にはCloudflare Artifactsも利用しています。 Cloudflare Workersへのデプロイ 実際のo5sでは、リポジトリのルートに wrangler.toml を配置しています。設定の一部を抜粋すると、次のようになっています。 name = "o5s" main = "crates/worker/build/index.js" compatibility_date = "2026-10-01" [build] command = "[ -f build.sh ] || cd crates/worker; sh build.sh" [placement] region = "aws:us-east-1" 設定ファイル全体はこちらで確認できます。main にはビルドされたWorkerのエントリーポイントを、build.command にはRustコードをWasmへコンパイルするためのコマンドを設定しています。 また、今回のアプリケーションはPostgreSQLを利用しているため、CloudflareのHyperdriveを使用しています。Hyperdriveは、Workersから外部のPostgreSQLなどへアクセスするためのサービスで、接続プーリングなどを提供しています。 デプロイコマンド RustからWorkers向けのWasmを生成するために、worker-build を使います。 cargo install worker-build o5sでは、必要なビルド処理をスクリプトにまとめています。 # Cloudflareへのログイン npx wrangler login # デプロイ npx wrangler deploy PostgreSQLのテーブル作成やSecretの登録などは事前に済ませる必要がありますが、アプリケーションの更新自体はこのコマンドで完了します。デプロイ後は、独自ドメインを持っていなくても、Cloudflareが提供する workers.dev サブドメインに公開されます。 さらに、GitHub連携のWorkers Buildsを使えば、main ブランチへのpushに合わせて自動的にビルド・デプロイすることもできます。 料金がかなり安い 2026年10月時点のCloudflare Workersの料金は、以下のようになっています。 | 項目 | Free | Paid（Standard） | |---|---|---| | 基本料金 | $0 | $5/月 | | リクエスト数 | 10万/日 | 1,000万/月を含む | | 超過リクエスト | 無料枠超過後は制限 | 100万回あたり$0.30 | | CPU時間 | 1リクエストあたり10ms | 月3,000万CPU-msを含む | | CPU時間の追加料金 | — | 100万CPU-msあたり$0.02 | > 出典：Cloudflare Workers Pricing 例えば、1日1万リクエスト程度の小規模なWeb APIであれば、CPU時間などの制限にも収まる限り、無料枠だけで運用できるので、個人開発や小規模なWebアプリケーションにとっては、かなり余裕のある数字ではないでしょうか。 もちろん、実際の料金はCPU使用時間に依存します。また、PostgreSQLやCloudflare Artifactsなどの外部サービス・追加機能の費用は別途考える必要があります。 RustでWorkersを使う際の注意点 もちろん、Cloudflare WorkersがすべてのRustアプリケーションに最適というわけではありません。実際に使ってみて、特に注意が必要だと感じた点もあります。 1. すべてのRust crateが使えるわけではない WorkersではRustをWasmにコンパイルして実行するため、Linuxのシステムコールやネイティブライブラリに依存するcrateは、そのままでは動作しない場合があります。 例えば、通常のサーバー環境向けに実装されたファイル操作やネットワーク処理では、Workers向けの実装への切り替えが必要になることがあります。Cloudflareは対応crateに関するガイドも公開しています。 2. データベースは別途用意する必要がある Workers自体は、一般的なLinuxサーバーのようにローカルディスクへ永続的なデータを書き込んで保存する環境ではありません。そのため、データの永続化には、Cloudflare D1、KV、R2、あるいは外部のPostgreSQLなどを使用する必要があります。 まとめ 今回、Rust製のWebアプリケーションであるo5sをCloudflare Workersへデプロイしてみて、RustのWebアプリケーションにとってかなり魅力的な選択肢だと感じました。RustでWebサービスを作っていて、デプロイ先を探している方は、ぜひ一度Cloudflare Workersを試してみてください。 参考リンク - Cloudflare Workers - Cloudflare Workers - Rust - Cloudflare Workers Pricing - workers-rs - o5s - GitHub - h5i - GitHub