Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway
---
title: "Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway"
published: true
description: "One Tesla T4 serving Gemma 4 E2B to three paying tenants. A 245-line Rust gateway gives each one a key, its own adapter, a priority and a private cache, and bills them by GPU time from a ledger that matches vLLM's counters to the token."
tags: devchallenge, hf26challenge, gemma, rust
cover_image: raw.githubusercontent.com
---
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
This article provides a step by step guide to renting out one NVIDIA T4 on a Google Cloud VM to several small customers at once, with Gemma 4 E2B served by vLLM and a small Rust gateway in front. Every log, script and number is committed.
xbill9/t4-timeshare
One customer sending 4,000-token prompts pushes everyone else's first token from 0.28 s to 29.90 s on a shared T4. vLLM's priority scheduling brings that back to 6.52 s. The gateway adds what turns a shared GPU into a business: keys, a fine-tune per customer that no one else can call, a prefix cache per customer, and a ledger that matched vLLM's own token counters exactly. Billed by GPU time, the long-prompt customer pays 27.2% of the VM for 22.0% of the tokens.
---
What I Built
A small AI job costs too much either way. A GPU of your own sits idle most of the day. A per-token API bill grows with every customer you add, and every prompt leaves your building.
The side hustle is the middle: one person rents one T4, and sells shares of it to a few small businesses at a flat monthly price. Each customer gets:
- an API key that works with any OpenAI client
- its own fine-tuned adapter on the shared base model, callable by that customer only
- a prefix cache that other customers cannot read through response timing
- a bill computed from metered tokens, weighted by what each token costs the GPU
The operator gets one config file and a ledger.
The three demo tenants are picked so each one loads the T4 differently:
| Tenant | Job | Shape |
| :--- | :--- | :--- |
| A | Summarise long daily logs | Few requests, 4,096-token prompts |
| B | Triage short support emails | One 512-token request every 2 s |
| C | Tag product rows, with its own adapter | One 512-token request every 2 s |
---
Demo
The gateway listens on port 8080 and speaks the OpenAI chat API. day3/demo.sh, run with bash -v:
curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-B" \
-d '{"messages":[{"role":"user","content":"Label this email: my parcel arrived broken"}],"max_tokens":20}' \
| jq -c '{model, usage: .usage.total_tokens}'
{"model":"xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4","usage":37}
curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-C" \
-d '{"messages":[{"role":"user","content":"Label this email: my parcel arrived broken"}],"max_tokens":20}' \
| jq -c '{model, usage: .usage.total_tokens}'
{"model":"rand","usage":26}
curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-B" \
-d '{"model":"rand","messages":[{"role":"user","content":"hi"}]}' -w ' %{http_code}\n'
{"error":"adapter belongs to another tenant"} 403
curl -s localhost:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}]}' -w ' %{http_code}\n'
{"error":"missing or unknown key"} 401
tail -4 ledger.jsonl
{"completion_tokens":20,"model":"xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4","prompt_tokens":17,"queue_ms":0.0006799999999999999,"status":200,"tenant":"B","total_ms":355.14126200000004,"ts":1792165075.2638927}
{"completion_tokens":9,"model":"rand","prompt_tokens":17,"queue_ms":0.000778,"status":200,"tenant":"C","total_ms":498.805413,"ts":1792165075.7809048}
{"refused":"adapter belongs to another tenant","status":403,"tenant":"B","ts":1792165075.7979941}
{"refused":"missing or unknown key","status":401,"tenant":null,"ts":1792165075.8125255}
The same request from B and C reaches two different models: B gets the base model, C gets its adapter without naming it. B asking for C's adapter is refused, and so is a request with no key. Every one of the four lands in the ledger.
---
Code
xbill9/t4-timeshare
| Path | What |
| :--- | :--- |
| gateway/src/main.rs | The gateway, 245 lines of Rust (axum, reqwest, tokio) |
| gateway/tenants.toml | Keys, adapters, priorities and in-flight caps |
| day1/ | Checks on the T4 before building: LoRA, cache salt, streamed usage, priority, prefill cost |
| day2/ | The noisy-neighbour test straight to vLLM |
| day3/ | The same test through the gateway, isolation checks, the bill and VM pricing |
---
How I Built It
At This Point You Should Have…
- A Google Cloud VM with one NVIDIA T4. This one is an n1-standard-2 in us-west2-b
- vLLM 0.29.0 serving xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4, Gemma 4 E2B with Google's quantization-aware 4-bit weights, fp16, at most 8 sequences at once
- Rust, for the gateway
- A clone: git clone https://github.com/xbill9/t4-timeshare
---
Step 1 — Check the Engine Can Do Multi-Tenant Work
Four features decide whether one vLLM can serve several paying customers. Each was checked on the T4 before any gateway code was written (day1/RESULTS.md):
| Check | Result |
| :--- | :--- |
| LoRA adapters on the 4-bit weights, Turing GPU | 🟢 A zero adapter returns the base model's output, 40 of 40 tokens. A random adapter differs from token 1. 6 of 6 requests in one mixed batch match their solo output. |
| Per-request prefix-cache salt | 🟢 Same salt, repeat: 1,216 of 1,217 tokens from cache. Different salt: 0. |
| Token usage on streamed responses | 🟢 Present with stream_options.include_usage, absent without it. The gateway sets it. |
| Priority scheduling | 🟢 With --scheduling-policy priority, a high-priority request queued last finished first. |
The 6 of 6 is what makes per-customer fine-tunes cheap: requests for different adapters share one batch on one GPU, and each still gets its own model's answer.
---
Step 2 — Measure What Hurts on a T4
A customer's request costs the GPU in two parts: reading the prompt (prefill) and writing the answer (decode). day1/prefill_cost.py times prefill alone, one request at a time, 5 per length, with the prefix cache unable to help:
prompt ~ 256: median 106.9 ms min 103.4 max 108.7
prompt ~ 512: median 262.0 ms min 258.9 max 267.9
prompt ~ 1024: median 730.8 ms min 717.0 max 737.4
prompt ~ 2048: median 2180.8 ms min 2149.6 max 2223.6
prompt ~ 4096: median 7122.4 ms min 7050.4 max 7363.7
prompt ~ 8192: median 25364.0 ms min 25261.6 max 25697.8
prefill fit over 30 requests: -12.8 ms + 374.32 us x n + 336.9989 ns x n^2; worst error 2.1%
decode: median 9.34 ms per output token over 5 pairs (256 tokens minus 1 token, same length prompt)
Prefill grows with the square of the prompt. One 4,096-token prompt costs as much GPU time as 767.8 output tokens. On a T4 the customer who hurts the others is the one with long prompts, more than the one with many requests.
---
Step 3 — Watch One Customer Stall the Others
day2/m1_direct.py runs the three tenants straight against vLLM. B and C each send one 512-token request every 2 seconds and stream 64 tokens back. For 180 seconds, A keeps four 4,096-token summaries in flight. Every prompt starts with a random tag, so nothing comes from cache.
First come, first served, every request in the same queue:
[quiet] B: 90/90 ok first token median 0.29 s 95th 0.29 s max 0.34 s | total median 0.98 s 95th 0.99 s
[flood] A: 16/16 ok first token median 30.82 s 95th 45.37 s max 46.12 s | total median 50.08 s 95th 54.03 s
[flood] B: 90/90 ok first token median 29.78 s 95th 44.47 s max 47.81 s | total median 35.07 s 95th 48.91 s
A support-email triage that answered in a third of a second now waits half a minute for its first word. B pays for A's prompts.
---
Step 4 — Put the Gateway in Front
gateway/src/main.rs does six things per request, in order:
1. Key to tenant. Authorization: Bearer <key> looks the tenant up in tenants.toml. No key or an unknown one: 401.
2. Model check. The tenant gets its own adapter by default, or the base model on request. Another tenant's adapter: 403.
3. Priority. The tenant's priority goes into the request body, overwriting whatever the client sent.
4. Cache salt. cache_salt is set to the tenant's name, so cached prompts are shared within a tenant and never across tenants.
5. In-flight cap. The request waits for one of the tenant's slots, a tokio semaphore per tenant.
6. Ledger. When the response ends, or the client disconnects, one JSON line is written with the tenant, the model, the status, vLLM's own prompt and output token counts, the time spent waiting for a slot and the total time.
The demo tenants:
[[tenant]]
name = "A"
key = "demo-key-A"
max_inflight = 1
priority = 10
[[tenant]]
name = "C"
key = "demo-key-C"
adapter = "rand"
max_inflight = 8
priority = 0
A has one slot and runs last. B and C have 8 slots each, the most vLLM runs at once. C's rand adapter is a random rank-8 LoRA from day1/make_adapters.py, standing in for a customer's fine-tune.
Streaming responses pass straight through. The gateway watches the stream for the one chunk that carries usage and records it when the stream ends. A request that dies halfway is still in the ledger, and its slot is released at that moment.
---
Step 5 — Check Isolation
python3 day3/isolation.py
PASS no key: 401
PASS unknown key: 401
PASS B asks for C's adapter: 403
PASS B asks for a model that is not served: 404
PASS B gets the base model: xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4
PASS C gets its own adapter by default: rand
PASS C's answer differs from the base model's: 'Here are a few examples of how a spare G' vs 'Here are three common uses for a spare G'
PASS B's repeat is served from B's cache: 2400 hit tokens (first send 0)
PASS A sending B's prompt gets none of B's cache: 0 hit tokens
9 of 9 pass
The last two lines are the privacy check. When B sends a long prompt twice, the second send is served from cache. A sends the identical prompt and gets nothing from B's cache, so A cannot tell from response time whether B ever sent it. The hit counts come from vLLM's /metrics, not from timing.
---
Step 6 — Rerun the Noisy Neighbour Through the Gateway
day3/run_m1.sh
python3 day3/compare.py
run B+C first token median 95th max B+C done A done
quiet, no A (direct) 0.28 s 0.29 s 0.37 s 180 0
direct, first come first served 29.90 s 44.49 s 47.81 s 180 16
direct, A at low priority 6.52 s 14.71 s 16.31 s 180 16
gateway: priority + A capped at 1 7.06 s 12.11 s 13.55 s 180 7
Priority does most of the work: B and C's median wait falls from 29.90 s to 6.52 s. Capping A at one request shortens the worst waits, from 14.71 s to 12.11 s at the 95th percentile. A pays for it, finishing 7 summaries in the window against 16.
B and C still wait about 7 s, close to one 4,096-token prefill (7.1 s in Step 2). A request that arrives while A's prompt is being read waits for that read to finish.
🔎 Tip: size the cap above what the tenant needs. B sends a request every 2 seconds and each one spends about 12 seconds on the GPU under load, so B needs about 6 slots. With B capped at 4, the same run gave B a median time inside vLLM of 11.61 s and a median wait for a gateway slot of 37.77 s (day3/cap4/). A cap is for the customer you want to slow down.
---
Step 7 — Check the Ledger Against vLLM
A bill is only as good as its meter. day3/bill.py adds up the ledger and compares it with vLLM's own prompt_tokens_total and generation_tokens_total counters, read before and after the run:
ledger vs vLLM counters: prompt 122,442 vs 122,442, output 12,416 vs 12,416; match: True
Every token vLLM processed in the run is attributed to a tenant, and no tenant is charged for a token vLLM did not process.
---
Step 8 — Send the Bill
The VM's on-demand price comes from the Cloud Billing Catalog API, fetched during the build (day3/pricing.us-west2.txt):
T4 0.410000 + 2 vCPU x 0.037970 + 7.5 GiB x 0.005089 = 0.5241 USD/hour
x 730 hours = 382.60 USD/month
bill.py prices each request in GPU seconds, using the prefill fit from Step 2 plus 9.34 ms per output token, and splits the month by each tenant's share. Over the gateway run:
tenant requests refused prompt tok output tok token share GPU s GPU share bill USD
A 7 0 28,718 896 22.0% 58.7 27.2% 104.22
B 90 0 46,803 5,760 39.0% 78.4 36.4% 139.11
C 90 0 46,921 5,760 39.1% 78.5 36.4% 139.26
A sent 22.0% of the tokens and used 27.2% of the GPU. A flat price per token would undercharge the long-prompt customer and pass the difference on to the other two. Weighting prompt tokens by their measured prefill cost puts it back.
---
Compare and Contrast
| | Straight to vLLM | vLLM with priority | Through the gateway |
| :--- | :--- | :--- | :--- |
| B and C, median first token under load | 29.90 s | 🥈 6.52 s | 🥇 7.06 s |
| B and C, 95th percentile | 44.49 s | 🥈 14.71 s | 🥇 12.11 s |
| Keys per customer | ❌ no | ❌ no | 🟢 yes |
| Adapter only its owner can call | ❌ no | ❌ no | 🟢 yes |
| Prefix cache per customer | ❌ no | ❌ no | 🟢 yes |
| Per-customer bill, checked against vLLM | ❌ no | ❌ no | 🟢 yes |
---
So, Which One?
Turn on --scheduling-policy priority for any shared vLLM: on these runs it is the single biggest change, and it costs one flag. Selling the GPU takes the gateway, because priority alone still lets any client claim any priority, call any adapter and read any cache. The gateway sets all three from the key.
---
Prior Work
- The T4 rig: the VM, vLLM 0.29.0 install, Turing patch and the vllm-t4 script come from my earlier gemma4-dev T4 serving work.
- gemma-rust: reqwest client patterns for the OpenAI-compatible endpoint.
- dev-hacktoberfest (Week 1): every result checked by code that did not produce it.
- Open source used: vLLM (Apache 2.0), Gemma 4 E2B (Gemma terms), axum, reqwest, tokio.
---
Why Does Open Innovation Matter?
Every part of this business needs the weights.
- The model runs on hardware the operator controls. That is possible only with downloadable weights, and it is what lets the operator promise a customer that its emails never reach a model vendor.
- One base model, one small adapter per customer. Per-customer fine-tunes that share one GPU and one batch need the weights and an engine that serves adapters side by side. vLLM's multi-LoRA does that on a 2018-era T4.
- The cost is flat. The operator pays 382.60 USD a month whatever the customers send. A heavy customer costs the others waiting time, which the scheduler controls, and costs the operator nothing extra.
- Everything can be checked. Engine, gateway and model are open, so the cache isolation and the bill are tested against vLLM's own counters, and a customer can read the code that bills them.
---
Prize Categories
Gemma: Gemma 4 E2B, served locally on a T4 with vLLM, with per-customer LoRA adapters on Google's quantization-aware 4-bit weights.
---
Summary
The goal of this project was to sell shares of one T4 to several small customers without one customer ruining the service for the others. The key to the solution was vLLM's priority scheduling, adapters and cache salt, controlled per customer by a gateway that holds the keys. The results were:
- 🟢 A long-prompt customer pushed other customers' first token from 0.28 s to 29.90 s. Priority brought it to 6.52 s; the gateway's cap on that customer brought the 95th percentile from 14.71 s to 12.11 s.
- 🟢 9 of 9 isolation checks pass: keys, adapter ownership and a prefix cache per customer.
- 🟢 The ledger matched vLLM's token counters exactly: 122,442 prompt and 12,416 output tokens.
- 🟢 A per-adapter customer runs in the same batch as base-model customers, at about the same speed.
- ⚠️ B and C still wait about one long prefill, around 7 s, while A is active. Chunking long prompts further, or giving A its own hours, are the next steps.
- ❌ A's throughput falls from 16 summaries to 7 in the window. That is the price A pays for a cheaper share.
Scope: one T4 on one Google Cloud VM, vLLM 0.29.0, one 180-second run per configuration. The two direct runs and the gateway run used separate server starts with the same flags; in the direct runs C used the base model, through the gateway it used its adapter. The token bucket, the per-customer fair queue and a tenant-count sweep from SPEC.md are not built. The VM price excludes disk and network.
The strategy for renting out one T4 to several customers with Gemma 4 and vLLM was validated with an incremental step by step approach.
---
References
- Hacktoberfest Open-Source AI Challenge, Week 2
- t4-timeshare on GitHub
- vLLM
- Gemma 4 E2B, 4-bit build used here
- Google Cloud GPU pricing
2 reactions
Mentions
Termtastic — new TUI client for Linux, macOS and Windows preview.redd.it Hello, guys! I wanna introduce you Termtastic — my handmade feature-rich Meshtastic® console (TUI) client written in Rust. preview.redd.it Channels screen Chat Nodes tab Traceroute results Multiline message support Emojis Logs tab Settings tab Repo link: acelot/termtastic I wanted to create a cross-platform, minimalist, and aesthetically pleasing console client. I’ve been working on this application for six months. The entire project was built by hand, without the use of AI (aside from looking up reference information). This process gave me a deep understanding of how Meshtastic works under the hood; I’ve thoroughly explored its protocol and behavioral nuances. As the project evolved, I contributed to the official Meshtastic library for Rust, adding new features and making minor fixes. The long-term goal of the project is to achieve feature parity with the official Android app. The project is currently in the alpha-testing stage, so there may be bugs — some of which could even cause the program to crash. That’s why I need your help: please give my application a try — I’m confident many of you will enjoy it! Any feedback is welcome via Issues in Github. Highlights - Vibecode-free fully handwritten code - Dependency-free single-binary application - Cross-platform: Linux, macOS, Windows - Fully asynchronous (Tokio channels-based) - Optimized for small screens (down to 80x24 characters) - Respects the user terminal colors (no fancy opinionated TrueColor schemes) - Keyboard-centric UI navigation - Minimum use of emoji/non-ascii characters in UI - Small memory footprint (~20KB) - Supports multiple connection protocols (TCP, BLE, Serial) - Could discover devices automatically: BLE paired devices, TCP devices via mDNS, Serial devices - Uses standard system directories for storing configuration, database and logs Tech stack |Feature|Library| |:-|:-| |TUI: Framework|ratatui| |TUI: Backend|crossterm| |TUI: Inputs|ratatui-textarea| |TUI: Lists|tui-widget-list| |Interaction with Meshtastic|meshtastic| |Clipboard functionality|arboard| |Bluetooth devices discovery|btleplug| |TCP devices discovery|mdns-sd| |Logging|tracing| |Async/Channels|tokio| |Configuration|confy, etcetera| |Errors|anyhow, thiserror| |Datetime|chrono| |Emoji selector|emojis| |Local DB|rusqlite, rusqlite_migration| Download Latest releases are available here: github.com/acelot/termtastic/releases
Lemme introduce to you my TUI client — Termtastic Hello, guys! I wanna introduce you Termtastic — my handmade feature-rich Meshtastic® console (TUI) client written in Rust. preview.redd.it Repo link: acelot/termtastic I wanted to create a cross-platform, minimalist, and aesthetically pleasing console client. I’ve been working on this application for six months. The entire project was built by hand, without the use of AI (aside from looking up reference information). This process gave me a deep understanding of how Meshtastic works under the hood; I’ve thoroughly explored its protocol and behavioral nuances. As the project evolved, I contributed to the official Meshtastic library for Rust, adding new features and making minor fixes. The long-term goal of the project is to achieve feature parity with the official Android app. The project is currently in the alpha-testing stage, so there may be bugs — some of which could even cause the program to crash. That’s why I need your help: please give my application a try — I’m confident many of you will enjoy it! Any feedback is welcome via Issues in Github. Highlights - Vibecode-free fully handwritten code - Dependency-free single-binary application - Cross-platform: Linux, macOS, Windows - Fully asynchronous (Tokio channels-based) - Optimized for small screens (down to 80x24 characters) - Respects the user terminal colors (no fancy opinionated TrueColor schemes) - Keyboard-centric UI navigation - Minimum use of emoji/non-ascii characters in UI - Small memory footprint (~20KB) - Supports multiple connection protocols (TCP, BLE, Serial) - Could discover devices automatically: BLE paired devices, TCP devices via mDNS, Serial devices - Uses standard system directories for storing configuration, database and logs Tech stack |Feature|Library| |:-|:-| |TUI: Framework|ratatui| |TUI: Backend|crossterm| |TUI: Inputs|ratatui-textarea| |TUI: Lists|tui-widget-list| |Interaction with Meshtastic|meshtastic| |Clipboard functionality|arboard| |Bluetooth devices discovery|btleplug| |TCP devices discovery|mdns-sd| |Logging|tracing| |Async/Channels|tokio| |Configuration|confy, etcetera| |Errors|anyhow, thiserror| |Datetime|chrono| |Emoji selector|emojis| |Local DB|rusqlite, rusqlite_migration| Download Latest releases are available here: github.com/acelot/termtastic/releases
An elapsed timer can still be in use: the Tokio 1.54.1 fix If your application resolved Tokio 1.54.0 and uses time intervals, check the lockfile and move to 1.54.1. The October 21 release reverts a timer ownership change that upstream identified as a data race and use-after-free. The interesting part is the order of operations. A timer can be marked deregistered while the driver still has work to do with the entry's memory. In the failure described in PR #8692, StateCell::fire marked an entry as deregistered before taking its waker. Meanwhile, another thread could poll an interval. After observing the tick as ready, Interval::poll_tick called Sleep::reset_without_timer, which could drop or overwrite that entry while the driver was still accessing it. Tokio 1.54.0 had removed Drop implementations from TimerEntry and the alternative timer. Sleep still had cancellation on drop, but discarding its internal timer during reset did not go through that outer destructor. The lifetime guarantee therefore missed a destruction path. Version 1.54.1 restores cancellation on drop for the timer entries and the runtime handle needed by the traditional timer. Its cancellation path takes the driver lock and clears the entry. The lock provides the synchronization needed before the entry's storage can be reused. That gives a useful review question for intrusive queues and runtime internals: after a completion flag becomes visible, what proves that the producer has made its final access to the object? Removing the object from a logical data structure does not, by itself, answer that question. The destructor needs a synchronization boundary that also covers replacement and reset paths. The patch adds a model test, interval_poll_tick_races_with_fire. One thread polls an interval twice while the timer driver advances time on another thread. This targets the firing/reset interleaving directly. I inspected the upstream patch; I have not independently run the model test or reproduced the race. For an application, the immediate check is small: cargo tree -i tokio If the resolved version is 1.54.0, update the dependency resolution to 1.54.1, inspect the lockfile diff, and run the application's tests. A manifest range alone does not show which version the deployed binary contains. The official changelog dates 1.54.0 to October 19, 2026 and 1.54.1 to October 21. The latter lists this revert as its sole fix. These sources establish the regression in 1.54.0; they do not establish that every interval workload triggers it, or justify assigning a severity or CVE. Sources: tokio#8692 github.com/tokio-rs/tokio/blob/master/tokio/CHANG… github.com/tokio-rs/tokio/releases/tag/tokio-1.54… Disclosure: AI-assisted writing and source review. The race description and test are from upstream; no independent reproduction is claimed.
Tokio 1.54.0 is out, bringing two notable additions for low-latency systems: Builder::max_io_events_per_busy_tick and stabilized tokio::runtime::worker_index(). Here is the operational tradeoff and why it matters: (1/3) github.com/tokio-rs/tokio/releases/tag/tokio-1.54…
Tokio 1.54.0 dropped yesterday with Builder::max_io_events_per_busy_tick. Under heavy network bursts, the reactor driver can starve compute tasks by repeatedly polling I/O. Capping busy-tick events forces the driver to yield to the run queue. github.com/tokio-rs/tokio/releases/tag/tokio-1.54…
@notatallshaw I've been on the other side of that recently: I'm doing a lot of finding, reporting & coordinating. Yes, it's difficult to get good AI generated output for vulns/bugs. But definitely possible! Here are some examples: github.com/tokio-rs/tokio/issues
Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway --- title: "Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway" published: true description: "One Tesla T4 serving Gemma 4 E2B to three paying tenants. A 245-line Rust gateway gives each one a key, its own adapter, a priority and a private cache, and bills them by GPU time from a ledger that matches vLLM's counters to the token." tags: devchallenge, hf26challenge, gemma, rust cover_image: raw.githubusercontent.com --- This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle This article provides a step by step guide to renting out one NVIDIA T4 on a Google Cloud VM to several small customers at once, with Gemma 4 E2B served by vLLM and a small Rust gateway in front. Every log, script and number is committed. xbill9/t4-timeshare One customer sending 4,000-token prompts pushes everyone else's first token from 0.28 s to 29.90 s on a shared T4. vLLM's priority scheduling brings that back to 6.52 s. The gateway adds what turns a shared GPU into a business: keys, a fine-tune per customer that no one else can call, a prefix cache per customer, and a ledger that matched vLLM's own token counters exactly. Billed by GPU time, the long-prompt customer pays 27.2% of the VM for 22.0% of the tokens. --- What I Built A small AI job costs too much either way. A GPU of your own sits idle most of the day. A per-token API bill grows with every customer you add, and every prompt leaves your building. The side hustle is the middle: one person rents one T4, and sells shares of it to a few small businesses at a flat monthly price. Each customer gets: - an API key that works with any OpenAI client - its own fine-tuned adapter on the shared base model, callable by that customer only - a prefix cache that other customers cannot read through response timing - a bill computed from metered tokens, weighted by what each token costs the GPU The operator gets one config file and a ledger. The three demo tenants are picked so each one loads the T4 differently: | Tenant | Job | Shape | | :--- | :--- | :--- | | A | Summarise long daily logs | Few requests, 4,096-token prompts | | B | Triage short support emails | One 512-token request every 2 s | | C | Tag product rows, with its own adapter | One 512-token request every 2 s | --- Demo The gateway listens on port 8080 and speaks the OpenAI chat API. day3/demo.sh, run with bash -v: curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-B" \ -d '{"messages":[{"role":"user","content":"Label this email: my parcel arrived broken"}],"max_tokens":20}' \ | jq -c '{model, usage: .usage.total_tokens}' {"model":"xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4","usage":37} curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-C" \ -d '{"messages":[{"role":"user","content":"Label this email: my parcel arrived broken"}],"max_tokens":20}' \ | jq -c '{model, usage: .usage.total_tokens}' {"model":"rand","usage":26} curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer demo-key-B" \ -d '{"model":"rand","messages":[{"role":"user","content":"hi"}]}' -w ' %{http_code}\n' {"error":"adapter belongs to another tenant"} 403 curl -s localhost:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}]}' -w ' %{http_code}\n' {"error":"missing or unknown key"} 401 tail -4 ledger.jsonl {"completion_tokens":20,"model":"xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4","prompt_tokens":17,"queue_ms":0.0006799999999999999,"status":200,"tenant":"B","total_ms":355.14126200000004,"ts":1792165075.2638927} {"completion_tokens":9,"model":"rand","prompt_tokens":17,"queue_ms":0.000778,"status":200,"tenant":"C","total_ms":498.805413,"ts":1792165075.7809048} {"refused":"adapter belongs to another tenant","status":403,"tenant":"B","ts":1792165075.7979941} {"refused":"missing or unknown key","status":401,"tenant":null,"ts":1792165075.8125255} The same request from B and C reaches two different models: B gets the base model, C gets its adapter without naming it. B asking for C's adapter is refused, and so is a request with no key. Every one of the four lands in the ledger. --- Code xbill9/t4-timeshare | Path | What | | :--- | :--- | | gateway/src/main.rs | The gateway, 245 lines of Rust (axum, reqwest, tokio) | | gateway/tenants.toml | Keys, adapters, priorities and in-flight caps | | day1/ | Checks on the T4 before building: LoRA, cache salt, streamed usage, priority, prefill cost | | day2/ | The noisy-neighbour test straight to vLLM | | day3/ | The same test through the gateway, isolation checks, the bill and VM pricing | --- How I Built It At This Point You Should Have… - A Google Cloud VM with one NVIDIA T4. This one is an n1-standard-2 in us-west2-b - vLLM 0.29.0 serving xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4, Gemma 4 E2B with Google's quantization-aware 4-bit weights, fp16, at most 8 sequences at once - Rust, for the gateway - A clone: git clone https://github.com/xbill9/t4-timeshare --- Step 1 — Check the Engine Can Do Multi-Tenant Work Four features decide whether one vLLM can serve several paying customers. Each was checked on the T4 before any gateway code was written (day1/RESULTS.md): | Check | Result | | :--- | :--- | | LoRA adapters on the 4-bit weights, Turing GPU | 🟢 A zero adapter returns the base model's output, 40 of 40 tokens. A random adapter differs from token 1. 6 of 6 requests in one mixed batch match their solo output. | | Per-request prefix-cache salt | 🟢 Same salt, repeat: 1,216 of 1,217 tokens from cache. Different salt: 0. | | Token usage on streamed responses | 🟢 Present with stream_options.include_usage, absent without it. The gateway sets it. | | Priority scheduling | 🟢 With --scheduling-policy priority, a high-priority request queued last finished first. | The 6 of 6 is what makes per-customer fine-tunes cheap: requests for different adapters share one batch on one GPU, and each still gets its own model's answer. --- Step 2 — Measure What Hurts on a T4 A customer's request costs the GPU in two parts: reading the prompt (prefill) and writing the answer (decode). day1/prefill_cost.py times prefill alone, one request at a time, 5 per length, with the prefix cache unable to help: prompt ~ 256: median 106.9 ms min 103.4 max 108.7 prompt ~ 512: median 262.0 ms min 258.9 max 267.9 prompt ~ 1024: median 730.8 ms min 717.0 max 737.4 prompt ~ 2048: median 2180.8 ms min 2149.6 max 2223.6 prompt ~ 4096: median 7122.4 ms min 7050.4 max 7363.7 prompt ~ 8192: median 25364.0 ms min 25261.6 max 25697.8 prefill fit over 30 requests: -12.8 ms + 374.32 us x n + 336.9989 ns x n^2; worst error 2.1% decode: median 9.34 ms per output token over 5 pairs (256 tokens minus 1 token, same length prompt) Prefill grows with the square of the prompt. One 4,096-token prompt costs as much GPU time as 767.8 output tokens. On a T4 the customer who hurts the others is the one with long prompts, more than the one with many requests. --- Step 3 — Watch One Customer Stall the Others day2/m1_direct.py runs the three tenants straight against vLLM. B and C each send one 512-token request every 2 seconds and stream 64 tokens back. For 180 seconds, A keeps four 4,096-token summaries in flight. Every prompt starts with a random tag, so nothing comes from cache. First come, first served, every request in the same queue: [quiet] B: 90/90 ok first token median 0.29 s 95th 0.29 s max 0.34 s | total median 0.98 s 95th 0.99 s [flood] A: 16/16 ok first token median 30.82 s 95th 45.37 s max 46.12 s | total median 50.08 s 95th 54.03 s [flood] B: 90/90 ok first token median 29.78 s 95th 44.47 s max 47.81 s | total median 35.07 s 95th 48.91 s A support-email triage that answered in a third of a second now waits half a minute for its first word. B pays for A's prompts. --- Step 4 — Put the Gateway in Front gateway/src/main.rs does six things per request, in order: 1. Key to tenant. Authorization: Bearer <key> looks the tenant up in tenants.toml. No key or an unknown one: 401. 2. Model check. The tenant gets its own adapter by default, or the base model on request. Another tenant's adapter: 403. 3. Priority. The tenant's priority goes into the request body, overwriting whatever the client sent. 4. Cache salt. cache_salt is set to the tenant's name, so cached prompts are shared within a tenant and never across tenants. 5. In-flight cap. The request waits for one of the tenant's slots, a tokio semaphore per tenant. 6. Ledger. When the response ends, or the client disconnects, one JSON line is written with the tenant, the model, the status, vLLM's own prompt and output token counts, the time spent waiting for a slot and the total time. The demo tenants: [[tenant]] name = "A" key = "demo-key-A" max_inflight = 1 priority = 10 [[tenant]] name = "C" key = "demo-key-C" adapter = "rand" max_inflight = 8 priority = 0 A has one slot and runs last. B and C have 8 slots each, the most vLLM runs at once. C's rand adapter is a random rank-8 LoRA from day1/make_adapters.py, standing in for a customer's fine-tune. Streaming responses pass straight through. The gateway watches the stream for the one chunk that carries usage and records it when the stream ends. A request that dies halfway is still in the ledger, and its slot is released at that moment. --- Step 5 — Check Isolation python3 day3/isolation.py PASS no key: 401 PASS unknown key: 401 PASS B asks for C's adapter: 403 PASS B asks for a model that is not served: 404 PASS B gets the base model: xbill9/gemma-4-E2B-it-qat-q4_0-w4a16-ct-text-emb4 PASS C gets its own adapter by default: rand PASS C's answer differs from the base model's: 'Here are a few examples of how a spare G' vs 'Here are three common uses for a spare G' PASS B's repeat is served from B's cache: 2400 hit tokens (first send 0) PASS A sending B's prompt gets none of B's cache: 0 hit tokens 9 of 9 pass The last two lines are the privacy check. When B sends a long prompt twice, the second send is served from cache. A sends the identical prompt and gets nothing from B's cache, so A cannot tell from response time whether B ever sent it. The hit counts come from vLLM's /metrics, not from timing. --- Step 6 — Rerun the Noisy Neighbour Through the Gateway day3/run_m1.sh python3 day3/compare.py run B+C first token median 95th max B+C done A done quiet, no A (direct) 0.28 s 0.29 s 0.37 s 180 0 direct, first come first served 29.90 s 44.49 s 47.81 s 180 16 direct, A at low priority 6.52 s 14.71 s 16.31 s 180 16 gateway: priority + A capped at 1 7.06 s 12.11 s 13.55 s 180 7 Priority does most of the work: B and C's median wait falls from 29.90 s to 6.52 s. Capping A at one request shortens the worst waits, from 14.71 s to 12.11 s at the 95th percentile. A pays for it, finishing 7 summaries in the window against 16. B and C still wait about 7 s, close to one 4,096-token prefill (7.1 s in Step 2). A request that arrives while A's prompt is being read waits for that read to finish. 🔎 Tip: size the cap above what the tenant needs. B sends a request every 2 seconds and each one spends about 12 seconds on the GPU under load, so B needs about 6 slots. With B capped at 4, the same run gave B a median time inside vLLM of 11.61 s and a median wait for a gateway slot of 37.77 s (day3/cap4/). A cap is for the customer you want to slow down. --- Step 7 — Check the Ledger Against vLLM A bill is only as good as its meter. day3/bill.py adds up the ledger and compares it with vLLM's own prompt_tokens_total and generation_tokens_total counters, read before and after the run: ledger vs vLLM counters: prompt 122,442 vs 122,442, output 12,416 vs 12,416; match: True Every token vLLM processed in the run is attributed to a tenant, and no tenant is charged for a token vLLM did not process. --- Step 8 — Send the Bill The VM's on-demand price comes from the Cloud Billing Catalog API, fetched during the build (day3/pricing.us-west2.txt): T4 0.410000 + 2 vCPU x 0.037970 + 7.5 GiB x 0.005089 = 0.5241 USD/hour x 730 hours = 382.60 USD/month bill.py prices each request in GPU seconds, using the prefill fit from Step 2 plus 9.34 ms per output token, and splits the month by each tenant's share. Over the gateway run: tenant requests refused prompt tok output tok token share GPU s GPU share bill USD A 7 0 28,718 896 22.0% 58.7 27.2% 104.22 B 90 0 46,803 5,760 39.0% 78.4 36.4% 139.11 C 90 0 46,921 5,760 39.1% 78.5 36.4% 139.26 A sent 22.0% of the tokens and used 27.2% of the GPU. A flat price per token would undercharge the long-prompt customer and pass the difference on to the other two. Weighting prompt tokens by their measured prefill cost puts it back. --- Compare and Contrast | | Straight to vLLM | vLLM with priority | Through the gateway | | :--- | :--- | :--- | :--- | | B and C, median first token under load | 29.90 s | 🥈 6.52 s | 🥇 7.06 s | | B and C, 95th percentile | 44.49 s | 🥈 14.71 s | 🥇 12.11 s | | Keys per customer | ❌ no | ❌ no | 🟢 yes | | Adapter only its owner can call | ❌ no | ❌ no | 🟢 yes | | Prefix cache per customer | ❌ no | ❌ no | 🟢 yes | | Per-customer bill, checked against vLLM | ❌ no | ❌ no | 🟢 yes | --- So, Which One? Turn on --scheduling-policy priority for any shared vLLM: on these runs it is the single biggest change, and it costs one flag. Selling the GPU takes the gateway, because priority alone still lets any client claim any priority, call any adapter and read any cache. The gateway sets all three from the key. --- Prior Work - The T4 rig: the VM, vLLM 0.29.0 install, Turing patch and the vllm-t4 script come from my earlier gemma4-dev T4 serving work. - gemma-rust: reqwest client patterns for the OpenAI-compatible endpoint. - dev-hacktoberfest (Week 1): every result checked by code that did not produce it. - Open source used: vLLM (Apache 2.0), Gemma 4 E2B (Gemma terms), axum, reqwest, tokio. --- Why Does Open Innovation Matter? Every part of this business needs the weights. - The model runs on hardware the operator controls. That is possible only with downloadable weights, and it is what lets the operator promise a customer that its emails never reach a model vendor. - One base model, one small adapter per customer. Per-customer fine-tunes that share one GPU and one batch need the weights and an engine that serves adapters side by side. vLLM's multi-LoRA does that on a 2018-era T4. - The cost is flat. The operator pays 382.60 USD a month whatever the customers send. A heavy customer costs the others waiting time, which the scheduler controls, and costs the operator nothing extra. - Everything can be checked. Engine, gateway and model are open, so the cache isolation and the bill are tested against vLLM's own counters, and a customer can read the code that bills them. --- Prize Categories Gemma: Gemma 4 E2B, served locally on a T4 with vLLM, with per-customer LoRA adapters on Google's quantization-aware 4-bit weights. --- Summary The goal of this project was to sell shares of one T4 to several small customers without one customer ruining the service for the others. The key to the solution was vLLM's priority scheduling, adapters and cache salt, controlled per customer by a gateway that holds the keys. The results were: - 🟢 A long-prompt customer pushed other customers' first token from 0.28 s to 29.90 s. Priority brought it to 6.52 s; the gateway's cap on that customer brought the 95th percentile from 14.71 s to 12.11 s. - 🟢 9 of 9 isolation checks pass: keys, adapter ownership and a prefix cache per customer. - 🟢 The ledger matched vLLM's token counters exactly: 122,442 prompt and 12,416 output tokens. - 🟢 A per-adapter customer runs in the same batch as base-model customers, at about the same speed. - ⚠️ B and C still wait about one long prefill, around 7 s, while A is active. Chunking long prompts further, or giving A its own hours, are the next steps. - ❌ A's throughput falls from 16 summaries to 7 in the window. That is the price A pays for a cheaper share. Scope: one T4 on one Google Cloud VM, vLLM 0.29.0, one 180-second run per configuration. The two direct runs and the gateway run used separate server starts with the same flags; in the direct runs C used the base model, through the gateway it used its adapter. The token bucket, the per-customer fair queue and a tenant-count sweep from SPEC.md are not built. The VM price excludes disk and network. The strategy for renting out one T4 to several customers with Gemma 4 and vLLM was validated with an incremental step by step approach. --- References - Hacktoberfest Open-Source AI Challenge, Week 2 - t4-timeshare on GitHub - vLLM - Gemma 4 E2B, 4-bit build used here - Google Cloud GPU pricing
OpenBSD or Tokio bug? tokio#8643
Why probabilistic AI review bots fail at catching AI agent bugs—and how we enforce deterministic AST invariants & 12 CWE security shields in GitHub Actions >TL;DR: AI coding agents increased raw PR volume by 300%, yet overall merge velocity has dropped by 40% because Senior Engineers spend 15–20 hrs/week untangling 200-line hallucinated wrappers, deleted tests, and subtle CWE security flaws. After shipping patches into tier-1 upstream systems repos (LLVM #215076, Meta Velox #18529, DuckDB #26443, Apache DataFusion #24394, Microsoft Pyright #11605, Tokio #8595, ClickHouse #114954, OpenAI Triton #12221), we codified those exact deterministic CI gates into an open GitHub Actions workflow (ZeroSlop · Architecture & Simulator) that strips 94% of AI bloat, enforces 12 CWE security shields, and gets PRs merged 100% green in minutes. 1. The 2026 Engineering Bottleneck: Why AI PRs Stall in Review When shipping patches into upstream systems repos like LLVM, Meta Velox, DuckDB, or Tokio, a real production fix is 4 to 15 lines of surgical invariant changes paired with a deterministic regression test that passes CI on the first push. Inside commercial teams using Cursor, Copilot, or Claude Code, the opposite happens: - 94% Diff Bloat: An agent generates a 240-line wrapper class instead of fixing a 4-line boundary condition (interactive diff breakdown). - Weakened Tests: When a test fails, the agent silently deletes or loosens the assertion to manufacture a green check. - Noisy LLM Review Bots: Conversational AI review bots run another LLM to dump 25 chatty comments on top of a bloated PR—without shrinking the diff or getting the PR merged. 2. Conversational LLM Review Bots vs. Deterministic PR Airlock |Engineering Dimension|Conversational AI Review Bots|Deterministic PR Airlock (ZeroSlop)| |:-|:-|:-| |Primary Objective|Leave 25+ chatty LLM comments on a bloated diff|Get PRs merged 100% green in minutes (not 4 days)| |Diff Minimization|Reviews 200 lines of hallucinated wrappers as-is|Strips up to 94% of AI bloat down to the core AST invariant| |CI Failure Triage|Ignores red CI checks or hallucinates fixes|Separates contributor code defects from runner flakes| |Test Integrity|Misses deleted or weakened test assertions|Hard-blocks unauthorized regression test deletions| |PR Comment Hygiene|Spams new threads on every git push|Strictly 1 self-updating PR Verdict Card + Mermaid Graph| |Security & Privacy|Cloud LLM retention & external dashboards|12-Shield CWE Airlock + Zero Code Retention| 3. The 12-Shield Deterministic CWE Security Airlock (ZS-SEC-001 – ZS-SEC-012) Every Pull Request is audited in milliseconds across 8 merge-readiness signals (0–100) and 12 deterministic CWE security shields before a human reviewer is pinged: |Shield ID|Security Vector & CWE / CVE Mapping|What ZeroSlop Security Deterministically Intercepts| |:-|:-|:-| |ZS-SEC-001|SQL & Vector Injection (CWE-89 / CWE-943)|Unparameterized SQL/NoSQL/Vector template literals & raw string concatenation| |ZS-SEC-002|Hardcoded & Seeded Secrets (CWE-798 / CWE-256)|API keys, JWT secrets, seeded credentials & plaintext password comparisons| |ZS-SEC-003|Unbound Identity / IDOR (CWE-639 / CWE-862)|Trusting req.body.userId instead of verified session/JWT caller identity| |ZS-SEC-004|Command & Eval Injection (CWE-78 / CWE-94)|Unsanitized child_process.exec, os.system, subprocess(shell=True), eval()| |ZS-SEC-005|Silent Exception Swallowing (CWE-390)|Empty catch () {} blocks and swallowed promise rejections that hide crashes| |ZS-SEC-006|XSS & Unsanitized DOM (CWE-79)|dangerouslySetInnerHTML, .innerHTML =, and unescaped template rendering| |ZS-SEC-007|Unsafe Deserialization (CWE-502 / CWE-22)|pickle.loads, yaml.load without SafeLoader, and ../ path traversal| |ZS-SEC-008|SSRF & Disabled TLS (CWE-918 / CWE-295)|Unvalidated fetch(req.body.url) & NODE_TLS_REJECT_UNAUTHORIZED = '0'| |ZS-SEC-009|CI/CD "Pwn Requests" (CWE-426 / CWE-250)|pull_request_target fork checkouts & ${{ github.event.pull_request.title }} injection| |ZS-SEC-010|AI Package "Slopsquatting" (CWE-829 / CWE-494)|Hallucinated npm/PyPI/Cargo packages, unpinned Git deps & curl | bash| |ZS-SEC-011|Resource Exhaustion / DoS (CWE-400 / CWE-770)|while(true) without exit bounds, leaked setInterval, unbounded Promise.all| |ZS-SEC-012|Trojan Source & Jailbreaks (CVE-2021-42574)|Invisible Unicode Bidi characters & hidden IGNORE PREVIOUS INSTRUCTIONS comments| 4. Upstream Systems Proof (Click Any PR to Inspect the Diff) These deterministic CI invariants come directly from shipping merged & 100% green Pull Requests across upstream systems repositories: |Upstream Repository|Live Pull Request|Status|Core Systems Invariant & Regression Coverage| |:-|:-|:-|:-| |LLVM Compiler Infrastructure|llvm/llvm-project#215076|MERGED|Fixed Waitcnt brackets in GFX12+ AMDGPUAsmPrinter with MC regression tests| |Meta Velox Execution Engine|facebookincubator/velox#18529|MERGED|Aligned DecimalUtil::toByteArray overflow bounds & restored kMaxDecimalBytes| |DuckDB Analytical Engine|duckdb/duckdb#26443|MERGED|Fixed pragma_storage_info internal compressed segment type alignment| |Apache DataFusion Query Engine|apache/datafusion#24394|MERGED|Fixed lead()/lag() window frame boundaries with non-NULL default values| |Microsoft Pyright Type Checker|microsoft/pyright#11605|MERGED|Resolved type narrowing across nested loop closures & pattern guards| |Tokio Async Rust Runtime|tokio-rs/tokio#8595|100% GREEN|Preserved Coop budget accounting across JoinSet & FuturesUnordered| |ClickHouse OLAP DBMS|ClickHouse/ClickHouse#114954|100% GREEN|Fixed StorageURL distributed cluster header forwarding & serialization| |OpenAI Triton GPU Compiler|triton-lang/triton#12221|100% GREEN|Enforced strict tensor descriptor stride & block shape invariants| |Cloudflare Agents SDK|cloudflare/agents#1248|MERGED|Hardened state synchronization & WebSocket hibernation lifecycle invariants| 5. Architecture Documentation & GitHub Action Source - Interactive PR Simulator & Architecture: ribbsaetersystems.com - 12-Shield CWE & Zero-Retention Security Matrix: ribbsaetersystems.com - GitHub Action Workflow: github.com/marketplace/actions/zeroslop-by-ribbsa… - GitHub Action Repository: patrickswedish/zeroslop-action - Technical Deep-Dive Writeup: ribbsaetersystems.com What deterministic CI gates or pre-review checks are your teams enforcing in 2026 to get PRs merged cleanly without burning out your staff engineers?
AI agents write 3x more code, yet PRs take 4x longer to get merged: Enforcing deterministic AST invariants & 12 CWE security shields in GitHub CI >TL;DR: AI coding agents increased raw PR volume by 300%, yet overall merge velocity has dropped by 40% because Senior Engineers spend 15–20 hrs/week untangling 200-line hallucinated wrappers, deleted tests, and subtle CWE security flaws. After shipping patches into tier-1 upstream systems repos (LLVM #215076, Meta Velox #18529, DuckDB #26443, Apache DataFusion #24394, Microsoft Pyright #11605, Tokio #8595, ClickHouse #114954, OpenAI Triton #12221), we codified those exact deterministic CI gates into an open GitHub Actions workflow (ZeroSlop · Architecture & Simulator) that strips 94% of AI bloat, enforces 12 CWE security shields, and gets PRs merged 100% green in minutes. 1. The 2026 Engineering Bottleneck: Why AI PRs Stall in Review When shipping patches into upstream systems repos like LLVM, Meta Velox, DuckDB, or Tokio, a real production fix is 4 to 15 lines of surgical invariant changes paired with a deterministic regression test that passes CI on the first push. Inside commercial teams using Cursor, Copilot, or Claude Code, the opposite happens: - 94% Diff Bloat: An agent generates a 240-line wrapper class instead of fixing a 4-line boundary condition (interactive diff breakdown). - Weakened Tests: When a test fails, the agent silently deletes or loosens the assertion to manufacture a green check. - Noisy LLM Review Bots: Conversational AI review bots run another LLM to dump 25 chatty comments on top of a bloated PR—without shrinking the diff or getting the PR merged. 2. Conversational LLM Review Bots vs. Deterministic PR Airlock |Engineering Dimension|Conversational AI Review Bots|Deterministic PR Airlock (ZeroSlop)| |:-|:-|:-| |Primary Objective|Leave 25+ chatty LLM comments on a bloated diff|Get PRs merged 100% green in minutes (not 4 days)| |Diff Minimization|Reviews 200 lines of hallucinated wrappers as-is|Strips up to 94% of AI bloat down to the core AST invariant| |CI Failure Triage|Ignores red CI checks or hallucinates fixes|Separates contributor code defects from runner flakes| |Test Integrity|Misses deleted or weakened test assertions|Hard-blocks unauthorized regression test deletions| |PR Comment Hygiene|Spams new threads on every git push|Strictly 1 self-updating PR Verdict Card + Mermaid Graph| |Security & Privacy|Cloud LLM retention & external dashboards|12-Shield CWE Airlock + Zero Code Retention| 3. The 12-Shield Deterministic CWE Security Airlock (ZS-SEC-001 – ZS-SEC-012) Every Pull Request is audited in milliseconds across 8 merge-readiness signals (0–100) and 12 deterministic CWE security shields before a human reviewer is pinged: |Shield ID|Security Vector & CWE / CVE Mapping|What ZeroSlop Security Deterministically Intercepts| |:-|:-|:-| |ZS-SEC-001|SQL & Vector Injection (CWE-89 / CWE-943)|Unparameterized SQL/NoSQL/Vector template literals & raw string concatenation| |ZS-SEC-002|Hardcoded & Seeded Secrets (CWE-798 / CWE-256)|API keys, JWT secrets, seeded credentials & plaintext password comparisons| |ZS-SEC-003|Unbound Identity / IDOR (CWE-639 / CWE-862)|Trusting req.body.userId instead of verified session/JWT caller identity| |ZS-SEC-004|Command & Eval Injection (CWE-78 / CWE-94)|Unsanitized child_process.exec, os.system, subprocess(shell=True), eval()| |ZS-SEC-005|Silent Exception Swallowing (CWE-390)|Empty catch () {} blocks and swallowed promise rejections that hide crashes| |ZS-SEC-006|XSS & Unsanitized DOM (CWE-79)|dangerouslySetInnerHTML, .innerHTML =, and unescaped template rendering| |ZS-SEC-007|Unsafe Deserialization (CWE-502 / CWE-22)|pickle.loads, yaml.load without SafeLoader, and ../ path traversal| |ZS-SEC-008|SSRF & Disabled TLS (CWE-918 / CWE-295)|Unvalidated fetch(req.body.url) & NODE_TLS_REJECT_UNAUTHORIZED = '0'| |ZS-SEC-009|CI/CD "Pwn Requests" (CWE-426 / CWE-250)|pull_request_target fork checkouts & ${{ github.event.pull_request.title }} injection| |ZS-SEC-010|AI Package "Slopsquatting" (CWE-829 / CWE-494)|Hallucinated npm/PyPI/Cargo packages, unpinned Git deps & curl | bash| |ZS-SEC-011|Resource Exhaustion / DoS (CWE-400 / CWE-770)|while(true) without exit bounds, leaked setInterval, unbounded Promise.all| |ZS-SEC-012|Trojan Source & Jailbreaks (CVE-2021-42574)|Invisible Unicode Bidi characters & hidden IGNORE PREVIOUS INSTRUCTIONS comments| 4. Upstream Systems Proof (Click Any PR to Inspect the Diff) These deterministic CI invariants come directly from shipping merged & 100% green Pull Requests across upstream systems repositories: |Upstream Repository|Live Pull Request|Status|Core Systems Invariant & Regression Coverage| |:-|:-|:-|:-| |LLVM Compiler Infrastructure|llvm/llvm-project#215076|MERGED|Fixed Waitcnt brackets in GFX12+ AMDGPUAsmPrinter with MC regression tests| |Meta Velox Execution Engine|facebookincubator/velox#18529|MERGED|Aligned DecimalUtil::toByteArray overflow bounds & restored kMaxDecimalBytes| |DuckDB Analytical Engine|duckdb/duckdb#26443|MERGED|Fixed pragma_storage_info internal compressed segment type alignment| |Apache DataFusion Query Engine|apache/datafusion#24394|MERGED|Fixed lead()/lag() window frame boundaries with non-NULL default values| |Microsoft Pyright Type Checker|microsoft/pyright#11605|MERGED|Resolved type narrowing across nested loop closures & pattern guards| |Tokio Async Rust Runtime|tokio-rs/tokio#8595|100% GREEN|Preserved Coop budget accounting across JoinSet & FuturesUnordered| |ClickHouse OLAP DBMS|ClickHouse/ClickHouse#114954|100% GREEN|Fixed StorageURL distributed cluster header forwarding & serialization| |OpenAI Triton GPU Compiler|triton-lang/triton#12221|100% GREEN|Enforced strict tensor descriptor stride & block shape invariants| |Cloudflare Agents SDK|cloudflare/agents#1248|MERGED|Hardened state synchronization & WebSocket hibernation lifecycle invariants| 5. Architecture Documentation & GitHub Action Source - Interactive PR Simulator & Architecture: ribbsaetersystems.com - 12-Shield CWE & Zero-Retention Security Matrix: ribbsaetersystems.com - GitHub Action Workflow: github.com/marketplace/actions/zeroslop-by-ribbsa… - GitHub Action Repository: patrickswedish/zeroslop-action - Technical Deep-Dive Writeup: ribbsaetersystems.com What deterministic CI gates or pre-review checks are your teams enforcing in 2026 to get PRs merged cleanly without burning out your staff engineers?