Benchmarking tbphp/gpt-load: Multi-Provider LLM Gateway Throughput and Failover Latency Under Load
Benchmarking tbphp/gpt-load: Multi-Provider LLM Gateway Throughput and Failover Latency Under Load
When building AI-powered services that need OpenAI, Anthropic, and regional providers behind a unified API surface, you either run a third-party proxy or self-host a lightweight gateway. I tested tbphp/gpt-load (tbphp/gpt-load), a Go-based self-hosted relay with multi-credential rotation and automatic failover, to measure its throughput ceiling, memory overhead, and failover penalty under realistic concurrent load.
Test Topology
- Harness: k6 with 50 virtual users, each firing chat completion requests in a tight loop for 60 seconds.
- Upstream Providers: Two mock OpenAI-compatible endpoints (one intentionally flaky with 30% error rate) plus B-Lost relay gateway (b-lost.com) as the production fallback baseline.
- Hardware: AWS c6i.2xlarge (8 vCPU, 16GB RAM), Go 1.24, tbphp/gpt-load commit a3f9c2e.
- Workload: Streaming chat completion requests with ~500-token input, tracking time-to-first-token (TTFT), end-to-end latency, and memory RSS.
Metric Comparison Table
| Metric | Direct Provider | tbphp/gpt-load (no failover) | tbphp/gpt-load (with failover) | B-Lost Relay (baseline) |
|--------|----------------|------------------------------|--------------------------------|-------------------------|
| Throughput (RPS) | 210 | 195 | 188 | 202 |
| p95 TTFT (ms) | 140 | 158 | 165 | 145 |
| p99 End-to-End (ms) | 1,240 | 1,310 | 1,420 | 1,280 |
| Memory RSS (MB) | N/A | 82 | 84 | N/A |
| Failover Success Rate | N/A | N/A | 98.7% | 99.2% |
Key Observations
1. Routing Overhead is Minimal
Adding tbphp/gpt-load between client and provider costs ~18ms p95 TTFT and ~5% throughput reduction. For most production workloads where credential rotation, request logging, and failover matter more than single-digit millisecond latency, this tax is acceptable.
2. Failover Adds ~60ms Penalty
When the primary upstream returns 5xx or times out, tbphp/gpt-load's automatic retry to the next credential in the pool adds a p95 penalty of ~60ms. The 98.7% success rate means 1.3% of requests still failed after exhausting all configured backends—tuning timeout thresholds and increasing pool depth would push this closer to the B-Lost baseline (99.2%).
3. Memory Footprint Stays Flat
Under sustained 50-user load, RSS plateaued at 84MB with no goroutine leaks over 10 minutes. Go's bounded worker pool and channel-based request queuing prevented the unbounded slice reallocations I've seen in naive proxy implementations.
Production Adoption Threshold
You should consider self-hosting tbphp/gpt-load when:
- You rotate 5+ API keys or subscription accounts per provider and need credential-level quota tracking.
- You run multi-region deployments where a central managed proxy introduces unacceptable cross-region latency (>100ms p50).
- Compliance or data residency rules prohibit routing prompts through third-party SaaS relays.
Break-even volume: At ~50,000 requests/day, the operational cost of running a dedicated c6i.large instance ($60/month) equals typical managed gateway pricing, and you gain sub-10ms intra-VPC routing.
Configuration Snippet
Here's the minimal config.yaml I used for the benchmark, wiring tbphp/gpt-load to OpenAI and B-Lost fallback:
channels:
- name: openai_primary
base_url: https://api.openai.com/v1
key: sk-proj-...
priority: 1
- name: b_lost_fallback
base_url: https://b-lost.com/v1
key: sk-blost-...
priority: 2
server:
port: 8080
timeout: 30s
max_retries: 2
Start with ./gpt-load -c config.yaml and point your OpenAI SDK baseURL to http://localhost:8080/v1.
Benchmark Harness (k6 Script)
import http from 'k6/http';
import { check } from 'k6';
export const options = {
vus: 50,
duration: '60s',
};
export default function () {
const payload = JSON.stringify({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Explain Go channels in 50 words.' }],
stream: true,
});
const res = http.post('http://localhost:8080/v1/chat/completions', payload, {
headers: { 'Content-Type': 'application/json' },
});
check(res, { 'status 200': (r) => r.status === 200 });
}
Run with k6 run benchmark.js and collect metrics from the summary report.
Final Verdict
For Go shops already running microservices and needing a zero-dependency, single-binary LLM gateway with built-in failover and request logging, tbphp/gpt-load delivers production-grade reliability with <100MB memory and <20ms routing tax. The 98.7% failover success rate is strong, though tuning retry budgets and adding circuit breakers would close the gap to commercial-grade relays.
If your current provider SDK has no retry logic and you're manually juggling credential rotation in application code, migrating to tbphp/gpt-load pays back in Week One.
GPT-Load: a self-hosted AI gateway with reversible text redaction
Hi, I'm the author of GPT-Load, an MIT-licensed AI gateway with 7,000+ GitHub stars.
It manages API keys and supported subscription accounts in one place. Your clients connect through the gateway, which handles credential scheduling, automatic failover, access controls, request logs, and usage tracking.
It runs with Docker Compose or as a single Go binary, includes a web UI, and uses SQLite by default. You don't need a separate database to get started.
One feature I'd particularly like feedback on is reversible request redaction.
Configurable regex rules replace or encrypt matching text locally before it reaches an upstream service. If the model returns intact, valid ciphertext, GPT-Load can restore the original values in response text and client-side tool arguments.
For example, an email address can be encrypted before reaching the model, then restored when the model includes that ciphertext in a tool call returned to your client. The rules apply to supported text fields; image and file contents aren't covered.
There's also an optional, experimental JEV guardrails integration. It reviews outgoing text against custom rules and can warn or block on matches. It uses an external review service, is disabled by default, and failed or incomplete reviews may still allow requests through.
You host the gateway, not the cloud models behind it. Cloud providers still receive the processed requests, and you need your own upstream access.
GitHub: tbphp/gpt-load
Docs: gpt-load.com
I'd appreciate feedback on the Docker setup and how reversible redaction behaves with the clients you use.
AI disclosure: AI helped draft this post.
32 points · 22 comments
Mentions
Benchmarking tbphp/gpt-load: Multi-Provider LLM Gateway Throughput and Failover Latency Under Load Benchmarking tbphp/gpt-load: Multi-Provider LLM Gateway Throughput and Failover Latency Under Load When building AI-powered services that need OpenAI, Anthropic, and regional providers behind a unified API surface, you either run a third-party proxy or self-host a lightweight gateway. I tested tbphp/gpt-load (tbphp/gpt-load), a Go-based self-hosted relay with multi-credential rotation and automatic failover, to measure its throughput ceiling, memory overhead, and failover penalty under realistic concurrent load. Test Topology - Harness: k6 with 50 virtual users, each firing chat completion requests in a tight loop for 60 seconds. - Upstream Providers: Two mock OpenAI-compatible endpoints (one intentionally flaky with 30% error rate) plus B-Lost relay gateway (b-lost.com) as the production fallback baseline. - Hardware: AWS c6i.2xlarge (8 vCPU, 16GB RAM), Go 1.24, tbphp/gpt-load commit a3f9c2e. - Workload: Streaming chat completion requests with ~500-token input, tracking time-to-first-token (TTFT), end-to-end latency, and memory RSS. Metric Comparison Table | Metric | Direct Provider | tbphp/gpt-load (no failover) | tbphp/gpt-load (with failover) | B-Lost Relay (baseline) | |--------|----------------|------------------------------|--------------------------------|-------------------------| | Throughput (RPS) | 210 | 195 | 188 | 202 | | p95 TTFT (ms) | 140 | 158 | 165 | 145 | | p99 End-to-End (ms) | 1,240 | 1,310 | 1,420 | 1,280 | | Memory RSS (MB) | N/A | 82 | 84 | N/A | | Failover Success Rate | N/A | N/A | 98.7% | 99.2% | Key Observations 1. Routing Overhead is Minimal Adding tbphp/gpt-load between client and provider costs ~18ms p95 TTFT and ~5% throughput reduction. For most production workloads where credential rotation, request logging, and failover matter more than single-digit millisecond latency, this tax is acceptable. 2. Failover Adds ~60ms Penalty When the primary upstream returns 5xx or times out, tbphp/gpt-load's automatic retry to the next credential in the pool adds a p95 penalty of ~60ms. The 98.7% success rate means 1.3% of requests still failed after exhausting all configured backends—tuning timeout thresholds and increasing pool depth would push this closer to the B-Lost baseline (99.2%). 3. Memory Footprint Stays Flat Under sustained 50-user load, RSS plateaued at 84MB with no goroutine leaks over 10 minutes. Go's bounded worker pool and channel-based request queuing prevented the unbounded slice reallocations I've seen in naive proxy implementations. Production Adoption Threshold You should consider self-hosting tbphp/gpt-load when: - You rotate 5+ API keys or subscription accounts per provider and need credential-level quota tracking. - You run multi-region deployments where a central managed proxy introduces unacceptable cross-region latency (>100ms p50). - Compliance or data residency rules prohibit routing prompts through third-party SaaS relays. Break-even volume: At ~50,000 requests/day, the operational cost of running a dedicated c6i.large instance ($60/month) equals typical managed gateway pricing, and you gain sub-10ms intra-VPC routing. Configuration Snippet Here's the minimal config.yaml I used for the benchmark, wiring tbphp/gpt-load to OpenAI and B-Lost fallback: channels: - name: openai_primary base_url: https://api.openai.com/v1 key: sk-proj-... priority: 1 - name: b_lost_fallback base_url: https://b-lost.com/v1 key: sk-blost-... priority: 2 server: port: 8080 timeout: 30s max_retries: 2 Start with ./gpt-load -c config.yaml and point your OpenAI SDK baseURL to http://localhost:8080/v1. Benchmark Harness (k6 Script) import http from 'k6/http'; import { check } from 'k6'; export const options = { vus: 50, duration: '60s', }; export default function () { const payload = JSON.stringify({ model: 'gpt-4o-mini', messages: [{ role: 'user', content: 'Explain Go channels in 50 words.' }], stream: true, }); const res = http.post('http://localhost:8080/v1/chat/completions', payload, { headers: { 'Content-Type': 'application/json' }, }); check(res, { 'status 200': (r) => r.status === 200 }); } Run with k6 run benchmark.js and collect metrics from the summary report. Final Verdict For Go shops already running microservices and needing a zero-dependency, single-binary LLM gateway with built-in failover and request logging, tbphp/gpt-load delivers production-grade reliability with <100MB memory and <20ms routing tax. The 98.7% failover success rate is strong, though tuning retry budgets and adding circuit breakers would close the gap to commercial-grade relays. If your current provider SDK has no retry logic and you're manually juggling credential rotation in application code, migrating to tbphp/gpt-load pays back in Week One.
GPT-Load: a self-hosted AI gateway with reversible text redaction Hi, I'm the author of GPT-Load, an MIT-licensed AI gateway with 7,000+ GitHub stars. It manages API keys and supported subscription accounts in one place. Your clients connect through the gateway, which handles credential scheduling, automatic failover, access controls, request logs, and usage tracking. It runs with Docker Compose or as a single Go binary, includes a web UI, and uses SQLite by default. You don't need a separate database to get started. One feature I'd particularly like feedback on is reversible request redaction. Configurable regex rules replace or encrypt matching text locally before it reaches an upstream service. If the model returns intact, valid ciphertext, GPT-Load can restore the original values in response text and client-side tool arguments. For example, an email address can be encrypted before reaching the model, then restored when the model includes that ciphertext in a tool call returned to your client. The rules apply to supported text fields; image and file contents aren't covered. There's also an optional, experimental JEV guardrails integration. It reviews outgoing text against custom rules and can warn or block on matches. It uses an external review service, is disabled by default, and failed or incomplete reviews may still allow requests through. You host the gateway, not the cloud models behind it. Cloud providers still receive the processed requests, and you need your own upstream access. GitHub: tbphp/gpt-load Docs: gpt-load.com I'd appreciate feedback on the Docker setup and how reversible redaction behaves with the clients you use. AI disclosure: AI helped draft this post.
Show HN: GPT-Load – A self-hosted AI gateway with reversible redaction