concurrency-limits

Building an In-Memory Circuit Breaker for Webhook Ingress: Shedding Load Without Losing Events Webhooks connect cloud systems. Payment providers like Stripe, code hosts like GitHub and commerce platforms like Shopify all use them to tell your services that something changed. When a sender has a bad day, you pay for it. A bulk export, a backfill or a burst of retries can hit your ingress endpoint, and the endpoint is the first thing that fails. When a burst arrives, worker threads or goroutines block on slow downstream calls and connection pools saturate. Memory fills with queued request bodies and garbage-collection pressure pushes latency higher. Health checks start failing, the load balancer moves traffic to the remaining nodes, and those nodes tip over too. This guide describes an in-memory admission controller for webhook ingress. It combines an adaptive concurrency limit, a CoDel-style delay signal and priority-aware shedding. It also covers what “not losing events” means when you reject a request, because that depends on how each sender retries. 1. How an ingress meltdown unfolds Webhook senders enforce short response deadlines, and each provider sets its own: | Sender | Documented behavior | | --- | --- | | GitHub | Expects a 2XX within 10 seconds, otherwise it terminates the connection and counts a failed delivery. It does not redeliver automatically; failed deliveries must be redelivered manually or via the API. | | Shopify | One-second connection timeout and five-second timeout for the whole request. Failed deliveries are retried 8 times over 4 hours. | | Slack (Events API) | Expects a valid response within three seconds. | | Stripe (live mode) | Retries for up to 3 days with exponential backoff. | Because the deadlines are short, a slow endpoint produces timeouts, and timeouts produce more traffic. A sender that retries adds those retries to the stream of fresh events. A sender that doesn’t retry, like GitHub, leaves you with lost events and manual redelivery. The failure then plays out in a predictable order: 1. Connection saturation: new connections queue in the socket backlog. 2. Worker starvation: handlers blocked on database calls hold their threads or goroutines. 3. Memory growth: request bodies pile up while waiting for a worker. 4. Latency amplification: garbage-collection work and contention raise latency, which causes more sender timeouts. [Upstream Sender] ──(flood / retries)──> [Load Balancer] │ ▼ [Ingress Worker Node] ├── Connection queue (full) ├── Workers (blocked) └── Memory / GC (pressure) ──> [meltdown] The first defense is architectural. GitHub’s documentation recommends putting a queue in front of slow work so the server can respond immediately and process the payload in the background. Shopify gives the same advice for bursts. Admission control is the second line of defense. It protects the thin “validate, enqueue, acknowledge” path itself. 2. Why the admission decision should be local Resilience4j’s circuit breaker is the standard tool for outbound calls. It is an in-process library and doesn’t need Redis or any other shared store. The same reasoning applies at ingress: - No extra dependency on the hot path. Consulting a shared store for every inbound request adds a network round trip and a new failure mode. If that store is down, your ingress should still be able to shed load. - The overload is local. Thread pools, memory and database connections are per-node resources. A node-local signal reflects what that node can actually handle. - Senders only see HTTP. They react to status codes and timing, so the cleanest signal you can give them is a fast, explicit rejection. The trade-off is that each node decides independently. With uneven load balancing, one node may shed while another has spare capacity. Per-node metrics (section 7) show you when that happens. 3. Two mechanisms: adaptive concurrency and queue delay Adaptive concurrency limits Little’s Law says that concurrency equals throughput times latency. A fixed limit such as “100 concurrent requests” goes stale as soon as latency changes. If the database slows down, 100 is too high. If it speeds up, 100 is too low. Netflix’s open-source concurrency-limits library, described in their 2018 post “Performance Under Load”, adjusts the limit from observed latency instead. Its gradient algorithm uses: newLimit = gradient × currentLimit + queueSize In the library’s Gradient2 variant, the gradient is the long-term RTT divided by the current RTT, clamped to the range 0.5 to 1.0. The result is then smoothed (default factor 0.2): gradient = max(0.5, min(1.0, longTermRTT / currentRTT)) newLimit = gradient × currentLimit + queueSize newLimit = currentLimit × (1 − smoothing) + newLimit × smoothing When latency rises above its long-term average, the gradient drops below 1 and the limit shrinks. When latency is stable, the queueSize term lets the limit creep upward. It can be a fixed value or a function of the limit, such as a square root. Queue delay (CoDel-style) Counting 5xx errors reacts late, because by the time errors appear the damage is done. A better early signal is how long requests wait for a slot. CoDel (Controlled Delay) was designed by Kathleen Nichols and Van Jacobson as an active queue management algorithm for network buffers (RFC 8289). It tracks a packet’s sojourn time, the time spent in the queue. It acts only when the minimum delay over a sliding window stays above a target, which allows short bursts but prevents a persistent standing queue. RFC 8289’s defaults are a 5 ms target and a 100 ms interval. Facebook adapted a variant of CoDel for request queues, as described in “Fail at Scale”. Its open-source Wangle library shows the idea. Within each interval it tracks the minimum queueing delay and declares overload if that minimum exceeded the target. While overloaded, it sheds requests that have waited longer than twice the target. This is an adaptation rather than textbook CoDel. Treat 5 ms and 100 ms as starting points and tune them against your handler’s normal service time. 4. Architecture Requests pass through classification and then a limiter that both enforces the adaptive limit and measures queue delay: Incoming webhook request │ ▼ ┌──────────────────────────────────────────┐ │ 1. Priority classifier │ │ (route or verified partner identity) │ └──────────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────┐ │ 2. Admission: slot free? │ │ yes → admit; no → queue briefly, or │ │ shed if overloaded (by priority) │ └──────────────────────────────────────────┘ │ (slot acquired) ▼ ┌──────────────────────────────────────────┐ │ 3. Thin handler: verify signature, │ │ enqueue durably, return 2xx │ └──────────────────────────────────────────┘ │ ▼ Release slot → feed RTT back to the limit Classify by something you control, such as the endpoint path or the partner identified after signature verification. Don’t trust a priority header that arrives from the public internet, because anyone can set it. 5. A working Go sketch The following sketch uses only the standard library. It combines a Gradient2-style limit with a Wangle-style overload flag. The parameter values are illustrative and need tuning against your own traffic. package ingress import ( "math" "net/http" "sync" "time" ) type Priority int const ( PriorityLow Priority = iota PriorityDefault PriorityHigh ) type Config struct { InitialLimit float64 // e.g. 20 MinLimit float64 // e.g. 4 MaxLimit float64 // e.g. 200 Smoothing float64 // e.g. 0.2 TargetDelay time.Duration // e.g. 5ms Interval time.Duration // e.g. 100ms MaxWait time.Duration // longest a request may wait for a slot, e.g. 2 * TargetDelay MaxQueue int // hard cap on waiting requests } type waiter struct { ready chan struct{} granted bool } type Limiter struct { cfg Config mu sync.Mutex limit float64 inflight int waiters []*waiter shortRTT float64 // seconds, fast EWMA longRTT float64 // seconds, slow EWMA intervalEnd time.Time minDelay time.Duration overloaded bool } func NewLimiter(cfg Config) *Limiter { return &Limiter{cfg: cfg, limit: cfg.InitialLimit} } // Acquire returns a release func and true if the request is admitted. func (l *Limiter) Acquire(p Priority) (func(time.Duration), bool) { l.mu.Lock() if l.inflight < int(l.limit) && len(l.waiters) == 0 { l.inflight++ l.observeDelay(0) l.mu.Unlock() return l.release, true } if l.shouldShed(p) { l.mu.Unlock() return nil, false } w := &waiter{ready: make(chan struct{})} l.waiters = append(l.waiters, w) l.mu.Unlock() start := time.Now() timer := time.NewTimer(l.cfg.MaxWait) defer timer.Stop() timedOut := false select { case <-w.ready: case <-timer.C: timedOut = true } l.mu.Lock() defer l.mu.Unlock() l.observeDelay(time.Since(start)) if timedOut && !w.granted { l.removeWaiter(w) return nil, false } return l.release, true } func (l *Limiter) shouldShed(p Priority) bool { switch p { case PriorityHigh: return len(l.waiters) >= l.cfg.MaxQueue // only when the queue is full case PriorityDefault: return l.overloaded || len(l.waiters) >= l.cfg.MaxQueue default: // PriorityLow: no queueing at all once slots are exhausted return true } } // observeDelay tracks the minimum queueing delay per interval (CoDel-style). func (l *Limiter) observeDelay(d time.Duration) { now := time.Now() if now.After(l.intervalEnd) { l.overloaded = l.minDelay > l.cfg.TargetDelay l.minDelay = d l.intervalEnd = now.Add(l.cfg.Interval) return } if d < l.minDelay { l.minDelay = d } } func (l *Limiter) release(rtt time.Duration) { l.mu.Lock() defer l.mu.Unlock() l.inflight-- l.updateLimit(rtt.Seconds()) for len(l.waiters) > 0 && l.inflight < int(l.limit) { w := l.waiters[0] l.waiters = l.waiters[1:] w.granted = true l.inflight++ close(w.ready) } } // updateLimit is a simplified Gradient2-style update. func (l *Limiter) updateLimit(rtt float64) { if rtt <= 0 { return } if l.longRTT == 0 { l.longRTT, l.shortRTT = rtt, rtt } l.shortRTT += 0.2 * (rtt - l.shortRTT) l.longRTT += 0.01 * (rtt - l.longRTT) gradient := math.Max(0.5, math.Min(1.0, l.longRTT/l.shortRTT)) next := gradient*l.limit + math.Sqrt(l.limit) next = l.limit*(1-l.cfg.Smoothing) + next*l.cfg.Smoothing l.limit = math.Max(l.cfg.MinLimit, math.Min(l.cfg.MaxLimit, next)) } func (l *Limiter) removeWaiter(w *waiter) { for i, x := range l.waiters { if x == w { l.waiters = append(l.waiters[:i], l.waiters[i+1:]...) return } } } // Guard wraps a webhook handler with admission control. func Guard(l *Limiter, classify func(*http.Request) Priority, next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { release, ok := l.Acquire(classify(r)) if !ok { w.Header().Set("Retry-After", "10") http.Error(w, "service temporarily overloaded", http.StatusServiceUnavailable) return } start := time.Now() defer func() { release(time.Since(start)) }() next.ServeHTTP(w, r) }) } Some limits of this sketch: - Waiters are served in FIFO order. Facebook’s design switches to LIFO once a queue forms, so the newest requests, which are most likely to still meet their sender’s deadline, go first. - RTT is measured for every request. In production you would typically exclude requests that failed fast. - A single mutex is fine at moderate rates. Profile before optimizing it. 6. Rejecting requests without losing events Whether a rejected webhook is lost depends on the sender, so you need to know how each one behaves: - Senders with retries: Stripe retries with exponential backoff for up to 3 days in live mode. Shopify retries 8 times over 4 hours. - Senders without automatic retries: GitHub does not redeliver automatically. If you shed a GitHub delivery, you need to redeliver it manually or through the API once you recover. - Subscription risk: Shopify deletes a webhook subscription created through the Admin API after 8 consecutive failures, and Stripe emails you and eventually disables an endpoint that fails for multiple days. A long shedding episode can cost you the subscription itself. For these reasons, shedding should be short, targeted and backed by a reconciliation job. Shopify itself recommends building one if you can’t afford to miss webhooks. The job periodically fetches data you may have missed through the provider’s API. Status codes - 503 Service Unavailable with Retry-After: the standard way to signal temporary overload. RFC 9110 defines Retry-After as either a number of seconds or an HTTP date. - 429 Too Many Requests: RFC 6585 defines it for rate violations by a particular sender. It also accepts Retry-After. - Don’t just drop connections. An abrupt reset gives the sender no information, and it will treat the delivery as a failed attempt. A clean 503 tells it explicitly that the failure is temporary. Do senders actually honor Retry-After? Some do. Webhook infrastructure vendors such as Hookdeck and Kit document that they respect Retry-After on 429 and 503, and AWS EventBridge API destinations are documented to as well. Svix advises that good senders will respect it when you return a 429. Stripe’s public documentation describes exponential backoff but doesn’t mention Retry-After, so check its behavior in your delivery logs rather than assuming. Because support varies, treat the header as a hint and rely on the sender’s own backoff as the fallback. Add jitter to the values you return, for example 5 to 15 seconds instead of a constant 10. Otherwise senders that honor the header will retry in lockstep. 7. Priority-based shedding Not all webhooks are equally urgent. A payment confirmation matters more than a bulk analytics sync. Classify by endpoint or authenticated partner, then apply different behavior per tier. The sketch above implements this policy: | Tier | Examples | Behavior in the sketch | | --- | --- | --- | | PriorityHigh | Payment confirmations, security alerts | Queue for a slot; shed only when the wait queue is full. | | PriorityDefault | Standard operational updates | Queue, but shed once the CoDel-style overload flag is set. | | PriorityLow | Bulk syncs, analytics, non-essential notifications | Shed immediately when no slot is free. | This is a policy choice rather than a standard. Choose tiers based on which events you can safely replay later. 8. Observability Since each node decides for itself, you need per-node metrics. Export them to Prometheus or OpenTelemetry: 1. webhook_ingress_concurrency_limit: gauge for the current adaptive limit. 2. webhook_ingress_inflight_requests: gauge for requests being processed. 3. webhook_ingress_shed_total{priority}: counter of rejected requests. 4. webhook_ingress_queue_delay_seconds: histogram of time spent waiting for a slot. Use millisecond-scale buckets, since the targets are in the single-digit-millisecond range. 5. webhook_ingress_overloaded: gauge (0 or 1) for the overload flag. Alert on a sustained rise in shed counts or on queue delay staying above target. Both mean you should look at downstream capacity, not just the limiter. 9. Checklist - Acknowledge fast: Verify the signature, enqueue durably and return 2xx. Do the real work asynchronously. - Keep admission state local: Avoid a shared-store call on every inbound request. - Use a latency-driven limit: An adaptive, gradient-style limit beats a fixed number. - Watch queue delay, not just errors: Use a CoDel-style minimum-delay signal, and tune the target to your handler. - Reject explicitly: Return 503 (or 429 for per-sender limits) with a jittered Retry-After. - Know your senders: Document each provider’s timeout, retry window and subscription-deletion rules. - Reconcile: Run a job that backfills events missed during shedding, and redeliver GitHub deliveries manually. - Classify safely: Derive priority from routes or verified identity, never from unauthenticated headers. References - Stripe webhooks documentation: <docs.stripe.com> - GitHub, “Best practices for using webhooks”: <docs.github.com> - Shopify, “Deliver webhooks through HTTPS”: <shopify.dev> - Slack Events API response deadline: <vincentschmalbach.com> - Netflix Technology Blog, “Performance Under Load”: <netflixtechblog.medium.com> - Netflix concurrency-limits library: <Netflix/concurrency-limits> - RFC 8289, “Controlled Delay Active Queue Management”: <datatracker.ietf.org> - CoDel pseudocode (ACM Queue): <queue.acm.org> - Facebook Wangle CoDel implementation: <github.com/facebook/wangle/blob/master/wangle/con…> - Svix, “Webhook retry strategies”: <svix.com> - Hookdeck, “Webhook Delivery Guarantees”: <hookdeck.com> --- Originally published at instawebhook.com

LLM API Slows to 30s, Your Server Holds 600 Requests: Little's Law and a Load Shedder - Book: System Design Pocket Guide: Fundamentals - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Your LLM provider is up. Every call just takes 30 seconds instead of one. Nothing errors. Your health check is green. And your Node process is slowly filling up with requests that are all waiting on the same slow upstream. The loud failures make the status pages. DeepSeek logged an incident at 8:11 Beijing time on October 15, and chat and API users got "the server was temporarily unavailable" until a fix moved to monitoring at 12:49, per TechNode. Five days later OpenAI posted authentication errors for some users near Illinois, API included, for an hour, fixed by rolling back a configuration change. The slow afternoon usually never gets an entry. You can predict exactly how full your process gets with one line of arithmetic, and you can stop it with a counter and a fast 503, checked against a simulation you can run. The arithmetic: L = λW Little's Law, proved by John Little in 1961, says that in a stable system the average number of items inside (L) equals the arrival rate (λ) times the average time each one spends inside (W). It does not care about the arrival distribution or the order things are served in. That is why it is so useful for back-of-the-envelope capacity work. Apply it to an endpoint that proxies an LLM call: - Normal day: 20 requests per second, 1 second per call. L = 20 × 1 = 20 requests in flight. - Upstream slows down: same 20 requests per second, 30 seconds per call. L = 20 × 30 = 600 requests in flight. Your traffic did not change. Your code did not change. Your concurrency went up 30x because W did. Each of those 600 requests holds a socket to the client, a socket to the provider, the prompt, the buffered response, and a promise chain in memory. Users rarely wait 30 seconds either. Some hit refresh, which raises λ while W stays high. Netflix's concurrency-limits library builds on the same identity: limit = average RPS × average latency. Flip it around and you get the two levers you have. You can cap W (timeouts) or you can cap L directly (a concurrency limit). λ belongs to your users. !A pipe of identical request tokens backing up behind a narrow, slow valve at the far end When W goes from 1 s to 30 s, the pipe holds 30 times as much, with no extra traffic. A simulation you can run Real load tests against a slow LLM are expensive and hard to reproduce. A deterministic, tick-based simulation is neither. It steps time forward 100 ms at a time, sends 20 requests per second, and makes the upstream take 30 seconds per call between second 30 and second 90. It also needs a way to die. Here the server has a 400 MB budget and each in-flight request holds 1 MB. Both are assumptions, picked to make the failure visible; measure your own. When the budget is exceeded, the process crashes, loses everything it was holding, and takes 10 seconds to come back. Save the file as sim.ts and run it with tsx; there are no dependencies. // sim.ts: run with `npx tsx sim.ts` const TICK = 100; // ms per simulation step const END = 180_000; // simulate three minutes const RPS = 20; // requests arriving per second const PATIENCE = 15_000; // a user gives up after 15 s const TARGET = 2_000; // "healthy" upstream latency const HEAP_MB = 400; // memory budget (assumption) const MB_PER_REQ = 1; // held per in-flight request const RESTART = 10_000; // crash-to-ready time // Fast, then 30 s per call for one minute, then fast. const upstream = (t: number) => t >= 30_000 && t < 90_000 ? 30_000 : 1_000; type Policy = { name: string; timeout: number; // Infinity = wait forever limit: () => number; // Infinity = no cap onSlow?: () => void; // a call passed TARGET onFast?: (inFlight: number) => void; }; type Req = { start: number; end: number; slow: boolean }; Each tick of run() does three things: settle what is in flight, admit new arrivals, and check the memory budget. Append it to the same sim.ts. function run(p: Policy) { let inFlight: Req[] = []; let downUntil = 0; let due = 0; // fractional arrivals carried between ticks const s = { ok: 0, late: 0, timedOut: 0, shed: 0, lost: 0, crashes: 0, peak: 0 }; for (let t = 0; t < END; t += TICK) { // finish, time out, or flag what is in flight const still: Req[] = []; for (const r of inFlight) { if (r.end <= t) { const took = r.end - r.start; if (took <= PATIENCE) s.ok++; else s.late++; if (took <= TARGET) p.onFast?.(inFlight.length); } else if (t - r.start >= p.timeout) { s.timedOut++; } else { if (!r.slow && t - r.start > TARGET) { r.slow = true; p.onSlow?.(); } still.push(r); } } inFlight = still; // new arrivals this tick (exact for any RPS) due += (RPS * TICK) / 1000; for (; due >= 1; due--) { if (t < downUntil) { s.lost++; continue; } if (inFlight.length >= p.limit()) { s.shed++; continue; } inFlight.push({ start: t, end: t + upstream(t), slow: false }); } s.peak = Math.max(s.peak, inFlight.length); // out of memory: the process dies with its requests if (inFlight.length * MB_PER_REQ > HEAP_MB) { s.crashes++; s.lost += inFlight.length; inFlight = []; downUntil = t + RESTART; } } return s; } The counters mean: ok answered within the user's patience, late answered after the user left (you paid for those tokens and nobody read them), timedOut your server gave up on, shed rejected instantly, lost dropped by a crash or refused while restarting. Three policies that fail differently Start with what a naive proxy does: no limit, no timeout. Then add a timeout, then a fixed limit. const unbounded: Policy = { name: "unbounded", timeout: Infinity, limit: () => Infinity, }; const timeoutOnly: Policy = { name: "timeout only", timeout: PATIENCE, limit: () => Infinity, }; const fixed50: Policy = { name: "limit 50 + timeout", timeout: PATIENCE, limit: () => 50, }; You can predict the unbounded run before you execute it. Little's Law says the in-flight count heads toward 600, but the budget is 400, so the process dies about 20 seconds into the slowdown (20 per second × 20 seconds = 400). It restarts empty, refills at the same rate, and runs into the same wall. Every crash throws away hundreds of requests that had each been waiting for many seconds. The timeout-only run caps W at 15 seconds, so L tops out near 20 × 15 = 300, under the budget. No crash. That is a real improvement, and it shows that a timeout is the cheapest fix there is. The catch is in timedOut: during the slow minute, every single user waits 15 seconds to get an error. And the cap only holds while λ holds. Set RPS = 40, the refresh-button scenario, and 40 × 15 = 600 puts you back over the budget. The fixed limit caps L at 50 no matter what λ does. Most users in the slow minute land in shed: they get a 503 right away instead of a 15-second spinner. The AWS Builders' Library puts the reason plainly: "Early rejection is important because it's the cheapest place to drop excess traffic, but it comes at a cost to visibility." That second half is why you count and graph every shed request. The weakness is the number 50. You picked it for one latency, and the upstream decides what latency you get. !A gatekeeper valve letting a few request tokens through to a slow upstream and bouncing the rest straight back A concurrency limit turns a slow upstream into fast rejections instead of a growing queue. Shedding in a real Node server Here is the same policy in an HTTP handler. The rejection carries Retry-After, which on a 503 tells the client how long the service is expected to be unavailable, as a number of seconds or a date. import { createServer } from "node:http"; const LIMIT = 50; const TIMEOUT_MS = 15_000; let inFlight = 0; createServer(async (req, res) => { if (inFlight >= LIMIT) { res.writeHead(503, { "Retry-After": "5" }); res.end("busy, retry shortly\n"); return; } inFlight++; try { const up = await fetch(process.env.UPSTREAM_URL!, { signal: AbortSignal.timeout(TIMEOUT_MS), }); res.writeHead(up.status); res.end(await up.text()); } catch { res.writeHead(504); res.end("upstream failed or timed out\n"); } finally { inFlight--; } }).listen(3000); The check runs before any work: no body parsing, no auth lookup, no upstream socket. The finally is the part people get wrong; if the counter does not come down on every path, the limit slowly leaks until the server sheds everything. On the client side, honour the header and add jitter, or 50 rejected clients all retry in the same second. Google's SRE book describes the client half too: a per-client retry budget where a request is retried only while retries stay below 10% of traffic. AIMD: let the upstream pick the limit Additive increase, multiplicative decrease is the feedback rule from TCP congestion control: grow slowly while things are fine, cut sharply on a sign of congestion. Netflix's AIMDLimit does this for concurrency: multiply the limit by 0.9 on a drop or a call slower than its timeout, add 1 on success, but only grow when at least half the current limit is in use. function aimd(min = 5, max = 200): Policy { let limit = 20; return { name: "aimd + timeout", timeout: PATIENCE, limit: () => Math.floor(limit), onSlow: () => { limit = Math.max(min, limit * 0.9); }, onFast: (inFlight) => { if (inFlight * 2 >= limit) { limit = Math.min(max, limit + 1); } }, }; } const policies = [unbounded, timeoutOnly, fixed50, aimd()]; console.table( policies.map((p) => ({ policy: p.name, ...run(p) })), ); The main change from Netflix's version: the sim flags a call as slow the moment it passes TARGET, not when it finally returns. With a 30-second upstream, a completion-based signal arrives 30 seconds after the trouble started. Two seconds in is when you want to know. Run all four and compare the shed and timedOut columns for the last two rows. During the slow minute the AIMD limit drops toward its floor of 5, so far fewer users sit through a 15-second wait for an error, and the in-flight count stays tiny. When the upstream recovers, fast completions push the limit back up by one each, and the "half in use" guard stops it from inflating while traffic is light. TARGET is the knob that carries your judgement. At 2 seconds, an upstream that settles at 10 seconds would also be treated as broken, even though a patient user could still be served. Set it from your own latency data, not from this post. What to take to your service Do the arithmetic for your worst endpoint first: peak RPS times the worst latency your provider has shown you. If that number of in-flight requests would hurt, you need a timeout today and a concurrency limit that answers with a fast 503 and Retry-After. AIMD is the next step once you are tired of guessing the right number. Then change upstream, RPS and HEAP_MB in the sim to match your own service and see which row you are living in. --- If this was useful Little's Law, timeouts, load shedding and backpressure are the parts of system design that only feel abstract until a provider has a slow afternoon. The System Design Pocket Guide: Fundamentals covers those building blocks, with the capacity arithmetic that tells you where your service breaks before your users find it. 

Adaptive Concurrency Limits in Go: Dynamically Protecting Dependencies Under Load Most services eventually need a concurrency limit. You might start with something simple: const maxConcurrentRequests = 100 A semaphore ensures that no more than 100 requests execute simultaneously. It works. Until the conditions change. Your service gets more CPU resources after scaling. A downstream database becomes slower. A new deployment changes request processing time. Traffic patterns shift throughout the day. The limit that worked yesterday might be too restrictive today. Or worse, it might be too high and allow a dependency to become overloaded. Static concurrency limits are easy to implement, but they require continuous tuning. What if a service could adjust its concurrency limit automatically based on how the dependency behaves? That's the idea behind adaptive concurrency limiting. Instead of relying entirely on a predefined number, the system observes request latency and failure signals, estimates when queueing is happening, and adjusts the number of requests it allows in flight. In this article, we'll explore: - Why fixed concurrency limits eventually become difficult to maintain - The relationship between concurrency, throughput, and latency - How TCP congestion control inspired adaptive concurrency algorithms - How to implement a latency-aware concurrency limiter in Go - How to integrate it into an HTTP client - How adaptive limits interact with backpressure, bulkheads, and retry budgets - What to monitor and test before using adaptive concurrency in production 1. The Problem with Fixed Concurrency Limits Imagine a Go service that calls an external market data API. You initially configure a concurrency limit of 50. limiter := NewFixedLimiter(50) Under normal traffic, the dependency responds in 40ms. Everything looks healthy. Then traffic increases. The dependency starts processing requests more slowly. Its response time grows to 150ms, then 300ms. Your service still allows 50 concurrent requests. The problem is that concurrency and latency are connected. When requests take longer to complete, existing requests occupy concurrency slots for longer. New requests accumulate, queues grow, and latency increases further. A fixed limit cannot automatically distinguish between: - A dependency that has additional capacity - A dependency approaching saturation - A dependency already experiencing queueing - A dependency recovering from an overload You can increase the limit manually, but that risks overloading the dependency. You can decrease it, but that might unnecessarily reduce throughput. The challenge is finding a concurrency level that allows high throughput without creating excessive queueing. 2. Why Concurrency Matters More Than RPS Alone Rate limiting and concurrency limiting solve different problems. A rate limiter controls how many requests can start during a period. A concurrency limiter controls how many requests can be in progress simultaneously. Consider two situations: | Scenario | Request rate | Average latency | Approx. concurrency | |---|---:|---:|---:| | Healthy dependency | 500 req/s | 40ms | 20 | | Slow dependency | 500 req/s | 200ms | 100 | Both scenarios have the same incoming request rate. But the second requires five times as many concurrent requests to sustain that throughput. This relationship is described by Little's Law: [ L = \lambda W ] Where: - (L) is the average number of items in the system - (\lambda) is the average arrival rate - (W) is the average time spent in the system For request processing: [ Concurrency \approx Throughput \times Latency ] This explains why a fixed RPS limit is not always sufficient to protect a dependency. A service can receive an acceptable number of requests per second while still accumulating too many in-flight operations because response times have increased. Adaptive concurrency focuses on this relationship. 3. The Idea Behind Adaptive Concurrency The central idea is straightforward: 1. Measure request latency. 2. Maintain an estimate of normal, unloaded latency. 3. Compare current latency against that baseline. 4. Increase concurrency when the dependency has spare capacity. 5. Reduce concurrency when latency indicates queueing or failures. This resembles congestion control in TCP. TCP does not blindly send unlimited data. It maintains a congestion window and adjusts how much data can be in flight based on network conditions. Adaptive concurrency applies a similar idea to service requests. The concurrency limit becomes a dynamic window. <figure> <img src="dummyimage.com" alt="Adaptive concurrency control loop" /> </figure> There are several algorithms worth understanding. AIMD Additive Increase, Multiplicative Decrease. - Increase the limit gradually when requests succeed. - Reduce it significantly when failures or overload signals appear. For example: Current limit: 50 Successful samples: 50 → 51 → 52 → 53 Overload detected: 53 → 37 AIMD is relatively simple and works well when the system has reliable failure signals. However, it may react only after a dependency starts rejecting requests or timing out. Vegas Vegas uses latency as an early signal of congestion. It estimates queueing using the difference between observed latency and a no-load baseline. A simplified queue estimate is: [ Q = L \times \left(1 - \frac{RTT_{baseline}}{RTT_{current}}\right) ] Where: - (Q) is estimated queue size - (L) is the current concurrency limit - (RTT_{baseline}) is the observed no-load latency - (RTT_{current}) is the current measured latency When latency stays close to the baseline, the algorithm can increase concurrency. When latency grows significantly, it reduces the limit. Gradient2 Gradient2 compares short-term and long-term latency averages to detect changes in the latency trend. It is useful when latency baselines drift over time or workloads are bursty. Instead of relying only on a minimum latency measurement, it uses smoothed averages to identify growing queueing pressure. For a deeper implementation reference, Netflix's concurrency-limits project is worth exploring: Netflix/concurrency-limits For this article, we'll build a smaller Vegas-inspired controller. It is an educational implementation, not a complete reproduction of Netflix's Vegas or Gradient2 algorithms. 4. Building an Adaptive Limiter in Go Let's implement a limiter that: - Tracks in-flight requests - Enforces a dynamic concurrency limit - Measures request duration - Collects latency samples in windows - Estimates queueing using a baseline latency - Increases or decreases the limit based on observed behavior We'll use a fail-fast admission strategy. When the current concurrency limit is reached, the limiter rejects the new request instead of allowing an unbounded queue to form. Step 1: Define the limiter package adaptive import ( "errors" "sync" "time" ) var ErrLimitExceeded = errors.New( "adaptive concurrency limit exceeded", ) type Limiter struct { mu sync.Mutex limit int minLimit int maxLimit int inflight int peak int baseline time.Duration sampleCount int sampleTotal time.Duration dropped bool windowSize int alpha float64 beta float64 } func New( initialLimit int, minLimit int, maxLimit int, ) *Limiter { if minLimit < 1 || initialLimit < minLimit || maxLimit < initialLimit { panic("invalid concurrency limits") } return &Limiter{ limit: initialLimit, minLimit: minLimit, maxLimit: maxLimit, windowSize: 20, alpha: 2, beta: 4, } } The limiter maintains two important pieces of information: Current limit: The maximum number of requests allowed simultaneously. In-flight count: The number of requests currently executing. We also maintain latency statistics and a baseline. The windowSize determines how many completed requests we collect before updating the concurrency limit. Using a window helps avoid changing the limit after every individual request. Step 2: Acquire a permit The limiter needs to decide whether a new request can execute. func (l *Limiter) Acquire() (func(bool), error) { l.mu.Lock() if l.inflight >= l.limit { l.mu.Unlock() return nil, ErrLimitExceeded } l.inflight++ if l.inflight > l.peak { l.peak = l.inflight } l.mu.Unlock() startedAt := time.Now() var once sync.Once done := func(dropped bool) { once.Do(func() { elapsed := time.Since(startedAt) l.mu.Lock() defer l.mu.Unlock() l.inflight-- l.recordSample(elapsed, dropped) }) } return done, nil } The returned function releases the permit and reports the outcome. For example: done, err := limiter.Acquire() if err != nil { return err } defer done(false) return callDependency() If the operation experiences a dependency-level overload signal, we can report that separately: done, err := limiter.Acquire() if err != nil { return err } defer func() { done(isOverloadError(err)) }() err = callDependency() return err The sync.Once protects against accidentally releasing the same permit twice. The caller must invoke the returned function exactly once in normal control flow. The sync.Once makes repeated invocations harmless. Step 3: Record latency samples Now we need to collect completed request samples. func (l *Limiter) recordSample( latency time.Duration, dropped bool, ) { l.sampleCount++ l.sampleTotal += latency if dropped { l.dropped = true } if l.sampleCount < l.windowSize { return } average := l.sampleTotal / time.Duration(l.sampleCount) l.updateLimit(average) l.sampleCount = 0 l.sampleTotal = 0 l.dropped = false l.peak = 0 } Instead of adjusting concurrency after every request, we wait until a complete sample window is collected. This reduces sensitivity to individual latency spikes. A production implementation may use more sophisticated rolling windows, exponential moving averages, or percentile-based measurements. Step 4: Implement the control algorithm Now comes the interesting part. We compare the current average latency against the baseline. func (l *Limiter) updateLimit( currentLatency time.Duration, ) { if currentLatency <= 0 { return } if l.baseline == 0 || currentLatency < l.baseline { l.baseline = currentLatency return } if l.dropped { l.decrease() return } queueEstimate := float64(l.limit) * (1 - float64(l.baseline)/float64(currentLatency)) // Increase only when the previous window // demonstrated meaningful concurrency usage. if l.peak*2 < l.limit { return } switch { case queueEstimate < l.alpha: l.increase() case queueEstimate > l.beta: l.decrease() } } The queue estimate is an approximation of the queueing pressure observed at the dependency. When current latency is close to the baseline, the estimated queue remains small. When current latency increases, the estimated queue grows. We use two thresholds: - alpha: Below this value, the dependency appears to have spare capacity. - beta: Above this value, queueing pressure is considered too high. The gap between the thresholds creates a stability region where the limit does not change. This helps avoid unnecessary oscillation. Step 5: Increase and decrease concurrency The final methods enforce the configured boundaries. func (l *Limiter) increase() { if l.limit < l.maxLimit { l.limit++ } } func (l *Limiter) decrease() { next := int(float64(l.limit) * 0.7) if next < l.minLimit { next = l.minLimit } l.limit = next } The limiter uses: - Additive increase: Grow gradually. - Multiplicative decrease: Reduce more aggressively when congestion appears. This is intentionally conservative. Increasing the limit by one allows the dependency to demonstrate additional capacity without creating a large traffic spike. Reducing the limit by 30% provides a faster response to overload. These values are configuration choices, not universal production defaults. Step 6: Expose the current limit We should expose the current state for observability. func (l *Limiter) Stats() ( limit int, inflight int, baseline time.Duration, ) { l.mu.Lock() defer l.mu.Unlock() return l.limit, l.inflight, l.baseline } Now we can inspect the limiter without exposing its internal synchronization. For example: limit, inflight, baseline := limiter.Stats() log.Printf( "adaptive limiter: limit=%d inflight=%d baseline=%s", limit, inflight, baseline, ) 5. Integrating the Limiter with an HTTP Client Let's apply the limiter to an external dependency. type MarketDataClient struct { httpClient *http.Client limiter *adaptive.Limiter } func (c *MarketDataClient) Fetch( ctx context.Context, symbol string, ) error { done, err := c.limiter.Acquire() if err != nil { return err } var dropped bool defer func() { done(dropped) }() req, err := http.NewRequestWithContext( ctx, http.MethodGet, "https://api.example.com/market/"+symbol, nil, ) if err != nil { return err } resp, err := c.httpClient.Do(req) if err != nil { dropped = isDependencyOverload(err) return err } defer resp.Body.Close() if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= http.StatusServiceUnavailable { dropped = true return fmt.Errorf( "market data dependency returned %d", resp.StatusCode, ) } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf( "unexpected status: %d", resp.StatusCode, ) } return nil } The example classifies overload responses separately from other errors. This distinction matters. A validation error, for example, should not necessarily cause the adaptive limiter to reduce its concurrency limit. Likewise, a network timeout might be an overload signal in one system but a network infrastructure issue in another. The failure classifier should reflect the dependency's actual behavior. The HTTP client should also have appropriate timeouts and connection settings. An adaptive concurrency limiter is not a replacement for transport-level protection. 6. What Happens During an Overload? Let's walk through a simplified scenario. Assume the limiter starts with a concurrency limit of 50. The dependency initially responds in 40ms. As traffic increases, the limiter observes the following: | Stage | Average latency | Concurrency limit | |---|---:|---:| | Initial | 40ms | 50 | | Increased traffic | 45ms | 51 | | Higher utilization | 55ms | 52 | | Queueing begins | 120ms | 36 | | Dependency recovers | 45ms | Gradually increases | The important behavior is not the exact numbers. The limiter detects that latency is increasing relative to the baseline and reduces the amount of work it allows in flight. As the dependency recovers, the limiter can gradually explore additional capacity again. This is a feedback control loop. However, it is important to understand that the limiter does not instantly discover the perfect concurrency limit. It continuously estimates a useful operating point based on observed signals. 7. Adaptive Concurrency vs. Other Resilience Patterns Adaptive concurrency is not a replacement for the patterns we've discussed in previous articles. It complements them. | Mechanism | Responsibility | |---|---| | Backpressure | Controls incoming work | | Load shedding | Rejects work when capacity is exceeded | | Bulkhead | Isolates resources and dependencies | | Retry budget | Limits additional retry traffic | | Adaptive concurrency | Adjusts the concurrency ceiling based on observed conditions | Consider a service that communicates with three dependencies: - Market Data API - Portfolio Database - Reporting Service Each dependency has different latency characteristics and capacity. Using one global concurrency limit would couple their behavior. A better approach is to create an independent adaptive limiter for each dependency. type DependencyLimiters struct { MarketData *adaptive.Limiter Portfolio *adaptive.Limiter Reporting *adaptive.Limiter } This gives each dependency its own feedback loop. If the reporting service becomes slow, its concurrency limit can decrease without automatically reducing the capacity available to market data requests. This is where adaptive concurrency and bulkheads work particularly well together. 8. Important Production Considerations The example implementation demonstrates the core idea, but a production adaptive limiter needs additional safeguards. Baseline latency can become stale A latency baseline is only useful when it represents a reasonably unloaded system. If the baseline is established while the dependency is already overloaded, the algorithm may incorrectly treat unhealthy latency as normal. Production implementations should carefully manage baseline updates and resets. Traffic must be representative A dependency may expose different latency characteristics for different operations. For example: - Reading a cached market price - Fetching historical prices - Calculating a portfolio - Executing a database aggregation Combining all these operations into one latency signal can produce misleading results. Consider separate limiters or traffic partitions when workloads have significantly different behavior. The algorithm needs enough samples Adaptive control depends on observations. A low-traffic service might not generate enough samples to update its limit reliably. A new instance also needs a warm-up strategy. Slow start, conservative initial limits, and minimum sample requirements can help. Avoid uncontrolled oscillation A limiter that constantly increases and decreases its capacity can introduce instability. Useful techniques include: - Sampling windows - Exponential smoothing - Separate increase and decrease thresholds - Bounded increase and decrease rates - Minimum and maximum concurrency limits - Jitter when multiple independent clients share a dependency Local limits multiply across replicas This is especially important in distributed deployments. Imagine ten replicas, each with an adaptive concurrency limit of 100. The dependency may receive up to 1,000 concurrent requests. Each instance is making decisions based on its own observations. The aggregate behavior is not automatically coordinated. Local adaptive concurrency is often sufficient for protecting individual instances and reacting to local conditions. But strict global dependency quotas may require a shared dispatcher, centralized coordination, or provider-side quota enforcement. Do not confuse concurrency with connection limits An HTTP transport's connection pool and an application's concurrency limiter protect different resources. A request may hold an application permit while waiting for a connection. Conversely, HTTP/2 can multiplex multiple requests over one connection. Configure both based on the actual workload. 9. Testing Adaptive Concurrency Adaptive algorithms should be tested as control systems, not only as ordinary utility functions. At minimum, test these behaviors: - The initial concurrency limit is respected. - Requests are rejected when the current limit is reached. - Permits are released exactly once. - Successful low-latency windows allow gradual growth. - Rising latency causes the limit to decrease. - Overload signals trigger a decrease. - The limit never exceeds its configured maximum. - The limit never falls below its configured minimum. - Concurrent acquisitions are race-safe. - Insufficient sample windows do not trigger premature changes. Run the race detector: go test -race ./... Also consider deterministic algorithm tests. Instead of using real HTTP requests and sleeping for arbitrary durations, separate the control algorithm from the admission mechanism. Feed known latency samples into the controller and verify the resulting limit. For example: Baseline: 40ms Window 1: 42ms → increase Window 2: 45ms → increase Window 3: 120ms → decrease Window 4: 130ms → decrease Window 5: 45ms → gradual recovery This makes algorithm behavior easier to reproduce and debug. Finally, run load tests with changing dependency latency. A useful test should simulate: 1. A healthy dependency. 2. Gradually increasing latency. 3. Sudden overload. 4. Partial recovery. 5. A complete dependency outage. Measure throughput, p95/p99 latency, rejection rate, and concurrency limit changes. The goal is not simply to maximize throughput. The goal is to maintain useful throughput without allowing latency and queueing to grow uncontrollably. 10. Observability An adaptive limiter introduces a new operational component. You need visibility into how it makes decisions. Useful metrics include: | Metric | Type | Purpose | |---|---|---| | adaptive_concurrency_limit | Gauge | Current concurrency ceiling | | adaptive_inflight_requests | Gauge | Active requests | | adaptive_request_latency_seconds | Histogram | Observed dependency latency | | adaptive_rejected_total | Counter | Requests rejected at admission | | adaptive_limit_increase_total | Counter | Limit increase events | | adaptive_limit_decrease_total | Counter | Limit decrease events | | adaptive_baseline_latency_seconds | Gauge | Estimated unloaded latency | These metrics help answer important questions: - Is the limiter constantly operating at its maximum? - Is the concurrency limit frequently decreasing? - Is the dependency latency increasing? - Are requests being rejected before reaching the dependency? - Is the limiter recovering too slowly? A particularly useful dashboard plots three signals together: 1. Current concurrency limit 2. In-flight requests 3. Dependency latency Looking at them independently can hide the relationship between capacity and latency. Avoid high-cardinality labels such as raw URLs, user IDs, and request IDs. 11. When Should You Use Adaptive Concurrency? Adaptive concurrency is not necessary for every service. A fixed limit is often the better choice when: - Dependency capacity is stable and well understood. - Traffic patterns are predictable. - The service has low request volume. - The cost of an adaptive control loop is not justified. - A simple semaphore already provides sufficient protection. Adaptive concurrency becomes more attractive when: - Dependency capacity changes frequently. - Traffic is bursty or unpredictable. - Services scale horizontally. - Latency is a reliable indicator of queueing. - Manual concurrency tuning is difficult. - You need to protect dependencies before they start rejecting requests. Start with a fixed limit if it is sufficient. Introduce adaptive behavior when operational evidence shows that static configuration is becoming a limitation. Conclusion A fixed concurrency limit assumes that system capacity is relatively predictable. In distributed systems, that assumption does not always hold. Dependencies slow down, traffic changes, instances scale, and resource availability evolves. Adaptive concurrency provides a feedback loop that adjusts the amount of work allowed in flight based on observed latency and failure signals. In Go, the core mechanism is straightforward: track active requests, collect latency samples, estimate congestion, and adjust the concurrency ceiling. The difficult part is not implementing a semaphore. It is building a stable controller that reacts to real overload without constantly oscillating or making incorrect decisions from noisy measurements. Don't just limit how much work your service can handle. Let it learn when it should slow down.

Your Rate Limiter Is a Stampede at the Window Boundary The fixed window lets double the traffic through at the boundary, the distributed counter is an agreement problem in disguise, and the token bucket is the honest answer. The page came at 00:00:37 on a Tuesday. The database was falling over and the rate limiter's dashboard said everything was fine. Limit: 5,000 requests per minute. Busiest minute on record: 4,800. The limiter was telling the truth. The database was dying anyway. Somewhere around 23:59:59, twenty-five hundred requests landed in the last second of one window. At 00:00:01, twenty-three hundred more landed in the first second of the next. Each window reported "under the limit." The database received forty-eight hundred requests in about two seconds. That is the stampede. It is not a bug in your limiter. It is the design, working exactly as specified. Two Bad Defaults and a Third Option That Ships Most services I see run one of two defaults, and both fail the same way: quietly, at the boundary, under load you actually care about. The first default is no limiter at all. The API is open, the clients are known, until the day someone is not. One misconfigured cron job, one retry loop with a short fuse, one partner's viral moment, and your database is doing the job the limiter was supposed to do. This wins for simplicity. You pay for it with the first incident. The second default is the fixed-window counter. One key per client, one counter, one expiry. At the top of each minute the counters reset. It is simple, it is cheap, and it has a hole you can drive a stampede through: the limit is enforced per window, but your downstream experiences time as a continuum. Anything that straddles the boundary gets counted twice against nothing. I understand why the second default survives. It is one INCR and one TTL, and it fits in a middleware in an afternoon. When your downstream can absorb twice the limit for one window, it is arguably fine. But "twice the limit, briefly, at :00, every hour" is a strange thing to leave to chance, and chance decides which clients fire at the boundary. The third option is to stop rating the average and start rating the burst: pick the algorithm by the worst second you can tolerate, not the average minute you want. Here is the gap this piece fills. Every tutorial shows you five algorithms and a Redis snippet. Almost none of them say which guarantee each algorithm actually gives you at the boundary, what breaks when the counter has to live on more than one machine, or what the client is supposed to do when you say no. Those three questions are the whole job. Name the Algorithms by Their Guarantee A rate limiter is a clock and a counter. The counter is honest. The clock is the part that lies. Fixed window. One counter per key, reset every N seconds. Guarantee: at most L requests per window. The part nobody prints on the box: up to 2L requests in any N-second span that straddles a boundary. The numbers, on a simulated clock: from rate_limiter import FixedWindow lim = FixedWindow(limit=100, window_s=60.0) allowed = 0 for _ in range(100): allowed += lim.allow(59.9) # last tenth of a second of window N for _ in range(100): allowed += lim.allow(60.1) # first tenth of a second of window N+1 print(allowed) # 200 Two hundred requests allowed inside a 0.2-second span, under a limit of 100 per 60 seconds. The limiter kept its promise. The promise was the problem. Drawn as a timeline, the hole is obvious: 59.0s 59.5s 59.9s | 60.0s 60.1s 61.0s | requests: 2,500 ##|## 2,300 window N count: ... 4,700 -> 5,000, then reset window N+1 count: 0 -> 2,300 ... Each window, the limiter says: under the limit. The database, across the boundary: 4,800 requests in two seconds. Sliding window log. Store a timestamp per request, evict anything older than N seconds, count what remains. Guarantee: at most L requests in any rolling N-second span, exactly. The price is memory: one timestamp per request per key until it expires. Sliding window counter. Keep two counters, current and previous window, and estimate: count = current + previous * (1 - elapsed_fraction). Approximately L per rolling window, no 2x blowup, two integers of state; Cloudflare described this design for their edge in 2017. The estimate errs in the friendly direction, which should make you slightly nervous. Token bucket. Two numbers: tokens in the bucket, and the last refill time. Each request costs one token. Tokens refill at a steady rate, and the bucket caps accumulation. Guarantee: long-run average of R requests per second, with bursts up to the bucket size allowed on purpose. The burst is not a bug here. It is the parameter you chose. Here it is on a real clock: from rate_limiter import TokenBucket b = TokenBucket(capacity=10, refill_per_s=2.0) burst = sum(b.allow() for _ in range(15)) # burst of 15 -> 10 allowed immediately, 5 denied time.sleep(3.0) after = sum(b.allow() for _ in range(10)) # after 3s idle, 10 quick requests -> 6 allowed Ten through immediately, five denied, and three idle seconds buy you exactly six tokens back. The math is boring in the way that production systems should be boring. Leaky bucket. The mirror image: requests pour in, the bucket drains at a fixed rate, overflow is rejected or queued. As a meter it behaves like the token bucket. As a queue, it smooths traffic on the way out instead of rejecting it on the way in. That is what nginx does with limit_req: excess requests are delayed to fit the rate, not dropped, unless you say otherwise. Rate the burst, not the average: the bucket size is the decision, the refill rate is just the budget. Real Tools With Real Commands Theory is cheap. Here is what the tools actually give you. nginx implements the leaky bucket, and the two knobs that matter are burst and nodelay: limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s; server { location /api/ { limit_req zone=mylimit burst=20 nodelay; proxy_pass http://backend; } } Ten per second sustained, twenty extra as a burst. With nodelay, the burst is processed immediately. Without it, excess requests are delayed so the backend still sees a smooth 10 per second. That one flag decides whether your limiter protects latency or throughput, and they are not the same thing. Redis is where most hand-rolled limiters live, and the classic INCR pattern has a race the Redis docs themselves call out: a client that crashes between INCR and EXPIRE leaves a counter with no expiry, a leak that only grows. The fix is a Lua script so the check, increment, and expiry happen atomically: -- KEYS[1]: counter key, ARGV[1]: limit, ARGV[2]: window seconds local current = redis.call("INCR", KEYS[1]) if current == 1 then redis.call("EXPIRE", KEYS[1], ARGV[2]) end if current > tonumber(ARGV[1]) then return {0, current} -- denied end return {1, current} -- allowed One round trip, atomic, no leak. If your limiter is not atomic, you do not have a limiter. You have a suggestion. redis-cell is the serious version: Brandur Leach's module implements GCRA, the Generic Cell Rate Algorithm, the token bucket done with exact virtual scheduling time instead of approximate refill math. One command does the whole job: CL.THROTTLE user123 15 30 60 That reads: bursts of 15, sustained 30 per 60 seconds, for key user123. The reply is five integers: limited or not, the limit, remaining, seconds until retry, seconds until full reset. Everything a client needs to behave well, in one round trip. The headers are finally getting standardized, slowly. draft-ietf-httpapi-ratelimit-headers, draft 11 as of May 2026 and still not an RFC, defines two fields: RateLimit-Policy, which advertises the quota ("default";q=100;w=60), and RateLimit, which reports current state ("default";r=15;t=23). Cloudflare adopted the draft in September 2025; GitHub still emits the legacy X-RateLimit-* family, 5,000 requests an hour for authenticated users. The draft itself warns that a naive fixed-window reset encourages cyclic burst-pause behavior: the stampede wearing formal clothes. Emit the headers even before the standard settles. A client that can see the limit can pace itself; a client flying blind can only guess, and guesses come in bursts. The client side is the half everyone forgets. Netflix's concurrency-limits library runs an adaptive limiter inside the client: it watches round-trip times against a long-term minimum and shrinks its own concurrency budget when latency climbs, backing off before the server has to say no. The server-side limiter is the guardrail. The client-side limiter is the driver who slows down before the curve. You want both. To see your own limiter misbehave, generate load and watch the headers: hey -q 300 -c 50 -z 20s https://staging.example.com/api/search curl -sI https://staging.example.com/api/search | grep -i -E "ratelimit|retry-after" If the 429s cluster at the top of the minute and your p99 spikes with them, you have found your stampede. Ten minutes of measurement, and the argument is over. The Distributed Question Nobody Prices In One machine, one counter, one clock: easy. The moment the counter has to live on more than one machine, you have an agreement problem wearing a rate-limiting costume. The naive answer is to divide: N nodes, each enforces limit/N. Traffic is never even, so the node holding the hot key throttles legitimate users while the other N-1 sit on idle quota. You paid for the full limit and you use the smallest slice. It fails the other direction too: a client that spreads requests across nodes gets N times the limit, because nobody counts the total. The standard answer is the shared counter: every node checks the same Redis. The counter is now correct and the hot path pays a network round trip on every request. Redis is single-threaded, so your "distributed" limiter has a single-threaded heart, and that heart is the throughput ceiling of your API. Correctness costs a round trip; the round trip costs latency plus a new bottleneck. There is no version where the distributed counter is free. The honest answers, in increasing order of machinery: 1. Sticky routing. Send each key to the same node and keep the counter local. Exact counts, memory-speed hot path, and the agreement problem moves into the router. Works until a node dies and its keys move, and every moved key gets a fresh counter and a fresh burst. 2. Local buckets, small. Each node runs a token bucket at limit/N. You accept the uneven-traffic waste in exchange for no hot-path round trip. This is what most large APIs actually do; the waste is the price of the latency. 3. Shared counter via GCRA. redis-cell's CL.THROTTLE, one round trip, exact. Pay the latency when the downstream is expensive enough to deserve it: a database, a GPU endpoint, a third-party API that bills per call. Notice the shape: who holds the counter, who agrees on the count, what happens when a holder dies. The same questions as leader election and quorum, with smaller numbers. The limiter is a tiny consensus system about how many requests happened, and like every consensus system, it charges you for the agreement. Where This Breaks No hedging. These are the failure modes I have watched in production. - A limiter cannot tell a good burst from a bad one. Deploys, cron jobs firing at :00, a legitimate flash of traffic: all of it looks like an attack. You will 429 your own deploy pipeline at least once. Plan the allowlist before the incident, not during it. - Retry-After without jitter synchronizes the retries. Every denied client waits the same number of seconds and returns at the same instant. You built a second stampede out of the first one's survivors. Jitter the wait, always. - The limiter sits on the hot path. A shared Redis counter adds a round trip to every request, including the 99 percent that were never going to be limited. Measure that latency. It is the real price of the feature. - Per-IP limits punish everyone behind a NAT. Per-key limits miss aggregate abuse. Key cardinality is unbounded, because an attacker rotates keys faster than you can count them. Cap the keys or the limiter is a memory leak with extra steps. - Fixed windows punish the innocent at the boundary and reward the strategic: fire at :59 and :01 and the limiter applauds. Token buckets need per-key state and a refill clock per key. - Skewed clocks move the boundary per node. Monotonic clocks for measuring rates, wall clocks for agreeing on windows, never the reverse. - The headers are advisory. The IETF draft says the values MAY be ignored, and "5 remaining" promises nothing. Emit them anyway, but never build correctness on a client reading them. - Limiters protect rate, not cost. One hundred cheap cache hits and one hundred full-table scans count the same. If the work per request varies by orders of magnitude, weight the tokens by cost. Build It If, Skip It If Build it if you have a shared expensive downstream, noisy multi-tenant neighbors, or a public API with clients you did not choose. Those are the three situations where one actor's burst becomes everyone's outage, and the limiter is cheaper than the incident. Skip it if the service is single-tenant and internal behind auth, with clients you can fix. Fix the client. Also skip it if traffic is already shaped upstream: your CDN or WAF probably rate-limits better than your application will. The minimal viable version fits in an afternoon. An in-process token bucket per key, no Redis, no shared anything. On deny, return 429 with a jittered Retry-After. Teach exactly one client to read the header and back off. Run hey against staging and watch the boundary. Only add the shared counter when you have more than one node and traffic uneven enough to notice. Most teams add Redis on day one and debug clock skew on day thirty. Reverse that order. Rate the Burst, Not the Average Pick the burst size first. Write it in the config next to the rate, as a number you chose instead of a default you inherited. Then generate load against staging, watch the window boundary, and let the measurement argue with the dashboard. The average is the easy part. The boundary is the whole job. What is the worst 429 you have ever caused, or received? The one where the limiter was technically correct and the outage happened anyway. I read every reply. Resources 1. RateLimit header fields for HTTP, draft-ietf-httpapi-ratelimit-headers: draft 11, May 2026, still a draft; defines the RateLimit and RateLimit-Policy fields. 2. redis-cell by Brandur Leach: Redis module implementing GCRA; the CL.THROTTLE command is the serious version of the Lua script above. 3. nginx limit_req module documentation: leaky bucket with burst and nodelay, and the difference between delaying and dropping. 4. Netflix concurrency-limits: client-side adaptive limiting with the gradient algorithm; the other half of the story. 5. RFC 6585: defines status 429 Too Many Requests. 6. RFC 9110: HTTP semantics, including the Retry-After field. 7. Related pieces in this series: retry storms, backpressure, idempotency keys, clock skew, and the quorum tax.