Netflix

Building an In-Memory Circuit Breaker for Webhook Ingress: Shedding Load Without Losing Events Webhooks connect cloud systems. Payment providers like Stripe, code hosts like GitHub and commerce platforms like Shopify all use them to tell your services that something changed. When a sender has a bad day, you pay for it. A bulk export, a backfill or a burst of retries can hit your ingress endpoint, and the endpoint is the first thing that fails. When a burst arrives, worker threads or goroutines block on slow downstream calls and connection pools saturate. Memory fills with queued request bodies and garbage-collection pressure pushes latency higher. Health checks start failing, the load balancer moves traffic to the remaining nodes, and those nodes tip over too. This guide describes an in-memory admission controller for webhook ingress. It combines an adaptive concurrency limit, a CoDel-style delay signal and priority-aware shedding. It also covers what “not losing events” means when you reject a request, because that depends on how each sender retries. 1. How an ingress meltdown unfolds Webhook senders enforce short response deadlines, and each provider sets its own: | Sender | Documented behavior | | --- | --- | | GitHub | Expects a 2XX within 10 seconds, otherwise it terminates the connection and counts a failed delivery. It does not redeliver automatically; failed deliveries must be redelivered manually or via the API. | | Shopify | One-second connection timeout and five-second timeout for the whole request. Failed deliveries are retried 8 times over 4 hours. | | Slack (Events API) | Expects a valid response within three seconds. | | Stripe (live mode) | Retries for up to 3 days with exponential backoff. | Because the deadlines are short, a slow endpoint produces timeouts, and timeouts produce more traffic. A sender that retries adds those retries to the stream of fresh events. A sender that doesn’t retry, like GitHub, leaves you with lost events and manual redelivery. The failure then plays out in a predictable order: 1. Connection saturation: new connections queue in the socket backlog. 2. Worker starvation: handlers blocked on database calls hold their threads or goroutines. 3. Memory growth: request bodies pile up while waiting for a worker. 4. Latency amplification: garbage-collection work and contention raise latency, which causes more sender timeouts. [Upstream Sender] ──(flood / retries)──> [Load Balancer] │ ▼ [Ingress Worker Node] ├── Connection queue (full) ├── Workers (blocked) └── Memory / GC (pressure) ──> [meltdown] The first defense is architectural. GitHub’s documentation recommends putting a queue in front of slow work so the server can respond immediately and process the payload in the background. Shopify gives the same advice for bursts. Admission control is the second line of defense. It protects the thin “validate, enqueue, acknowledge” path itself. 2. Why the admission decision should be local Resilience4j’s circuit breaker is the standard tool for outbound calls. It is an in-process library and doesn’t need Redis or any other shared store. The same reasoning applies at ingress: - No extra dependency on the hot path. Consulting a shared store for every inbound request adds a network round trip and a new failure mode. If that store is down, your ingress should still be able to shed load. - The overload is local. Thread pools, memory and database connections are per-node resources. A node-local signal reflects what that node can actually handle. - Senders only see HTTP. They react to status codes and timing, so the cleanest signal you can give them is a fast, explicit rejection. The trade-off is that each node decides independently. With uneven load balancing, one node may shed while another has spare capacity. Per-node metrics (section 7) show you when that happens. 3. Two mechanisms: adaptive concurrency and queue delay Adaptive concurrency limits Little’s Law says that concurrency equals throughput times latency. A fixed limit such as “100 concurrent requests” goes stale as soon as latency changes. If the database slows down, 100 is too high. If it speeds up, 100 is too low. Netflix’s open-source concurrency-limits library, described in their 2018 post “Performance Under Load”, adjusts the limit from observed latency instead. Its gradient algorithm uses: newLimit = gradient × currentLimit + queueSize In the library’s Gradient2 variant, the gradient is the long-term RTT divided by the current RTT, clamped to the range 0.5 to 1.0. The result is then smoothed (default factor 0.2): gradient = max(0.5, min(1.0, longTermRTT / currentRTT)) newLimit = gradient × currentLimit + queueSize newLimit = currentLimit × (1 − smoothing) + newLimit × smoothing When latency rises above its long-term average, the gradient drops below 1 and the limit shrinks. When latency is stable, the queueSize term lets the limit creep upward. It can be a fixed value or a function of the limit, such as a square root. Queue delay (CoDel-style) Counting 5xx errors reacts late, because by the time errors appear the damage is done. A better early signal is how long requests wait for a slot. CoDel (Controlled Delay) was designed by Kathleen Nichols and Van Jacobson as an active queue management algorithm for network buffers (RFC 8289). It tracks a packet’s sojourn time, the time spent in the queue. It acts only when the minimum delay over a sliding window stays above a target, which allows short bursts but prevents a persistent standing queue. RFC 8289’s defaults are a 5 ms target and a 100 ms interval. Facebook adapted a variant of CoDel for request queues, as described in “Fail at Scale”. Its open-source Wangle library shows the idea. Within each interval it tracks the minimum queueing delay and declares overload if that minimum exceeded the target. While overloaded, it sheds requests that have waited longer than twice the target. This is an adaptation rather than textbook CoDel. Treat 5 ms and 100 ms as starting points and tune them against your handler’s normal service time. 4. Architecture Requests pass through classification and then a limiter that both enforces the adaptive limit and measures queue delay: Incoming webhook request │ ▼ ┌──────────────────────────────────────────┐ │ 1. Priority classifier │ │ (route or verified partner identity) │ └──────────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────┐ │ 2. Admission: slot free? │ │ yes → admit; no → queue briefly, or │ │ shed if overloaded (by priority) │ └──────────────────────────────────────────┘ │ (slot acquired) ▼ ┌──────────────────────────────────────────┐ │ 3. Thin handler: verify signature, │ │ enqueue durably, return 2xx │ └──────────────────────────────────────────┘ │ ▼ Release slot → feed RTT back to the limit Classify by something you control, such as the endpoint path or the partner identified after signature verification. Don’t trust a priority header that arrives from the public internet, because anyone can set it. 5. A working Go sketch The following sketch uses only the standard library. It combines a Gradient2-style limit with a Wangle-style overload flag. The parameter values are illustrative and need tuning against your own traffic. package ingress import ( "math" "net/http" "sync" "time" ) type Priority int const ( PriorityLow Priority = iota PriorityDefault PriorityHigh ) type Config struct { InitialLimit float64 // e.g. 20 MinLimit float64 // e.g. 4 MaxLimit float64 // e.g. 200 Smoothing float64 // e.g. 0.2 TargetDelay time.Duration // e.g. 5ms Interval time.Duration // e.g. 100ms MaxWait time.Duration // longest a request may wait for a slot, e.g. 2 * TargetDelay MaxQueue int // hard cap on waiting requests } type waiter struct { ready chan struct{} granted bool } type Limiter struct { cfg Config mu sync.Mutex limit float64 inflight int waiters []*waiter shortRTT float64 // seconds, fast EWMA longRTT float64 // seconds, slow EWMA intervalEnd time.Time minDelay time.Duration overloaded bool } func NewLimiter(cfg Config) *Limiter { return &Limiter{cfg: cfg, limit: cfg.InitialLimit} } // Acquire returns a release func and true if the request is admitted. func (l *Limiter) Acquire(p Priority) (func(time.Duration), bool) { l.mu.Lock() if l.inflight < int(l.limit) && len(l.waiters) == 0 { l.inflight++ l.observeDelay(0) l.mu.Unlock() return l.release, true } if l.shouldShed(p) { l.mu.Unlock() return nil, false } w := &waiter{ready: make(chan struct{})} l.waiters = append(l.waiters, w) l.mu.Unlock() start := time.Now() timer := time.NewTimer(l.cfg.MaxWait) defer timer.Stop() timedOut := false select { case <-w.ready: case <-timer.C: timedOut = true } l.mu.Lock() defer l.mu.Unlock() l.observeDelay(time.Since(start)) if timedOut && !w.granted { l.removeWaiter(w) return nil, false } return l.release, true } func (l *Limiter) shouldShed(p Priority) bool { switch p { case PriorityHigh: return len(l.waiters) >= l.cfg.MaxQueue // only when the queue is full case PriorityDefault: return l.overloaded || len(l.waiters) >= l.cfg.MaxQueue default: // PriorityLow: no queueing at all once slots are exhausted return true } } // observeDelay tracks the minimum queueing delay per interval (CoDel-style). func (l *Limiter) observeDelay(d time.Duration) { now := time.Now() if now.After(l.intervalEnd) { l.overloaded = l.minDelay > l.cfg.TargetDelay l.minDelay = d l.intervalEnd = now.Add(l.cfg.Interval) return } if d < l.minDelay { l.minDelay = d } } func (l *Limiter) release(rtt time.Duration) { l.mu.Lock() defer l.mu.Unlock() l.inflight-- l.updateLimit(rtt.Seconds()) for len(l.waiters) > 0 && l.inflight < int(l.limit) { w := l.waiters[0] l.waiters = l.waiters[1:] w.granted = true l.inflight++ close(w.ready) } } // updateLimit is a simplified Gradient2-style update. func (l *Limiter) updateLimit(rtt float64) { if rtt <= 0 { return } if l.longRTT == 0 { l.longRTT, l.shortRTT = rtt, rtt } l.shortRTT += 0.2 * (rtt - l.shortRTT) l.longRTT += 0.01 * (rtt - l.longRTT) gradient := math.Max(0.5, math.Min(1.0, l.longRTT/l.shortRTT)) next := gradient*l.limit + math.Sqrt(l.limit) next = l.limit*(1-l.cfg.Smoothing) + next*l.cfg.Smoothing l.limit = math.Max(l.cfg.MinLimit, math.Min(l.cfg.MaxLimit, next)) } func (l *Limiter) removeWaiter(w *waiter) { for i, x := range l.waiters { if x == w { l.waiters = append(l.waiters[:i], l.waiters[i+1:]...) return } } } // Guard wraps a webhook handler with admission control. func Guard(l *Limiter, classify func(*http.Request) Priority, next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { release, ok := l.Acquire(classify(r)) if !ok { w.Header().Set("Retry-After", "10") http.Error(w, "service temporarily overloaded", http.StatusServiceUnavailable) return } start := time.Now() defer func() { release(time.Since(start)) }() next.ServeHTTP(w, r) }) } Some limits of this sketch: - Waiters are served in FIFO order. Facebook’s design switches to LIFO once a queue forms, so the newest requests, which are most likely to still meet their sender’s deadline, go first. - RTT is measured for every request. In production you would typically exclude requests that failed fast. - A single mutex is fine at moderate rates. Profile before optimizing it. 6. Rejecting requests without losing events Whether a rejected webhook is lost depends on the sender, so you need to know how each one behaves: - Senders with retries: Stripe retries with exponential backoff for up to 3 days in live mode. Shopify retries 8 times over 4 hours. - Senders without automatic retries: GitHub does not redeliver automatically. If you shed a GitHub delivery, you need to redeliver it manually or through the API once you recover. - Subscription risk: Shopify deletes a webhook subscription created through the Admin API after 8 consecutive failures, and Stripe emails you and eventually disables an endpoint that fails for multiple days. A long shedding episode can cost you the subscription itself. For these reasons, shedding should be short, targeted and backed by a reconciliation job. Shopify itself recommends building one if you can’t afford to miss webhooks. The job periodically fetches data you may have missed through the provider’s API. Status codes - 503 Service Unavailable with Retry-After: the standard way to signal temporary overload. RFC 9110 defines Retry-After as either a number of seconds or an HTTP date. - 429 Too Many Requests: RFC 6585 defines it for rate violations by a particular sender. It also accepts Retry-After. - Don’t just drop connections. An abrupt reset gives the sender no information, and it will treat the delivery as a failed attempt. A clean 503 tells it explicitly that the failure is temporary. Do senders actually honor Retry-After? Some do. Webhook infrastructure vendors such as Hookdeck and Kit document that they respect Retry-After on 429 and 503, and AWS EventBridge API destinations are documented to as well. Svix advises that good senders will respect it when you return a 429. Stripe’s public documentation describes exponential backoff but doesn’t mention Retry-After, so check its behavior in your delivery logs rather than assuming. Because support varies, treat the header as a hint and rely on the sender’s own backoff as the fallback. Add jitter to the values you return, for example 5 to 15 seconds instead of a constant 10. Otherwise senders that honor the header will retry in lockstep. 7. Priority-based shedding Not all webhooks are equally urgent. A payment confirmation matters more than a bulk analytics sync. Classify by endpoint or authenticated partner, then apply different behavior per tier. The sketch above implements this policy: | Tier | Examples | Behavior in the sketch | | --- | --- | --- | | PriorityHigh | Payment confirmations, security alerts | Queue for a slot; shed only when the wait queue is full. | | PriorityDefault | Standard operational updates | Queue, but shed once the CoDel-style overload flag is set. | | PriorityLow | Bulk syncs, analytics, non-essential notifications | Shed immediately when no slot is free. | This is a policy choice rather than a standard. Choose tiers based on which events you can safely replay later. 8. Observability Since each node decides for itself, you need per-node metrics. Export them to Prometheus or OpenTelemetry: 1. webhook_ingress_concurrency_limit: gauge for the current adaptive limit. 2. webhook_ingress_inflight_requests: gauge for requests being processed. 3. webhook_ingress_shed_total{priority}: counter of rejected requests. 4. webhook_ingress_queue_delay_seconds: histogram of time spent waiting for a slot. Use millisecond-scale buckets, since the targets are in the single-digit-millisecond range. 5. webhook_ingress_overloaded: gauge (0 or 1) for the overload flag. Alert on a sustained rise in shed counts or on queue delay staying above target. Both mean you should look at downstream capacity, not just the limiter. 9. Checklist - Acknowledge fast: Verify the signature, enqueue durably and return 2xx. Do the real work asynchronously. - Keep admission state local: Avoid a shared-store call on every inbound request. - Use a latency-driven limit: An adaptive, gradient-style limit beats a fixed number. - Watch queue delay, not just errors: Use a CoDel-style minimum-delay signal, and tune the target to your handler. - Reject explicitly: Return 503 (or 429 for per-sender limits) with a jittered Retry-After. - Know your senders: Document each provider’s timeout, retry window and subscription-deletion rules. - Reconcile: Run a job that backfills events missed during shedding, and redeliver GitHub deliveries manually. - Classify safely: Derive priority from routes or verified identity, never from unauthenticated headers. References - Stripe webhooks documentation: <docs.stripe.com> - GitHub, “Best practices for using webhooks”: <docs.github.com> - Shopify, “Deliver webhooks through HTTPS”: <shopify.dev> - Slack Events API response deadline: <vincentschmalbach.com> - Netflix Technology Blog, “Performance Under Load”: <netflixtechblog.medium.com> - Netflix concurrency-limits library: <Netflix/concurrency-limits> - RFC 8289, “Controlled Delay Active Queue Management”: <datatracker.ietf.org> - CoDel pseudocode (ACM Queue): <queue.acm.org> - Facebook Wangle CoDel implementation: <github.com/facebook/wangle/blob/master/wangle/con…> - Svix, “Webhook retry strategies”: <svix.com> - Hookdeck, “Webhook Delivery Guarantees”: <hookdeck.com> --- Originally published at instawebhook.com

LLM API Slows to 30s, Your Server Holds 600 Requests: Little's Law and a Load Shedder - Book: System Design Pocket Guide: Fundamentals - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Your LLM provider is up. Every call just takes 30 seconds instead of one. Nothing errors. Your health check is green. And your Node process is slowly filling up with requests that are all waiting on the same slow upstream. The loud failures make the status pages. DeepSeek logged an incident at 8:11 Beijing time on October 15, and chat and API users got "the server was temporarily unavailable" until a fix moved to monitoring at 12:49, per TechNode. Five days later OpenAI posted authentication errors for some users near Illinois, API included, for an hour, fixed by rolling back a configuration change. The slow afternoon usually never gets an entry. You can predict exactly how full your process gets with one line of arithmetic, and you can stop it with a counter and a fast 503, checked against a simulation you can run. The arithmetic: L = λW Little's Law, proved by John Little in 1961, says that in a stable system the average number of items inside (L) equals the arrival rate (λ) times the average time each one spends inside (W). It does not care about the arrival distribution or the order things are served in. That is why it is so useful for back-of-the-envelope capacity work. Apply it to an endpoint that proxies an LLM call: - Normal day: 20 requests per second, 1 second per call. L = 20 × 1 = 20 requests in flight. - Upstream slows down: same 20 requests per second, 30 seconds per call. L = 20 × 30 = 600 requests in flight. Your traffic did not change. Your code did not change. Your concurrency went up 30x because W did. Each of those 600 requests holds a socket to the client, a socket to the provider, the prompt, the buffered response, and a promise chain in memory. Users rarely wait 30 seconds either. Some hit refresh, which raises λ while W stays high. Netflix's concurrency-limits library builds on the same identity: limit = average RPS × average latency. Flip it around and you get the two levers you have. You can cap W (timeouts) or you can cap L directly (a concurrency limit). λ belongs to your users. !A pipe of identical request tokens backing up behind a narrow, slow valve at the far end When W goes from 1 s to 30 s, the pipe holds 30 times as much, with no extra traffic. A simulation you can run Real load tests against a slow LLM are expensive and hard to reproduce. A deterministic, tick-based simulation is neither. It steps time forward 100 ms at a time, sends 20 requests per second, and makes the upstream take 30 seconds per call between second 30 and second 90. It also needs a way to die. Here the server has a 400 MB budget and each in-flight request holds 1 MB. Both are assumptions, picked to make the failure visible; measure your own. When the budget is exceeded, the process crashes, loses everything it was holding, and takes 10 seconds to come back. Save the file as sim.ts and run it with tsx; there are no dependencies. // sim.ts: run with `npx tsx sim.ts` const TICK = 100; // ms per simulation step const END = 180_000; // simulate three minutes const RPS = 20; // requests arriving per second const PATIENCE = 15_000; // a user gives up after 15 s const TARGET = 2_000; // "healthy" upstream latency const HEAP_MB = 400; // memory budget (assumption) const MB_PER_REQ = 1; // held per in-flight request const RESTART = 10_000; // crash-to-ready time // Fast, then 30 s per call for one minute, then fast. const upstream = (t: number) => t >= 30_000 && t < 90_000 ? 30_000 : 1_000; type Policy = { name: string; timeout: number; // Infinity = wait forever limit: () => number; // Infinity = no cap onSlow?: () => void; // a call passed TARGET onFast?: (inFlight: number) => void; }; type Req = { start: number; end: number; slow: boolean }; Each tick of run() does three things: settle what is in flight, admit new arrivals, and check the memory budget. Append it to the same sim.ts. function run(p: Policy) { let inFlight: Req[] = []; let downUntil = 0; let due = 0; // fractional arrivals carried between ticks const s = { ok: 0, late: 0, timedOut: 0, shed: 0, lost: 0, crashes: 0, peak: 0 }; for (let t = 0; t < END; t += TICK) { // finish, time out, or flag what is in flight const still: Req[] = []; for (const r of inFlight) { if (r.end <= t) { const took = r.end - r.start; if (took <= PATIENCE) s.ok++; else s.late++; if (took <= TARGET) p.onFast?.(inFlight.length); } else if (t - r.start >= p.timeout) { s.timedOut++; } else { if (!r.slow && t - r.start > TARGET) { r.slow = true; p.onSlow?.(); } still.push(r); } } inFlight = still; // new arrivals this tick (exact for any RPS) due += (RPS * TICK) / 1000; for (; due >= 1; due--) { if (t < downUntil) { s.lost++; continue; } if (inFlight.length >= p.limit()) { s.shed++; continue; } inFlight.push({ start: t, end: t + upstream(t), slow: false }); } s.peak = Math.max(s.peak, inFlight.length); // out of memory: the process dies with its requests if (inFlight.length * MB_PER_REQ > HEAP_MB) { s.crashes++; s.lost += inFlight.length; inFlight = []; downUntil = t + RESTART; } } return s; } The counters mean: ok answered within the user's patience, late answered after the user left (you paid for those tokens and nobody read them), timedOut your server gave up on, shed rejected instantly, lost dropped by a crash or refused while restarting. Three policies that fail differently Start with what a naive proxy does: no limit, no timeout. Then add a timeout, then a fixed limit. const unbounded: Policy = { name: "unbounded", timeout: Infinity, limit: () => Infinity, }; const timeoutOnly: Policy = { name: "timeout only", timeout: PATIENCE, limit: () => Infinity, }; const fixed50: Policy = { name: "limit 50 + timeout", timeout: PATIENCE, limit: () => 50, }; You can predict the unbounded run before you execute it. Little's Law says the in-flight count heads toward 600, but the budget is 400, so the process dies about 20 seconds into the slowdown (20 per second × 20 seconds = 400). It restarts empty, refills at the same rate, and runs into the same wall. Every crash throws away hundreds of requests that had each been waiting for many seconds. The timeout-only run caps W at 15 seconds, so L tops out near 20 × 15 = 300, under the budget. No crash. That is a real improvement, and it shows that a timeout is the cheapest fix there is. The catch is in timedOut: during the slow minute, every single user waits 15 seconds to get an error. And the cap only holds while λ holds. Set RPS = 40, the refresh-button scenario, and 40 × 15 = 600 puts you back over the budget. The fixed limit caps L at 50 no matter what λ does. Most users in the slow minute land in shed: they get a 503 right away instead of a 15-second spinner. The AWS Builders' Library puts the reason plainly: "Early rejection is important because it's the cheapest place to drop excess traffic, but it comes at a cost to visibility." That second half is why you count and graph every shed request. The weakness is the number 50. You picked it for one latency, and the upstream decides what latency you get. !A gatekeeper valve letting a few request tokens through to a slow upstream and bouncing the rest straight back A concurrency limit turns a slow upstream into fast rejections instead of a growing queue. Shedding in a real Node server Here is the same policy in an HTTP handler. The rejection carries Retry-After, which on a 503 tells the client how long the service is expected to be unavailable, as a number of seconds or a date. import { createServer } from "node:http"; const LIMIT = 50; const TIMEOUT_MS = 15_000; let inFlight = 0; createServer(async (req, res) => { if (inFlight >= LIMIT) { res.writeHead(503, { "Retry-After": "5" }); res.end("busy, retry shortly\n"); return; } inFlight++; try { const up = await fetch(process.env.UPSTREAM_URL!, { signal: AbortSignal.timeout(TIMEOUT_MS), }); res.writeHead(up.status); res.end(await up.text()); } catch { res.writeHead(504); res.end("upstream failed or timed out\n"); } finally { inFlight--; } }).listen(3000); The check runs before any work: no body parsing, no auth lookup, no upstream socket. The finally is the part people get wrong; if the counter does not come down on every path, the limit slowly leaks until the server sheds everything. On the client side, honour the header and add jitter, or 50 rejected clients all retry in the same second. Google's SRE book describes the client half too: a per-client retry budget where a request is retried only while retries stay below 10% of traffic. AIMD: let the upstream pick the limit Additive increase, multiplicative decrease is the feedback rule from TCP congestion control: grow slowly while things are fine, cut sharply on a sign of congestion. Netflix's AIMDLimit does this for concurrency: multiply the limit by 0.9 on a drop or a call slower than its timeout, add 1 on success, but only grow when at least half the current limit is in use. function aimd(min = 5, max = 200): Policy { let limit = 20; return { name: "aimd + timeout", timeout: PATIENCE, limit: () => Math.floor(limit), onSlow: () => { limit = Math.max(min, limit * 0.9); }, onFast: (inFlight) => { if (inFlight * 2 >= limit) { limit = Math.min(max, limit + 1); } }, }; } const policies = [unbounded, timeoutOnly, fixed50, aimd()]; console.table( policies.map((p) => ({ policy: p.name, ...run(p) })), ); The main change from Netflix's version: the sim flags a call as slow the moment it passes TARGET, not when it finally returns. With a 30-second upstream, a completion-based signal arrives 30 seconds after the trouble started. Two seconds in is when you want to know. Run all four and compare the shed and timedOut columns for the last two rows. During the slow minute the AIMD limit drops toward its floor of 5, so far fewer users sit through a 15-second wait for an error, and the in-flight count stays tiny. When the upstream recovers, fast completions push the limit back up by one each, and the "half in use" guard stops it from inflating while traffic is light. TARGET is the knob that carries your judgement. At 2 seconds, an upstream that settles at 10 seconds would also be treated as broken, even though a patient user could still be served. Set it from your own latency data, not from this post. What to take to your service Do the arithmetic for your worst endpoint first: peak RPS times the worst latency your provider has shown you. If that number of in-flight requests would hurt, you need a timeout today and a concurrency limit that answers with a fast 503 and Retry-After. AIMD is the next step once you are tired of guessing the right number. Then change upstream, RPS and HEAP_MB in the sim to match your own service and see which row you are living in. --- If this was useful Little's Law, timeouts, load shedding and backpressure are the parts of system design that only feel abstract until a provider has a slow afternoon. The System Design Pocket Guide: Fundamentals covers those building blocks, with the capacity arithmetic that tells you where your service breaks before your users find it. 

Adaptive Concurrency Limits in Go: Dynamically Protecting Dependencies Under Load Most services eventually need a concurrency limit. You might start with something simple: const maxConcurrentRequests = 100 A semaphore ensures that no more than 100 requests execute simultaneously. It works. Until the conditions change. Your service gets more CPU resources after scaling. A downstream database becomes slower. A new deployment changes request processing time. Traffic patterns shift throughout the day. The limit that worked yesterday might be too restrictive today. Or worse, it might be too high and allow a dependency to become overloaded. Static concurrency limits are easy to implement, but they require continuous tuning. What if a service could adjust its concurrency limit automatically based on how the dependency behaves? That's the idea behind adaptive concurrency limiting. Instead of relying entirely on a predefined number, the system observes request latency and failure signals, estimates when queueing is happening, and adjusts the number of requests it allows in flight. In this article, we'll explore: - Why fixed concurrency limits eventually become difficult to maintain - The relationship between concurrency, throughput, and latency - How TCP congestion control inspired adaptive concurrency algorithms - How to implement a latency-aware concurrency limiter in Go - How to integrate it into an HTTP client - How adaptive limits interact with backpressure, bulkheads, and retry budgets - What to monitor and test before using adaptive concurrency in production 1. The Problem with Fixed Concurrency Limits Imagine a Go service that calls an external market data API. You initially configure a concurrency limit of 50. limiter := NewFixedLimiter(50) Under normal traffic, the dependency responds in 40ms. Everything looks healthy. Then traffic increases. The dependency starts processing requests more slowly. Its response time grows to 150ms, then 300ms. Your service still allows 50 concurrent requests. The problem is that concurrency and latency are connected. When requests take longer to complete, existing requests occupy concurrency slots for longer. New requests accumulate, queues grow, and latency increases further. A fixed limit cannot automatically distinguish between: - A dependency that has additional capacity - A dependency approaching saturation - A dependency already experiencing queueing - A dependency recovering from an overload You can increase the limit manually, but that risks overloading the dependency. You can decrease it, but that might unnecessarily reduce throughput. The challenge is finding a concurrency level that allows high throughput without creating excessive queueing. 2. Why Concurrency Matters More Than RPS Alone Rate limiting and concurrency limiting solve different problems. A rate limiter controls how many requests can start during a period. A concurrency limiter controls how many requests can be in progress simultaneously. Consider two situations: | Scenario | Request rate | Average latency | Approx. concurrency | |---|---:|---:|---:| | Healthy dependency | 500 req/s | 40ms | 20 | | Slow dependency | 500 req/s | 200ms | 100 | Both scenarios have the same incoming request rate. But the second requires five times as many concurrent requests to sustain that throughput. This relationship is described by Little's Law: [ L = \lambda W ] Where: - (L) is the average number of items in the system - (\lambda) is the average arrival rate - (W) is the average time spent in the system For request processing: [ Concurrency \approx Throughput \times Latency ] This explains why a fixed RPS limit is not always sufficient to protect a dependency. A service can receive an acceptable number of requests per second while still accumulating too many in-flight operations because response times have increased. Adaptive concurrency focuses on this relationship. 3. The Idea Behind Adaptive Concurrency The central idea is straightforward: 1. Measure request latency. 2. Maintain an estimate of normal, unloaded latency. 3. Compare current latency against that baseline. 4. Increase concurrency when the dependency has spare capacity. 5. Reduce concurrency when latency indicates queueing or failures. This resembles congestion control in TCP. TCP does not blindly send unlimited data. It maintains a congestion window and adjusts how much data can be in flight based on network conditions. Adaptive concurrency applies a similar idea to service requests. The concurrency limit becomes a dynamic window. <figure> <img src="dummyimage.com" alt="Adaptive concurrency control loop" /> </figure> There are several algorithms worth understanding. AIMD Additive Increase, Multiplicative Decrease. - Increase the limit gradually when requests succeed. - Reduce it significantly when failures or overload signals appear. For example: Current limit: 50 Successful samples: 50 → 51 → 52 → 53 Overload detected: 53 → 37 AIMD is relatively simple and works well when the system has reliable failure signals. However, it may react only after a dependency starts rejecting requests or timing out. Vegas Vegas uses latency as an early signal of congestion. It estimates queueing using the difference between observed latency and a no-load baseline. A simplified queue estimate is: [ Q = L \times \left(1 - \frac{RTT_{baseline}}{RTT_{current}}\right) ] Where: - (Q) is estimated queue size - (L) is the current concurrency limit - (RTT_{baseline}) is the observed no-load latency - (RTT_{current}) is the current measured latency When latency stays close to the baseline, the algorithm can increase concurrency. When latency grows significantly, it reduces the limit. Gradient2 Gradient2 compares short-term and long-term latency averages to detect changes in the latency trend. It is useful when latency baselines drift over time or workloads are bursty. Instead of relying only on a minimum latency measurement, it uses smoothed averages to identify growing queueing pressure. For a deeper implementation reference, Netflix's concurrency-limits project is worth exploring: Netflix/concurrency-limits For this article, we'll build a smaller Vegas-inspired controller. It is an educational implementation, not a complete reproduction of Netflix's Vegas or Gradient2 algorithms. 4. Building an Adaptive Limiter in Go Let's implement a limiter that: - Tracks in-flight requests - Enforces a dynamic concurrency limit - Measures request duration - Collects latency samples in windows - Estimates queueing using a baseline latency - Increases or decreases the limit based on observed behavior We'll use a fail-fast admission strategy. When the current concurrency limit is reached, the limiter rejects the new request instead of allowing an unbounded queue to form. Step 1: Define the limiter package adaptive import ( "errors" "sync" "time" ) var ErrLimitExceeded = errors.New( "adaptive concurrency limit exceeded", ) type Limiter struct { mu sync.Mutex limit int minLimit int maxLimit int inflight int peak int baseline time.Duration sampleCount int sampleTotal time.Duration dropped bool windowSize int alpha float64 beta float64 } func New( initialLimit int, minLimit int, maxLimit int, ) *Limiter { if minLimit < 1 || initialLimit < minLimit || maxLimit < initialLimit { panic("invalid concurrency limits") } return &Limiter{ limit: initialLimit, minLimit: minLimit, maxLimit: maxLimit, windowSize: 20, alpha: 2, beta: 4, } } The limiter maintains two important pieces of information: Current limit: The maximum number of requests allowed simultaneously. In-flight count: The number of requests currently executing. We also maintain latency statistics and a baseline. The windowSize determines how many completed requests we collect before updating the concurrency limit. Using a window helps avoid changing the limit after every individual request. Step 2: Acquire a permit The limiter needs to decide whether a new request can execute. func (l *Limiter) Acquire() (func(bool), error) { l.mu.Lock() if l.inflight >= l.limit { l.mu.Unlock() return nil, ErrLimitExceeded } l.inflight++ if l.inflight > l.peak { l.peak = l.inflight } l.mu.Unlock() startedAt := time.Now() var once sync.Once done := func(dropped bool) { once.Do(func() { elapsed := time.Since(startedAt) l.mu.Lock() defer l.mu.Unlock() l.inflight-- l.recordSample(elapsed, dropped) }) } return done, nil } The returned function releases the permit and reports the outcome. For example: done, err := limiter.Acquire() if err != nil { return err } defer done(false) return callDependency() If the operation experiences a dependency-level overload signal, we can report that separately: done, err := limiter.Acquire() if err != nil { return err } defer func() { done(isOverloadError(err)) }() err = callDependency() return err The sync.Once protects against accidentally releasing the same permit twice. The caller must invoke the returned function exactly once in normal control flow. The sync.Once makes repeated invocations harmless. Step 3: Record latency samples Now we need to collect completed request samples. func (l *Limiter) recordSample( latency time.Duration, dropped bool, ) { l.sampleCount++ l.sampleTotal += latency if dropped { l.dropped = true } if l.sampleCount < l.windowSize { return } average := l.sampleTotal / time.Duration(l.sampleCount) l.updateLimit(average) l.sampleCount = 0 l.sampleTotal = 0 l.dropped = false l.peak = 0 } Instead of adjusting concurrency after every request, we wait until a complete sample window is collected. This reduces sensitivity to individual latency spikes. A production implementation may use more sophisticated rolling windows, exponential moving averages, or percentile-based measurements. Step 4: Implement the control algorithm Now comes the interesting part. We compare the current average latency against the baseline. func (l *Limiter) updateLimit( currentLatency time.Duration, ) { if currentLatency <= 0 { return } if l.baseline == 0 || currentLatency < l.baseline { l.baseline = currentLatency return } if l.dropped { l.decrease() return } queueEstimate := float64(l.limit) * (1 - float64(l.baseline)/float64(currentLatency)) // Increase only when the previous window // demonstrated meaningful concurrency usage. if l.peak*2 < l.limit { return } switch { case queueEstimate < l.alpha: l.increase() case queueEstimate > l.beta: l.decrease() } } The queue estimate is an approximation of the queueing pressure observed at the dependency. When current latency is close to the baseline, the estimated queue remains small. When current latency increases, the estimated queue grows. We use two thresholds: - alpha: Below this value, the dependency appears to have spare capacity. - beta: Above this value, queueing pressure is considered too high. The gap between the thresholds creates a stability region where the limit does not change. This helps avoid unnecessary oscillation. Step 5: Increase and decrease concurrency The final methods enforce the configured boundaries. func (l *Limiter) increase() { if l.limit < l.maxLimit { l.limit++ } } func (l *Limiter) decrease() { next := int(float64(l.limit) * 0.7) if next < l.minLimit { next = l.minLimit } l.limit = next } The limiter uses: - Additive increase: Grow gradually. - Multiplicative decrease: Reduce more aggressively when congestion appears. This is intentionally conservative. Increasing the limit by one allows the dependency to demonstrate additional capacity without creating a large traffic spike. Reducing the limit by 30% provides a faster response to overload. These values are configuration choices, not universal production defaults. Step 6: Expose the current limit We should expose the current state for observability. func (l *Limiter) Stats() ( limit int, inflight int, baseline time.Duration, ) { l.mu.Lock() defer l.mu.Unlock() return l.limit, l.inflight, l.baseline } Now we can inspect the limiter without exposing its internal synchronization. For example: limit, inflight, baseline := limiter.Stats() log.Printf( "adaptive limiter: limit=%d inflight=%d baseline=%s", limit, inflight, baseline, ) 5. Integrating the Limiter with an HTTP Client Let's apply the limiter to an external dependency. type MarketDataClient struct { httpClient *http.Client limiter *adaptive.Limiter } func (c *MarketDataClient) Fetch( ctx context.Context, symbol string, ) error { done, err := c.limiter.Acquire() if err != nil { return err } var dropped bool defer func() { done(dropped) }() req, err := http.NewRequestWithContext( ctx, http.MethodGet, "https://api.example.com/market/"+symbol, nil, ) if err != nil { return err } resp, err := c.httpClient.Do(req) if err != nil { dropped = isDependencyOverload(err) return err } defer resp.Body.Close() if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= http.StatusServiceUnavailable { dropped = true return fmt.Errorf( "market data dependency returned %d", resp.StatusCode, ) } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf( "unexpected status: %d", resp.StatusCode, ) } return nil } The example classifies overload responses separately from other errors. This distinction matters. A validation error, for example, should not necessarily cause the adaptive limiter to reduce its concurrency limit. Likewise, a network timeout might be an overload signal in one system but a network infrastructure issue in another. The failure classifier should reflect the dependency's actual behavior. The HTTP client should also have appropriate timeouts and connection settings. An adaptive concurrency limiter is not a replacement for transport-level protection. 6. What Happens During an Overload? Let's walk through a simplified scenario. Assume the limiter starts with a concurrency limit of 50. The dependency initially responds in 40ms. As traffic increases, the limiter observes the following: | Stage | Average latency | Concurrency limit | |---|---:|---:| | Initial | 40ms | 50 | | Increased traffic | 45ms | 51 | | Higher utilization | 55ms | 52 | | Queueing begins | 120ms | 36 | | Dependency recovers | 45ms | Gradually increases | The important behavior is not the exact numbers. The limiter detects that latency is increasing relative to the baseline and reduces the amount of work it allows in flight. As the dependency recovers, the limiter can gradually explore additional capacity again. This is a feedback control loop. However, it is important to understand that the limiter does not instantly discover the perfect concurrency limit. It continuously estimates a useful operating point based on observed signals. 7. Adaptive Concurrency vs. Other Resilience Patterns Adaptive concurrency is not a replacement for the patterns we've discussed in previous articles. It complements them. | Mechanism | Responsibility | |---|---| | Backpressure | Controls incoming work | | Load shedding | Rejects work when capacity is exceeded | | Bulkhead | Isolates resources and dependencies | | Retry budget | Limits additional retry traffic | | Adaptive concurrency | Adjusts the concurrency ceiling based on observed conditions | Consider a service that communicates with three dependencies: - Market Data API - Portfolio Database - Reporting Service Each dependency has different latency characteristics and capacity. Using one global concurrency limit would couple their behavior. A better approach is to create an independent adaptive limiter for each dependency. type DependencyLimiters struct { MarketData *adaptive.Limiter Portfolio *adaptive.Limiter Reporting *adaptive.Limiter } This gives each dependency its own feedback loop. If the reporting service becomes slow, its concurrency limit can decrease without automatically reducing the capacity available to market data requests. This is where adaptive concurrency and bulkheads work particularly well together. 8. Important Production Considerations The example implementation demonstrates the core idea, but a production adaptive limiter needs additional safeguards. Baseline latency can become stale A latency baseline is only useful when it represents a reasonably unloaded system. If the baseline is established while the dependency is already overloaded, the algorithm may incorrectly treat unhealthy latency as normal. Production implementations should carefully manage baseline updates and resets. Traffic must be representative A dependency may expose different latency characteristics for different operations. For example: - Reading a cached market price - Fetching historical prices - Calculating a portfolio - Executing a database aggregation Combining all these operations into one latency signal can produce misleading results. Consider separate limiters or traffic partitions when workloads have significantly different behavior. The algorithm needs enough samples Adaptive control depends on observations. A low-traffic service might not generate enough samples to update its limit reliably. A new instance also needs a warm-up strategy. Slow start, conservative initial limits, and minimum sample requirements can help. Avoid uncontrolled oscillation A limiter that constantly increases and decreases its capacity can introduce instability. Useful techniques include: - Sampling windows - Exponential smoothing - Separate increase and decrease thresholds - Bounded increase and decrease rates - Minimum and maximum concurrency limits - Jitter when multiple independent clients share a dependency Local limits multiply across replicas This is especially important in distributed deployments. Imagine ten replicas, each with an adaptive concurrency limit of 100. The dependency may receive up to 1,000 concurrent requests. Each instance is making decisions based on its own observations. The aggregate behavior is not automatically coordinated. Local adaptive concurrency is often sufficient for protecting individual instances and reacting to local conditions. But strict global dependency quotas may require a shared dispatcher, centralized coordination, or provider-side quota enforcement. Do not confuse concurrency with connection limits An HTTP transport's connection pool and an application's concurrency limiter protect different resources. A request may hold an application permit while waiting for a connection. Conversely, HTTP/2 can multiplex multiple requests over one connection. Configure both based on the actual workload. 9. Testing Adaptive Concurrency Adaptive algorithms should be tested as control systems, not only as ordinary utility functions. At minimum, test these behaviors: - The initial concurrency limit is respected. - Requests are rejected when the current limit is reached. - Permits are released exactly once. - Successful low-latency windows allow gradual growth. - Rising latency causes the limit to decrease. - Overload signals trigger a decrease. - The limit never exceeds its configured maximum. - The limit never falls below its configured minimum. - Concurrent acquisitions are race-safe. - Insufficient sample windows do not trigger premature changes. Run the race detector: go test -race ./... Also consider deterministic algorithm tests. Instead of using real HTTP requests and sleeping for arbitrary durations, separate the control algorithm from the admission mechanism. Feed known latency samples into the controller and verify the resulting limit. For example: Baseline: 40ms Window 1: 42ms → increase Window 2: 45ms → increase Window 3: 120ms → decrease Window 4: 130ms → decrease Window 5: 45ms → gradual recovery This makes algorithm behavior easier to reproduce and debug. Finally, run load tests with changing dependency latency. A useful test should simulate: 1. A healthy dependency. 2. Gradually increasing latency. 3. Sudden overload. 4. Partial recovery. 5. A complete dependency outage. Measure throughput, p95/p99 latency, rejection rate, and concurrency limit changes. The goal is not simply to maximize throughput. The goal is to maintain useful throughput without allowing latency and queueing to grow uncontrollably. 10. Observability An adaptive limiter introduces a new operational component. You need visibility into how it makes decisions. Useful metrics include: | Metric | Type | Purpose | |---|---|---| | adaptive_concurrency_limit | Gauge | Current concurrency ceiling | | adaptive_inflight_requests | Gauge | Active requests | | adaptive_request_latency_seconds | Histogram | Observed dependency latency | | adaptive_rejected_total | Counter | Requests rejected at admission | | adaptive_limit_increase_total | Counter | Limit increase events | | adaptive_limit_decrease_total | Counter | Limit decrease events | | adaptive_baseline_latency_seconds | Gauge | Estimated unloaded latency | These metrics help answer important questions: - Is the limiter constantly operating at its maximum? - Is the concurrency limit frequently decreasing? - Is the dependency latency increasing? - Are requests being rejected before reaching the dependency? - Is the limiter recovering too slowly? A particularly useful dashboard plots three signals together: 1. Current concurrency limit 2. In-flight requests 3. Dependency latency Looking at them independently can hide the relationship between capacity and latency. Avoid high-cardinality labels such as raw URLs, user IDs, and request IDs. 11. When Should You Use Adaptive Concurrency? Adaptive concurrency is not necessary for every service. A fixed limit is often the better choice when: - Dependency capacity is stable and well understood. - Traffic patterns are predictable. - The service has low request volume. - The cost of an adaptive control loop is not justified. - A simple semaphore already provides sufficient protection. Adaptive concurrency becomes more attractive when: - Dependency capacity changes frequently. - Traffic is bursty or unpredictable. - Services scale horizontally. - Latency is a reliable indicator of queueing. - Manual concurrency tuning is difficult. - You need to protect dependencies before they start rejecting requests. Start with a fixed limit if it is sufficient. Introduce adaptive behavior when operational evidence shows that static configuration is becoming a limitation. Conclusion A fixed concurrency limit assumes that system capacity is relatively predictable. In distributed systems, that assumption does not always hold. Dependencies slow down, traffic changes, instances scale, and resource availability evolves. Adaptive concurrency provides a feedback loop that adjusts the amount of work allowed in flight based on observed latency and failure signals. In Go, the core mechanism is straightforward: track active requests, collect latency samples, estimate congestion, and adjust the concurrency ceiling. The difficult part is not implementing a semaphore. It is building a stable controller that reacts to real overload without constantly oscillating or making incorrect decisions from noisy measurements. Don't just limit how much work your service can handle. Let it learn when it should slow down.

Circuit breakers: when to stop calling a service Imagine a payments service that starts timing out at 14:02. Your app keeps sending requests. Each one waits two seconds and fails. By 14:04 the worker pool is full, and checkout is down because one dependency is sick. A circuit breaker stops the sending. After a set number of failures it returns an error without making the call. Your app keeps the capacity it would have spent on calls that cannot succeed. The calls that cannot succeed A breaker wraps a remote call and watches the result. Past a threshold it stops calling. Azure's architecture guide states the purpose: the pattern "helps prevent an application from repeatedly trying to run an operation that's likely to fail." Martin Fowler credits Michael Nygard's book Release It! with popularizing the pattern. In Fowler's account, the breaker trips once failures reach a threshold. Then "all further calls to the circuit breaker return with an error, without the protected call being made at all." You save the wait, not the error. A failing call still holds a connection and a worker until its timeout fires. Azure warns that blocked requests "might hold critical system resources, such as memory, threads, and database connections." Enough of them and unrelated parts of the app fail too. Example: a checkout service calls payments with a 2-second timeout. Payments hangs. At 50 checkout requests a second, 100 workers sit waiting on the same broken dependency, and the pool is gone in under a minute. !Call-path diagram: your app calls the breaker, which forwards the call to payments while closed. When the breaker is open, the forward call is crossed out and an error returns to the app at once. The breaker sits between the caller and the dependency; when it opens, the call stops there and the error comes back at once. Closed, open, half-open The breaker is a small state machine. Closed. Calls pass. The breaker counts failures inside a rolling window, a moving slice of time such as the last 10 seconds. Open. Calls fail at once. Nothing reaches the dependency, and a timer runs. Half-open. When the timer ends, a few trial calls pass. A success closes the breaker and resets the count. A failure reopens it and restarts the timer. Azure explains why the third state earns its place. A recovering service "might be able to support a limited volume of requests until the recovery is complete." A flood of work can knock it over again. Failures open the breaker, the cooldown leaves it half-open, and the trial call decides the next state. What trips it Start with a count of failures in a window. Azure describes a counter that resets at intervals, so the occasional error does not open the circuit. The threshold trips only when enough failures land inside the period. Fowler suggests a rate instead: open at a 50% failure rate. Different errors can have different limits, such as "a threshold of 10 for timeouts but 3 for connection failures." Count only errors that say the dependency is unhealthy. A 404 for a missing record, or a rejected input, is a normal answer. Fowler is direct: "Not all errors should trip the circuit." Scope the breaker to one dependency, often to one operation. Wide scope causes a problem: a single breaker over every call to a service will open for a fault that affects one endpoint. a breaker with three states // A breaker for one dependency. The numbers are examples. function breaker(fn, { threshold = 3, cooldown = 30000 /* ms */ } = {}) { let failures = 0; let openedAt = 0; return async (...args) => { const open = openedAt > 0 && Date.now() - openedAt < cooldown; if (open) throw new Error("circuit open"); try { const result = await fn(...args); failures = 0; openedAt = 0; return result; } catch (err) { if (++failures >= threshold) openedAt = Date.now(); throw err; } }; } The next call after the cooldown is the probe. A success resets the breaker. A failure starts the cooldown again. What the caller sees An open breaker returns an error at once. The caller needs a plan for that error. Fowler lists the choices: fail the operation, use a workaround, show stale data, or put the work on a queue for later. Example: a product page calls recommendations and inventory. With the recommendations breaker open, the page renders without the recommendations strip and keeps the buy button. The inventory call still runs. One broken dependency costs a strip, not the sale. Hystrix, Netflix's breaker library, calls this fallback and graceful degradation. It also rejects a call when its pool of threads is full, instead of queueing work that will wait. Where the breaker lives You can put the breaker in the client. Hystrix, Netflix's library, and resilience4j, a Java library, wrap each call in process. The breaker knows which operation it protects, so per-endpoint thresholds are possible. You can put it in the network. Envoy's outlier detection is passive health checking: it removes hosts that fail or slow down from the load-balancing set. Envoy caps how many hosts it ejects, and each repeat lengthens the time a host stays out. The two see different things. A proxy sees host health, not what the operation means. A client library sees the operation, but each instance keeps its own count unless you share state. It does not fix the service A breaker protects your capacity. It does not repair the dependency, and it does not make a failed call succeed. The half-open probe exists to notice recovery, nothing more. Traffic returns when the breaker closes. If every instance probes at the same moment, the recovering service gets a wave. Stagger the probes, or let one instance probe first. Order matters with retries. Retries must respect the breaker. Azure says retry logic must watch what the breaker returns. Retries should stop "if the circuit breaker indicates that a fault isn't transient." Log every state change with the failure count that caused it. An open breaker is a signal about the dependency. Fowler argues that operations staff should be able to trip or reset a breaker by hand. Without a breaker, the slowest dependency decides how much of your capacity it gets. With one, you decide how many calls to waste before you stop. Takeaway A timeout bounds one call. A breaker bounds the number of calls, and protects your capacity, not the dependency. Common questions Is a circuit breaker the same as a timeout? No. A timeout bounds one call. A breaker stops new calls to a dependency that keeps failing. Use both: the timeout ends the call, and the breaker ends the attempts. Which errors should open it? Only errors that mean the dependency is unhealthy: timeouts, connection errors, and 5xx responses. A 404 or a rejected input is a normal result and should not count. What happens to calls already in flight? They finish or hit their timeout. The breaker only blocks new calls. Sources - Azure Architecture Center: Circuit Breaker pattern - Martin Fowler: CircuitBreaker - Netflix Hystrix wiki - Envoy: Outlier detection --- _Originally published at awahab.dev._

VMAFなんてあるんだ初めて知った Netflix/vmaf 作ったことないジャンルの開発は色んなことをしれて面白い

@Swizec you can actually read every line. its not hard when the code is readable. i keep posting this example to folks (it might not be the best-architected, but oh wow is it readable) - Netflix/dispatch

Netflix vmaf: Perceptual video quality assessment based on multi-method fusion Netflix/vmaf (news.ycombinator.com)

Netflix vmaf: Perceptual video quality assessment based on multi-method fusion Netflix/vmaf (news.ycombinator.com)

Netflix vmaf: Perceptual video quality assessment based on multi-method fusion

VMAF is super cool, although not the end-all-be-all of video quality measurement for sure. I'm currently evaluating switching over to the new v1.0 models. The (unsolicited) slop PRs on that repository are a sight to behold, though. The same slopper posted an aggressive response to my PR: vmaf#1586