Hedged Requests in Go: Reducing Tail Latency Without Overloading Dependencies
Most requests in a distributed system finish quickly.
A few don't.
Your API might have a median response time of 40ms, while its p99 latency reaches 900ms. Most users get a fast response, but a small percentage experience significantly longer waits.
These slow requests are particularly problematic in systems that depend on multiple services.
A single slow dependency can delay an entire operation, even when every other component is responding normally.
One way to address this problem is request hedging.
Instead of waiting indefinitely for an unusually slow request, the client starts another copy after a configured delay. Whichever request succeeds first wins, and the remaining requests are canceled.
This technique can significantly reduce tail latency, but it comes with an important trade-off: it generates additional traffic.
If every slow request triggers another request without limits, the mechanism designed to improve latency can overload the dependency it's trying to protect.
In this article, we'll explore:
- What tail latency is and why averages hide it
- How hedged requests work
- The difference between hedging and retries
- How to implement hedged requests in Go
- How to cancel losing requests safely
- How to integrate hedging with retry budgets and adaptive concurrency
- When hedging helps and when it makes things worse
- What to monitor and test in production
1. The Problem: Tail Latency
Let's say an API receives 10,000 requests.
Its response-time distribution looks like this:
| Percentile | Response time |
|---|---:|
| p50 | 40ms |
| p90 | 75ms |
| p95 | 120ms |
| p99 | 900ms |
| p99.9 | 2.5s |
Looking only at the average or median would suggest that the service is performing well.
But 1% of 10,000 requests means 100 requests take at least 900ms.
For latency-sensitive applications, these outliers matter.
Consider an API that aggregates data from five independent services:
API Request
|
+-----------+-----------+
| | |
Service A Service B Service C
| | |
Service D Service E |
| | |
+-----------+-----------+
|
Final Response
If the API needs all five responses, its total latency is constrained by the slowest dependency.
Even if four services respond in 30ms, one service taking 800ms can delay the entire operation.
This is one reason tail latency becomes increasingly important as distributed systems grow.
Jeff Dean and Luiz André Barroso discuss this problem in their influential paper, The Tail at Scale. They describe techniques for building responsive services despite unpredictable latency in individual components.
Reference: research.google
2. What Are Hedged Requests?
A hedged request is a duplicate request sent while the original request is still in progress.
The client does not immediately send multiple copies.
Instead, it waits for a configurable period.
If the original request hasn't completed, the client sends another request, ideally to a different replica.
The first successful response wins.
Time
|
| Primary request
| ────────────────────────────────>
|
| Wait: 100ms
|
| Hedge request
| ─────────────────────>
|
| Hedge response: 160ms
| Primary response: 450ms
|
| Return hedge response
| Cancel primary request
|
The important detail is that the second request starts before the first one fails.
This is fundamentally different from a traditional retry.
A retry waits for an unsuccessful attempt before trying again.
A hedge assumes that an unusually slow request might be an outlier and starts another attempt proactively.
Why does this work?
Imagine two replicas:
- Replica A is experiencing temporary contention.
- Replica B is healthy.
The primary request goes to A.
After 100ms, it still hasn't completed, so the client sends a hedge to B.
If B responds in 60ms, the client returns that response instead of waiting for A.
The total latency becomes approximately 160ms instead of 450ms.
The original request might still finish later, but its result is no longer needed.
3. Hedging vs. Retrying
These mechanisms are related, but they solve slightly different problems.
| Mechanism | When the next attempt starts | Primary goal |
|---|---|---|
| Retry | After an attempt fails | Recover from transient failures |
| Hedging | While an attempt is still running | Reduce tail latency |
| Timeout | After a deadline expires | Bound waiting time |
| Circuit breaker | When a dependency is unhealthy | Stop sending harmful traffic |
A retry responds to failure.
A hedge responds to excessive waiting.
This distinction matters because hedging can increase downstream load even when the original request would eventually succeed.
For that reason, hedging should generally be limited to safe, idempotent operations, especially reads.
4. Choosing the Hedging Delay
The hedging delay is one of the most important configuration decisions.
If the delay is too short, the client duplicates too many requests.
If it is too long, the hedge provides little latency improvement.
A reasonable starting point is to use a measured latency percentile for a particular operation.
For example:
| Percentile | Latency |
|---|---:|
| p50 | 40ms |
| p90 | 70ms |
| p95 | 100ms |
| p99 | 450ms |
A 100ms hedging delay means that requests exceeding the p95 threshold become candidates for hedging.
Under stable conditions, approximately 5% of requests may reach that threshold.
This does not mean exactly 5% of requests will generate a hedge: requests can complete at the threshold boundary, and actual behavior depends on the latency distribution and scheduling.
The important principle is to trigger hedging in the tail, not in the middle of normal traffic.
The original Tail at Scale paper discusses using a high-percentile latency threshold to limit duplicate traffic while reducing extreme latency. Its examples demonstrate why carefully selected hedging delays can achieve substantial tail-latency improvements with relatively modest additional load.
Source: andrew.cmu.edu
5. Implementing Hedged Requests in Go
Let's build a small hedging executor.
Our implementation will support:
- An immediate primary request
- A configurable hedging delay
- A maximum number of attempts
- Context cancellation
- First-successful-response semantics
- Retryable error classification
- An optional shared retry budget
We'll use Go generics so the executor can return any response type.
Step 1: Define the policy
package hedge
import (
"context"
"errors"
"fmt"
"time"
)
var (
ErrInvalidPolicy = errors.New("invalid hedging policy")
ErrBudgetDenied = errors.New("hedging budget exhausted")
)
type Budget interface {
TryTake() bool
}
type Policy struct {
Delay time.Duration
MaxAttempts int
Budget Budget
ShouldHedge func(error) bool
}
type result[T any] struct {
value T
err error
}
The Budget interface allows us to reuse the retry budget implementation from the previous article.
The original request does not consume a hedge token.
Every additional request does.
We also use ShouldHedge to distinguish errors that should terminate the operation from errors that may allow another attempt.
For example, a validation error should not trigger another request.
Step 2: Implement the executor
func Do[T any](
ctx context.Context,
policy Policy,
operation func(context.Context, int) (T, error),
) (T, error) {
var zero T
if policy.Delay <= 0 ||
policy.MaxAttempts < 1 ||
policy.MaxAttempts > 3 ||
operation == nil ||
policy.ShouldHedge == nil {
return zero, ErrInvalidPolicy
}
if err := ctx.Err(); err != nil {
return zero, err
}
hedgeCtx, cancel := context.WithCancel(ctx)
defer cancel()
results := make(chan result[T], policy.MaxAttempts)
launched := 0
inFlight := 0
var lastErr error
startAttempt := func() {
launched++
inFlight++
attempt := launched
go func() {
value, err := operation(hedgeCtx, attempt)
results <- result[T]{
value: value,
err: err,
}
}()
}
// The primary request starts immediately.
startAttempt()
var timer *time.Timer
var timerC <-chan time.Time
if policy.MaxAttempts > 1 {
timer = time.NewTimer(policy.Delay)
timerC = timer.C
defer timer.Stop()
}
for {
select {
case <-ctx.Done():
return zero, ctx.Err()
case res := <-results:
inFlight--
if res.err == nil {
// First successful response wins.
return res.value, nil
}
lastErr = res.err
if !policy.ShouldHedge(res.err) {
return zero, res.err
}
if inFlight == 0 &&
launched == policy.MaxAttempts {
return zero, lastErr
}
case <-timerC:
if launched >= policy.MaxAttempts {
timerC = nil
continue
}
if policy.Budget != nil &&
!policy.Budget.TryTake() {
// No more additional requests.
timerC = nil
if inFlight == 0 {
if lastErr != nil {
return zero, lastErr
}
return zero, ErrBudgetDenied
}
continue
}
startAttempt()
if launched < policy.MaxAttempts {
timer.Reset(policy.Delay)
} else {
timerC = nil
}
}
}
}
Let's examine the important decisions.
The primary request starts immediately.
We don't wait for a timer before sending the first request.
Additional requests start only after the configured delay.
If the primary finishes successfully before the delay expires, the executor returns without sending a hedge.
The first successful response wins.
As soon as a successful result arrives, the function returns and cancels the shared context.
Errors are classified.
A fatal error can terminate the operation immediately. A retryable error can be treated as non-final while another attempt is still running.
The number of attempts is bounded.
The implementation intentionally limits the example to three total attempts, including the original request.
This is a conservative default for the example, not a universal limit.
Step 3: Use the executor
Imagine a service that reads market data from multiple replicas.
type Price struct {
Symbol string
Value float64
}
func fetchPrice(
ctx context.Context,
attempt int,
) (Price, error) {
replica := selectReplica(attempt)
return replica.Fetch(ctx, "AAPL")
}
We can now use the hedging executor:
policy := hedge.Policy{
Delay: 100 * time.Millisecond,
MaxAttempts: 2,
Budget: retryBudget,
ShouldHedge: isRetryableError,
}
price, err := hedge.Do(
ctx,
policy,
fetchPrice,
)
if err != nil {
return err
}
fmt.Printf(
"%s: %.2f\n",
price.Symbol,
price.Value,
)
The primary request goes to the first replica.
If it hasn't completed after 100ms, the executor checks the budget and starts another request.
If the second replica responds first, the result is returned and the first request is canceled.
The example assumes that selectReplica routes attempts to distinct, suitable replicas. Sending both copies to the same overloaded instance may provide little benefit.
6. Cancellation Is Not the Same as Stopping Work
This is one of the most important details in a hedging implementation.
When the winning request completes, the executor calls cancel() on the shared context.
The losing requests receive cancellation through their contexts.
But cancellation is cooperative.
It does not forcibly terminate a goroutine, an HTTP request, a database query, or server-side processing.
For example, a dependency may continue executing a query even after the client disconnects.
A correct operation should propagate its context:
func fetch(
ctx context.Context,
client *http.Client,
) error {
req, err := http.NewRequestWithContext(
ctx,
http.MethodGet,
"https://api.example.com/prices/AAPL",
nil,
)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
return nil
}
The server should also respect request cancellation where possible.
Otherwise, the client may stop waiting while the dependency continues doing unnecessary work.
This is especially important for expensive database queries and CPU-intensive operations.
7. Why a Hedge Budget Matters
Hedging creates additional traffic.
Imagine a service receiving 10,000 requests per second.
If 5% reach the hedging threshold, that could mean approximately 500 additional attempts per second.
Now imagine the dependency is already degraded.
More concurrent requests may increase queueing, which increases latency, which triggers more hedges.
The mechanism can create the same kind of amplification problem that we discussed in the Retry Budgets article.
This is why the hedge budget should be shared across requests.
A simple rule:
- The original request is always allowed, subject to normal admission controls.
- Each additional hedge requires a token.
- Successful normal traffic replenishes the budget.
- If the budget is exhausted, no new hedges are launched.
For example:
budget := retry.NewRetryBudget(
20,
0.1,
)
The exact budget should be tuned according to the dependency's spare capacity and the service's traffic patterns.
The important part is that hedging is not an unlimited permission to duplicate requests.
It is an additional workload that must be controlled.
8. Hedging and Adaptive Concurrency
Hedging and adaptive concurrency work particularly well together, but they need to share a consistent view of downstream capacity.
Adaptive concurrency adjusts how many requests are allowed in flight based on observed dependency behavior.
Hedging creates additional in-flight requests.
If the adaptive limiter counts only original requests, hedged requests can bypass the intended concurrency ceiling.
That defeats the purpose of the limiter.
A safer integration is:
1. The primary request acquires a concurrency permit.
2. A hedge must acquire its own permit.
3. If no permit is available, the hedge is skipped.
4. The winning request returns its result.
5. Every attempt releases its own permit when it actually completes.
There is an important implementation detail here.
Do not release the losing request's concurrency permit merely because the caller stopped waiting for it.
The permit must remain held until the underlying operation actually completes or otherwise confirms that the protected work has stopped.
Otherwise, the limiter undercounts active downstream work.
This is also why a shared cancellation context alone is not enough to guarantee that hedged work has disappeared.
9. Hedging Requires Idempotency
Sending the same operation more than once is not always safe.
Consider these operations:
| Operation | Hedging suitability |
|---|---|
| Read a market price | Generally suitable |
| Read a portfolio | Generally suitable |
| Fetch cached configuration | Generally suitable |
| Create a payment | Unsafe without idempotency |
| Submit an order | Unsafe without deduplication |
| Send an email | May produce duplicates |
| Update an account balance | Unsafe without strong safeguards |
A hedge can reach two different servers before either has completed.
Both servers may execute the operation.
Canceling one request does not guarantee that its side effects are rolled back.
For operations with side effects, use an idempotency key or another server-side deduplication mechanism.
Even then, the server must correctly enforce the idempotency contract.
Hedging should generally be introduced first for read-only operations where duplicate execution is safe.
10. Where Should Hedged Requests Be Sent?
Sending a duplicate request to the same server is often ineffective.
If that server is experiencing CPU contention, a long queue, or an overloaded database connection pool, the second request may suffer from the same problem.
Ideally, the hedge should target an independent replica.
But independence is not guaranteed.
Two replicas may share:
- The same database
- The same network path
- The same cache
- The same storage system
- The same overloaded region
This is called correlated failure or correlated latency.
If both requests depend on the same struggling resource, hedging may increase load without meaningfully reducing latency.
Replica selection should consider failure domains and actual dependency topology.
11. Observability: Measure the Cost of Hedging
A hedging system needs more than a request-latency dashboard.
You need to understand whether the reduction in tail latency justifies the additional work.
Useful metrics include:
| Metric | Type | Purpose |
|---|---|---|
| hedged_requests_total | Counter | Number of additional attempts |
| hedge_wins_total | Counter | Requests where a hedge returned first |
| hedge_budget_denied_total | Counter | Hedges rejected by the budget |
| hedge_canceled_total | Counter | Losing attempts canceled by the client |
| hedge_attempts_total | Counter | Total attempts, including originals |
| hedge_delay_seconds | Histogram | Time before a hedge is launched |
| request_latency_seconds | Histogram | End-to-end request latency |
A useful derived metric is the hedge win rate:
[
HedgeWinRate =
\frac{HedgeWins}{HedgedRequests}
]
If hedges rarely win, you may be generating additional traffic without much benefit.
Also compare:
- p50 latency
- p95 latency
- p99 latency
- Downstream request volume
- Dependency CPU and memory usage
- Cancellation effectiveness
- Error rate
The goal is not to minimize p99 at any cost.
It is to improve tail latency while keeping the additional workload within the dependency's capacity.
12. Testing Hedged Requests
Concurrency behavior needs deterministic tests.
At minimum, test these scenarios:
- The primary request succeeds before the hedging delay.
- The primary request is slow and the hedge wins.
- The primary request fails with a retryable error.
- A fatal error cancels outstanding attempts.
- The hedge budget prevents additional requests.
- The parent context is canceled.
- The maximum number of attempts is respected.
- Losing operations receive cancellation.
- A losing operation that ignores cancellation does not release its concurrency permit prematurely.
A simple test for the winning hedge could use controlled channels instead of arbitrary sleeps:
func TestHedgeWins(t *testing.T) {
primaryStarted := make(chan struct{})
hedgeStarted := make(chan struct{})
ctx, cancel := context.WithTimeout(
context.Background(),
time.Second,
)
defer cancel()
policy := Policy{
Delay: 20 * time.Millisecond,
MaxAttempts: 2,
ShouldHedge: func(error) bool {
return true
},
}
result, err := Do(
ctx,
policy,
func(
ctx context.Context,
attempt int,
) (string, error) {
if attempt == 1 {
close(primaryStarted)
<-ctx.Done()
return "", ctx.Err()
}
close(hedgeStarted)
return "hedge response", nil
},
)
if err != nil {
t.Fatal(err)
}
if result != "hedge response" {
t.Fatalf(
"unexpected result: %s",
result,
)
}
<-primaryStarted
<-hedgeStarted
}
The test verifies that the second request can win and that the operation completes successfully.
In a complete test suite, also verify that the primary operation observes cancellation.
Run the race detector:
go test -race ./...
Finally, use load tests with a dependency that deliberately introduces variable latency.
A useful experiment compares:
1. Hedging disabled.
2. Hedging enabled with a conservative delay.
3. Hedging enabled with an aggressive delay.
4. Hedging enabled with a shared budget.
5. Hedging enabled with adaptive concurrency.
Compare both latency improvements and additional downstream traffic.
13. Production Checklist
Before introducing hedged requests into a production service:
- [ ] Hedge only safe, idempotent operations.
- [ ] Select a delay from measured latency distributions.
- [ ] Use a strict maximum attempt count.
- [ ] Require a shared budget for additional attempts.
- [ ] Ensure hedges target independent replicas where possible.
- [ ] Apply concurrency limits to every attempt.
- [ ] Propagate context cancellation.
- [ ] Ensure permits remain held while work is still active.
- [ ] Respect the original request deadline.
- [ ] Measure hedge win rate and additional traffic.
- [ ] Test correlated latency and dependency overload.
- [ ] Roll out gradually and compare against a no-hedging baseline.
Conclusion
Tail latency is one of the most difficult problems in distributed systems because a small number of unusually slow operations can dominate the response time of an entire request.
Hedged requests provide a way to reduce that impact.
By launching a second request after a carefully selected delay, a client can avoid waiting for an unusually slow replica and use the first successful response.
But hedging is not free.
Every additional request consumes capacity, and uncontrolled duplication can make an already degraded dependency even slower.
The right approach combines measured hedging delays, idempotent operations, cancellation, retry budgets, and adaptive concurrency limits.
The goal isn't to send more requests. It's to get the response sooner without making the system slower for everyone else.
---
References
- The Tail at Scale — Google Research
- Request Hedging — gRPC Documentation
- gRPC Retry and Hedging Design
Retry Storms: Simulate How a 30-Second Blip Becomes an Outage That Never Ends
- Book: System Design Pocket Guide: Fundamentals
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
The database fails over. Thirty seconds later it is back, healthy, with plenty of capacity for normal traffic. Your dashboards stay red for the next hour anyway. Every node is at 100% CPU and almost nothing succeeds. Someone restarts the database, which changes nothing. Someone else finally turns off the retries in the API gateway, and the whole system recovers in about a minute.
That shape has a name. In 2021, Nathan Bronson and three co-authors called it a metastable failure in a HotOS '21 paper: a trigger pushes the system into a bad state, and that state persists even when the trigger is removed. A follow-up study at OSDI '22, Metastable Failures in the Wild, went through public incident reports and found that at least 4 of 15 major AWS outages in the last decade were metastable failures.
Retries are one of the most common things that keep the bad state going. The simulator below reproduces the effect with numbers you can check by hand, then adds the fix that production RPC stacks ship: a retry budget.
The multiplication nobody adds up
Each retry policy looks harmless in isolation. "Try three times" is a sensible default for a service calling a flaky dependency. The problem is that your stack has several services, and each one has its own sensible default.
The AWS Builders' Library article on timeouts and retries walks through a five-deep stack with three retries at each layer and puts it plainly: if each layer retries independently, load on the database goes up 243x, "making it unlikely to ever recover." With three layers and three tries each, the bottom of the stack sees up to 3 × 3 × 3 = 27 attempts for every user request.
Here is why 27 is the exact count when every call fails. Layer A calls B up to 3 times. Each of those B calls makes up to 3 calls to C. Each of those makes up to 3 calls to the database. When the database fails every call, the attempts count up like an odometer, and the database sees all 27 before the user gets an error.
Your normal traffic might be 70% of the database's capacity. Multiply it by 27 and you need 19 databases. The fault that started it is gone, and the retries are now the outage.
Build the simulator
The model is deliberately small. A database serves 100 requests per second, oldest first. Users send 70 new requests per second. A caller waits one second, then gives up and retries. A request the caller abandoned stays in the database's queue, and the database still does the work when it gets there, which is how real servers behave without deadline propagation. For 30 seconds, the database serves nothing.
Because a 3-deep stack of 3 tries collapses to a flat count of 27 attempts when every call fails, the simulator models the whole stack as one caller with max_attempts = 27. It runs on plain Python 3.8 or later, no dependencies:
# retry_storm.py: one 30-second blip, three retry policies
import math
from collections import defaultdict, deque
CAPACITY = 100 # requests the database serves per second
BASE_LOAD = 70 # new user requests per second
TIMEOUT = 1 # seconds a caller waits before giving up
BLIP = range(60, 90) # 30 seconds with zero capacity
DURATION = 3600 # one simulated hour
def simulate(max_attempts, budget=None):
queue = deque() # FIFO of [enqueued, attempt, n]
by_tick = defaultdict(list) # enqueue tick -> its batches
due = defaultdict(list) # tick -> [(attempt, n)] retries
queued = useful = served = 0
peak_load = peak_queue = 0
last_bad = None
def enqueue(now, attempt, n):
nonlocal queued
batch = [now, attempt, n]
queue.append(batch)
by_tick[now].append(batch)
queued += n
for now in range(DURATION):
# 1. new requests, then the retries the budget allows
enqueue(now, 1, BASE_LOAD)
sent = BASE_LOAD
room = math.inf
if budget is not None:
room = round(BASE_LOAD * budget)
for attempt, n in due.pop(now, []):
n = min(n, room)
room -= n
sent += n
if n:
enqueue(now, attempt, n)
peak_load = max(peak_load, sent)
# 2. the database works through the oldest requests
cap = 0 if now in BLIP else CAPACITY
while cap and queue:
batch = queue[0]
take = min(cap, batch[2])
if now - batch[0] <= TIMEOUT:
useful += take
served += take
batch[2] -= take
queued -= take
cap -= take
if batch[2] == 0:
queue.popleft()
# 3. callers still waiting past the timeout give up
for batch in by_tick.pop(now - TIMEOUT, []):
attempt, n = batch[1], batch[2]
if n and attempt < max_attempts:
due[now + 1].append((attempt + 1, n))
peak_queue = max(peak_queue, queued)
if now >= BLIP.start and queued > CAPACITY * TIMEOUT:
last_bad = now
if last_bad == DURATION - 1:
recovered = None
else:
recovered = last_bad + 1 - BLIP.stop
return (recovered, peak_load / BASE_LOAD, peak_queue,
1 - useful / served)
def report(label, max_attempts, budget=None):
rec, load, peak_q, waste = simulate(max_attempts, budget)
when = "never" if rec is None else f"after {rec}s"
print(f"{label:<16} recovered {when:<11} "
f"peak load {load:4.1f}x peak queue {peak_q:>9,} "
f"wasted {waste:4.0%}")
report("no retries", 1)
report("3 layers x 3", 3 ** 3)
report("same + budget", 3 ** 3, budget=0.1)
A few things to know before you run it. Time moves in one-second ticks, and requests travel in batches so an hour of millions of queued requests stays fast. "Recovered" means the queue has dropped to one timeout's worth of work (100 requests) and stays there, because past that point a new request gets served before its caller gives up. "Wasted" is the share of the database's work that went to requests whose caller had already walked away.
Predict the numbers, then run it
Work each run out on paper first. That is the point of a model this small: if the output disagrees with your arithmetic, one of you has a bug, and finding it teaches you more than the run does.
No retries. During the blip, 30 seconds × 70 requests = 2,100 requests pile up. Afterwards the database drains them at 100 − 70 = 30 per second. 2,000 requests at 30 per second is roughly 67 seconds, so expect recovery a little over a minute after the blip ends. Peak load: 1.0x.
Three layers, three tries. Every request queued during the blip times out, and so does every retry, because each one lands at the back of a growing queue. After 26 rounds of retries, every second carries 70 new requests plus 26 × 70 retries, which is 1,890 requests per second against a capacity of 100. The queue grows by about 1,790 every second. It never drains, so expect recovered never, a peak load of 27.0x and a wasted-work figure close to 100%: the database spends the hour busy, almost entirely on answers nobody is waiting for. This is the metastable state. The blip ended at second 90, capacity came back, and the system stayed down because of its own retries.
Same stack plus a 10% budget. Retries can never exceed 10% of new requests, so arrivals are capped at 77 per second. The backlog after the blip is about 2,300, draining at 100 − 77 = 23 per second. That is around 95 to 100 seconds. Peak load: 1.1x.
Run python3 retry_storm.py and compare. The budget run recovers a little slower than the no-retry run, because retries still eat some capacity. That is the cost of the budget, and it is small. The difference between a minute and a half and never is the whole argument.
What a retry budget is
A retry budget caps retries as a fraction of real traffic, instead of capping them per request. Per-request limits ("try three times") say nothing about how many requests are failing at once. A budget does, because it is shared by every request a client sends.
This is not a new idea. The production stacks you already use ship it:
- The Google SRE book's chapter on handling overload describes two limits: up to three attempts per request, and a per-client budget where a request is only retried while the client's retry ratio is below 10%. The chapter says the budget reduces worst-case growth from "somewhere just below 3X" to "just 1.1x in the general case."
- gRPC's retry design (gRFC A6) has retry throttling as a token bucket. Every failed RPC removes one token, every success adds tokenRatio (0.1 in the example config), and retries stop when the count drops to half of maxTokens. In a storm where everything fails, the bucket empties fast and retries switch off by themselves.
- Finagle's RetryBudget by default lets about 20% of requests retry immediately, with a floor of 10 retries per second so quiet clients can still retry at all.
The gRPC and SRE versions also attack the layering problem from the other side. The SRE chapter says requests should only be retried at the layer immediately above the one rejecting them, and an overloaded backend can return an "overloaded; don't retry" error so the layers above stop multiplying. gRPC sends a grpc-previous-rpc-attempts header on retries, so a server can see that a request is already a retry.
One caveat about the simulator: it applies the budget to every extra attempt the database sees. A real stack only gets that if each layer runs its own budget (three layers at 10% gives at most 1.1³, about 1.33x) or if retries are tagged so lower layers can count them. Either works. A budget in one service and blind retries in the other two does not.
Experiments worth running
Change one constant at a time and predict the result first.
- Set BASE_LOAD = 90. A 10% budget then puts 99 requests per second on a database that serves 100. One request per second of headroom against a backlog of about 2,950 is a drain of most of the simulated hour. Budgets need spare capacity to work. That is a capacity-planning problem.
- Set budget=0.2, Finagle's default ratio. Work out the drain rate before you run it.
- Make the blip 5 seconds instead of 30. Predict whether the naive run still tips over. In this model, a trigger tips the system over once the backlog grows past one timeout's worth of work, however small the trigger looks.
- Add a queue cap in enqueue that rejects requests past, say, 500 queued. Rejections fail fast instead of wasting work, which changes the numbers. Then check whether retries alone can still keep the system down.
Backoff and jitter deserve a mention here too, because they are the first thing people reach for. Exponential backoff spreads the 27 attempts over a longer window, which lowers the per-second peak. It does not change the count: each failing request still sends 27 attempts to the bottom of the stack. The budget is what limits the total.
Add up the retry policies along your deepest call path and multiply them. If the product is bigger than the headroom your bottom dependency has, a short blip can turn into a long outage, and the place to fix it is the retry policy.
---
If this was useful
Retry budgets only work next to timeouts, rate limits and capacity headroom. System Design Pocket Guide: Fundamentals walks through those building blocks and the availability trade-offs behind them. If this simulator made you want to multiply out your own call paths, start there.
1 comment
Designing gRPC APIs for Mobile: Deadline Propagation, Channel Pooling, and the Retry Budget That Prevents Thundering Herds
---
title: "gRPC for Mobile: Deadlines, Channel Pools, and the Retry Budget That Stops Thundering Herds"
published: true
description: "Master gRPC for mobile clients: propagate deadlines through your service mesh, safely pool channels on Android and iOS, and cap retry traffic before a thundering herd takes down your backend."
tags: android, ios, mobile, api
canonical_url: https://mvpfactory.co/blog/grpc-mobile-deadlines-channel-pools-retry-budgets
---
What you will build
By the end of this guide, you will have a production-grade gRPC configuration for your mobile client: deadlines that cascade correctly through your service mesh, a channel pool that does not leak on Android or iOS, and a retry strategy that stays well-behaved when your network degrades. This is the exact config that would have prevented an 18-minute backend outage — which we will walk through to make the lessons concrete.
Prerequisites
- Familiarity with gRPC concepts (channels, stubs, service configs)
- An Android project using grpc-kotlin or an iOS project using grpc-swift
- A backend with at least two service layers (API gateway + downstream service)
---
The incident that changed how I think about gRPC
At 11pm on a Tuesday, a stadium Wi-Fi handoff triggered 400,000 simultaneous retries. The backend was down for 18 minutes. The root cause was three missing lines of config.
Most teams treat gRPC like a slightly faster REST call. It is not. The deadline, retry, and connection models are fundamentally different — and each interacts with the others in ways that only surface under load. Here is the minimal setup to get this right.
---
Step 1 — Deadline propagation: cascade or die
Here is the gotcha that will save you hours: a deadline is not a timeout. A timeout fires client-side and abandons the call. A deadline propagates through every hop of your service mesh via the grpc-timeout header, letting upstream services abandon work that will never reach the client anyway.
Set deadlines on the stub, not the channel:
// Android — set a 2.5s deadline on the stub, not the channel
val response = stub
.withDeadlineAfter(2500, TimeUnit.MILLISECONDS)
.getHealthData(request)
Your server deadline must always be shorter than your client deadline. Here is a cascade pattern I use in every project:
| Layer | Deadline | Why |
| :--- | :---: | :--- |
| Mobile client | 3000ms | User-facing tolerance |
| API gateway | 2700ms | Buffer for network RTT |
| Backend service | 2400ms | Buffer for internal hops |
| Database query | 1800ms | Leave room for retries |
A 300ms cascade buffer at each hop is conservative. On low-latency internal networks, 150ms is usually enough.
---
Step 2 — Channel pooling: size it or leak it
gRPC multiplexes requests over a single HTTP/2 connection per channel. One channel per backend endpoint is usually correct on mobile. Before you reach for a pool, understand the cost: each channel holds a TCP connection, negotiates TLS, and maintains a flow control window. On Android, an unclosed channel holds a wake lock. On iOS, it fights the system's network interface scheduler.
A practical ceiling is 2–4 channels per host. Here is the iOS setup:
// iOS — pool of 2 channels for the primary API host
let pool = GRPCChannelPool.with(
target: .host("api.example.com", port: 443),
transportSecurity: .tls(GRPCTLSConfiguration.makeClientDefault()),
eventLoopGroup: PlatformSupport.makeEventLoopGroup(loopCount: 2)
)
Two channels handle roughly 200 concurrent streams comfortably. Going beyond 4 per host rarely improves throughput — and it meaningfully increases your server-side connection count, which bites hard when running behind a service mesh with per-replica connection limits. Always close channels in onDestroy / deinit.
---
Step 3 — The retry budget: two mechanisms, not one
This is where most implementations break down. The docs do not make this clear, but there are two distinct concepts here, and conflating them is exactly how you end up with an 18-minute outage.
The client-side retryPolicy controls per-call behavior:
// gRPC service config — per-call retry behavior, client-side only
val serviceConfig = """{
"retryPolicy": {
"maxAttempts": 3,
"initialBackoff": "0.5s",
"maxBackoff": "10s",
"backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"]
}
}"""
This limits retries per call. It does not limit total retry traffic across your fleet. If 400,000 clients each retry 3 times simultaneously, you have multiplied your load by 3x at the worst possible moment.
Fleet-wide enforcement requires server-side controls. The most practical approach: track the ratio of retries (identifiable via the grpc-previous-rpc-attempts header) to total requests at your gateway, and shed excess retry traffic above 10% of total RPC volume. With this budget enforced at the gateway, your effective traffic multiplier under a partial outage is capped at 1.1x regardless of fleet size.
Exponential backoff with jitter is necessary. It is not sufficient.
---
Gotchas
- Deadline on the channel vs. the stub. Channel-level deadlines apply to every call for the lifetime of the channel. Stub-level deadlines are per-call. Always use the stub.
- Missing grpc-previous-rpc-attempts at the gateway. Without this header check, your gateway cannot distinguish organic traffic from a retry storm.
- Pool size creep. Teams see head-of-line blocking and double their pool size. Measure server-side connection count first — you are almost always hitting a concurrency limit, not a connection limit.
- iOS deinit timing. Swift ARC does not guarantee deinit timing under memory pressure. Explicitly close pool channels in viewDidDisappear or your scene lifecycle, not only in deinit.
---
Conclusion
Three things prevent the outage:
1. Deadlines on the stub, cascaded 10–15% tighter at each service layer so upstream services abandon work before the client times out.
2. Channel pool capped at 2–4 per host, always closed in onDestroy / deinit.
3. Client-side retryPolicy combined with a gateway-level retry budget — the config handles per-call behavior, the gateway enforces the fleet-wide ceiling.
If you are building a mobile app that communicates with users throughout the day — something like HealthyDesk, which sends break reminders and guided desk exercises to developers on a schedule — this pattern is exactly what keeps your notification and sync services standing when a large cohort of users reconnects simultaneously.
Relevant docs: gRPC retry design proposal (A6) · grpc-swift · grpc-kotlin
Mentions
Hedged Requests in Go: Reducing Tail Latency Without Overloading Dependencies Most requests in a distributed system finish quickly. A few don't. Your API might have a median response time of 40ms, while its p99 latency reaches 900ms. Most users get a fast response, but a small percentage experience significantly longer waits. These slow requests are particularly problematic in systems that depend on multiple services. A single slow dependency can delay an entire operation, even when every other component is responding normally. One way to address this problem is request hedging. Instead of waiting indefinitely for an unusually slow request, the client starts another copy after a configured delay. Whichever request succeeds first wins, and the remaining requests are canceled. This technique can significantly reduce tail latency, but it comes with an important trade-off: it generates additional traffic. If every slow request triggers another request without limits, the mechanism designed to improve latency can overload the dependency it's trying to protect. In this article, we'll explore: - What tail latency is and why averages hide it - How hedged requests work - The difference between hedging and retries - How to implement hedged requests in Go - How to cancel losing requests safely - How to integrate hedging with retry budgets and adaptive concurrency - When hedging helps and when it makes things worse - What to monitor and test in production 1. The Problem: Tail Latency Let's say an API receives 10,000 requests. Its response-time distribution looks like this: | Percentile | Response time | |---|---:| | p50 | 40ms | | p90 | 75ms | | p95 | 120ms | | p99 | 900ms | | p99.9 | 2.5s | Looking only at the average or median would suggest that the service is performing well. But 1% of 10,000 requests means 100 requests take at least 900ms. For latency-sensitive applications, these outliers matter. Consider an API that aggregates data from five independent services: API Request | +-----------+-----------+ | | | Service A Service B Service C | | | Service D Service E | | | | +-----------+-----------+ | Final Response If the API needs all five responses, its total latency is constrained by the slowest dependency. Even if four services respond in 30ms, one service taking 800ms can delay the entire operation. This is one reason tail latency becomes increasingly important as distributed systems grow. Jeff Dean and Luiz André Barroso discuss this problem in their influential paper, The Tail at Scale. They describe techniques for building responsive services despite unpredictable latency in individual components. Reference: research.google 2. What Are Hedged Requests? A hedged request is a duplicate request sent while the original request is still in progress. The client does not immediately send multiple copies. Instead, it waits for a configurable period. If the original request hasn't completed, the client sends another request, ideally to a different replica. The first successful response wins. Time | | Primary request | ────────────────────────────────> | | Wait: 100ms | | Hedge request | ─────────────────────> | | Hedge response: 160ms | Primary response: 450ms | | Return hedge response | Cancel primary request | The important detail is that the second request starts before the first one fails. This is fundamentally different from a traditional retry. A retry waits for an unsuccessful attempt before trying again. A hedge assumes that an unusually slow request might be an outlier and starts another attempt proactively. Why does this work? Imagine two replicas: - Replica A is experiencing temporary contention. - Replica B is healthy. The primary request goes to A. After 100ms, it still hasn't completed, so the client sends a hedge to B. If B responds in 60ms, the client returns that response instead of waiting for A. The total latency becomes approximately 160ms instead of 450ms. The original request might still finish later, but its result is no longer needed. 3. Hedging vs. Retrying These mechanisms are related, but they solve slightly different problems. | Mechanism | When the next attempt starts | Primary goal | |---|---|---| | Retry | After an attempt fails | Recover from transient failures | | Hedging | While an attempt is still running | Reduce tail latency | | Timeout | After a deadline expires | Bound waiting time | | Circuit breaker | When a dependency is unhealthy | Stop sending harmful traffic | A retry responds to failure. A hedge responds to excessive waiting. This distinction matters because hedging can increase downstream load even when the original request would eventually succeed. For that reason, hedging should generally be limited to safe, idempotent operations, especially reads. 4. Choosing the Hedging Delay The hedging delay is one of the most important configuration decisions. If the delay is too short, the client duplicates too many requests. If it is too long, the hedge provides little latency improvement. A reasonable starting point is to use a measured latency percentile for a particular operation. For example: | Percentile | Latency | |---|---:| | p50 | 40ms | | p90 | 70ms | | p95 | 100ms | | p99 | 450ms | A 100ms hedging delay means that requests exceeding the p95 threshold become candidates for hedging. Under stable conditions, approximately 5% of requests may reach that threshold. This does not mean exactly 5% of requests will generate a hedge: requests can complete at the threshold boundary, and actual behavior depends on the latency distribution and scheduling. The important principle is to trigger hedging in the tail, not in the middle of normal traffic. The original Tail at Scale paper discusses using a high-percentile latency threshold to limit duplicate traffic while reducing extreme latency. Its examples demonstrate why carefully selected hedging delays can achieve substantial tail-latency improvements with relatively modest additional load. Source: andrew.cmu.edu 5. Implementing Hedged Requests in Go Let's build a small hedging executor. Our implementation will support: - An immediate primary request - A configurable hedging delay - A maximum number of attempts - Context cancellation - First-successful-response semantics - Retryable error classification - An optional shared retry budget We'll use Go generics so the executor can return any response type. Step 1: Define the policy package hedge import ( "context" "errors" "fmt" "time" ) var ( ErrInvalidPolicy = errors.New("invalid hedging policy") ErrBudgetDenied = errors.New("hedging budget exhausted") ) type Budget interface { TryTake() bool } type Policy struct { Delay time.Duration MaxAttempts int Budget Budget ShouldHedge func(error) bool } type result[T any] struct { value T err error } The Budget interface allows us to reuse the retry budget implementation from the previous article. The original request does not consume a hedge token. Every additional request does. We also use ShouldHedge to distinguish errors that should terminate the operation from errors that may allow another attempt. For example, a validation error should not trigger another request. Step 2: Implement the executor func Do[T any]( ctx context.Context, policy Policy, operation func(context.Context, int) (T, error), ) (T, error) { var zero T if policy.Delay <= 0 || policy.MaxAttempts < 1 || policy.MaxAttempts > 3 || operation == nil || policy.ShouldHedge == nil { return zero, ErrInvalidPolicy } if err := ctx.Err(); err != nil { return zero, err } hedgeCtx, cancel := context.WithCancel(ctx) defer cancel() results := make(chan result[T], policy.MaxAttempts) launched := 0 inFlight := 0 var lastErr error startAttempt := func() { launched++ inFlight++ attempt := launched go func() { value, err := operation(hedgeCtx, attempt) results <- result[T]{ value: value, err: err, } }() } // The primary request starts immediately. startAttempt() var timer *time.Timer var timerC <-chan time.Time if policy.MaxAttempts > 1 { timer = time.NewTimer(policy.Delay) timerC = timer.C defer timer.Stop() } for { select { case <-ctx.Done(): return zero, ctx.Err() case res := <-results: inFlight-- if res.err == nil { // First successful response wins. return res.value, nil } lastErr = res.err if !policy.ShouldHedge(res.err) { return zero, res.err } if inFlight == 0 && launched == policy.MaxAttempts { return zero, lastErr } case <-timerC: if launched >= policy.MaxAttempts { timerC = nil continue } if policy.Budget != nil && !policy.Budget.TryTake() { // No more additional requests. timerC = nil if inFlight == 0 { if lastErr != nil { return zero, lastErr } return zero, ErrBudgetDenied } continue } startAttempt() if launched < policy.MaxAttempts { timer.Reset(policy.Delay) } else { timerC = nil } } } } Let's examine the important decisions. The primary request starts immediately. We don't wait for a timer before sending the first request. Additional requests start only after the configured delay. If the primary finishes successfully before the delay expires, the executor returns without sending a hedge. The first successful response wins. As soon as a successful result arrives, the function returns and cancels the shared context. Errors are classified. A fatal error can terminate the operation immediately. A retryable error can be treated as non-final while another attempt is still running. The number of attempts is bounded. The implementation intentionally limits the example to three total attempts, including the original request. This is a conservative default for the example, not a universal limit. Step 3: Use the executor Imagine a service that reads market data from multiple replicas. type Price struct { Symbol string Value float64 } func fetchPrice( ctx context.Context, attempt int, ) (Price, error) { replica := selectReplica(attempt) return replica.Fetch(ctx, "AAPL") } We can now use the hedging executor: policy := hedge.Policy{ Delay: 100 * time.Millisecond, MaxAttempts: 2, Budget: retryBudget, ShouldHedge: isRetryableError, } price, err := hedge.Do( ctx, policy, fetchPrice, ) if err != nil { return err } fmt.Printf( "%s: %.2f\n", price.Symbol, price.Value, ) The primary request goes to the first replica. If it hasn't completed after 100ms, the executor checks the budget and starts another request. If the second replica responds first, the result is returned and the first request is canceled. The example assumes that selectReplica routes attempts to distinct, suitable replicas. Sending both copies to the same overloaded instance may provide little benefit. 6. Cancellation Is Not the Same as Stopping Work This is one of the most important details in a hedging implementation. When the winning request completes, the executor calls cancel() on the shared context. The losing requests receive cancellation through their contexts. But cancellation is cooperative. It does not forcibly terminate a goroutine, an HTTP request, a database query, or server-side processing. For example, a dependency may continue executing a query even after the client disconnects. A correct operation should propagate its context: func fetch( ctx context.Context, client *http.Client, ) error { req, err := http.NewRequestWithContext( ctx, http.MethodGet, "https://api.example.com/prices/AAPL", nil, ) if err != nil { return err } resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() return nil } The server should also respect request cancellation where possible. Otherwise, the client may stop waiting while the dependency continues doing unnecessary work. This is especially important for expensive database queries and CPU-intensive operations. 7. Why a Hedge Budget Matters Hedging creates additional traffic. Imagine a service receiving 10,000 requests per second. If 5% reach the hedging threshold, that could mean approximately 500 additional attempts per second. Now imagine the dependency is already degraded. More concurrent requests may increase queueing, which increases latency, which triggers more hedges. The mechanism can create the same kind of amplification problem that we discussed in the Retry Budgets article. This is why the hedge budget should be shared across requests. A simple rule: - The original request is always allowed, subject to normal admission controls. - Each additional hedge requires a token. - Successful normal traffic replenishes the budget. - If the budget is exhausted, no new hedges are launched. For example: budget := retry.NewRetryBudget( 20, 0.1, ) The exact budget should be tuned according to the dependency's spare capacity and the service's traffic patterns. The important part is that hedging is not an unlimited permission to duplicate requests. It is an additional workload that must be controlled. 8. Hedging and Adaptive Concurrency Hedging and adaptive concurrency work particularly well together, but they need to share a consistent view of downstream capacity. Adaptive concurrency adjusts how many requests are allowed in flight based on observed dependency behavior. Hedging creates additional in-flight requests. If the adaptive limiter counts only original requests, hedged requests can bypass the intended concurrency ceiling. That defeats the purpose of the limiter. A safer integration is: 1. The primary request acquires a concurrency permit. 2. A hedge must acquire its own permit. 3. If no permit is available, the hedge is skipped. 4. The winning request returns its result. 5. Every attempt releases its own permit when it actually completes. There is an important implementation detail here. Do not release the losing request's concurrency permit merely because the caller stopped waiting for it. The permit must remain held until the underlying operation actually completes or otherwise confirms that the protected work has stopped. Otherwise, the limiter undercounts active downstream work. This is also why a shared cancellation context alone is not enough to guarantee that hedged work has disappeared. 9. Hedging Requires Idempotency Sending the same operation more than once is not always safe. Consider these operations: | Operation | Hedging suitability | |---|---| | Read a market price | Generally suitable | | Read a portfolio | Generally suitable | | Fetch cached configuration | Generally suitable | | Create a payment | Unsafe without idempotency | | Submit an order | Unsafe without deduplication | | Send an email | May produce duplicates | | Update an account balance | Unsafe without strong safeguards | A hedge can reach two different servers before either has completed. Both servers may execute the operation. Canceling one request does not guarantee that its side effects are rolled back. For operations with side effects, use an idempotency key or another server-side deduplication mechanism. Even then, the server must correctly enforce the idempotency contract. Hedging should generally be introduced first for read-only operations where duplicate execution is safe. 10. Where Should Hedged Requests Be Sent? Sending a duplicate request to the same server is often ineffective. If that server is experiencing CPU contention, a long queue, or an overloaded database connection pool, the second request may suffer from the same problem. Ideally, the hedge should target an independent replica. But independence is not guaranteed. Two replicas may share: - The same database - The same network path - The same cache - The same storage system - The same overloaded region This is called correlated failure or correlated latency. If both requests depend on the same struggling resource, hedging may increase load without meaningfully reducing latency. Replica selection should consider failure domains and actual dependency topology. 11. Observability: Measure the Cost of Hedging A hedging system needs more than a request-latency dashboard. You need to understand whether the reduction in tail latency justifies the additional work. Useful metrics include: | Metric | Type | Purpose | |---|---|---| | hedged_requests_total | Counter | Number of additional attempts | | hedge_wins_total | Counter | Requests where a hedge returned first | | hedge_budget_denied_total | Counter | Hedges rejected by the budget | | hedge_canceled_total | Counter | Losing attempts canceled by the client | | hedge_attempts_total | Counter | Total attempts, including originals | | hedge_delay_seconds | Histogram | Time before a hedge is launched | | request_latency_seconds | Histogram | End-to-end request latency | A useful derived metric is the hedge win rate: [ HedgeWinRate = \frac{HedgeWins}{HedgedRequests} ] If hedges rarely win, you may be generating additional traffic without much benefit. Also compare: - p50 latency - p95 latency - p99 latency - Downstream request volume - Dependency CPU and memory usage - Cancellation effectiveness - Error rate The goal is not to minimize p99 at any cost. It is to improve tail latency while keeping the additional workload within the dependency's capacity. 12. Testing Hedged Requests Concurrency behavior needs deterministic tests. At minimum, test these scenarios: - The primary request succeeds before the hedging delay. - The primary request is slow and the hedge wins. - The primary request fails with a retryable error. - A fatal error cancels outstanding attempts. - The hedge budget prevents additional requests. - The parent context is canceled. - The maximum number of attempts is respected. - Losing operations receive cancellation. - A losing operation that ignores cancellation does not release its concurrency permit prematurely. A simple test for the winning hedge could use controlled channels instead of arbitrary sleeps: func TestHedgeWins(t *testing.T) { primaryStarted := make(chan struct{}) hedgeStarted := make(chan struct{}) ctx, cancel := context.WithTimeout( context.Background(), time.Second, ) defer cancel() policy := Policy{ Delay: 20 * time.Millisecond, MaxAttempts: 2, ShouldHedge: func(error) bool { return true }, } result, err := Do( ctx, policy, func( ctx context.Context, attempt int, ) (string, error) { if attempt == 1 { close(primaryStarted) <-ctx.Done() return "", ctx.Err() } close(hedgeStarted) return "hedge response", nil }, ) if err != nil { t.Fatal(err) } if result != "hedge response" { t.Fatalf( "unexpected result: %s", result, ) } <-primaryStarted <-hedgeStarted } The test verifies that the second request can win and that the operation completes successfully. In a complete test suite, also verify that the primary operation observes cancellation. Run the race detector: go test -race ./... Finally, use load tests with a dependency that deliberately introduces variable latency. A useful experiment compares: 1. Hedging disabled. 2. Hedging enabled with a conservative delay. 3. Hedging enabled with an aggressive delay. 4. Hedging enabled with a shared budget. 5. Hedging enabled with adaptive concurrency. Compare both latency improvements and additional downstream traffic. 13. Production Checklist Before introducing hedged requests into a production service: - [ ] Hedge only safe, idempotent operations. - [ ] Select a delay from measured latency distributions. - [ ] Use a strict maximum attempt count. - [ ] Require a shared budget for additional attempts. - [ ] Ensure hedges target independent replicas where possible. - [ ] Apply concurrency limits to every attempt. - [ ] Propagate context cancellation. - [ ] Ensure permits remain held while work is still active. - [ ] Respect the original request deadline. - [ ] Measure hedge win rate and additional traffic. - [ ] Test correlated latency and dependency overload. - [ ] Roll out gradually and compare against a no-hedging baseline. Conclusion Tail latency is one of the most difficult problems in distributed systems because a small number of unusually slow operations can dominate the response time of an entire request. Hedged requests provide a way to reduce that impact. By launching a second request after a carefully selected delay, a client can avoid waiting for an unusually slow replica and use the first successful response. But hedging is not free. Every additional request consumes capacity, and uncontrolled duplication can make an already degraded dependency even slower. The right approach combines measured hedging delays, idempotent operations, cancellation, retry budgets, and adaptive concurrency limits. The goal isn't to send more requests. It's to get the response sooner without making the system slower for everyone else. --- References - The Tail at Scale — Google Research - Request Hedging — gRPC Documentation - gRPC Retry and Hedging Design
Retry Storms: Simulate How a 30-Second Blip Becomes an Outage That Never Ends - Book: System Design Pocket Guide: Fundamentals - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- The database fails over. Thirty seconds later it is back, healthy, with plenty of capacity for normal traffic. Your dashboards stay red for the next hour anyway. Every node is at 100% CPU and almost nothing succeeds. Someone restarts the database, which changes nothing. Someone else finally turns off the retries in the API gateway, and the whole system recovers in about a minute. That shape has a name. In 2021, Nathan Bronson and three co-authors called it a metastable failure in a HotOS '21 paper: a trigger pushes the system into a bad state, and that state persists even when the trigger is removed. A follow-up study at OSDI '22, Metastable Failures in the Wild, went through public incident reports and found that at least 4 of 15 major AWS outages in the last decade were metastable failures. Retries are one of the most common things that keep the bad state going. The simulator below reproduces the effect with numbers you can check by hand, then adds the fix that production RPC stacks ship: a retry budget. The multiplication nobody adds up Each retry policy looks harmless in isolation. "Try three times" is a sensible default for a service calling a flaky dependency. The problem is that your stack has several services, and each one has its own sensible default. The AWS Builders' Library article on timeouts and retries walks through a five-deep stack with three retries at each layer and puts it plainly: if each layer retries independently, load on the database goes up 243x, "making it unlikely to ever recover." With three layers and three tries each, the bottom of the stack sees up to 3 × 3 × 3 = 27 attempts for every user request. Here is why 27 is the exact count when every call fails. Layer A calls B up to 3 times. Each of those B calls makes up to 3 calls to C. Each of those makes up to 3 calls to the database. When the database fails every call, the attempts count up like an odometer, and the database sees all 27 before the user gets an error. Your normal traffic might be 70% of the database's capacity. Multiply it by 27 and you need 19 databases. The fault that started it is gone, and the retries are now the outage. Build the simulator The model is deliberately small. A database serves 100 requests per second, oldest first. Users send 70 new requests per second. A caller waits one second, then gives up and retries. A request the caller abandoned stays in the database's queue, and the database still does the work when it gets there, which is how real servers behave without deadline propagation. For 30 seconds, the database serves nothing. Because a 3-deep stack of 3 tries collapses to a flat count of 27 attempts when every call fails, the simulator models the whole stack as one caller with max_attempts = 27. It runs on plain Python 3.8 or later, no dependencies: # retry_storm.py: one 30-second blip, three retry policies import math from collections import defaultdict, deque CAPACITY = 100 # requests the database serves per second BASE_LOAD = 70 # new user requests per second TIMEOUT = 1 # seconds a caller waits before giving up BLIP = range(60, 90) # 30 seconds with zero capacity DURATION = 3600 # one simulated hour def simulate(max_attempts, budget=None): queue = deque() # FIFO of [enqueued, attempt, n] by_tick = defaultdict(list) # enqueue tick -> its batches due = defaultdict(list) # tick -> [(attempt, n)] retries queued = useful = served = 0 peak_load = peak_queue = 0 last_bad = None def enqueue(now, attempt, n): nonlocal queued batch = [now, attempt, n] queue.append(batch) by_tick[now].append(batch) queued += n for now in range(DURATION): # 1. new requests, then the retries the budget allows enqueue(now, 1, BASE_LOAD) sent = BASE_LOAD room = math.inf if budget is not None: room = round(BASE_LOAD * budget) for attempt, n in due.pop(now, []): n = min(n, room) room -= n sent += n if n: enqueue(now, attempt, n) peak_load = max(peak_load, sent) # 2. the database works through the oldest requests cap = 0 if now in BLIP else CAPACITY while cap and queue: batch = queue[0] take = min(cap, batch[2]) if now - batch[0] <= TIMEOUT: useful += take served += take batch[2] -= take queued -= take cap -= take if batch[2] == 0: queue.popleft() # 3. callers still waiting past the timeout give up for batch in by_tick.pop(now - TIMEOUT, []): attempt, n = batch[1], batch[2] if n and attempt < max_attempts: due[now + 1].append((attempt + 1, n)) peak_queue = max(peak_queue, queued) if now >= BLIP.start and queued > CAPACITY * TIMEOUT: last_bad = now if last_bad == DURATION - 1: recovered = None else: recovered = last_bad + 1 - BLIP.stop return (recovered, peak_load / BASE_LOAD, peak_queue, 1 - useful / served) def report(label, max_attempts, budget=None): rec, load, peak_q, waste = simulate(max_attempts, budget) when = "never" if rec is None else f"after {rec}s" print(f"{label:<16} recovered {when:<11} " f"peak load {load:4.1f}x peak queue {peak_q:>9,} " f"wasted {waste:4.0%}") report("no retries", 1) report("3 layers x 3", 3 ** 3) report("same + budget", 3 ** 3, budget=0.1) A few things to know before you run it. Time moves in one-second ticks, and requests travel in batches so an hour of millions of queued requests stays fast. "Recovered" means the queue has dropped to one timeout's worth of work (100 requests) and stays there, because past that point a new request gets served before its caller gives up. "Wasted" is the share of the database's work that went to requests whose caller had already walked away. Predict the numbers, then run it Work each run out on paper first. That is the point of a model this small: if the output disagrees with your arithmetic, one of you has a bug, and finding it teaches you more than the run does. No retries. During the blip, 30 seconds × 70 requests = 2,100 requests pile up. Afterwards the database drains them at 100 − 70 = 30 per second. 2,000 requests at 30 per second is roughly 67 seconds, so expect recovery a little over a minute after the blip ends. Peak load: 1.0x. Three layers, three tries. Every request queued during the blip times out, and so does every retry, because each one lands at the back of a growing queue. After 26 rounds of retries, every second carries 70 new requests plus 26 × 70 retries, which is 1,890 requests per second against a capacity of 100. The queue grows by about 1,790 every second. It never drains, so expect recovered never, a peak load of 27.0x and a wasted-work figure close to 100%: the database spends the hour busy, almost entirely on answers nobody is waiting for. This is the metastable state. The blip ended at second 90, capacity came back, and the system stayed down because of its own retries. Same stack plus a 10% budget. Retries can never exceed 10% of new requests, so arrivals are capped at 77 per second. The backlog after the blip is about 2,300, draining at 100 − 77 = 23 per second. That is around 95 to 100 seconds. Peak load: 1.1x. Run python3 retry_storm.py and compare. The budget run recovers a little slower than the no-retry run, because retries still eat some capacity. That is the cost of the budget, and it is small. The difference between a minute and a half and never is the whole argument. What a retry budget is A retry budget caps retries as a fraction of real traffic, instead of capping them per request. Per-request limits ("try three times") say nothing about how many requests are failing at once. A budget does, because it is shared by every request a client sends. This is not a new idea. The production stacks you already use ship it: - The Google SRE book's chapter on handling overload describes two limits: up to three attempts per request, and a per-client budget where a request is only retried while the client's retry ratio is below 10%. The chapter says the budget reduces worst-case growth from "somewhere just below 3X" to "just 1.1x in the general case." - gRPC's retry design (gRFC A6) has retry throttling as a token bucket. Every failed RPC removes one token, every success adds tokenRatio (0.1 in the example config), and retries stop when the count drops to half of maxTokens. In a storm where everything fails, the bucket empties fast and retries switch off by themselves. - Finagle's RetryBudget by default lets about 20% of requests retry immediately, with a floor of 10 retries per second so quiet clients can still retry at all. The gRPC and SRE versions also attack the layering problem from the other side. The SRE chapter says requests should only be retried at the layer immediately above the one rejecting them, and an overloaded backend can return an "overloaded; don't retry" error so the layers above stop multiplying. gRPC sends a grpc-previous-rpc-attempts header on retries, so a server can see that a request is already a retry. One caveat about the simulator: it applies the budget to every extra attempt the database sees. A real stack only gets that if each layer runs its own budget (three layers at 10% gives at most 1.1³, about 1.33x) or if retries are tagged so lower layers can count them. Either works. A budget in one service and blind retries in the other two does not. Experiments worth running Change one constant at a time and predict the result first. - Set BASE_LOAD = 90. A 10% budget then puts 99 requests per second on a database that serves 100. One request per second of headroom against a backlog of about 2,950 is a drain of most of the simulated hour. Budgets need spare capacity to work. That is a capacity-planning problem. - Set budget=0.2, Finagle's default ratio. Work out the drain rate before you run it. - Make the blip 5 seconds instead of 30. Predict whether the naive run still tips over. In this model, a trigger tips the system over once the backlog grows past one timeout's worth of work, however small the trigger looks. - Add a queue cap in enqueue that rejects requests past, say, 500 queued. Rejections fail fast instead of wasting work, which changes the numbers. Then check whether retries alone can still keep the system down. Backoff and jitter deserve a mention here too, because they are the first thing people reach for. Exponential backoff spreads the 27 attempts over a longer window, which lowers the per-second peak. It does not change the count: each failing request still sends 27 attempts to the bottom of the stack. The budget is what limits the total. Add up the retry policies along your deepest call path and multiply them. If the product is bigger than the headroom your bottom dependency has, a short blip can turn into a long outage, and the place to fix it is the retry policy. --- If this was useful Retry budgets only work next to timeouts, rate limits and capacity headroom. System Design Pocket Guide: Fundamentals walks through those building blocks and the availability trade-offs behind them. If this simulator made you want to multiply out your own call paths, start there. 
Designing gRPC APIs for Mobile: Deadline Propagation, Channel Pooling, and the Retry Budget That Prevents Thundering Herds --- title: "gRPC for Mobile: Deadlines, Channel Pools, and the Retry Budget That Stops Thundering Herds" published: true description: "Master gRPC for mobile clients: propagate deadlines through your service mesh, safely pool channels on Android and iOS, and cap retry traffic before a thundering herd takes down your backend." tags: android, ios, mobile, api canonical_url: https://mvpfactory.co/blog/grpc-mobile-deadlines-channel-pools-retry-budgets --- What you will build By the end of this guide, you will have a production-grade gRPC configuration for your mobile client: deadlines that cascade correctly through your service mesh, a channel pool that does not leak on Android or iOS, and a retry strategy that stays well-behaved when your network degrades. This is the exact config that would have prevented an 18-minute backend outage — which we will walk through to make the lessons concrete. Prerequisites - Familiarity with gRPC concepts (channels, stubs, service configs) - An Android project using grpc-kotlin or an iOS project using grpc-swift - A backend with at least two service layers (API gateway + downstream service) --- The incident that changed how I think about gRPC At 11pm on a Tuesday, a stadium Wi-Fi handoff triggered 400,000 simultaneous retries. The backend was down for 18 minutes. The root cause was three missing lines of config. Most teams treat gRPC like a slightly faster REST call. It is not. The deadline, retry, and connection models are fundamentally different — and each interacts with the others in ways that only surface under load. Here is the minimal setup to get this right. --- Step 1 — Deadline propagation: cascade or die Here is the gotcha that will save you hours: a deadline is not a timeout. A timeout fires client-side and abandons the call. A deadline propagates through every hop of your service mesh via the grpc-timeout header, letting upstream services abandon work that will never reach the client anyway. Set deadlines on the stub, not the channel: // Android — set a 2.5s deadline on the stub, not the channel val response = stub .withDeadlineAfter(2500, TimeUnit.MILLISECONDS) .getHealthData(request) Your server deadline must always be shorter than your client deadline. Here is a cascade pattern I use in every project: | Layer | Deadline | Why | | :--- | :---: | :--- | | Mobile client | 3000ms | User-facing tolerance | | API gateway | 2700ms | Buffer for network RTT | | Backend service | 2400ms | Buffer for internal hops | | Database query | 1800ms | Leave room for retries | A 300ms cascade buffer at each hop is conservative. On low-latency internal networks, 150ms is usually enough. --- Step 2 — Channel pooling: size it or leak it gRPC multiplexes requests over a single HTTP/2 connection per channel. One channel per backend endpoint is usually correct on mobile. Before you reach for a pool, understand the cost: each channel holds a TCP connection, negotiates TLS, and maintains a flow control window. On Android, an unclosed channel holds a wake lock. On iOS, it fights the system's network interface scheduler. A practical ceiling is 2–4 channels per host. Here is the iOS setup: // iOS — pool of 2 channels for the primary API host let pool = GRPCChannelPool.with( target: .host("api.example.com", port: 443), transportSecurity: .tls(GRPCTLSConfiguration.makeClientDefault()), eventLoopGroup: PlatformSupport.makeEventLoopGroup(loopCount: 2) ) Two channels handle roughly 200 concurrent streams comfortably. Going beyond 4 per host rarely improves throughput — and it meaningfully increases your server-side connection count, which bites hard when running behind a service mesh with per-replica connection limits. Always close channels in onDestroy / deinit. --- Step 3 — The retry budget: two mechanisms, not one This is where most implementations break down. The docs do not make this clear, but there are two distinct concepts here, and conflating them is exactly how you end up with an 18-minute outage. The client-side retryPolicy controls per-call behavior: // gRPC service config — per-call retry behavior, client-side only val serviceConfig = """{ "retryPolicy": { "maxAttempts": 3, "initialBackoff": "0.5s", "maxBackoff": "10s", "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"] } }""" This limits retries per call. It does not limit total retry traffic across your fleet. If 400,000 clients each retry 3 times simultaneously, you have multiplied your load by 3x at the worst possible moment. Fleet-wide enforcement requires server-side controls. The most practical approach: track the ratio of retries (identifiable via the grpc-previous-rpc-attempts header) to total requests at your gateway, and shed excess retry traffic above 10% of total RPC volume. With this budget enforced at the gateway, your effective traffic multiplier under a partial outage is capped at 1.1x regardless of fleet size. Exponential backoff with jitter is necessary. It is not sufficient. --- Gotchas - Deadline on the channel vs. the stub. Channel-level deadlines apply to every call for the lifetime of the channel. Stub-level deadlines are per-call. Always use the stub. - Missing grpc-previous-rpc-attempts at the gateway. Without this header check, your gateway cannot distinguish organic traffic from a retry storm. - Pool size creep. Teams see head-of-line blocking and double their pool size. Measure server-side connection count first — you are almost always hitting a concurrency limit, not a connection limit. - iOS deinit timing. Swift ARC does not guarantee deinit timing under memory pressure. Explicitly close pool channels in viewDidDisappear or your scene lifecycle, not only in deinit. --- Conclusion Three things prevent the outage: 1. Deadlines on the stub, cascaded 10–15% tighter at each service layer so upstream services abandon work before the client times out. 2. Channel pool capped at 2–4 per host, always closed in onDestroy / deinit. 3. Client-side retryPolicy combined with a gateway-level retry budget — the config handles per-call behavior, the gateway enforces the fleet-wide ceiling. If you are building a mobile app that communicates with users throughout the day — something like HealthyDesk, which sends break reminders and guided desk exercises to developers on a schedule — this pattern is exactly what keeps your notification and sync services standing when a large cohort of users reconnects simultaneously. Relevant docs: gRPC retry design proposal (A6) · grpc-swift · grpc-kotlin