grpc

Hedged Requests in Go: Reducing Tail Latency Without Overloading Dependencies Most requests in a distributed system finish quickly. A few don't. Your API might have a median response time of 40ms, while its p99 latency reaches 900ms. Most users get a fast response, but a small percentage experience significantly longer waits. These slow requests are particularly problematic in systems that depend on multiple services. A single slow dependency can delay an entire operation, even when every other component is responding normally. One way to address this problem is request hedging. Instead of waiting indefinitely for an unusually slow request, the client starts another copy after a configured delay. Whichever request succeeds first wins, and the remaining requests are canceled. This technique can significantly reduce tail latency, but it comes with an important trade-off: it generates additional traffic. If every slow request triggers another request without limits, the mechanism designed to improve latency can overload the dependency it's trying to protect. In this article, we'll explore: - What tail latency is and why averages hide it - How hedged requests work - The difference between hedging and retries - How to implement hedged requests in Go - How to cancel losing requests safely - How to integrate hedging with retry budgets and adaptive concurrency - When hedging helps and when it makes things worse - What to monitor and test in production 1. The Problem: Tail Latency Let's say an API receives 10,000 requests. Its response-time distribution looks like this: | Percentile | Response time | |---|---:| | p50 | 40ms | | p90 | 75ms | | p95 | 120ms | | p99 | 900ms | | p99.9 | 2.5s | Looking only at the average or median would suggest that the service is performing well. But 1% of 10,000 requests means 100 requests take at least 900ms. For latency-sensitive applications, these outliers matter. Consider an API that aggregates data from five independent services: API Request | +-----------+-----------+ | | | Service A Service B Service C | | | Service D Service E | | | | +-----------+-----------+ | Final Response If the API needs all five responses, its total latency is constrained by the slowest dependency. Even if four services respond in 30ms, one service taking 800ms can delay the entire operation. This is one reason tail latency becomes increasingly important as distributed systems grow. Jeff Dean and Luiz André Barroso discuss this problem in their influential paper, The Tail at Scale. They describe techniques for building responsive services despite unpredictable latency in individual components. Reference: research.google 2. What Are Hedged Requests? A hedged request is a duplicate request sent while the original request is still in progress. The client does not immediately send multiple copies. Instead, it waits for a configurable period. If the original request hasn't completed, the client sends another request, ideally to a different replica. The first successful response wins. Time | | Primary request | ────────────────────────────────> | | Wait: 100ms | | Hedge request | ─────────────────────> | | Hedge response: 160ms | Primary response: 450ms | | Return hedge response | Cancel primary request | The important detail is that the second request starts before the first one fails. This is fundamentally different from a traditional retry. A retry waits for an unsuccessful attempt before trying again. A hedge assumes that an unusually slow request might be an outlier and starts another attempt proactively. Why does this work? Imagine two replicas: - Replica A is experiencing temporary contention. - Replica B is healthy. The primary request goes to A. After 100ms, it still hasn't completed, so the client sends a hedge to B. If B responds in 60ms, the client returns that response instead of waiting for A. The total latency becomes approximately 160ms instead of 450ms. The original request might still finish later, but its result is no longer needed. 3. Hedging vs. Retrying These mechanisms are related, but they solve slightly different problems. | Mechanism | When the next attempt starts | Primary goal | |---|---|---| | Retry | After an attempt fails | Recover from transient failures | | Hedging | While an attempt is still running | Reduce tail latency | | Timeout | After a deadline expires | Bound waiting time | | Circuit breaker | When a dependency is unhealthy | Stop sending harmful traffic | A retry responds to failure. A hedge responds to excessive waiting. This distinction matters because hedging can increase downstream load even when the original request would eventually succeed. For that reason, hedging should generally be limited to safe, idempotent operations, especially reads. 4. Choosing the Hedging Delay The hedging delay is one of the most important configuration decisions. If the delay is too short, the client duplicates too many requests. If it is too long, the hedge provides little latency improvement. A reasonable starting point is to use a measured latency percentile for a particular operation. For example: | Percentile | Latency | |---|---:| | p50 | 40ms | | p90 | 70ms | | p95 | 100ms | | p99 | 450ms | A 100ms hedging delay means that requests exceeding the p95 threshold become candidates for hedging. Under stable conditions, approximately 5% of requests may reach that threshold. This does not mean exactly 5% of requests will generate a hedge: requests can complete at the threshold boundary, and actual behavior depends on the latency distribution and scheduling. The important principle is to trigger hedging in the tail, not in the middle of normal traffic. The original Tail at Scale paper discusses using a high-percentile latency threshold to limit duplicate traffic while reducing extreme latency. Its examples demonstrate why carefully selected hedging delays can achieve substantial tail-latency improvements with relatively modest additional load. Source: andrew.cmu.edu 5. Implementing Hedged Requests in Go Let's build a small hedging executor. Our implementation will support: - An immediate primary request - A configurable hedging delay - A maximum number of attempts - Context cancellation - First-successful-response semantics - Retryable error classification - An optional shared retry budget We'll use Go generics so the executor can return any response type. Step 1: Define the policy package hedge import ( "context" "errors" "fmt" "time" ) var ( ErrInvalidPolicy = errors.New("invalid hedging policy") ErrBudgetDenied = errors.New("hedging budget exhausted") ) type Budget interface { TryTake() bool } type Policy struct { Delay time.Duration MaxAttempts int Budget Budget ShouldHedge func(error) bool } type result[T any] struct { value T err error } The Budget interface allows us to reuse the retry budget implementation from the previous article. The original request does not consume a hedge token. Every additional request does. We also use ShouldHedge to distinguish errors that should terminate the operation from errors that may allow another attempt. For example, a validation error should not trigger another request. Step 2: Implement the executor func Do[T any]( ctx context.Context, policy Policy, operation func(context.Context, int) (T, error), ) (T, error) { var zero T if policy.Delay <= 0 || policy.MaxAttempts < 1 || policy.MaxAttempts > 3 || operation == nil || policy.ShouldHedge == nil { return zero, ErrInvalidPolicy } if err := ctx.Err(); err != nil { return zero, err } hedgeCtx, cancel := context.WithCancel(ctx) defer cancel() results := make(chan result[T], policy.MaxAttempts) launched := 0 inFlight := 0 var lastErr error startAttempt := func() { launched++ inFlight++ attempt := launched go func() { value, err := operation(hedgeCtx, attempt) results <- result[T]{ value: value, err: err, } }() } // The primary request starts immediately. startAttempt() var timer *time.Timer var timerC <-chan time.Time if policy.MaxAttempts > 1 { timer = time.NewTimer(policy.Delay) timerC = timer.C defer timer.Stop() } for { select { case <-ctx.Done(): return zero, ctx.Err() case res := <-results: inFlight-- if res.err == nil { // First successful response wins. return res.value, nil } lastErr = res.err if !policy.ShouldHedge(res.err) { return zero, res.err } if inFlight == 0 && launched == policy.MaxAttempts { return zero, lastErr } case <-timerC: if launched >= policy.MaxAttempts { timerC = nil continue } if policy.Budget != nil && !policy.Budget.TryTake() { // No more additional requests. timerC = nil if inFlight == 0 { if lastErr != nil { return zero, lastErr } return zero, ErrBudgetDenied } continue } startAttempt() if launched < policy.MaxAttempts { timer.Reset(policy.Delay) } else { timerC = nil } } } } Let's examine the important decisions. The primary request starts immediately. We don't wait for a timer before sending the first request. Additional requests start only after the configured delay. If the primary finishes successfully before the delay expires, the executor returns without sending a hedge. The first successful response wins. As soon as a successful result arrives, the function returns and cancels the shared context. Errors are classified. A fatal error can terminate the operation immediately. A retryable error can be treated as non-final while another attempt is still running. The number of attempts is bounded. The implementation intentionally limits the example to three total attempts, including the original request. This is a conservative default for the example, not a universal limit. Step 3: Use the executor Imagine a service that reads market data from multiple replicas. type Price struct { Symbol string Value float64 } func fetchPrice( ctx context.Context, attempt int, ) (Price, error) { replica := selectReplica(attempt) return replica.Fetch(ctx, "AAPL") } We can now use the hedging executor: policy := hedge.Policy{ Delay: 100 * time.Millisecond, MaxAttempts: 2, Budget: retryBudget, ShouldHedge: isRetryableError, } price, err := hedge.Do( ctx, policy, fetchPrice, ) if err != nil { return err } fmt.Printf( "%s: %.2f\n", price.Symbol, price.Value, ) The primary request goes to the first replica. If it hasn't completed after 100ms, the executor checks the budget and starts another request. If the second replica responds first, the result is returned and the first request is canceled. The example assumes that selectReplica routes attempts to distinct, suitable replicas. Sending both copies to the same overloaded instance may provide little benefit. 6. Cancellation Is Not the Same as Stopping Work This is one of the most important details in a hedging implementation. When the winning request completes, the executor calls cancel() on the shared context. The losing requests receive cancellation through their contexts. But cancellation is cooperative. It does not forcibly terminate a goroutine, an HTTP request, a database query, or server-side processing. For example, a dependency may continue executing a query even after the client disconnects. A correct operation should propagate its context: func fetch( ctx context.Context, client *http.Client, ) error { req, err := http.NewRequestWithContext( ctx, http.MethodGet, "https://api.example.com/prices/AAPL", nil, ) if err != nil { return err } resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() return nil } The server should also respect request cancellation where possible. Otherwise, the client may stop waiting while the dependency continues doing unnecessary work. This is especially important for expensive database queries and CPU-intensive operations. 7. Why a Hedge Budget Matters Hedging creates additional traffic. Imagine a service receiving 10,000 requests per second. If 5% reach the hedging threshold, that could mean approximately 500 additional attempts per second. Now imagine the dependency is already degraded. More concurrent requests may increase queueing, which increases latency, which triggers more hedges. The mechanism can create the same kind of amplification problem that we discussed in the Retry Budgets article. This is why the hedge budget should be shared across requests. A simple rule: - The original request is always allowed, subject to normal admission controls. - Each additional hedge requires a token. - Successful normal traffic replenishes the budget. - If the budget is exhausted, no new hedges are launched. For example: budget := retry.NewRetryBudget( 20, 0.1, ) The exact budget should be tuned according to the dependency's spare capacity and the service's traffic patterns. The important part is that hedging is not an unlimited permission to duplicate requests. It is an additional workload that must be controlled. 8. Hedging and Adaptive Concurrency Hedging and adaptive concurrency work particularly well together, but they need to share a consistent view of downstream capacity. Adaptive concurrency adjusts how many requests are allowed in flight based on observed dependency behavior. Hedging creates additional in-flight requests. If the adaptive limiter counts only original requests, hedged requests can bypass the intended concurrency ceiling. That defeats the purpose of the limiter. A safer integration is: 1. The primary request acquires a concurrency permit. 2. A hedge must acquire its own permit. 3. If no permit is available, the hedge is skipped. 4. The winning request returns its result. 5. Every attempt releases its own permit when it actually completes. There is an important implementation detail here. Do not release the losing request's concurrency permit merely because the caller stopped waiting for it. The permit must remain held until the underlying operation actually completes or otherwise confirms that the protected work has stopped. Otherwise, the limiter undercounts active downstream work. This is also why a shared cancellation context alone is not enough to guarantee that hedged work has disappeared. 9. Hedging Requires Idempotency Sending the same operation more than once is not always safe. Consider these operations: | Operation | Hedging suitability | |---|---| | Read a market price | Generally suitable | | Read a portfolio | Generally suitable | | Fetch cached configuration | Generally suitable | | Create a payment | Unsafe without idempotency | | Submit an order | Unsafe without deduplication | | Send an email | May produce duplicates | | Update an account balance | Unsafe without strong safeguards | A hedge can reach two different servers before either has completed. Both servers may execute the operation. Canceling one request does not guarantee that its side effects are rolled back. For operations with side effects, use an idempotency key or another server-side deduplication mechanism. Even then, the server must correctly enforce the idempotency contract. Hedging should generally be introduced first for read-only operations where duplicate execution is safe. 10. Where Should Hedged Requests Be Sent? Sending a duplicate request to the same server is often ineffective. If that server is experiencing CPU contention, a long queue, or an overloaded database connection pool, the second request may suffer from the same problem. Ideally, the hedge should target an independent replica. But independence is not guaranteed. Two replicas may share: - The same database - The same network path - The same cache - The same storage system - The same overloaded region This is called correlated failure or correlated latency. If both requests depend on the same struggling resource, hedging may increase load without meaningfully reducing latency. Replica selection should consider failure domains and actual dependency topology. 11. Observability: Measure the Cost of Hedging A hedging system needs more than a request-latency dashboard. You need to understand whether the reduction in tail latency justifies the additional work. Useful metrics include: | Metric | Type | Purpose | |---|---|---| | hedged_requests_total | Counter | Number of additional attempts | | hedge_wins_total | Counter | Requests where a hedge returned first | | hedge_budget_denied_total | Counter | Hedges rejected by the budget | | hedge_canceled_total | Counter | Losing attempts canceled by the client | | hedge_attempts_total | Counter | Total attempts, including originals | | hedge_delay_seconds | Histogram | Time before a hedge is launched | | request_latency_seconds | Histogram | End-to-end request latency | A useful derived metric is the hedge win rate: [ HedgeWinRate = \frac{HedgeWins}{HedgedRequests} ] If hedges rarely win, you may be generating additional traffic without much benefit. Also compare: - p50 latency - p95 latency - p99 latency - Downstream request volume - Dependency CPU and memory usage - Cancellation effectiveness - Error rate The goal is not to minimize p99 at any cost. It is to improve tail latency while keeping the additional workload within the dependency's capacity. 12. Testing Hedged Requests Concurrency behavior needs deterministic tests. At minimum, test these scenarios: - The primary request succeeds before the hedging delay. - The primary request is slow and the hedge wins. - The primary request fails with a retryable error. - A fatal error cancels outstanding attempts. - The hedge budget prevents additional requests. - The parent context is canceled. - The maximum number of attempts is respected. - Losing operations receive cancellation. - A losing operation that ignores cancellation does not release its concurrency permit prematurely. A simple test for the winning hedge could use controlled channels instead of arbitrary sleeps: func TestHedgeWins(t *testing.T) { primaryStarted := make(chan struct{}) hedgeStarted := make(chan struct{}) ctx, cancel := context.WithTimeout( context.Background(), time.Second, ) defer cancel() policy := Policy{ Delay: 20 * time.Millisecond, MaxAttempts: 2, ShouldHedge: func(error) bool { return true }, } result, err := Do( ctx, policy, func( ctx context.Context, attempt int, ) (string, error) { if attempt == 1 { close(primaryStarted) <-ctx.Done() return "", ctx.Err() } close(hedgeStarted) return "hedge response", nil }, ) if err != nil { t.Fatal(err) } if result != "hedge response" { t.Fatalf( "unexpected result: %s", result, ) } <-primaryStarted <-hedgeStarted } The test verifies that the second request can win and that the operation completes successfully. In a complete test suite, also verify that the primary operation observes cancellation. Run the race detector: go test -race ./... Finally, use load tests with a dependency that deliberately introduces variable latency. A useful experiment compares: 1. Hedging disabled. 2. Hedging enabled with a conservative delay. 3. Hedging enabled with an aggressive delay. 4. Hedging enabled with a shared budget. 5. Hedging enabled with adaptive concurrency. Compare both latency improvements and additional downstream traffic. 13. Production Checklist Before introducing hedged requests into a production service: - [ ] Hedge only safe, idempotent operations. - [ ] Select a delay from measured latency distributions. - [ ] Use a strict maximum attempt count. - [ ] Require a shared budget for additional attempts. - [ ] Ensure hedges target independent replicas where possible. - [ ] Apply concurrency limits to every attempt. - [ ] Propagate context cancellation. - [ ] Ensure permits remain held while work is still active. - [ ] Respect the original request deadline. - [ ] Measure hedge win rate and additional traffic. - [ ] Test correlated latency and dependency overload. - [ ] Roll out gradually and compare against a no-hedging baseline. Conclusion Tail latency is one of the most difficult problems in distributed systems because a small number of unusually slow operations can dominate the response time of an entire request. Hedged requests provide a way to reduce that impact. By launching a second request after a carefully selected delay, a client can avoid waiting for an unusually slow replica and use the first successful response. But hedging is not free. Every additional request consumes capacity, and uncontrolled duplication can make an already degraded dependency even slower. The right approach combines measured hedging delays, idempotent operations, cancellation, retry budgets, and adaptive concurrency limits. The goal isn't to send more requests. It's to get the response sooner without making the system slower for everyone else. --- References - The Tail at Scale — Google Research - Request Hedging — gRPC Documentation - gRPC Retry and Hedging Design

gRPC-Web changes the transport, not the authorization boundary gRPC-Web changes the transport, not the authorization boundary A browser cannot speak native gRPC in the same way as a Go or Java client. The gRPC-Web protocol adapts calls for browser constraints and commonly carries request metadata in HTTP headers and response metadata in a trailer frame inside the body. That adaptation creates a useful bridge for web applications, but it does not move the authorization decision into the browser or into the protocol adapter. The first design mistake is treating CORS as authentication. CORS controls which browser origins may read a cross-origin response. It does not prove who sent a request, and it does not stop a server-side client or a script outside the browser. A public CORS policy on a gRPC-Web endpoint can be reasonable if the endpoint still requires a validated bearer token, a session cookie with CSRF protection, or mutual TLS at a trusted proxy. The identity check has to happen before the request reaches service code. The second mistake is trusting a header because it is familiar. A gateway may add x-user-id, x-tenant-id or an authorization context after validating a token. If the same gateway forwards a client-supplied copy of that header, downstream services may see two values or the attacker's value. Gateways should remove untrusted identity headers, authenticate the request, and inject a new context. Services should reject requests that arrive without the expected gateway identity marker and should use the marker only on a network path that ordinary clients cannot reach. Method-level authorization still matters after a valid token is established. A token can be valid for the application while the caller lacks permission for a specific service method or resource. Mapping an HTTP path to a gRPC method is convenient, but the policy should use the canonical service and method names and the resource identifiers in the request. Wildcard access such as * is a change to the authorization model, not a harmless convenience for generated clients. Session cookies need a separate review. gRPC-Web clients often use ordinary POST requests, so browsers may attach cookies automatically. The proxy must apply CSRF defenses to state-changing calls, set SameSite deliberately, and avoid accepting cookies on endpoints that also accept a bearer token unless the precedence rule is clear. If both credentials are accepted, the stronger identity must not be silently replaced by a weaker one from a query parameter or custom header. The wire format also changes what intermediaries see. Metadata becomes headers, and some trailers appear in the response body. A reverse proxy can terminate TLS, enforce request size limits, attach tracing metadata, and forward the adapted response. It can also break streaming, compress messages incorrectly, or remove CORS response headers. Test these behaviors through the same ingress path used in production. A successful call from a local gRPC client says nothing about a browser call through an identity-aware proxy. Logging should connect the user identity, the client application, the gRPC method, the resource, and the authorization result without recording access tokens. When a call is denied, distinguish an expired credential, a valid credential with insufficient scope, a policy mismatch, and a malformed request. That difference matters during incident response because each condition points to a different control. The practical rule is simple: authenticate at the edge, authorize at the service, and verify both through the browser-facing path. gRPC-Web is a compatibility layer. A secure deployment still depends on ordinary HTTP protections, explicit method policy, and an intermediary that knows which identity headers it owns. References - gRPC-Web protocol specification - gRPC metadata guide

gRPCも関係してたんか grpc#43699

gRPC shows up when REST starts breaking under its own weight Picture this: you've got fifteen microservices talking to each other. Each one exposes its own REST API, each one has its own JSON with its own naming conventions, and every time someone changes a field in the payments service, three other teams find out when everything blows up in production. You document with Swagger, but the Swagger goes stale. You write HTTP clients by hand, or generate something with OpenAPI that works "more or less." And when you need service A to ask service B something ten thousand times a second, the overhead of JSON + HTTP/1.1 starts hurting your infrastructure bill. That specific pain — inter-service communication with contracts that break silently and performance that doesn't scale — is exactly what gRPC comes to solve. It's not a trend. Google designed it to solve that exact problem inside its own infrastructure, and then released it as an open source project. Today it shows up in three independent awesome lists, which, in the curation system I've been building out through this series, is a strong signal: it's not noise, it's real community consensus. If you've been working with REST your whole career (like I spent a good chunk of time managing infrastructure before diving fully into development), the first time you see gRPC in action it changes how you think. It's not "another HTTP framework." It's a different paradigm for how your services talk to each other. What it does gRPC is a Remote Procedure Call framework that runs over HTTP/2 and uses Protocol Buffers (protobuf) as its default serialization mechanism. Instead of exposing REST endpoints and letting every client build its own request, you define a contract — a .proto file — and from that you generate client and server code in whatever language you need. Go, Java, Python, C++, Node, whatever. Same contract, multiple languages, and the field names and types are enforced by the generated code instead of living only in a doc someone forgot to update. Here's what a basic contract looks like: // pedidos.proto // We define the service and the messages it will exchange syntax = "proto3"; package pedidos; // The service exposes an RPC method, as if it were a remote function service PedidosService { rpc ObtenerPedido (PedidoRequest) returns (PedidoResponse); } // Messages are like DTOs, but typed and versionable message PedidoRequest { string id_pedido = 1; } message PedidoResponse { string id_pedido = 1; double monto_total = 2; string estado = 3; } From that file, the protobuf compiler (protoc) automatically generates serialization classes and client/server stubs. If you come from the Java world (like me, where I started my career as a developer), this is going to sound similar to what SOAP did with WSDL, but without the dead weight of XML and with noticeably better performance. On the server side in Java, the implementation looks like this: // We implement the service automatically generated from the .proto public class PedidosServiceImpl extends PedidosServiceGrpc.PedidosServiceImplBase { @Override public void obtenerPedido(PedidoRequest request, StreamObserver<PedidoResponse> responseObserver) { // The real logic would go here, querying the database PedidoResponse response = PedidoResponse.newBuilder() .setIdPedido(request.getIdPedido()) .setMontoTotal(1500.50) .setEstado("CONFIRMADO") .build(); // onNext sends the response, onCompleted closes the stream responseObserver.onNext(response); responseObserver.onCompleted(); } } Notice there's no JSON anywhere. The payload is serialized in binary, which means smaller payloads and faster parsing. And since it runs over HTTP/2, you get multiplexing of requests over a single TCP connection, native bidirectional streaming, and header compression — the structural reasons gRPC tends to outperform REST under high load. Why it's on the list The consensus signal — three independent awesome lists mentioning it — isn't a coincidence. gRPC solves a problem that comes up fast with microservices: the lack of strong contracts between services. With REST + JSON, the contract is a convention that lives in devs' heads and in documentation that goes stale. With gRPC, the contract is generated code. If you break a field, you find out at compile time, not in production on a Friday at six in the evening. The other big reason is performance. Code generation eliminates manual serialization boilerplate — you don't write JSON parsers by hand, you don't maintain duplicate DTOs in every language. And binary protobuf over HTTP/2 is the combination that gives gRPC an edge over traditional REST under high load, which is exactly the scenario where microservices start to struggle. It's the kind of tool that doesn't shine in a five-minute demo, but becomes indispensable when you have dozens of services talking to each other at real volume. That's where the JVM ecosystem (if you work with Spring Boot, there's solid support via grpc-spring-boot-starter) benefits from this approach versus maintaining fifteen hand-built REST clients. When NOT to use it Now, let's be honest: gRPC isn't free, and this is the part people skip. Protobuf and the async/streaming model are a different mental model from request/response REST — if your team hasn't touched this before, budget real ramp-up time for it, not an afternoon. Debugging is also more tedious: you can't just hit a gRPC endpoint with curl like you did with REST, you need extra tools like grpcurl or BloomRPC to inspect traffic. It also doesn't make sense if you're going to consume your API directly from a browser (you need gRPC-Web with an intermediate proxy, another layer of complexity) or if you're building a public API that third parties will consume without any control over their stack — there, REST + OpenAPI is still more friendly. If your system is two or three services with low traffic, you're probably adding complexity you don't need. For those cases, stick with REST or look at lighter alternatives like tRPC if you're in a TypeScript monorepo. Wrap-up gRPC is the kind of tool you shouldn't adopt because it's trendy, but because you have the specific problem it solves: inter-service communication at scale, with strong contracts and performance that matters. That's why it made it through this series' filters — it's not hype, it's real technical consensus, and it comes with a real learning cost you should plan for, not skip. If you liked this analysis, this is just entry #13 of Awesome Curated: The Tools, where I go through tools that passed our curation system's filter. If you're into infrastructure and runtimes, you might also like the post on Node.js and its event loop, or if you're into networking, Sniffnet for monitoring network traffic without losing your mind. The full series is at /blog/series/awesome-curated-tools. --- This article was originally published on juanchi.dev

gRPC no es moda: es el contrato que REST nunca te dio Imaginate esto: tenés quince microservicios hablando entre sí. Cada uno expone su API REST, cada uno tiene su propio JSON con sus propias convenciones de nombres, y cada vez que alguien cambia un campo en el servicio de pagos, tres equipos más se enteran cuando todo explota en producción. Documentás con Swagger, pero el Swagger se desactualiza. Escribís clientes HTTP a mano, o generás algo con OpenAPI que funciona "más o menos". Y cuando necesitás que el servicio A le pregunte algo al servicio B diez mil veces por segundo, el overhead de JSON + HTTP/1.1 empieza a dolerte en la factura de infraestructura. Ese dolor específico — comunicación inter-servicios con contratos que se rompen en silencio y performance que no escala — es el que gRPC viene a resolver. No es una moda. Google lo diseñó para resolver ese problema puntual dentro de su propia infraestructura, y después lo liberó como proyecto open source. Hoy aparece en tres awesome lists independientes, lo cual en el sistema de curación que vengo laburando en esta serie es señal fuerte: no es ruido, es consenso real de la comunidad. Si vos venís de laburar con REST toda la vida (como yo laburé un buen tiempo administrando infraestructura antes de meterme de lleno al desarrollo), la primera vez que ves gRPC en acción te cambia la cabeza. No es "otro framework HTTP". Es un paradigma distinto para pensar cómo se hablan tus servicios. Qué hace gRPC es un framework de Remote Procedure Calls que corre sobre HTTP/2 y usa Protocol Buffers (protobuf) como mecanismo de serialización por default. En vez de exponer endpoints REST y que cada cliente arme su propio request, definís un contrato — un archivo .proto — y de ahí se genera código cliente y servidor en el lenguaje que necesites: Go, Java, Python, C++, Node. El mismo contrato, múltiples lenguajes, menos espacio para que cada equipo interprete el JSON a su manera. Mirá cómo se ve un contrato básico: // pedidos.proto // Definimos el servicio y los mensajes que va a intercambiar syntax = "proto3"; package pedidos; // El servicio expone un método RPC, como si fuera una función remota service PedidosService { rpc ObtenerPedido (PedidoRequest) returns (PedidoResponse); } // Los mensajes son como DTOs, pero tipados y versionables message PedidoRequest { string id_pedido = 1; } message PedidoResponse { string id_pedido = 1; double monto_total = 2; string estado = 3; } De ese archivo, el compilador de protobuf (protoc) genera automáticamente las clases de serialización y los stubs de cliente/servidor. Si vos venís del mundo Java (como yo, que arranqué ahí mi carrera como developer), esto te va a sonar parecido a lo que hacía SOAP con WSDL, pero sin el peso muerto de XML. Del lado del servidor en Java, la implementación queda así: // Implementamos el servicio generado automáticamente desde el .proto public class PedidosServiceImpl extends PedidosServiceGrpc.PedidosServiceImplBase { @Override public void obtenerPedido(PedidoRequest request, StreamObserver<PedidoResponse> responseObserver) { // Acá iría la lógica real, consultando la base de datos PedidoResponse response = PedidoResponse.newBuilder() .setIdPedido(request.getIdPedido()) .setMontoTotal(1500.50) .setEstado("CONFIRMADO") .build(); // onNext envía la respuesta, onCompleted cierra el stream responseObserver.onNext(response); responseObserver.onCompleted(); } } Fijate que no hay JSON por ningún lado. El payload va serializado en binario, lo cual en teoría significa payloads más chicos y parsing más rápido que un JSON equivalente. Y como corre sobre HTTP/2, tenés multiplexación de requests sobre una sola conexión TCP, streaming bidireccional nativo y header compression — las piezas que, en el paper y la documentación oficial de gRPC, explican la ventaja frente a REST + HTTP/1.1 en escenarios de tráfico alto. No tengo benchmarks propios corridos para este post; lo dejo como lo que es, una ventaja de diseño documentada por el proyecto, no una medición mía. Por qué está en la lista La señal de consenso — tres awesome lists independientes mencionándolo — no es casualidad. gRPC apunta a un problema que cualquiera que laburó con microservicios a escala conoce: la falta de contratos fuertes entre servicios. Con REST + JSON, el contrato es una convención que vive en la cabeza de los devs y en una documentación que se desactualiza. Con gRPC, el contrato es código generado: si cambiaste un campo y rompiste algo, el compilador te lo marca, no lo descubrís en producción un viernes a las seis de la tarde. La otra razón es que el code generation elimina el boilerplate de serialización manual — no escribís parsers de JSON a mano, no mantenés DTOs duplicados en cada lenguaje. Eso es valor concreto, independiente de cuánto gane o no en performance tu caso puntual. Es el tipo de herramienta que no brilla en un demo de cinco minutos, pero que tiene sentido cuando tenés una arquitectura con decenas de servicios hablando entre sí. Ahí es donde el ecosistema JVM (si laburás con Spring Boot, hay soporte vía grpc-spring-boot-starter) se beneficia de este approach frente a mantener quince clientes REST hechos a mano. Cuándo NO usarlo Ahora, seamos honestos: gRPC no es gratis, y esto es lo que más me interesa dejar claro. Protobuf y los patrones async/streaming tienen una curva de entrada que no es trivial — si tu equipo nunca laburó con esto, contá con perder tiempo entendiendo por qué el stub generado no hace lo que esperabas en el primer intento. No tengo un número de días para darte porque depende del equipo, pero es una friccion real, no un detalle menor. El debugging distribuido también es más tedioso: no podés pegarle con curl a un endpoint gRPC como hacías con REST, necesitás herramientas extra como grpcurl o BloomRPC para inspeccionar tráfico. Tampoco tiene sentido si tu API la vas a consumir directamente desde un browser (necesitás gRPC-Web con un proxy intermedio, otra capa de complejidad) o si estás armando una API pública que terceros van a consumir sin control sobre su stack — ahí REST + OpenAPI sigue siendo más amigable. Si tu sistema son dos o tres servicios con tráfico bajo, probablemente estés agregando complejidad que no necesitás. Para esos casos, quedate con REST o mirá alternativas más livianas como tRPC si estás en un monorepo TypeScript. Cierre gRPC tiene sentido cuando tenés el problema específico que resuelve: comunicación inter-servicios a escala, con contratos fuertes y una curva de aprendizaje que estás dispuesto a pagar. No lo adoptaría solo porque aparece en tres awesome lists — esa señal te dice que la comunidad lo validó, no que tu equipo esté listo para el costo de entrada. Esto es la entrega #13 de Awesome Curated: The Tools, donde voy desmenuzando herramientas que pasaron el filtro de nuestro sistema de curación. Si te interesa el lado de infraestructura y runtimes, capaz te cope también el post sobre Node.js y su event loop, o si andás por el lado de redes, Sniffnet para monitorear tráfico sin volverte loco. La serie completa está en /blog/series/awesome-curated-tools. --- Este artículo fue publicado originalmente en juanchi.dev

[POTENTIAL FIX] Android cloud save crashes — Found the cause using ADB (tested on Android 16) [FIX FOUND] Idle Skilling Android cloud save crashing — gRPC/SSL error After hours of troubleshooting, I managed to get Idle Skilling working again on my Samsung S25 Ultra. Device information: - Phone: Samsung Galaxy S25 Ultra - Android: 16 - One UI: 8.5 - Game: Idle Skilling (Google Play version) The problem Every time I tried to initialise my cloud save, Google Smart Lock would open and Idle Skilling would crash. After that, the game would repeatedly crash on startup. My original save worked perfectly through the official Google Play Games application on PC, so I knew my progress wasn't corrupted. The actual error I connected my phone to ADB and captured the following crash: java.lang.AssertionError: Method getAlpnSelectedProtocol not supported The error originated from io.grpc.okhttp while connecting to firestore.googleapis.com. It appears to be an incompatibility involving the game's older gRPC networking implementation and Android's restrictions on internal APIs. The workaround WARNING: This changes an Android security/compatibility restriction system-wide. It is an experimental workaround, not a permanent fix, and may have security or stability implications. Proceed at your own risk. 1. Download Google's official Android SDK Platform-Tools on your PC. 2. Enable Developer options and Wireless debugging (or USB debugging) on your Android phone. 3. Connect your phone to ADB. 4. Check your current policy: adb shell settings get global hidden_api_policy Mine originally returned null. 5. Run: adb shell settings put global hidden_api_policy 1 6. Force-stop Idle Skilling: adb shell am force-stop com.lavaflame.IdleSkilling 7. Reopen Idle Skilling and try loading your existing cloud save. My results - Default policy (null) → Game crashes. - Policy set to 1 → Game works. - Reverted to null → Game crashes again. - Re-enabled 1 → Game works again. I also tested restarting my phone, force-stopping the game, re-enabling Google Password Manager's Auto sign-in and switching networks. The game continued working. IMPORTANT: Check your crash logs first I strongly encourage anyone experiencing this issue to capture their Android crash logs before attempting the workaround. Not every Idle Skilling crash necessarily has the same cause. This workaround was tested against a specific gRPC/SSL error, and we don't know whether it'll help with unrelated crashes. Here's how to check: 1. Install Android SDK Platform-Tools on your PC. 2. Connect your phone through USB or Wireless debugging. 3. Open a terminal in the Platform-Tools folder. 4. Run: adb logcat -b crash -c 5. Launch Idle Skilling and reproduce the crash. 6. Run: adb logcat -b crash -d > idle-skilling-crash.txt 7. Open the generated text file and look for: FATAL EXCEPTION: grpc-okhttp-0 java.lang.AssertionError: Method getAlpnSelectedProtocol not supported io.grpc.okhttp.OkHttpProtocolNegotiator If you see those errors, you may be experiencing the same underlying issue. If your logs show a different exception, please share the relevant stack trace (after removing personal information) so we can determine whether it's related. Additional Smart Lock issue Before applying the workaround, I also had to temporarily disable Auto sign-in under Google Password Manager to get the Google account selection screen to appear. After applying the workaround and restoring my cloud save, I was able to re-enable Auto sign-in without further crashes. Reverting the workaround To restore Android's original default hidden API policy: adb shell settings delete global hidden_api_policy This is important because policy 1 relaxes Android's hidden API restrictions for all applications, not just Idle Skilling. Technical references Related gRPC bug: grpc-java#7519 Android hidden API documentation: developer.android.com Android Platform-Tools: developer.android.com Final notes This workaround consistently resolved the crash on my S25 Ultra, but I haven't verified it on other devices or Android versions. Ideally, the game's developer can fix the underlying issue by updating its outdated networking dependencies. If you try this, please reply with: - Your phone model. - Your Android version. - Whether your crash log matches mine. - Whether the workaround worked. Hopefully this helps other people recover their Idle Skilling progress!

gRPC — Your Handler Is a Class You wrote the contract in a .proto file. Then the implementation turned into a function handed to server.addService(), closing over whatever it needed because a function has nowhere to put a constructor. That function can be a class now. The container builds it. The adapter puts it on the wire. Familiar, on purpose @inversifyjs/grpc-js builds a @grpc/grpc-js server from classes registered in an Inversify container. Decorators, status errors, guards, middleware, interceptors, pipes, and error filters live in @inversifyjs/grpc-core. Mark the method. Return the value. Let the pipeline run. Return the message import { RPC, Service } from '@inversifyjs/grpc-core'; import { type HeroRequest, type HeroResponse, HeroServiceService, } from './generated/hero.js'; @Service(HeroServiceService) export class HeroService { @RPC('getHero') public getHero(call: { request: HeroRequest }): HeroResponse { return { name: call.request.id, }; } } @Service() stores the generated service definition and applies @injectable(). @RPC('getHero') is the key ts-proto emits for rpc GetHero when you generate with outputServices=grpc-js. A unary method returns the response message, and the adapter sends it. Bind HeroService before build(). The adapter registers every bound @Service() class and skips the ones you left unbound. build() hands back the @grpc/grpc-js Server. Opening the port stays in your code, so a test can bind an ephemeral port and production can pass ServerCredentials.createSsl. Already constructed a Server? Pass it to the adapter. Your services are registered on that instance, and build() returns it. One proto, four conversations syntax = "proto3"; package hero.v1; service HeroService { rpc GetHero (HeroRequest) returns (HeroResponse); } service HeroUploadService { rpc UploadHeroes (stream UploadRequest) returns (UploadResponse); } service HeroListService { rpc ListHeroes (HeroRequest) returns (stream HeroResponse); } service HeroChatService { rpc Chat (stream ChatMessage) returns (stream ChatMessage); } message HeroRequest { string id = 1; } message HeroResponse { string name = 1; } message UploadRequest { string name = 1; } message UploadResponse { repeated string names = 1; } message ChatMessage { string text = 1; } requestStream and responseStream on the service definition choose how the method sees the call. | Kind | Request | Response | Your method | | --- | --- | --- | --- | | Unary | one message | one message | Returns the response | | Client streaming | a stream | one message | Returns the response when the input ends | | Server streaming | one message | a stream | Writes each message, then calls end() | | Bidirectional | a stream | a stream | Reads and writes the same call | Unary and client-streaming methods can return the message and let the adapter send it. Return undefined when you have already finished the call yourself, for example through @Callback(). Server-streaming and bidirectional methods own the call. You write the messages, and the adapter leaves the return value alone. A server stream is a few writes and an end(): import { type ServerWritableStream } from '@grpc/grpc-js'; import { RPC, Service } from '@inversifyjs/grpc-core'; import { heroListServiceDefinition, type HeroRequest, type HeroResponse, } from './loadHeroServiceDefinition.js'; @Service(heroListServiceDefinition) export class HeroListService { @RPC('ListHeroes') public listHeroes( call: ServerWritableStream<HeroRequest, HeroResponse>, ): void { const id: string = call.request.id; call.write({ name: `${id}-a`, }); call.write({ name: `${id}-b`, }); call.end(); } } That sample loads hero.proto with @grpc/proto-loader, so @RPC() uses the name from the file, ListHeroes. Generate the same service with ts-proto and outputServices=grpc-js, and the key follows the generated definition, the way GetHero became getHero above. The string you pass to @RPC() has to be a key of the definition you passed to @Service(). The constructor was the point import { RPC, Service } from '@inversifyjs/grpc-core'; import { inject, injectable } from 'inversify'; import { type HeroRequest, type HeroResponse, heroServiceDefinition, } from './loadHeroServiceDefinition.js'; @injectable() export class HeroRepository { public findName(id: string): string { return id; } } @Service(heroServiceDefinition) export class RepositoryHeroService { readonly #heroRepository: HeroRepository; constructor(@inject(HeroRepository) heroRepository: HeroRepository) { this.#heroRepository = heroRepository; } @RPC('GetHero') public getHero(call: { request: HeroRequest }): HeroResponse { return { name: this.#heroRepository.findName(call.request.id), }; } } Bind the repository. Bind the service. GetHero uses the repository the container passed into the constructor, the same way any other class in the process gets its collaborators. @Service() accepts a scope of 'Singleton', 'Transient', or 'Request'. Leave it off and the container default applies. @RPC() methods are inherited. A subclass method with the same RPC name replaces the parent, and @Service() goes on the class you bind. The call runs a pipeline Each RPC goes through the same sequence: 1. Pre-handler middleware. Global, then class, then method. 2. Guards, in that same order. 3. Interceptors. Class interceptors wrap method interceptors, which wrap global interceptors. 4. Pipes on decorated parameters. Global pipes run first. 5. The method. 6. Post-handler middleware. Class, then method, then global. A guard that returns false finishes the call with PERMISSION_DENIED. Throw a GrpcError from a guard, middleware, an interceptor, a pipe, or the method, and the client receives that status, on unary calls and on streams. An error that is not a GrpcError, and that no error filter handles, becomes status UNKNOWN with the details Unknown. With logging enabled, the original error is written to the log. import { NotFoundGrpcError, RPC, Service } from '@inversifyjs/grpc-core'; import { type HeroRequest, type HeroResponse, heroServiceDefinition, } from './loadHeroServiceDefinition.js'; @Service(heroServiceDefinition) export class NotFoundHeroService { @RPC('GetHero') public getHero(call: { request: HeroRequest }): HeroResponse { if (call.request.id === 'missing') { throw new NotFoundGrpcError(`Hero ${call.request.id} was not found`); } return { name: call.request.id, }; } } Every non-OK gRPC status has a class: NotFoundGrpcError, UnauthenticatedGrpcError, UnavailableGrpcError, and the rest. Pass your own details, and optional metadata, when the default string is too vague. An error filter translates one of your own exceptions into one of those statuses. Throw a GrpcError from catch, and that is what the client sees. Method filters are chosen before class filters, which are chosen before global filters. Interceptors can reshape a returned unary or client-streaming message before it is sent. A custom parameter decorator can hand the method a value computed from the call, and pipes can transform that value before the method runs. @Call() and @Callback() are the built-in way to take the raw call or the unary callback when you want them. Register global handlers on the adapter before build(). Switch it on npm install inversify reflect-metadata @grpc/grpc-js @bufbuild/protobuf @inversifyjs/grpc-core @inversifyjs/grpc-js npm install -D grpc-tools ts-proto Enable experimentalDecorators and emitDecoratorMetadata. Import reflect-metadata once, before any decorated class loads. Leave logger unset and build() prints each service and the kind of each RPC. logger: false keeps that quiet. Pass a Logger to use yours. The getting started guide walks through code generation and the server bootstrap. From there: services, RPCs, guards, and errors. --- A service definition that deserves a better server? Open an issue on GitHub or bring it to Discord.

io_uring's async writes really can cause some unexpected problems 4/4 grpc-java#11954 vert.x#5077