grpc

ثغرة في gRPC (C++ وgrpcio) مع xDS: الـ routing بيقرا host والـ RBAC بيقرا :authority. client من غير auth يبعت قيمتين مختلفتين ويعدّي من RBAC rule قافلة الـ route. حدّث لـ 1.84.2 أو 1.85.1، أو خلّي الـ proxy يرفض أي request فيها host و:authority مختلفين. github.com/grpc/grpc/security/advisories/GHSA-v2c…

Hedged Requests in Go: Reducing Tail Latency Without Overloading Dependencies Most requests in a distributed system finish quickly. A few don't. Your API might have a median response time of 40ms, while its p99 latency reaches 900ms. Most users get a fast response, but a small percentage experience significantly longer waits. These slow requests are particularly problematic in systems that depend on multiple services. A single slow dependency can delay an entire operation, even when every other component is responding normally. One way to address this problem is request hedging. Instead of waiting indefinitely for an unusually slow request, the client starts another copy after a configured delay. Whichever request succeeds first wins, and the remaining requests are canceled. This technique can significantly reduce tail latency, but it comes with an important trade-off: it generates additional traffic. If every slow request triggers another request without limits, the mechanism designed to improve latency can overload the dependency it's trying to protect. In this article, we'll explore: - What tail latency is and why averages hide it - How hedged requests work - The difference between hedging and retries - How to implement hedged requests in Go - How to cancel losing requests safely - How to integrate hedging with retry budgets and adaptive concurrency - When hedging helps and when it makes things worse - What to monitor and test in production 1. The Problem: Tail Latency Let's say an API receives 10,000 requests. Its response-time distribution looks like this: | Percentile | Response time | |---|---:| | p50 | 40ms | | p90 | 75ms | | p95 | 120ms | | p99 | 900ms | | p99.9 | 2.5s | Looking only at the average or median would suggest that the service is performing well. But 1% of 10,000 requests means 100 requests take at least 900ms. For latency-sensitive applications, these outliers matter. Consider an API that aggregates data from five independent services: API Request | +-----------+-----------+ | | | Service A Service B Service C | | | Service D Service E | | | | +-----------+-----------+ | Final Response If the API needs all five responses, its total latency is constrained by the slowest dependency. Even if four services respond in 30ms, one service taking 800ms can delay the entire operation. This is one reason tail latency becomes increasingly important as distributed systems grow. Jeff Dean and Luiz André Barroso discuss this problem in their influential paper, The Tail at Scale. They describe techniques for building responsive services despite unpredictable latency in individual components. Reference: research.google 2. What Are Hedged Requests? A hedged request is a duplicate request sent while the original request is still in progress. The client does not immediately send multiple copies. Instead, it waits for a configurable period. If the original request hasn't completed, the client sends another request, ideally to a different replica. The first successful response wins. Time | | Primary request | ────────────────────────────────> | | Wait: 100ms | | Hedge request | ─────────────────────> | | Hedge response: 160ms | Primary response: 450ms | | Return hedge response | Cancel primary request | The important detail is that the second request starts before the first one fails. This is fundamentally different from a traditional retry. A retry waits for an unsuccessful attempt before trying again. A hedge assumes that an unusually slow request might be an outlier and starts another attempt proactively. Why does this work? Imagine two replicas: - Replica A is experiencing temporary contention. - Replica B is healthy. The primary request goes to A. After 100ms, it still hasn't completed, so the client sends a hedge to B. If B responds in 60ms, the client returns that response instead of waiting for A. The total latency becomes approximately 160ms instead of 450ms. The original request might still finish later, but its result is no longer needed. 3. Hedging vs. Retrying These mechanisms are related, but they solve slightly different problems. | Mechanism | When the next attempt starts | Primary goal | |---|---|---| | Retry | After an attempt fails | Recover from transient failures | | Hedging | While an attempt is still running | Reduce tail latency | | Timeout | After a deadline expires | Bound waiting time | | Circuit breaker | When a dependency is unhealthy | Stop sending harmful traffic | A retry responds to failure. A hedge responds to excessive waiting. This distinction matters because hedging can increase downstream load even when the original request would eventually succeed. For that reason, hedging should generally be limited to safe, idempotent operations, especially reads. 4. Choosing the Hedging Delay The hedging delay is one of the most important configuration decisions. If the delay is too short, the client duplicates too many requests. If it is too long, the hedge provides little latency improvement. A reasonable starting point is to use a measured latency percentile for a particular operation. For example: | Percentile | Latency | |---|---:| | p50 | 40ms | | p90 | 70ms | | p95 | 100ms | | p99 | 450ms | A 100ms hedging delay means that requests exceeding the p95 threshold become candidates for hedging. Under stable conditions, approximately 5% of requests may reach that threshold. This does not mean exactly 5% of requests will generate a hedge: requests can complete at the threshold boundary, and actual behavior depends on the latency distribution and scheduling. The important principle is to trigger hedging in the tail, not in the middle of normal traffic. The original Tail at Scale paper discusses using a high-percentile latency threshold to limit duplicate traffic while reducing extreme latency. Its examples demonstrate why carefully selected hedging delays can achieve substantial tail-latency improvements with relatively modest additional load. Source: andrew.cmu.edu 5. Implementing Hedged Requests in Go Let's build a small hedging executor. Our implementation will support: - An immediate primary request - A configurable hedging delay - A maximum number of attempts - Context cancellation - First-successful-response semantics - Retryable error classification - An optional shared retry budget We'll use Go generics so the executor can return any response type. Step 1: Define the policy package hedge import ( "context" "errors" "fmt" "time" ) var ( ErrInvalidPolicy = errors.New("invalid hedging policy") ErrBudgetDenied = errors.New("hedging budget exhausted") ) type Budget interface { TryTake() bool } type Policy struct { Delay time.Duration MaxAttempts int Budget Budget ShouldHedge func(error) bool } type result[T any] struct { value T err error } The Budget interface allows us to reuse the retry budget implementation from the previous article. The original request does not consume a hedge token. Every additional request does. We also use ShouldHedge to distinguish errors that should terminate the operation from errors that may allow another attempt. For example, a validation error should not trigger another request. Step 2: Implement the executor func Do[T any]( ctx context.Context, policy Policy, operation func(context.Context, int) (T, error), ) (T, error) { var zero T if policy.Delay <= 0 || policy.MaxAttempts < 1 || policy.MaxAttempts > 3 || operation == nil || policy.ShouldHedge == nil { return zero, ErrInvalidPolicy } if err := ctx.Err(); err != nil { return zero, err } hedgeCtx, cancel := context.WithCancel(ctx) defer cancel() results := make(chan result[T], policy.MaxAttempts) launched := 0 inFlight := 0 var lastErr error startAttempt := func() { launched++ inFlight++ attempt := launched go func() { value, err := operation(hedgeCtx, attempt) results <- result[T]{ value: value, err: err, } }() } // The primary request starts immediately. startAttempt() var timer *time.Timer var timerC <-chan time.Time if policy.MaxAttempts > 1 { timer = time.NewTimer(policy.Delay) timerC = timer.C defer timer.Stop() } for { select { case <-ctx.Done(): return zero, ctx.Err() case res := <-results: inFlight-- if res.err == nil { // First successful response wins. return res.value, nil } lastErr = res.err if !policy.ShouldHedge(res.err) { return zero, res.err } if inFlight == 0 && launched == policy.MaxAttempts { return zero, lastErr } case <-timerC: if launched >= policy.MaxAttempts { timerC = nil continue } if policy.Budget != nil && !policy.Budget.TryTake() { // No more additional requests. timerC = nil if inFlight == 0 { if lastErr != nil { return zero, lastErr } return zero, ErrBudgetDenied } continue } startAttempt() if launched < policy.MaxAttempts { timer.Reset(policy.Delay) } else { timerC = nil } } } } Let's examine the important decisions. The primary request starts immediately. We don't wait for a timer before sending the first request. Additional requests start only after the configured delay. If the primary finishes successfully before the delay expires, the executor returns without sending a hedge. The first successful response wins. As soon as a successful result arrives, the function returns and cancels the shared context. Errors are classified. A fatal error can terminate the operation immediately. A retryable error can be treated as non-final while another attempt is still running. The number of attempts is bounded. The implementation intentionally limits the example to three total attempts, including the original request. This is a conservative default for the example, not a universal limit. Step 3: Use the executor Imagine a service that reads market data from multiple replicas. type Price struct { Symbol string Value float64 } func fetchPrice( ctx context.Context, attempt int, ) (Price, error) { replica := selectReplica(attempt) return replica.Fetch(ctx, "AAPL") } We can now use the hedging executor: policy := hedge.Policy{ Delay: 100 * time.Millisecond, MaxAttempts: 2, Budget: retryBudget, ShouldHedge: isRetryableError, } price, err := hedge.Do( ctx, policy, fetchPrice, ) if err != nil { return err } fmt.Printf( "%s: %.2f\n", price.Symbol, price.Value, ) The primary request goes to the first replica. If it hasn't completed after 100ms, the executor checks the budget and starts another request. If the second replica responds first, the result is returned and the first request is canceled. The example assumes that selectReplica routes attempts to distinct, suitable replicas. Sending both copies to the same overloaded instance may provide little benefit. 6. Cancellation Is Not the Same as Stopping Work This is one of the most important details in a hedging implementation. When the winning request completes, the executor calls cancel() on the shared context. The losing requests receive cancellation through their contexts. But cancellation is cooperative. It does not forcibly terminate a goroutine, an HTTP request, a database query, or server-side processing. For example, a dependency may continue executing a query even after the client disconnects. A correct operation should propagate its context: func fetch( ctx context.Context, client *http.Client, ) error { req, err := http.NewRequestWithContext( ctx, http.MethodGet, "https://api.example.com/prices/AAPL", nil, ) if err != nil { return err } resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() return nil } The server should also respect request cancellation where possible. Otherwise, the client may stop waiting while the dependency continues doing unnecessary work. This is especially important for expensive database queries and CPU-intensive operations. 7. Why a Hedge Budget Matters Hedging creates additional traffic. Imagine a service receiving 10,000 requests per second. If 5% reach the hedging threshold, that could mean approximately 500 additional attempts per second. Now imagine the dependency is already degraded. More concurrent requests may increase queueing, which increases latency, which triggers more hedges. The mechanism can create the same kind of amplification problem that we discussed in the Retry Budgets article. This is why the hedge budget should be shared across requests. A simple rule: - The original request is always allowed, subject to normal admission controls. - Each additional hedge requires a token. - Successful normal traffic replenishes the budget. - If the budget is exhausted, no new hedges are launched. For example: budget := retry.NewRetryBudget( 20, 0.1, ) The exact budget should be tuned according to the dependency's spare capacity and the service's traffic patterns. The important part is that hedging is not an unlimited permission to duplicate requests. It is an additional workload that must be controlled. 8. Hedging and Adaptive Concurrency Hedging and adaptive concurrency work particularly well together, but they need to share a consistent view of downstream capacity. Adaptive concurrency adjusts how many requests are allowed in flight based on observed dependency behavior. Hedging creates additional in-flight requests. If the adaptive limiter counts only original requests, hedged requests can bypass the intended concurrency ceiling. That defeats the purpose of the limiter. A safer integration is: 1. The primary request acquires a concurrency permit. 2. A hedge must acquire its own permit. 3. If no permit is available, the hedge is skipped. 4. The winning request returns its result. 5. Every attempt releases its own permit when it actually completes. There is an important implementation detail here. Do not release the losing request's concurrency permit merely because the caller stopped waiting for it. The permit must remain held until the underlying operation actually completes or otherwise confirms that the protected work has stopped. Otherwise, the limiter undercounts active downstream work. This is also why a shared cancellation context alone is not enough to guarantee that hedged work has disappeared. 9. Hedging Requires Idempotency Sending the same operation more than once is not always safe. Consider these operations: | Operation | Hedging suitability | |---|---| | Read a market price | Generally suitable | | Read a portfolio | Generally suitable | | Fetch cached configuration | Generally suitable | | Create a payment | Unsafe without idempotency | | Submit an order | Unsafe without deduplication | | Send an email | May produce duplicates | | Update an account balance | Unsafe without strong safeguards | A hedge can reach two different servers before either has completed. Both servers may execute the operation. Canceling one request does not guarantee that its side effects are rolled back. For operations with side effects, use an idempotency key or another server-side deduplication mechanism. Even then, the server must correctly enforce the idempotency contract. Hedging should generally be introduced first for read-only operations where duplicate execution is safe. 10. Where Should Hedged Requests Be Sent? Sending a duplicate request to the same server is often ineffective. If that server is experiencing CPU contention, a long queue, or an overloaded database connection pool, the second request may suffer from the same problem. Ideally, the hedge should target an independent replica. But independence is not guaranteed. Two replicas may share: - The same database - The same network path - The same cache - The same storage system - The same overloaded region This is called correlated failure or correlated latency. If both requests depend on the same struggling resource, hedging may increase load without meaningfully reducing latency. Replica selection should consider failure domains and actual dependency topology. 11. Observability: Measure the Cost of Hedging A hedging system needs more than a request-latency dashboard. You need to understand whether the reduction in tail latency justifies the additional work. Useful metrics include: | Metric | Type | Purpose | |---|---|---| | hedged_requests_total | Counter | Number of additional attempts | | hedge_wins_total | Counter | Requests where a hedge returned first | | hedge_budget_denied_total | Counter | Hedges rejected by the budget | | hedge_canceled_total | Counter | Losing attempts canceled by the client | | hedge_attempts_total | Counter | Total attempts, including originals | | hedge_delay_seconds | Histogram | Time before a hedge is launched | | request_latency_seconds | Histogram | End-to-end request latency | A useful derived metric is the hedge win rate: [ HedgeWinRate = \frac{HedgeWins}{HedgedRequests} ] If hedges rarely win, you may be generating additional traffic without much benefit. Also compare: - p50 latency - p95 latency - p99 latency - Downstream request volume - Dependency CPU and memory usage - Cancellation effectiveness - Error rate The goal is not to minimize p99 at any cost. It is to improve tail latency while keeping the additional workload within the dependency's capacity. 12. Testing Hedged Requests Concurrency behavior needs deterministic tests. At minimum, test these scenarios: - The primary request succeeds before the hedging delay. - The primary request is slow and the hedge wins. - The primary request fails with a retryable error. - A fatal error cancels outstanding attempts. - The hedge budget prevents additional requests. - The parent context is canceled. - The maximum number of attempts is respected. - Losing operations receive cancellation. - A losing operation that ignores cancellation does not release its concurrency permit prematurely. A simple test for the winning hedge could use controlled channels instead of arbitrary sleeps: func TestHedgeWins(t *testing.T) { primaryStarted := make(chan struct{}) hedgeStarted := make(chan struct{}) ctx, cancel := context.WithTimeout( context.Background(), time.Second, ) defer cancel() policy := Policy{ Delay: 20 * time.Millisecond, MaxAttempts: 2, ShouldHedge: func(error) bool { return true }, } result, err := Do( ctx, policy, func( ctx context.Context, attempt int, ) (string, error) { if attempt == 1 { close(primaryStarted) <-ctx.Done() return "", ctx.Err() } close(hedgeStarted) return "hedge response", nil }, ) if err != nil { t.Fatal(err) } if result != "hedge response" { t.Fatalf( "unexpected result: %s", result, ) } <-primaryStarted <-hedgeStarted } The test verifies that the second request can win and that the operation completes successfully. In a complete test suite, also verify that the primary operation observes cancellation. Run the race detector: go test -race ./... Finally, use load tests with a dependency that deliberately introduces variable latency. A useful experiment compares: 1. Hedging disabled. 2. Hedging enabled with a conservative delay. 3. Hedging enabled with an aggressive delay. 4. Hedging enabled with a shared budget. 5. Hedging enabled with adaptive concurrency. Compare both latency improvements and additional downstream traffic. 13. Production Checklist Before introducing hedged requests into a production service: - [ ] Hedge only safe, idempotent operations. - [ ] Select a delay from measured latency distributions. - [ ] Use a strict maximum attempt count. - [ ] Require a shared budget for additional attempts. - [ ] Ensure hedges target independent replicas where possible. - [ ] Apply concurrency limits to every attempt. - [ ] Propagate context cancellation. - [ ] Ensure permits remain held while work is still active. - [ ] Respect the original request deadline. - [ ] Measure hedge win rate and additional traffic. - [ ] Test correlated latency and dependency overload. - [ ] Roll out gradually and compare against a no-hedging baseline. Conclusion Tail latency is one of the most difficult problems in distributed systems because a small number of unusually slow operations can dominate the response time of an entire request. Hedged requests provide a way to reduce that impact. By launching a second request after a carefully selected delay, a client can avoid waiting for an unusually slow replica and use the first successful response. But hedging is not free. Every additional request consumes capacity, and uncontrolled duplication can make an already degraded dependency even slower. The right approach combines measured hedging delays, idempotent operations, cancellation, retry budgets, and adaptive concurrency limits. The goal isn't to send more requests. It's to get the response sooner without making the system slower for everyone else. --- References - The Tail at Scale — Google Research - Request Hedging — gRPC Documentation - gRPC Retry and Hedging Design

gRPC-Web changes the transport, not the authorization boundary gRPC-Web changes the transport, not the authorization boundary A browser cannot speak native gRPC in the same way as a Go or Java client. The gRPC-Web protocol adapts calls for browser constraints and commonly carries request metadata in HTTP headers and response metadata in a trailer frame inside the body. That adaptation creates a useful bridge for web applications, but it does not move the authorization decision into the browser or into the protocol adapter. The first design mistake is treating CORS as authentication. CORS controls which browser origins may read a cross-origin response. It does not prove who sent a request, and it does not stop a server-side client or a script outside the browser. A public CORS policy on a gRPC-Web endpoint can be reasonable if the endpoint still requires a validated bearer token, a session cookie with CSRF protection, or mutual TLS at a trusted proxy. The identity check has to happen before the request reaches service code. The second mistake is trusting a header because it is familiar. A gateway may add x-user-id, x-tenant-id or an authorization context after validating a token. If the same gateway forwards a client-supplied copy of that header, downstream services may see two values or the attacker's value. Gateways should remove untrusted identity headers, authenticate the request, and inject a new context. Services should reject requests that arrive without the expected gateway identity marker and should use the marker only on a network path that ordinary clients cannot reach. Method-level authorization still matters after a valid token is established. A token can be valid for the application while the caller lacks permission for a specific service method or resource. Mapping an HTTP path to a gRPC method is convenient, but the policy should use the canonical service and method names and the resource identifiers in the request. Wildcard access such as * is a change to the authorization model, not a harmless convenience for generated clients. Session cookies need a separate review. gRPC-Web clients often use ordinary POST requests, so browsers may attach cookies automatically. The proxy must apply CSRF defenses to state-changing calls, set SameSite deliberately, and avoid accepting cookies on endpoints that also accept a bearer token unless the precedence rule is clear. If both credentials are accepted, the stronger identity must not be silently replaced by a weaker one from a query parameter or custom header. The wire format also changes what intermediaries see. Metadata becomes headers, and some trailers appear in the response body. A reverse proxy can terminate TLS, enforce request size limits, attach tracing metadata, and forward the adapted response. It can also break streaming, compress messages incorrectly, or remove CORS response headers. Test these behaviors through the same ingress path used in production. A successful call from a local gRPC client says nothing about a browser call through an identity-aware proxy. Logging should connect the user identity, the client application, the gRPC method, the resource, and the authorization result without recording access tokens. When a call is denied, distinguish an expired credential, a valid credential with insufficient scope, a policy mismatch, and a malformed request. That difference matters during incident response because each condition points to a different control. The practical rule is simple: authenticate at the edge, authorize at the service, and verify both through the browser-facing path. gRPC-Web is a compatibility layer. A secure deployment still depends on ordinary HTTP protections, explicit method policy, and an intermediary that knows which identity headers it owns. References - gRPC-Web protocol specification - gRPC metadata guide

gRPCも関係してたんか grpc#43699

Building a Spark Connect Client in OCaml because I'm crazy and you're not But Lisa, surely you must have a good reason for using OCaml? No, not really. Well, yes. Sort of. In a previous blog post I wrote a cheeky little guide to Spark Connect Clients, which I enjoyed writing thoroughly and was pretty well received. But it ended on a warning: not all Connect Clients are built equally, and it highly depends on the language you're using and the love and care of the maintainers. Current gold standards include the Rust Client, for instance, which gives you full interoperability with Rust native engines like Polars or DuckDB. More on that in a follow up post, but it's pretty cool. The follow-up question is what it takes to build one then and how to test it. And to be honest, I've always defaulted to OCaml as sort of a joke language- partially from it's niche quality and mostly from my secret love for the finance industry and beasts like Jane Street. OCaml is a useful language for the demonstration because it lacks all of the built in capabilities that would otherwise make life easy: there is no Arrow library, no concept of the gRPC protocol at all, and a protobuf compiler is its own project with its own parser rather than a plugin for Google's protoc. Every piece of the Connect client has to be manually built; making you realize how good the Go or Rust clients have it in comparison because of their respectively rich ecosystems. Naturally, I vibe coded most of this with my good friend Claude Code, and if you want to skip the entire droll on the implementation journey you can skip ahead to the repo here. Every Spark Connect client essentially has the same four basic requirements: compile the published protos, make a streaming gRPC call, build relation trees, and decode Arrow batches. The Connect server itself does so much of the work for you in this architecture that the only parts costing real effort are the two the protocol leaves to the client implementation: its protobuf toolchain and its Arrow codec. The Roles and Responsibilities of a Connect Client The client sits between the DataFrame calls and the server, and it never processes data itself. It turns method calls into a protobuf tree, sends the tree, and turns the Arrow that comes back into values: !diagram 1 The analyzer, Catalyst and the executors are the server's. The rest is the client's. Step 1: Getting the protos to compile The protocol is its protobuf definitions, published in the Spark repository under sql/connect/common/src/main/protobuf/spark/connect/. A client starts by copying them out of the tag it targets (I used v4.2.0) and compiling them with whatever protobuf toolchain its language has. They import google/protobuf/any.proto and timestamp.proto, and Spark's error details also need google/rpc/status.proto and error_details.proto. None of these are in the Spark repo. If the language's toolchain does not ship them, each is a few lines to write by hand, and wire-compatible as long as the field numbers match. This step is "mechanical" in Go, Rust, Swift and Java, where protoc and a maintained plugin do it. In OCaml the compiler is ocaml-protoc, a from-scratch parser and generator, and Spark's protos hit several things it disagrees with. Each fix is one line in a post-generation fix-up script: | Problem | Where | Fix | | ---------------------------------------------------------------------------------- | --------------------------------------------------------- | ---------------------------------------- | | Single-quoted strings (syntax = 'proto3';), legal but rare | every Spark proto | rewrite to double quotes when vendoring | | reserved "userId", "sessionId"; (field name reservations) | relations.proto | drop the lines; they have no wire effect | | A one-line default emitted before the function it calls exists | the mutually recursive ML relation messages | move the line below its dependency | | Key and payload emitted in the wrong order for a oneof case whose message is empty | Catalog.CurrentDatabase, CurrentCatalog, ClearCache | swap the two generated lines | The oneof bug surfaced the way every protobuf mistake surfaces. Three catalog calls, and only those three, came back from the server as UNKNOWN: Application error processing RPC, with Invalid protobuf byte sequence in the server log. That message means the server could not parse the request at all, and the diagnostic is to dump the bytes: Catalog{current_database} generated: 00 0a length 0, then the key for field 1 Catalog{current_database} correct: 0a 00 key for field 1, then length 0 AnalyzePlan{spark_version} generated: 52 00 same shape, inlined elsewhere, comes out right ocaml-protoc encodes back to front (payload first, then its key, so a nested message's length is known before the key is written), and the standalone encoder for an empty message had the two lines in the wrong order. Nothing else in the API was affected: the bug lives in the encoder generated for a message that is nothing but a oneof, and Catalog is the only such message on the request path with empty cases. A client in Go never meets this, because protoc has been exercised by every Google service for a decade; a client in OCaml meets it immediately. The protobuf compiler is the first dependency the protocol pushes onto the host language, and the Spark protos are a stress test for it. Step 2: Speak gRPC gRPC is small once HTTP/2 is available, and OCaml does have HTTP/2 (h2, with TLS through ocaml-tls and the system trust store). A call is a POST to /spark.connect.SparkConnectService/<Method> with two headers that matter, content-type: application/grpc and te: trailers, plus authorization: Bearer <token> when a token is set. The body in each direction is a sequence of messages, each framed as a compression flag byte, a big-endian length, and the protobuf bytes: let frame payload = let n = String.length payload in let b = Bytes.create (5 + n) in Bytes.set b 0 '\000'; (* not compressed *) Bytes.set_int32_be b 1 (Int32.of_int n); (* length *) Bytes.blit_string payload 0 b 5 n; Bytes.unsafe_to_string b The status comes back in HTTP/2 trailers, not in the response status line (which is always 200), as grpc-status and a percent-encoded grpc-message. Spark adds grpc-status-details-bin, base64 of a google.rpc.Status, which is where the useful error information lives. The reader side is the mirror of frame: buffer the body, peel off messages as their lengths complete, and finish when the trailers arrive. let status_of_headers (h : H2.Headers.t) = match H2.Headers.get h "grpc-status" with | None -> None | Some code -> let code = try int_of_string code with _ -> 2 in let message = Option.value ~default:"" (Option.map percent_decode (H2.Headers.get h "grpc-message")) in let details = Option.map base64_decode (H2.Headers.get h "grpc-status-details-bin") in Some { code; message; details } Spark Connect only uses two call shapes: unary (AnalyzePlan, Config, Interrupt, ReleaseSession, FetchErrorDetails, and friends) and server-streaming (ExecutePlan, ReattachExecute). A unary call is a streaming call where the client reads one message. There are no client-streaming or bidirectional methods except AddArtifacts, which is only needed to upload files to the server. That is the whole transport. It worked on the first query, and almost none of the effort in this project went into it. Step 3: A session, and the first RPC Every request carries the same identity block: a session_id the client mints (a UUID), a user_context with a user_id, and a client_type string that is effectively a user agent. After the first reply the client also records the server_side_session_id the server sent and echoes it back as client_observed_server_side_session_id on every later request, which is how the server notices if it has restarted underneath the client. The hello world of Connect is AnalyzePlan with the spark_version option, because it needs no plan: let analyze t analyze = let req = P.Base.make_analyze_plan_request ~session_id:t.session_id ?client_observed_server_side_session_id:t.server_side_session_id ~user_context:(user_context t) ~client_type:t.client_type ~analyze () in unary t ~method_:"AnalyzePlan" ~encode:P.Base.encode_pb_analyze_plan_request ~decode:P.Base.decode_pb_analyze_plan_response req >|= fun r -> observe t r.server_side_session_id; r let version t = Client.analyze t.client B.Spark_version >|= fun r -> match r.result with Some (B.Spark_version v) -> v.version | _ -> failwith "unexpected response" server version: 4.2.0 Once that prints, the protobuf toolchain, the HTTP/2 stack, the gRPC framing and the session plumbing are all known good, and everything after it is vocabulary. Step 4: Build plans instead of running them Plan building is the most clerical step of the seven. A Relation is a common block holding a plan_id plus a oneof of every operator the server understands: Read, Project, Filter, Join, Aggregate, Sort, Limit, SetOp, Sample, Deduplicate, Range, Sql, LocalRelation, and a long tail. A DataFrame method is a function that wraps the current relation in a new one, and that is the entire DataFrame API: let of_rel_type client rel_type = { client; rel = R.make_relation ~common:(R.make_relation_common ~plan_id:(Client.fresh_plan_id client) ()) ~rel_type () } let filter t cond = of_rel_type t.client (R.Filter (R.make_filter ~input:t.rel ~condition:cond ())) let join ?(how = `Inner) t other ~on = of_rel_type t.client (R.Join (R.make_join ~left:t.rel ~right:other.rel ~join_condition:on ~join_type:(join_type_pb how) ())) Expressions are the same idea one level down. A column reference is an UnresolvedAttribute holding a string; a literal is a Literal with a typed oneof; and every function and every operator is an UnresolvedFunction called by name, so a + b is the function + applied to two arguments and a == b is ==. The client never resolves anything: let fn ?(distinct = false) name args = of_expr_type (E.Unresolved_function (E.make_expression_unresolved_function ~function_name:name ~arguments:args ~is_distinct:distinct ())) let add a b = fn "+" [ a; b ] let eq a b = fn "==" [ a; b ] let gt a b = fn ">" [ a; b ] There is no analyzer in the client: Column.col "nope" is a string in a message until the server sees the plan, which is why a typo fails late under Connect. The client cannot check the name; it has no catalog to check it against. The size of this layer depends on how much the server does, and the server does nearly everything. Each of the following is a server message rather than client code: | What you'd expect to implement client-side | What Connect actually does | | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Rendering the table for show() | ShowString is a relation; the server formats the ASCII table and returns it as one string cell | | count() | an Aggregate with count(1); the client reads back one row | | Schema, printSchema(), explain() | AnalyzePlan with Schema, TreeString or Explain; the server returns the type tree, the formatted text, or the plan string (the only way to see one, since Catalyst isn't on your machine) | | listTables(), currentDatabase(), tableExists() | relations under the Catalog message; they execute like a query and return rows | | sql() for DDL vs for a query | a SqlCommand; the server runs it eagerly and, for a query, hands back a relation to keep building on | | Named and positional SQL parameters | literals attached to the SqlCommand; the server binds :name and ? | | Telling a.id from b.id in a self-join | every relation carries a client-assigned plan_id, and an attribute can name the plan it belongs to | The plan_id is the one piece of real design in this step. Because the client assigns plan_ids, it can mint a column reference bound to a particular DataFrame, and the server resolves it against that side of the join even when both sides have the same column names: let col t name = Column.col_of_plan ~plan_id:(plan_id t) name let a = Dataframe.alias people "a" and b = Dataframe.alias people "b" in Dataframe.join a b ~on:Column.O.(Dataframe.col a "id" = Dataframe.col b "id" + Column.int 1) Step 5: Execute, and decode the Arrow that comes back ExecutePlan takes a Plan (either a Relation to query or a Command to run) and returns a stream of ExecutePlanResponse messages. Each one has a response_type oneof: arrow_batch carries data, sql_command_result carries the relation a SqlCommand produced, result_complete is the end marker, and progress and metrics messages can be ignored at first. The collect loop is a fold over the stream: let collect_arrow t = Client.execute t.client (root t) >>= fun stream -> let schema = ref None and batches = ref [] in Lwt_stream.iter (fun (r : B.execute_plan_response) -> match r.response_type with | Some (B.Arrow_batch ab) -> let sc, bs = Arrow_ipc.read_stream (Bytes.unsafe_to_string ab.data) in (match sc with Some s when !schema = None -> schema := Some s | _ -> ()); batches := List.rev_append bs !batches | _ -> ()) stream >|= fun () -> (!schema, List.rev !batches) Each arrow_batch in ab.data is a complete Arrow IPC stream, a schema message followed by one record batch, not a bare batch sharing a schema across the response. So the decoder is a full IPC stream reader. In a language with an Arrow library this step is one function call. OCaml does not have one (the package on opam binds the C++ library, which defeats the point of a thin client), so this is where the project stopped being clerical. An IPC stream is a sequence of messages, each framed as a continuation marker, a metadata length, a FlatBuffers Message, and a body of raw buffers: FF FF FF FF <int32 metadata length> <FlatBuffer: Message{version, header type, header, bodyLength}> <body> The Message header is a Schema (a vector of Fields, each with a name, a nullability flag, a type union and child fields) or a RecordBatch (a row count, a vector of field nodes giving each column's length and null count, and a vector of buffers giving offsets into the body). Decoding a batch is a depth-first walk: for each field, take the next node, take its validity bitmap buffer, take its data buffer(s), and recurse into children for lists, structs and maps. FlatBuffers itself is a vtable-indirected binary format and reading it is a page or so of offset arithmetic. Then there is one decoder per type. The decoder is the largest file in the project not because Arrow is complicated but because Spark produces many Arrow types: | Spark type | What arrives in the Arrow batch | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------- | | tinyint, smallint, int, bigint | Int with bit width and signedness; a fixed-width buffer | | float, double | FloatingPoint with precision | | string, binary | Utf8 / Binary: an offsets buffer plus a data buffer (and Large variants with 64-bit offsets) | | decimal(p, s) | Decimal: 128-bit (or 256-bit) little-endian two's complement, which needs a bignum-to-decimal-string conversion | | date | Date32, days since the epoch | | timestamp | Timestamp in microseconds, with the session time zone as a string | | timestamp_ntz | the same with no time zone | | interval year to month | Interval(YEAR_MONTH) | | interval day to second | Duration(MICROSECOND) | | array, struct, map | List, Struct, Map: child columns with their own nodes and validity bitmaps | Decimal decoding was the only algorithmic piece, and only because the natural answer (the zarith bignum library) is a C dependency I did not want in a client whose point is having none. Performance without tuning was acceptable: a two-column projection over three million rows (a bigint and a SHA-1 string) came back and decoded in 1.63 seconds on localhost, with the client building a boxed value per cell, the least efficient representation available. The server's time dominates. Step 6: Sending data to the server createDataFrame has to send rows to the server, and Connect's LocalRelation message carries them as, again, an Arrow IPC stream. A client that builds a DataFrame from local values therefore needs an Arrow writer, including a FlatBuffers builder for the schema and record batch metadata: let create_dataframe ?schema t ~columns rows = let data = Arrow_writer.encode ?schema ~columns rows in of_rel_type t (R.Local_relation (R.make_local_relation ~data:(Bytes.unsafe_of_string data) ())) The builder is the fiddly half. FlatBuffers are written back to front with vtables, alignment padding and forward offsets, and a byte out of place produces a Java stack trace on the server rather than anything that points at the byte. So the acceptance test for the writer was not the Spark server, which is forgiving, but pyarrow's validate(full=True), which is not. The stream passed with every Spark type the decoder reads, including nested lists, structs and decimals. The server infers the Spark schema from the Arrow schema, so a local table of Value.Int, Value.String and Value.Float cells comes up as bigint, string and double with no DDL string needed (a DDL string can be passed for narrower types). Every official client carries both halves of this codec; they borrow them from an Arrow library the language already had. Step 7: Errors, the way Spark sends them The grpc-status-details-bin trailer is base64 of a google.rpc.Status, whose details is a list of Any. Spark puts a google.rpc.ErrorInfo in it, and the metadata map on that is where everything useful lives: classes (the JVM exception class chain as a JSON array), sqlState, errorId, and messageParameters. The error id goes to FetchErrorDetails for the server-side stack trace and query context. let status = P.Status.decode_pb_status (Pbrt.Decoder.of_string raw) in let info = List.find_map (fun (a : P.Any.any) -> if ends_with ~suffix:"google.rpc.ErrorInfo" a.type_url then Some (P.Error_details.decode_pb_error_info (Pbrt.Decoder.of_string (Bytes.to_string a.value))) else None) status.details in (* info.metadata: ("classes", "[...]"), ("sqlState", "42P01"), ("errorId", "..."), ... *) The Spark error condition itself is not a separate field; it is the bracketed prefix of the message, so a client lifts it with a trivial parse. The OCaml client's output for a query on a table that does not exist, with no string matching beyond that bracket: Spark error INTERNAL (org.apache.spark.sql.catalyst.ExtendedAnalysisException): [TABLE_OR_VIEW_NOT_FOUND] The table or view `nope` cannot be found. ... SQLSTATE: 42P01; line 1 pos 14; 'Project [*] +- 'UnresolvedRelation [nope], [], false error_class = TABLE_OR_VIEW_NOT_FOUND, sql_state = 42P01, and the exception classes as a list. The server supplies all of it; the client's job is to keep it. Putting it together With those seven pieces the client reads like every other Connect client, because they all converge on the proto's shape: open Lwt.Infix open Spark_connect let () = Lwt_main.run begin Session.connect "sc://localhost:15002" >>= fun spark -> let people = Session.create_dataframe spark ~columns:[ "id"; "name" ] [ [ Value.Int 1; Value.String "ann" ]; [ Value.Int 2; Value.String "bob" ] ] in let orders = Session.create_dataframe spark ~columns:[ "pid"; "amount" ] [ [ Value.Int 1; Value.Float 10.0 ]; [ Value.Int 1; Value.Float 5.5 ] ] in let per_person = Dataframe.join people orders ~on:Column.O.(Column.col "id" = Column.col "pid") |> fun df -> Dataframe.group_by df [ Column.col "name" ] |> fun g -> Dataframe.Grouped.agg g [ Column.alias (F.sum (Column.col "amount")) "total" ] in Dataframe.show per_person >>= fun () -> Session.close spark end +----+-----+ |name|total| +----+-----+ | ann| 15.5| +----+-----+ The two local tables went up inside the plan as LocalRelations, the join and aggregate went up as relation nodes around them, the server formatted the table and returned it as a one-cell Arrow batch, and the whole exchange was a single ExecutePlan call, since nothing before show touches the server. Pointed at a Databricks endpoint with sc://host:443/;token=..., the same client turns TLS on because the token is present, which is what the connection string spec asks for and what every client reimplements. The SQL path is the only one with a wrinkle: sql() sends a SqlCommand and executes eagerly, so CREATE TABLE runs right away, and for a query the reply carries a sql_command_result relation that the client wraps as the DataFrame to keep building on. Config is a unary call with get/set/unset operations, the catalog is relations, writers are WriteOperation (v1, with save modes) and WriteOperationV2 (create/replace/append) commands, and persist is an AnalyzePlan option. Testing its Production Readiness So, here begets the real question: is this actually usable? Well, maybe. As much as I'd love Jane Street to plug this in and hire me to maintain it (though, I'm pretty happy at Databricks right now), there's still some coverage issues, particularly around UDFs which are highly variable anyways. For my own due diligence, I tested the client two ways: every message it can build, exercised against more than one Spark release, and execution on a real cluster of the kind of work I naively imagine someone at Jane Street would reach for OCaml to do. Every message, tested against three Spark releases Coverage is precisely measurable for a Connect client, because the protocol is a finite list of protobuf messages. The repo has a script that reads the generated bindings, finds which relations, expressions, commands, catalog calls and analyze requests the client builds, follows each one back to the public function that builds it, and checks whether a test calls that function. Its output is docs/protocol-coverage.md, regenerated rather than maintained by hand: | Protocol area | Implemented | Of those, tested | | -------------------------------- | ----------- | ---------------- | | Relations (DataFrame operations) | 33 of 61 | all | | Expressions | 14 of 22 | all | | Literal types | 18 of 22 | all | | Catalog | 21 of 36 | all | | AnalyzePlan requests | 11 of 14 | all | | Config operations | 7 of 7 | all | | Commands | 4 of 20 | all | The same suite passes against Spark 4.1.3, 4.2.0 and 4.3.0-rc1. The unimplemented messages are mostly the runtime half of the protocol (streaming, UDFs, ML, pipelines), plus a tail of statistics relations like crosstab and approx_quantile that I have not needed yet. The tests found bugs that a demo doesn't make obvious. na_fill sent whole numbers as 32-bit integer literals, which the server's fill rejects outright (PySpark always sends longs, so it never meets this). Sort orders without an explicit null ordering went up as "unspecified", so an ascending sort put nulls last where Spark and PySpark put them first. And show() did not end with a newline, so two tables printed in a row ran together. The suite also had a bug of its own: it compared a TIMESTAMP literal without pinning the session time zone, so it passed on every server running in UTC (all of my Docker ones) and failed on the one running natively on my machine. Case Study: A small options desk OCaml's best-known home outside academia is finance, where trading firms write their pricing and risk code in it, wirh the same firms keep their market history in data lakes that Spark reads. OCaml and Spark here have historically met through Python or a SQL string. So my E2E is a small options desk with one question: should 30-day index options be priced off realized volatility, measured from recent prices, or off the VIX, the market's implied volatility? The pricing code is borrowed. It is the closed-form Black-Scholes module from Case Ollila's option-pricing-model-comparison (MIT), typed OCaml with Greeks in trader units and every numeric routine written from scratch, vendored unchanged into the example. The market data is public: daily S&P 500, Nasdaq Composite and Dow Jones closes, the VIX, and the 3-month Treasury yield, all from FRED. The whole thing runs as one OCaml program, spark-demos/demos/08_spark_connect_ocaml, against a standalone Spark 4.2 cluster with Delta Lake: !diagram 2 The program is a pipeline, and each step that can check itself does so against an independent calculation: !diagram 3 1. Market data through a Python adapter. The adapter is a short Python data source that downloads FRED series, one partition per series, installed on every node as a pyspark_fred package so Spark discovers it with no per-session registration. From OCaml the adapter is a format name: let fred spark series = Session.read spark |> fun r -> Session.Reader.format r "fred" |> fun r -> Session.Reader.option r "series" (String.concat "," series) |> fun r -> Session.Reader.load r The client never learns that Python is involved: it sends a Read relation naming a format, the executors run the adapter, and the same Arrow batches come back that the decoder reads for any query. The ingest lands ten years of closes in a Delta table (ten years because that is what FRED's agreement with S&P Dow Jones Indices lets it publish for their series, and the job trims the Nasdaq to the same window): +---------+--------+----------+----------+ | series|count(1)| min(date)| max(date)| +---------+--------+----------+----------+ | DJIA| 2514|2016-10-10|2026-10-09| |NASDAQCOM| 2515|2016-10-10|2026-10-09| | SP500| 2514|2016-10-10|2026-10-09| +---------+--------+----------+----------+ 2. The heavy part, in Spark. Daily log returns come from a lag over each series, realized volatility is the standard deviation of the last 21 returns annualized by the square root of 252 trading days, and drawdown is each close against the running maximum, all window functions built in OCaml and evaluated on the cluster: let by_day = Column.Window.partition_by [ col "series" ] |> fun w -> Column.Window.with_order_by w [ col "date" ] in let last_21 = Column.Window.rows_between by_day (Column.Window.Value (Column.int (-20))) Column.Window.current_row in let daily = Dataframe.with_columns prices [ ("ret", F.log Column.O.(col "value" / Column.over (F.lag (col "value")) by_day)) ] |> fun d -> Dataframe.with_column d "rv21" Column.O.(Column.over (F.stddev (col "ret")) last_21 * F.sqrt (Column.int 252)) +---------+----------+--------+--------+------------+----------------+-------------+ | series| date| spot|rv21_pct|avg_rv21_pct|max_drawdown_pct|worst_day_pct| +---------+----------+--------+--------+------------+----------------+-------------+ | DJIA|2026-10-09|51654.95| 10.2| 14.3| -37.1| -13.8| |NASDAQCOM|2026-10-09|27366.17| 14.2| 19.4| -36.4| -13.1| | SP500|2026-10-09| 7811.54| 10.0| 15.0| -33.9| -12.8| +---------+----------+--------+--------+------------+----------------+-------------+ The worst single day for all three is 16 March 2020. The deepest drawdown is March 2020 for the S&P 500 and the Dow, but December 2022 for the Nasdaq, whose 2022 bear market went further than the Covid crash. Joining the S&P 500's trailing realized volatility to the VIX day by day answers half the desk's question: over the decade the VIX sat above realized volatility on 86.1% of trading days, by 3.7 volatility points on average. That gap is a rough measure of the volatility risk premium, the price the market charges for insurance. 3. Pricing, in OCaml. The program pulls the latest close and realized volatility for each index, plus the current VIX and T-bill yield, and hands them to the pricer as its author intended (I set the dividend yield to zero to keep the example small): let m = { T.spot; strike = spot; ttm = 30. /. 365.; rate; div_yield = 0.0; vol = rv series } in let g = BS.greeks kind m in (BS.price kind m, g.delta, BS.vega_per_point g.vega, BS.theta_per_day g.theta) risk-free (3m T-bill) 4.23%, VIX 15.4 index kind contracts strike vol premium delta vega/pt theta/day SP500 call 20 7811.54 10.0% 103.40 0.554 8.85 -1.96 SP500 put -10 7811.54 10.0% 76.28 -0.446 8.85 -1.06 NASDAQCOM call 5 27366.17 14.2% 492.76 0.542 31.13 -9.03 DJIA put 8 51654.95 10.2% 516.15 -0.447 58.56 -7.22 SP500 30-day ATM call: 103.40 at realized vol (10.0%), 151.41 at VIX (15.4%) The same at-the-money S&P call is worth 103.40 at realized volatility and 151.41 at the VIX, 46% more, which is the volatility risk premium expressed in dollars. Put-call parity, the pricer's own check that its calls and puts are consistent with each other, holds for every index to within 5e-13. The priced book then goes back to Spark the other way, as a LocalRelation built from OCaml values and written as a Delta table. 4. The risk layer, as a metric view. Spark 4.2 added metric views (SPARK-54403, SPARK-54405): a view defined in YAML as a source, dimensions, and measures, where each measure is an aggregate defined once and queried with measure() under whatever grouping the query asks for. A risk layer has the same shape, so the desk's exposures are defined once: version: "0.1" source: desk_book dimensions: - name: index_name expr: index_name - name: kind expr: kind measures: - name: premium_usd expr: cast(round(sum(premium * contracts * 100)) as bigint) - name: dollar_delta expr: cast(round(sum(delta * contracts * 100 * spot)) as bigint) - name: vega_usd_per_pt expr: cast(round(sum(vega_pt * contracts * 100)) as bigint) - name: theta_usd_per_day expr: cast(round(sum(theta_day * contracts * 100)) as bigint) The client has no code for metric views at all. Defining one is a SQL statement, and querying one from the DataFrame API is a function call, because measure is a function the server resolves: Dataframe.group_by (Session.table spark "desk_risk") [ col "index_name" ] |> fun g -> Dataframe.Grouped.agg g (List.map (fun m -> Column.alias (Column.fn "measure" [ col m ]) m) [ "premium_usd"; "dollar_delta"; "vega_usd_per_pt"; "theta_usd_per_day" ]) +----------+-----------+------------+---------------+-----------------+ |index_name|premium_usd|dollar_delta|vega_usd_per_pt|theta_usd_per_day| +----------+-----------+------------+---------------+-----------------+ | DJIA| 412920| -18467411| 46844| -5775| | NASDAQCOM| 246379| 7417769| 15563| -4514| | SP500| 130507| 12138611| 8853| -2867| +----------+-----------+------------+---------------+-----------------+ The check computes the same four sums directly from the book table and compares them row by row. They match. 5. The next day. The desk adds to its S&P calls and closes the Nasdaq position, as a MERGE from a DataFrame of fills built in OCaml. Delta keeps the previous book, and VERSION AS OF reads it back: yesterday: [DJIA, put, 8] [NASDAQCOM, call, 5] [SP500, call, 20] [SP500, put, -10] today: [DJIA, put, 8] [SP500, call, 25] [SP500, put, -10] | Check | Compared against | Result | |---|---|---| | Put-call parity, each index | the pricer's own arbitrage identity | pass | | Priced book written to Delta | the number of positions sent | pass | | Risk measures from the metric view | the same aggregation on the book table | pass | | Time travel to yesterday's book | the book before the MERGE | pass | On the cluster, every Spark job succeeded, with the tasks split close to evenly between the two workers' executors. What's left on my to do list, and where the official clients spend their time What this client does not do is also a map of where Connect stops being a plan-and-Arrow protocol and starts being a runtime: - Reattachable execution. The official clients use ReattachExecute and ReleaseExecute so a dropped connection can resume a long query from the last response it saw. This one uses plain ExecutePlan, which Spark 4 servers fully support, but a flaky network on an hour-long query loses the query. It is the first thing to add. - User-defined functions. A UDF in Connect is serialized code shipped to the server: pickled Python, or a Scala closure with its JARs uploaded through AddArtifacts. It only works when the server can run the client's language, and there is no OCaml runtime on a Spark executor. This client will never have UDFs; it has the whole SQL function library and higher-order functions with lambdas. Code that already lives on the server is a different matter: the desk's market-data adapter is Python installed on the executors, and any client can call it by name. - Streaming, MLlib and pipeline commands. The messages are generated and reachable, but each is a stateful API rather than a plan. - Dictionary-encoded or compressed Arrow batches. Spark does not send them by default; the decoder raises if it ever meets one. Each item is either resilience or a place where the client has to be a runtime instead of a plan builder. The line between those halves is the line Connect draws. The first half is small enough to build in a language the Spark project will never target. The second half is where the Go, Swift and Rust clients are spending their effort, and it is the half to check before building on any of them. Further reading - Spark Connect overview and the protobuf definitions that are the real spec - gRPC over HTTP/2, the framing and trailer rules from step 2 - Arrow IPC streaming format and the FlatBuffers schema it's described in - spark-connect-ocaml, the client this post walks through, and the official Go, Rust and Swift clients to compare it against

gRPC shows up when REST starts breaking under its own weight Picture this: you've got fifteen microservices talking to each other. Each one exposes its own REST API, each one has its own JSON with its own naming conventions, and every time someone changes a field in the payments service, three other teams find out when everything blows up in production. You document with Swagger, but the Swagger goes stale. You write HTTP clients by hand, or generate something with OpenAPI that works "more or less." And when you need service A to ask service B something ten thousand times a second, the overhead of JSON + HTTP/1.1 starts hurting your infrastructure bill. That specific pain — inter-service communication with contracts that break silently and performance that doesn't scale — is exactly what gRPC comes to solve. It's not a trend. Google designed it to solve that exact problem inside its own infrastructure, and then released it as an open source project. Today it shows up in three independent awesome lists, which, in the curation system I've been building out through this series, is a strong signal: it's not noise, it's real community consensus. If you've been working with REST your whole career (like I spent a good chunk of time managing infrastructure before diving fully into development), the first time you see gRPC in action it changes how you think. It's not "another HTTP framework." It's a different paradigm for how your services talk to each other. What it does gRPC is a Remote Procedure Call framework that runs over HTTP/2 and uses Protocol Buffers (protobuf) as its default serialization mechanism. Instead of exposing REST endpoints and letting every client build its own request, you define a contract — a .proto file — and from that you generate client and server code in whatever language you need. Go, Java, Python, C++, Node, whatever. Same contract, multiple languages, and the field names and types are enforced by the generated code instead of living only in a doc someone forgot to update. Here's what a basic contract looks like: // pedidos.proto // We define the service and the messages it will exchange syntax = "proto3"; package pedidos; // The service exposes an RPC method, as if it were a remote function service PedidosService { rpc ObtenerPedido (PedidoRequest) returns (PedidoResponse); } // Messages are like DTOs, but typed and versionable message PedidoRequest { string id_pedido = 1; } message PedidoResponse { string id_pedido = 1; double monto_total = 2; string estado = 3; } From that file, the protobuf compiler (protoc) automatically generates serialization classes and client/server stubs. If you come from the Java world (like me, where I started my career as a developer), this is going to sound similar to what SOAP did with WSDL, but without the dead weight of XML and with noticeably better performance. On the server side in Java, the implementation looks like this: // We implement the service automatically generated from the .proto public class PedidosServiceImpl extends PedidosServiceGrpc.PedidosServiceImplBase { @Override public void obtenerPedido(PedidoRequest request, StreamObserver<PedidoResponse> responseObserver) { // The real logic would go here, querying the database PedidoResponse response = PedidoResponse.newBuilder() .setIdPedido(request.getIdPedido()) .setMontoTotal(1500.50) .setEstado("CONFIRMADO") .build(); // onNext sends the response, onCompleted closes the stream responseObserver.onNext(response); responseObserver.onCompleted(); } } Notice there's no JSON anywhere. The payload is serialized in binary, which means smaller payloads and faster parsing. And since it runs over HTTP/2, you get multiplexing of requests over a single TCP connection, native bidirectional streaming, and header compression — the structural reasons gRPC tends to outperform REST under high load. Why it's on the list The consensus signal — three independent awesome lists mentioning it — isn't a coincidence. gRPC solves a problem that comes up fast with microservices: the lack of strong contracts between services. With REST + JSON, the contract is a convention that lives in devs' heads and in documentation that goes stale. With gRPC, the contract is generated code. If you break a field, you find out at compile time, not in production on a Friday at six in the evening. The other big reason is performance. Code generation eliminates manual serialization boilerplate — you don't write JSON parsers by hand, you don't maintain duplicate DTOs in every language. And binary protobuf over HTTP/2 is the combination that gives gRPC an edge over traditional REST under high load, which is exactly the scenario where microservices start to struggle. It's the kind of tool that doesn't shine in a five-minute demo, but becomes indispensable when you have dozens of services talking to each other at real volume. That's where the JVM ecosystem (if you work with Spring Boot, there's solid support via grpc-spring-boot-starter) benefits from this approach versus maintaining fifteen hand-built REST clients. When NOT to use it Now, let's be honest: gRPC isn't free, and this is the part people skip. Protobuf and the async/streaming model are a different mental model from request/response REST — if your team hasn't touched this before, budget real ramp-up time for it, not an afternoon. Debugging is also more tedious: you can't just hit a gRPC endpoint with curl like you did with REST, you need extra tools like grpcurl or BloomRPC to inspect traffic. It also doesn't make sense if you're going to consume your API directly from a browser (you need gRPC-Web with an intermediate proxy, another layer of complexity) or if you're building a public API that third parties will consume without any control over their stack — there, REST + OpenAPI is still more friendly. If your system is two or three services with low traffic, you're probably adding complexity you don't need. For those cases, stick with REST or look at lighter alternatives like tRPC if you're in a TypeScript monorepo. Wrap-up gRPC is the kind of tool you shouldn't adopt because it's trendy, but because you have the specific problem it solves: inter-service communication at scale, with strong contracts and performance that matters. That's why it made it through this series' filters — it's not hype, it's real technical consensus, and it comes with a real learning cost you should plan for, not skip. If you liked this analysis, this is just entry #13 of Awesome Curated: The Tools, where I go through tools that passed our curation system's filter. If you're into infrastructure and runtimes, you might also like the post on Node.js and its event loop, or if you're into networking, Sniffnet for monitoring network traffic without losing your mind. The full series is at /blog/series/awesome-curated-tools. --- This article was originally published on juanchi.dev

gRPC no es moda: es el contrato que REST nunca te dio Imaginate esto: tenés quince microservicios hablando entre sí. Cada uno expone su API REST, cada uno tiene su propio JSON con sus propias convenciones de nombres, y cada vez que alguien cambia un campo en el servicio de pagos, tres equipos más se enteran cuando todo explota en producción. Documentás con Swagger, pero el Swagger se desactualiza. Escribís clientes HTTP a mano, o generás algo con OpenAPI que funciona "más o menos". Y cuando necesitás que el servicio A le pregunte algo al servicio B diez mil veces por segundo, el overhead de JSON + HTTP/1.1 empieza a dolerte en la factura de infraestructura. Ese dolor específico — comunicación inter-servicios con contratos que se rompen en silencio y performance que no escala — es el que gRPC viene a resolver. No es una moda. Google lo diseñó para resolver ese problema puntual dentro de su propia infraestructura, y después lo liberó como proyecto open source. Hoy aparece en tres awesome lists independientes, lo cual en el sistema de curación que vengo laburando en esta serie es señal fuerte: no es ruido, es consenso real de la comunidad. Si vos venís de laburar con REST toda la vida (como yo laburé un buen tiempo administrando infraestructura antes de meterme de lleno al desarrollo), la primera vez que ves gRPC en acción te cambia la cabeza. No es "otro framework HTTP". Es un paradigma distinto para pensar cómo se hablan tus servicios. Qué hace gRPC es un framework de Remote Procedure Calls que corre sobre HTTP/2 y usa Protocol Buffers (protobuf) como mecanismo de serialización por default. En vez de exponer endpoints REST y que cada cliente arme su propio request, definís un contrato — un archivo .proto — y de ahí se genera código cliente y servidor en el lenguaje que necesites: Go, Java, Python, C++, Node. El mismo contrato, múltiples lenguajes, menos espacio para que cada equipo interprete el JSON a su manera. Mirá cómo se ve un contrato básico: // pedidos.proto // Definimos el servicio y los mensajes que va a intercambiar syntax = "proto3"; package pedidos; // El servicio expone un método RPC, como si fuera una función remota service PedidosService { rpc ObtenerPedido (PedidoRequest) returns (PedidoResponse); } // Los mensajes son como DTOs, pero tipados y versionables message PedidoRequest { string id_pedido = 1; } message PedidoResponse { string id_pedido = 1; double monto_total = 2; string estado = 3; } De ese archivo, el compilador de protobuf (protoc) genera automáticamente las clases de serialización y los stubs de cliente/servidor. Si vos venís del mundo Java (como yo, que arranqué ahí mi carrera como developer), esto te va a sonar parecido a lo que hacía SOAP con WSDL, pero sin el peso muerto de XML. Del lado del servidor en Java, la implementación queda así: // Implementamos el servicio generado automáticamente desde el .proto public class PedidosServiceImpl extends PedidosServiceGrpc.PedidosServiceImplBase { @Override public void obtenerPedido(PedidoRequest request, StreamObserver<PedidoResponse> responseObserver) { // Acá iría la lógica real, consultando la base de datos PedidoResponse response = PedidoResponse.newBuilder() .setIdPedido(request.getIdPedido()) .setMontoTotal(1500.50) .setEstado("CONFIRMADO") .build(); // onNext envía la respuesta, onCompleted cierra el stream responseObserver.onNext(response); responseObserver.onCompleted(); } } Fijate que no hay JSON por ningún lado. El payload va serializado en binario, lo cual en teoría significa payloads más chicos y parsing más rápido que un JSON equivalente. Y como corre sobre HTTP/2, tenés multiplexación de requests sobre una sola conexión TCP, streaming bidireccional nativo y header compression — las piezas que, en el paper y la documentación oficial de gRPC, explican la ventaja frente a REST + HTTP/1.1 en escenarios de tráfico alto. No tengo benchmarks propios corridos para este post; lo dejo como lo que es, una ventaja de diseño documentada por el proyecto, no una medición mía. Por qué está en la lista La señal de consenso — tres awesome lists independientes mencionándolo — no es casualidad. gRPC apunta a un problema que cualquiera que laburó con microservicios a escala conoce: la falta de contratos fuertes entre servicios. Con REST + JSON, el contrato es una convención que vive en la cabeza de los devs y en una documentación que se desactualiza. Con gRPC, el contrato es código generado: si cambiaste un campo y rompiste algo, el compilador te lo marca, no lo descubrís en producción un viernes a las seis de la tarde. La otra razón es que el code generation elimina el boilerplate de serialización manual — no escribís parsers de JSON a mano, no mantenés DTOs duplicados en cada lenguaje. Eso es valor concreto, independiente de cuánto gane o no en performance tu caso puntual. Es el tipo de herramienta que no brilla en un demo de cinco minutos, pero que tiene sentido cuando tenés una arquitectura con decenas de servicios hablando entre sí. Ahí es donde el ecosistema JVM (si laburás con Spring Boot, hay soporte vía grpc-spring-boot-starter) se beneficia de este approach frente a mantener quince clientes REST hechos a mano. Cuándo NO usarlo Ahora, seamos honestos: gRPC no es gratis, y esto es lo que más me interesa dejar claro. Protobuf y los patrones async/streaming tienen una curva de entrada que no es trivial — si tu equipo nunca laburó con esto, contá con perder tiempo entendiendo por qué el stub generado no hace lo que esperabas en el primer intento. No tengo un número de días para darte porque depende del equipo, pero es una friccion real, no un detalle menor. El debugging distribuido también es más tedioso: no podés pegarle con curl a un endpoint gRPC como hacías con REST, necesitás herramientas extra como grpcurl o BloomRPC para inspeccionar tráfico. Tampoco tiene sentido si tu API la vas a consumir directamente desde un browser (necesitás gRPC-Web con un proxy intermedio, otra capa de complejidad) o si estás armando una API pública que terceros van a consumir sin control sobre su stack — ahí REST + OpenAPI sigue siendo más amigable. Si tu sistema son dos o tres servicios con tráfico bajo, probablemente estés agregando complejidad que no necesitás. Para esos casos, quedate con REST o mirá alternativas más livianas como tRPC si estás en un monorepo TypeScript. Cierre gRPC tiene sentido cuando tenés el problema específico que resuelve: comunicación inter-servicios a escala, con contratos fuertes y una curva de aprendizaje que estás dispuesto a pagar. No lo adoptaría solo porque aparece en tres awesome lists — esa señal te dice que la comunidad lo validó, no que tu equipo esté listo para el costo de entrada. Esto es la entrega #13 de Awesome Curated: The Tools, donde voy desmenuzando herramientas que pasaron el filtro de nuestro sistema de curación. Si te interesa el lado de infraestructura y runtimes, capaz te cope también el post sobre Node.js y su event loop, o si andás por el lado de redes, Sniffnet para monitorear tráfico sin volverte loco. La serie completa está en /blog/series/awesome-curated-tools. --- Este artículo fue publicado originalmente en juanchi.dev

[POTENTIAL FIX] Android cloud save crashes — Found the cause using ADB (tested on Android 16) [FIX FOUND] Idle Skilling Android cloud save crashing — gRPC/SSL error After hours of troubleshooting, I managed to get Idle Skilling working again on my Samsung S25 Ultra. Device information: - Phone: Samsung Galaxy S25 Ultra - Android: 16 - One UI: 8.5 - Game: Idle Skilling (Google Play version) The problem Every time I tried to initialise my cloud save, Google Smart Lock would open and Idle Skilling would crash. After that, the game would repeatedly crash on startup. My original save worked perfectly through the official Google Play Games application on PC, so I knew my progress wasn't corrupted. The actual error I connected my phone to ADB and captured the following crash: java.lang.AssertionError: Method getAlpnSelectedProtocol not supported The error originated from io.grpc.okhttp while connecting to firestore.googleapis.com. It appears to be an incompatibility involving the game's older gRPC networking implementation and Android's restrictions on internal APIs. The workaround WARNING: This changes an Android security/compatibility restriction system-wide. It is an experimental workaround, not a permanent fix, and may have security or stability implications. Proceed at your own risk. 1. Download Google's official Android SDK Platform-Tools on your PC. 2. Enable Developer options and Wireless debugging (or USB debugging) on your Android phone. 3. Connect your phone to ADB. 4. Check your current policy: adb shell settings get global hidden_api_policy Mine originally returned null. 5. Run: adb shell settings put global hidden_api_policy 1 6. Force-stop Idle Skilling: adb shell am force-stop com.lavaflame.IdleSkilling 7. Reopen Idle Skilling and try loading your existing cloud save. My results - Default policy (null) → Game crashes. - Policy set to 1 → Game works. - Reverted to null → Game crashes again. - Re-enabled 1 → Game works again. I also tested restarting my phone, force-stopping the game, re-enabling Google Password Manager's Auto sign-in and switching networks. The game continued working. IMPORTANT: Check your crash logs first I strongly encourage anyone experiencing this issue to capture their Android crash logs before attempting the workaround. Not every Idle Skilling crash necessarily has the same cause. This workaround was tested against a specific gRPC/SSL error, and we don't know whether it'll help with unrelated crashes. Here's how to check: 1. Install Android SDK Platform-Tools on your PC. 2. Connect your phone through USB or Wireless debugging. 3. Open a terminal in the Platform-Tools folder. 4. Run: adb logcat -b crash -c 5. Launch Idle Skilling and reproduce the crash. 6. Run: adb logcat -b crash -d > idle-skilling-crash.txt 7. Open the generated text file and look for: FATAL EXCEPTION: grpc-okhttp-0 java.lang.AssertionError: Method getAlpnSelectedProtocol not supported io.grpc.okhttp.OkHttpProtocolNegotiator If you see those errors, you may be experiencing the same underlying issue. If your logs show a different exception, please share the relevant stack trace (after removing personal information) so we can determine whether it's related. Additional Smart Lock issue Before applying the workaround, I also had to temporarily disable Auto sign-in under Google Password Manager to get the Google account selection screen to appear. After applying the workaround and restoring my cloud save, I was able to re-enable Auto sign-in without further crashes. Reverting the workaround To restore Android's original default hidden API policy: adb shell settings delete global hidden_api_policy This is important because policy 1 relaxes Android's hidden API restrictions for all applications, not just Idle Skilling. Technical references Related gRPC bug: grpc-java#7519 Android hidden API documentation: developer.android.com Android Platform-Tools: developer.android.com Final notes This workaround consistently resolved the crash on my S25 Ultra, but I haven't verified it on other devices or Android versions. Ideally, the game's developer can fix the underlying issue by updating its outdated networking dependencies. If you try this, please reply with: - Your phone model. - Your Android version. - Whether your crash log matches mine. - Whether the workaround worked. Hopefully this helps other people recover their Idle Skilling progress!