starlette

Optimizing Async Event Loops: How I Dropped Server Shutdown Latency from 200ms to <1ms in Uvicorn Optimizing Async Event Loops: How I Dropped Server Shutdown Latency from 200ms to 1ms in Uvicorn In high-concurrency microservices, server shutdown latency is often treated as an afterthought. Engineers spend months shaving microseconds off request handlers, yet accept multi-second teardown delays during rolling container deployments, CI runner teardowns, and automated test harnesses. While profiling ASGI server lifecycles in Uvicorn, the lightning-fast ASGI server powering FastAPI and Starlette, I investigated an architectural latency tax: idle shutdown requests could experience 100–200ms of delay when the server relied on periodic sleep polling. By exploring a wakeable asyncio.Event synchronization primitive in connection with Uvicorn PR #3238, I investigated how to reduce this delay from approximately 200ms to sub-millisecond response times. The reported benchmark results include a median latency of 0.42ms, representing an approximately 235× improvement over a 98.6ms baseline. Here is the root-cause analysis, the event-loop mechanics, the architectural comparison, and the reported benchmark numbers. --- 1. The Anatomy of the Bug: The 200ms Sleep Tax Uvicorn's main server lifecycle coordinates socket binding, lifespan events, signal handling, and connection management. In a polling-based design, the server repeatedly checks whether it should exit: # Simplified polling-based implementation while not self.should_exit: await asyncio.sleep(0.2) Whenever an operator requests shutdown, changes the exit state, or sends a termination signal, the server must eventually resume execution and inspect that state. If the coroutine is suspended inside asyncio.sleep(), the shutdown request may remain unnoticed until the sleep completes. The Problem with Periodic Sleep Polling Time (ms) ─────────────────────────────────────────────► 0 ms 100 ms 200 ms │ │ │ ▼ ▼ ▼ Sleep begins Timer expires Next check ▲ │ Shutdown requested at 42 ms │ └──── Up to 158 ms waiting for the timer The actual delay depends on the polling interval, when the shutdown request arrives, and how quickly the event loop schedules the coroutine. Why This Matters Unpredictable latency: A shutdown request arriving immediately after a sleep begins may wait almost the entire interval. Compounding test-suite overhead: Repeatedly starting and stopping application servers can accumulate unnecessary waiting time. Container termination: During Kubernetes rollouts and other containerized deployments, prompt shutdown signaling helps the application begin its termination procedure sooner. It does not, by itself, guarantee faster completion of graceful shutdown. Developer productivity: Reducing unnecessary teardown delays can improve integration-test turnaround and local development workflows. --- 2. Architectural Comparison Before: Polling Loop The following sequence illustrates a polling-based implementation. sequenceDiagram autonumber participant Loop as asyncio Event Loop participant Server as Uvicorn Server Loop participant Caller as Shutdown Trigger Server->>Loop: await asyncio.sleep(0.2) Note over Server,Loop: Coroutine is suspended Caller->>Server: Request shutdown Note over Server: Exit state changes Loop-->>Server: Sleep completes Server->>Server: Check should_exit Server->>Server: Begin graceful shutdown The problem is that changing a Boolean flag does not automatically wake a coroutine suspended in asyncio.sleep(). The coroutine must resume before it can inspect the updated state. After: Event-Driven Wake-Up An event-driven design provides an explicit notification mechanism. sequenceDiagram autonumber participant Loop as asyncio Event Loop participant Server as Server Loop participant Event as asyncio.Event participant Caller as Shutdown Trigger Server->>Event: await event.wait() Note over Server,Event: Coroutine suspends until signaled Caller->>Event: event.set() Event-->>Loop: Schedule waiting coroutine Loop-->>Server: Resume coroutine Server->>Server: Check shutdown state Server->>Server: Begin graceful shutdown Python's asyncio.Event is designed for signaling between asynchronous tasks. Instead of periodically checking whether shutdown has been requested, the server can wait for a notification and resume when the event is set. Important: Event-driven notification removes the need to wait for the polling interval, but the coroutine still depends on event-loop scheduling. Sub-millisecond latency is a benchmark result, not a guaranteed property of asyncio.Event. --- 3. The Implementation The objective is to make shutdown signaling responsive without busy-waiting or continuously consuming CPU resources. The following examples illustrate the design. They are simplified examples, not a verified reproduction of the exact production diff in PR #3238. Step 1: Add a Shutdown Event import asyncio class Server: def __init__(self) -> None: self.should_exit = False self.stop_event: asyncio.Event | None = None The event represents a notification that the server's main loop should stop waiting. Step 2: Initialize the Event During Startup async def serve(self) -> None: self.stop_event = asyncio.Event() await self.startup() if self.should_exit: await self.shutdown() return await self.main_loop() await self.shutdown() Initializing the event within the asynchronous lifecycle makes its ownership and initialization point explicit. A production implementation must also account for startup failures, repeated invocations, and the server's existing lifecycle guarantees. Step 3: Replace Periodic Polling A simple event-driven loop can look like this: async def main_loop(self) -> None: assert self.stop_event is not None while not self.should_exit: await self.stop_event.wait() This example assumes that the event is the only thing the loop needs to await. If the server must also perform periodic work, manage reloads, or observe worker state, that work must be incorporated into the design rather than silently removed. For example, a timeout can be used when periodic maintenance is required: async def main_loop(self) -> None: assert self.stop_event is not None while not self.should_exit: try: await asyncio.wait_for( self.stop_event.wait(), timeout=0.5, ) except asyncio.TimeoutError: # Perform required periodic maintenance. pass Python's asyncio.wait_for provides timeout-based waiting. A timeout should not be added without a reason. If no periodic work is needed, waiting directly on the event avoids unnecessary timer wake-ups. Step 4: Signal the Event During Shutdown def request_shutdown(self) -> None: self.should_exit = True if self.stop_event is not None: self.stop_event.set() Calling set() wakes tasks waiting on the event. If shutdown can be requested from another thread, the implementation must respect asyncio's thread-safety requirements. A common pattern is: loop.call_soon_threadsafe(stop_event.set) See the official asyncio event-loop documentation for details. Critical Edge Cases Lazy initialization: Create and use the event in an appropriate asynchronous lifecycle context, accounting for the Python versions and process model supported by the application. Periodic maintenance: If the loop is responsible for health checks, worker monitoring, or reload behavior, preserve those responsibilities in the new design. Signal handling: Ensure termination signals reliably reach the event-loop thread and trigger the appropriate shutdown procedure. Repeated shutdown requests: Make shutdown handling idempotent so multiple signals do not cause inconsistent lifecycle transitions. Graceful shutdown: Waking the main loop is not equivalent to completing shutdown. Active requests, connection draining, application lifespan hooks, and resource cleanup may still take time. --- 4. Benchmark Results The following table contains the measurements reported in this investigation. These figures have not been independently verified here against the linked pull request. Reproduce the benchmark against the relevant source revision before presenting the results as independently confirmed findings. | Metric | Legacy Polling | Event-Driven Approach | Reported Improvement | | ------------------------ | -------------: | --------------------: | -------------------: | | Minimum shutdown latency | 12.4 ms | 0.18 ms | 68.9× | | Median (p50) | 98.6 ms | 0.42 ms | 234.8× | | 99th percentile (p99) | 198.2 ms | 0.89 ms | 222.7× | | Idle CPU utilization | 0.02% | 0.02% | No reported change | | 100-test teardown suite | 18.42 s | 0.12 s | 153.5× | The speedup ratios are calculated from the reported latency values and rounded. Shutdown Latency Distribution Lower latency is better. Legacy polling (p99) ████████████████████████████████████████ 198.2 ms Event-driven approach (p99) ▏ 0.89 ms How to Interpret These Results The reported median decreases from 98.6ms to 0.42ms. That corresponds to approximately: 98.6 / 0.42 = 234.8× faster The reported p99 decreases from 198.2ms to 0.89ms, corresponding to approximately 222.7× faster. However, a reproducible performance report should document: - The exact source commit and Python version - Operating system and hardware - The shutdown-trigger mechanism - The number of iterations and warm-up procedure - Whether the measurement covers idle-loop wake-up or complete graceful shutdown - Whether active connections and application hooks are involved - The method used to measure CPU utilization - Raw measurements and percentile calculation methods The reported 100-test teardown result also requires an isolated benchmark to establish whether the improvement comes entirely from shutdown signaling or from other differences in test execution. --- 5. Key Engineering Lessons 1. Avoid Unnecessary Polling in Event-Driven Systems asyncio.sleep() is useful for scheduled work and periodic tasks. However, when a coroutine is waiting for a specific state transition, an explicit synchronization primitive may be more appropriate than repeatedly checking a Boolean flag. The right choice depends on the lifecycle requirements, not a blanket rule that sleeping is always an anti-pattern. 2. Low CPU Usage Does Not Mean Low Latency A suspended coroutine can consume almost no CPU while still introducing significant response delay. Performance engineering must measure both resource utilization and responsiveness. 3. Shutdown Responsiveness and Shutdown Duration Are Different A server can respond to a shutdown request quickly but still take time to drain connections and clean up resources. Measure these phases separately to avoid optimizing the wrong bottleneck. 4. Performance Claims Need Reproducible Evidence Benchmark improvements are most useful when other engineers can reproduce them. A strong report includes the source revision, test harness, environment, sample count, raw results, and measurement methodology. 5. Small Lifecycle Improvements Can Compound In CI environments and automated integration tests, unnecessary waits may be repeated many times. Even a small improvement can become meaningful when multiplied across a large number of server lifecycle operations. --- Conclusion Server shutdown latency deserves the same engineering attention as request-processing latency. Periodic polling can delay a shutdown response when a coroutine must wait for a timer before checking its state. An event-driven approach using Python's asyncio.Event provides an explicit notification mechanism that can remove this polling delay. The implementation must still preserve signal handling, worker monitoring, active-connection management, and graceful-shutdown behavior. The broader lesson is simple: Measure the lifecycle, identify the actual bottleneck, and choose synchronization primitives that match the behavior your system needs. --- Article Author Attribution Written by Raj Gahoi B.Tech CSE | Open-Source Contributor | Software Developer Connect with me on GitHub · LinkedIn --- References - Uvicorn repository - Uvicorn PR #3238 - Python asyncio synchronization primitives - Python asyncio tasks and timeouts - Python asyncio event loop - FastAPI repository - Starlette repository

Starlette 1.10.0 is out! 🚀 - Faster URL-encoded form parsing with less memory - url_path_for() keeps values that look like placeholders - Handled exceptions no longer pin requests in memory Have a nice weekend, and happy coding! github.com/Kludex/starlette/releases/tag/1.10.0

Starlette 1.8.0 made the Python 3.10 goodbye official. Python 3.11 or later is now required, and HTTP 103 Early Hints gets `Request.send_early_hints()`. #Python #ASGI github.com/Kludex/starlette/blob/main/docs/releas… Nandann can help: nandann.com t.co

vLLM 0.32 stops letting any website call your model server Run vLLM 0.31 with default flags and it tells every browser that any website may call it. A page on some unrelated site can have its JavaScript send chat requests to the vLLM server on your laptop or your office network, and read the answers, because the server replies Access-Control-Allow-Origin: *. vLLM 0.32 turns that off. The change landed on October 11 in a commit titled "Restrict default CORS origins" and shipped in the 0.32.0rc0 release candidate on October 12. The current stable release, 0.31.0, still has the wildcard default, and so did releases as far back as 0.6. The default flips from everyone to no one The --allowed-origins flag now defaults to [] instead of ["*"], in both the Python server and the Rust frontend. vLLM hands the value straight to Starlette's CORSMiddleware, together with --allowed-methods, --allowed-headers and --allow-credentials, whose defaults (["*"], ["*"] and false) haven't changed. With an empty list, a browser's preflight gets a 400 with no Access-Control-Allow-Origin header, so the browser never sends the JSON POST that would follow. Clients that aren't browsers don't do CORS. curl, the OpenAI SDK, your gateway and any agent running on a server keep working after the upgrade exactly as before. Who is exposed today, and who breaks on upgrade You're exposed on 0.31 if a vLLM server runs without --api-key anywhere a browser can reach it: localhost on a developer laptop, a GPU box on the office network, a cluster service reachable over the VPN. Any page that person opens can spend your GPU time and read the output. Setting --api-key helps, because the page can't supply the key, but the key only guards the /v1, /v2, /inference and /cohere prefixes. vLLM's security guide lists other inference routes, /invocations, /score and /pooling among them, that stay open even with a key. You break on upgrade if a web page served from a different origin calls vLLM straight from the browser: an internal chat UI, a playground on another port, a notebook front end. Those calls start failing with a CORS error in the browser console. Origins have to match exactly on scheme, host and port, so http://localhost:3000 and http://127.0.0.1:3000 count as two different origins. The Python server passes no origin pattern to Starlette, so there's no *.example.com shortcut: list every origin. Set the flag on 0.31 now Don't wait for 0.32. Set the origins yourself today and the upgrade becomes a no-op. The value is a JSON list: # Only these browser origins may call the server; everyone else is denied vllm serve Qwen/Qwen3-8B \ --api-key "$VLLM_API_KEY" \ --allowed-origins '["https://chat.internal.example"]' If no browser should ever talk to vLLM directly, which is the normal setup behind a gateway, pass --allowed-origins '[]' and you get the 0.32 behavior on 0.31. Then check from the outside, the way a browser would. This preflight asks whether a page on another site may send an authenticated JSON POST: curl -si -X OPTIONS http://vllm.internal:8000/v1/chat/completions \ -H 'Origin: https://some-other-site.example' \ -H 'Access-Control-Request-Method: POST' \ -H 'Access-Control-Request-Headers: authorization,content-type' \ | grep -i -e '^HTTP' -e 'access-control-allow-origin' A locked-down server answers 400 with no access-control-allow-origin line. If you see *, or the origin you sent echoed back, any website can use that server. CORS is not access control CORS only decides what a browser lets a page do. Requests that don't need a preflight, such as a plain GET, still reach vLLM and run. The browser just hides the response from the page. In the test for this article, GET /v1/models from a foreign origin still returns 200 under the new default, only without the header. So keep --api-key, keep the server off the public internet, and put a reverse proxy in front that exposes only the routes you use, as vLLM's security guide recommends. Never pair ["*"] with --allow-credentials. Starlette then echoes back whichever origin asked, which is the wildcard again, now with cookies allowed. List the origins instead. Your ingress may be answering for vLLM. If NGINX, Envoy or an API gateway in front of the server adds its own CORS headers or handles OPTIONS itself, vLLM's default never comes into play. Run the curl check against the URL your users' browsers actually hit, not against the pod. Do this today Run that preflight against every vLLM endpoint you own, using the hostname a browser would see. Wherever it comes back with *, add --allowed-origins with the exact origins of your UIs, or '[]' if you have none, and roll the deployment. Checked against the vLLM v0.32.0rc0, v0.31.0 and v0.6.0 source (cli_args.py, middleware/register.py) and the security guide in the repo. The CORS behavior was reproduced with Starlette 1.6.0 and uvicorn 0.53.0 in a stand-in app that wires CORSMiddleware with vLLM's arguments; the code and tests are in articles/examples/vllm-cors in this site's repo. InferPrice didn't run vLLM itself or test it on GPU hardware. Sources - [vLLM commit fcae5c9: "[Security] Restrict default CORS origins" (#61009), Oct 11, 2026](github.com/vllm-project/vllm/commit/fcae5c98870ff…) - vLLM v0.32.0rc0: FrontendArgs in cli_args.py (allowed_origins, api_key) - vLLM v0.32.0rc0: CORS middleware registration (register.py) - vLLM v0.31.0: cli_args.py with the ["*"] default - vLLM security guide (docs/usage/security.md at v0.32.0rc0): API key limitations and reverse proxy advice - Starlette 1.6.0: CORSMiddleware source - MDN: Cross-Origin Resource Sharing (CORS) --- Originally published at inferprice.pages.dev.