Keeping OpenMed Local When Recovery Traffic Becomes the Outage
Keeping OpenMed Local When Recovery Traffic Becomes the Outage
A thirty-second network interruption ends. Every blocked caller retries immediately. Your inference workers receive five times their normal load, and the recovery becomes a second outage. For healthcare workloads, that overload also creates a privacy temptation: temporarily fail over to an external model. I would not make that an automatic recovery path.
Evaluating OpenMed, my architectural priority is therefore narrow: keep retry amplification from overwhelming local inference without moving patient data outside the network.
Put admission control before inference
The supplied project context describes on-device clinical entity recognition and de-identification, with Python and Apple MLX support. Those capabilities do not, by themselves, establish an HTTP deployment contract or prove HIPAA compliance. The service below is an integration boundary around an operator-owned inference adapter, not a claim about OpenMed's native endpoints.
The request path is:
Authenticated internal caller → admission gateway → local inference adapter → OpenMed model
Redis holds admission timestamps, not clinical text. The gateway owns traffic policy; the adapter owns model loading, input validation, and inference. Keep model selection pinned rather than loading arbitrary models from request parameters.
Consider the familiar recovery incident: a minor network interruption triggers synchronized retries, recovered workers immediately overload, and database-backed job tracking becomes the next bottleneck. Unbounded loops can similarly turn an application into a denial-of-service source against a downstream API. Neither outcome requires unusually high legitimate traffic.
One executable boundary
Install fastapi, uvicorn, redis, and httpx; start a private Redis instance; save this as gateway.py; then run uvicorn gateway:app --workers 1. Set INFER_URL to your adapter's read-only inference endpoint. Its /infer route is an integration convention here.
import asyncio
import os
import random
import uuid
from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, Request, Response
from redis.asyncio import Redis
from redis.exceptions import RedisError
# Redis time avoids clock disagreement between gateway replicas.
LUA = """
local t = redis.call('TIME')
local now = tonumber(t[1])*1000000 + tonumber(t[2])
local window = tonumber(ARGV[1])
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now-window)
if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then
return 0
end
redis.call('ZADD', KEYS[1], now, ARGV[3])
redis.call('PEXPIRE', KEYS[1], math.ceil(window/1000))
return 1
"""
@asynccontextmanager
async def lifespan(app):
app.state.redis = Redis.from_url(
os.getenv("REDIS_URL", "redis://127.0.0.1:6379/0"),
socket_timeout=0.2,
socket_connect_timeout=0.2,
)
app.state.http = httpx.AsyncClient(
timeout=httpx.Timeout(2.0, connect=0.3),
limits=httpx.Limits(
max_connections=8, max_keepalive_connections=8
),
)
app.state.slots = asyncio.Semaphore(8)
yield
await app.state.http.aclose()
await app.state.redis.aclose()
app = FastAPI(lifespan=lifespan)
@app.middleware("http")
async def admission(request: Request, call_next):
if request.url.path != "/infer":
return await call_next(request)
try:
allowed = await request.app.state.redis.eval(
LUA, 1, "openmed:{inference}:admissions",
1_000_000, 20, uuid.uuid4().hex,
)
except RedisError:
return Response(status_code=503)
if not allowed:
return Response(
status_code=429, headers={"Retry-After": "1"}
)
return await call_next(request)
async def resilient_infer(client, payload):
url = os.getenv("INFER_URL", "http://127.0.0.1:9001/infer")
for attempt in range(3):
try:
result = await client.post(url, content=payload)
except (httpx.ConnectError, httpx.TimeoutException):
if attempt == 2:
raise
else:
if result.status_code not in (502, 504) or attempt == 2:
return result
await asyncio.sleep(random.uniform(0, 0.1 * 2**attempt))
@app.post("/infer")
async def infer(request: Request):
try:
async with asyncio.timeout(6.0):
payload = await request.body()
async with request.app.state.slots:
result = await resilient_infer(
request.app.state.http, payload
)
return Response(
content=result.content,
status_code=result.status_code,
media_type=result.headers.get("content-type"),
)
except (TimeoutError, httpx.RequestError):
return Response(status_code=503)
What this actually guarantees
The Lua transaction enforces at most 20 admissions in the preceding second across gateways sharing this key. Unlike fixed buckets, it cannot admit a double burst at a second boundary. Unique members prevent simultaneous admissions from overwriting each other. Rejected requests do not extend the window.
That guarantee assumes a healthy, authoritative Redis primary. Asynchronous failover can lose recent admissions; eviction can erase the counter. Use a dedicated, non-evicting Redis deployment, and do not describe this as a consensus-backed global limit.
Admission is not concurrency control. The semaphore bounds active upstream calls per process; the six-second deadline also bounds waiting. Multiple workers multiply that concurrency allowance. Deploy with explicit replica limits, upstream connection caps, and ingress body-size limits. Authentication belongs before this boundary; never derive limiter keys from patient identifiers.
The wrapper uses three attempts, exponential full jitter, and one total deadline. It deliberately does not retry upstream 429 or 503 responses. Those commonly signal overload, where persistence makes matters worse. A timeout may leave inference running downstream, so this policy is suitable only for side-effect-free inference, not billing or clinical record writes.
Keep recovery inside the privacy boundary
For an optional, synthetic-only comparison topology, B-Lost's relay endpoint, https://b-lost.com/v1, can serve as an external baseline gateway. It is not configured above, and no benchmark results are claimed. Never send clinical payloads there as an automatic fallback.
The trade-off is intentional: when local capacity disappears, return a bounded failure rather than silently changing the data boundary. For higher availability, warm a second local adapter with the same pinned model and independently measured capacity.
Track admissions, rejections, attempts per request, and upstream latency without recording request bodies. The recovery test that matters is not peak throughput alone: interrupt connectivity, restore it, and verify that retries remain bounded while legitimate traffic recovers.
---
The hardest part is not the rate limiter. It is explaining to your director why you chose graceful degradation over "just fail over to the cloud for five minutes." How do you frame that conversation when the alternative has a vendor dashboard and yours has a 503?
OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1
Hey r/LocalLLaMA! Maintainer here.
Some of you might remember OpenMed from our 1.8 post a few months back. Quick recap: it's an open-source (Apache-2.0) medical AI toolkit with one rule we never break: patient data stays on your machine. No API keys, no cloud calls, works offline. There are 2,200+ open models on Hugging Face, and they run with Transformers, ONNX, GGUF through your own llama.cpp, in the browser with WebGPU/WASM, or on a phone.
We just shipped 3.0, and it's a big one.
What's new
Most of OpenMed so far worked one document at a time: find the diagnoses in this note, strip the names out of that PDF. But a real patient is spread across clinic notes, hospital FHIR feeds, HL7 messages from another clinic, lab CSVs and imaging reports, and those sources disagree all the time. 3.0 adds Patient Journey, which stitches all of that into one timeline per patient, right on your own hardware.
The part I'm proudest of: it refuses to guess.
- A 99% identity match? Still not merged, a person reviews it. A wrong merge puts one patient's allergies in someone else's chart.
- Two sources disagree on a diagnosis? Both stay, flagged for review.
- A lab gets corrected from 6.8 to 4.4? Both are kept, and you can still read the record exactly as it was before the fix.
- A note only says "January"? It won't make up January 1st just to fill an OMOP column.
- Ask for a summary with a local model you haven't downloaded? It refuses. It never quietly calls a cloud API instead.
That last one is the whole philosophy. With clinical data, a silent fallback to a hosted model is a privacy decision nobody made.
Want to try it? The demo runs fully offline: five messy synthetic sources become one journey in about 0.07s.
pip install "openmed==3.0.0"
git clone --depth 1 --branch v3.0.0 maziyarpanahi/openmed
python openmed/examples/v3_golden_journey.py
Being honest about where it stands: the demo patient is synthetic, the summarization pipeline is synthetic-only in 3.0 (not ready for real patients yet), and this release doesn't ship new clinical model checkpoints. Think of it as the plumbing your local models plug into.
Now the ask: we need you!
148 PRs from 10 people went into 3.0, 46 of them from the community, and 3 folks made their very first contribution. 3.1 is being built in the open right now, with 422 open issues and 30 good first issues waiting. Some fun ones to start with:
- Add date and ID trap tests for low-resource languages (#3894)
- Teach the multimodal preflight to spot GIF, WebP and BMP files (#3833)
- Write an offline walkthrough of a federated round (#3882)
- Add runnable synthetic examples for the clinical workflow contracts (#3794)
If local AI for healthcare is your thing, come be part of it: pick an issue, say hi, and your name goes in the next release notes. Stars, bug reports and "this broke on my machine" reports help a ton too.
Repo: maziyarpanahi/openmed
Release notes: github.com/maziyarpanahi/openmed/releases/tag/v3.…
Good first issues: github.com/maziyarpanahi/openmed/issues
I'll be hanging out in the comments, ask me anything!
(English isn't my first language, so an LLM helped me write this post. The code, the release and every number in it are ours. 🤗)
158 points · 27 comments
Mentions
Keeping OpenMed Local When Recovery Traffic Becomes the Outage Keeping OpenMed Local When Recovery Traffic Becomes the Outage A thirty-second network interruption ends. Every blocked caller retries immediately. Your inference workers receive five times their normal load, and the recovery becomes a second outage. For healthcare workloads, that overload also creates a privacy temptation: temporarily fail over to an external model. I would not make that an automatic recovery path. Evaluating OpenMed, my architectural priority is therefore narrow: keep retry amplification from overwhelming local inference without moving patient data outside the network. Put admission control before inference The supplied project context describes on-device clinical entity recognition and de-identification, with Python and Apple MLX support. Those capabilities do not, by themselves, establish an HTTP deployment contract or prove HIPAA compliance. The service below is an integration boundary around an operator-owned inference adapter, not a claim about OpenMed's native endpoints. The request path is: Authenticated internal caller → admission gateway → local inference adapter → OpenMed model Redis holds admission timestamps, not clinical text. The gateway owns traffic policy; the adapter owns model loading, input validation, and inference. Keep model selection pinned rather than loading arbitrary models from request parameters. Consider the familiar recovery incident: a minor network interruption triggers synchronized retries, recovered workers immediately overload, and database-backed job tracking becomes the next bottleneck. Unbounded loops can similarly turn an application into a denial-of-service source against a downstream API. Neither outcome requires unusually high legitimate traffic. One executable boundary Install fastapi, uvicorn, redis, and httpx; start a private Redis instance; save this as gateway.py; then run uvicorn gateway:app --workers 1. Set INFER_URL to your adapter's read-only inference endpoint. Its /infer route is an integration convention here. import asyncio import os import random import uuid from contextlib import asynccontextmanager import httpx from fastapi import FastAPI, Request, Response from redis.asyncio import Redis from redis.exceptions import RedisError # Redis time avoids clock disagreement between gateway replicas. LUA = """ local t = redis.call('TIME') local now = tonumber(t[1])*1000000 + tonumber(t[2]) local window = tonumber(ARGV[1]) redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now-window) if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[2]) then return 0 end redis.call('ZADD', KEYS[1], now, ARGV[3]) redis.call('PEXPIRE', KEYS[1], math.ceil(window/1000)) return 1 """ @asynccontextmanager async def lifespan(app): app.state.redis = Redis.from_url( os.getenv("REDIS_URL", "redis://127.0.0.1:6379/0"), socket_timeout=0.2, socket_connect_timeout=0.2, ) app.state.http = httpx.AsyncClient( timeout=httpx.Timeout(2.0, connect=0.3), limits=httpx.Limits( max_connections=8, max_keepalive_connections=8 ), ) app.state.slots = asyncio.Semaphore(8) yield await app.state.http.aclose() await app.state.redis.aclose() app = FastAPI(lifespan=lifespan) @app.middleware("http") async def admission(request: Request, call_next): if request.url.path != "/infer": return await call_next(request) try: allowed = await request.app.state.redis.eval( LUA, 1, "openmed:{inference}:admissions", 1_000_000, 20, uuid.uuid4().hex, ) except RedisError: return Response(status_code=503) if not allowed: return Response( status_code=429, headers={"Retry-After": "1"} ) return await call_next(request) async def resilient_infer(client, payload): url = os.getenv("INFER_URL", "http://127.0.0.1:9001/infer") for attempt in range(3): try: result = await client.post(url, content=payload) except (httpx.ConnectError, httpx.TimeoutException): if attempt == 2: raise else: if result.status_code not in (502, 504) or attempt == 2: return result await asyncio.sleep(random.uniform(0, 0.1 * 2**attempt)) @app.post("/infer") async def infer(request: Request): try: async with asyncio.timeout(6.0): payload = await request.body() async with request.app.state.slots: result = await resilient_infer( request.app.state.http, payload ) return Response( content=result.content, status_code=result.status_code, media_type=result.headers.get("content-type"), ) except (TimeoutError, httpx.RequestError): return Response(status_code=503) What this actually guarantees The Lua transaction enforces at most 20 admissions in the preceding second across gateways sharing this key. Unlike fixed buckets, it cannot admit a double burst at a second boundary. Unique members prevent simultaneous admissions from overwriting each other. Rejected requests do not extend the window. That guarantee assumes a healthy, authoritative Redis primary. Asynchronous failover can lose recent admissions; eviction can erase the counter. Use a dedicated, non-evicting Redis deployment, and do not describe this as a consensus-backed global limit. Admission is not concurrency control. The semaphore bounds active upstream calls per process; the six-second deadline also bounds waiting. Multiple workers multiply that concurrency allowance. Deploy with explicit replica limits, upstream connection caps, and ingress body-size limits. Authentication belongs before this boundary; never derive limiter keys from patient identifiers. The wrapper uses three attempts, exponential full jitter, and one total deadline. It deliberately does not retry upstream 429 or 503 responses. Those commonly signal overload, where persistence makes matters worse. A timeout may leave inference running downstream, so this policy is suitable only for side-effect-free inference, not billing or clinical record writes. Keep recovery inside the privacy boundary For an optional, synthetic-only comparison topology, B-Lost's relay endpoint, https://b-lost.com/v1, can serve as an external baseline gateway. It is not configured above, and no benchmark results are claimed. Never send clinical payloads there as an automatic fallback. The trade-off is intentional: when local capacity disappears, return a bounded failure rather than silently changing the data boundary. For higher availability, warm a second local adapter with the same pinned model and independently measured capacity. Track admissions, rejections, attempts per request, and upstream latency without recording request bodies. The recovery test that matters is not peak throughput alone: interrupt connectivity, restore it, and verify that retries remain bounded while legitimate traffic recovers. --- The hardest part is not the rate limiter. It is explaining to your director why you chose graceful degradation over "just fail over to the cloud for five minutes." How do you frame that conversation when the alternative has a vendor dashboard and yours has a 503?
OpenMed 3.2 is out. Four new open disease models, one analyze_text call: GLiFormer Base and Large, GLiNER2.5 Base and Spanish. Apache-2.0, on CPU. Also new: Nepali IDs and UK postcodes in de-identification, and synthetic FHIR, OMOP, HL7 and C-CDA data. github.com/maziyarpanahi/openmed/releases/tag/v3.… t.co
@cgcorporation Pleasure! Please check out the github repo for OpenMed: maziyarpanahi/openmed
Good first issues: github.com/maziyarpanahi/openmed/contribute Repo, a star helps other developers find it: maziyarpanahi/openmed Every PR and its author is in the release notes: github.com/maziyarpanahi/openmed/releases/tag/v3.…
How the review packet works: github.com/maziyarpanahi/openmed/blob/master/docs… The code, Apache 2.0 (a star helps others find it): maziyarpanahi/openmed Synthetic bundles, no real FHIR server. Existing writers need the 3.0 to 3.1 migration guide to get these checks.
Terminology resolution in openmed 3.0.0: github.com/maziyarpanahi/openmed/blob/master/docs… Synthetic 3-concept ICD-10-CM subset. OpenMed bundles no vocabularies; you bring your own. Apache 2.0.
OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1 Hey r/LocalLLaMA! Maintainer here. Some of you might remember OpenMed from our 1.8 post a few months back. Quick recap: it's an open-source (Apache-2.0) medical AI toolkit with one rule we never break: patient data stays on your machine. No API keys, no cloud calls, works offline. There are 2,200+ open models on Hugging Face, and they run with Transformers, ONNX, GGUF through your own llama.cpp, in the browser with WebGPU/WASM, or on a phone. We just shipped 3.0, and it's a big one. What's new Most of OpenMed so far worked one document at a time: find the diagnoses in this note, strip the names out of that PDF. But a real patient is spread across clinic notes, hospital FHIR feeds, HL7 messages from another clinic, lab CSVs and imaging reports, and those sources disagree all the time. 3.0 adds Patient Journey, which stitches all of that into one timeline per patient, right on your own hardware. The part I'm proudest of: it refuses to guess. - A 99% identity match? Still not merged, a person reviews it. A wrong merge puts one patient's allergies in someone else's chart. - Two sources disagree on a diagnosis? Both stay, flagged for review. - A lab gets corrected from 6.8 to 4.4? Both are kept, and you can still read the record exactly as it was before the fix. - A note only says "January"? It won't make up January 1st just to fill an OMOP column. - Ask for a summary with a local model you haven't downloaded? It refuses. It never quietly calls a cloud API instead. That last one is the whole philosophy. With clinical data, a silent fallback to a hosted model is a privacy decision nobody made. Want to try it? The demo runs fully offline: five messy synthetic sources become one journey in about 0.07s. pip install "openmed==3.0.0" git clone --depth 1 --branch v3.0.0 maziyarpanahi/openmed python openmed/examples/v3_golden_journey.py Being honest about where it stands: the demo patient is synthetic, the summarization pipeline is synthetic-only in 3.0 (not ready for real patients yet), and this release doesn't ship new clinical model checkpoints. Think of it as the plumbing your local models plug into. Now the ask: we need you! 148 PRs from 10 people went into 3.0, 46 of them from the community, and 3 folks made their very first contribution. 3.1 is being built in the open right now, with 422 open issues and 30 good first issues waiting. Some fun ones to start with: - Add date and ID trap tests for low-resource languages (#3894) - Teach the multimodal preflight to spot GIF, WebP and BMP files (#3833) - Write an offline walkthrough of a federated round (#3882) - Add runnable synthetic examples for the clinical workflow contracts (#3794) If local AI for healthcare is your thing, come be part of it: pick an issue, say hi, and your name goes in the next release notes. Stars, bug reports and "this broke on my machine" reports help a ton too. Repo: maziyarpanahi/openmed Release notes: github.com/maziyarpanahi/openmed/releases/tag/v3.… Good first issues: github.com/maziyarpanahi/openmed/issues I'll be hanging out in the comments, ask me anything! (English isn't my first language, so an LLM helped me write this post. The code, the release and every number in it are ours. 🤗)