nats-io

Dark Matter Engine: A Sloth That Hunts for Missing Answers at 2 AM What I Built Every spec I have ever read answers the questions its author thought of. The expensive bugs live in the ones they didn't. Here is a real sentence from a small checkout spec I wrote for this project: > If the gateway responds with a 500 error, the service retries the charge up to 3 times with exponential backoff. Reasonable. Probably even tested. Now picture week three of a sprint, when someone asks: what if the gateway charges the card, and the response never arrives? The retry fires. Does the customer pay twice? Nothing in the docs says, because nobody asked. That missing answer is what I went hunting for. I call it dark matter: you can't see it directly, but you can find the hole by looking at everything around it. The Dark Matter Engine is an offline scan that reads your docs, asks "what could go wrong here?", and reports the questions your documentation never settled. It runs at 2 AM. A local Gemma does the reading. Plain Python checks its work. TL;DR. The Dark Matter Engine is an offline scan that reads your docs and lists the failure questions they never answer. A local Gemma 3 (4B) model proposes and judges scenarios, and plain Python checks every quote it relies on. A Task Scheduler job runs it at 2 AM. On my nine planted test pairs the default check flagged 6 of 9 omissions and wrongly flagged 2 of 45 documented behaviors; a stricter mode flagged 9 of 9 but wrongly flagged 23. On a real design document it found one genuine unanswered question and produced eight findings I had to throw out. It's a prototype that produces candidates for a human to review, not verdicts. MIT licensed, Python standard library only. Who gets to be lazier: anyone who reviews specs or design docs and would rather not brainstorm every failure case by hand. Point it at a folder of Markdown or text. For each behavior the docs describe, it proposes a failure scenario, searches the docs for an answer, and reports whether the text actually establishes one. My demo is two versions of the checkout spec. Version 1 documents retries on a gateway error and says nothing about a lost response. Version 2 adds one section on idempotency keys. For the scenario "the gateway charges the card but the response is lost and the client retries": - Version 1: Partially answered. Gemma cites only the sentence about the normal payment call and asks the right question: how does the system ensure the order is charged exactly once when retries are involved? The spec never says, and the engine flags it as unresolved. - Version 2: Answered, with the exact sentence and its line number: the service returns the stored result and never charges the order a second time. The verdicts are generated from the specification text, not hard-coded for these two versions. A warning before the screenshots: this is a prototype that produces candidates for a human to review, not verdicts. On a real design document from another project it found one genuine unanswered question and also produced eight findings I had to throw out. The details are below, because they're the most interesting part. !Dark Matter Engine report for version 1 of the checkout spec: the lost-response scenario is marked Partially answered, with the question to resolve and the cited sentence from spec.md line 4. !Report for version 2: the same scenario is marked Answered, citing the idempotency-key sentence at spec.md line 16, which says the service never charges the order a second time. Demo {% embed youtube.com %} !Results table from the offline demo run: the lost-response scenario changes from Partially answered in version 1 to Answered in version 2, with benchmark rows below. To reproduce it yourself: ollama pull gemma3:4b python dark_matter.py demo --quick That scans the checkout pair and runs the benchmark on it (about 5 minutes on my laptop), then writes one results page you can open in a browser. It's quicker than the 8.5-minute scan I mention below because the demo uses pre-written scenarios, while a plain scan first has the model propose scenarios from every section. I also left the full result files from my runs in the repo. Code {% embed Emmimal/dark-matter-engine %} MIT licensed. Python standard library only; Ollama is the one external runtime. The logs, result files and the offline recording are in the repo's evidence/ folder. Model: gemma3:4b, digest 68ceab7e468a. How I Built It I used Google's open-weight Gemma 3 (4B), run locally through Ollama, with temperature 0 and a fixed seed. Everything else is standard-library Python. The steady part: how it avoids inventing gaps Ask a language model "what's missing?" and it will cheerfully make things up. So the work is split: 1. Gemma proposes scenarios from sections the docs actually contain. 2. Plain lexical search fetches evidence before the model judges anything. 3. Gemma classifies: Answered, Partially answered or Not established. (A Contradicted status exists in the code, but my corpus has no contradiction case, so I haven't tested it.) 4. Python checks that every quote exists in the retrieved text. A quote it can't find makes the finding Unverified, never a gap. 5. An Answered claim gets a second check. A Not established result gets a second search with a hand-written synonym list, then is classified again. Gemma decides what the text establishes. Python decides whether Gemma is allowed to say so. The slow part: does it work? I wrote nine small spec pairs, each with one planted omission. That's 54 scenarios: 9 planted gaps and 45 documented behaviors. The expected answers live in a separate file the scanner never reads. The first two pairs were used to tune the prompts. Three more were run once after tuning, with expected answers written beforehand but not committed before the run. For the last four, I committed the expected answers before running any model on them (commit d11c484). I wrote those four after the worker-crash failure below, so they deliberately include similar traps. I changed no prompt or engine logic after seeing results on them; I only added per-scenario reporting. | Method | Planted gaps flagged | Documented behaviors wrongly flagged | |---|---|---| | Gemma, default check | 6 of 9 | 2 of 45 | | Gemma, stricter two-step check | 9 of 9 | 23 of 45 | | No-model word-coverage baseline (threshold 0.5) | 9 of 9 | 32 of 45 | | Same baseline (threshold 0.4) | 6 of 9 | 19 of 45 | "Flagged" means Not established or Partially answered. The baseline is a simple rule: how many of the scenario's words appear in the top three retrieved passages. I'm showing its whole sweep, not one hand-picked setting: thresholds 0.6, 0.7 and 0.8 also catch 9 of 9, but flag 37, 42 and 45 of the 45 documented behaviors. At the same 6 of 9, the baseline wrongly flags 19 of 45 and the default check flags 2. These are two different methods, so this compares outcomes on my test set. It doesn't show that Gemma is better in general, and it says nothing about stronger retrieval than my simple one. What it does show is what the model contributes here: far fewer false alarms. These are single runs on a small corpus that I partly built while tuning, so read the counts, not percentages. The last four pairs are the stronger evidence, but even they are not an independent evaluation. Two runs on the same machine gave identical results, which shows determinism, not robustness. What a run costs, on one Windows laptop (I didn't record the hardware): | Run | Time | |---|---| | One real scan, one spec, the model writes its own scenarios | about 8.5 minutes | | demo --quick, checkout pair, end to end | about 5 minutes | | 24 benchmark scenarios, default check | about 20 minutes | | The same 24 scenarios, stricter two-step check | about 35 minutes | | API fees | $0 | Where it breaks My favorite failure. For "a worker process crashes while running a job", the default check let Gemma quote the sentence about retrying a job that raises an error, and call the scenario Answered. The quote was real. The inference wasn't. Retrying after an error says nothing about what happens when the whole process dies. My citation check can't catch that, because the sentence exists. A yes/no relevance check I added didn't catch it either. The stricter two-step check does, along with two other missed gaps, but it also flags about half of the documented behaviors. So I left the default alone instead of crowning whichever mode looked best on my own data. What I learned: when a small model checks its own work, how strict you make the check is a product decision. It's a dial between missing gaps and crying wolf, and no setting of it is free. A real spec I didn't write To see whether that failure was just a quirk of my own test pairs, I ran the default settings once, with nothing tuned, on a design document from another project: ADR-22 from NATS, "JetStream Publish Retries on No Responders" (Apache-2.0; I did not copy it into my repo). The engine reported 12 scenarios and 0 potential gaps. (Its report counts only "Not established" as a potential gap; the one "Partially answered" finding sat in a separate section.) Then I read every finding against the document. The labels are one person's judgment, and they're in the repo. - 3 were correct and properly supported. - 1 was a genuine unanswered question: the ADR says what a publish returns after a timeout or after a last "no responders" error, but not what happens if the last attempt fails with some other error. The engine labeled it Partially answered, not a gap. - 5 were marked Answered, but the quote they cited didn't support the claim. - 3 were scenarios that contradict the spec or aren't about it. One said the client "does not attempt to send the message again", the opposite of the ADR's whole point. Another was about a Go timer failing to start. So one real finding and three correct confirmations, buried among eight that don't hold up. It's the same "real quote, unsupported inference" failure, on a document nobody tuned for, plus the scenario generator inventing hazards. This is one document labeled by one person, not a benchmark. Other caveats - My first run on the first two pairs went badly: Gemma labeled one planted omission "Partially answered" and called the other "Answered". Strictly, it found neither. I then added the relevance check, partial-answer reporting and a second metric ("unresolved", meaning Not established or Partially answered), using only those two pairs. That metric was defined after the first run. - On a plain scan, where the model writes its own scenarios, it invents requirements the spec never states. In my scheduled run, one flagged "gap" was a false alarm. Treat generated scenarios as prompts for review, not findings. - One export scenario in my own corpus is arguably ambiguous and got flagged in both versions. I left it as written. - Markdown and text only. No comparison with closed models. Bugs I hit along the way - Gemma copied my prompt's placeholder. In an early real run, the report's "question to resolve" was literally the example text from my own prompt. The prompt no longer contains placeholder text, and any question that shares no words with its scenario is replaced by a plain fallback. - My relevance check was too literal. It rejected the right answer to the crash scenario in version 2 because the spec says "lease expires", not "crash". I loosened it, and the looser check then accepted the wrong quote in version 1. That is how I ended up with the dial described above. - Windows Task Scheduler caps the command at 261 characters. My first scheduled command was far longer, so the engine now writes a short wrapper script and schedules that. Task Scheduler also refuses to start on battery by default, so I had to change that setting for a 2 AM run to work on a laptop. Why Does Open Innovation Matter? - The docs stay on my machine. Specs for systems that don't exist yet are exactly what I wouldn't paste into a hosted API. The engine refuses any model host that isn't localhost. It also patches Python's socket calls so a non-loopback connection or DNS lookup inside its own process raises an error, and a test checks that. That covers the Python process only, not what Ollama does. - It works offline. I turned Wi-Fi off, used a failing ping google.com to show it was off, and ran the demo. It finished with the same result. The ping only shows the network was down; the completed run is the evidence. - No API fees. - The weights are pinned. I recorded the model digest, so the benchmark can be reproduced instead of drifting under me. The model tag is a command-line flag, though I only benchmarked gemma3:4b. I never compared it with a closed model, so I'm not claiming this beats one on accuracy. My argument is privacy, offline use, cost and control. The Lazy Payoff I do not run it. After one setup command, Windows Task Scheduler does. python dark_matter.py schedule ./docs --at 02:00 On Oct 22 the task fired by itself at 2:00 AM and had written the report and a log line two seconds later. That was fast because the docs hadn't changed, so it reused the saved result. When they do change, a real scan takes its time: about 8.5 minutes for one spec on my laptop. Nobody is waiting, so slow is fine. A sloth would approve. After setup, the number of things I do each day is zero. As for the payoff: I ran it on my own spec corpus and, once, on a real design document. That run surfaced one genuine unanswered question, and also eight findings I had to read and throw out. So I can't honestly tell you how many hours it saves a team. What happens when the gateway charges the card and the response disappears? Version 1 of my spec had no answer. Version 2 did, after I added the idempotency behavior. That is the gap Dark Matter Engine is built to surface. It can't promise to find every missing answer, and it still produces false alarms. But it can read documentation overnight and leave me questions worth reviewing. The sloth moves slowly. At 2 AM, nobody has to remind it to get started. Prize Categories Best Use of Gemma: a local Gemma decides whether each quote establishes the required behavior. Python validates every citation and every output format. I kept those jobs separate on purpose.

Spring Nats 4 is now officially released. The wait is finally over. For more details, please see: github.com/nats-io/spring-boot-4-nats/releases/ta… t.co

Beyond Kafka and Redis, Part 2: An AI Chat Backend on NATS 2.15 In part 1 I walked through the NATS building blocks one at a time: subjects, request-reply, JetStream streams, the Key-Value store, and the Object Store. That post answered "what can NATS replace?" This one builds a real backend with it. The system is the backend for an AI chat product. It has a GPU work queue, token streaming to the browser, conversation history, per-tenant usage metering, run timeouts, and worker ownership of conversations. In a typical stack that's Kafka or SQS for the queue, Redis for counters and locks, a cron service for timeouts, Postgres for history, and a WebSocket gateway for streaming. NATS has shipped four minor releases since part 1: 2.11, 2.12, 2.14, and 2.15 (2.13 was skipped). Several of the new features close gaps that used to push you toward one of those other systems. Each section below takes one problem, shows the feature that handles it, and shows code I ran against nats-server 2.15.0 with nats.js 3.4.0. The full scripts are in nats-backbone-examples. What's new since part 1 | Release | Feature | What it handles | |---|---|---| | 2.11 | Per-message TTL (Nats-TTL) | Expiring one message without a stream-wide MaxAge | | 2.11 | Per-key TTL in KV | Locks and leases that expire on their own | | 2.11 | Priority groups: overflow, pinned_client | Sending work to preferred workers first | | 2.11 | Consumer pause | Stopping delivery during a deploy without deleting the consumer | | 2.12 | Atomic batch publish | Writing several messages all-or-nothing | | 2.12 | Distributed counters | INCR across regions without double counting | | 2.12 | Delayed messages (@at) | Timeouts and reminders without a cron service | | 2.12 | prioritized priority policy | Failover that moves work sooner than overflow | | 2.14 | Recurring schedules (@every, cron) and subject sampling | Periodic jobs and downsampling | | 2.14 | Atomic schedule cancel | Clearing a timeout in the same publish that finishes the work | | 2.14 | Fast-ingest batch publish | High-throughput ingest with server-side flow control | | 2.14 | Consumer reset | Reprocessing from a sequence without recreating the consumer | | 2.14 | Mirroring WorkQueue and Interest streams | Disaster recovery for queues | | 2.15 | Stream backup v2, evacuate, desired-state metalayer | Safer node replacement and backups | | 2.15 | Default limit of 1,000 consumers per stream | Protection from runaway consumer creation | A note on part 1 I've fixed three code errors in part 1. The KV ttl option in nats.js is in milliseconds (part 1 passed nanoseconds, which set it to about 114 years). KV keys can't contain :. Optimistic locking uses kv.update(), not a previousRevision option. If you copied the KV examples, check them again. The subject layout Every piece of state in this backend lives on a subject. The layout is the design, so it comes first: inference.jobs.<tenant> INFERENCE work queue for GPU workers chat.<conv>.run.<run>.chunk CHAT streamed tokens, 1 hour TTL chat.<conv>.run.<run>.done CHAT end of a reply chat.<conv>.run.<run>.deadline CHAT schedule that fires the timeout chat.<conv>.run.<run>.timeout CHAT written by the schedule history.<conv> HISTORY committed conversation turns usage.<region>.<tenant>.tokens USAGE_<REGION> per-region counters usage.all.<tenant>.tokens USAGE_GLOBAL aggregated counters $KV.leases.<conv> KV "leases" which worker owns a conversation A few rules shaped it. Each run gets its own subjects, so a client can replay exactly one reply. The tenant or conversation ID sits in a fixed token position, so permissions and filters can match on it. Things with different lifetimes live in different streams: tokens are short-lived, history is permanent, and counters have their own stream type. 1. A GPU queue with paid overflow You run inference on your own GPUs. When they fall behind, you want to rent burst capacity, but only while there's a real backlog. With Kafka or SQS you'd build this yourself: watch the queue depth, scale a second worker pool up and down, and hope the metrics lag doesn't cost you. Priority groups (2.11) put this rule in the consumer. With the overflow policy, each pull can include a min_pending threshold. The server answers that pull only when the consumer has at least that many messages waiting. await jsm.streams.add({ name: "INFERENCE", subjects: ["inference.jobs.>"], retention: "workqueue", }); await jsm.consumers.add("INFERENCE", { durable_name: "gpu-workers", ack_policy: AckPolicy.Explicit, priority_policy: "overflow", priority_groups: ["gpu"], }); const consumer = await js.consumers.get("INFERENCE", "gpu-workers"); // Your own GPUs: always pull. const onPrem = await consumer.fetch({ max_messages: 10, expires: 2000, group: "gpu" }); // Rented burst capacity: only gets work while 25 or more jobs are waiting. const burst = await consumer.fetch({ max_messages: 10, expires: 2000, group: "gpu", min_pending: 25, }); With 60 jobs queued and both pools pulling 10 at a time: round 0: on-prem=10 burst=10 pending=40 round 1: on-prem=10 burst=8 pending=22 round 2: on-prem=10 burst=0 pending=12 round 3: on-prem=10 burst=0 pending=2 round 4: on-prem=2 burst=0 pending=0 The burst pool stopped mid-round once the backlog dropped under 25. The on-prem pool finished the rest. Two other policies cover related cases. pinned_client sends everything to one client until it stops pulling, which suits a single active processor with a hot standby. prioritized (2.12) lets each pull carry a priority number, and the server serves lower numbers first. The catch: once a consumer has a priority policy, every pull must name the group. A pull without group is rejected with Priority Group missing. 2. Streaming replies that survive a reconnect Token streaming is where most chat backends get fragile. The usual setup sends tokens over a WebSocket or SSE connection from the server that runs the model. If the browser drops for two seconds, or the user reloads, those tokens are gone. Teams patch this with Redis Streams or a buffer table, plus resume logic. In this design the responder publishes each token chunk into the CHAT stream. The browser reads with a consumer, keeps the stream sequence of the last chunk it rendered, and resumes after that sequence when it reconnects. The stream is the buffer. Chunks are only useful for a short time, but CHAT also holds things you may want longer. Per-message TTL (2.11) handles that: each chunk carries its own lifetime, and the rest of the stream keeps its normal limits. await jsm.streams.add({ name: "CHAT", subjects: ["chat.>"], allow_msg_ttl: true, allow_msg_schedules: true, }); // Responder: one message per token, each kept for an hour. for (const token of tokens) { await js.publish(`${base}.chunk`, JSON.stringify({ token }), { ttl: "1h" }); } await js.publish(`${base}.done`, JSON.stringify({ chunks: tokens.length })); The reader resumes from a cursor: async function readRun(base: string, afterSeq = 0) { const c = await js.consumers.get("CHAT", { filter_subjects: [`${base}.chunk`, `${base}.done`], ...(afterSeq ? { deliver_policy: DeliverPolicy.StartSequence, opt_start_seq: afterSeq + 1 } : {}), }); // ...consume, record m.seq after each rendered chunk, stop at ".done" } Reading four tokens, dropping, and resuming from the saved cursor: before disconnect: NATS keeps the reply (cursor 5) after reconnect: until the client catches up The cursor is a stream sequence, not a token index. Other subjects in CHAT take sequence numbers too, which is why the fourth token sits at sequence 5. The catch: a TTL deletes the message whether or not anyone read it. Set it longer than the longest time a client can be away and still expect to resume. Also, allow_msg_ttl can't be turned off once it's on. In a browser, nats.js connects over WebSockets and the same code works. The extra work is around it: a cursor that survives a page reload (IndexedDB is enough), replay that finishes before you render live tokens, and one connection shared across components instead of one per component. I've been building a side project for that part, NATSail. It's a TypeScript runtime with React, RxJS, and Effect adapters, and its repo has runnable chat apps on these patterns. The AI chat example streams Vercel AI SDK and TanStack AI replies over JetStream and has buttons that drop the connection or reload the page mid-reply. You don't need it for anything in this post. 3. Run deadlines without a cron service A model call can stall. Without a deadline, the user watches a spinner forever. The usual fix is a timer in the responder process (lost when it crashes) or a delayed job in a separate queue (another system). Message schedules (2.12, extended in 2.14) let the stream fire the timer. When a run starts, the responder publishes a schedule message with an @at time. If the run isn't finished by then, the server writes a message to the target subject. When the run finishes, the .done publish cancels the schedule in the same operation. // When the run starts await js.publish(`${base}.deadline`, JSON.stringify({ reason: "deadline" }), { schedule: { specification: new Date(Date.now() + 30_000), target: `${base}.timeout`, }, }); // When the run finishes: store ".done" and remove the schedule together await js.publish(`${base}.done`, JSON.stringify({ chunks: n }), { cancelSchedule: { scheduleSubject: `${base}.deadline` }, }); I ran two runs with short deadlines. run-1 finished. run-2 sent three tokens and stopped: timeouts fired: [ 'chat.conv-42.run.run-2.timeout' ] timeout body: {"reason":"deadline"} The timeout lands in the same stream as the chunks. A browser that's already reading the run sees it in order and can show "the model stopped responding" instead of a spinner. A cleanup worker can subscribe to chat.*.run.*.timeout and mark the run failed. The schedule survives a responder crash because it lives in the stream, replicated like any other message. That's the property the in-process timer was missing. Schedules do more than one-shot timers. @every 5m and 6-field cron expressions (with a time zone) arrived in 2.14. A schedule with source set publishes the latest message from another subject on each tick, which gives you downsampling: copy the latest value of a busy sensor subject into an hourly subject, for example. The catch: each schedule needs its own subject, and the target must be a different subject in the same stream. A stream with schedules can't use discard: new or be a mirror. 4. Writing a conversation turn all at once One turn in an agent conversation is several records: the user message, each tool call, each tool result, and the assistant's answer. If the writer crashes partway, history holds half a turn, and the next model call gets a broken context. In Postgres you'd use a transaction. With plain JetStream you'd write one large message per turn, or accept the risk. Atomic batch publish (2.12) stores a group of messages as a unit. Nothing is visible until the commit, and a failed batch stores nothing. The more interesting part is concurrency. Two agents (or two retries of one agent) can both read the conversation at sequence 4 and both try to write the next turn. Put a lastSubjectSequence expectation on the first message of the batch, and only one turn commits. await jsm.streams.add({ name: "HISTORY", subjects: ["history.>"], allow_atomic: true, }); async function appendTurn(conv: string, expectedLastSeq: number, entries: Entry[]) { const subject = `history.${conv}`; const [first, ...rest] = entries; const last = rest.pop()!; const batch = await js.startBatch(subject, JSON.stringify(first), { expect: { lastSubjectSequence: expectedLastSeq }, }); for (const e of rest) batch.add(subject, JSON.stringify(e)); return batch.commit(subject, JSON.stringify(last)); } Writing the first turn, then two competing second turns: turn 1 committed: batch of 4, last seq 4 committed at 6 rejected: batch didn't contain number of published messages messages stored: 6 Four messages from turn 1, two from the winning turn 2, and nothing from the loser. The losing agent re-reads the conversation and decides what to do. The catch: a batch is capped at 1,000 messages by default, and the server drops a batch that goes 10 seconds without a new message. The commit's ack is the only proof the batch landed. Atomic batches also don't work on a stream with persist_mode: async. If what you need is throughput rather than all-or-nothing, look at fast-ingest batch publish (2.14, allow_batched). The server paces the publisher with flow control and acks in groups. You choose whether a gap aborts the batch or is reported and skipped. 5. Token usage per tenant, across regions Usage metering is usually a Redis INCRBY per tenant, then a job that copies totals into the billing database. Run in two regions and you either pick one region's Redis as the source of truth (cross-region latency on every request) or sum two Redis instances yourself. Distributed counters (2.12) turn a stream into a counter. You publish an increment in the Nats-Incr header, and the server stores the running total for that subject. Counter streams can source from other counter streams, and the server tracks what it has already counted from each source, so the global total doesn't double count. for (const region of ["us", "eu"]) { await jsm.streams.add({ name: `USAGE_${region.toUpperCase()}`, subjects: [`usage.${region}.>`], allow_msg_counter: true, allow_direct: true, }); } await jsm.streams.add({ name: "USAGE_GLOBAL", subjects: [], allow_msg_counter: true, allow_direct: true, sources: [ { name: "USAGE_US", subject_transforms: [{ src: "usage.us.>", dest: "usage.all.>" }] }, { name: "USAGE_EU", subject_transforms: [{ src: "usage.eu.>", dest: "usage.all.>" }] }, ], }); async function addTokens(region: string, tenant: string, n: number) { const h = headers(); h.set("Nats-Incr", n >= 0 ? `+${n}` : `${n}`); await js.publish(`usage.${region}.${tenant}.tokens`, "", { headers: h }); } async function read(stream: string, subject: string) { const m = await jsm.direct.getMessage(stream, { last_by_subj: subject }); return m ? m.json<{ val: string }>().val : "0"; } After 1,200 and 800 tokens in the US, and 450 tokens minus a 50-token refund in the EU: us: 2000 eu: 400 global: 2400 In production each regional stream lives in its own cluster, and USAGE_GLOBAL sources across them. Each region writes locally, and the global view converges. The value is a string ({"val":"2400"}) so it can go past JavaScript's 2^53 limit. The examples set the header by hand to show the protocol. The @synadiaorbit/counters package wraps it. The catch: counter streams must use limits retention, can't be turned off after creation, and can't be combined with per-message TTL, schedules, mirrors, or discard: new. Keep counters in their own streams. 6. One worker per conversation Only one agent worker should act on a conversation at a time. That's a lease: take it if it's free, keep renewing while you work, and let it expire if you crash. The standard tool is Redis SET NX PX plus a Lua script for safe renewal. Per-key TTL (2.11) does this in the KV store. kv.create() fails if the key exists, and its third argument sets a TTL on that key only. The bucket's markerTTL makes the server leave a marker when a key expires, so watchers learn about it. const kv = await new Kvm(nc).create("leases", { markerTTL: 60_000 }); async function acquire(conv: string, worker: string) { try { return await kv.create(conv, worker, "3s"); // returns the revision } catch { return 0; // someone else holds it } } // The KV client has no "update with TTL", so publish to the bucket's subject. // The expectation makes renewal fail if someone else took the lease. async function renew(conv: string, worker: string, revision: number) { const ack = await js.publish(`$KV.leases.${conv}`, worker, { ttl: "3s", expect: { lastSubjectSequence: revision }, }); return ack.seq; } The renewal shows something useful about KV: a bucket is a stream named KV_<bucket>, and each key is the subject $KV.<bucket>.<key>. When the KV client doesn't expose an option, the stream publish API usually does. Worker A takes the lease, renews once, and then crashes: worker-a acquired at rev 1 watch: conv-42 PUT worker-a worker-b acquire: refused worker-a renewed, rev 2 watch: conv-42 PUT worker-a worker-a stops renewing (crash)... watch: conv-42 PURGE worker-b acquire: 4 The PURGE entry is the expiry marker. A supervisor watching the bucket can see a lease expire and reassign the conversation right away instead of waiting for someone to poll. The catch: a lease is a liveness hint, not a fence. A worker that pauses for longer than the TTL can wake up and keep working after someone else took over. Check the lease revision before side effects that matter, the same as with any lease. Operating it The newer releases also made day-two operations easier. Here's what I'd use in this backend. Pause a consumer during a deploy (2.11). Messages keep landing in the stream, and delivery resumes on its own at the deadline or when you call resume. await jsm.consumers.pause("EMBED", "indexer", new Date(Date.now() + 15 * 60_000)); await jsm.consumers.resume("EMBED", "indexer"); Reprocess after a bug fix (2.14). Before, you deleted and recreated the consumer at a start sequence. Now you reset it in place, and it keeps its name and config: const r = await jsm.consumers.reset("EMBED", "indexer", 4); console.log(r.num_pending); // 7: sequences 4 through 10 are pending again Back up a stream in a readable format (2.15). Backup v2 stores messages in a format the CLI can parse and edit, rather than the on-disk format. It also works for memory-backed streams, which the old format didn't support. Drain a node before you replace it (2.15). The evacuate endpoints move every stream and consumer off a server, keeping their replica counts. Scaling and moving streams now runs on a desired-state loop, so an interrupted move resumes instead of getting stuck. Upgrade from 2.14.7 or later before going to 2.15. Watch the new consumer limit (2.15). Streams now default to a maximum of 1,000 consumers. That includes ephemeral consumers and KV watchers. If every browser tab opens its own consumer on CHAT, you'll reach it. Raise max_consumers on that stream on purpose, or share one consumer per page instead of one per component. Plan for the 2.16 ack subject change. Starting in 2.16, ack and flow-control subjects include the JetStream domain and an account hash. If your permissions or account exports list $JS.ACK.<stream>.> (or anything narrower), update them first. A catch-all $JS.ACK.> is unaffected. On 2.14 or 2.15 you can test the new format now with feature_flags { js_ack_fc_v2: true }. Where NATS still isn't the answer Part 1 said NATS won't replace Postgres. That's still true, and building this backend made the line clearer. - Queries. HISTORY replays one conversation in order. It can't answer "all conversations where the agent called the weather tool last week." Project the stream into a database for that. - Large files. The Object Store is good for model configs, adapters, and attachments in the megabytes to low gigabytes. For a data lake, use object storage built for it. - Client support for the newest features. nats.js 3.4 covers schedules and atomic batches in the main package, but fast ingest comes from the Synadia Orbit @synadiaorbit/fastingest package. Other languages reach several of these features through Orbit too, and fast-ingest support is still landing in Python, Java, and .NET. Check your language before you design around a feature. The rest of the backend runs on one NATS cluster. Every stream above can be replicated with num_replicas: 3, and each feature uses the same stream mechanics, so the same backups, mirrors, and monitoring cover all of it. Try it brew install nats-server nats-io/nats-tools/nats nats-server -js You need server 2.14 or later for every example (2.15 to match the output above). Then run the examples from this post with Node 22 or later: git clone https://github.com/thedonmon/nats-backbone-examples cd nats-backbone-examples/part-2 npm install npx tsx 01-overflow.ts Each script creates the streams it needs and deletes them on exit. Set NATS_URL if your server isn't on 127.0.0.1:4222. Resources - NATS 2.15 upgrade guide, with the 2.14 and 2.12 guides - ADR-42: Priority groups - ADR-49: Distributed counters - ADR-50: Batch publishing - ADR-51: Message schedules - nats.js - Examples for this post - NATSail If you're running any of these patterns in production, or you found a case where they break down, I'd like to hear about it.

@nats_io Congrats on v2.15! Just opened a small nats.go fix for a Close()/Drain() race where a closed conn could get resurrected into DRAINING_PUBS: nats.go#2163

Opened a PR on nats-io/nats.go: if Close() raced a Drain(), the drain goroutine could flip the closed connection back to DRAINING_PUBS and sit in a 5s flush. The fix rechecks isClosed() under the lock first, with a regression test. nats.go#2163 #golang