sholajegede

How to Stop Concurrent Jobs From Spending the Same Usage Limit Twice Let's say you have a customer on a plan of 100,000 tokens a month. Five of their jobs start at the same moment, and each job expects to use 40,000. Each job checks the account before it starts. Each check sees 100,000 left, so each job runs. The five jobs spend 200,000 tokens but the plan allowed only 100,000. Every check was correct when it ran, and the customer still ended up 100,000 over. This failure is easy to miss, because it needs no bug in the check. It needs only two jobs that start close together and take a long time to finish. AI agents and long-running jobs fit that description well, and they are the work that costs the most. So, how do you make a usage limit hold when many expensive jobs start at once, when some of them cost more than expected, and when some of them never finish? This post answers that question with a small system I built called Earmark. Earmark is a gate that sits in front of expensive work. It holds part of the customer's limit before a job starts, stops the job at what it holds, and charges the real cost when the job ends. The plan limits come from Kinde, a platform for authentication and billing, but the design needs only two things from a provider: a way to read a limit, and a way to report usage. Why a limit check is not a limit A limit is the most a customer may use in one billing cycle. A billing cycle is one month of usage. A check is a read: it looks at what the customer has used so far and compares that number with the limit. A check is a read, and a limit has to be a promise. The difference shows up when work takes time. Usage is counted when a job ends, so a job that is still running has not spent anything yet. A new job that checks the account cannot see it. Four things go wrong, and each one needs a different part of the design. !Five jobs against one limit: check then run, compared with hold, run and settle Every job passes the check. This is the case from the start of this post. Five jobs look at the same remaining capacity at the same moment, and none of them can see the others. A job costs more than expected. The estimate was 10,000 tokens, and the model used 15,000. A check at the start cannot stop this, because the job was allowed in at the start. A job fails halfway. It used 60% of its estimate and then stopped. If you charge nothing, you give the work away. If you charge the estimate, you bill for work that did not happen. A worker disappears. The process is killed in the middle of a job. Whatever the system set aside for that job stays set aside, and the customer is locked out of their own limit. What the provider gives you A gate needs a number to enforce, and Kinde is where that number lives. In Kinde, a plan has features, and a feature can be metered. A metered feature is billed by consumption and has a usage limit that you set for each plan. Kinde's billing concepts page describes features and limits, and its metered usage page describes how to report consumption. The Kinde pages I read about plans and billing concepts do not say whether Kinde blocks work when a limit is reached. Kinde's guide on billing for AI and LLM-based APIs covers cost control for this kind of work, but it does not describe holding capacity before a request runs. So I treat Kinde as the source of the limit and let the gate enforce it. Earmark reads the limit from the customer's entitlements. An entitlement is a feature that a plan grants to a customer. The API reference for the billing entitlements endpoint lists GET /api/v1/billing/entitlements, which takes a customer_id and returns each entitlement with a feature_code and an entitlement_limit_max. The call needs the read:billing_entitlements scope. A null maximum means the plan sets no maximum, and Earmark treats that as no limit. Settled usage goes the other way, from the gate to the meter. The meter is the provider's running total for the cycle. Earmark sends the settled total as an absolute value, which sets the meter to that number, so a repeated report changes nothing. Proving that the meter matches what happened is a separate problem, and I covered it in my earlier post on reconciling metered usage. Reserve: check and hold in one step The first design decision is the most important one. The check and the hold must be one transaction. If a job checks first and holds second, two jobs can still check at the same moment. A hold is a part of the limit that is promised to a running job. Earmark keeps one budget row for each account and cycle. The row has two numbers: used, which is final usage, and held, which is the sum of the holds of running jobs. The gate counts both. export function admit(b: Budget, amount: number): Admission { const remaining = Math.max(0, b.limit - b.used - b.held); if (amount > remaining) { return { admitted: false, remaining, reason: `The job needs ${amount} and ${remaining} is left.` }; } return { admitted: true, remaining: remaining - amount }; } The function is pure, which means the same input always gives the same answer, so a test can check each rule alone. The database call around it reads the budget row, runs admit, and writes the new held value in the same transaction. That transaction is what makes the five-job race safe. Convex, the database I used, runs mutations as serializable transactions. Its documentation describes optimistic concurrency control: a mutation that read data which another mutation changed is run again. When two mutations touch the same budget row, one of them commits first, and the other is run again and sees the new held value. Any database with serializable transactions, or a conditional update on one row, gives you the same guarantee. !The decision a job passes through before it is allowed to start Each job also sends a key. If the same key arrives twice, the gate returns the same job and holds nothing new. This matters because a client that times out will try again, and a second hold for the same job would take capacity that no job uses. Run: the hold is also the cap Once a job holds 40,000, the hold works as a limit on the job. The worker charges each step against the hold before it starts the next step. export function chargeStep(held: number, used: number, tokens: number) { const room = Math.max(0, held - used); const charged = Math.max(0, Math.min(Math.floor(tokens), room)); return { charged, capped: charged < tokens }; } If a step would pass the hold, the gate charges only what is left and marks the job as capped. The job stops. This handles the job that costs more than expected: the customer cannot be charged for more than the hold, and no job can take capacity that another job holds. A capped job loses its unfinished work, and that is a real cost. I chose it on purpose. The alternative is to let a job pass its hold and hope that capacity is free, which brings back the first problem in a smaller form. Every charge also renews the hold. Earmark gives each hold a lease, which is a time limit that a sign of life renews. The lease lasts two minutes. A worker that is alive and charging keeps its hold. A worker that stops does not. Settle: pay for what was used When a job ends, the gate settles it. It charges what the job used and returns the rest of the hold to the budget. export function settle(held: number, used: number) { const charged = Math.min(used, held); return { charged, released: held - charged }; } A job that held 40,000 and used 31,200 is charged 31,200, and 8,800 goes back. A job that failed halfway is charged for the half it did. Partial completion needs no special case, because every ending is the same step with a different used value. The idea is older than usage billing. A card authorization holds an amount on a customer's card before the final price is known. Stripe's documentation on placing a hold describes the same pattern: you capture up to the held amount, and the rest of the hold is released. Settling a job is the same move with tokens. A worker that disappears never calls settle, so something else must. A sweeper runs on a schedule and looks for running jobs whose lease has run out. It settles each one as expired, at the usage it had recorded, and returns the rest of the hold. A worker that comes back late finds its job already settled, and the gate ignores it. Settling a job twice does nothing, so the late call and the sweeper cannot charge the same job twice. !A job from hold to settlement, including the three ways it can end early Fail closed when the limit is unknown A gate is only as good as its number. The limit comes from the provider, and the provider can be slow or down. Earmark keeps the last limit it read. It uses that limit as it is for 60 seconds, so it asks the provider at most once a minute for each account. If the provider cannot be read, Earmark keeps using the last known limit for up to 10 minutes. After that, it refuses new work until the limit can be read again. I think refusing is the right default for expensive work. A wrong refusal costs the customer a short delay. A wrong admission costs real money, and it may spend past a limit that changed while the gate could not see it. For cheap work I would choose differently, because the price of a mistake is small. The cost of the work should set the default. A plan can also change while jobs run. Earmark does not change a running job's hold when the plan changes. The new limit applies to the next admission. Keep the budget row small Every admission and every settlement writes the same budget row, so that row is a serial point. This is the price of an exact limit, and it is the right price. The design goal is to keep everything else off that row. Convex's error documentation names two causes of write conflicts: frequent writes to one document, and mutations that read a whole table or a wide range. Both apply here, so Earmark follows three rules. First, the rate limit is a counter and not a scan. Earmark allows 120 new jobs a minute for each account. Counting recent job rows would make every running job conflict with every new admission. Earmark keeps a small counter on the account's billing row instead. Second, the report state lives in its own table. Whether a new total is waiting for the provider is not part of admission, so it does not belong on the budget row. Third, a conflict is retried only where retrying is safe. Admission is safe to repeat because it is keyed, and settlement is safe because a job settles once. Each retry waits a random time that grows, up to one second, and gives up after eight tries. What happened when I ran it I ran the system in two ways. The first is the Concurrency Lab, a page in the app that starts five jobs of 40,000 at the same moment against a plan of 100,000. It runs the same jobs twice: once with a plain check, and once with a hold. !The Concurrency Lab: the plain check spent 200,000 and the hold spent 80,000 With a plain check, all five jobs started and 200,000 was spent, which is 100,000 over the plan. With a hold, two jobs started, three were refused with the amount that was left, and 80,000 was spent. This is the case from the start of this post. The second run is a load script. It sends 120 jobs of 1,000 tokens, 50 requests at a time, against an account that has room for exactly 100 of them. !Results of the load run: 100 started, 20 refused, 0 failed, 100,000 spent The gate admitted 100 jobs and refused 20. No request failed, and the run spent 100,000 tokens, which is all the plan had left and not one token more. Nothing was still held when the jobs ended. The run took 6.7 seconds, about 18 requests a second. The median time to start a job was 1,316 ms, the 95th percentile was 3,717 ms, and the 99th was 5,099 ms. Limits This system has edges, and I want to name them. - The numbers come from a laptop. I ran the load script from a laptop in Lagos against a Convex cloud development deployment. The latency includes that distance. The run shows that the limit holds under concurrency, not what a production host will do. - One account is one serial point. The load run used one account, and every admission wrote one budget row. Many accounts run in parallel because each has its own row, but I did not measure that. - The worker is simulated. The jobs are real database records that hold, charge and settle. The work itself is a loop that charges tokens. I did not run an actual model, and I did not kill a real process. The crash path is tested with a simulated crash. - I did not run it against a live Kinde account. The limit read and the usage report are tested with fake responses. I took the entitlements endpoint from its API reference, so the scope and field names still need a live check. - A hard cap costs work. A job that reaches its hold stops, and the customer loses what it had not finished. A softer policy is possible, but it brings the overspend back. - The gate is only as good as the estimate. An estimate that is too low means more capped jobs. An estimate that is too high locks capacity, and the gate refuses a job even though room exists. Earmark does not learn estimates. You have to supply a good one. - Plan changes and caching are only touched on. Earmark keeps a limit for a minute and uses the new one at the next admission. Keeping every service in agreement during a plan change is a larger problem than this post covers. Three questions to ask of your own gate Before you copy this design, ask these questions of the gate you have now. Does your check also take a hold? If it only reads usage, it is a check and not a limit. What happens to the hold when a worker stops? If the answer is "nothing", one crash can lock a customer out until someone notices. How far apart are your estimates and your real costs? Pull the last month of finished jobs and compare the two. The size of that gap tells you how much capacity your holds will lock up and how often your jobs will be capped. Run it yourself The code is in the repository. It runs with no billing account, using a built-in stand-in for the plan limit. In the project folder, run these commands: npm install npx convex dev npm run dev Open /app and open the Concurrency Lab. To repeat the load run, start with a fresh account and run node scripts/load.mjs --jobs 120 --estimate 1000 --limit 100000 --concurrency 50 --url http://localhost:3000, using the port that npm run dev prints. The 49 tests run with npm test. FAQ Why hold the estimate instead of charging it? A hold is a promise that can be returned. A charge is final. When the job ends early or costs less, the unused hold goes back to the customer's limit, and the customer pays only for what the job used. Why not lock the whole account while a job runs? A lock makes every other job wait, even when the plan has plenty of room. A hold takes only the part that the job needs, so many jobs can run at once until the limit is full. What if the estimate is wrong? If it is too low, the job is capped at its hold and stops. If it is too high, the extra hold goes back when the job settles, but it blocks other jobs while the job runs. Set the estimate close to what the job costs at the high end of normal. Why a lease and not a timeout on the job? A job can run for a long time and still be healthy. A lease asks a different question: has the worker shown a sign of life recently? Every charge renews it, so long jobs keep their hold and dead workers lose it. Does this need Kinde? No. The gate needs a way to read a limit and a way to report settled usage. Kinde supplies both in the version I built. The built-in stand-in in the repository shows the same behavior with no account. Why send settled usage and not holds to the meter? A hold is a promise and not usage. If the meter showed holds, a job that was refused or that returned most of its hold would still change the customer's invoice. Sources - The Kinde billing model, Kinde docs - About plans, Kinde docs - Commonly used billing concepts and terms, Kinde docs - Add metered usage for a customer via API, Kinde docs - Billing for AI and LLM-based APIs: cost control strategies, Kinde - Kinde Billing Entitlements API reference, apis.io - Optimistic concurrency control, Convex docs - Errors and warnings, Convex docs - Earmark source code

How to Reconcile Metered Usage Billing When Events Arrive Late, Twice, or Never A short link app has a simple job: count the clicks and bill the owner for them. The app counts each click in its own database. The billing provider keeps a second count, called a meter, and the invoice comes from the meter. Both counts start equal, but nothing keeps them equal. A click can reach the database after the total was sent. A retry can send the same usage twice. A report can fail on a timeout and never be sent again. Each case leaves the meter wrong, and the customer sees the error only as an invoice that is too high or too low. The hard question is this: how do you prove that the meter matches what really happened, and fix it when it does not? This article answers that question with a small system I call Notch. Notch is a short link and QR code app that bills per click. It keeps a ledger of every click, sends totals to the meter, and runs a reconciler that compares the two, repairs a difference, and keeps the proof. The billing provider in my test is Kinde, a platform for authentication and billing, but the design needs only two things from a provider: a way to set a total, and a way to read it back. Why the meter and the ledger drift apart A ledger is the list of events your app wrote down, one row for each click. A meter is the provider's running total for a billing cycle, which is one month of usage. The ledger is the record, and the meter is a copy that you keep up to date. Three things break that copy.  Usage arrives late. The app sends a total at 10:00. A click is recorded at 10:01. Until the next report, the meter is one click behind. If the cycle closes before that report, the meter stays wrong, and a closed cycle is the one you can no longer change. Usage arrives twice. A retry runs after a timeout, or a second job writes to the same meter. If the write adds usage, the second write adds it again. The customer pays for clicks that happened once. Usage never arrives. A report fails on a timeout or a rate limit, and nothing sends it again. The meter stays low. Nothing fails loudly, so nobody sees it. All three look the same from the provider's side. The meter holds a number, and the provider cannot tell you whether the number is right. Only the app has the record to check it against, which is why the first rule of the design is that the ledger is the record. What the provider lets you do A design has to match what the provider actually accepts, so I read the Kinde documentation and then tested each behavior on my own account before I relied on it. Kinde's usage endpoint takes a meter_type_code. The value absolute sets the running total to the number you send, and it is the default. The value delta adds the number to the running total. That difference decides the whole design. An absolute write is safe to repeat, because the second write sets the same total. A delta write is not, because a retry adds the usage again, and I confirmed this by writing the same delta twice and reading the total move twice. Stripe takes the opposite route. Each Stripe meter event has an identifier that you can set, and Stripe uses it to prevent a second count of the same event. Kinde's documentation shows no such key, so with Kinde the safe write is the absolute one. The write is only half of the API. To compare the meter with the ledger, the app must read the total back. Kinde's read endpoint returns the current cycle by default, and all_cycles=true returns every record. The read needs the read:meter_usage scope, which is separate from the write scope. Three more facts shaped the design: - A meter_usage_timestamp is stored on the record, but it does not select a billing cycle or change the total. A write that I backdated became its own record, and the live total did not move. - Each write is a record. After about 1,200 writes, the total still read correctly. - Kinde bills recurring metered charges in arrears, and it starts each cycle on the customer's sign-up date, not on the first of the month. The ledger is the record Every click becomes one row in the ledger, and the row has a key that the app makes before it writes anything. The redirect to the destination goes out first. The write happens after the response, and it carries the same key on every try. A second try with the same key adds nothing, so a retry cannot count a click twice. A click that a person or a bot should not have caused is not deleted. The ledger is append only, so a correction is a new row with units: -1, written into the cycle of the click it corrects. The total is the sum of all rows, and the history stays readable.  One more detail keeps the ledger fast. If every click updated one counter row, many clicks at once would fight over that row. Notch splits each counter over 64 rows and adds them on read. Each click picks one shard at random, so two clicks rarely touch the same row. The click and its counter update commit in one transaction, and the same transaction marks the cycle as changed. That mark is what connects the ledger to the meter. A cycle is marked once, and more clicks while it is marked add no more writes. The mark says one thing: the meter may be behind, so send the total again. The reporter sends totals, not clicks Every minute the reporter takes up to 100 marked cycles, deletes the marks, and reads the ledger total for each cycle. It sends each total to the meter with an absolute write, 10 at a time. Sending totals fixes two of the three failures at once. A repeated report is harmless, because it sets the same number. A late click is also harmless, because it marks the cycle again and the next report carries the new total. The third failure, the report that never arrives, needs one more step. If a write fails or gets a 429, the reporter marks the cycle again after the loop ends. The next minute tries again. Kinde documents a limit on concurrent requests and a RateLimit-Reset header on a 429, which gives the seconds until the limit resets. I never saw a 429 in my tests, so that path is covered by tests with a fake meter, not by a real rejection. If a cycle has already closed, the reporter does not write. A closed cycle goes to a review list, because a write to a closed cycle has an effect that I could not confirm. The reconciler proves the copy The reporter keeps the meter close. It cannot prove the meter is right, because it only knows what it sent. The reconciler does the proof, and it runs nightly and on demand. For each cycle, the reconciler reads every ledger row and adds them up. It checks that sum against the running counters, and it rebuilds the counters if they differ. It then compares three numbers: the ledger total, the last total the app sent, and the total the provider holds.  The comparison is a short pure function, so a test can check each case alone: export function decide(e: Evidence): Decision { if (e.provider === null) return { verdict: "unreadable", repair: null, /* ... */ }; if (e.provider === e.ledger) return { verdict: "match", repair: null, /* ... */ }; if (e.closed) return { verdict: "closed_cycle_drift", repair: null, /* ... */ }; return { verdict: e.provider > e.ledger ? "over_billed" : "under_billed", repair: e.ledger, // the ledger is the record /* ... */ }; } The function never trusts the meter over the ledger. When the two differ in an open cycle, the repair is always the ledger total. The cause text depends on the evidence: if the last report failed, the reconciler says so, and if the meter moved without a report from the app, it says that a report was counted twice or something else wrote to the meter. A repair is not finished when the write returns 200. The reconciler reads the meter back and checks that it now equals the ledger. Only then does it record "repaired and confirmed". If the read back differs, the run ends with an error. Every run stores evidence, which is one row with the three numbers, the verdict, the cause, and the read back. A run on the same numbers adds no new row, so running the reconciler twice leaves the record unchanged. That makes the evidence cheap to keep and useful to show: the answer to "was this customer billed correctly in October" is a row, not a memory. The meter's calendar is not yours One mistake would corrupt every number above. A cycle named by calendar month would put a click on 20 October into 2026-10 while the provider counts it in a cycle that started on 20 September. On 1 November the app would then write a small October-only total over the meter's real cycle. Kinde starts cycles on the customer's sign-up date. Notch therefore takes an agreement start when you connect an account, and it names each cycle by the month it starts in. Take an agreement start of 20 October at 15:29 UTC. A click on 19 November belongs to the cycle that began on 20 October, and a click on 21 November belongs to the next one. Two rules follow. The start must fall on the 1st to the 28th of a month, because a start on the 31st has no clear meaning in a short month. The start is set before the first click and refused afterward, because changing it later would move old clicks into different cycles. Kinde's FAQ says that only monthly cycles are supported, so Notch handles monthly cycles only. What happened with live clicks I ran the system against a real Kinde account in two ways. First, I sent 5,000 clicks to a short link at 100 at a time, from a laptop dev server. All 5,000 were recorded. The server handled about 97 redirects a second, with a median of 455 ms, a 95th percentile of 916 ms and a 99th percentile of 1,633 ms. The reconciler then found a match at 6,048 in the ledger, the sent log and the meter, with 0 corrections. That first run proves that the loop works when nothing is wrong. The second run shows what happens when something is. After a later run of 2,100 clicks, the ledger held 8,457 clicks, and the meter held 8,031. The meter was 426 clicks behind. The reconciler marked the cycle as under-billed, wrote 8,457 with an absolute write, read it back, and stored the evidence. The next run matched at 8,457 and added no new row.  Those are the numbers from the runs I watched: - 5,000 of 5,000 clicks were recorded at 100 at a time. - The 2,100 click run reached about 65 redirects a second, with a median of 274 ms and a 99th percentile of 705 ms. - The reconciler repaired a gap of 426 clicks and confirmed the repair by reading the meter back. The cause text that the reconciler stored reads "The ledger changed after the last report. A late click or a correction arrived." That is the late case from the start of this post, found and closed with evidence, and nobody had to notice it. Limits This system has edges, and I want to name them. - The reconciler proves the copy, not the capture. A click that never reached the ledger is invisible to it. The load run counts what was sent against what was recorded, and that is a separate check. - Only the current cycle can be read. Notch reads the current cycle from Kinde. A closed cycle that drifts is flagged for a person, and the reconciler does not repair it. - The cycle rollover is untested. My test agreement starts a new cycle on 21 November. Until that date passes, I have not seen what Kinde does at the exact moment of the change. - I have not seen a real 429. The 1,200 writes that I sent at about 180 a second all returned 200. The retry path rests on Kinde's documentation and on tests. - I do not know what a past-month write does to an invoice. Kinde's documentation does not say. Stripe's documentation says that a late event may be accepted without changing an invoice that is already final. Check your own provider before you rely on a late write. - The load numbers come from a laptop. They show that the design holds under load, not what a production host will do. - One provider was tested. The design needs a set operation and a read, but I only ran it against Kinde. The rule that holds it together The ledger is the fact, and the meter is a copy of it. Once you treat them that way, a late click becomes a new mark, a repeated report becomes the same number, and a failed report becomes a retry. Each case is the same case: the copy is behind, so send the total again and check it. To run it yourself, clone the repository and run npm test for the 58 tests. To measure your own deployment, run node scripts/load.mjs <short code> --clicks 5000 --concurrency 100. Then compare the ledger with the meter in the reconciliation panel, and look for a gap that you did not cause. Do let me know what you find in the comments. FAQ Why send a total instead of each click? A total is safe to repeat and safe to send late. A per-click write needs a unique key at the provider for every click. Where a provider has no such key, a repeated write counts twice. What if the provider only supports delta writes? Keep the ledger and send each change with an idempotency key if the provider accepts one. Stripe does, through the meter event identifier. If the provider has neither an absolute write nor a key, you can only detect a drift and not repair it safely. Why not trust the meter after a repair? A write that returns 200 says that the provider accepted the request. It does not say what total the provider now holds. Reading the value back is the only proof. Does the cycle anchor matter if I bill on the first of the month? No. With no anchor, a cycle is a UTC calendar month, which matches a provider that starts every cycle on the first. The anchor matters when the provider starts cycles on the customer's sign-up date. How often should the reconciler run? Run it nightly, plus on demand after an incident. It reads every ledger row, so run it in pages. Notch does 50 accounts a page, 5 at a time. Sources - Add metered usage for a customer via API, Kinde docs - The Kinde billing model, Kinde docs - Billing FAQ, Kinde docs - API rate limits, Kinde docs - Record usage for billing with the API, Stripe docs - Notch source code

How to Keep Billing State Correct When Webhook Events Repeat, Arrive Late, or Fail When building apps in production, the billing provider you decide to use tells your app what happened to a customer, such as a payment that succeeded, a plan that changed, or a subscription that ended. Your app then turns each message into a change in its own database: a plan column, a monthly allowance, or a flag that lets someone in. Those messages travel over a network that duplicates, delays and drops, so the same payment message can arrive three times, a cancellation can arrive after the upgrade that replaced it, and a server can stop half way through a message. Each case goes on to corrupt a customer's state in a different way, and none of them shows up when you test with one clean message. Which brings me to a question a developer friend of mine asked me last week: how does an app keep a customer's plan and allowance correct when the events that change them can repeat, arrive out of order, or stop in the middle? This post builds the answer as a small pipeline: an endpoint that stores each event before it answers, a queue in Postgres that holds the stored events, and a worker that applies them with effects that are safe to repeat. To show this in practice, I built an example app called "Bottle", it's a reminder app with a free plan and a paid plan using Kinde under the hood to send billing events. Kinde is a platform for authentication and billing, and here it is the billing provider whose plan and payment changes the app must follow. After building the app, I tested the pipeline against the events Kinde sent for my own account and I'll be breaking it all down in this article. What goes wrong when a handler applies each message as it arrives First, we will start with the handler most apps write, which reads the message, changes the database, and answers 200 (OK). That works on a quiet day with a clean network, but three things can potentially break it. !Three ways one billing event goes wrong, and what the pipeline does in each case The same event arrives more than once. If the app applies a payment and the 200 answer never reaches the provider, the provider sends the payment again. The handler adds the monthly allowance a second time. And will do the same a third time if a third copy arrives. An old event arrives after a newer one. A customer upgrades, so the old plan is cancelled. The cancellation happened first, but a retry delays it, and it arrives last. A handler that only trusts the order of arrival will set the customer back to Free. The server stops in the middle. A plan change needs two writes: the plan, then the allowance that goes with it. If the process stops between them, the customer is on Pro with a Free balance. The provider retries, but by then the state is already split. I wrote a handler that applies each message on arrival and ran nine deliveries of three events through it. The nine are the plan change three times, the payment five times, and one old cancellation. The worker stops once during the first plan change, and this is what the account looked like along the way: - After delivery 2, the plan was Pro and the balance was still 10, because the process stopped before the allowance write. - After delivery 7, the plan was Free, because the old cancellation arrived last and won. - After delivery 9, the balance was 2,950. The correct balance is 500. The account ended on Pro only because the plan change happened to arrive a third time. The handler had no rule that made it come out right, and the rest of this post builds those rules. What the provider promises A fix has to match what the provider actually does, so I read Kinde's webhook documentation before I designed anything. These are the parts that matter for correctness: - The body of each request is a JWT (a signed token) with the content type application/jwt. You check its signature against the keys Kinde publishes at /.well-known/jwks on your Kinde domain. - Every request carries a webhook-id header. Kinde describes it as unique to a webhook and an event, and the same on every attempt. The docs tell you to use it as the idempotency key. - A 200 answer means delivered. Any other answer, or no answer, makes Kinde try again. - The retry schedule is roughly: immediately, then after 5 seconds, 30 seconds, 2 minutes, 10 minutes, 1 hour and 4 hours, then every 12 hours until 36 hours have passed. - An endpoint that fails for 72 hours is disabled, and the account owner gets an email. That list explains the three failures. The promise is "we keep trying until we hear 200", so a copy arrives whenever your 200 gets lost, even if your app applied the event correctly. A retry that waits 10 minutes also arrives after every event sent in those 10 minutes, which is how order is lost. The docs do not promise an order for events on one endpoint, and the retry schedule shows why no one could. The pipeline assumes no order and expects copies. The event itself carries what the handler needs. Here is a customer.plan_changed event from my account, with the fields I use and the rest left out: { "type": "customer.plan_changed", "event_id": "event_01a1…", "timestamp": "2026-10-21T02:29:24.481368869+11:00", "event_timestamp": "2026-10-21T02:29:23.403633+11:00", "data": { "user_id": "kp_4940…", "customer_id": "customer_0198…", "customer_type": "user", "plan": { "key": "customer_pro_plan" } } } The plan is an object, and its key is at data.plan.key. The user and customer ids tell the handler which account to change. The two timestamps matter later, in the section on ordering. The pipeline The design follows from two facts. An event can arrive many times, so the app needs one stored copy and a counter. An event can also arrive at any time, so the app must apply it in a way that does not depend on when. Four rules do that work. !The pipeline: the endpoint stores each event, a worker applies it, and every effect is safe to repeat Rule 1: Store the event, then answer The endpoint does three things and nothing else: it checks the signature, writes the event to a table, and answers 200. Applying the event happens after the answer. If the signature is not valid, the endpoint answers 401 and stores nothing. If the database write fails, the endpoint answers with an error, and Kinde tries again later. The 200 therefore means one thing: the event is stored. It does not mean the event is applied. That difference is what lets Bottle survive a crash, because a stored event can run again and a lost event cannot. The signature check is one call to the jose library, which fetches Kinde's keys once and caches them. After the check, the event goes into inbox_events: INSERT INTO inbox_events (source, dedupe_key, type, payload, occurred_at, …) VALUES ($1, $2, $3, $4::jsonb, $5, …) ON CONFLICT (source, dedupe_key) DO UPDATE SET seen_count = inbox_events.seen_count + 1 RETURNING id, (xmax = 0) AS inserted Rule 2: One key for each event The webhook-id header is the dedupe key, and the table has a unique constraint on it. The first copy inserts a row. Every later copy hits the conflict, adds one to seen_count, and does nothing else. The last line of the query tells the caller which case happened. In Postgres, xmax is 0 for a row the statement just created, so (xmax = 0) is true only for a new event. Bottle also writes one row to a receipts table for every delivery. That gives the app a record of all nine copies even though only three events exist. I considered keeping seen ids in memory or in a cache. Both lose the ids when the process restarts, and a restart is exactly when copies arrive. The key has to live in the same database as the event. Rule 3: Apply each event in one transaction, with a lease A worker claims stored events and applies them. Two workers can run at once, for example when the endpoint starts one after its answer and a timer starts another, and they must not take the same event, so the claim uses a clause Postgres provides for this: UPDATE inbox_events SET status = 'processing', attempts = attempts + 1, locked_until = now() + interval '30 seconds' WHERE id IN ( SELECT id FROM inbox_events WHERE (status IN ('received', 'failed') AND next_attempt_at <= now()) OR (status = 'processing' AND locked_until < now()) ORDER BY occurred_at, id FOR UPDATE SKIP LOCKED LIMIT 25) RETURNING * SKIP LOCKED makes a second worker pass over rows the first worker holds. The 30-second lease handles a worker that dies. When the lease ends, the row is claimed again. The worker then runs the handler and marks the event applied inside one transaction. Either both happen or neither does. If the process stops half way, the database rolls back, and the event runs again from the start. That removes the third failure from the list, because the plan write and the allowance write can no longer be separated. When a handler throws, the event goes back to failed and waits. The waits are 5 seconds, 30 seconds, 2 minutes, 10 minutes and 1 hour. After six attempts the event is marked dead, which keeps it visible without running it forever. I chose Postgres over a message queue for one reason. The claim, the effect and the mark must share a transaction, and a separate queue cannot join one. Billing events are also low in volume, so one more service would add cost without adding safety. Rule 4: Make every effect safe to repeat The first three rules reduce how often a handler runs twice, but only this rule makes a second run harmless, and it has to protect two effects. The allowance is a ledger row, and the row has a key made of the account and the month. Writing it again does nothing: INSERT INTO ledger (account_id, period, delta, reason, entry_key) VALUES ($1, $2, $3, $4, 'pro:' || $1 || ':' || $2) ON CONFLICT (entry_key) DO NOTHING Both the payment handler and the plan change handler try to write this same row. Whichever event arrives first grants the Pro allowance, and the other finds the row and skips it. That is why a payment that arrives before its plan change does no harm. The plan change grants the allowance when it comes. The plan has its own protection, and it depends on time rather than arrival. Arrival order is not the order things happened Each stored event has an occurred_at time taken from the event, not from the moment the copy arrived. A plan change applies only if its time is newer than the plan change already stored: UPDATE accounts SET plan = $2, plan_changed_at = $3 WHERE id = $1 AND plan_changed_at < $3 RETURNING id If the update changes no row, the handler records ignored: a newer plan change is already stored and stops. !Arrival order against event time: the old cancellation arrives seventh and is ignored The diagram shows the effect. The cancellation happened three minutes before the upgrade, but it arrives seventh. The database already holds a newer plan change, so the cancellation does nothing and the customer stays on Pro. The event has two timestamps. In the two events I stored, event_timestamp was about one second earlier than timestamp. I read the first as the time of the change and the second as the time Kinde sent the message. Bottle uses event_timestamp when it is present and falls back to timestamp. The choice matters, because the guard compares times, and a time that means "when it was sent" would reorder events that were sent late. Replay and ignored events Two features come from the same idea: a stored event is a fact you can look at again. An ignored event keeps its reason. When a handler chooses to do nothing, it writes why. The log shows "ignored: a newer plan change is already stored", not a silent skip. An event the handler cannot use shows up the same way. A replay puts a stored event back in the queue. It sets the status to received and lets the worker run it again. Replay is safe because of Rule 4. A replayed payment finds its ledger row and writes nothing. A replayed plan change finds a newer change and is ignored. Replay is how you run stored events again after a handler changes or after an outage, and it needs no new code path. In Bottle, each billing event on the account page has a Replay link. The route behind it only accepts events that belong to the signed-in account, because replay should not let one customer requeue another customer's events. !The account page lists each billing event with its result and a Replay link What happened with live events The tests cover the rules, but I also wanted events sent by Kinde itself. In Kinde, I moved my own user from Pro to the Starter plan and back to Pro. Kinde sent the events to Bottle's live endpoint. To test a late delivery, I then replayed the older Starter change after the Pro change had applied. The log on the account page shows the result in the screenshot above. The Pro change reads "plan set to pro, Pro allowance granted". The older Starter change reads "ignored: a newer plan change is already stored". The account shows the Pro plan and 500 bottles for the month. The same nine deliveries from the first section run through the pipeline on Bottle's Engine Room page. The page uses the same intake code and handlers as the live endpoint, on a fresh test account each time. !The Engine Room runs nine deliveries of three events against the pipeline !Steps 2 to 8: a crash, a retry, three dropped payments and a late cancellation The numbers are these: - 9 copies arrived and 3 events were stored. The other 6 copies were counted and dropped. - 1 plan change stopped in the middle. It stayed stored, and the next copy ran it to the end. - The late cancellation was ignored, because a newer plan change was stored. - The account ended on Pro with a balance of 500, and the allowance was granted once. The handler that applies on arrival ended with 2,950. That is 2,450 more bottles than the plan allows, and nothing in the database records how they got there. Limits This design has edges. - Order depends on the provider's clock. The guard compares event times from Kinde. Two plan changes with the same timestamp are treated as a tie, and the second one is ignored. - The live log shows two event types. The plan change and the cancellation came from real Kinde events. The payment handlers and the overdue-invoice handler follow the documented event names and are covered by tests, but I did not watch Kinde send them. - The token's timestamp has no age limit. Kinde's docs mention checking the timestamp for replay protection. Bottle does not reject old tokens. The dedupe table already refuses a second copy of any webhook-id, but it does not expire a token. - A dead event needs a person. After six attempts, an event is marked dead and stays visible. Bottle does not page anyone. A production system needs an alert on that status. - Postgres is the queue. It suits the low volume of billing events. A stream of millions of events a day is a job for a message queue. The rule that holds it together The stored event is the fact, and a delivery is one attempt to tell you about it. Once the app treats it that way, a copy becomes a counter, a late event becomes a comparison of times, and a crash becomes a retry. None of the three needs special code, because each one is the same case: a stored event that has not been applied yet. To run it yourself, clone the repository, run npm test for the 23 tests, and run npm run dev for the app. The Engine Room page runs the nine deliveries in your browser. After testing it out yourself, do drop a comment below telling me your findings, even if you test it with a completely different billing provider. FAQ Why answer 200 before the event is applied? The provider stops retrying once it hears 200. Applying an event can be slow or can fail. If the app answers only after applying, every failure becomes a retry of an event that may be half applied. Storing first makes the answer fast and keeps the event safe. Which key should I dedupe on? Use the key the provider documents. For Kinde, that is the webhook-id header, which stays the same across attempts. Store it with a unique constraint in the same database as the event. Can I use a queue like SQS instead of Postgres? Yes, but the claim, the effect and the "applied" mark can no longer share one transaction. You then need the effects to be safe to repeat on their own, which Rule 4 requires anyway. How do I test retries and reordering? Write down a sequence of deliveries that includes copies, a late event and a crash, and run it against the handler. The Engine Room page does this, and the repository has the same sequence as a test. Sources - About webhooks, Kinde docs - Manage customer activity webhooks, Kinde docs - Bottle source code

Repo, give it a star: sholajegede/ferry