The webhook that credits twice

A payment processor delivers every result at least once. The interesting engineering is in what happens when your side fails halfway through.

4 min readGoldVault · Webhook stage

Every card processor I have integrated makes the same promise about webhooks: at least once. Not exactly once. If they do not hear a success from you within some window, they send it again. That is the correct promise for them to make, because the alternative is losing a payment result in a network blip, and a lost result is worse than a duplicated one. But it means the duplicate is your problem, and on a payments platform a duplicate means a customer credited twice.

The usual answer is a deduplication key: take some identifier from the delivery, store it, refuse to process anything you have already seen. Most implementations stop there, and most of them have a hole in the middle.

The hole

Picture the sequence. The delivery arrives. You claim the dedup slot so a retry will be refused. You start crediting the customer. The credit fails: a lock timeout, a bad row, a deploy landing at the wrong second. You return an error. The processor retries, as promised. Your handler sees a claimed slot and refuses it. The customer is now never credited, and nothing in your system knows.

That is not a theoretical shape. It is what falls out of writing the dedup check first and the credit second, which is how everyone writes it, because that is the order you think about it in. The dedup key protects against the double credit and quietly creates the lost one.

What the platform does instead

The operation id is deterministic. It is derived from the delivery itself, so a retry of the same delivery produces the same id and collides with its own first attempt rather than being treated as new. A slot is claimed on arrival, as usual. Then the part that matters, on the newer processor's handler: if the credit fails, the slot is released and the response is a 503, not a 200 with an error inside. The older handler releases its slot on the chargeback path and answers 500, and on a failed credit it still logs and answers 200, which is on the list. A 503 is the one status that says, plainly, try again later. The processor does exactly that, the slot is free, and the retry gets to be the attempt that succeeds.

So a retry is refused only when the first attempt actually finished. Where the credit function fails, the claim is undone. Two early returns after the claim, for a delivery with no customer identifier or no matching row, still answer 200 and leave the slot held: the same silent loss through a smaller doorway, and the next fix. That is the whole idea, and it fits in a sentence, but it is the sentence most implementations do not contain.

The things around it

The body of a webhook is unverified until you have looked up the right tenant's signing secret, and the only link back to a customer is a session id sitting inside that unverified body. So the handler has to find the tenant first, from the one field in the unverified body it can be looked up by, verify the signature with that tenant's secret, and only then trust anything else in the payload. One processor's documented signature scheme turned out to be wrong; the real one was reverse-engineered from a captured sandbox delivery, and the code says so, because a future reader will otherwise trust the docs and break it.

An unknown tenant and a bad signature return the identical response. If they differed, someone could probe the endpoint and learn which tenant identifiers exist. That is a small thing to get right and an annoying thing to have gotten wrong.

The costs are real. The deterministic id is a hand-rolled UUIDv5 rather than a dependency. The first version of the signature lookup was a loose scan for any header with signature in its name, which also matched the hosting platform's own proxy signature header; it was caught against a captured delivery and is now an exact-name lookup, with the reason written beside it. The signature lookup says why in its comment. The UUIDv5 says only that it takes no dependency, and should say more, so that a helpful person does not clean it up.

Why it belongs on this page

None of this is exotic. It is a handful of decisions, each obvious once stated, that only get made when the same person owns the endpoint, the ledger it writes to, and the customer support inbox that hears about it when it goes wrong. That is the case for one pair of hands, made with a webhook.

Written by Dylan Pulver, who built the platform this decision lives in. Software engineer and entrepreneur, Toronto.

More notes
  • A ledger you can edit from a route handler is not a ledgerWhy every movement of value on the payments platform goes through a Postgres function, and what it costs to keep it that way.
  • Four rows that were not theirsHardening multi-tenant row-level security in four reversible phases, verified by impersonating a member and counting what they could see.
  • The bug inside the fixA payout race, the fix for it, the bug inside that fix, and why it is the best argument I have for one person owning the whole path.
  • The sale that arrives with no referrerA creator shares a link on Instagram, the buyer taps it, installs the app and purchases. Nothing in that chain carries the creator's name across. Here is what does.
  • The minimum that belongs to someone elseHeld commissions are released to creators once a brand's payout minimum is met. Group the money by creator, the obvious way, and one brand's minimum ends up holding another brand's money.
  • The commission that must not mint twiceA buyer pays and the platform owes a creator a commission. Between those two facts sit a colluding pair, a call that arrives twice, and a cart with three items on one payment.
  • The rule that has to be written twiceFirestore security rules do not cascade to subcollections. Forget that in one place and a single query returns every private message on the platform.
  • Thirty days in the ledgerSplitting a payment at charge time is simpler and wrong: a refund after the creator is paid is a clawback nobody enjoys. Holding the money creates a different set of problems, and each piece of machinery around the hold answers one of them.
  • Arbitrary but consistentA gym's assessment answers become rules that swap an exercise for a member before a session. Two rules can disagree about the same movement. The code says who wins, and the comment admits how.