The bug inside the fix

A payout race, the fix for it, the bug inside that fix, and why it is the best argument I have for one person owning the whole path.

3 min readGoldVault · Payout stage

A customer cashes out. The platform asks the processor to send the money. Two requests for the same payout arrive together, as they do when someone double-taps or a client retries, and nothing stops both from reaching the processor. That is a double payment, in real money, and it is the kind of bug that decides whether a payments platform gets to exist.

The fix

The redemption row is flipped to an initiated state atomically, before the processor is called. An update with a condition: set the state to initiated where the state is still pending, or failed from an earlier attempt, and return the row. Two requests race; the database serialises them; exactly one update matches and returns a row; the other returns nothing and is told the payout is already in progress. Only the winner calls the processor, and every failure branch after that point compensates by flipping the row back. Textbook, and correct.

The bug inside it

It shipped, and every payout started failing as a conflict. Every single one, including the first request with no race at all. The update was matching, the state was flipping, the processor was never being called, and the customer was being told someone else got there first.

The cause was in the HTTP data layer between the client library and the database, a layer below the code that had been written. When you ask that library to update rows and return them, it re-applies your filter to the rows that come back. The filter said state equals pending. The returned row's state was now initiated. So the library, helpfully, filtered the winning row out of its own result, the code saw an empty result, and an empty result meant somebody else had won.

Nothing in the documentation says this. It was found by running payouts end to end, the commit says so, and explaining it meant reading below the code that had been written until the behaviour was visible, then restructuring the query so the returned row was fetched in a way the filter could not touch. The fix to the fix was a few lines. The fix landed twenty-six hours after the previous commit to that file, and two weeks after the race fix it was inside.

Why this is the argument

Consider how this goes on a team of specialists. A backend engineer writes the atomic flip and is right. A frontend engineer sees every payout return a conflict and files a bug against the backend. The backend engineer reproduces it, confirms the update works in SQL, and sends it back. Somebody eventually suspects the client library, but nobody owns the client library, and nobody's job description includes reading its source. The bug lives for a sprint, then two, while payouts are manual.

When one person owns the checkout, the route handler, the query, the database function, the data layer's behaviour and the row the customer sees afterwards, there is nowhere for the bug to hide between people. There is nobody to hand it to. You read the source because it is the next thing on the path, and the path is yours.

That is what the line at the top of this site means. Not that one person types faster than five. That there is a kind of bug, and a kind of decision, that only gets found by somebody who is standing on every layer at once. This was one of them.

Written by Dylan Pulver, who built the platform this decision lives in. Software engineer and entrepreneur, Toronto.

More notes
  • The webhook that credits twiceA payment processor delivers every result at least once. The interesting engineering is in what happens when your side fails halfway through.
  • A ledger you can edit from a route handler is not a ledgerWhy every movement of value on the payments platform goes through a Postgres function, and what it costs to keep it that way.
  • Four rows that were not theirsHardening multi-tenant row-level security in four reversible phases, verified by impersonating a member and counting what they could see.
  • The sale that arrives with no referrerA creator shares a link on Instagram, the buyer taps it, installs the app and purchases. Nothing in that chain carries the creator's name across. Here is what does.
  • The minimum that belongs to someone elseHeld commissions are released to creators once a brand's payout minimum is met. Group the money by creator, the obvious way, and one brand's minimum ends up holding another brand's money.
  • The commission that must not mint twiceA buyer pays and the platform owes a creator a commission. Between those two facts sit a colluding pair, a call that arrives twice, and a cart with three items on one payment.
  • The rule that has to be written twiceFirestore security rules do not cascade to subcollections. Forget that in one place and a single query returns every private message on the platform.
  • Thirty days in the ledgerSplitting a payment at charge time is simpler and wrong: a refund after the creator is paid is a clawback nobody enjoys. Holding the money creates a different set of problems, and each piece of machinery around the hold answers one of them.
  • Arbitrary but consistentA gym's assessment answers become rules that swap an exercise for a member before a session. Two rules can disagree about the same movement. The code says who wins, and the comment admits how.