/blog

11 min read

Idempotency keys, explained with one retry storm

A payment endpoint, a slow database and a client that retried every 200 ms. What idempotency keys fix and what they don't.

A client sends POST /charges. The server charges the card, then the response times out on the way back. The client does the sensible thing and retries. Now the card is charged twice.

What a key buys you

The client makes up a unique key per logical operation and sends it with every attempt. The server records the key with the result, and a repeat of the same key gets the stored result instead of a second charge.

POST /charges
Idempotency-Key: 7f9c2e1a-4b8d-4e0f-9a51-3c2d1b0e8f77
Content-Type: application/json

{"amount": 4200, "currency": "usd", "source": "tok_visa"}

Where it goes wrong

  • Storing the key after the side effect. A crash in between leaves no record, so the retry charges again.
  • Accepting the same key with a different body. Reject it with 422; it is almost always a client bug.
  • Letting two attempts with the same key run at once. Take a row lock or an advisory lock on the key first.

The shape that held up

Insert the key with a pending status inside the same transaction that does the work, update it to done with the response, and have concurrent attempts wait or return 409. Expire keys after a day, not never: the table grows faster than you expect.

$ subscribe

Want the next one?

Leave your email and new posts from Stack & Heap come to your inbox.

$ subscribe --email

Get new posts by email

Leave your address and new posts from Stack & Heap come to your inbox.

Rather write back? hello@stackandheap.example