A client sends POST /charges. The server charges the card, then the response times out on the way back. The client does the sensible thing and retries. Now the card is charged twice.
What a key buys you
The client makes up a unique key per logical operation and sends it with every attempt. The server records the key with the result, and a repeat of the same key gets the stored result instead of a second charge.
POST /charges
Idempotency-Key: 7f9c2e1a-4b8d-4e0f-9a51-3c2d1b0e8f77
Content-Type: application/json
{"amount": 4200, "currency": "usd", "source": "tok_visa"}
Where it goes wrong
- Storing the key after the side effect. A crash in between leaves no record, so the retry charges again.
- Accepting the same key with a different body. Reject it with
422; it is almost always a client bug. - Letting two attempts with the same key run at once. Take a row lock or an advisory lock on the key first.
The shape that held up
Insert the key with a pending status inside the same transaction that does the work, update it to done with the response, and have concurrent attempts wait or return 409. Expire keys after a day, not never: the table grows faster than you expect.