Idempotency & Preventing Double Payments

Networks fail in ambiguous ways. A client sends “charge $50”, the connection drops, and it can’t tell whether the charge happened. If it retries blindly, the customer may be charged twice.

An operation is idempotent if doing it several times has the same effect as doing it once. Payment providers make non-idempotent operations safe with an idempotency key.

Diagram

The first request with a key does the work and saves the result; retries with the same key get the saved result.

How it works

The client generates a unique key (a UUID) per logical operation and sends it in an Idempotency-Key header.
The server tries to insert the key into a store with a unique constraint, marked “in progress”.
If the insert succeeds, it performs the operation and saves the response against the key.
If the key already exists and is complete, it returns the saved response without doing the work again.
If it exists and is still in progress, it returns 409 Conflict so the client waits and retries.
Keys expire after a retention period, such as 24 hours.

Key concepts

Naturally idempotent: GET, PUT (replace) and DELETE should be idempotent by design. POST usually is not.
Idempotency key: Client-generated ID for one logical operation, reused on every retry of that operation.
At-least-once delivery: Message brokers may deliver twice, so consumers must also be idempotent.
Exponential backoff + jitter: Retries wait longer each time, with randomness, so clients don’t retry in lockstep.

Trade-offs

Benefits

Retries become safe for clients and message consumers
No duplicate charges, orders or emails
Simplifies failure handling everywhere

Costs

Needs a durable key store with a unique constraint
Must decide how long keys live and what counts as “same request”

Use it when

Payments, transfers, order placement, any operation with money or side effects
Message consumers in at-least-once systems

Common mistakes

Generating a new key on each retry, which defeats the purpose
Checking “does the key exist?” and inserting it in two steps (race condition) — use an atomic insert
Reusing a key with a different request body — reject it

Interview questions

Q1Why isn’t a database transaction enough to prevent double charges?

The duplicate is a second, separate request. Each transaction succeeds on its own. You need a key that links the two requests as the same operation.

Q2Is DELETE /orders/42 idempotent?

Yes in effect: after the first call the order is gone, and later calls leave it gone. The status code may differ (204 then 404), but the state is the same.

Q3How do you make a message consumer idempotent?

Record processed message IDs in the same transaction as the business change, and skip messages whose ID is already recorded.