Why one click becomes two charges, and the patterns that stop it

Why one click becomes two

The gap between request and response has a name. Distributed systems call it the two generals problem: two parties separated by an unreliable channel can never be certain the other received a message, because the acknowledgement travels the same channel and can also be lost. A network partition, a dropped packet, or a database that pauses under load all end the same way, in a timeout with no reply. The sender cannot tell a request that never arrived apart from one that succeeded and whose acknowledgement went missing. That is an ambiguous failure, and it is the whole problem in one sentence.

Faced with that uncertainty, systems retry, usually with exponential backoff and jitter so a fleet of clients does not hammer a recovering server in lockstep. Retry logic hides in four places, and each can fire the same operation twice: a client retry after a timeout, a payment gateway's webhook replay, a message queue's redelivery after a missed acknowledgement, and the user who double-clicks because the spinner never moved. Any layer that adds retries and queued jobs multiplies the duplicates the receiver must absorb.

Safe versus idempotent methods

The protocol already made these promises before you wrote a line of handler code. RFC 9110 grades methods along two independent axes. A safe method makes no observable change to server state, while an idempotent method produces the same effect whether it arrives once or fifty times. Safe is the stricter claim: a GET reads and mutates nothing, so it is safe, and safety implies idempotency for free. PUT and DELETE change state, so they are not safe, but repeating either lands on the same result. POST keeps neither promise, which is the one the protocol will not keep for you, and PATCH keeps neither by default. Idempotency is a property of the operation's effect, not a keyword you switch on.

The fixes that quietly make it worse

Reach for the obvious reflexes and each one fails in a way that looks fine in a tutorial and breaks under load. Adding retries without a stable key does not remove duplicates, it manufactures them: a retry storm hammers an endpoint that already succeeded. Widening a deduplication window by timestamp assumes writes arrive far enough apart to tell apart, which two events a millisecond apart do not. Waiting for the problem to matter guarantees it lands at peak volume, when retries and concurrency are both at their worst.

At-least-once, at-most-once, and the exactly-once myth

Every message transport picks a delivery guarantee, and that choice decides how your retries behave. Three semantics are on offer, and only two of them are physically real. The names describe how many times a message can arrive, not how many times it ought to.

No transport delivers exactly-once over a lossy network. Once an acknowledgement can be lost, the sender is forced to choose between never retrying, which loses messages, or retrying, which risks duplicates. SQS FIFO queues and Kafka's transactional producer narrow the duplicate window and manage ordering, but neither closes it: the consumer still has to be idempotent.

Redesign for natural idempotency

The cheapest dedup layer is the one you never write. Before reaching for keys or a ledger, check whether the operation is naturally idempotent: whether applying it twice leaves the system in the same state as applying it once. That property is not luck. It follows from a single design choice, whether you assert an absolute state or perform a relative operation.

An absolute assignment declares the end-state you want. A relative operation mutates whatever it finds, so its result depends on how many times it ran, which is the one thing a retry cannot control. The test is mechanical: if you can describe the outcome as a value the row should hold, the write is an assertion; if you can only describe it as a delta, it is an operation, and every replay changes the answer. Re-expressing the second form as the first often removes the dedup layer completely: rather than add fifty, record the intended balance, or move the account to a named state. This is where modelling the operation as a state transition earns its keep. When each request asserts a target state against a state machine, replays converge instead of compounding, and idempotency keys stop being necessary. Event sourcing reaches the same guarantee from another direction: an event carries its own identity, so applying it twice is caught by that identity rather than by its effect.

The pattern has a boundary. Some operations are genuinely new events with no absolute state to assert. A fresh charge against a card is not a status you can set; each one is a distinct fact. Those cases are why the next three mechanisms exist.

Let the database settle it

Section 3 left a race running. Two requests for the same payment arrive within milliseconds, both run the same "already processed" SELECT, both find nothing because neither has committed yet, and both proceed to INSERT. The check was a suggestion, not a guarantee. Application-level SELECT-then-INSERT cannot close this: the gap between reading and writing is exactly where the race condition lives.

The fix is to stop asking the application to arbitrate and let the database do it. A unique constraint on the idempotency key means the two inserts collide at the storage layer, where writes are serialised. One commits; the second violates the index and fails predictably. Pair that with UPSERT and the failure becomes graceful rather than an exception you have to catch. In PostgreSQL, INSERT ... ON CONFLICT turns the collision into a controlled outcome:

CREATE UNIQUE INDEX idx_payments_idem ON payments (idempotency_key);

INSERT INTO payments (idempotency_key, invoice_id, amount)
VALUES ('inv-8842-charge', 8842, 14900)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;

The second request hits the conflict, changes nothing, and returns the row the first one wrote. MySQL and MariaDB reach the same result with ON DUPLICATE KEY UPDATE. This is the arbiter that makes an idempotent api concurrency-safe: the write itself decides, not a check that ran moments earlier. The same shape underpins duplicate-invoice detection, where the goal is one paid invoice per source document. Making the database the source of truth is the difference between spotting a duplicate invoice before it is paid and guaranteeing it can never be paid twice, even under concurrent load.

Idempotency keys: claim, store, replay

The unique index from the previous section governed keys the database already owned. An idempotency key extends the same move to keys the client owns: the caller generates a UUID, sends it in the Idempotency-Key header, and repeats it verbatim on every retry of that one logical request. Stripe's API popularised the pattern and the header now sits in an IETF draft, so the shape is close to settled. The server's job is to recognise a repeat and refuse to run the write a second time.

The claim is the same concurrency-safe insert as before: write the key to a column carrying a unique index (or SETNX the key in Redis) and let the first request through. A duplicate arriving while the original is still in flight collides on that constraint and receives a 409 conflict rather than a second execution. In Laravel, that logic fits in one middleware sitting in front of the write endpoint.

public function handle(Request $request, Closure $next)
{
    $key = $request->header('Idempotency-Key');
    if (! $key) {
        return $next($request); // unkeyed request, nothing to guard
    }

    $scope = $request->user()->id . ':' . $request->path();
    $fingerprint = hash('sha256', $request->getContent());

    // Claim: unique index on (scope, key) is the arbiter
    $record = IdempotencyKey::firstOrCreate(
        ['scope' => $scope, 'key' => $key],
        ['fingerprint' => $fingerprint, 'response' => null],
    );

    if (! $record->wasRecentlyCreated) {
        abort_if($record->fingerprint !== $fingerprint, 409, 'Key reused with a different payload');
        abort_if($record->response === null, 409, 'Original request still in flight');
        return response()->json(json_decode($record->response, true)); // replay
    }

    $response = $next($request);
    $record->update(['response' => $response->getContent()]);
    return $response;
}

Two details keep the record honest. Payload fingerprinting stores a hash of the body alongside the key and rejects a reused key that carries a different body with a 409, since that signals a client bug rather than a retry. Key scoping per user and per endpoint stops one caller's UUID shadowing another's. Store each record with a TTL of 24 to 72 hours: long enough to outlast any sane retry window, short enough to stop the table growing without bound.

Deduplicating webhooks and queue messages

External providers do not send each event once. Stripe retries a failing webhook for roughly three days; Shopify retries repeatedly over roughly 48 hours. Any endpoint receiving webhooks from an external provider will therefore see the same event arrive more than once, and the fix is the arbiter from earlier: a deduplication table keyed on the provider's stable event ID, written before you process anything, so a replay collides and drops out. Store it the moment the request lands, not after the work completes, or a crash between the two leaves the event unrecorded and open to reprocessing.

CREATE TABLE processed_events (
    event_id    TEXT PRIMARY KEY,   -- provider's stable ID, e.g. evt_1Pm...
    source      TEXT NOT NULL,      -- 'stripe', 'shopify'
    received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Claim the event on receipt, before any processing.
INSERT INTO processed_events (event_id, source)
VALUES ('evt_1PmXqL...', 'stripe')
ON CONFLICT (event_id) DO NOTHING;
-- Zero rows affected means a replay: acknowledge 200 and stop.

Queue systems narrow the window without closing it. Amazon SQS FIFO queues accept a message deduplication ID that suppresses repeats inside their dedup interval, and Kafka's idempotent producer stops a producer retry writing the same record twice. Neither removes the need for an idempotent consumer, which is the exactly-once point restated: the broker thins the duplicates, your handler still has to tolerate them. On the job side, Laravel's ShouldBeUnique keeps a duplicate dispatch off the queue and WithoutOverlapping stops two copies running at once, though both are TTL locks rather than a substitute for the event-ID record.

Which mechanism, when

The four mechanisms map to four different shapes of duplicate problem, which is why the ordering has run cheapest first. Redesign to natural idempotency whenever you can restate the operation as an assertion about an end-state, because that removes the duplicate at the API layer before anything downstream has to guard against it. When you cannot, push the guard down: a unique constraint enforces it at the database layer, idempotency keys enforce it at the boundary of a genuinely new event, and consumer deduplication absorbs redelivery at the queue layer. These stack rather than compete. A real system commonly asserts an end-state where it can, backs it with a business key, issues keys for fresh charges, and still deduplicates inbound webhooks.

Software you trust versus software you babysit

Every layer beneath your code promises at-least-once delivery and nothing stronger. The network will retry, and duplicates will arrive. Correctness under those duplicates is something you build into the design, not a property you can buy. The work stays cheap when you take it in order: reshape the operation to be naturally idempotent first, reach for a unique constraint or UPSERT when the state genuinely changes, and add idempotency keys only where nothing else fits.

Get that right and retries stop being dangerous. A duplicate charge stops being a class of incident and becomes something the system rejects on its own. That is the felt difference. Software you trust keeps running correctly while the network misbehaves. Software you babysit waits for someone to notice the double payment and reconcile the damage by hand.