Webhook Retry Logic That Doesn't Duplicate Data: A Real Idempotency Bug

A payment provider's webhook fired, our handler took slightly longer than the provider's timeout to respond because of an unrelated database slowdown that day, and the provider — reasonably, per its own documented retry policy — sent the exact same webhook again a few seconds later, assuming the first attempt had failed to reach us. Our handler processed both, because nothing in it distinguished "a genuinely new payment event" from "the same payment event delivered twice." One order got marked paid, and its fulfillment logic ran, twice.

Why webhook retries are a documented, expected behavior, not an edge case

Nearly every mature webhook-sending service — payment providers, messaging platforms, CI systems — documents a retry policy for exactly this scenario: if the receiving server doesn't respond with a success status within some timeout, or responds with an error, the sender retries, sometimes several times over an escalating backoff schedule. This is correct, sensible behavior on the sender's side; from their perspective, they have no way to know whether their original request actually reached you and failed to process, or never reached you at all — retrying is the only reasonable choice available to them without more information. But it means any webhook handler that isn't explicitly built to handle duplicate delivery isn't handling a rare edge case by ignoring this — it's failing a documented, expected, and eventually inevitable part of how webhooks actually work in production.

The handler as it existed when the bug hit

class PaymentWebhookController
{
    public function handle(Request $request)
    {
        $event = $this->verifySignature($request);

        if ($event->type === 'payment.succeeded') {
            $order = Order::findOrFail($event->order_id);
            $order->update(['status' => 'paid']);

            $this->fulfillmentService->triggerFulfillment($order);
            $this->notificationService->sendReceiptEmail($order);
        }

        return response()->json(['received' => true]);
    }
}

Nothing here checks whether this specific event had already been processed. Two deliveries of the same underlying payment event triggered fulfillment twice and sent the customer two receipt emails for what was, from the payment provider's side, a single actual payment — the provider was doing exactly what its documented retry policy said it would do, and our handler had no defense against that documented behavior at all.

The fix: an idempotency key, checked before any side effect runs

Every webhook payload from a well-designed provider includes a unique event ID, distinct from the underlying business object's ID (the order ID, in this case) — specifically meant to be used for exactly this deduplication purpose. Recording which event IDs have already been processed, and checking that record before doing anything else, closes the gap structurally rather than hoping retries just don't happen:

class PaymentWebhookController
{
    public function handle(Request $request)
    {
        $event = $this->verifySignature($request);

        $alreadyProcessed = ProcessedWebhookEvent::where('event_id', $event->id)->exists();

        if ($alreadyProcessed) {
            return response()->json(['received' => true, 'note' => 'duplicate, skipped']);
        }

        DB::transaction(function () use ($event) {
            ProcessedWebhookEvent::create(['event_id' => $event->id]);

            if ($event->type === 'payment.succeeded') {
                $order = Order::findOrFail($event->order_id);
                $order->update(['status' => 'paid']);

                $this->fulfillmentService->triggerFulfillment($order);
                $this->notificationService->sendReceiptEmail($order);
            }
        });

        return response()->json(['received' => true]);
    }
}

The event ID gets recorded inside the same database transaction as the side effects it guards, which matters more than it might look like at first glance — recording it outside the transaction, or after the side effects instead of before, reopens exactly the same race condition the fix is meant to close.

The race condition this alone still doesn't close

A subtler problem: if two retried requests for the same event arrive close enough together, both could pass the exists() check before either one finishes writing its own ProcessedWebhookEvent record — a classic check-then-act race condition, where the check and the action aren't atomic with respect to each other. The fix needs a database-level uniqueness guarantee, not just an application-level check, to be genuinely safe under concurrent delivery rather than just usually-safe:

Schema::create('processed_webhook_events', function (Blueprint $table) {
    $table->id();
    $table->string('event_id')->unique(); // the actual safety net
    $table->timestamps();
});

try {
    ProcessedWebhookEvent::create(['event_id' => $event->id]);
} catch (UniqueConstraintViolationException $e) {
    return response()->json(['received' => true, 'note' => 'duplicate, skipped']);
}

The database's own unique constraint is what actually closes the race condition, not the application-level exists() check — that check is a fast, cheap first pass that avoids doing unnecessary work in the common case, while the unique constraint is the real guarantee that holds even under truly concurrent, near-simultaneous delivery, which an application-level check alone can never fully protect against.

Generating a test that actually proves the fix, not just describes it

public function test_duplicate_webhook_delivery_only_processes_once(): void
{
    $order = Order::factory()->create(['status' => 'pending']);
    $payload = $this->paymentSucceededPayload($order, eventId: 'evt_test_123');

    $this->postJson('/webhooks/payment', $payload)->assertOk();
    $this->postJson('/webhooks/payment', $payload)->assertOk(); // exact same event, sent twice

    $this->assertEquals(1, ProcessedWebhookEvent::where('event_id', 'evt_test_123')->count());
    Notification::assertSentTimes(ReceiptEmail::class, 1); // not 2
}

This test sends the literal same payload twice, on purpose, and asserts the side effect (the receipt email) fired exactly once — the actual behavior the bug violated, made concrete and checkable, rather than trusting the fix by inspection alone. Testing against the real event ID rather than the order ID matters here specifically, since it's what confirms the deduplication is keyed on the right identifier.

The same shape of problem, in a completely different context

This bug is structurally the same category of mistake as caching a response without accounting for what happens if the same key gets written twice concurrently, covered from a different angle in building a simple in-memory cache with expiry — both are cases where an operation needs to behave correctly under repetition or concurrency, not just under the single, clean, sequential case that's easiest to picture and easiest to test by hand. The database-level unique constraint here plays the same structural role a proper atomic check-and-set would play in a concurrent cache implementation: application-level logic alone, however careful, can't fully close a race condition that the underlying storage layer needs to guarantee instead.

Where idempotency keys apply beyond payment webhooks

• Any webhook receiver — messaging platforms, CI/CD triggers, third-party integrations — should assume retries will happen eventually and design for it from the start, not add deduplication reactively after the first duplicate-processing incident.
• Client-initiated idempotency keys (a key your own API accepts from a client, rather than one a webhook sender provides) solve the same class of problem for your own outbound APIs — a client retrying a form submission after a slow or dropped response shouldn't create two records either, and the same unique-constraint pattern applies directly.
• A quick way to generate a test idempotency key for manual testing or a fixture, when you need a random-looking unique value and don't want to reach for a full UUID library for a one-off script, is our random number generator — useful for quickly producing distinct test values while working through exactly this kind of deduplication logic by hand.

The underlying lesson generalizes past webhooks specifically: any operation with a real side effect — charging money, sending an email, triggering fulfillment — that can plausibly be invoked more than once for the same logical event needs an explicit, structurally enforced way to recognize "I've already done this," not just a hope that duplicate invocations won't happen in practice. They will, eventually, exactly the way ours did.

Cleaning up the processed-events table before it grows forever

One detail easy to overlook once the deduplication logic itself is working: the processed_webhook_events table grows by one row for every single webhook received, indefinitely, with nothing ever removing old rows on its own. For a payment provider sending a meaningful volume of events daily, that table can grow into the millions of rows within a year or two if left unattended, which eventually affects both storage cost and the speed of the uniqueness check itself as the index backing it grows. Most webhook providers only actually retry within a bounded window — commonly somewhere in the range of 24 to 72 hours after the original delivery — which means retaining deduplication records indefinitely is protecting against a retry scenario that, per the provider's own documented behavior, can't actually occur past that window.

// A scheduled cleanup job, run daily
ProcessedWebhookEvent::where('created_at', '<', now()->subDays(7))->delete();

A seven-day retention window here is deliberately generous relative to most providers' actual retry windows, trading a small amount of extra storage for comfortable margin against any provider whose retry policy runs a bit longer than typical or isn't fully documented in enough detail to rely on precisely.

Verifying the fix survived a second real retry, not just the test suite

The genuine confirmation that this fix actually worked came a few weeks later when the exact same class of delay-then-retry scenario happened again — a different unrelated slowdown caused our handler to respond slowly enough to trigger a real retry from the same payment provider. This time, the second delivery correctly logged as a skipped duplicate, and monitoring confirmed exactly one fulfillment trigger and exactly one receipt email for that order, matching the passing test written for this exact scenario. Having both the automated test and a real production retry validate the same expected behavior was worth more, as confirmation, than either one alone would have been — the test proves the logic is correct in isolation, and the real retry proves it holds up under the actual, sometimes-messier conditions of production traffic hitting a fix for the first time.