Idempotent Payment Webhooks: Duplicates and Retries

A payment provider tells your system what happened by calling a URL you control. That call is a webhook, and it is the least reliable part of most payment integrations. Deliveries are retried, so the same event can arrive twice. They are not guaranteed to arrive in order. Your server can crash halfway through handling one.

A handler that assumes “one event, one delivery, in order” works in testing and fails in production, usually by sending a customer two receipts or marking a refunded order as paid. This article builds one that stays correct when those assumptions fail.

The code is TypeScript on Node.js 22 with its built-in (still experimental) node:sqlite. We ran this code; the imports, types and the orders and outbox tables are left out here. With several processes writing to one SQLite file, set PRAGMA busy_timeout or writers will fail with SQLITE_BUSY. The ideas apply to any language and to Postgres or MySQL. The event names are illustrative; use your provider’s.

What can go wrong

Most providers deliver webhooks at least once. If your endpoint returns an error, times out, or the response is lost, the provider tries again later. From your side that means:

Duplicates. The same event ID arrives two or more times, sometimes at nearly the same moment.
Reordering. A refund can arrive before the payment that preceded it.
Early arrival. The event reaches you before your code has saved the order it refers to.
Partial failure. You update the database, then crash before sending the receipt.
Forgery. Anyone who knows the URL can post to it.

1. Verify the signature against the raw body

Check the signature first. Providers sign the exact bytes they sent, so verify the raw request body, not JSON that has been parsed and serialised again. Re-serialising changes whitespace and key order, and the signature no longer matches. In Express, mount the raw body parser on the webhook route before any JSON parser, and decode the resulting Buffer to a string only for the check.

Many providers use the same scheme: an HMAC-SHA256 of a timestamp and the body, sent in a header. Stripe’s looks like t=1700000000,v1=<hex>, and its Node library rejects timestamps older than 300 seconds by default, which stops a captured request being replayed later.

export function verify(rawBody: string, header: string | undefined, secret: string, now = Date.now(), toleranceSeconds = 300): boolean {
  const pairs = (header ?? "").split(",").map((p) => p.split("=") as [string, string]);
  const t = pairs.find(([k]) => k === "t")?.[1] ?? "";
  const timestamp = Number(t);
  if (!t || !Number.isFinite(timestamp) || Math.abs(now / 1000 - timestamp) > toleranceSeconds) return false;

  const expected = createHmac("sha256", secret).update(`${t}.${rawBody}`).digest();
  // A header may carry several v1 signatures (for example during secret rotation).
  return pairs
    .filter(([k]) => k === "v1")
    .some(([, v]) => {
      const given = Buffer.from(v ?? "", "hex");
      return given.length === expected.length && timingSafeEqual(given, expected);
    });
}

timingSafeEqual throws if the buffers differ in length, so the length check comes first. It also compares in constant time, so an attacker cannot learn the signature byte by byte from response times. A missing or malformed header returns false rather than throwing. In production, prefer the provider’s own library; this sketch shows what it does.

2. Deduplicate with a unique event ID

Every event has a unique ID. Store it under a primary key and let the database decide whether you have seen it:

CREATE TABLE webhook_events (
  id          TEXT PRIMARY KEY,
  type        TEXT NOT NULL,
  received_at TEXT NOT NULL DEFAULT (datetime('now'))
);

INSERT INTO webhook_events (id, type) VALUES (?, ?) ON CONFLICT (id) DO NOTHING;

If the insert added a row, the event is new. If it changed zero rows, you have handled it already. Do not write this as “SELECT, then INSERT if missing”: two simultaneous deliveries can both see no row and both carry on. A single insert against a unique key has no such gap. In Postgres at the default READ COMMITTED level, a second transaction inserting the same key waits for the first to finish, then does nothing if the first committed.

3. Record the ID and apply the effect in one transaction

The order of operations decides what a crash does to you. Record the ID first and a crash loses the event, because the retry sees a known ID and skips it. Do the work first and a crash repeats the work. Put both in one transaction, and either both happen or neither does:

export function handle(db: DatabaseSync, event: Event): "processed" | "duplicate" {
  db.exec("BEGIN IMMEDIATE");
  try {
    const seen = db
      .prepare("INSERT INTO webhook_events (id, type) VALUES (?, ?) ON CONFLICT (id) DO NOTHING")
      .run(event.id, event.type);
    if (seen.changes === 0) {
      db.exec("COMMIT");
      return "duplicate";
    }

    if (event.type === "payment.succeeded") {
      const r = db
        .prepare(`UPDATE orders SET status = 'paid' WHERE id = ? AND status = 'pending'`)
        .run(event.data.orderId);
      if (r.changes > 0) {
        db.prepare("INSERT INTO outbox (kind, order_id) VALUES ('receipt', ?)").run(event.data.orderId);
      } else {
        requireOrder(db, event.data.orderId);
      }
    } else if (event.type === "payment.refunded") {
      const r = db
        .prepare(`UPDATE orders SET status = 'refunded' WHERE id = ? AND status IN ('pending', 'paid')`)
        .run(event.data.orderId);
      if (r.changes === 0) requireOrder(db, event.data.orderId);
    }
    db.exec("COMMIT");
    return "processed";
  } catch (err) {
    db.exec("ROLLBACK");
    throw err;
  }
}

BEGIN IMMEDIATE takes SQLite’s write lock up front, so the duplicate check and the update cannot interleave with another writer. In Postgres, a normal transaction does the job.

If anything throws, the rollback removes the event ID too. Return a 5xx and the retry is processed as if it were the first attempt. We tested this with a trigger that makes the outbox insert fail: no ID was stored, the order stayed pending, and the event processed normally once the trigger was gone. We also ran four processes delivering the same 200 events at once against one SQLite file, and got 200 event rows, 200 receipts and 200 paid orders. SQLite has one writer at a time, so that shows the logic is sound, not how Postgres behaves under load.

4. Guard state changes, so order does not matter

The ID table stops one event running twice. It does nothing about two different events arriving in the wrong order. For that, make each update conditional on the current state, as the WHERE clauses above do. A payment moves an order from pending to paid. A refund moves it from pending or paid to refunded.

If the refund arrives first, the order becomes refunded. When the late payment event follows, status = 'pending' no longer matches, nothing changes and no receipt is queued. We tested exactly that sequence, and the order ended up refunded with no receipt. A second payment event with a new ID is harmless for the same reason: the order is already paid.

Write down the legal transitions for each object and let the WHERE clause enforce them. If an event carries only a hint that something changed, fetch the current object from the provider’s API and reconcile against that.

5. Do not swallow events that arrive too early

A guard that matches nothing has two possible meanings: the order has already moved on, or it does not exist yet. The first is fine. The second is the webhook beating your own code to the database, and treating it as success is a bug. The event ID is stored, the retry is rejected as a duplicate, and the order stays pending for ever.

That is what requireOrder is for:

function requireOrder(db: DatabaseSync, orderId: string) {
  if (!db.prepare("SELECT 1 FROM orders WHERE id = ?").get(orderId)) throw new Error(`unknown order ${orderId}`);
}

The throw rolls back the transaction, including the event ID, and the endpoint returns a 5xx. The provider retries, by which time the order should exist. If your provider retries quickly, save the order before sending the customer to pay, and alert on repeated failures so a genuinely unknown order does not retry in silence. We tested that sequence too: the first attempt failed, nothing was stored, and the retry marked the order paid.

6. Send side effects through an outbox

A database transaction cannot include an email or a call to another service. If you send the receipt inside the transaction and it then rolls back, the customer has a receipt for an unpaid order. Instead, write a row to an outbox table in the same transaction, as above. A separate worker reads unsent rows, performs the action and marks each row sent.

This gives you at-least-once delivery, not exactly-once. If the worker crashes after sending but before marking the row, it sends again on restart. For the final hop, pass an idempotency key to the receiving service when it supports one, or make the action harmless to repeat.

7. Answer fast, and return the right status

Do the minimum in the request: verify, run the transaction, return 200. Slow work belongs in the outbox worker. Providers generally treat a slow response as a failure and retry, which creates the duplicates you are guarding against.
Return 200 for a duplicate and for event types you do not handle. Otherwise the provider keeps retrying events you will never act on.
Return a 4xx for a bad or missing signature. Do not process the request.
Return a 5xx when your own processing failed. Most providers retry on any non-2xx response, so check your provider’s documentation.

None of this is difficult, but all of it is easy to skip when the first version works. Test the awkward cases deliberately: duplicate, reordered, early, and a crash mid-handler.

If you’re building or repairing a payment integration, API or background job system, see our backend development service.

Idempotent payment webhooks: handling duplicates, retries and out-of-order events