Idempotency keys: the cheapest reliability win

Every API that takes money, sends emails, or creates side effects should support idempotency keys. It’s one of the cheapest reliability features you can add — and one of the most under-implemented.

The problem

Imagine you’re calling our API to create an invoice. You send the POST, the network hiccups, you don’t get a response. Did the invoice get created? Maybe. Do you retry? If you do, you might create two invoices. If you don’t, you might create zero.

Stripe popularized the solution: clients send an Idempotency-Key header with each mutating request. The server stores the first response keyed by that header. Any subsequent request with the same key returns the cached response instead of executing again.

curl -X POST https://api.tenavora.com/v1/invoices \
  -H "Idempotency-Key: 01HXPC..." \
  -H "Content-Type: application/json" \
  -d '{"amount": 50000}'

Retry the same call → same response. Network hiccup → safe to retry. No duplicate invoices, no manual reconciliation.

How we implemented it

The middleware lives in Tenavora.Modules.AsyncReliability.Idempotency. The flow:

1. Read Idempotency-Key header. If absent, pass through unchanged.
2. Hash the request body + path + method. This guards against a client reusing the same key for a different request — we want to fail loudly, not silently return the wrong response.
3. Look up (client_id, key) in the idempotency_records table.
   • Found + completed: return cached status + body.
   • Found + in-progress: return 409 with Retry-After. (Race between concurrent retries.)
   • Not found: insert a row in in-progress state, run the handler, update with the response.

The table:

CREATE TABLE idempotency_records (
    client_id        uuid NOT NULL,
    key              text NOT NULL,
    request_hash     bytea NOT NULL,
    status           text NOT NULL CHECK (status IN ('in-progress','completed')),
    response_status  int,
    response_body    bytea,
    created_at       timestamptz NOT NULL DEFAULT now(),
    completed_at     timestamptz,
    PRIMARY KEY (client_id, key)
);

CREATE INDEX idx_idempotency_expiry
  ON idempotency_records (created_at);

A background job sweeps records older than 24 hours. Clients can retry within that window; after that we expect them to have given up.

Edge cases we hit

• Long-running handlers: a handler that takes 30 seconds will block retries with 409 the entire time. We added a Retry-After header proportional to elapsed time so clients back off intelligently.

• Streaming responses: large response bodies don’t fit in the response_body column comfortably. We cap at 1 MB and return 422 if exceeded (caller shouldn’t retry that anyway).

• Cross-tenant key reuse: the key is scoped per client_id, so two tenants can use the same key string without colliding. Important — keys are often UUIDs from the client side, but treating them as global would be a privacy hole.

When to use it

Every mutating endpoint, ideally. We mark it required (via OpenAPI) on endpoints that:

• Charge money or change financial state
• Send emails, SMS, or webhooks
• Trigger workflows that side-effect external systems

For pure CRUD where re-creating the same record is harmless, optional. For GETs, never — they’re idempotent by definition.

50 lines of middleware, one table, one background job. Pays for itself the first time a partner integration double-fires a webhook.