Idempotency when HTTP timeouts lie about server-side effects

A client sends a POST request to create an order. The server commits the order, but the connection stalls before the response arrives. The client times out. Did the operation fail?

The only defensible answer from the client is unknown. An HTTP timeout tells you the client did not receive a satisfactory response before its deadline; it does not roll back the server's transaction. If the client blindly retries, it may create the same order twice.

Use the interactive retry lab to see how this ambiguous result changes with timeouts, attempt counts, and deduplication.

Take a synthetic scenario with these timing assumptions:

• The client timeout is 350 ms.
• The server commits at 600 ms.
• Response delivery adds 100 ms.
• The caller is configured for two additional retries.

The original request is still running when the client deadline expires. Under the lab's simplified assumption that every attempt reaches the server and eventually commits, three submissions can produce three writes. The client may still report a timeout despite all of them completing.

This is not a performance benchmark or a model of every HTTP stack. Real cancellation, connection state, queues, server concurrency, and late response handling will vary.

Instead of asking the server to execute the same logical action again, the client supplies a unique operation identifier. The same identifier is reused across retries of that one action. For a different action, generate a new key.

A server can associate the key with a fingerprint of the intended request, its processing state, and its final response. When the same request is retried, the server can replay the outcome without repeating the side effect. Reusing a key for a different request body should be rejected.

The emerging Idempotency-Key HTTP header is documented by MDN as non-standard/experimental. Do not assume every service supports that exact header or guarantees identical semantics.

POST /orders
Content-Type: application/json
Idempotency-Key: 1f65cbe7-55de-43c7-a1f2-88c35f470432

{"sku":"book-123","quantity":1}

The header by itself provides zero protection if the backend ignores it or stores keys without an atomic operation claim.

For a system with transactional storage, record the action under a uniqueness constraint scoped appropriately to the tenant/account and operation. The store must arbitrate concurrent arrivals.

CREATE TABLE operation_requests (
  tenant_id text NOT NULL,
  operation_key text NOT NULL,
  request_fingerprint text NOT NULL,
  status text NOT NULL,
  response_json jsonb,
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, operation_key)
);

The table is a schema illustration, not a complete transaction. The implementation still must define how the operation and result are recorded atomically, or reconciled safely after a crash. It needs retention limits, request fingerprinting, replay/ownership checks, concurrency control, and error recovery.

Important cases:

1. First request: Claim the operation key, perform the mutation once, and store its terminal result.
2. Duplicate while processing: Do not execute twice. Return an explicit in-progress response or wait under a bounded policy.
3. Duplicate after success: Return the recorded result, subject to the API's published semantics.
4. Same key with different parameters: Reject the mismatch; never treat it as a new valid request.
5. Expired key: Apply a documented retention and replay policy; avoid silent reuse ambiguity.

For high-value operations, the database transaction and externally visible side effects (such as messaging or payments) may also need an outbox or provider-specific idempotency support. A local table alone cannot roll back a call already accepted by an external system.

Retries have side effects on availability even when logically safe. Limit attempts and use backoff with jitter, timeouts at each dependency, and a total deadline. A service in distress can be made worse by synchronized bursts of retries.

Do not retry every error automatically. A deterministic validation error, authorization failure, or conflicting idempotency key should not get treated like a transient 503.

The API retry simulator models one type of failure: a server commits despite the client's timeout. Toggle the idempotency-key checkbox and observe the number of server writes. Even when the server records only one write, the client may still have an ambiguous result until it queries a safe operation-status endpoint.

• Generate a unique key for every new logical mutation.
• Reuse that key across all retries of the same operation.
• Scope the key and request fingerprint to the authenticated tenant.
• Make deduplication atomic across workers, not just in-memory.
• Persist an unambiguous terminal outcome or reconciliation state.
• Test concurrent duplicates and crashes between commit and response.
• Do not claim exactly-once delivery simply because a header exists.

Editorial note: Reference design prepared with AI assistance; examples use synthetic times and simplified transactions. They do not describe measured production incidents.