ZombieMutex

30 results · ↑ ↓ navigate · Enter open · Esc close

All writing
Engineering#Backend#API Reliability#Idempotency#Distributed Systems#Reproducible Tests
November 6, 20263 min read

Idempotency under 100 competing requests: a reproducible in-process experiment

An observable test for duplicate mutation suppression, tenant isolation, mismatch rejection, and retained errors—plus what a production persistence layer must add.

By

Share

Send this post or copy the direct link.

Suggest a correction

Suppose a client sends an HTTP mutation and times out. It may be tempting to generate a new operation ID and retry, but the first mutation may have committed. The right question is whether the server can safely recognize a retry as the same logical operation.

This behavior was reproduced using a small dependency-free Node.js test harness. The source and test fixtures are at ZombieMutex Idempotency Engine. This is synthetic, in-process code; it is not a database-backed distributed production service.

Setup

The implementation accepts a tenant ID, one operation key, exact request bytes, a retention period, and a callback representing the side effect. It computes a SHA-256 fingerprint of the request bytes. For the same tenant/key/fingerprint, every caller receives the first execution's Promise rather than invoking another callback.

The key detail is ordering: the registry inserts the pending Promise in its map before the asynchronous callback starts. Within one process, there is no second window for a fresh caller to claim the same key.

Run it

text
git clone https://github.com/DeadPineapple/zombiemutex-idempotency-engine.git
cd zombiemutex-idempotency-engine
npm test

The main test schedules 100 calls for the same logical request:

typescript
const results = await Promise.all(
  Array.from({length: 100}, () =>
    registry.execute(
      {tenant: "t1", key: "operation-001", requestBytes: request},
      async () => { effects++; return 201; }
    )
  )
);
assert.equal(effects, 1);

The code is available verbatim in the public repository, including the test's actual request string.

Observed local outcomes

Test caseVerified result
100 concurrently scheduled identical callsSide-effect callback executed once
Same key, different request bytesConflict rejected
Same operation key in two different tenantsSeparate executions allowed
Failure with an uncertain commitThe failure is retained for later calls during the retention window
Key reused after expiryA later execution becomes possible
Invalid key or TTLRejected before dispatch

All six local tests passed in Node.js 24.21. These outcomes describe logic, not network throughput, tail latency, crash consistency or DB isolation guarantees.

Where the simple model fails

The local map is neither shared between service instances nor persistent across restarts. Two workers using different maps will both execute the callback. A process crash after an external payment is accepted but before a durable result is recorded may also leave an ambiguous outcome.

A production design needs a transactionally enforced claim for the tenant-scoped operation key, a stable request fingerprint, an explicit in-progress state, a durable terminal outcome or reconciliation status, a documented retention policy, and an outbox or downstream idempotency support for external effects.

The public repository contains an illustrative PostgreSQL table schema. It is intentionally not presented as a complete claim-and-commit transaction or a tested real database implementation.

The next useful experiment

Stand up PostgreSQL, send two requests in independent worker processes, and inject failure after the business mutation but before response persistence. Check whether the system can recover an unambiguous operation status without duplicating effects. That would provide evidence beyond the current in-memory reference.

You can also explore the shareable retry simulator and the companion HTTP timeout explanation.


Provenance: A new AI-assisted ZombieMutex reference repository, with measured pass/fail assertions from its local test suite. No customer transaction or production exactly-once guarantee is claimed.

Related technical notes with overlapping topics.