Exactly-Once Has Boundaries

Last week, I argued that a retry budget is the only retry control that gets stricter as conditions get worse. It bounds what retries cost you in availability.

It says nothing about what they cost you in correctness.

This week: idempotency—why delivery, execution, and business effects are three different contracts; why deduplication has a scope and a lifetime; and why the idempotency store is one of the most under-planned dependencies in regional failover designs.

A payouts platform. This is a composite scenario, but every failure mode in it is real, and I have walked through each one with teams.

Sellers get paid on a schedule. A request comes off a queue. An orchestration service calls a payout API. The payout API calls a bank rail and records the result in a ledger.

One Friday evening, the payout API started returning 504s. It was not necessarily failing. It was slow—slow enough that the caller’s timeout fired while the work continued on the other side of that timeout.

The orchestration service retried.

Some sellers were paid twice.

The team had done the work—or thought it had. It had idempotency keys and a table to store them. But the key was minted inside the retry loop: a fresh UUID for every attempt, passed dutifully down the stack. Every retry reached the payout API wearing a different name.

They had built a very expensive UUID generator and called it idempotency.

They fixed that in an afternoon. Three weeks later, a smaller set of sellers was paid twice again. This time, the requests came from a dead-letter queue redriven the next morning, roughly fourteen hours after the original failures.

The keys were correct. The retention window was not.

Two incidents, one mistake underneath them: treating idempotency as a property of a code path when it is a contract among an intent, a key, retained state, scope, and the side effect itself.

Get any one of those wrong and the others cannot save you.

“Exactly once” is not one thing

Three ideas are often compressed into one phrase:

• Delivery semantics: How many times can a transport deliver a message?

• Execution semantics: How many times can a task or workflow step run?

• Effect semantics: How many times can the business outcome happen?

These are not interchangeable.

An at-least-once transport can still produce one business effect if the consumer and every side-effecting dependency share a sound idempotency contract. Conversely, a workflow described as exactly-once does not automatically make a payment provider, email gateway, or bank rail execute an external effect only once.

AWS documentation is useful here because it states the boundaries explicitly.

Amazon SQS standard queues use at-least-once delivery: a message can be delivered more than once and may arrive out of order. A visibility timeout reduces concurrent processing, but it is not a deduplication guarantee. The default is 30 seconds; the maximum is 12 hours from receipt; and SQS still does not guarantee that a standard-queue message will be delivered only once during that interval.

Amazon Kinesis Data Streams documents duplicate paths on both sides. If a producer times out after PutRecord commits but before the acknowledgement arrives, a retry can create a second record with the same data and a different sequence number. A consumer restart can also resume from the last checkpoint and reprocess records.

For standard AWS Lambda functions invoked asynchronously, Lambda retries function errors twice by default. The service documentation also warns that an event can occasionally be delivered more than once even when no function error occurs. Retry behavior differs for synchronous calls, event source mappings, throttling, and durable functions, so “Lambda retries twice” is not a universal rule.

AWS Step Functions makes the same distinction. Standard Workflows follow an exactly-once workflow-execution model unless Retry behavior is configured. Asynchronous Express Workflows use at-least-once execution, while Synchronous Express Workflows use at-most-once execution. Those are orchestration contracts. A retried task can still call an external dependency more than once, so the downstream effect needs its own safety mechanism.

At-least-once is not a weakness. It is an honest contract telling you where application responsibility begins.

Every practical deduplication mechanism has a scope and a lifetime

Consider the windows built into common mechanisms:

• Amazon SQS FIFO: repeated SendMessage calls with the same deduplication ID are suppressed within a five-minute deduplication interval.

• Amazon SNS FIFO: a message published with the same deduplication ID within the five-minute interval is accepted but not delivered again.

• Amazon DynamoDB TransactWriteItems: a ClientRequestToken is valid for ten minutes after the first request completes. After that, the same token is treated as a new request.

• Powertools for AWS Lambda (Python): idempotency records expire after 3,600 seconds by default. The value is configurable; one hour is a default, not a service limit.

These mechanisms are doing exactly what their contracts say. The architectural mistake is assuming that a producer retry window is also a business replay window.

A real replay horizon is rarely measured only in seconds. It includes:

• client and SDK retries;

• queue visibility timeouts and redrive delays;

• backlog recovery after an outage;

• deployment rollback and consumer restart;

• manual DLQ redrive;

• partner or customer resubmission;

• batch reprocessing days later; and

• regional failover and catch-up.

The question is not merely, “Is idempotency enabled?”

The question is: How long after the original intent can another attempt still arrive, and will every side-effecting component still recognize it?

If a legitimate replay can arrive after your record expires, the duplicate will look new.

For high-consequence operations, the right answer may be to retain a durable business identifier in the ledger—not to rely on a short-lived cache entry. Storage cleanup is an operational choice. It is also a correctness decision.

The key must name the intent, not the attempt

An idempotency key identifies a business decision made once: “pay seller 847 for settlement 2026-W36.” It does not identify a particular HTTP request, Lambda invocation, or queue delivery.

Every retry of that decision must carry the same key, unchanged, all the way to the component that performs the effect.

Good keys usually originate at the boundary where the intent becomes stable. A random UUID can work, but only if it is generated once and persisted before retries begin. A deterministic business key can be even stronger when the domain already has a natural uniqueness rule, such as (seller_id, settlement_id), enforced by a unique constraint.

The receiver should bind the key to a canonical request fingerprint. Reusing the same key with a different amount, currency, seller, or destination account must fail loudly rather than return the result of an unrelated request.

The latest IETF HTTPAPI work on the Idempotency-Key header captures these ideas: a key must be unique and must not be reused with a different payload. It also recommends conflict-style handling for a duplicate that arrives while the original is still running and an unprocessable-entity response when the same key is reused for different content.

One precision point matters: as of this writing, the latest document is an expired Internet-Draft under IETF working-group review, not an RFC. Treat it as useful design guidance and an emerging convention, not as a finalized HTTP standard. Also note that its suggested 409 Conflict and 422 Unprocessable Content responses are recommendations, not universal protocol requirements.

An atomic claim is necessary—but it is not the payment

This is the most important correction to many idempotency examples.

A conditional database write can atomically claim an intent. It cannot make that write atomic with a call to an external bank or payment provider.

The safe shape looks like this:

1. Read the idempotency key that names the intent.
2. Compute a canonical fingerprint of the request.
3. Atomically create {key, fingerprint, state: IN_PROGRESS, expires_at}
   only if no record exists for the key:
     exists, same fingerprint, COMPLETED    -> return the stored result
     exists, same fingerprint, IN_PROGRESS  -> conflict, do not proceed
     exists, different fingerprint          -> reject the request
4. Call the provider, passing the same key.
5. Store the result and set state: COMPLETED.

In DynamoDB, step 3 can use a conditional PutItem, for example with attribute_not_exists(pk). That prevents two workers in the same consistency scope from both acquiring a new key.

But there is still a hard failure boundary:

3. Claim the key                 -> record: IN_PROGRESS
4. Call the provider             -> the payment happens
   Crash, or the response is lost, before step 5
5. Never runs                    -> record stays IN_PROGRESS

Now the local record says IN_PROGRESS, while the external world may already contain the effect. Blindly expiring the lock and trying again can duplicate the payment. Permanently keeping it IN_PROGRESS can suppress a payment that never happened.

A conditional write alone cannot resolve this ambiguity.

You need at least one of the following:

• the external provider accepts the same idempotency key and deduplicates the actual effect;

• the provider exposes a lookup by your stable business reference, allowing reconciliation before retry;

• the effect and idempotency record live in the same transactional database and can commit together;

• an outbox, saga, or durable workflow coordinates transitions, with idempotent participants and explicit reconciliation or compensation; or

• where none of those exists, you choose and document the trade-off between possible loss and possible duplication, then route ambiguity for human or automated reconciliation.

Exactly-once business effect is an end-to-end property. No local table can unilaterally grant it to an external system.

One more implementation detail: do not rely on physical TTL deletion to define validity. Store an expiry timestamp and evaluate it in application logic. For example, DynamoDB TTL deletion is asynchronous, while Powertools checks the timestamp in the idempotency record itself.

What idempotency does not buy you

Four boundaries are worth saying out loud.

1. Send-side deduplication is not receive-side deduplication

The SQS FIFO five-minute window protects producer retries of SendMessage. It does not prevent a consumer from processing a message, crashing before deletion, and processing it again after the visibility timeout.

Those are two different duplicate paths. They need two different controls.

2. Content-based deduplication only covers the content that is hashed

SQS FIFO generates its content-based deduplication ID from a SHA-256 hash of the message body, excluding message attributes. SNS FIFO likewise excludes message attributes from its content-based hash.

If two distinct business events have identical bodies but differ only in attributes, content-based deduplication can suppress the second event. The feature is working as designed; the identity model is wrong.

3. Durable orchestration does not make arbitrary side effects idempotent

AWS Lambda durable functions can deduplicate execution starts when you supply an execution name. Completed steps are replayed from checkpoints. But steps use at-least-once execution by default, and an interrupted step can run again before its completion is checkpointed.

That means payment, messaging, and other side-effecting code still needs an idempotency key, an appropriate at-most-once strategy, or reconciliation.

4. A token protects one request shape within one documented scope

Amazon EC2 RunInstances is a useful example. Its client-token behavior can be regional or zonal depending on the request. The same token can create resources in a different Region, and under zonal idempotency it can create resources in another Availability Zone. Reusing a token with different parameters in the same relevant scope can produce IdempotentParameterMismatch.

The lesson is not that all AWS idempotency behaves this way. The lesson is to read the scope of each API contract rather than infer a global guarantee from the word “idempotent.”

The regional dependency nobody drew

Your application’s idempotency store is stateful, and it sits in the correctness path of every protected mutation. Yet it often appears on no failover diagram.

Imagine a regional evacuation:

1. The original Region accepts a payout and records its key.

2. The response is lost.

3. Traffic fails over.

4. The request is replayed in the second Region.

5. The second Region cannot see the first Region’s idempotency state.

The failover succeeded. The payout may happen twice.

A regional evacuation: the replayed request reaches a second Region that cannot see the first Region’s idempotency state.

Replicating the table changes the question; it does not remove it.

DynamoDB global tables using multi-Region eventual consistency replicate asynchronously and resolve concurrent updates with last-writer-wins. Conditional writes are evaluated against the local replica’s version. During replication lag, two Regions can each accept what appears locally to be the first claim for the same key. Transactions are ACID only in the Region where they originate, not across Regions.

DynamoDB also supports multi-Region strong consistency. It synchronously replicates a successful write to at least one other Region, and conditional writes evaluate against the latest item version. That is a materially different contract, with topology, availability, latency, cost, and migration considerations; it is not a flag to switch casually after the incident. For example, an MRSC global table must use exactly three Regions, implemented as three replicas or two replicas plus a witness, and its consistency mode is chosen when the table is created.

The design question is broader than product selection:

Does the recovery point objective of the idempotency state match the recovery point objective of the business state it protects?

If the ledger survives a Region loss but the deduplication memory does not, the architecture can be highly available and financially unsafe at the same time.

A review checklist worth putting in the design document

Before calling an operation idempotent, write down:

1. Intent: What exact business decision does the key identify?

2. Generation: Where is the key created, and is it persisted before retries begin?

3. Propagation: Does the same key reach every side-effecting dependency?

4. Fingerprint: Which fields are bound to the key, and what happens if they change?

5. Concurrency: What happens when two matching requests overlap?

6. Completion: Is the original result stored and returned on duplicate requests?

7. Ambiguity: How do you reconcile a crash after the external effect but before local completion?

8. Retention: Is the record kept longer than the longest automated or manual replay horizon?

9. Scope: Is deduplication per process, account, tenant, queue, Availability Zone, Region, or global?

10. Failover: Can two Regions both accept the first claim?

11. Recovery: Do backups restore the business data and its deduplication history to a compatible point?

12. Testing: Have you injected timeouts before and after the effect, lost acknowledgements, concurrent retries, expired keys, DLQ redrives, and regional failover?

If one of those answers is “we assume,” the guarantee is unfinished.

The lesson, generalized

Idempotency is where teams discover that resilience and correctness are not the same property. Retrying can improve completion rates while quietly increasing the chance of duplicate effects.

The general rule is simple:

Any guarantee implemented through remembered state is bounded by what is remembered, for how long, and where that memory is authoritative.

Name the intent.

Reuse the key.

Bind it to the request.

Retain it for the replay you will actually perform—not only the retry you hope to see.

Make local state transitions atomic. Treat external effects as a separate failure boundary. Reconcile ambiguity. And ensure the memory survives the failure it exists to protect you from.

That is not “exactly once” as a slogan.

It is a correctness contract you can test.

What’s next

Both incidents in this issue arrived through the same door: a replay performed later by someone doing the responsible thing.

Next week: the dead-letter queue as a resilience liability. Why a DLQ is a decision deferred rather than a failure handled; what a poison message costs the consumer that keeps retrying it; how redrive interacts with every window in this issue; and why “we’ll replay it in the morning” is an architectural commitment almost nobody has tested.