Mostly One Leader

In the last issue of this newsletter I argued that every saga eventually grows a sweeper. Somewhere behind the happy path, a process has to find the transactions that never quite finished and decide what to do with them. I ended with the problem that creates: you have now given a scheduled process permission to repair your business state, and you need to be very sure about what happens when two copies turn up for work.

This week, let us follow one of those copies through the forty-one seconds in which it loses its job without finding out.

THE WORKER THAT CAME BACK

Imagine a payments platform with a nightly reconciler. This is a fictional example, with illustrative timings, rather than an account of an AWS or customer incident.

The reconciler looks for authorisations with no matching settlement. Once a grace period has passed, it requests release of the hold. The team has been sensible about introducing it: two weeks in dry run, a dashboard showing divergence, and compensating operations designed to tolerate retries. They have also put a lease in DynamoDB so that only one reconciler is supposed to work at a time. The holder renews every ten seconds; the lease lasts thirty.

On Tuesday night, worker A reads a batch of authorisations and prepares its next update. Then its runtime enters a long garbage collection pause. For forty-one seconds, the process makes no progress. It does not renew the lease, receive a warning or reconsider the batch sitting in memory. Everything it knew before the pause is still waiting for it afterwards.

Meanwhile, worker B is doing exactly what the team wanted a standby to do. It observes that the lease can be acquired, takes ownership and starts reconciling. From the team’s point of view, failover has worked.

Then A resumes.

Its next instruction is still its next instruction. It submits the work it prepared before the pause, while B is already processing the same population of authorisations. Some release requests are duplicated; others reflect state that has since changed. Whether those requests cause duplicate financial effects depends on the provider and the idempotency design. But the team already has a problem: decisions made under an obsolete claim to ownership are still reaching the write path.

It would be easy to describe this as a failed distributed lock. Yet the lease allowed a new owner after the old one stopped renewing, exactly as intended. The missing part was what should happen when the old owner returned.

WHAT THE LEASE COULD NOT TELL A

From B’s perspective, the handover is clear: the lease record says B owns the current term. From A’s perspective, nothing has happened since it checked its right to work. There is no contradiction in the database. There are two processes operating with different information.

The AWS Builders’ Library discusses this uncomfortable property of leader election: a system can mostly have one leader while still encountering periods with none or two. It also calls out pauses between checking a lock and doing the work. Kubernetes client-go makes the boundary explicit too: its leader election implementation does not guarantee that only one client is acting as leader.

Our fictional team might respond by adding another lease check immediately before each update. That narrows the window during normal execution, but A can still pause after the check. A shorter lease makes B take over sooner; it does not make A stop sooner. Asking the worker more frequently whether it is still entitled to write cannot close a gap in which the worker itself may disappear.

There is a further detail in this DynamoDB example. The service is evaluating conditions against stored values; it is not running a lease clock on the application’s behalf. The application supplies the time used in its expiration comparison. Missing three renewals might be a useful warning, but the acquisition protocol, its time assumptions and the stored expiration value determine when another worker can take over.

To protect the write, the team needs to move the decision to somewhere that can reject A even when A still believes it is doing its job.

FOLLOW THE OLD WRITE ALL THE WAY TO THE DATABASE

Give A’s leadership term a number: 6. When B acquires the lease, atomically advance that number to 7. Each worker carries its number with the work it submits. These numbers are the fencing tokens, or epochs, and they let a destination distinguish a delayed former owner from the current one.

But simply attaching the number is not enough. Suppose each authorisation remembers the highest epoch it has accepted and rejects anything lower. If B has already written to an authorisation with epoch 7, A’s epoch-6 write will fail there. That is useful protection. Now follow A to an authorisation B has not reached yet: its highest recorded epoch may still be 6, and A’s old write can pass.

This is where a convincing-looking example can quietly promise too much. A highest-seen fence rejects earlier generations once a newer generation has reached that resource. Acquiring epoch 7 in a lease record does not, on its own, update every destination.

For the database portion of our reconciler, we can instead check the current leadership record and update the authorisation in one DynamoDB transaction. DynamoDB supports this across distinct items in tables in the same AWS account and Region.

The transaction has two parts. First, confirm that the lease record still names this worker and its epoch. Second, change the authorisation only if its version and business state still match what the worker observed. Both succeed together, or neither does. In conceptual pseudocode:

TransactWriteItems
  ConditionCheck  lease record
    ConditionExpression: #owner = :me AND epoch = :my_epoch
  Update          authorisation
    UpdateExpression:    SET #state = :release_requested, version = version + :one
    ConditionExpression: version = :observed_version AND #state = :observed_state

Run the Tuesday-night sequence again. B acquires epoch 7. A wakes up and submits this transaction with epoch 6. Its leadership check fails, so its authorisation update does not commit. It no longer matters whether B has touched that particular authorisation. The authoritative ownership record is part of the decision.

There is an exact boundary to that guarantee: A’s transaction can commit if it is ordered before B’s takeover. Once takeover commits, A cannot pass the old-generation check. This gives us an ordering between ownership changes and protected writes; it does not give us a promise that all writes cease at the precise wall-clock instant a lease expires.

Adding “expires_at > now” does not magically create that promise either. In a DynamoDB condition, “now” is a value the caller supplied. A request can be delayed after capturing it. The distinction matters: lease expiration makes takeover possible, while advancing the authoritative epoch fences the previous term in this design.

The acquisition protocol therefore deserves as much care as the transaction. Use a conditional update to acquire an expired lease and increment its epoch atomically; renewals must verify the same owner and epoch and preserve that epoch. Keep the counter in a persistent record, with a unique owner identifier for each acquisition attempt. Deleting the record and starting its counter again would let an old number become current. DynamoDB provides the update primitives, but clock assumptions, delayed requests and uncertain responses remain application responsibilities.

This is a design sketch for one regional coordination domain. Every protected write must participate, and a replicated table does not turn it into a cross-Region lock. Those are constraints worth knowing before the first failure, when changing the design is still an ordinary engineering task.

THE PAYMENT HAS NOT BEEN RELEASED YET

Look again at the transaction: it records release_requested. It does not say released, because changing a DynamoDB item cannot atomically complete a call to an external payment provider.

That brings our team to the next boundary. They can record an outbox command in the same transaction and arrange for it to be delivered, but the delivery worker can also retry. The provider may perform the release and lose the response. The command therefore needs a stable business-operation identifier and an appropriate downstream idempotency contract. Fencing has stopped a superseded reconciler from committing new database decisions; it has not solved every way an external operation can be repeated.

The same care is needed when comparing storage services. S3’s If-Match checks an object’s ETag, while If-None-Match can require that the object does not already exist. Those are valuable conditional-write mechanisms, but neither header independently tells S3 who holds the current leadership term.

EBS Multi-Attach offers another, more literal example through NVMe reservations on io1 and io2 volumes. Hosts can register keys and use reservations to control access, including preempting a previous holder. The storage software must actually establish and manage the appropriate reservation; enabling support does not establish exclusive ownership for it. The documentation also describes the creation-date defaults and the detach-and-reattach procedure needed for older volumes.

The useful question in each case is the one our team had not asked at the beginning: when A comes back, what exactly will refuse its operation?

TUESDAY NIGHT IS NOT THE ONLY WAY TO GET TWO WORKERS

Once the team has reproduced the pause, it is tempting to file this under an unusual runtime failure and move on. But now that they are looking for overlapping work, they can find less dramatic ways to produce it.

Their schedule can deliver twice: EventBridge Scheduler provides at-least-once delivery. A queue can deliver a message again: SQS standard queues do not promise exclusivity even during the visibility timeout. A deployment can introduce a second process deliberately: an ECS rolling deployment with maximumPercent set to 200 can run two tasks for a service whose desired count is one. None of those mechanisms has to malfunction to expose the assumption.

If the reconciliation job fits a workflow, Step Functions Standard Workflows may remove some of the coordination machinery the team would otherwise maintain. Standard has an exactly-once execution model, with configured Retry behaviour allowing tasks or states to run again. Asynchronous Express uses at-least-once execution; synchronous Express uses at-most-once execution.

For Standard, starting the same running execution with the same name and input returns the original response. Different input under that name, or a closed execution within the name-reuse restriction period, produces ExecutionAlreadyExists. A deterministic name for a logical reconciliation run can therefore help suppress duplicate starts.

It still needs to be the same logical run. Two differently named workflows can touch the same authorisation, and a task that retries an external payment operation still needs to handle the possibility that the first attempt succeeded. The attraction of a managed workflow is how much coordination it can remove from the team’s workload, provided they understand where its guarantees end.

WHEN AN EARLIER CHECK BECOMES STALE

There is a public incident that makes the timing problem unusually concrete. AWS’s report on the 19–20 October 2025 DynamoDB disruption in US East (N. Virginia), using the report’s PDT dates, describes a race in DNS automation.

Three independent DNS Enactors ran across Availability Zones. One was unusually delayed while applying a plan. Another applied a newer plan and initiated cleanup. The delayed Enactor then replaced the regional endpoint’s newer plan with its older one: the freshness check it had performed at the start no longer protected that update. Cleanup deleted the older plan, removing the endpoint’s IP addresses and leaving an inconsistent state that required manual repair.

The connection to our example is the interval between validation and action. I would not infer a particular lease implementation, or prescribe an undisclosed remedy, from that account. What it shows is why redundant automation also needs correct ordering between updates and cleanup. Having another actor available does not make an earlier check remain valid.

GIVE THE STORY A DIFFERENT ENDING

Back at our fictional payments platform, the most useful test now begins by stopping A at precisely the wrong moment. Let it read its batch and check its lease. Pause it. Allow B to take over. Then resume A before B has reached the authorisation A wants to change.

With the transactional ownership check, A’s stale update is refused. The team can follow that result through the rest of the system: verify that no unfenced write path bypasses it, check that repeated commands use the same business-operation identifier, and test what happens when the provider’s response disappears. They can also check that the current leader makes the right business decision, because a perfectly fenced worker can still be wrong.

The difference is visible in how the failure ends. A may still wake up believing it owns the work. The database no longer has to accept that belief. B can continue, and the old worker’s return becomes a rejected operation instead of another reconciliation problem.

That is the design I want to see before calling a sweeper safe: one whose correctness survives the return of the worker everyone else has already replaced.

---

WHAT’S NEXT

Our current leader can now establish its right to write. Next week, it discovers whether there is enough capacity to do so.

Quotas, throttling and the capacity nobody reserved: what happens when ten thousand clients reconnect during recovery, why an account-wide limit can follow you into your failover plan, and how “we failed over, then immediately failed” becomes a perfectly plausible outcome.