Retries are Multipliers

Weeks ago I argued the safest fallback is often none at all. This week: the two dials everyone turns instead.

A retry is a load multiplier before it is a reliability feature, and the number that matters is not simply how many times one caller retrie, it is how many retries the whole system can afford. AWS is changing retry defaults across its SDKs and tools later this year. Worth understanding before the new behavior lands on you.

A composite incident I have walked through with customers more than once, because it repeats.

A payments platform. Mobile app, API edge, an orchestration service, a ledger service, a database. Five hops from the customer’s thumb to the row that matters. Every team had done the responsible thing: timeouts on every call, up to three attempts with exponential backoff, written down and reviewed.

One Thursday the ledger’s database got slow. Not down. Slow. A query plan went bad after a statistics refresh, and the ninety-ninth percentile went from tens of milliseconds to most of a second.

Then came the arithmetic. The ledger’s timeouts fired, so the ledger retried. The orchestration service’s timeouts fired, so it retried — each of its attempts becoming a fresh set of ledger attempts. The edge retried. The mobile app, written by a different team in a different building, retried too, because a spinner that gives up feels like a bug.

The database was not handling a slightly elevated load. It was handling the product of four independent retry policies, each entirely reasonable on its own. With three total attempts at each of those four retrying layers, the worst-case path can produce 3⁴ = 81 database attempts for one original customer operation.

The incident channel stared at a dashboard that said “database overloaded.” True, and exactly backwards. It was slow because it was overloaded and overloaded because it was slow. Nobody made a bad decision. Everybody made the same good one, and the good decisions multiplied.

A Retry Is a Decision About Someone Else’s Capacity

Here is the reframing to carry out of this issue. A retry is not, in the first instance, a reliability feature. It is a request for more of someone else’s capacity, made at the exact moment that capacity may be scarcest.

The Amazon Builders’ Library puts it in one word: “Retries are ‘selfish’”. The full sentence is worth having verbatim: “When a client retries, it spends more of the server’s time to get a higher chance of success”.

Your availability goes up. Someone else’s headroom goes down. That trade is often correct, but it is still a trade.

The same source draws the line: “Where failures are rare or transient, that’s not a problem.” A retry against one unlucky packet loss or connection reset is nearly free. But, as the same source says, “when failures are caused by overload, retries that increase load can make matters significantly worse.”

The policy that saves you on a good day can become the one that hurts you on a bad one — and a 500 does not carry a field explaining whether the dependency stubbed its toe or is drowning. Your retry logic is guessing, every time.

The Multiplication Table Nobody Prints

The most sobering number in AWS’s published resilience material comes from the Builders’ Library. It asks you to “consider a system where the customer’s call causes a five-deep stack of service calls. It ends with a query to a database, and three retries at each layer.”

AWS then shows the consequence: when each of five layers independently makes three attempts, the database can see 3⁵ = 243 times the load from the original request. Two hundred and forty-three.

My payments example above has four retrying layers, so its equivalent worst-case multiplier is 3⁴ = 81. The AWS example adds one more independently retrying layer and gets 243. Either number is enough to ruin your Thursday.

This is also why terminology matters. In most AWS SDK configuration, max_attempts = 3 means one initial request plus up to two retries: three total attempts. Other systems use “retry attempts” differently. AWS Step Functions, for example, defines MaxAttempts as the number of retries after the original state execution. Count attempts, not labels.

The AWS Well-Architected Reliability Pillar names the broader problem as a first-class anti-pattern in REL05-BP03: “Retrying at multiple layers of your application stack in a manner which compounds retry attempts further consuming resources in a retry storm.”

It also names the endgame: configure maximum retry counts or elapsed-time limits so retries do not create backlogs capable of producing metastable failures. Metastable is the polite word for a system that stays down after the original trigger is gone, held there by its own traffic.

The remedy in the Builders’ Library is unfashionably simple: for low-cost control-plane and data-plane operations, Amazon’s best practice is generally to retry at a single point in the stack. One layer owns the retry. Everyone else fails and propagates the result.

Yes, retrying only higher in the stack may waste work the lower layers already completed. Wasting some work beats multiplying all of it by 81, 243, or whatever number your architecture quietly produces.

That is a design decision, not a config value. Go and count how many layers between your customer and your data store are retrying today. The honest answer is usually “more than the diagram shows,” because SDKs retry automatically and nobody counted them as another retrying participant.

Backoff Spaces the Flood. Jitter De-correlates It. A Budget Bounds It.

Exponential backoff and jitter answer a different question than the one most people are asking.

REL05-BP03 is clear on the mechanics: use exponential backoff to place progressively longer intervals between retries, introduce jitter so those retries do not synchronize, and limit the maximum number of retries.

Jitter matters because synchronization is dangerous. Thousands of clients that fail together should not all wake up together.

Full jitter, from AWS’s own analysis of the variants, is usually the shape to reach for — and it is the model used by AWS SDK standard retry behavior:

// full jitter: randomize across the WHOLE window, not around it
sleep = random(0, min(cap, base × 2^attempt))

// the SDK form, with a hard ceiling
delay = random(0, 1) × min(20,000 ms, base_delay × 2^retry)

This is very different from simply waiting one second, then two, then four. If every client failed at the same instant and uses the same deterministic schedule, they can return together at the same instant. Randomizing across the whole backoff window breaks the herd apart.

Jitter can also reduce total work in contention scenarios because spreading attempts increases the chance that some succeed earlier instead of generating another synchronized round.

But there is an important boundary. Backoff and jitter do not themselves create a system-wide retry budget. During a sustained failure where every permitted attempt continues to fail, and no enclosing deadline expires, they mainly redistribute the allowed attempts in time.

They control timing and correlation. They do not answer the larger question: how much retrying are we prepared to allow before the retry mechanism itself becomes part of the failure?

That is where a retry budget enters.

The Budget Is the Real Control — and Yours Is About to Change

Amazon has used token-bucket retry limiting in AWS SDK behavior since 2016. The Builders’ Library describes the idea plainly: retries consume a finite local resource so a client cannot continue retrying aggressively forever.

Standard retry mode later standardized that concept across AWS SDKs, and on 20 May 2026 AWS announced another significant update to retry behavior across AWS SDKs and the AWS Command Line Interface.

The new defaults are already available behind an opt-in environment variable and are scheduled to become the default no sooner than November 2026. To enable the new behavior today in supported SDKs and tools, set AWS_NEW_RETRIES_2026=true.

Under the 2026 defaults, standard becomes the default retry mode across the SDKs and tools covered by the rollout. For general AWS service clients, the default maximum is three total attempts: one initial request plus up to two retries.

What changes is the shape of the waiting and the price of retrying.

Transient failures, including things such as connection failures, DNS failures, and HTTP 500, 502, 503 and 504 responses — now use a 50 millisecond base delay. That replaces SDK-specific transient defaults that previously varied from roughly 10 milliseconds to 1,000 milliseconds.

The logic is sensible: a genuine transient blip should not automatically consume a full second of your latency budget.

Throttling responses use a 1,000 millisecond base because throttling is the dependency explicitly telling you to slow down. Both follow full-jitter exponential backoff against a hard 20-second backoff ceiling.

DynamoDB and DynamoDB Streams are tuned separately, with a 25 millisecond transient base and four maximum attempts.

Then comes the number I think matters most: the retry quota starts with 500 tokens. It is typically scoped to an SDK client instance — exact implementation details can vary by SDK, and it is not shared globally across processes or hosts.

A throttling retry costs 5 tokens. A transient retry under the new behavior costs 14. Previously that transient cost was 5. That change is much more consequential than the difference between 50 milliseconds and 100 milliseconds of backoff.

A successful retry restores the token cost associated with that successful retry. A clean first-attempt success restores one token. There is a subtle but important consequence: if an earlier retry failed and a later retry succeeds, the cost of the earlier failed retry remains spent.

For ordinary operations, once the retry quota is depleted, the SDK stops issuing retries and returns the error to the caller. Long-polling operations have slightly different handling so a depleted quota does not turn a polling loop into a hot spin, but the principle is the same: retry aggressiveness collapses as sustained failure consumes the budget.

Work through those costs and the intent falls out. With three maximum attempts, AWS calculates that the quota begins draining under sustained transient failures at roughly 22% of requests, and at roughly 32% when the failures are throttling responses.

Below those levels, normal successes replenish enough quota to absorb ordinary noise. Above them, the client’s ability to keep retrying begins to disappear deliberately. As AWS describes the behavior, the application starts failing fast rather than waiting through retries that are increasingly unlikely to succeed.

That gives you circuit-breaker-like retry suppression without introducing an explicit open / half-open / closed state machine. That does not mean it requires no testing. It absolutely does.

The distinction is that retry eligibility changes progressively as quota is consumed rather than requiring the application to manage a separate dependency state transition.

That matters because the Builders’ Library is deliberately cautious about explicit circuit breakers. They introduce modal behavior into systems, can be difficult to test, and can add recovery time when the dependency has recovered but the circuit has not yet decided to trust it again.

A token bucket is not stateless and it is not magic. It is simply a different control surface.

What a Retry Budget Does Not Buy You

There are three honest boundaries.

First, it is not global. The bucket is typically local to an SDK client instance and is not shared across processes or hosts. Ten thousand hosts each holding a fresh 500-token retry quota can still, collectively, flatten a dependency.

Client-side budgets protect a dependency from an individual caller continuing to retry without limit. They do not create a fleet-wide admission-control system. Fleet-scale protection still requires controls at the scope where the traffic converges: throttling, load shedding, admission control, concurrency limits, or other server-side protection.

Second, it does not make retries safe. It makes them bounded.

REL05-BP03 is blunt about the correctness half: “Verify that services are idempotent before implementing retries”. It also names another anti-pattern: retrying service calls that are not idempotent and therefore creating duplicated side effects.

A retry budget limits what retry amplification can cost your availability. It says nothing about whether you just charged the same card twice.

Third, it does not fix your timeouts.

Every retry policy sits on top of a timeout policy. Too long, and your retry never gets a useful turn while requests, threads, connections or concurrency slots accumulate. Too short, and you manufacture failures out of healthy-but-slow responses.

REL05-BP05 refuses to give you one magic number and gives the instruction that matters instead: set connection and request timeouts appropriately, verify them systematically, and do not blindly trust defaults because defaults do not know your workload.

You normally need to think separately about connection establishment and the lifetime of the request itself.

If you want a method rather than a vibe, use Amazon’s own starting point. For controlled service-to-service calls, choose an acceptable false-timeout rate — Amazon gives 0.1% as an example — then examine the corresponding downstream latency percentile.

If you tolerate 0.1% false timeouts, that might point you toward roughly the p99.9 latency of the dependency. That is a starting point, not a universal formula.

Internet-facing clients need additional network-latency allowance. Very tight p50-to-p99.9 distributions may need padding. You also need to understand whether the timeout primitive you are using includes DNS resolution, TCP connection setup, TLS negotiation and other work that happens before your application thinks the request has really started.

A timeout should be a deliberate point in the latency behavior of the dependency plus the surrounding network path. Anything else is probably a round number somebody typed in 2019.

The Retry You Do Not Control

Here is the uncomfortable part, and the reason I opened with a mobile app.

You can budget every retry inside your own estate and still be taken down by clients you will never ship code to: a partner integration, a browser, a phone running a build from eighteen months ago, an SDK somebody wrapped in another SDK.

They will retry on your worst day, sometimes without jitter, without a budget, and without any knowledge of what your dependency is experiencing.

Two things help. Neither of them is another client-side retry policy.

The first is to tell them in-band, using mechanisms the protocol already gives you.

HTTP defines Retry-After. RFC 9110 §10.2.3 defines the header and its use with responses including 503 Service Unavailable. RFC 6585 §4 defines 429 Too Many Requests and allows a 429 response to include Retry-After, telling the client how long to wait before trying again.

Some clients ignore it. Send it anyway. It costs a header and turns a guess into an instruction for every client willing to listen.

The second is to stop treating that boundary as a place where you negotiate and start treating it as a place where you enforce. If your survival depends on a stranger’s retry loop being polite, you have a hope, not an architecture.

Audit the defaults you inherited, too. AWS Step Functions is a good example of why terminology matters.

When you configure a retrier but omit its timing fields, the retrier defaults to:

"IntervalSeconds": 1 "MaxAttempts": 3 "BackoffRate": 2.0 "JitterStrategy": "NONE"

Full jitter is one field away — "JitterStrategy": "FULL" — but it is not the default.

Step Functions also uses MaxAttempts differently from AWS SDK max_attempts. For a Step Functions retrier, "MaxAttempts": 3 means up to three retry attempts after the original execution. For an AWS SDK configured with max_attempts = 3, it means three total attempts including the initial request.

Same number, different arithmetic. That is precisely the sort of default that disappears inside an architecture diagram.

The Lesson, Generalized

Every retry is a bet that the failure you just saw was isolated. When it is, the bet pays and nobody notices. When it is not, you have added load to something already failing under load — and multiplied that decision by every layer that made it independently.

So stop asking only how many times you should retry.

Ask instead who in this call path is allowed to retry at all. What counts as a retryable failure? How many total attempts can one operation create? How much retry traffic can one client generate before it stops helping? What is the end-to-end latency deadline? And what happens when the system reaches those limits?

Backoff controls when the retry happens. Jitter controls how correlated retries are across clients. Maximum attempts bound amplification for a single operation. A retry quota controls how aggressively a client keeps retrying as sustained failure becomes widespread.

Those are different controls solving different problems.

And this is not really about retries. Any behavior that is locally rational and multiplicative at system scale needs a bound at the scope where that multiplication happens — not just another locally reasonable default.

Because every participant will be able to prove, afterward, that their own number was reasonable.

What’s Next

A retry budget bounds what retries cost you in availability. It says nothing about what they cost you in correctness.

Next week: idempotency, and the exactly-once story we keep telling ourselves. Why at-least-once delivery is all most of the stack actually promises, what an idempotency key has to cover to be worth having, why deduplication windows are a resilience decision disguised as a storage decision, and how a retry that is perfectly safe for a read becomes a duplicated payment the moment someone adds a side effect.