Reachability Is Not Readiness

Anycast solves the addressing problem in a regional failover. It does nothing about the two problems that actually take you down — detection and readiness. Here is why a “Global Accelerator in front of two Regions” diagram is a traffic plan, not a recovery plan.

A composite incident I have walked through with more than one customer starts the same way: on a whiteboard, with a confident rectangle labelled “Global Accelerator” drawn across the top of two Regions.

The team was a payments platform. They had done the responsible thing and stood up a second AWS Region, replicated their data, and put AWS Global Accelerator (GA) in front of both stacks so that clients would talk to two static anycast IP addresses instead of chasing DNS records around the world. The mental model in the room was clean and satisfying: primary Region gets sick, health checks fail, anycast quietly steers everyone to the secondary Region, nobody notices. We scheduled a game day to prove it.

We pulled the primary Region’s application tier — not the load balancer, the application behind it. And traffic kept hammering the sick Region.

The Application Load Balancer (ALB) was still answering. Its health check was a shallow HTTPS check on a static page, and that page was perfectly happy to return 200 while the target pool behind it returned 500s to every real request. From Global Accelerator’s point of view, that endpoint was healthy. Anycast did exactly what it was told. It was told the wrong thing. That gap — a liveness probe answering a readiness question — is the whole reason next week’s issue exists, but hold that thought.

The second half of the story is worse, and I have seen it independently: teams whose failover does trigger, cleanly, in under two minutes — and who then watch the secondary Region fall over because it had been sized as a cold idea rather than a warm system, and it met production traffic for the first time at the worst possible moment.

Reachability is not readiness — and that is the whole article

Here is the reframing I ask people to sit with. Regional failover feels like an addressing problem: how do we get clients to stop talking to Region A and start talking to Region B without waiting on DNS caches we do not control? Anycast is a genuinely elegant answer to that specific question. But addressing was never the part that took you down.

The parts that take you down are detection — did we correctly and quickly notice that Region A cannot actually serve? — and readiness — is Region B genuinely able to carry the load the instant traffic lands? Global Accelerator moves packets. It does not decide whether the destination is ready to receive them, and it is only as smart about “sick” as the health signal you hand it.

Treat GA as a disaster-recovery button and you have quietly conflated “traffic will arrive” with “the service will work.” Those are different promises, and only one of them is on AWS’s side of the shared-responsibility line.

What anycast actually buys you — and what it quietly does not

It is worth being precise about the real value, because it is real. Global Accelerator gives you two static IPv4 anycast addresses by default (four for a dual-stack accelerator: two IPv4 and two IPv6), advertised from the AWS edge network. A client anywhere in the world enters at the nearest edge location, and its traffic then rides the AWS global backbone to your endpoints rather than the open internet. Global Accelerator terminates the client’s TCP connection at the edge and opens a fresh one to your endpoint, which is where a chunk of the latency and jitter improvement comes from.

The failover-relevant magic is that the client’s IP never changes. There is no DNS Time To Live (TTL) to expire, no stub resolver stubbornly caching a dead record for an hour because it decided your 60-second TTL was a suggestion. When GA decides to send new connections elsewhere, it just… does, at the network layer.

That is the addressing problem, solved. Notice what is not in that paragraph: any statement about whether Region B is ready, any statement about your data, any statement about the correctness of the health signal driving the decision. Anycast changed where packets go and how fast the change propagates. It changed nothing about the two hard questions.

The health check is the whole ballgame — and it is often lying

This is the part that surprises even experienced teams, so let me be exact about it, because the exactness is the point.

When your Global Accelerator endpoint is an Application Load Balancer or a Network Load Balancer (NLB), Global Accelerator does not run its own health check against it. It reuses the health checks already configured on that load balancer. The health-check fields you see on the GA endpoint group — interval, protocol, port, path — apply only to Amazon Elastic Compute Cloud (EC2) instance and Elastic IP endpoints. For ALB and NLB endpoints, those fields are inert.

Read that again if you run GA in front of load balancers, because it means your entire regional-failover decision is delegated to whatever health check you configured on the load balancer months ago and have not looked at since. If that check is a shallow TCP probe, or an HTTP GET / that returns 200 from a stub page while your actual dependencies are down, Global Accelerator will faithfully keep routing to a Region that cannot serve a single real request. That is precisely the gray failure that ended our game day. GA was not broken. It was honest about a lie.

For EC2 and Elastic IP endpoints, where GA does own the check, the knobs are worth knowing:

Endpoint group health check (EC2 / Elastic IP endpoints only)
  HealthCheckIntervalSeconds : 10 | 30      (default 30)
  ThresholdCount             : 1..10        (default 3)
  HealthCheckProtocol        : TCP | HTTP | HTTPS   (default TCP)
  HealthCheckPort            : listener port unless set
  HealthCheckPath            : "/"          (HTTP/HTTPS only)

# For ALB / NLB endpoints these fields are ignored.
# GA reuses the load balancer's OWN health check instead.

Endpoint group traffic dial (every endpoint type)
  TrafficDialPercentage      : 0..100       (default 100)

That ThresholdCount cuts both ways, and it is worth internalizing: the same count of consecutive checks that must fail before a healthy endpoint is declared unhealthy must also succeed before an unhealthy one is allowed back. Detection has a floor on the way down and a deliberate delay on the way back up — the second one is what stops a flapping dependency from yo-yoing your traffic between Regions.

The lesson generalizes past GA: a failover system is only as truthful as its weakest health signal, and the default health signals almost everywhere in our industry are optimized to answer “is the process up?” not “can this system do its job?” Those are not the same question, and your customers only ever ask the second one.

Failover has a floor, and “fail open” is not on your side

Even when the signal is honest, anycast does not make detection instantaneous. On the endpoint types where GA owns the check, the fastest honest configuration is a 10-second interval with a threshold of 1 — and the defaults are a 30-second interval with a threshold of 3, which is on the order of a minute and a half before an endpoint is even declared unhealthy, before you add convergence time. Anycast eliminated the DNS-caching tail. It did not eliminate detection latency. If your recovery-time objective is measured in seconds, the health-check math has to be part of the design conversation, not an afterthought.

Then there is the behavior almost nobody reads about until it bites: what Global Accelerator does when it cannot find a healthy home. When an endpoint group has no healthy endpoints with non-zero weight, GA looks to other endpoint groups — the closest ones, up to three more Regions — and it ignores the traffic dial while doing so, so a group you had dialed to zero is still a failover candidate. And if it finds no healthy endpoint anywhere, GA does not hold traffic back to protect you. It fails open: it routes to a random endpoint in the closest group and lets your stack sort it out.

Fail-open is the right default for a global traffic director — a network layer that black-holes traffic when confused is worse than one that keeps trying. But it is emphatically not a safety feature for you. It means “everything looks unhealthy” and “everything is fine” produce the same externally visible behavior: traffic still flowing. If you were counting on GA to be the thing that notices total failure and stops the bleeding, it will instead cheerfully keep the bleeding going.

One more sharp edge. Failover only governs new connections. Global Accelerator does not move established connections; they stay pinned to the endpoint they landed on until the connection resets or the client opens a new one. For request/response HTTP that is usually invisible. For long-lived connections — WebSockets, gRPC streams, database-style sessions — a “successful” failover can leave a large population of clients welded to the failing Region for as long as those connections live. The dashboard says failover succeeded. A meaningful slice of your users disagree.

Readiness is a different question, and it belongs to you

Notice the through-line. Every boundary above is Global Accelerator declining to answer a question it was never designed to answer. Is the health signal telling the truth? Is the standby Region warm and scaled? Has data replication caught up enough that failing over will not serve stale or lost writes? Are the established connections drained? None of these are addressing questions, and anycast is an addressing tool.

This is where readiness controls live — the discipline of continuously proving, before the incident, that the recovery target can actually take the traffic, and gating failover on that proof rather than on a TCP handshake. AWS has primitives in this space, and they are worth a serious look, but the primitive is not the point I want to leave you with. The point is that you cannot buy readiness as a side effect of buying fast rerouting. Readiness is a property you have to build, test, and keep warm, and no amount of anycast will manufacture it on the day.

The honest posture is to say plainly what Global Accelerator commits to and what it does not. It commits to getting client traffic onto the AWS backbone at the nearest edge, to a stable pair of IP addresses, and to steering new connections toward endpoints its health signal considers healthy, with a well-defined and fast convergence once it decides. It does not commit to the truthfulness of that health signal, to the readiness of the destination, to moving your in-flight connections, or to holding traffic back when everything is on fire. Every one of those is your architecture’s job. Draw that on the whiteboard, and the rectangle at the top stops being a recovery plan and becomes what it actually is: a very good traffic plan that your recovery plan is allowed to use.

The lesson, generalized

The failure mode here is not specific to Global Accelerator or even to AWS. It is the habit of buying a component that solves one crisp, visible sub-problem and unconsciously crediting it with solving the whole messy problem it sits near. Anycast makes rerouting look effortless, and “looks effortless” is exactly the property that lulls a team into skipping the game day that would have shown them the health check was shallow and the standby was cold.

Resilient systems are built by people who can state, without flinching, the precise boundary of every component they depend on — what it promises, and the first thing it stops promising. The teams that survive their bad Fridays are not the ones with the most impressive architecture diagrams. They are the ones who pulled the plug on purpose, in daylight, and were unsurprised by what happened next.

What’s next

Next week I want to stay on detection and pick apart the health check itself — the difference between a liveness probe, a readiness probe, and a dependency-aware health check, why “deep” health checks can turn a single sick dependency into a self-inflicted regional outage, and how to design a health signal that is honest about failure without being hair-triggered about it. The signal is where most failover stories are quietly won or lost, and it deserves an issue of its own.