Your Health Check Is a Trigger, Not a Test

A thorough health check feels safer than a shallow one. It is often the exact opposite. The depth that lets you evict one sick server is the same depth that lets one sick dependency evict your entire fleet at once. Here is how to build a health signal that is honest about failure without turning a dependency’s bad minute into your regional outage.

A composite incident I have walked through with more than one team always seems to land on a Friday, and it always starts with a health check somebody was proud of.

The team ran a retail checkout service — a couple of Availability Zones, a fleet of instances behind an Application Load Balancer (ALB), the whole thing managed by an Auto Scaling group (ASG). At some point a thoughtful engineer had upgraded the health check. The old one was a bare HTTP probe of a static page, and it had once kept traffic flowing to a box whose application had quietly wedged while the port stayed open. So they made the new one deep. The /health endpoint now opened a connection to the database, touched the session cache, and called the downstream authorization service before it would return 200. If an instance could not actually complete a checkout, it would say so. Thorough. Honest. Exactly what the previous postmortem had asked for.

At nine o’clock on a Friday night, the authorization service had a bad ninety seconds. A deploy, a throttle, a garbage-collection pause — it does not matter which. It did not go down. It got slow.

And every instance in the fleet failed its health check at the same instant.

Not one box. All of them. They were healthy in every way a customer holding a full cart would care about — the web tier was up, the database was fine, the cache was fine — but they all called the same slow dependency inside /health, so they all timed out inside /health, together. The ALB did exactly what it is built to do and marked the targets unhealthy. And because the ASG’s health-check type was wired to the load balancer, Auto Scaling reached its own honest conclusion — the entire fleet is sick — and began terminating and replacing every instance in it. Into a dependency brownout, on a Friday night, the platform started tearing down the one tier that was still working.

The authorization service recovered in about ninety seconds. The outage lasted a great deal longer than ninety seconds.

A health check is a trigger, not a test

Here is the reframing I ask people to sit with, because it inverts the instinct that caused the incident.

We talk about health checks as though they were tests — small experiments that discover the truth about a server. That framing is exactly why “deeper” feels “safer”: a more thorough test surely tells you more, and more is better. Right?

But a health check is not sitting quietly reporting the truth to a dashboard. It is wired to a weapon. Its output pulls a trigger — evict this target from the load balancer, terminate this instance and launch a replacement, steer Amazon Route 53 away from this endpoint. So the question that actually matters is not “how much truth does this check discover?” It is “what happens the moment this check fails, and how many things can make it fail at the same time?”

Once you see the check as a trigger rather than a test, the design rule falls out on its own. The depth that lets a check catch a genuinely broken server is the same depth that lets a single shared dependency fail the check on every server at once. Depth buys you sensitivity. Sensitivity to a shared cause is correlation. And correlation is precisely the thing that turns a one-box problem into a fleet-wide one.

Three checks, three blast radii

The AWS Builders’ Library article on implementing health checks draws a distinction I find clarifying, and it is worth restating in terms of blast radius rather than thoroughness.

A liveness check confirms the smallest possible thing: the server is listening on its port and accepting new TCP connections, perhaps returning a basic HTTP response. It catches a dead process. It cannot catch a process that is up and lying.

A local health check goes one step further and tests things that are true or false on that host alone — is the disk writable, are the critical worker processes alive, did configuration load correctly. The important property, in the library’s own words, is that these checks are “unlikely to fail on many servers in the fleet simultaneously.” Each server’s answer is independent of its neighbors’. One bad box fails; the boxes beside it do not.

A dependency health check reaches outward — to the database, the cache, a peer service — and reports whether that interaction actually works. This is the deep check, and it genuinely catches things the others miss: expired credentials, a broken network path, a dependency that is fine for the fleet at large but unreachable from this one subnet. But it carries a property the local check does not. In the library’s words again, it “can also have false positives when there are problems with the dependency itself.” When the dependency has a bad minute, every server that checks it fails together. The blast radius of a dependency check is not one host. It is every host that shares the dependency — which, for a shared dependency, is all of them.

Same two words, “health check.” Three completely different blast radii. The mistake in the checkout story was not building a dependency check. It was wiring a dependency check to a fleet-wide trigger.

Why “deep” quietly means “correlated”

Watch what the depth did once it was connected to automation.

The ALB marked the targets unhealthy — correct, per its configuration. Then two separate mechanisms compounded it. First, the ASG was configured to use ELB health for instance health, so a load-balancer verdict of “unhealthy” became an Auto Scaling verdict of “replace,” and the group started churning the whole fleet during the one window when it should have held perfectly still. The AWS networking guidance on choosing health checks is blunt about this exact trap: adding deep checks to an ASG means a transient dependency failure “could cause every registered target to fail their ELB health checks simultaneously,” and the standing best practice is equally blunt: “don’t add ‘deep’ ELB health checks to your ASG.”

Second, consider what a load balancer does when every target is unhealthy at once. It fails open — it gives up on the health signal and sends traffic to all targets anyway, on the theory that a black hole is worse than a coin flip — the right instinct for a traffic director, but read what it means for you: “all targets healthy” and “all targets unhealthy” collapse into the same externally visible behavior. Traffic still flows. The deep check you added to protect yourself has, in the failure that matters most, no effect at all — except the collateral damage it caused on the way there.

This is the honest boundary worth stating out loud. A deeper health check does not make a correlated failure safer. It makes a correlated failure more likely to fire your automation, and the automation is where the real damage lives.

# Load-balancer health check → drives eviction and (if ASG uses ELB health) termination.
# Keep it LOCAL. No dependency calls.
GET /health
    return 200 if (disk_writable and config_loaded and workers_alive)
    else 503

# Dependency-aware check → drives ALARMS and humans, NOT automated eviction.
# Scraped by monitoring on its own schedule; never by the ALB.
GET /health/deep
    call database, auth, cache
    emit metrics; page on a sustained, fleet-wide failure

# ALB target group defaults that decide how fast the trigger pulls:
#   HealthCheckIntervalSeconds : 30    (range 5-300)
#   HealthCheckTimeoutSeconds  : 5
#   HealthyThresholdCount      : 5
#   UnhealthyThresholdCount    : 2     -> ~60s to evict at the defaults
#   Matcher (success codes)    : 200

The signal chooses the consequence — so choose the consequence first

Different health checks pull different triggers, and each trigger has its own temperament. It is worth matching the depth of a check to the violence of what it controls.

An ALB eviction is cheap and reversible: a target drops out and rejoins the moment it passes HealthyThresholdCount checks again. An ASG termination is expensive and permanent: the instance is gone, and you pay full cold-start time to replace it. The same signal should not drive both — a signal aggressive enough to be useful for the cheap action is dangerous wired to the expensive one.

Amazon Route 53 sits at the other end of the spectrum, and its designers clearly thought hard about correlation. A Route 53 health check runs from a global fleet of checkers on a configurable interval — standard at 30 seconds or fast at 10 — and here is the part worth internalizing: Route 53 considers an endpoint healthy as long as more than 18% of its health checkers report it healthy. The system is deliberately biased toward calling an endpoint up, because at DNS scale a false “down” that pulls a Region out of rotation is far more destructive than a false “up.” It is a fail-toward-availability posture worth copying: the more powerful the trigger, the more evidence it should demand before it fires.

Honest without being hair-triggered

So how do you build a signal that tells the truth about failure without flinching at every twitch? A few principles have held up for me.

Match the check’s depth to its trigger’s blast radius. Local checks — the ones that fail independently, one box at a time — are safe to wire to automated eviction and replacement. Dependency checks belong on a different circuit entirely: let them drive alarms, dashboards, and a human’s attention, not the automatic destruction of capacity. The Builders’ Library makes the same split, recommending you keep the fleet-killing automation on shallow, local signals and handle dependency health through “external monitoring systems” where a person can apply judgment before anything gets torn down. And do not leave the other gap open: if your load-balancer check is deep and therefore not wired to the ASG, a box whose application has died will never be replaced — so give a monitoring watcher the job of spotting locally-broken instances (a heartbeat table feeding SetInstanceHealth is AWS’s own suggested pattern) with strict velocity limits so it can never mass-terminate.

Do the timing math before you tune the thresholds, not after. At the ALB defaults — a 30-second interval and an unhealthy threshold of 2 — a target takes roughly a minute to be declared unhealthy, before any convergence. Tighten those numbers and you detect real failures faster; you also convert more transient blips into evictions. There is no setting that is both fast and calm; choose the trade-off on purpose against your recovery-time objective (RTO), rather than inheriting it by accident.

Give the check a memory. A single failed probe is noise; a dependency that has been unreachable for thirty seconds is a signal. Requiring several consecutive failures, and — where the platform allows it — degrading gracefully rather than hard-failing when a non-critical dependency is the one that is sick, keeps a hiccup from being indistinguishable from a heart attack. A checkout service whose recommendation engine is down is not an unhealthy checkout service. Do not let it report that it is.

And say plainly what a passing check does not prove. A green health check means the instance believes it can serve. It does not mean the Region behind it is warm and scaled, that replication has caught up, or that the thing your customer is about to do will actually succeed. The check is a necessary signal. It was never a sufficient one, and the teams that get burned are the ones who quietly promoted it from the first to the second.

The lesson, generalized

The failure here is not specific to load balancers, or to Auto Scaling, or to AWS. It is a habit of thought: mistaking a control signal for an observation. The instant a measurement is wired to an automatic action, its most important property stops being how accurate it is and becomes how many things can make it fire at once — because that number is the size of your worst day.

Thoroughness is not a virtue in a trigger. Independence is. The health check that saves you is not the one that knows the most about the world; it is the one that fails one server at a time, tells the truth about that server, and refuses to speak for the whole fleet just because the whole fleet happens to share a dependency having a bad ninety seconds.