Failover Weekly No. 9
Lies Cloud Architects Tell Themselves at 2 AM...
Every cloud architect has a moment — usually at 2 AM during an incident — where they stare at a dashboard and realize that something they swore was resilient absolutely is not.
I have collected these moments. Not from strangers. From myself. From colleagues. From postmortems that read like confessions. Consider this newsletter a public service: a roast of the resilience delusions we all carry, so you can laugh now instead of cry later.
LIE #1: “WE’RE MULTI-AZ, SO WE’RE FINE”
Ah yes. The classic. You deployed to three Availability Zones. Your Auto Scaling Group spans all of them. Your load balancer has health checks. You are invincible.
Except your database is a single-AZ RDS instance because “multi-AZ doubles the cost and we’ll migrate later.” Your secrets are cached locally on boot and never refreshed. And your health check returns 200 OK if the process is alive — not if it can actually serve traffic.
You are not multi-AZ. You are multi-AZ with a single point of failure wearing a trench coat.
The fix is boring: actually test an AZ failure. Shut one down during business hours. Watch what screams. Fix what screams. Repeat until nothing screams. Most teams have never done this once.
LIE #2: “OUR DR PLAN IS DOCUMENTED”
It is documented. In a Confluence page last updated in 2023 by someone who no longer works at the company. The runbook references a Jenkins pipeline that was decommissioned, an IAM role that was rotated, and a Slack channel that has been archived.
Your DR plan is not a plan. It is a historical artifact. It belongs in a museum next to your original architecture diagram that still shows a monolith.
A disaster recovery plan you have not tested is a disaster recovery wish. And wishes do not restore databases.
The test: can a new team member who has never seen the runbook execute it successfully in under your target RTO? If the answer is “probably not,” that is your DR maturity level. Probably not.
LIE #3: “WE HAVE MONITORING”
You have 4,000 CloudWatch alarms. 3,800 of them are in ALARM state and have been for six months. Nobody remembers why. Everyone ignores them.
You also have a Grafana dashboard that looks like mission control at NASA. It is beautiful. It has 47 panels. Nobody checks it unless there is already an incident, at which point everyone stares at it while saying “what am I looking at?”
Monitoring without actionable alerting is just data hoarding. You are not observing your system. You are collecting telemetry for fun, like a birdwatcher who never identifies the birds.
The rule is simple: if an alarm fires and nobody responds, delete it. If a dashboard exists and nobody checks it proactively, question why it exists. Observability is not a quantity game. It is a signal-to-noise game, and right now the noise is winning.
LIE #4: “WE’LL FAIL OVER MANUALLY”
The plan is that during an outage, a senior engineer will assess the situation, make a judgment call, and execute the failover to the secondary region.
That senior engineer is on vacation. The backup is in a meeting. The third-in-line is asleep because it is 2 AM in their timezone. The runbook says “contact the database team” and the database team’s on-call rotation expired last quarter.
By the time someone with the right permissions, the right context, and the right courage to press the failover button is available, your RTO has been exceeded by a factor of four and your CFO is texting your CTO in all caps.
Manual failover is not a resilience strategy. It is a hope-based strategy with extra steps. Automate the detection. Automate the decision. Automate the execution. Let humans handle the things humans are good at, like writing blameless postmortems that are only slightly blameful.
LIE #5: “OUR ARCHITECTURE IS RESILIENT BECAUSE WE USE KUBERNETES”
Kubernetes will restart your pods. Kubernetes will reschedule your workloads. Kubernetes is basically magic.
Kubernetes also just scheduled all your replicas onto the same node because you forgot to set pod anti-affinity rules. Your persistent volumes are tied to a single AZ. Your ingress controller is a single point of failure that nobody thinks about because “it’s infrastructure.” And your cluster autoscaler takes four minutes to provision new nodes while your traffic spike lasts thirty seconds.
Kubernetes is a powerful platform. It is not a resilience architecture. It is a container orchestrator that will faithfully orchestrate your containers into exactly the failure mode you configured, which is usually “all eggs in one basket, but with YAML.”
LIE #6: “THE CLOUD PROVIDER HANDLES THAT”
My personal favorite.
The cloud provider handles infrastructure. They keep the lights on, the disks spinning, and the hypervisors patched. They provide Availability Zones, Regions, managed services, and an SLA that promises you service credits if things break.
They do not handle your application’s ability to survive when their infrastructure has a bad day. That is your job. The shared responsibility model is not just a security concept — it is a resilience concept. AWS designs S3 to offer 99.999999999% durability. They do not guarantee that your application will gracefully handle the 30-second S3 API latency spike that precedes an incident.
Every major cloud outage reveals the same pattern: the provider’s infrastructure degrades, and then customers discover which parts of their resilience strategy were real and which parts were assumptions.
THE PUNCHLINE
Here is the thing about all six of these lies: I have believed every single one of them at some point in my career. We all have. Resilience is not about being right from day one. It is about being honest about what you have not tested, what you have assumed, and what would actually happen at 2 AM on a Friday when the one person who knows how everything works is unreachable.
The gap between your architecture diagram and your production reality is where incidents live. Close the gap, and the 2 AM calls get shorter. Ignore it, and you will keep learning these lessons the hard way — one postmortem at a time.
At least the postmortems make great newsletter content.
Get Failover Weekly by email
One essay every two weeks, plus a short note when a new article is published here.