Failover Weekly No. 2
When the Network Breaks
What Three Major Outages Taught Us About Resilient Architecture
Between July and October 2025, three of the world’s most sophisticated infrastructure providers — AWS, Microsoft Azure, and Cloudflare — experienced significant outages. None were caused by hardware failures or cyberattacks. All three were triggered by configuration-driven cascading failures in networking and infrastructure control systems.
A DNS automation race condition. A corrupt configuration that overwrote its own rollback path. A dormant BGP misconfiguration that lay invisible for 38 days before making a global DNS resolver vanish from the internet.
These weren’t edge cases. They were well-understood failure modes operating at unprecedented scale. And the architectural lessons they expose apply to every organization running production workloads in the cloud.
THE INCIDENTS
AWS US-EAST-1 (October 19–20, 2025): A latent race condition in DynamoDB’s DNS automation published an empty DNS record for the DynamoDB regional endpoint. EC2’s host lease manager (DWFM) depended on DynamoDB, so the failure cascaded into EC2 launch failures, network state propagation delays, and NLB health check failures. A single empty DNS record became a 14+ hour cascading service disruption. The system entered congestive collapse — leases timed out faster than they could be renewed, turning a DNS issue into a compute recovery crisis.
Azure Front Door (October 29, 2025): Nine days later, a configuration change produced inconsistent metadata during rollout. It passed all staged health check validations because the failure mode was asynchronous — the corruption only manifested after propagation completed. Worse: the corrupted configuration had propagated into the ‘last known good’ snapshot, forcing engineers to manually sanitize the rollback configuration before it could be deployed — adding critical delay during an active global outage. Azure Portal, Microsoft 365, and Microsoft Entra ID (Azure AD) were hit by an outage lasting over eight hours.
Cloudflare 1.1.1.1 (July 14, 2025): A configuration change on June 6 inadvertently linked the 1.1.1.1 resolver’s IP prefixes to a non-production service topology. The misconfiguration sat dormant for 38 days. On July 14, an unrelated change triggered a global configuration refresh, causing BGP withdrawal of those prefixes from every production data center worldwide. The resolver vanished from the internet for 62 minutes.
THE COMMON DNA: THREE ANTI-PATTERNS
Despite operating at different layers and involving different technologies, these incidents share the same architectural DNA.
Control plane / data plane coupling. In all three cases, the data plane’s ability to route traffic depended on the control plane’s ability to manage configuration correctly. When the control plane produced incorrect state — through a race condition, corrupt propagation, or dormant misconfiguration — the data plane had no independent mechanism to continue operating. This is the most dangerous coupling in distributed systems. The control plane changes state; the data plane moves packets. When the data plane requires live control-plane interaction to forward traffic, you have a single point of failure with extra steps.
Global blast radius from configuration changes. A single configuration change propagated globally in two of the three incidents — either atomically (Cloudflare’s BGP withdrawal hit every data center simultaneously) or through staged rollout that lacked meaningful health gating (Azure’s corrupted LKG). None contained the blast radius to a subset of infrastructure before expanding.
Monitoring that depends on the system being monitored. In both the AWS and Azure incidents, the monitoring and health-check systems that should have detected the failure were themselves impacted by it. Azure’s health checks passed because the failure was asynchronous. AWS’s NLB health check subsystem was degraded by the same cascade, failing targets that were healthy. If your alerting pipeline runs on the same infrastructure it’s monitoring, it will fail silently at precisely the moment you need it most.
BUILDING THE DEFENSIVE ARCHITECTURE
These anti-patterns have concrete countermeasures — documented, available, and implementable today.
Decouple your data plane from your control plane. Amazon Application Recovery Controller (ARC) provides data-plane-level routing controls that operate independently of the Route 53 control plane. Pre-provision routing control states so that in a control plane degradation scenario, you can steer traffic at the data plane level. For DNS specifically, implement multi-provider DNS with at least two independent authorities and multi-signer DNSSEC (RFC 8901) to maintain cryptographic integrity across providers without sharing private keys.
Contain blast radius with cell-based architecture. Design infrastructure as a collection of independent cells — each with its own load balancer, compute, database replica, and cache layer. Deploy configuration changes cell by cell, validating health metrics before proceeding. Each cell gets its own CI/CD pipeline. A deployment to Cell A shares no mutable state with Cell B. The bulkhead pattern from naval architecture is the mental model: watertight compartments that prevent a hull breach from sinking the entire ship.
Monitor from outside your blast radius. Deploy CloudWatch Synthetics canaries in separate AWS accounts for cross-account monitoring. Use Route 53 health checks from multiple AWS Regions — these operate independently of your regional infrastructure. For BGP, subscribe to stream monitoring services and alert on unexpected route withdrawals for your prefixes. The Cloudflare incident proved that dormant misconfigurations can withdraw your prefixes without any external hijack — your monitoring must cover both external threats and internal misconfigurations.
THE UNCOMFORTABLE TRUTH
These outages were not caused by unprecedented technical challenges. They were caused by well-understood failure modes applied at unprecedented scale. At hyperscale, cascading failures are statistically inevitable. The question is whether your architecture can contain the blast radius when they occur.
Cell-based architectures, control plane and data plane decoupling, multi-provider DNS, and out-of-band monitoring are not premium features — they are baseline requirements for any system where downtime costs more than the engineering effort to prevent it.
Recovery in resilient systems should feel boring. If your incident response requires heroics, your architecture has already failed.
For the full L400 deep-dive with architecture diagrams and detailed RCA analysis, read the complete article on cloudpunk.blog:
These patterns don’t fix themselves.
Get Failover Weekly by email
One essay every two weeks, plus a short note when a new article is published here.