Three Cats, Zero Downtime

What my household cluster taught me about high availability

It’s Friday. You’ve survived another week of on-call, and somewhere a dashboard is quietly turning yellow in protest. Let’s not look at it.

Instead, let me introduce you to the most resilient distributed system I’ve ever operated. It has three nodes, runs 24/7, requires no patching, and has never once paged me at a reasonable hour. I am referring, of course, to my cats.

After years of careful observation — read: being woken at 4am — I’ve concluded they understand resilience engineering better than most architecture review boards. Here’s the curriculum.

1. No single point of failure. There are three of them. At any given moment one is asleep, one is plotting, and one is actively violating a service-level objective. Redundancy isn’t a feature here, it’s a lifestyle.

2. Health checks, every 30 seconds. They walk across my keyboard at fixed intervals to confirm I’m still alive and capable of dispensing food. If I fail to respond, they escalate. Loudly.

3. Failover is instant. Primary food source (bowl) returns empty? Traffic reroutes to the secondary (me) before the cat has finished processing the error. Sub-second RTO. Your DR plan should be so lucky.

4. Chaos engineering is a way of life. I have a one-cat Chaos Monkey named Kronos who periodically unplugs things to verify the system recovers. He does not file a change request. Netflix would hire him on the spot.

5. The thundering herd problem is real. Open one tin of food and watch four independent clients converge on a single resource simultaneously, with zero backoff and no rate limiting. I have witnessed cascading failure. It started with a can opener.

6. Split-brain happens. Two cats — Nemesi and Ares — one sunbeam. Both are absolutely convinced they are the primary. There is no quorum, no leader election, only conflict and, eventually, fur.

7. Retries without exponential backoff are a cautionary tale. When a cat wants a door opened, it does not back off gracefully. It retries at full rate, immediately, forever. Engineers, observe closely what not to do.

8. Graceful degradation under load. Even at 3am, with the human asleep and staffing at a minimum, core services remain available. Specifically: knocking small objects off high surfaces. The blast radius is the floor.

The lesson buried in all of this — and there is one, because this is still a technical newsletter — is that resilience was never about a single clever component. It’s about assuming everything fails, building in redundancy, testing your failure modes on purpose, and designing so that when (not if) a node goes down, the system keeps serving. The cats just internalized this long before we gave it acronyms.

And speaking of creatures who grasp resilience instinctively — I went and wrote a whole book about the human-engineered version.

Cloud Networking and Resilience (Apress) officially landed in paperback yesterday, with the ebook already out in the wild. It’s the BGP, DNS, failover, and multi-region material that keeps real systems standing — minus the fur. The part I’m proudest of: every penny of my royalties goes to charities helping cats in distress. So you’ll level up your resilience architecture and help a cat having a worse week than your primary region. The book is dedicated to one cat in particular, who taught me most of this, his name is Ade.

Buy the book. Help a cat. Tell your CAB about Kronos.

Have a resilient weekend — may your failovers be boring and your sunbeams undisputed.

— Cris