Reaped, Not Broken

A default timeout quietly dropped from five days to 350 seconds, and connections that teams swore were healthy started dying in the dark. The fix is old and boring. The lesson is neither.

This is a composite incident I have now walked through with more than one team, because it keeps arriving in the same shape.

A financial-services customer ran a nightly reconciliation job. Nothing exotic. A worker in a private subnet opened a pooled connection to a database, pulled a large result set, then sat mostly idle for long stretches while it chewed through business logic, occasionally reaching back to that same connection to write results. This had run every night for close to two years. By every dashboard they had, it was boring.

Then they refreshed their compute fleet to a newer instance generation. Same machine image, same code, same security groups, same Amazon Virtual Private Cloud (Amazon VPC). Nothing in their infrastructure-as-code changed in any way they considered meaningful. And within a week, the reconciliation job began failing intermittently — not every night, but often enough to page someone. The symptom was maddening: after a quiet stretch, the application would pull a connection from the pool, try to use it, and hang, eventually surfacing a generic I/O timeout. A retry almost always worked. The pool “recovered.” Nobody could reproduce it on demand.

They spent two days staring at the database. The database was fine. They spent another day suspecting a firewall change. There was none.

The thing that had changed was a number nobody on the team had ever thought about: the default idle timeout for connection tracking on their instances.

The connection was never really open — it was a claim your application kept making

Here is the reframe I want you to carry out of this issue.

When your application holds an idle TCP connection, it feels like you possess a thing — a durable pipe between two endpoints that exists until someone deliberately closes it. That mental model is comfortable and wrong. A TCP connection is not an object that lives in the network. It is a shared fiction, agreed upon by two endpoints and — this is the part we forget — silently refereed by every stateful device on the path between them: security groups, Network Address Translation gateways (NAT gateways), load balancers, firewalls.

An idle connection is not a fact. It is a claim each end keeps making about state the middle has quietly agreed to remember. And memory in the middle is finite, so the middle sets a timer. When the timer expires, the middle forgets. Neither endpoint is told. The next time your application speaks into that connection, it is speaking into a pipe that one or more devices on the path no longer believe exists.

Your connection didn’t fail. It was reaped. And it was reaped precisely because it was healthy enough to sit still.

What connection tracking actually is — and why it has an opinion about your idle sockets

On Amazon Elastic Compute Cloud (Amazon EC2), the referee closest to your instance is security group connection tracking, often shortened to conntrack. It is what lets a security group be stateful: when you allow an inbound flow, conntrack remembers it so the return traffic is automatically permitted without a matching outbound rule. That same machinery runs on the Nitro system alongside VPC Flow Logs and network metering, and — this is the crux — it has finite capacity. Every tracked connection consumes an entry, and each instance type has a ceiling on how many entries it can hold.

To keep that table from filling with ghosts, conntrack ages out idle flows. Each protocol has its own idle timeout. For an established TCP connection, that timeout historically defaulted to 432,000 seconds — five days. Five days is so long that, in practice, no idle connection ever hit it. The table effectively never reaped a live-but-quiet TCP flow, and a generation of us built systems on that unspoken assumption without ever knowing we had made it.

Then the default changed.

The 350-second cliff, and the exact edge it sits on

On sixth-generation Nitro instances — Nitro v6, the newest generation at time of writing — the default established-TCP idle timeout for connection tracking dropped from 432,000 seconds to 350 seconds. Instances on Nitro v5 and earlier retain the old five-day behavior. There is even a specific carve-out: the P6e-GB200 instance type keeps the 432,000-second default. So this is not a fleet-wide flag day. It is a change that arrives quietly, per instance, the moment you happen to launch onto v6 hardware — which is exactly why my reconciliation customer walked into it by simply refreshing their fleet.

Sit with the size of that jump. Five days to under six minutes. Any connection your application holds open but idle for longer than 350 seconds is now, on this hardware, a candidate to be forgotten by the very thing that makes your security group stateful. A pooled database connection between bursts of work. A message-broker consumer waiting on a slow topic. A long-poll to a downstream service. A batch worker thinking between phases. None of these are bugs. All of them are now exposed.

And the reason it presents as an intermittent hang rather than a clean error is worth naming precisely. When conntrack forgets an established flow, it does not send a courtesy reset to either endpoint. Your kernel still believes the socket is ESTABLISHED. Your connection pool still lists it as healthy. The fiction persists on both ends while the middle has moved on. Only when you try to send data does the truth surface — as silence, then a timeout.

It is worth being honest that this is not unique to conntrack, and that is the deeper point. A NAT gateway enforces its own fixed idle timeout of 350 seconds on TCP flows, and it is not configurable. When a NAT gateway times a flow out, it returns a reset packet only when a resource behind it next tries to continue the connection — not proactively. A Network Load Balancer (NLB) defaults its TCP idle timeout to 350 seconds as well, though there you can raise it to as high as 6,000 seconds (the fixed 350 for TLS listeners aside). The number 350 is not a coincidence you should lean on; it is a convergence you should respect. Multiple independent layers on your path have each decided, on their own, that a few minutes of silence is enough to reclaim the memory.

The exhaustion failure mode, which looks nothing like a timeout

There is a second, nastier way this bites, and it is the one that turns a quiet annoyance into an incident.

Conntrack capacity is finite. If idle or half-dead connections accumulate faster than they are reaped — or if you naively “fix” the reaping by pushing your idle timeout back up toward five days without also closing connections properly — you can exhaust the table. When that happens, the instance cannot track new connections, and new connection attempts are dropped. The symptoms are the ones that send teams down the wrong path: connection timeouts, connection-refused errors, and 504s from load balancers, all on an instance whose CPU, memory, and disk look perfectly healthy.

The good news is that this is observable before it becomes an outage, if you are watching. The Nitro network stack exposes two metrics you can read with ethtool: conntrack_allowance_available, the number of tracked connections still available on that instance, and conntrack_allowance_exceeded, the count of packets dropped because you hit the ceiling. If the first is trending toward zero or the second is anything but zero, you are living on borrowed time. One caution worth knowing: certain flows are always tracked regardless of your security group rules — egress-only internet gateways, Global Accelerator accelerators, AWS PrivateLink interface endpoints, Lambda’s Hyperplane elastic network interfaces, and DynamoDB gateway endpoints — and a DynamoDB gateway endpoint connection consumes two conntrack entries, not one. Your accounting needs to reflect that.

Keepalives — the cheapest insurance you are almost certainly not buying

The durable fix is not to fight the reaper. It is to stop being idle. If your connection exchanges a packet more often than the shortest idle timeout on its path, no referee ever has cause to forget it.

That is what TCP keepalives are for, and here is the trap: keepalives are off by default at the socket, and even where an application enables them, the operating system defaults are useless for cloud paths. The Linux kernel’s default keepalive idle time is 7,200 seconds — two hours, a value that traces back to the guidance in RFC 1122 from an era of very different networks. Two hours against a 350-second reaper is not protection; it is a coin flip you lose. Bring the kernel’s keepalive interval comfortably under the tightest timeout on the path — starting a probe at 240 seconds or less is a sane target against a 350-second floor.

In /etc/sysctl.d/, probe well before the 350s reaper:

# start probing after 240s idle
net.ipv4.tcp_keepalive_time = 240
# 60s between probes
net.ipv4.tcp_keepalive_intvl = 60
# give up after 3 unanswered
net.ipv4.tcp_keepalive_probes = 3

Kernel settings are the backstop, not the whole answer. Enable keepalives explicitly in your connection libraries and pools, because a kernel keepalive only applies to a socket that asked for it. And where you genuinely need long-lived idle connections and cannot make them chatty, you have a second lever: the established-TCP idle timeout for connection tracking is itself configurable per Elastic Network Interface (ENI), from 60 seconds up to the old 432,000, across every Nitro generation, via ModifyNetworkInterfaceAttribute. Raise it deliberately, for a workload you understand — not as a reflex to silence the pages, and never without also being sure those connections get closed when the work is done, or you have simply traded a reaping problem for an exhaustion problem.

What this posture does not fix

Be clear about the boundary, because that is where teams get a false sense of safety.

Tuning keepalives and idle timeouts keeps a healthy connection from being reaped for the crime of sitting still. It does nothing else. It will not save a connection whose far end has genuinely gone away — a keepalive probe against a dead peer is how you discover that, not how you prevent it. It does not make your application resilient to a connection that dies mid-request; that is what request-level timeouts, retries with backoff, and idempotency keys are for. It does not protect against conntrack exhaustion caused by a leak in your own pooling logic. And it does nothing for the failure modes above connection state entirely — a slow query, a poisoned cache, a downstream that is up but wrong.

Keepalives buy you exactly one guarantee: that when your connection is quiet, the network’s referees keep believing it exists. Everything else is still your job.

The general lesson — a timeout is a contract, and every layer signs its own

Strip away the Nitro specifics and here is the shape of it. Every stateful element on a path holds an opinion about how long silence is allowed to mean “still alive,” and those opinions are set independently, by different teams, at different layers, often years apart. Your application has one. Your connection pool has another. The security group, the NAT gateway, the load balancer, the far-end server each have their own. Resilience is not achieved by any single one being generous. It is achieved by making them agree — and by arranging that the liveliness signal (a keepalive) fires faster than the strictest silence any of them will tolerate.

The failure in my reconciliation story was never really a timeout. It was a disagreement. The application believed a connection was open for as long as it chose not to close it. The hardware underneath it had quietly signed a different contract. Nobody had reconciled the two, because for two years the two happened to overlap, and overlap is not agreement — it is luck that has not run out yet.

The instinct after an incident like this is to find the one number that changed and change it back. Resist it. The durable move is to write the contract down: know the tightest idle timeout on every path your long-lived connections traverse, set a keepalive that beats it with margin, and monitor the capacity metrics so exhaustion announces itself as a graph trending down rather than a pager going off. Defaults are decisions someone else made for their median case, on their schedule, and they will change again. A contract you understand survives the next default change. A number you copied does not.