aws
Root cause
Elastic Load Balancing keeps state data that describes each load balancer: how it is configured, how many nodes it should have, and where traffic should go. A running load balancer holds its configuration in memory, so it can keep serving traffic even if that stored state is briefly unavailable. That resilience is normally a good thing. On December 24, 2012, it became a trap.
An AWS developer running a maintenance process accidentally deleted a chunk of ELB state data in us-east-1. Nothing broke immediately, because the load balancers already running simply continued on their last-known configuration. The damage only surfaced when a load balancer needed to consult that state: when it scaled up under holiday load, when a customer created a new one, or when Auto Scaling modified it. Those operations produced misconfigured load balancers that dropped or misrouted traffic. So the outage did not arrive with a bang. It crept in over the evening as more and more load balancers reached for state that was no longer there, which is exactly why it took a while to diagnose and why the impact kept growing.
This slow-burn shape is what makes a state-deletion outage so much nastier than an obvious crash. A crashed component pages someone within seconds. A silent deletion of configuration data sits invisible until the system happens to need what was deleted, and by then the connection between cause (a maintenance action hours earlier) and symptom (load balancers failing now) is easy to miss. Recovery had its own subtlety: AWS could not just restore the state and walk away, because it also had to disable the workflows capable of re-triggering the deletion and then carefully re-apply correct configuration to every affected load balancer without knocking healthy ones offline in the process.
Business impact
The signature victim was Netflix, which streamed to a large share of North America from us-east-1 and leaned on ELB to spread that traffic. On Christmas Eve, one of the highest-viewership evenings of the year, Netflix went dark for millions of households. The timing could not have been worse: this was peak seasonal demand, the moment a streaming service most wants to be flawless, and the outage landed during the window when families had gathered specifically to watch. The public embarrassment became a turning point. Netflix accelerated its move toward multi-region architecture and doubled down on Chaos Engineering, deliberately injecting failure so an outage in one region could not take the whole service down. Its Simian Army tooling, and the wider industry practice of continuously testing failure in production, owes a real debt to this evening.
For everyone else, the lesson was that a control-plane mistake, not a hardware failure or a traffic spike, can be the thing that ruins your holiday. The trigger was not a natural disaster or a capacity crunch. It was a person running a normal maintenance job that had a bit more reach than it should have had, which is a category of risk you cannot capacity-plan your way out of. It echoes the operator-plus-powerful-tool pattern of the 2017 S3 outage: an authorized human, a capable tool, and a system that trusted the tool a little too much.
Prevention and lessons
- Protect control-plane state like production data. The trigger was a maintenance process that could reach and delete critical state directly. Lock down who and what can touch configuration stores, and require the same review you would demand before a destructive data operation.
- Beware failures that hide behind cached state. Systems that keep running on stale configuration mask the real problem until a scale or change event exposes it. Alert on the health of your state store itself, not only on request errors, so a silent deletion is caught early.
- Go multi-region for anything that cannot miss a holiday. Netflix's response was the correct one: assume a region can fail entirely. Multi-AZ vs Multi-Region walks through when the extra cost is justified, and revenue-critical evening traffic usually justifies it. Pair it with the DR checklist.
- Document the window and claim it. Hours of degraded load balancers can support a credit if you have the evidence. Estimate exposure on the SLA credit calculator, follow the credit playbook, and let our sponsor Next Signal track incident windows so you are not reconstructing a Christmas Eve timeline by hand.