aws
Root cause
The 2011 outage started with a network change that should have been routine. Engineers upgrading capacity in one us-east-1 Availability Zone needed to shift traffic off part of the primary EBS network. The correct move was to route it onto the redundant primary network. Instead, the traffic went to a low-capacity secondary network that was never meant to carry that volume. Bandwidth collapsed, and EBS nodes suddenly could not reach the replicas that keep each volume mirrored.
Here is where a safety feature became the failure. Every EBS node that loses its mirror is designed to find free space and rebuild a fresh copy, protecting your data. But when the network fault made a huge fraction of nodes lose their mirrors at the same moment, they all began hunting for space simultaneously. That stampede consumed the zone's spare capacity, so none of them could finish, and the ones that could not finish kept retrying. The result was a self-reinforcing re-mirroring storm.
Worse, the EBS control-plane API spanned the whole region, so as it backed up under the flood of requests, even healthy Availability Zones struggled to create or attach volumes. This is the detail that turned a single-zone hardware-network fault into a region-wide event. Customers who had diligently spread their workloads across multiple zones still found that the operations they needed to fail over, such as creating replacement volumes or launching new instances, ran through the same overloaded regional API. The blast radius, in other words, was not defined by where the fault started but by the reach of the control plane sitting above it. AWS also had to restore the health of that control plane before it could safely help individual volumes, which is a large part of why recovery stretched across days rather than hours.
Business impact
In 2011, us-east-1 was where a large share of the young web lived, and many of the era's marquee startups ran single-zone. Reddit, Quora, Foursquare, Heroku, and many others went dark or ran degraded for the better part of a day or more. For businesses that had bet everything on "the cloud just works," it was a brutal introduction to the fine print: availability is a property you have to design for, not a guarantee you can buy. The startups that recovered fastest were the handful that had already built to survive the loss of a single zone, and their competitors spent the following quarter doing the same. Some never fully recovered their reputations, having promised customers an "always on" service that visibly was not.
The event became the industry's first great lesson that "the cloud" is not a single magical resource: it is regions and zones with real, physical failure domains, and a single-zone design inherits a single zone's fate. It also exposed the danger of a region-wide control plane sitting above zone-isolated data, a theme that recurs in the 2015 DynamoDB event and the 2020 Kinesis outage. More than a decade of AWS resilience engineering, from cell-based architectures to zone-independent control planes, traces directly back to the post-mortem AWS published after this outage, which set the template for the detailed, blameless incident write-ups the whole industry now expects.
Prevention and lessons
- Design across Availability Zones, not just within one. Customers who ran multi-AZ with the ability to fail over rode this out. Multi-AZ vs Multi-Region quantifies when the extra spend pays off, and it usually does for anything revenue-bearing.
- Make retries polite. A re-mirroring storm is a retry storm with a fancy name. Any system that reacts to failure by immediately retrying at full force can turn a small fault into a large one. Use exponential backoff, jitter, and concurrency caps everywhere clients retry.
- Do not let a shared control plane erase your zone isolation. Your data can be perfectly isolated per zone and still be unreachable if the API that manages it is regional and overloaded. Ask where your provider's and your own control planes actually live.
- Take the credit and keep the evidence. AWS auto-issued a 10-day credit here, but most outages require you to file. Check exposure on the SLA credit calculator, follow the credit playbook, and let our sponsor Next Signal watch for breaches so you are not reconstructing timelines by hand.