awsRoot cause
DynamoDB manages the DNS records for its regional endpoints with automation - a DNS Planner that
computes the desired state and redundant DNS Enactors that apply it. On October 19–20, an unusual
timing interaction between two Enactors produced a classic race: a slow Enactor applied a stale
plan, a cleanup process then deleted the plan the other Enactor had just applied, and the system
ended up removing every IP address from dynamodb.us-east-1.amazonaws.com. The automation
could not self-repair from that state; humans had to intervene.
Two properties made this catastrophic rather than annoying. First, DNS sits in front of everything: an empty record doesn't degrade a service, it makes it unreachable. Second, DynamoDB in us-east-1 is not just a customer database - it is a dependency of AWS's own control planes. When it disappeared, services that customers think of as independent (EC2 launches, Lambda, IAM propagation, NLB health checking) discovered their shared foundation in public.
How the cascade unfolded
The incident is a textbook example of how recovery can hurt more than failure. Restoring the DNS record took about three hours. The next twelve were spent fighting physics:
- Retry storms. Every client that failed for three hours retried at once, hitting a service that was warming back up.
- Congestive collapse in EC2's launch path. The internal DropletWorkflow Manager, which leases the physical hosts EC2 launches onto, had lost its DynamoDB-backed state. As leases timed out en masse, re-establishing them queued behind itself, and new instance launches failed region-wide long after DynamoDB was healthy. Engineers ultimately had to throttle launches and restart components to break the feedback loop.
- Health-check flapping. Network Load Balancer health checks began failing healthy targets that were merely slow, removing real capacity at the worst moment - mitigated only when AWS disabled the automatic failover behavior mid-incident.
If your architecture retried aggressively that day, you were part of the problem - and your bill was part of the evidence.
Business impact
This was, by breadth, the most disruptive cloud incident since 2021: consumer apps, banks, airlines, and government services went dark or degraded for most of a business day, with public estimates of aggregate damage running into the billions. For individual AWS customers the impact concentrated in three places: anything stateful on DynamoDB in us-east-1, anything that needed to scale during the window (no new EC2 capacity meant no autoscaling), and anything fronted by an NLB whose healthy targets were being flapped out of service.
Prevention and lessons
The honest lesson is not "multi-cloud" - it is that us-east-1 is a single point of failure you chose. Concretely:
- Treat region as your blast-radius unit. Critical workloads need a tested path to run somewhere other than us-east-1. Our Multi-AZ vs Multi-Region guide covers when the cost is justified - Multi-AZ did nothing for you in this event.
- Bound your retries. Exponential backoff with jitter and circuit breakers is the difference between recovering in minutes and participating in a region-wide retry storm.
- Know your hidden DynamoDB dependencies. Many teams discovered mid-incident that "we don't use DynamoDB" was false - via Cognito, IAM, feature-flag vendors, or their own platform teams. Dependency mapping is DR work, not documentation work (the DR checklist has a concrete exercise for this).
- Watch the DynamoDB status page - not Twitter. During the event, status pages lagged reality by minutes; social media lagged by an hour and was mostly wrong about the cause.