The 2015 DynamoDB Outage: A Metadata Service Under Its Own Load

Root cause

DynamoDB spreads your tables across many storage servers, and every storage server has to know which partitions it owns. It learns that from an internal metadata service, and it periodically re-confirms its membership. Normally this is quiet background chatter. On September 20, 2015, a short network disruption in us-east-1 knocked a batch of storage servers out of membership at the same instant, and they all turned to the metadata service to get their assignments back.

The reason this became an outage rather than a blip is a change that had landed quietly beforehand. Global Secondary Indexes had grown table membership data, so each membership request now carried more information and took longer to process. Under the sudden surge, the metadata service could not answer everyone before their timeouts expired. A storage server that cannot confirm its membership takes itself out of service to avoid serving stale data, which is the safe choice individually, but collectively it meant more servers dropping out, retrying, and piling more load onto the exact service that was already saturated. It was a retry storm aimed at a metadata bottleneck.

The dangerous property here is that the trigger and the amplifier were separate events that only became lethal together. The network blip alone would have been forgettable. The larger membership requests alone would have been fine on a normal day. It took the two coinciding, a synchronized re-request the moment those requests had become slow and heavy, to tip the metadata service past the point where it could catch up. Breaking that loop required AWS to pause metadata processing entirely, which is a drastic step, so that the flood of retries would stop and the service could work through a backlog on its own terms before storage servers were allowed back.

Business impact

DynamoDB by 2015 was load-bearing not just for customers but for AWS itself. Parts of EC2 Auto Scaling, SQS, and the console stored state in DynamoDB, so a DynamoDB problem became an EC2 and SQS problem within the region. Customer apps that treated DynamoDB as their always-available primary store saw write and read errors for hours, and many discovered they had no fallback because they had trusted the "fully managed, highly available" label as if it meant "cannot fail."

The customer stories from that day are a catalog of missing degradation paths. Applications that returned a hard error to end users whenever a DynamoDB write failed had no graceful mode: every failed call became a failed user action. Teams whose Auto Scaling depended on the same region could not add capacity to absorb the retries their own clients were generating, so they were squeezed from both sides. The lesson that stuck was not "DynamoDB is unreliable," because on any ordinary day it is extraordinarily reliable. The lesson was that even an extraordinarily reliable dependency will have a bad hour eventually, and an application that has never decided what to do during that hour will decide badly in real time. The event is a direct ancestor of the 2020 Kinesis outage and the 2025 DynamoDB DNS event: a shared internal service, a synchronized surge, and a cascade into features that looked unrelated.

Prevention and lessons

1. Assume your managed database can have a bad hour. Even a five-nines service is not five-nines every month. Decide in advance how your app behaves when the primary store returns errors: shed load, serve stale reads, or queue writes for later.
2. Cap and jitter your retries. The failure amplified itself through synchronized re-requests. Any client that retries a failing dependency must use backoff, jitter, and a concurrency limit, or it becomes part of the outage. See the DR checklist.
3. Watch for scale creep in shared services. The Global Secondary Index change was healthy on its own, but it quietly raised the cost of a routine operation until a surge could no longer be absorbed. Track the size and latency of your hot-path metadata calls over time.
4. Turn five hours of errors into a claim. Elevated-error outages are claimable when you can show sustained failure rates. Model it on the SLA credit calculator, follow the credit playbook, and let our sponsor Next Signal capture the error-rate evidence automatically.