Skip to main content
awsdown
criticalEC2us-east-1

The December 2021 AWS us-east-1 Outage: How One Network Event Took Down Half the Internet

AWSDown Research · Incident analysisPublished December 12, 2021Updated May 2, 20268 min read

Timeline

  1. Impact begins

    An automated scaling activity triggers a surge of connection retries on the internal network that hosts AWS control-plane services, congesting the devices that bridge it to the main network.

  2. Investigating

    Consoles, EC2 APIs, and dependent services degrade. AWS’s own monitoring depends on the congested network, leaving operators partially blind and slowing diagnosis.

  3. Identified

    Engineers identify the internal-network congestion and begin throttling traffic and disabling the responsible scaling activity while restoring device health.

  4. Monitoring

    Network device congestion substantially recovers; consoles and APIs begin returning to normal, with a long tail of backlog processing in queue-based services.

  5. Resolved

    AWS reports services operating normally. Backlogged deliveries and delayed events continue draining into the evening.

Root cause

AWS runs (at least) two networks: the main network your workloads live on, and an internal network hosting foundational control-plane services, DNS, and - crucially - AWS's own monitoring. At 7:30 AM PST, an automated activity to scale one of the internal services triggered unexpected behavior in a large number of clients, which began aggressively retrying connections. The wave of retries overwhelmed the devices connecting the two networks, and previously untested queueing behavior on those devices made latency worse the more the network struggled.

The failure loop was self-sustaining: congestion caused failures, failures caused retries, retries caused congestion. And because AWS's operators observe AWS through systems on the affected network, the people fixing the outage were doing so with degraded dashboards - one reason public status updates lagged so badly that day.

Business impact

The data plane mostly held: already-running EC2 instances, load balancers, and S3 kept serving. What died was the ability to change anything - consoles, APIs, autoscaling, container launches, and the queue/event fabric (SQS, EventBridge) that glues serverless architectures together. Downstream, the outage reached consumer names (Ring, Roomba, Disney+, delivery logistics at Amazon itself) and thousands of SaaS products. If your incident response required launching new instances or rotating credentials via STS, you were stuck watching.

Prevention and lessons

  1. Static stability is the goal. Architectures that keep working when the control plane is unavailable - pre-provisioned capacity, no launch-time dependencies, cached credentials - rode this event out. Our DR checklist includes a control-plane-outage drill for exactly this failure class.
  2. Don't put your observability inside the blast radius. AWS's monitoring lived on the failing network; yours probably lives in the failing region. External synthetic checks (or a second region) are cheap insurance - this is also why our status page probes from outside.
  3. Retry like a good citizen. Unbounded client retries turned a scaling hiccup into a seven-hour event. Backoff with jitter and retry budgets protect both you and the shared infrastructure.
  4. Us-east-1 concentration risk, again. The event impaired global features homed in us-east-1 (STS global endpoint, console sign-in). Regionalize your dependencies where AWS offers the choice - see Multi-AZ vs Multi-Region.

SLA credit eligibility

Yes, for many control-plane-dependent workloads in us-east-1. Seven hours of impact in a month puts affected services below 99.0% only if fully down; most customers landed in the 10% credit tier for services like EC2 where API availability is covered. Claims required per-service evidence.

Work out what your bill was owed with the independent AWS SLA credit calculator, or automate breach detection with Next Signal (sponsor). For the full claim process, see the AWS credit-recovery playbook.

Sources: AWS official event summary (December 7, 2021), AWS Health Dashboard, CloudDowntime - cloud reliability research

Questions about this outage

What caused the AWS outage on December 7, 2021?

An automated scaling activity caused a surge of activity on AWS’s internal network - the one hosting its own control-plane and monitoring services. Congestion on the devices connecting that network to the main network degraded everything that crossed the boundary, including the consoles, EC2 APIs, and AWS’s own observability.

How long did the December 2021 AWS outage last?

Roughly seven hours of significant impact in us-east-1 (about 15:30 to 22:22 UTC), with a longer tail while queue backlogs drained.

Why couldn’t AWS see its own outage?

The monitoring systems AWS operators use ran on the same congested internal network that was failing. That circular dependency slowed both diagnosis and public communication - a lesson AWS later addressed and one worth applying to your own observability stack.

What broke for customers during the December 2021 event?

The management console, EC2/STS APIs, container launches (Fargate, ECS, EKS), queue-driven pipelines on SQS, EventBridge delivery, and consumer services from Ring to Disney+. Running EC2 instances largely kept running - the control plane, not the data plane, was down.

Next time, hear it from us first

Instant alerts when AWS incidents are detected - services and regions included.

Get outage alerts

More post-mortems

Lambda

AWS Lambda Outage in us-east-1 (June 13, 2023): What Happened

A degradation in the subsystem that manages Lambda’s execution capacity caused elevated invocation error rates in us-east-1 for about three hours. Because Lambda sits inside so many AWS features - including STS and parts of the console - the blast radius reached far beyond “serverless” workloads.

S3

The 2017 S3 Outage: A Typo That Broke the Web - Lessons That Still Apply

A playbook command executed with a mistyped parameter removed far more S3 index capacity than intended in us-east-1. The subsystems needed a full restart - something S3 hadn’t done in years at that scale - and for ~4 hours, a huge share of the web (including AWS’s own status dashboard) broke.