Skip to main content
awsdown
criticalS3us-east-1

The 2017 S3 Outage: A Typo That Broke the Web - Lessons That Still Apply

AWSDown Research · Incident analysisPublished March 6, 2017Updated April 10, 20267 min read

Timeline

  1. Trigger

    During debugging of an S3 billing-system slowdown, an operator runs a capacity-removal playbook command with a mistyped parameter, taking down a much larger set of index-subsystem servers than intended.

  2. Impact

    The S3 index and placement subsystems in us-east-1 lose quorum and require full restarts. GET, LIST, PUT, and DELETE requests fail region-wide; dependent AWS services (EBS, Lambda, new instance launches) degrade.

  3. Communication failure

    The AWS status dashboard cannot display the outage - its status icons are hosted on the S3 region that is down. AWS resorts to tweeting updates.

  4. Monitoring

    The index subsystem completes its restart and GET/LIST error rates recover.

  5. Resolved

    The placement subsystem finishes recovery; PUT/DELETE return to normal and dependent services drain their backlogs.

Root cause

The command itself was routine - S3 operators were debugging slow billing and intended to remove a small number of servers from one subsystem. One mistyped parameter widened the scope to a large set of servers backing two foundational subsystems: the index (metadata and location of every object in the region) and placement (where new objects go). Below a capacity floor, both required a full restart-and-verify cycle. S3 in us-east-1 had grown for years without those subsystems ever being fully restarted, so the restart took hours - the outage was less about the typo and more about an untested recovery path at massive scale.

Business impact

In 2017, "S3 us-east-1" was effectively "the internet's hard drive": Slack file uploads, Trello, Quora, Medium, IFTTT, countless images and JS bundles, and - infamously - AWS's own status dashboard icons all broke together. The event became the industry's standard case study in hidden single-region dependencies, and in why "the cloud is down" headlines are usually "one region of one provider is down."

Prevention and lessons

  1. Rate-limit destructive operations. AWS's own remediation - tooling that removes capacity slowly and refuses to cross safety floors - is directly portable: your delete/scale-down scripts should have the same guardrails.
  2. Exercise your cold-start path. The four hours were mostly restart time. Any system that has never been fully restarted since it was small has an unknown RTO. Game-day it - the DR checklist schedules this.
  3. Host your status page elsewhere. If your customer communication depends on the infrastructure that just failed, you have no customer communication. (It's why this site exists independently - see how status is measured.)
  4. Cross-region replication earns its bill. Customers with S3 CRR to another region served reads through the entire event. Multi-AZ vs Multi-Region quantifies when that trade is worth it.

SLA credit eligibility

Yes - four hours of regional S3 downtime breached the 99.9% monthly target. Four fully-down hours ≈ 99.45% monthly uptime, the 10% credit tier under the S3 SLA. Many affected customers never claimed it.

Work out what your bill was owed with the independent AWS SLA credit calculator, or automate breach detection with Next Signal (sponsor). For the full claim process, see the AWS credit-recovery playbook.

Sources: AWS official summary of the S3 disruption, Amazon S3 Service Level Agreement, CloudDowntime - cloud reliability research

Questions about this outage

What caused the 2017 S3 outage?

A member of the S3 team, following an established playbook to remove a small amount of billing-subsystem capacity, entered one command parameter incorrectly. The command removed a large set of servers supporting S3’s index and placement subsystems in us-east-1, which then required a full restart lasting roughly four hours.

How long was S3 down in the 2017 outage?

About four hours of major impact in us-east-1 (17:37–21:54 UTC on February 28, 2017), with GET/LIST recovering ~40 minutes before PUT/DELETE.

Why did the AWS status page show everything green during the outage?

The status dashboard’s health icons were stored on S3 in the affected region. AWS could not update its own status page and communicated via Twitter until they restored the capability - the canonical argument for hosting your status page off your own infrastructure.

Could the 2017 S3 outage happen again?

This exact failure is unlikely - AWS now limits how fast capacity can be removed and partitions the index subsystem into cells. But the underlying pattern (an authorized operator plus a powerful tool plus a typo) remains the root cause of major outages industry-wide, including several since.

Next time, hear it from us first

Instant alerts when AWS incidents are detected - services and regions included.

Get outage alerts

More post-mortems

Lambda

AWS Lambda Outage in us-east-1 (June 13, 2023): What Happened

A degradation in the subsystem that manages Lambda’s execution capacity caused elevated invocation error rates in us-east-1 for about three hours. Because Lambda sits inside so many AWS features - including STS and parts of the console - the blast radius reached far beyond “serverless” workloads.