S3 elevated error rates in us-east-1 (index subsystem restart)

Impact

A mistyped capacity-removal command took down S3’s index and placement subsystems in us-east-1 for roughly four hours, breaking large parts of the web.

Affected services (1)

Affected regions

Timeline

1. ResolvedFeb 28, 2017, 9:54 PM UTC

   Placement subsystem recovered; S3 operating normally.

2. MonitoringFeb 28, 2017, 9:18 PM UTC

   Index subsystem restarted; GET/LIST recovering.

3. IdentifiedFeb 28, 2017, 6:30 PM UTC

   Index subsystem capacity below operating floor after an operator command; full restart required.

4. InvestigatingFeb 28, 2017, 5:45 PM UTC

   High error rates for S3 requests in us-east-1.

Related reading