aws
Root cause
Kinesis Data Streams is built on a fleet of front-end servers, and each one keeps a thread open to every other server in the fleet so it can share and cache a map of which stream shards live where. That design is fine until the fleet gets large enough that the number of threads on a single server brushes up against the operating system's hard limit. On November 25, 2020, AWS added capacity to the us-east-1 front-end fleet. The extra servers pushed the existing ones past that thread ceiling, so they could no longer build a complete shard-map. Requests started failing, and because a front-end server needs a valid shard-map to route anything, the fleet degraded as a whole rather than shedding one broken node.
The insidious part was recovery. AWS could not simply restart everything: bringing servers back too quickly re-created the same thread pressure, because every new server immediately tried to open connections to every other server. The fix was to shrink the fleet, let the survivors rebuild a clean, complete shard-map, and then add servers back a handful at a time while watching the thread count on each one. That is why an outage triggered by a capacity addition took most of a business day to clear. It is worth sitting with that irony: the trigger was AWS trying to make Kinesis bigger and better, and the failure mode was baked into a design decision (all-to-all threading) that had worked fine for years right up until the fleet crossed an invisible line. Most serious outages look like this in hindsight. They are not a single dumb mistake but a healthy change meeting a long-dormant limit.
Business impact
Very few customers use Kinesis directly, yet nearly everyone in us-east-1 felt this event, because AWS runs its own control plane on Kinesis. CloudWatch publishes metric and log data through it, so dashboards went blank and alarms stopped firing at the exact moment operators needed them most. Cognito leaned on it for parts of authentication, so sign-in flows failed for apps that had nothing to do with streaming data. The AWS console, EventBridge, and portions of Lambda and Auto Scaling degraded for the same reason. Teams that had no idea they depended on Kinesis discovered that they did, all at once, mid-morning on the day before Thanksgiving.
The knock-on effects reached well beyond AWS proper. Because CloudWatch alarms feed Auto Scaling and countless third-party paging and observability tools, some customers could neither see their own systems clearly nor scale them in response to load. Popular consumer products that had no obvious tie to streaming, from smart-home apps to video services, reported errors during the window, simply because a link in their chain published telemetry or authenticated users through an affected AWS feature. The financial cost of a day like this is rarely the raw compute credits. It is the support tickets, the missed transactions, and the engineering hours spent chasing a problem that turned out to originate three vendors deep in a dependency you never chose. This is the same hidden-dependency theme as the 2021 us-east-1 outage and the 2025 DynamoDB DNS event: a single internal service sits under features that look unrelated, and the concentration of AWS control planes in us-east-1 makes that region the industry's most consequential single point of failure.
Prevention and lessons
- Blast-radius review routine capacity changes. The trigger here was a normal, approved scaling action, not a mistake. Treat additions to shared internal-style services (your own message buses, metadata stores, service discovery) with the same care as a deploy, and stage them.
- Do not let monitoring depend on the thing being monitored. CloudWatch failing during the outage blinded responders. Keep an out-of-band signal, whether a third-party monitor or a simple external heartbeat, so you can see an incident even when your primary telemetry is down. The DR checklist covers this.
- Design for slow, safe recovery. The long tail was deliberate re-scaling, not indecision. Systems that can only recover by carefully adding capacity back have a long RTO by nature, so measure and communicate it before you need it.
- Claim what you are owed. Eight regional hours is a clear SLA breach. Run the numbers on the SLA credit calculator and follow the credit playbook to file. Automated evidence collection for events like this is exactly what our sponsor Next Signal builds.