aws
Your architecture diagram shows a resilient system. The only way to know whether it actually is one is to break it on purpose, in a controlled way, and watch. That is chaos engineering: not random destruction, but disciplined experiments that turn "we think this fails over" into "we watched it fail over." On AWS, the tooling to do this safely is built in, and the payoff is finding the weak spots on your schedule instead of the provider's.
Why deliberately break things
Every incident in our outage archive has the same shape in hindsight: a dependency nobody had tested, an assumption that held right up until it did not. The October 2025 DynamoDB DNS failure took down teams that did not know they depended on DynamoDB through a third party. Chaos experiments exist to surface exactly that class of surprise before it surprises you at 3 a.m.
The mental shift is from "does it work?" to "what happens when this part fails?" You are not testing the happy path. You are testing the failure path, the one that only runs during incidents and therefore never gets exercised any other way.
The experiment loop
A chaos experiment is a small scientific method, run in five moves:
- Define steady state. Pick metrics that describe normal: p99 latency, error rate, checkouts per minute. Without a baseline you cannot tell whether an experiment broke anything.
- Form a hypothesis. State what you expect to stay true under the fault, for example: "if one AZ is impaired, the app keeps serving with under 1 percent errors and recovers within two minutes."
- Inject the smallest meaningful fault. Terminate one instance. Add latency to one dependency. Start small and low-blast-radius. The goal is learning, not spectacle.
- Observe and compare. Run it with the team watching dashboards. Did steady state hold? What recovered on its own? Where did a human have to step in?
- Fix and expand. File each weakness as owned engineering work, fix it, re-run to confirm, then widen the blast radius or pick a harder fault.
Using AWS Fault Injection Service
You do not have to build the tooling. AWS Fault Injection Service (FIS) runs fault-injection experiments as a managed service. It can:
- Terminate or stop EC2 instances and drain ECS/EKS tasks
- Stress CPU, memory, and disk on targets
- Inject network latency, packet loss, and blackhole traffic to endpoints
- Throttle API calls and simulate an Availability Zone impairment
The feature that makes it safe is stop conditions. You attach CloudWatch alarms to an experiment, and if steady state breaks past a threshold you set in advance, FIS aborts the run automatically. Combined with a tightly scoped target set (the blast radius), this lets you experiment without betting the whole system on your own reflexes.
| Fault to inject | What it tests | Real incident it mirrors |
|---|---|---|
| Terminate one instance | Auto Scaling replacement, LB health checks | Routine host failure |
| Impair one AZ | Multi-AZ failover, capacity headroom | AZ-level power/network event |
| Add latency to a dependency | Timeouts, retries, circuit breakers | Degraded downstream service |
| Blackhole a service endpoint | Graceful degradation, fallback paths | DNS/endpoint failure |
Pair this with the reliability wins in AWS Auto Scaling for Reliability: a chaos experiment is the only honest way to confirm your health checks and headroom actually absorb an AZ loss.
Game days: the human half
Automated experiments test your architecture. Game days test your people and process. A game day is a scheduled exercise where the team simulates a failure and practices the full response: detection, runbooks, communication, on-call handoff. It reveals gaps that no FIS experiment will, like a runbook that references a dashboard nobody can find or a break-glass credential that routes through the failing region.
The DR checklist prescribes quarterly game days against the three real failure classes: AZ loss, regional control-plane outage, and DNS/endpoint failure. Run them, measure the achieved recovery time against your target, and fix the gaps. A recovery plan that has never missed its target has never been tested hard enough.
Detection is part of resilience
You cannot respond to, or fail over from, an incident you have not noticed. Build independent detection so a real event does not depend on a customer ticket to reach you: external monitoring plus instant outage alerts typically beats the first support ticket by 20 to 40 minutes.
When you are ready to widen the aperture beyond your own systems, the reliability research at clouddowntime.com/data tracks how AWS actually fails across regions and services, and if a provider incident breaches an SLA you may be owed credits, checkable at cloudslacredit.com. For detection wired directly into your stack, Next Signal (sponsor) watches provider status so the game day you never scheduled does not become the incident you find out about last.