Skip to main content
awsdown
majorLambdaus-east-1

AWS Lambda Outage in us-east-1 (June 13, 2023): What Happened

AWSDown Research · Incident analysisPublished June 18, 2023Updated March 20, 20266 min read

Timeline

  1. Impact begins

    Lambda invocation error rates rise sharply in us-east-1. Event-driven pipelines (SQS/SNS/EventBridge triggers) begin accumulating backlogs.

  2. Investigating

    AWS acknowledges increased error rates and latencies across multiple services in us-east-1, including AWS STS and console sign-in flows that depend on Lambda internally.

  3. Identified

    Root cause identified in a subsystem responsible for capacity management of Lambda’s execution environment fleet. Mitigation begins.

  4. Monitoring

    Error rates improve as capacity management recovers; asynchronous invocation backlogs start draining.

  5. Resolved

    AWS reports Lambda invocation errors resolved and backlogs processed; dependent services return to normal.

Root cause

Lambda's promise is that capacity is someone else's problem - specifically, an internal capacity-management subsystem that keeps a warm fleet of execution environments matched to demand. On June 13, 2023, that subsystem degraded in us-east-1. Invocations that couldn't be placed onto healthy capacity failed or timed out, and the failure surfaced simultaneously in every AWS feature that quietly invokes Lambda on your behalf.

Business impact

Three distinct groups felt this event differently. Serverless-first products saw direct 5xx errors on their API paths. Traditional workloads were hit indirectly - console access and STS-issued credentials stumbled, which broke deploy pipelines and operator access at exactly the wrong time. And event-driven backends mostly experienced delay: queued events drained after recovery, which is the failure mode you want, but only if your consumers are idempotent and your queues have sensible retention.

Prevention and lessons

  1. Set concurrency floors and timeouts deliberately. During capacity events, generous function timeouts turn into user-facing hangs. Fail fast at the edge and degrade gracefully.
  2. Make async your default. The cleanest survivors were architectures where anything deferrable ran through queues with retries - late beats broken. Test the drain, though: a three-hour backlog arriving at once is its own load test.
  3. Keep an out-of-region operator path. Console/STS coupling means your break-glass access should not depend on us-east-1 - regional STS endpoints and pre-issued credentials are cheap (DR checklist, step 4).
  4. Log per-request outcomes. SLA claims for partial outages live or die on your own error-rate evidence - see the Lambda status page for how we track incident windows you can cite.

SLA credit eligibility

Borderline - claimable for accounts that documented sustained invocation failures. Lambda’s 99.95% monthly target allows ~21 minutes of full downtime; partial-failure math depends on your measured error rates, which is why keeping per-request logs matters.

Work out what your bill was owed with the independent AWS SLA credit calculator, or automate breach detection with Next Signal (sponsor). For the full claim process, see the AWS credit-recovery playbook.

Sources: AWS Health Dashboard, AWS Lambda Service Level Agreement, CloudDowntime - cloud reliability research

Questions about this outage

What caused the June 13, 2023 AWS Lambda outage?

AWS attributed it to a degradation in the internal subsystem that manages capacity for Lambda’s execution environments in us-east-1. With the fleet unable to scale properly, a portion of invocations failed or timed out for roughly three hours.

Why did an AWS Lambda outage break console logins and STS?

AWS builds AWS on AWS: pieces of STS token issuance and the management console invoke Lambda internally. When Lambda degraded in us-east-1, those features degraded with it - a reminder that “serverless” is in more of your dependency chain than your own code.

Were asynchronous Lambda invocations lost during the outage?

Generally no - asynchronous events were queued and retried, arriving late rather than never. Synchronous invocations (API-backed request paths) surfaced the errors directly to end users.

Did the June 2023 Lambda outage breach the SLA?

For heavily affected accounts, ~3 hours of elevated errors approaches the 99.95% monthly threshold (which allows ~21 minutes of downtime). Accounts that could document sustained failure rates had a claimable 10% credit case.

Next time, hear it from us first

Instant alerts when AWS incidents are detected - services and regions included.

Get outage alerts

More post-mortems

S3

The 2017 S3 Outage: A Typo That Broke the Web - Lessons That Still Apply

A playbook command executed with a mistyped parameter removed far more S3 index capacity than intended in us-east-1. The subsystems needed a full restart - something S3 hadn’t done in years at that scale - and for ~4 hours, a huge share of the web (including AWS’s own status dashboard) broke.