AWS Outage on DynamoDB 2025: A Race Condition in DNS Automation

Late on 19 October 2025, DynamoDB in the US-EAST-1 (N. Virginia) region started failing. By the time the last customer problem ended on 20 October, more than fourteen hours had passed and a long list of AWS services had been touched. The starting point was one empty DNS record.

DNS is the front door of every AWS service

DynamoDB keeps hundreds of thousands of DNS records for a region. They route traffic to load balancers, remove unhealthy capacity and let the service scale. Two small pieces of automation manage this:

• A DNS Planner watches the load balancers and produces a numbered plan: which IPs should be behind the endpoint.
• Several DNS Enactors (one per availability zone) pick up the latest plan and apply it to Route 53.

Enactors run independently, so the system has no single point of failure. It also means two of them can be at different stages at the same moment.

The race

AWS describes a latent defect that needed a very unlucky ordering:

1. Enactor A picked up a plan, then ran unusually slowly while retrying.
2. Meanwhile the Planner kept producing newer plans. Enactor B applied a newer plan quickly.
3. B then did its normal tidy-up and deleted old plans.
4. Enactor A finally got going and applied its old plan to the regional endpoint. Its safety check ("is this plan newer than what is applied?") had been made much earlier, before the delay, so it did not catch the stale plan.
5. B's clean-up then deleted that old plan, and all the IP addresses with it.

The endpoint dynamodb.us-east-1.amazonaws.com now resolved to nothing, and the automation was stuck in a state it could not repair by itself.

The check was true, then it stopped being true

That is the textbook time-of-check to time-of-use race:

# Illustrative, not AWS code.
def apply_plan(plan):
    if plan.version > current_version():     # check
        ...                                  # slow work, retries, delays
        publish_to_dns(plan)                 # use: current_version() may have moved on

The fix is to make the check and the write one atomic step, for example a conditional write that only succeeds if the version is still the one you saw.

The ripple effect

DynamoDB is used by many other AWS services, so the damage spread. AWS's summary describes three overlapping periods: DynamoDB API errors (11:48 PM to 2:40 AM PDT), failures to launch new EC2 instances (2:25 AM to 10:36 AM) and Network Load Balancer connection errors (5:30 AM to 2:09 PM). Recovery of one layer overloaded the next. When DynamoDB came back, EC2's internal management system had a huge backlog of work to catch up on, and that delayed things further.

What to take away

1. A check is not a lock. If something can change between "check" and "act", make the check part of the action.
2. Automation that deletes needs the most guard rails. The clean-up job removed the plan that was still in use.
3. Your dependencies have dependencies. Know which of your services lean on one regional endpoint, and what happens to them if it disappears.
4. Recovery is its own failure mode. Systems that were down for hours may be hit by a thundering herd of work when they return. Design for a slow, throttled restart.
5. Region isolation is a design choice. Many "global" systems still keep a control plane in us-east-1.