Case study · Cloud
AWS us-east-1: one empty DNS record, a day of cascading failures
Worldwide, October 2025. A race condition in DynamoDB's DNS automation leaves a region without an address, and many services fall with it.
In one sentenceOne empty DNS record in one region: fourteen and a half hours of disruption. The culprit was not a server, it was automation.
- Sector
- Cloud infrastructure, all sectors
- Region
- us-east-1 (N. Virginia)
- Date
- 19–20 October 2025
- Event window
- About 14.5 hours, from 11:48 PM to 2:20 PM PDT
- Cause according to AWS
- A latent race condition in DynamoDB's DNS management
Analysis based solely on public information. FZ Consultation was not involved in this incident and has no affiliation with the organizations named.
What happened
- 19 Oct, 11:48 PM PDT
DynamoDB API calls start failing in us-east-1: customers and AWS services cannot open new connections.
- 20 Oct, 12:38 AM
AWS identifies DynamoDB's DNS state as the source.
- 2:25 – 2:40 AM
All DNS information is restored; cached records expire and customers reconnect. DynamoDB is back after about 2 hours 52 minutes.
- 2:25 – 10:36 AM
EC2 instance launches keep failing for hours after DynamoDB has recovered.
- 5:30 AM – 2:09 PM
Network Load Balancers return connection errors because of health-check failures.
- 20 Oct, 2:20 PM
End of the event according to AWS. Some Redshift clusters stay impaired until 21 October.
What went wrong
According to the public sources listed below.
Automation that undid itself
According to AWS, two DNS Enactors collided: a delayed one applied an old plan after a newer one, and the cleanup then deleted the old plan and removed every address of the regional endpoint. The system was left inconsistent and needed manual intervention.
The fix was only half of the recovery
Once DNS was restored at 2:25 AM, EC2 launches kept failing until 10:36 AM and NLB errors lasted until 2:09 PM. Recovering an upstream service does not recover its dependents.
The tools to respond were inside the blast radius
IAM user sign-in to the AWS console failed for over an hour and a half, and access to Support Center cases was blocked for almost three hours.
Lessons for your organization
- 1
Find your hidden regional dependency
Many “global” services quietly depend on one region. Map them before the outage.
Know: measure what stopping would cost - 2
Test the provider's failure, not just yours
Simulate losing a region and time the real recovery, cascading dependencies included.
Prove: rehearse before the real day - 3
Keep an out-of-band channel
Plan how to reach teams, customers and your provider when your usual tools are unreachable.
Ransomware checklist
Check on Monday morning
Three questions to put to your own organization this week. If you cannot answer one of them in five minutes, that is your first task.
- Which of your services depend on a single cloud region, including for identity, queues or configuration?
- If your provider's console and support were unreachable, how would you reach your teams and customers?
- Have you ever simulated losing a region, not just a server, and timed the real recovery?
Sources
More case studies
- Jaguar Land Rover: weeks of idle factories and a state-backed loanUK, 2025. A cyberattack stops production for weeks and reaches over 5,000 organisations across the supply chain.
- Collins Aerospace: one check-in system, four airports on paperEurope, September 2025. Ransomware hits a shared check-in system and forces Heathrow, Brussels, Berlin and Dublin onto manual processes.
- Marks & Spencer: weeks without online salesUK, 2025. A cyberattack halts online orders and forces stores onto manual processes.