EC2 Host Failure
Incident Report for Aptible
Resolved
This incident has been resolved.
Posted Jul 13, 2021 - 11:24 EDT
Identified
Most of the affected instances and customer services have been fully restored. A total of 4 EC2 instances, associated with databases belonging to 2 customers, are still unhealthy. We're working on restoring these databases from backup to provide customers with an option for more rapid recovery.
Posted Jul 13, 2021 - 09:35 EDT
Update
We are continuing to investigate the issue. Currently, AWS is reporting an issue on their service dashboard [0]:

> 5:29 AM PDT We are investigating increased error rates and latencies for the EC2 APIs and connectivity issues for some instances in a single Availability Zone in the EU-CENTRAL-1 Region

This issue is currently preventing the recovery of existing instances as well as the launching of new instances. We are continuing to explore workarounds, but the extent of the outage is currently preventing most recovery techniques. As such, we are most likely waiting on AWS to mitigate the extent of the outage before we can complete resolving the issue.

[0] https://status.aws.amazon.com/#EU_block
Posted Jul 13, 2021 - 08:57 EDT
Investigating
We are investigating an EC2 dedicated host failure affecting a small number of apps and databases in eu-central-1. Affected apps are currently being restarted on healthy instances (apps scaled to 2 or more are automatically distributed across availability zones and will automatically failover).
Posted Jul 13, 2021 - 08:21 EDT
This incident affected: Aptible Deploy.