With an acknowledgement from AWS (copied below) we are marking this incident as resolved.
> Between 12:20 AM and 11:28 AM PDT, [on October 15th] we experienced intermittent failures in Route53 Health Checks impacting Target Health evaluation in US-EAST-1. The issue has been resolved and the service is operating normally.
Posted Oct 17, 2022 - 17:37 EDT
Update
The issue should be resolved for all endpoints that have been affected. We've also updated the behavior of the platform so that if you see this issue, running `aptible restart` on the affected app will update the configuration of all the app's endpoints to ignore health checks, and should thereby resolve the issue.
Posted Oct 15, 2022 - 14:32 EDT
Monitoring
All impacted endpoints that we have identified have been fixed by disabling their health checks.
If you have an Endpoint that seems to be impacted, please email support@aptible.com and include the domain name of the Endpoint in question.
It has been noticed that we can only detect the issue when HTTP requests are made to the Endpoint, and as the incident started in the middle of a weekend night, we were not able to identify Endpoints which were not in use.
We're reviewing and remediating additional Endpoints now, and will make an update to the platform to remove health checks entirely if AWS cannot fix the Route53 issue.