System Status
All Systems Operational
api.aptible.com ? Operational
90 days ago
100.0 % uptime
Today
auth.aptible.com ? Operational
90 days ago
100.0 % uptime
Today
gridiron.aptible.com ? Operational
90 days ago
100.0 % uptime
Today
dashboard.aptible.com ? Operational
90 days ago
100.0 % uptime
Today
Enclave Operational
90 days ago
100.0 % uptime
Today
AWS EC2 (Primary — Virginia) Operational
AWS EC2 (Secondary — Oregon) Operational
AWS EC2 (Secondary — California) Operational
Let's Encrypt ? Operational
Quay.io Registry Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
had a major outage
had a partial outage
Scheduled Maintenance
Dedicated stack maintenance May 22, 18:00-20:00 EDT
In anticipation of the scheduled retirement (by AWS) of several EC2 instances hosting Aptible resources, we will be replacing several impacted NAT instances, database containers, and app containers on Wednesday 5/22 between 22:00 and 23:59 UTC.

For any stack that did not already have multiple outbound IP addresses, an additional outbound IP has been added. The new outbound IP address will be used during the NAT replacement process for impacted stacks to ensure that there is not any downtime. Please update any location where you have the current outbound IP of your stacks whitelisted to include the additional outbound IP addresses for your stacks. You can find instructions for locating the outbound IP addresses for your dedicated stack here: https://www.aptible.com/documentation/enclave/reference/stacks/outbound-ips.html#determining-an-app-s-outbound-ip-address.

Any databases on an instance that is being replaced may be unavailable for 60-90 seconds during this maintenance window. We will be reaching out to customers with impacted databases in the next 24 hours.

Any apps on an instance that is being replaced will be restarted with a zero-downtime `aptible restart` operation.
Posted on May 20, 15:35 EDT
Uptime: api.aptible.com
Fetching
Uptime: auth.aptible.com
Fetching
Uptime: gridiron.aptible.com
Fetching
Past Incidents
May 21, 2019

No incidents reported today.

May 20, 2019

No incidents reported.

May 19, 2019

No incidents reported.

May 18, 2019
Resolved - The maintenance is complete. At this time, all app, build, bastion and database instances in shared-tenancy stacks have been replaced, and all shared-tenancy resources are running on instances with patched kernels.
May 18, 22:27 EDT
Update - We're beginning the scheduled database instance replacement now, and we'll update this status page once the maintenance is complete.
May 18, 18:46 EDT
Update - All app, build and bastion instances in shared-tenancy stacks have been replaced as of Thursday, May 16.

After continued investigation by our Security Team, we have decided to replace all database instances in shared stacks as well, and will complete this migration *on Saturday, May 18 between 18:00 and 22:00 EDT.* During this period, all shared-tenancy databases will be restarted on new instances, and this will result in a brief downtime (up to 60 seconds) for each shared-tenancy database.
May 17, 20:55 EDT
Investigating - Today, a new speculative execution attack affecting Intel processors was announced. The attack uses the Microarchitectural Data Sampling ("MDS") approach, and has been nicknamed "ZombieLoad" by the security community.

In order to ensure that customer environments are protected against this vulnerability, our Reliability Team is scheduling the process of re-launching customer containers and utility services (e.g., build instances, ephemeral SSH servers) on new EC2 instances that have been updated to the latest Linux kernel, which contains a mitigation against this attack.

To begin, the most vulnerable instances will be replaced — i.e., instances in shared environments where arbitrary code may be run. These include:

• App instances
• Build instances (where Docker images are built)
• Bastion instances (where ephemeral `aptible ssh` containers run)

The replacement operation should incur no downtime for your apps, but will require that we run `aptible restart` (a zero downtime operation) on each app in your shared environments.

We will continue to update this status page as we proceed with the patching process. We expect to begin replacing instances within the next 18 hours.

Here are a couple relevant links to learn more about this vulnerability:
• https://www.cyberus-technology.de/posts/2019-05-14-zombieload.html
• https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00233.html
May 14, 23:04 EDT
May 16, 2019
Resolved - This incident has been resolved.
May 16, 09:10 EDT
Identified - After working with the AWS Support Team, we've identified a regression in the AWS ELB API that's creating the root cause of this error. We're currently waiting for a full diagnosis and fix from AWS. In the meantime, we will continue to disable operations to apps using legacy ELB endpoints. We'll also update this status page as we have any additional information.
May 15, 22:07 EDT
Investigating - We are currently investigating an issue where headers are not being passed through correctly for some legacy ELB endpoints. We have disabled any operations to apps and endpoints using the using legacy ELB endpoint while we investigate.

Anyone who is effected by this outage should consider switching to the newer ALB endpoint as a quicker way to resolve this issue: https://www.aptible.com/documentation/enclave/reference/apps/endpoints/https-endpoints/alb-elb.html#upgrading-to-alb-from-elb
May 15, 16:57 EDT
May 13, 2019

No incidents reported.

May 12, 2019

No incidents reported.

May 11, 2019
Resolved - This incident has been resolved.
May 11, 10:13 EDT
Update - We have re-enabled all operations and are continuing to monitor upstream status at AWS.
May 11, 00:52 EDT
Monitoring - Our upstream provider (AWS Route 53) is reporting that the underlying issue is mostly resolved, although there still may be intermittent failures. We are continuing to monitor the situation.
May 11, 00:11 EDT
Investigating - We are observing timeouts during some operations for Apps, Databases, and Endpoints due to DNS registration timeouts.

Running Apps and Databases and existing Endpoints are not affected, but restart, provisioning, deprovisioning, and scaling operations are likely to timeout.

Our upstream provider (AWS Route 53) has reported that they are aware of the issue (https://status.aws.amazon.com/).
May 10, 16:10 EDT
May 9, 2019

No incidents reported.

May 8, 2019

No incidents reported.

May 7, 2019

No incidents reported.