What happened?

• Starting at 18:20 UTC Quay.io became increasingly slow to respond to Docker Registry queries.
• 18:20 UTC: A Redis Database provisioning operation failed. We started investigating immediately. The instance that was pulling the image was running an older version of Docker (1.6) than our current baseline (1.11), so we originally assumed that the issue might be on our end.
• 18:36 UTC: A second Redis Database provisioning operation failed, on a different instance, running a different version of Docker. We turned our investigation towards the registry (the only shared component in this context) and identified that Quay was indeed unresponsive.
• At this stage, Quay was successfully returning metadata when queried by the Docker daemon running on our instances, but timing out when returning layers. This is why Database provisioning operations would fail (Database image layers tend to not be cached locally because we have a large number of different database images), whereas App operations continued to succeed (those operations also need to pull from Quay, but there is a much smaller number of images, which are used a lot more frequently, so the layers tend to always be in cache).
• 18:36 UTC: We decided to failover to the Docker Hub, where all our Database images are replicated.
• 18:46 UTC: We completed the failover to the Docker Hub for all our Database images.
• 18:46 UTC: We turned to attention to other images hosted on Quay. Indeed, we offer Aptible-managed base Docker images via Quay (e.g. quay.io/aptible/ruby), which some customers depend on for their builds (in other words, they use them in their Dockerfile FROM line). Most of our base images are mirrored on the Docker Hub, so customers could switch to them to let their builds succeed, but 2 of them (which are seldom used) were missing, so we copied them over.
• 18:55 UTC: We identified that Quay was starting to timeout on metadata queries as well. This was going to affect App operations. As a result, we decided to failover to the Docker Hub for all our registry activity.
• 19:07 UTC: We completed the failover to the Docker Hub for all registry activity.
• 19:07 UTC - 21:35 UTC: We proactively reached out to all customers that were depending on a Quay-hosted base image to let them know they could switch to a mirror on the Docker Hub.

By the numbers:

• Database Operations failed (due to the outage affecting Quay registry layers): 2.
  • They were both retried successfully by our team within 30 minutes.
• App Operations failed (due to the outage affecting Quay registry metadata): 3.
  • These were retried by the customers a few minutes later.
• Customers proactively contacted when their build failed: 16.

Next steps:

• We've made improvements to our image replication process to ensure all images are replicated to the Docker Hub. This was completed on Friday, July 28, 2017.
• We've made improvements to our orchestrator code so that switching over from Quay to the Docker Hub (and vice versa) is even faster.