Resolved -
We have reached out individually to every Aptible Deploy customer running a MongoDB database on Deploy. Since most of these databases will be unaffected by the SSL configuration issue, we are not proactively restarting MongoDB databases. That said, we encourage all Aptible customers with MongoDB databases to verify that their apps are not reporting connectivity errors. If you see any connectivity errors, simply run: `aptible db:restart DATABASE_NAME`.
May 30, 15:32 EDT
Monitoring -
As of 18:10 UTC, all Elasticsearch, InfluxDB, CouchDB and RethinkDB databases have been restarted. We are continuing to monitor and evaluate whether to take further action for MongoDB, Redis and/or RabbitMQ databases.
May 30, 14:40 EDT
Update -
We've begun the process of restarting all Elasticsearch, InfluxDB, CouchDB and RethinkDB databases.
May 30, 11:00 EDT
Identified -
At 13:40 UTC today, our Reliability Team identified an issue affecting the SSL configuration for certain Aptible Deploy databases. Specifically, we identified an issue with the certificate bundle provided by our SSL vendor, Namecheap, for the `*.aptible.in` certificate we use to secure some databases via SSL.
Despite the fact that the certificate itself expires in February 2021, Namecheap provided us with a certificate bundle whose root certificate expired on May 30, 2020. We were not aware of this error until today, and Namecheap did not notify us about it.
Most database client libraries will still successfully connect to these databases even though the root certificate is expired, since the primary and issuer certificate are both valid. However, out of an abundance of caution, we are restarting all databases of the following types, to avoid issues:
* Elasticsearch
* InfluxDB
* CouchDB
* RethinkDB
In the unlikely event you see SSL-related issues raised by your app in connecting to databases of the following types, please simply restart your database using `aptible db:reload DATABASE_NAME`. This will automatically restart your database with the latest certificate (and bundle). Because of the very low probability that clients of these databases would be affected, we are choosing not to proactively restart them, as that might cause a brief disruption of service to customers.
* MongoDB
* Redis
* RabbitMQ
We will continue to update this issue as the database restarts mentioned above are completed, and if there are any changes to our planned remediation.
May 30, 10:55 EDT