System Status

February 2017

[Scheduled] Dedicated database maintenance
The scheduled maintenance has been completed.
Feb 10, 22:55-23:01 EST
New app release operations failing on some dedicated stacks
This incident has been resolved.
Feb 7, 01:27-01:51 EST
A small number of deployments failed with an internal error
During a routine software update, one server in one of Aptible's non-production infrastructure stacks didn't receive updated software, and as a result, app deployments that were scheduled there did not succeed. Due to the limited scope of the issue, only a very small number of apps and deployments were affected (respectively 7 apps over 14 deployments). If you're investigating a deployment failure, you can identify affected deployments by an internal error logged in your deployment output immediately after attempting to lock ports. Running apps were not affected (neither in production nor development environments).
Feb 6, 16:13 EST

January 2017

Deploys and Aptible SSH temporarily unavailable
This issue has been resolved.
Jan 19, 19:14-19:43 EST
EC2 outage affecting several environments
It appears this issue was either a monitoring false positive or very short-lived (despite Cloudwatch monitoring reporting issues, we did not observe any).
Jan 17, 12:17-12:31 EST
[Scheduled] Database SSL Certificate Rotation
The scheduled maintenance has been completed.
Jan 17, 00:43-09:53 EST
[Scheduled] Database SSL certificate rotation: dedicated databases
The scheduled maintenance has been completed.
Jan 8, 22:56-23:45 EST
[Scheduled] Database SSL certificate rotation: shared databases
The scheduled maintenance has been completed.
Jan 5, 22:17-23:10 EST

December 2016

[Scheduled] Legacy Shared Stack Security Updates
The scheduled maintenance has been completed.
Dec 6, 22:41-23:17 EST
Some development apps unavailable
This incident has been resolved.
Dec 1, 14:01-14:25 EST

November 2016

No incidents reported for this month.

October 2016

Git Server Availability
This incident has been resolved.
Oct 31, 17:35-17:47 EST
Issues provisioning new Managed HTTPS Endpoints
Let's Encrypt is back to fully operational.
Oct 31, 10:14-13:00 EST
Platform operations disabled due to ELB issues
This incident has been resolved.
Oct 24, 15:19-15:24 EST
Platform operations disabled due to EC2 outage
This incident has been resolved.
Oct 20, 10:31-11:24 EST
Some development apps unavailable
This incident has been resolved.
Oct 11, 23:04-23:33 EST
Git Server Availability
This incident has been resolved.
Oct 11, 15:45-15:48 EST
Some development apps unavailable
This incident has been resolved.
Oct 4, 14:09-14:21 EST

September 2016

Some development apps unavailable
This incident has been resolved.
Sep 27, 18:31-19:03 EST
7 development databases unavailable
All affected databases have been restored.
Sep 16, 05:43-05:51 EST
Small number of development apps unavailable
The affected apps were recovered.
Sep 16, 04:45-04:50 EST

August 2016

AWS issues affecting Aptible components
This incident has been resolved.
Aug 30, 10:06-10:58 EST
Small number of development databases unavailable
The 6 development databases that failed have been recovered.
Aug 30, 03:53-04:03 EST
High rate of 500 errors on api.aptible.com
This incident has been resolved.
Aug 20, 23:05-23:21 EST
Git server downtime
This incident has been resolved.
Aug 17, 15:30-15:37 EST

July 2016

EC2 outage affecting several environments
This incident has been resolved.
Jul 25, 21:27-23:07 EST
High operation failure rate due to EC2 API issues
As of the last status update at 20:37 UTC (4:37pm EDT), there have been no further EC2-related issues.
Jul 19, 14:52-21:24 EST
Some development apps unavailable
This incident has been resolved.
Jul 18, 10:15-10:24 EST
[Scheduled] Git server maintenance
This maintenance has been postponed until another date in the near future..
Jul 14, 22:02 EST
Some development databases unavailable
This incident has been resolved.
Jul 14, 12:38-12:49 EST
Some development apps unavailable
This incident has been resolved.
Jul 13, 12:17-13:35 EST
EC2 outage affecting several environments
This incident has been resolved.
Jul 8, 01:42-03:04 EST
Some development apps unavailable
This incident has been resolved.
Jul 6, 20:27-20:45 EST

June 2016

Some development apps unavailable
This incident has been resolved.
Jun 27, 17:40-18:21 EST
Small number of development apps unavailable
Due to an EC2 instance failure, a small number of developments apps became unavailable at 15:20 UTC (apps scaled to more than 1 container were not affected). 85% of affected apps were recovered within 10 minutes, and as of 15:40 UTC, all affected apps have been recovered.
Jun 24, 11:42 EST
Some development apps unavailable
This incident has been resolved.
Jun 19, 22:57 - Jun 20, 00:52 EST
Some development apps unavailable
This incident has been resolved.
Jun 17, 11:45-12:51 EST
A small number of development databases are unavailable
The 4 development databases that were unavailable earlier today are back up and running as of 5 minutes ago.
Jun 7, 06:33-06:49 EST
[Scheduled] Development database maintenance
The scheduled maintenance has been completed.
Jun 1, 22:24 - Jun 2, 00:48 EST
Aptible API down
This incident has been resolved.
Jun 1, 12:11-12:17 EST

May 2016

Some development deploys and SSH sessions failing
This incident has been resolved.
May 25, 15:59-19:13 EST

April 2016

Some development apps unavailable
This incident has been resolved.
Apr 20, 11:13-11:24 EST
Some development apps unavailable
This incident has been resolved.
Apr 18, 18:06-18:23 EST
Some development apps unavailable
All development apps affected by the recent EC2 instance failure have been restored.
Apr 8, 12:44-12:52 EST
Some development apps unavailable
The EC2 instance has been recovered and all affected development apps have been restored.
Apr 7, 11:09-11:28 EST

March 2016

Some development apps unavailable
This incident has been resolved.
Mar 10, 18:51-19:07 EST
Some development apps unavailable
This incident has been resolved.
Mar 10, 14:28-14:33 EST

February 2016

Some development apps unavailable
This incident has been resolved.
Feb 25, 14:20-14:41 EST
Some development apps unavailable
This incident has been resolved.
Feb 22, 19:39-19:48 EST
Some development apps unavailable
This incident has been resolved.
Feb 18, 17:15-17:58 EST
A low number of API requests are timing out
Our error capture suggests the few API failures we observed were due to network glitches. The problem has subsided for now. We've deployed additional monitoring to better track those errors and will consider additional remediation steps should the problem occur again.
Feb 5, 13:53-14:39 EST
Deploys temporarily unavailable
This incident has been resolved.
Feb 4, 20:21-20:47 EST
Some development apps unavailable
This incident has been resolved.
Feb 3, 15:09-15:42 EST
[Scheduled] Git server maintenance
The scheduled maintenance has been completed.
Feb 1, 22:03-22:27 EST

January 2016

Some apps unresponsive after restart/deploy
This incident has been resolved.
Jan 27, 16:35-20:03 EST
Some development apps unavailable
Incident resolved as of 6:02pm EST.
Jan 7, 17:42-18:45 EST

December 2015

Some development apps unavailable
Service is restored and affected apps have been restarted.
Dec 24, 13:16-14:29 EST
Some Development Apps Unavailable
All services have been restored as of 21:53 EST, and no further issues should exist.
Dec 9, 21:32-22:21 EST
[Scheduled] We're performing maintenance on our Git server which receives "git push" operations
The scheduled maintenance has been completed.
Dec 8, 14:00-15:00 EST

November 2015

EC2 instance failure affecting a subset of development databases
This incident has been resolved.
Nov 27, 13:12-13:54 EST
EC2 failure affecting a subset of development apps
This incident has been resolved.
Nov 21, 01:13-01:33 EST
Some Development Apps Unavailable
This incident has been resolved.
Nov 13, 18:14-18:55 EST
EC2 instance failure affecting a subset of development databases
This incident has been resolved.
Nov 9, 22:28-22:37 EST
Performance issues affecting some development apps
This incident has been resolved.
Nov 9, 20:28-22:27 EST
High error rate on api.aptible.com
This incident has been resolved.
Nov 5, 18:52-18:56 EST
Some Development Apps Unavailable
This incident has been resolved.
Nov 3, 18:15-19:24 EST

October 2015

Some development apps and databases unavailable
This incident has been resolved.
Oct 30, 08:02-10:41 EST
Some Development Apps Unavailable
All development apps have been restored.
Oct 27, 13:30-16:10 EST
Errors from api.aptible.com
A fix has been deployed and API errors have stopped. We'll continue to monitor the situation.
Oct 15, 15:10-15:19 EST
[Scheduled] Scheduled Production API Migration
The scheduled maintenance has been completed.
Oct 12, 22:32-23:05 EST
Some development apps unavailable
This incident has been resolved.
Oct 7, 12:52-13:45 EST

September 2015

API availability
The Aptible API was unavailable from approximately 3:00am EDT to 7:00am EDT this morning. Customers were unable to perform backend operations. Customer apps were not affected. We are investigating the root cause.
Sep 17, 07:17 EST
Some development databases unavailable
The EC2 instance has recovered and all development databases are now back up and running.
Sep 11, 17:27-17:43 EST
Intermittent AWS Rate Limit Errors
This incident has been resolved.
Sep 3, 15:23-16:07 EST
Some development apps unavailable
This incident has been resolved.
Sep 3, 11:43-12:39 EST

August 2015

[Scheduled] Aptible API Upgrade
The scheduled maintenance has been completed.
Aug 21, 17:30-18:11 EST

July 2015

Some Development Apps Unavailable
This incident has been resolved.
Jul 27, 12:36-13:14 EST
[Scheduled] Aptible Git Server Upgrade
The scheduled maintenance has been completed.
Jul 27, 10:45-10:49 EST
API Availability
This incident has been resolved.
Jul 21, 04:33-04:44 EST
Some development apps unavailable
This incident has been resolved.
Jul 11, 02:58-03:36 EST
Some development apps unavailable
This incident has been resolved.
Jul 9, 01:08-01:55 EST

June 2015

Some development apps unavailable
All development apps have been restored.
Jun 19, 14:53-15:02 EST
Dashboard availability interruption
This incident has been resolved.
Jun 8, 11:57-12:10 EST
Dashboard availability impaired
This incident has been resolved.
Jun 5, 14:59-15:31 EST
Some development apps and databases inaccessible
This incident has been resolved.
Jun 3, 08:15-08:53 EST

May 2015

Development instance under heavy load
All development servers are running under normal load and application logs are being delivered.
May 21, 14:59-16:43 EST
Build timeouts while pulling from Docker registry
We've deployed a fix to our system to work around most image pulls from quay.io. All builds should be back to normal.
May 18, 16:03-20:03 EST
Temporary database outage
This incident has been resolved.
May 4, 19:40-20:39 EST

April 2015

[Scheduled] Development Stack Maintenance (Docker Upgrade)
The scheduled maintenance has been completed.
Apr 25, 09:24-13:15 EST
Some development apps unavailable
All affected development apps have been restored.
Apr 24, 04:57-05:13 EST
Performance issues on deploy for some development apps
This incident has been resolved.
Apr 22, 14:39-17:06 EST
Some development apps unavailable
This incident has been resolved.
Apr 21, 20:50-21:05 EST
Some development apps unavailable
This incident has been resolved.
Apr 21, 14:47-16:23 EST
[Scheduled] Development Stack Maintenance: Brief App Downtime
The scheduled maintenance has been completed.
Apr 13, 22:02-22:28 EST
Development app outage affecting a subset of apps
This incident has been resolved.
Apr 13, 18:09-18:47 EST
Maintenance: Operations not possible on some development stacks
This incident has been resolved.
Apr 13, 09:35-10:23 EST

March 2015

Partial outage for restart operations on development apps
This incident has been resolved.
Mar 25, 14:19-17:02 EST
New deployments depending on Docker Hub-hosted images are failing
This incident has been resolved.
Mar 24, 12:48-13:59 EST

February 2015

No incidents reported for this month.

January 2015

Backend operations failing
This incident has been resolved.
Jan 28, 16:47-16:50 EST
Kernel Upgrade
This incident has been resolved.
Jan 15, 17:18-17:36 EST
Forced Restart Of Development Apps
This incident has been resolved.
Jan 9, 06:37-06:50 EST

December 2014

Stack Migration
This incident has been resolved.
Dec 18, 22:49 - Dec 19, 00:29 EST
Stack Migration
This incident has been resolved.
Dec 9, 23:35 - Dec 10, 00:42 EST

November 2014

Intermittent Dashboard Unavailability
This incident has been resolved.
Nov 29, 18:19-21:32 EST

October 2014

Intermittent API Unavailability
Marking as resolved pending root cause analysis.
Oct 24, 12:08-18:09 EST

September 2014

No incidents reported for this month.

August 2014

No incidents reported for this month.

July 2014

Dashboard Unavailability
This incident has been resolved.
Jul 31, 21:15-21:35 EST
git@beta.aptible.com Down
This incident has been resolved.
Jul 24, 18:59-19:17 EST

June 2014

Development Accounts Unavailable
All development VHOSTs are running, and all development account services are operational.
Jun 27, 16:07-16:53 EST

May 2014

No incidents reported for this month.

April 2014

No incidents reported for this month.

March 2014

No incidents reported for this month.

February 2014

No incidents reported for this month.

January 2014

No incidents reported for this month.