13th March 2018

Continuous Integration Major outage on travis-ci.com

Outage at third party build service prevented us from deploying updates to any LiveControl component from midday.

Resolved - The backlog has cleared. Systems have resumed normal operational status.
Mar 13, 20:59 UTC

Update - There is a still a backlog for sudo-enabled builds on travis-ci.com, but builds are processing at full capacity.
Mar 13, 20:13 UTC

Update - We are happy to report that our Mac infrastructure is now clear. A backlog remains on our sudo-enabled infrastructure. At the current pace, we expect the backlog to be cleared in approximately 1 hour.
Mar 13, 19:06 UTC

Monitoring - We are processing builds normally but we currently see backlogs on our Mac and sudo: required (i.e. GCE) infrastructures. On the other hand, our sudo:false (i.e. EC2) infrastructure is all clear. We'll continue to provide timely updates to the state of the backlogs. Thank you for your patience.
Mar 13, 18:41 UTC

Update - All platform services are running. We are now processing builds for private repositories on travis-ci.com. We'll continue to monitor as we begin to work through the backlog of builds.
Mar 13, 17:13 UTC

Update - We’ve reattached the database to our platform. www.travis-ci.com and api are back up, currently waiting on all platform services to come back up.
Mar 13, 17:06 UTC

Update - Database provisioning has completed earlier than expected. We now estimate that travis-ci.com should be fully functional by 18.00 UTC (14.00 EDT, 19.00 CET). Thanks for bearing with us.
Mar 13, 16:24 UTC

Identified - Database provisioning is around 50% complete. We continue to prepare for our system coming back online, and plan remediation work.
Mar 13, 15:54 UTC

Update - Database provisioning is around 1/3 complete. As it stands our best estimate for travis-ci.com being fully functional is 20.00 UTC (16.00 EDT, 21.00 CET). We will continue to provide regular updates.
Mar 13, 15:06 UTC

Update - Database provisioning from the recovery point is still underway. We are doing preparatory work to deal with increased demand when travis-ci.com comes back online. We also continue to investigate the original issue.
Mar 13, 14:29 UTC

Update - Database provisioning continues, and we are working closely with our database service provider to diagnose the issue.
Mar 13, 13:44 UTC

Update - A replacement database is being provisioned from a snapshot taken this morning. We will update as this process continues.
Mar 13, 13:13 UTC

Update - Customers have reported missing account data. We can confirm that we have seen the same symptoms.
Mar 13, 12:45 UTC

Investigating - We've identified a major outage on travis-ci.com. We're working to identify the problem and will update this issue shortly.
Mar 13, 12:34 UTC