9th November 2017

Full Service Outage

06:49AM - We are currently experiencing a full service outage. This is due to an on-going power grid issue in France, affecting multiple data centres and tens of thousands of businesses. The team are receiving updates from the data centre, we'll report back when we have more information.

09:25AM - ETA for services to be restored is 15 minutes.
SBG: 2 routing rooms are UP powered by gen. EDF still down. ETA: 15min

09:46AM - ETA for services to be restored is 1h-2h (we are in SBG1).
SBG: EDF repaired 1 line 20KV. the second is still down. All Gens are UP. 2 routing rooms coming UP. SBG2 will be UP in 15-20min (boot time). SBG1/SBG4: 1h-2h

11:30AM - ETA for services to be restored is between 5 minutes and 3-4 hours.
Power has been restored and services are restarting. Some customers are UP and others not yet. If your service is not yet UP, the recovery time is between 5 minutes and 3-4 hours. Our monitoring system allows us to know which customer is still impacted and we are working to fix them. [..] We are in the process of retrieving the details to provide you with information on the SBG recovery time of all services / customers. Also, we will give all the technical details on the origin of these 2 incidents.

12:25PM - Legacy Platform and LiveWeb service restored. IPs and Networking are coming online in stages. Awaiting Storage Area Network (SAN) for LiveControl.

16:15PM - Power and Network have been fully restored. Data centre team are working on resolving VMware Private Cloud issues.
We are gradually restarting all vCenters and all hosts.

16:48PM - All services are restored.