31st January 2019

Major outage

12:35PM - We're investigating an abnormal level of database connectivity issues.

12:46PM - Major outage affecting all LiveControl services

13:00PM - All services have been restored and we are investigating the root cause of the issue.

17:41PM - Performance of our relational database system deteriorated, exhibiting symptoms of memory (RAM) exhaustion, before breaking down entirely.

Analysis has shown that memory utilisation remained fixed just below 30% for some time, but that dynamically added memory above this was not being utilised as expected and this 30% utilisation represented the system operating at full capacity.

We are monitoring this closely, have adjusted our monitoring thresholds accordingly, are taking steps to reallocate as needed to ensure we do not reach these limits and will be performing maintenance during our next scheduled window on Monday 4th February to ensure all allocated memory is correctly available and can be utilised as expected.