30th August 2019

Partial outage

10:00 - Serious performance issues and intermittent outage affecting Speciality Fine Food Fair 2019 registration.

10:20 - Pipedrive integration failures causing persistent spike in write activity preventing registration form activity from getting required lock to write data. Pipedrive integration has been temporarily disabled following current executions to isolate issue.

10:40 - Background task workers temporarily disabled for Fresh Montgomery to isolate issue.

10:50 - Performance has improved and no failures detected in several minutes. We are beginning to provision additional resource and will be restoring background workers throughout the rest of the day, which will have backlogs to work through.

11:02 - Migrating IHS 2019 to separate database architecture to isolate issue.

11:04 - Finished migrating IHS 2019 to separate database architecture to isolate issue.

13:30 - An increase in traffic across our services, combined with higher than usual duration of response, is causing performance degradation now affecting all clients. Registrations are still being accepted and processed. Our plan to provision additional resource to address this is still being enacted and we expect to resolve the issues this evening.

15:00 - Demand now causing intermittent outage affecting all clients, background task workers temporarily disabled for all users to mitigate the issue.

15:30 - Background task workers re-enabled. Additional resource now at configuration management stage. We expect this to be completed and work to begin being reallocated by 16:30. Full resolution expected later this evening.

17:21 - Started gateway and firewall maintenance, expected to last 2-3 minutes.

17:24 - Gateway and firewall maintenance complete.