09:00 - We are responding to sustained DDoS activity which is having an impact on registration form and exhibitor hub performance.
8th September 2019
7th September 2019
6th September 2019
5th September 2019
4th September 2019
3rd September 2019
2nd September 2019
1st September 2019
31st August 2019
02:55 - Additional dedicated database resource has been provisioned for our most active clients. This is having a positive effect. Other components in our platform remain shared and have been assessed. Our gateway, which routes all traffic and takes care of SSL certificates, has had it's compute resource boosted as it was a factor during the spike in traffic yesterday afternoon. All front-end services are now online. We'll be bringing background services online through the morning.
09:30 - In response to performance issues this morning we've identified the cause to be background task build-up, due to reduced services, causing new issues during registration (pushing new tasks into the queue). We've rebuilt the tasks table and seeing improved performance through the registration forms now. All background tasks are now online. We’re continuing to look into the performance and any issues resulting from yesterday and after-effects of the reduced level of service.
13:20 - Unfortunately whilst carrying out additional maintenance today (regular tasks we usually complete in our maintenance window), to improve performance, a mistake was made which effected Speciality Fine Food Fair 2019. This has been rectified and all services are back online. Further work has been put on-hold during this key period and we will be reviewing the incident internally.
15:45 - We're continuing to see and receive reports of poor performance throughout registration forms and background import/export tasks. We're focusing on determining the root cause of the issue and will continue to provide updates as this progresses.
30th August 2019
10:00 - Serious performance issues and intermittent outage affecting Speciality Fine Food Fair 2019 registration.
10:20 - Pipedrive integration failures causing persistent spike in write activity preventing registration form activity from getting required lock to write data. Pipedrive integration has been temporarily disabled following current executions to isolate issue.
10:40 - Background task workers temporarily disabled for Fresh Montgomery to isolate issue.
10:50 - Performance has improved and no failures detected in several minutes. We are beginning to provision additional resource and will be restoring background workers throughout the rest of the day, which will have backlogs to work through.
11:02 - Migrating IHS 2019 to separate database architecture to isolate issue.
11:04 - Finished migrating IHS 2019 to separate database architecture to isolate issue.
13:30 - An increase in traffic across our services, combined with higher than usual duration of response, is causing performance degradation now affecting all clients. Registrations are still being accepted and processed. Our plan to provision additional resource to address this is still being enacted and we expect to resolve the issues this evening.
15:00 - Demand now causing intermittent outage affecting all clients, background task workers temporarily disabled for all users to mitigate the issue.
15:30 - Background task workers re-enabled. Additional resource now at configuration management stage. We expect this to be completed and work to begin being reallocated by 16:30. Full resolution expected later this evening.
17:21 - Started gateway and firewall maintenance, expected to last 2-3 minutes.
17:24 - Gateway and firewall maintenance complete.