Update on Performance Issues

All users will be aware that on the evening of Tuesday 27th September, PharmOutcomes started having significant performance issues. This was caused by a routine manual process by our third-party data centre failing that left us yesterday with only one-quarter of our processing power. Our fallback resilience systems and architectural design ensured that no data was lost when this happened.

To minimise disruption as much as possible, a number of platform features were disabled and this was communicated to all our users; however, at points during the day, the platform was overwhelmed by valid requests leaving users unable to use it for a few minutes or experiencing slow response times.

The EMIS Pinnacle team worked through the day to diagnose and fix the issue, which we have experience of and have remedied swiftly in the past; however, on this occasion, those remediations did not work, even with the database manufacturer's support. At 8pm last night we placed the platform in maintenance mode and the team continued through the night. At 4am we managed to restore one database node which has returned us to one-half of our normal processing power. Unfortunately, the remaining two quarters are not yet restored, despite best endeavours, but have made the decision to re-enable the platform at 7am this morning operating at one-half capacity.

We expect operating performance today to be an improvement on yesterday but not to the standard that we all expect from PharmOutcomes. We will be managing functionality, as we did yesterday, dependent upon performance and user experience, which we monitor constantly.

To avoid reduced performance through the day for our users, we will not be attempting to restore full capacity because the process would reduce the availability of computing resource to the levels we saw yesterday. We will be working through the day to further investigate why two database nodes are not re-synchronising and start work this evening on restoring them once more. We plan to do this work whilst the platform is still operational, as the load is not as significant in the evenings.

We do realise that our users rely upon the smooth operation of PharmOutcomes throughout their day, particularly at this time of the year and once again apologise for the frustration this must be causing. Please be assured that restoration of performance for you  remains our top priority.