Wednesday 24 August 2011

Routing Problem Affecting Some Services (resolved)

Due to a routing issue with one of our upsteam providers a small number of services may be currently unavailable. Engineers are working to resolve this issue as quickly as possible. We will post back here when more information is available.

Update:

This was caused by a power failure in one of the Manchester datacentres. Most systems failed over correctly, but some customer virtual servers were affected. These customers are being contacted individually.

Wednesday 10 August 2011

Authentication Service Outage (resolved)

There was a prolonged outage this morning of our authentication and directory service which affected many of our systems, most noticeably email.

Due to a double hardware failure, we scheduled emergency maintenance for two of our servers at 8am this morning, which took part of our virtualisation plaform offline. Affected customers were notified individually.

Unfortunately, for reasons we are still investigating, this led to a cascading failure which took our authentication and directory services offline, which affected other services, including email.

Thank you to all our customers for your patience throughout this extended outage. We are still investigating, and will take remedial steps based on what we find. Our authentication and directory services are critical to our infrastructure and are meant to failover seamlessly in the event of a single failure - today they did not.

Please accept our apologies for any inconvience caused.