MSO: Several Services Affected [Resolved]

From 16:23 until 17:03 there was an outage which affected several of our datacentre services. VOIP, some web and email services are know to have been affected. Unfortunately this also took down our phone systems, resulting in calls being diverted to staff mobiles. Cause: one of our virtualisation head units rebooted unexpectedly, taking down all related services. We are still investigating the cause. All services are now back online. If you are still experiencing any issues, please contact support@gladserv.com or call 0131 510 0340.

Major Service Outage: various services affected

We're having some network issues with out upstream that is affecting some services. Some web and email services are affected, as well as a small number of customer VMs. Will update this space as soon as we have more information.

Major Service Outage: DDOS attack

Several BT wholesale customers have reported line drops. We are also seeing packet loss to our data centre servers which is affecting many services, including our phones. Apologies to those who are trying to get through but can't.

Server Maintenance This Weekend

We will be conducting some server and network maintenance this weekend, on Sunday 9th November 2014 between 1500-1900 hrs GMT. There may be some distruptions to our shared hosting platform, email services and some customer servers during this period. We will try to keep any downtime to a minimum. We apologise for the short notice, but these are urgent datacentre works which are outside our control. We will update this blog with any relevant info.

IMAP Mail outage (alpha)

Earlier this morning we had an outage on one of our IMAP mailstores. This will have affected customers with email accounts on our "alpha" mailstore. Other customers and services were unaffected. The server is back online and all services are now functioning normally. Please contact support if you are still experiencing problems.

Routing Problem Affecting Some Services (resolved)

Due to a routing issue with one of our upsteam providers a small number of services may be currently unavailable. Engineers are working to resolve this issue as quickly as possible. We will post back here when more information is available.

Update:

This was caused by a power failure in one of the Manchester datacentres. Most systems failed over correctly, but some customer virtual servers were affected. These customers are being contacted individually.

Authentication Service Outage (resolved)

There was a prolonged outage this morning of our authentication and directory service which affected many of our systems, most noticeably email.

Due to a double hardware failure, we scheduled emergency maintenance for two of our servers at 8am this morning, which took part of our virtualisation plaform offline. Affected customers were notified individually.

Unfortunately, for reasons we are still investigating, this led to a cascading failure which took our authentication and directory services offline, which affected other services, including email.

Thank you to all our customers for your patience throughout this extended outage. We are still investigating, and will take remedial steps based on what we find. Our authentication and directory services are critical to our infrastructure and are meant to failover seamlessly in the event of a single failure - today they did not.

Please accept our apologies for any inconvience caused.