Momentary loss of power to Rack for HMW35 affected several websites
UPDATE: 06/18/2016 4:23am Pacific Time:
Server HMW35 has restored to the June 13th backup. The server is doing background maintence so things may be a bit slow but the websites and mail are back online and running. Please check back with this page for any new updates.
UPDATE: 06/17/2016 11:10pm Pacific Time:
Server HMW35 has aproxamatelly 5 hours left restoring to the June 13th backup. Once the server has been restored, it will do a fresh reboot and all the websites and email will be rolled back to June 13th. Please check back with this page for any new updates.
UPDATE: 06/16/2016 8:27am Pacific Time:
Due to the power outage and subsequent file corruption on various parts of the server, on Friday June 17th 2016 after 7:30pm Pacific Time the support team will be rolling back (restoring) server HMW35 to the June 13th backup.
What this means and what do you need to do:
1) Please do not make any changes to your website until the restore process has completed.
2) Please make sure that you have downloaded all of your email to your desktop computer even if you only use the webmail interface. If you are using IMAP, please make sure that all email in the INBOX AND SENT FOLDER is moved to the local portion of your mail programs computer. This insures that you do not loose the email of to this point in the IMAP and SMTP area.
3) Please turn off email checking starting Friday after 7:30pm Pacific Time as the email server will not respond during the restore.
4) This means any changes to websites or email will be rolled back to the June 13th backup.
5) Please make sure that you have your alternate email address on file with us. This can be updated by logging in to http://supportdatainc.com
6) PLEASE NOTE that part of the hosting services with Support Data Inc includes server restores from the backups at no charge. Will will attempt to make this process happen as fast as possible. However, we will not have an estimate time when the will be completed.
7) Please check back with the following announcements page for updates: http://supportdatainc.com/myaccount/announcements.php?id=54
UPDATE: 06/15/2016 6:10am Pacific Time:
While we have restored most of the affected websites, we are still working on several others where the MySQL databases will be restored from backups. Services may be intermitted. We will keep you posted here.
UPDATE: 06/14/2016 4:40pm Pacific Time:
We are still working on the issues which seams to also have affected some communication equipment as well. Services will be intermitted. We will keep you posted here.
On 06/14/2016 at 6:07 AM CST, while switching between redundant power sources for a scheduled maintenance at Dal05 server room 02, PDU4 encountered an issue which resulted in momentary loss of power. PDU4 supplies power to a subset of racks in server room 2 and only those racks were negatively impacted. Tech Operations personnel were immediately notified and building management proceeded to back out of the maintenance. Power was restored within minutes and at this time a majority of bare metal dedicated servers are back online. Tech support staff is working to restore service to a handful of bare metal systems that require more in depth troubleshooting. Support Data Inc is still working to restore service to all customer VSIs. Building management and equipment vendors are on site investigating the root cause of the issue encountered by PDU4. Currently, PDU4 is stable and operating in a bypass state. No additional work will be performed on electrical equipment until the root cause of this issue is ascertained.
Update: Support Data Inc continues to restore all customer VSIs. A majority of customer VSIs are online and support staff is working to resolve a handful of customer VSIs that are still offline.
Update: Building management is close to reaching a conclusion on the root cause for the event. Support Data Inc is coordinating to incorporate this into the RCA as soon as it is available.
Update: PDU4 remains stable and is operating in a bypass state until further details are available from the root cause investigation. Support Data Inc is coordinating with building management to complete a formal RFO that outlines the root cause of the incident.
Tuesday, June 14, 2016