Server HMW35 hardware issues being fixed (Resolved)

UPDATE Dec 28th 9:05am Pacific Time

The server is now rebooted to the December 23rd backup.  If the services are slower than normal this is because of the high load of all the service functions starting up.

We apologies for any inconvenience. Stay tuned and we appreciate your understanding.


— Management

 

---------------------------------------------------

 

UPDATE Dec 28th 8:57am Pacific Time

The server is now doing a verify on the data that was restored.  Once this portion of the restore is completed the server will reboot back into normal mode and all should be up and running again.

We apologies for any inconvenience. Stay tuned and we appreciate your understanding.


— Management

 

---------------------------------------------------

 

UPDATE Dec 28th 8:26am Pacific Time

The server is still doing the restore from the December 23rd backup.  Once the restore is completed the server will reboot back into normal mode and all should be up and running again.

At this time the restore is at 91%

We apologies for any inconvenience. Stay tuned and we appreciate your understanding.


— Management

 

---------------------------------------------------



UPDATE Dec 28th 1:36am Pacific Time

The server is still doing the restore from the December 23rd backup.  Once the restore is completed the server will reboot back into normal mode and all should be up and running again.

At this time the restore is at 38%

We apologies for any inconvenience. Stay tuned and we appreciate your understanding.


— Management

 

---------------------------------------------------



UPDATE Dec 27th 10:25pm Pacific Time

The server is still doing the restore from the December 23rd backup.  Once the restore is completed the server will reboot back into normal mode and all should be up and running again.

We apologies for any inconvenience. Stay tuned and we appreciate your understanding.


— Management

 

---------------------------------------------------

Those on server HMW35 may have noticed slow performance and intermittent connections affecting services on this server including email and website.


This was due to failing hard drives giving errors. So new hard drives were installed and data cloned from data on the live server.


Once this was done, the tech team then noticed that the hard drive controller was also failing so they replaced the controller card and cables for good measure.


Once this was done, we noticed that there was data corruption due to these bad parts so we are in the process of restoring data from the December 23rd backup which has no errors. This will bring the websites and email back to this restore point so you may be seeing email that you may have already read as well as may be missing email that you may not have downloaded yet.


We will keep this report page up to date until everything is back to normal.


Starting the beginning of January 2017 everyone on this server will be migrated to a brand new and faster server with an up to date operating system as well as newer security advancements. Like any computer, drives eventually fail and parts wear out and operating system need to be updated.


We apologies for any inconvenience. Stay tuned and we appreciate your understanding.


— Management

 

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW35
Priority - Medium
Date - 12/27/2016 20:30
Last Updated - 01/09/2017 22:03

Server HMW35 in a restore process (Resolved)

Server HMW35 is currentlly being restored to this mornings backup due to a operating system failure.  We will place updates here when we have new information.


*********  UPDATE 10/14/16 6:39 PM PACIFIC TIME *********

The current restore status is showing aproxamatelly 5 hours remaining.  Once the restore has completed the server will do a final verification of the files.

*********  UPDATE 10/15/16 12:46 PM PACIFIC TIME *********

The server has been restored to the backup.  All services are back up and running.

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW35
Priority - Medium
Date - 10/14/2016 16:15
Last Updated - 10/15/2016 01:09

Momentary loss of power to Rack for HMW35 aff (Resolved)

UPDATE: 06/18/2016 4:23am Pacific Time:
Server HMW35 has restored to the June 13th backup.  The server is doing background maintence so things may be a bit slow but the websites and mail are back online and running.  Please check back with this page for any new updates.

————


UPDATE: 06/17/2016 11:10pm Pacific Time:
Server HMW35 has aproxamatelly 5 hours left restoring to the June 13th backup.  Once the server has been restored, it will do a fresh reboot and all the websites and email will be rolled back to June 13th.  Please check back with this page for any new updates.

————


UPDATE: 06/16/2016 8:27am Pacific Time:


Due to the power outage and subsequent file corruption on various parts of the server, on Friday June 17th 2016 after 7:30pm Pacific Time the support team will be rolling back (restoring) server HMW35 to the June 13th backup.


What this means and what do you need to do:


1) Please do not make any changes to your website until the restore process has completed.


2) Please make sure that you have downloaded all of your email to your desktop computer even if you only use the webmail interface. If you are using IMAP, please make sure that all email in the INBOX AND SENT FOLDER is moved to the local portion of your mail programs computer. This insures that you do not loose the email of to this point in the IMAP and SMTP area.


3) Please turn off email checking starting Friday after 7:30pm Pacific Time as the email server will not respond during the restore.


4) This means any changes to websites or email will be rolled back to the June 13th backup.


5) Please make sure that you have your alternate email address on file with us. This can be updated by logging in to http://supportdatainc.com


6) PLEASE NOTE that part of the hosting services with Support Data Inc includes server restores from the backups at no charge. Will will attempt to make this process happen as fast as possible. However, we will not have an estimate time when the will be completed.


7) Please check back with the following announcements page for updates: http://supportdatainc.com/myaccount/announcements.php?id=54

————

UPDATE: 06/15/2016 6:10am Pacific Time:
While we have restored most of the affected websites, we are still working on several others where the MySQL databases will be restored from backups. Services may be intermitted. We will keep you posted here.

————

UPDATE: 06/14/2016 4:40pm Pacific Time:
We are still working on the issues which seams to also have affected some communication equipment as well. Services will be intermitted. We will keep you posted here.

————

On 06/14/2016 at 6:07 AM CST, while switching between redundant power sources for a scheduled maintenance at Dal05 server room 02, PDU4 encountered an issue which resulted in momentary loss of power. PDU4 supplies power to a subset of racks in server room 2 and only those racks were negatively impacted. Tech Operations personnel were immediately notified and building management proceeded to back out of the maintenance. Power was restored within minutes and at this time a majority of bare metal dedicated servers are back online. Tech support staff is working to restore service to a handful of bare metal systems that require more in depth troubleshooting. Support Data Inc is still working to restore service to all customer VSIs. Building management and equipment vendors are on site investigating the root cause of the issue encountered by PDU4. Currently, PDU4 is stable and operating in a bypass state. No additional work will be performed on electrical equipment until the root cause of this issue is ascertained.

Update: Support Data Inc continues to restore all customer VSIs. A majority of customer VSIs are online and support staff is working to resolve a handful of customer VSIs that are still offline.

Update: Building management is close to reaching a conclusion on the root cause for the event. Support Data Inc is coordinating to incorporate this into the RCA as soon as it is available.

Update: PDU4 remains stable and is operating in a bypass state until further details are available from the root cause investigation. Support Data Inc is coordinating with building management to complete a formal RFO that outlines the root cause of the incident.

 

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW35
Priority - High
Date - 06/14/2016 11:18
Last Updated - 06/18/2016 04:25

Network issues affecting server HMW35 (Resolved)

UPDATE:

A network router had failed and was quicklly replaced with a different router.  Traffic may be slow as traffic catches up for this devise.  But all 

--------------------------------------------------------

There is a network issue affecting server HMW35.  Technicians are on site looking into the issue.  

An update will be placed here once we have more information.

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW35
Priority - Critical
Date - 05/25/2015 20:28
Last Updated - 05/25/2015 20:53

Private Network Bandwidth Capacity Augment in (Resolved)

=======================================================================
Private Network Bandwidth Capacity Augment in DAL05 [Event Id:15613740] [12/19/2014]
Date: Friday, December 19, 2014 (12/19/2014)
Start Time: 06:00 AM UTC
End Time: 10:00 AM UTC
Services affected: Private Network Connectivity
Location: DAL05
Duration: 4 hours

======================================================================
In order to accommodate the continued growth and customer traffic demands on the private back-end network within the DAL05 datacenter, Network Engineers will be performing a maintenance to augment capacity between the BCRs (back-end customer routers) and MBRs (master back-end routers).  The work is scheduled to begin at 19-Dec-2014 06:00 UTC and complete by 19-Dec-2014 10:00 UTC.

While this maintenance window is scheduled for 4 hours, the actual network changes are only anticipated to take a few minutes per router.  Additional time has been allotted to permit for network verification testing while the changes are stepped through one router at-a-time.

During this maintenance, new additional network links are being added to the routers and customer traffic shifted over to them in a graceful fashion.  Therefore, the work is designed such that no customer disruption is anticipated.  Customer applications which are highly sensitive to sub-second variations in latency may experience a very brief impact as the traffic shifts occur.

Again, NO downtime is expected on the private network.
======================================================================

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW35
Priority - Low
Date - 12/19/2014 06:00 - 12/18/2014 23:27
Last Updated - 12/16/2014 12:25

Security Upgrades to server hmw35 (Resolved)

For accounts hosted on server hmw35 we will be doing security upgrades to the server today Wednesday Oct 8th 2014 and Thursday Oct 9th 2014.

For any accounts that have old software packages running that customers have not been keeping up to date, this could cause your old software (website) to break.  

If this happens, please open a support ticket here: https://supportdatainc.com/myaccount/submitticket.php so we may go over your options.

- Server Administrator

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW35
Priority - Medium
Date - 10/08/2014 12:54
Last Updated - 10/09/2014 15:13

Emergency Service update for server HMW35 (Resolved)

We are working on a Emergency Service update for server HMW35.  Server should be back on line shortlly.

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW35
Priority - Critical
Date - 04/20/2014 20:23 - 04/20/2014 20:55
Last Updated - 04/20/2014 21:05

Emergency Server Maintenance (Resolved)

We are doing an emergency server maintence to server hmw04.  We will update this feed when this server is brought back online.

Effects - This issue does not effect your relevant products/services
Affecting Server - HMW04
Priority - Critical
Date - 12/17/2013 09:37
Last Updated - 12/18/2013 22:22

Scheduled Maintenance [6410399] Firmware Upgr (Resolved)

=======================================================================
Scheduled Maintenance [6410399] Firmware Upgrades - Multiple Routers in DAL01 [10/10/2013]
Date: Thursday, October 10, 2013 (10/10/2013)
Start Time: 05:00 AM UTC
End Time: 09:00 AM UTC
Services affected: Server connectivity and back-end services
Location: DAL01
Duration: 4 hours

===================================================
Network Engineers, working in tandem with the hardware vendor, have
identified an issue on customer routers which provides connectivity for
your server(s) and back-end services. To prevent an unplanned outage
resulting in significant downtime, Engineers will be performing firmware
upgrades during a scheduled maintenance window.

During this maintenance, all servers in this datacenter will initially
experience a disruption in back-end services on the private network, such
as DNS resolution, iSCSI, NAS, data backup, and SSL/PPTP/IPSec managed VPN
access to servers while the MBR (master back-end router) is rebooted for
upgrade. Servers in the first "Pod" (behind fcr01/bcr01) will also
experience a disruption of back-end private and front-end public network
connectivity, and back-end private network connectivity to servers in
other pods will also be disrupted the initial step of this maintenance.

Afterward, servers in other Pods (behind the respective router pairs
listed below) will experience a disruption of both back-end private and
front-end public, network connectivity as those routers are rebooted in
the following sequence:

   mbr01(a/b).dal01 + fcr01.dal01 + bcr01.dal01 
   fcr02.dal01 + bcr02.dal01 
   fcr03.dal01 + bcr03.dal01 
   fcr04.dal01 + bcr04.dal01 
   fcr05.dal01 + bcr05.dal01 

In addition to the router maintenance, SAN-based CCI and VPX instances
will be shutdown/paused in order to maintain data integrity due the fact
that they rely on storage subsystems that reside on the backend network.
Once the routers have returned to service, all CCIs will be booted up and
brought back online.

While the maintenance window is set for 4 hours from 10-Oct-2013 5:00 - 09:00 UTC
(12:00AM - 4:00AM 10/10/2013 CDT datacenter local time), we expect the initial
services and first Pod disruption to last less than 30 minutes, and
disruption to each subsequent pod to last less than 30 minutes, as well.
We expect a disruption of up to 120 minutes for SAN-based CCI and
VPX instances.

===================================================

If you have any problems after this time frame with regard to connectivity,
or if you have any questions regarding the maintenance at any point, please
open a new ticket in the customer portal referencing this ticket. This ticket
is not being tracked by personnel and will be only used to update status of this
maintenance.

We appreciate your patience during this work and welcome any feedback.

Thank you,
Network Engineering
=======================================================================

Effects - This issue does not effect your relevant products/services
Affecting Other - Firmware Upgrades - Multiple Routers in DAL01 [10/10/2013]
Priority - Medium
Date - 10/10/2013 05:00 - 10/10/2013 09:00
Last Updated - 10/19/2013 17:42

 

rss View RSS Feed