[Done] Emergency maintenance on Web182

Posted in Downtime by

The filesystem on Web182 has gone into a read-only state. We are currently rebooting the server, and we will be running a filesystem check as soon as it is up. We will provide progress updates as soon as we have more information.

2011-10-28 12:18 UTC: The fsck is at 25%
2011-10-28 13:05 UTC: The pass 1A of fsck is done. Pass 1B started.
2011-10-28 14:31 UTC: The filesystem is fixed and server is back to operational status.

-
-

[Done] Scheduled maintenenace work on Web165, October 27th

Posted in Downtime by

Web165 will be taken down for a routine hard drive replacement on October 27th at 0900AM UTC. Expected downtime will be less than one hour. We will update this post as maintenance progresses.

2011-10-27 09:31 UTC: The disk has been replaced and the server is back to operational status.

-
-

[Done] Emergency maintenance on Web38

Posted in Downtime by

The filesystem on Web38 has gone into a read-only state. We are currently rebooting the server, and we will be running a filesystem check as soon as it is up. We will provide progress updates as soon as we have more information.

2011-10-24 17:23 UTC: We’re bring the server online in a rescue environment to run the fsck on the server now.

2011-10-24 17:57 UTC: The fsck is currently running.

2011-10-24 18:44 UTC: The fsck is still running.

2011-10-24 19:09 UTC: The fsck is finished and the server is back online.

-
-

[Done]MySQL issues on web193

Posted in Downtime by

We are experiencing MySQL issues on web193 relating to InnoDB tables which are affecting database server stability. We are working on restoring reliable MySQL functionality at this time.

Physical disk I/O throughput data has suggested that at the issue is occurring in part at the hardware level; as such, we are performing a RAID controller and disk swap. The RAID is currently rebuilding.

2011-10-20 06:25 UTC: MySQL is back and responsive. InnoDB tables have been made read-only pending further work to be performed after RAID rebuild completes.

2011-10-21 04:00 UTC: We will be taking the server down shortly for a full OS reload/restore from backup since the RAID is not rebuilding properly.

2011-10-24 15:17 UTC: We were able to change the hard drive on the server and have the RAID rebuild fully without reloading the OS. We’re verifying the state of MySQL and will update this post once we’ve verified MySQL is fully operational.

2011-10-25 05:35 UTC: MySQL is operational; we will now proceed to export existing databases to perform consistency maintenance. We will update this post once the databases have been restored.

2011-10-25 08:45 UTC: The databases have been restored and the server is now functioning properly again. All databases are readable and writeable now.

-
-

[Done] Emergency maintenance on Web7

Posted in Downtime by

We are bringing Web7 down for an emergency file system check; one of our hard drives is reporting that the file system needs to be checked. We will update this post as maintenance progresses.

2011-10-17 23:33 UTC: The server is now back online and responding normally.

-
-

[Done] Scheduled maintenance on Web300, Tuesday October 18

Posted in Downtime by

Web300 will be taken offline for a routine file system check on Tuesday, 18 October 2011 at 10:00 p.m. GMT. The maintenance should last approximately 1 hour. We will update this post as maintenance progresses.

2011-10-17 23:37 UTC: We’ve been able to unmount and run the file system check on the drive that required it with the server still running so we’re canceling this maintenance window.

-
-

[Done] Scheduled maintenenace work on Dweb78, October 18

Posted in Downtime by

Dweb78 will be taken down for a routine hard drive replacement on October 18th between 10PM UTC and 2AM UTC (October 19). Expected downtime will be less than one hour. We will update this post as maintenance progresses.

2011-10-18 05:42 UTC: Everything is operating normally.

-
-

[Done] Scheduled maintenance work on Web47, October 15th

Posted in Downtime by

Web47 will be taken down for a routine chassis swap on October 15th between 12PM UTC and 4PM UTC. Expected downtime will be less than one hour. We will update this post as maintenance progresses.

2011-10-15 00:40 UTC: The maintenance is complete and the server is back online.

-
-

[Done]Scheduled maintenance work on Web45, October 7

Posted in Downtime by

Web45 will be taken down for a disk replacement on October 7th at 15:00PM UTC. The downtime should not be more than one hour. We will update this post as maintenance progresses.

2011-10-07 16:21 UTC: The disk has been replaced and the RAID is now rebuilding.

2011-10-07 16:36 UTC: The rebuild progress is now over 70%.

2011-10-07 17:11 UTC: The server is back online at operational status.

-
-

[Done]Scheduled maintenance work on Web108, October 9

Posted in Downtime by

Web108 will be taken down for scheduled maintenance work on October 9th during the hours of 01:00 – 05:00 CDT (server time) for a RAID controller firmware upgrade. The downtime should not be more than one hour. We will update this post as maintenance progresses.

-
-