The filesystem on Web182 has gone into a read-only state. We are currently rebooting the server, and we will be running a filesystem check as soon as it is up. We will provide progress updates as soon as we have more information.
2011-10-28 12:18 UTC: The fsck is at 25%
2011-10-28 13:05 UTC: The pass 1A of fsck is done. Pass 1B started.
2011-10-28 14:31 UTC: The filesystem is fixed and server is back to operational status.
The filesystem on Web38 has gone into a read-only state. We are currently rebooting the server, and we will be running a filesystem check as soon as it is up. We will provide progress updates as soon as we have more information.
2011-10-24 17:23 UTC: We’re bring the server online in a rescue environment to run the fsck on the server now.
2011-10-24 17:57 UTC: The fsck is currently running.
2011-10-24 18:44 UTC: The fsck is still running.
2011-10-24 19:09 UTC: The fsck is finished and the server is back online.
We are experiencing MySQL issues on web193 relating to InnoDB tables which are affecting database server stability. We are working on restoring reliable MySQL functionality at this time.
Physical disk I/O throughput data has suggested that at the issue is occurring in part at the hardware level; as such, we are performing a RAID controller and disk swap. The RAID is currently rebuilding.
2011-10-20 06:25 UTC: MySQL is back and responsive. InnoDB tables have been made read-only pending further work to be performed after RAID rebuild completes.
2011-10-21 04:00 UTC: We will be taking the server down shortly for a full OS reload/restore from backup since the RAID is not rebuilding properly.
2011-10-24 15:17 UTC: We were able to change the hard drive on the server and have the RAID rebuild fully without reloading the OS. We’re verifying the state of MySQL and will update this post once we’ve verified MySQL is fully operational.
2011-10-25 05:35 UTC: MySQL is operational; we will now proceed to export existing databases to perform consistency maintenance. We will update this post once the databases have been restored.
2011-10-25 08:45 UTC: The databases have been restored and the server is now functioning properly again. All databases are readable and writeable now.
Web300 will be taken offline for a routine file system check on Tuesday, 18 October 2011 at 10:00 p.m. GMT. The maintenance should last approximately 1 hour. We will update this post as maintenance progresses.
2011-10-17 23:37 UTC: We’ve been able to unmount and run the file system check on the drive that required it with the server still running so we’re canceling this maintenance window.
Dweb78 will be taken down for a routine hard drive replacement on October 18th between 10PM UTC and 2AM UTC (October 19). Expected downtime will be less than one hour. We will update this post as maintenance progresses.
2011-10-18 05:42 UTC: Everything is operating normally.
Web108 will be taken down for scheduled maintenance work on October 9th during the hours of 01:00 – 05:00 CDT (server time) for a RAID controller firmware upgrade. The downtime should not be more than one hour. We will update this post as maintenance progresses.