We are experiencing MySQL issues on web193 relating to InnoDB tables which are affecting database server stability. We are working on restoring reliable MySQL functionality at this time.

Physical disk I/O throughput data has suggested that at the issue is occurring in part at the hardware level; as such, we are performing a RAID controller and disk swap. The RAID is currently rebuilding.

2011-10-20 06:25 UTC: MySQL is back and responsive. InnoDB tables have been made read-only pending further work to be performed after RAID rebuild completes.

2011-10-21 04:00 UTC: We will be taking the server down shortly for a full OS reload/restore from backup since the RAID is not rebuilding properly.

2011-10-24 15:17 UTC: We were able to change the hard drive on the server and have the RAID rebuild fully without reloading the OS. We’re verifying the state of MySQL and will update this post once we’ve verified MySQL is fully operational.

2011-10-25 05:35 UTC: MySQL is operational; we will now proceed to export existing databases to perform consistency maintenance. We will update this post once the databases have been restored.

2011-10-25 08:45 UTC: The databases have been restored and the server is now functioning properly again. All databases are readable and writeable now.