We’re investigating a problem with MySQL on Web193 and will update this post when we have more information.
Update: [20:13 CDT] – Our investigation into the MySQL problems has led to some emergency maintenance on the server as we look for the root cause. We’ll update this post as we get more information.
Update: [20:58 CDT] – The server is now back online and responding to requests. We are continuing to monitor the situation closely to verify that the server stays stable.
Update: [21:06 CDT] – We’ve taken the server mostly offline again to verify the integrity of the MySQL databases.
Update: [22:52 CDT] – We’ve found and repaired the error. We discovered some of the MySQL databases had been corrupted. We’ve run a check all on the databases and restored most of the databases that had become corrupted. Due the nature of the database errors some databases couldn’t be backed up and weren’t restored. We’ve restored the databases that were corrupted and could be restored from our latest backups.
Update: [05:36 CDT] – The problems with MySQL on Web193 are back. We’ve temporarily disabled InnoDB in MySQL while we are working on fixing the problem.
Update: [11:51 CDT] – The InnoDB engine is back up but all InnoDB tables are currently read-only. We are sill working on restoring full functionality.
Update: [00:30 CDT] – We are about to get the whole chassis for the server swapped by our data center.
Update: [07:06 CDT] – All of the new hardware is in place. InnoDB tables are going offline for a few minutes, hopefully for the last time, so that any remaining corrupted data can be fixed.
Update: [07:11 CDT] – All MySQL tables are now back online, free of any corruption and fully writeable.