[Fixed] MySQL down on Web344

Posted in Problems by

The MySQL service is currently down on Web344. We’re working to restore service and will update this post when we have more information.

2012-09-10 23:23 UTC: The MySQL service is up in a partial read-only state at this time (InnoDB data cannot be modified).

2012-09-11 00:04 UTC: We’re backing up all MySQL data at this time and will restore it to a clean MySQL installation.

2012-09-11 01:49 UTC: We’re about 50% through restoring the MySQL data.

2012-09-11 02:18 UTC: All MySQL data has been restored and the MySQL service is operating normally.


[Fixed] MySQL problems on Web193 – Emergency Maintenance

Posted in Problems by

We’re investigating a problem with MySQL on Web193 and will update this post when we have more information.

Update: [20:13 CDT] – Our investigation into the MySQL problems has led to some emergency maintenance on the server as we look for the root cause. We’ll update this post as we get more information.

Update: [20:58 CDT] – The server is now back online and responding to requests. We are continuing to monitor the situation closely to verify that the server stays stable.

Update: [21:06 CDT] – We’ve taken the server mostly offline again to verify the integrity of the MySQL databases.

Update: [22:52 CDT] – We’ve found and repaired the error. We discovered some of the MySQL databases had been corrupted. We’ve run a check all on the databases and restored most of the databases that had become corrupted. Due the nature of the database errors some databases couldn’t be backed up and weren’t restored. We’ve restored the databases that were corrupted and could be restored from our latest backups.

Update: [05:36 CDT] – The problems with MySQL on Web193 are back. We’ve temporarily disabled InnoDB in MySQL while we are working on fixing the problem.

Update: [11:51 CDT] – The InnoDB engine is back up but all┬áInnoDB tables are currently read-only. We are sill working on restoring full functionality.

Update: [00:30 CDT] – We are about to get the whole chassis for the server swapped by our data center.

Update: [07:06 CDT] – All of the new hardware is in place.┬áInnoDB tables are going offline for a few minutes, hopefully for the last time, so that any remaining corrupted data can be fixed.

Update: [07:11 CDT] – All MySQL tables are now back online, free of any corruption and fully writeable.