We will be performing a scheduled hardware change on Web183 Saturday, September 24 for approximately 1 hour during the 4 hour window of 0:00 CDT – 04:00 CDT. We will update this post as maintenance progresses.
Update: [0:30 CDT] – The server is now being taken offline for the hardware change.
Update: [01:10 CDT] – The server is back online and responding to requests.
There is a network issue at our Amsterdam data center, resulting in Web216, Web217, and Web218 being unreachable for the moment. We are working with our upstream provider to resolve the problem as soon as possible.
Update: [13:12 UTC] – Web217 and Web218 are back up. We are still working on bringing up Web216 as well.
We’re investigating a problem with MySQL on Web193 and will update this post when we have more information.
Update: [20:13 CDT] – Our investigation into the MySQL problems has led to some emergency maintenance on the server as we look for the root cause. We’ll update this post as we get more information.
Update: [20:58 CDT] – The server is now back online and responding to requests. We are continuing to monitor the situation closely to verify that the server stays stable.
Update: [21:06 CDT] – We’ve taken the server mostly offline again to verify the integrity of the MySQL databases.
Update: [22:52 CDT] – We’ve found and repaired the error. We discovered some of the MySQL databases had been corrupted. We’ve run a check all on the databases and restored most of the databases that had become corrupted. Due the nature of the database errors some databases couldn’t be backed up and weren’t restored. We’ve restored the databases that were corrupted and could be restored from our latest backups.
Update: [05:36 CDT] – The problems with MySQL on Web193 are back. We’ve temporarily disabled InnoDB in MySQL while we are working on fixing the problem.
Update: [11:51 CDT] – The InnoDB engine is back up but all InnoDB tables are currently read-only. We are sill working on restoring full functionality.
Update: [00:30 CDT] – We are about to get the whole chassis for the server swapped by our data center.
Update: [07:06 CDT] – All of the new hardware is in place. InnoDB tables are going offline for a few minutes, hopefully for the last time, so that any remaining corrupted data can be fixed.
Update: [07:11 CDT] – All MySQL tables are now back online, free of any corruption and fully writeable.
One of our datacenters will be performing maintenance work on its network infrastructure this Sunday between 05:00 UTC and 13:00 UTC.
The following servers will likely experience periods of high latency or packet loss and may experience complete loss of connectivity for brief moments during that period: Dweb74, Dweb75, Dweb76, Dweb77, Dweb78, Dweb79, Dweb81, Dweb84, Mail9, Mailbox6, Mailbox7, Web196, Web197, Web198, Web199, Web200, Web201, Web202, Web203, Web204, Web205, Web206, Web207, Web212, Web213, Web214, Web215, Web219, Web220, Web221, Web222, and the control panel.