Web233 stopped responding, and so we have rebooted into a rescue console and we are performing a filesystem check.  We will update this blog as maintenance progresses.

Update 23:09 UTC: Some of the data on the server appears to be corrupted so we are migrating the server to new hardware. The new server is already set up and we have started transferring data to it. The new server’s main IP is 108.168.213.86

Update 01:00 UTC: The data transfer is still ongoing

Update 03:45 UTC: The data transfer is still ongoing

Update 06:35 UTC: The transfer of the home directories is still ongoing

Update 10:07 UTC: All MySql and PostgreSQL databases have now been recovered on the new server. Home directories for usernames starting with the letters “a” to “q” have been recovered and we are working on recovering the rest of the home directories. The main services (MySql, PostgreSQL, Apache, Nginx) have been started on the new server.

Update 16:05 UTC: We are finishing to restore the last few home directories (starting with “w”)

Update 17:46 UTC: All data has been recovered and service is back to normal. If you notice any problem with your sites open a ticket and we’ll look into it asap. Note that if you use non-WebFaction DNS servers for your domains you will have to update them to point your domains at the new IP (108.168.213.86)