Web237’s disk went read-only shortly after the scheduled maintenance so we have taken it into rescue for  an immediate fsck.

We will update this post as the fsck progresses.

[2012-12-19 10:40 UTC] The fsck is upto 10% now.

[2012-12-19 13:06 UTC] The fsck is still running

[2012-12-19 15:58 UTC] fsck is at pass 1C.

[2012-12-19 18:12 UTC] Unfortunately fsck failed to complete. At this point we are going to copy the data to a spare server. We will update this post as the copy progresses.

[2012-12-19 20:06 UTC] The copy of the data to a new server is at 15%

[2012-12-19 21:18 UTC] The copy of the data to a new server is at 25%

[2012-12-19 22:59 UTC] The copy of the data to a new server is still ongoing

[2012-12-20 00:50 UTC] The copy of the data to a new server is at 35%

2012-12-20 02:26 UTC: The copy of the data to a new server is at 45%

2012-12-20 03:14 UTC: The copy of the data to a new server is at 50%

2012-12-20 05:02 UTC: The copy of the data to a new server is at 60%

2012-12-20 09:38 UTC: The copy of the data is still going on.

2012-12-20 10:35 UTC: After hanging for a while the copy is now going at full speed again. It’s at 70%

2012-12-20 11:33 UTC: We are going to start re-enabling websites for accounts whose data has been copied onto the new server. If you would like your websites to be re-enabled as a higher priority open a support ticket.

2012-12-20 13:37 UTC: Some sites are re-enabled on the new Web237 machine and we’re working through the other ones. Please note that the machine’s main IP changed from 108.59.11.111 to 75.126.24.82 so if you’re using external DNS servers you will have to update your DNS records. If you’re using WebFaction’s DNS servers you don’t need to do anything.

2012-12-20 16:49 UTC: Most websites have new been re-enabled. We’re working on re-enabling the last few ones.

2012-12-20 22:19 UTC: All websites have now been re-enabled for several hours. We’re marking this issue as fixed and we’ll email a post-mortem to all Web237 users tomorrow.