[Fixed] Web223 down

Posted in Downtime by

Web223 did not come back after a reboot. We are investigating the issue.

[Apr 15 01:51 UTC]  The server has a failed disk and  we are working with the datacenter to fix that before taking any further steps.

[Apr 15 02:09 UTC]  The disk has been replaced and the RAID is rebuilding but /home has FS errors too so we are fscking it.

[Apr 15 03:22 UTC]   We have stopped the fsck since it was really slow due to the RAID rebuild. We are letting the RAID rebuild before continuing with it.

[Apr 15 03:22 UTC]   We have stopped the fsck since it was really slow due to the RAID rebuild. We are letting the RAID rebuild before continuing with it.

[Apr 15 04:13 UTC] The RAID rebuild was going really slow so we have asked the datacenter to update the RAID firmware to see if that speeds up the rebuild.

[Apr 15 06:39 UTC] We have lowered the rebuild priority so that we can complete the fsck, it is at 13% now.

[Apr 15 08:23 UTC] The server was unintentionally rebooted by the datacenter while the fsck was running as they had to replace the drive since it stopped rebuilding. We have started the fsck again now.

[Apr 15 08:52 UTC] The fsck is at 64% now.

[Apr 15 10:39 UTC] The fsck on the disk didn’t succeed so we are setting up a new server now and restoring data to that server now.

[Apr 15 13:48 UTC] The new server is setup and we are restoring the data now.

[Apr 15 15:10 UTC] The data restoration is at about 25%. Note that the server’s main IP is changing from 62.212.65.160 to 5.153.9.62. If you’re using WebFaction’s DNS servers for your domains you don’t need to do anything but if you’re using third-party DNS servers then you will have to update your DNS records.

[Apr 15 18:42 UTC] Most data is restored and we’re working on postgresql databases.

[Apr 16 00:29 UTC] We’re still in the process of restoring data to the machine.

[Apr 16 10:01 UTC] The data has now been restored. If you notice any problems with your account open a support ticket and we’ll look into it asap.

-
-

[Fixed] Performance problems with web335

Posted in Downtime by

There are some performance problems with web335.

We are investigating the problem.

2013-03-17 18:43 UTC: We are taking the server down to run fsck on the filesystem.

2013-03-17 20:11 UTC: fsck was not required at this time. Performance issue seems to be under control. We are continuing to monitor for further issues.

2013-03-17 21:25 UTC: Performance issue has been resolved.

-
-

[Fixed] Web334 is offline

Posted in Downtime by

Web334 is offline from a kernel panic. Our sysadmins are looking into this now.

[27/02 20:57 UTC] The kernel has been upgraded and the server rebooted. We are monitoring for any issues.

-
-

[Fixed] Web28 down

Posted in Downtime by

The filesystem on web28 has gone read-only. We are taking the server down to perform and fsck to correct the problem.

2013-01-11 21:45 UTC – The filesystem check has completed and the server is back online.

-
-