WebFaction: network maintenance affecting several EU servers on July 1, 2015

Posted in Scheduled downtime by

Our upstream provider will be performing network maintenance on July 1, 2015 between 22:00 and 03:00 UTC the following morning. The following servers will be affected:

Dweb186, Web209, Web210, Web211, Web217, Web218, Web223, Web224, Web225, Web303, Web304, Web305, Web306, Web314, Web316, Web317, Web327, Web331, Web332, Web333, Web334, Web357, Web358, Web359, Web364, Web370, Web371, Web381, Web383, Web385, Web392, Web394, Web395, Web398, Web403, Web412, Web416, Web420, Web421, Web427, Web431, Web433, Web435, Web437, Web441, Web447, Web448, Web451, Web456, Web457, Web459, Web461, Web464, Web466, Web478, Web480, Web483, Web488

Connectivity to the above servers will be interrupted for upwards of 20 minutes during the maintenance window.

We apologize in advance for the inconvenience. Should you have any questions about this maintenance, feel free to open a support ticket or email support@webfaction.com and our support team will be happy to help.


[Done]Scheduled maintenance on Web381, February 13th, 2013.

Posted in Scheduled downtime by

Web381 will be taken down Wdnesday February 13th between 09:00 UTC and 12:00 UTC for a RAID adapter swap. We will update this post as maintenance progresses.

2013-02-13 12:24 UTC We had to start an offline RAID rebuild, it is now at 72%.

2013-02-13 15:45 UTC After rebuilding the RAID the OS was not able to boot up, we’re currently investigating the issue and considering a hardware replacement.

2013-02-13 17:24 UTC: The RAID controller is causing the kernel to panic on boot. We’ve currently got the server in a rescue environment and we are copying all data off of the current hard drives. Once this is finished we will have the entire server chassis swapped and begin restoring the data to the machine.

2013-02-13 18:33 UTC: The server chassis is currently being swapped.

2013-02-13 19:33 UTC: The relevant hardware has been swapped and we are working to bring the server back online in order to verify the hardware swap has corrected the problem.

2013-02-13 21:58 UTC: The new hardware has not helped the cause of the kernel panics. We’ve decided to bring the machine back online in a rescue environment to pull as much data off the machine as possible then have all of the hardware switched and restore from the backups we are taking now.

2013-02-14 00:14 UTC: We’ve replaced the entire machine including the hard disks. We are now preparing an OS reload.

2013-02-14 03:06 UTC: We’ve restored the mysql and postgres databases on the machine and all $HOME directories. We’re now vigorously testing the machine to verify the complete hardware swap has corrected the problem we were seeing.

2013-02-14 04:02 UTC: We’ve restored all cron jobs and the SSH fingerprint information so you will not see any warnings that the host key has changed. We’re taking the machine through one final test before allowing logins.

2013-02-14 04:05 UTC: The final test has completed and the server is now back online and functioning normally. We’ll continue to closely monitor the machine throughout the next few hours.


[Done]Web381 down

Posted in Downtime by

Web381 stopped responding and has not come back online following several attempted reboots. We’re working to restore service at this time and will update this post when we have more information.

2013-02-08 14:52 UTC: We’re still working on server to bring it online
2013-02-08 16:04 UTC: We’re reloading the server in order to bring it online, we were able to get all data from the server prior reload, so we don’t expect any data loss.

2013-02-08 16:40 UTC:The server is now back online and we are monitoring it closely.