WebFaction: network maintenance affecting several EU servers on July 1, 2015

Posted in Scheduled downtime by

Our upstream provider will be performing network maintenance on July 1, 2015 between 22:00 and 03:00 UTC the following morning. The following servers will be affected:

Dweb186, Web209, Web210, Web211, Web217, Web218, Web223, Web224, Web225, Web303, Web304, Web305, Web306, Web314, Web316, Web317, Web327, Web331, Web332, Web333, Web334, Web357, Web358, Web359, Web364, Web370, Web371, Web381, Web383, Web385, Web392, Web394, Web395, Web398, Web403, Web412, Web416, Web420, Web421, Web427, Web431, Web433, Web435, Web437, Web441, Web447, Web448, Web451, Web456, Web457, Web459, Web461, Web464, Web466, Web478, Web480, Web483, Web488

Connectivity to the above servers will be interrupted for upwards of 20 minutes during the maintenance window.

We apologize in advance for the inconvenience. Should you have any questions about this maintenance, feel free to open a support ticket or email support@webfaction.com and our support team will be happy to help.

-
-

[Done]Scheduled maintenance on Web223, May 24th, 2013.

Posted in Scheduled downtime by

Web223 will be taken down for a RAID controller swap Friday, May 24th, 2013 between 07:00UTC and 10:00UTC. We will update this post as maintenance progresses.

2013-05-24 08:20 UTC RAID controller and cables have been replaced, the server is back to operational status.

-
-

[Done]Emergency maintenance on Web223, May 22nd, 2013.

Posted in Downtime by

Web223 filesystem went read-only. We’re taking the server down and run a fsck. We will update this post as maintenance progresses.

2013-05-22 07:46 UTC the server is back at operational status

-
-

[Done]Emergency maintenance on Web223, May 14th, 2013.

Posted in Downtime by

Web223 filesystem is in read-only state. We’re currently taking the server down and run a fsck. We’ll update this post as maintenance progresses.

2013-05-14 10:49 UTC Fsck completed and the server is back to operational status.

-
-

[Fixed] Emergency maintenance on Web223, May 9th, 2013.

Posted in Downtime by

Web223 filesystem went read-only. We’re currently taking the server down to run a fsck. We will update this post as maintenance progresses.

[May  9 10:44 UTC]  The server is back after a quick fsck and is OK now.

-
-

[Done] PostgreSQL upgrade on Web22,Web23,Web223 and Web228

Posted in Scheduled downtime by

We are going to upgrade Postgresql 8.3 / PostGIS 1.3.6 to Postgresql 8.4 / PostGIS 1.5 on the following servers in the

upgrade interval mentioned alongside.

Web22    May 5, 2am to 4am UTC

Web228  May 5, 3am to 5am UTC

Web223  May 5, 4am to 6am UTC

Web23    May 5, 5am to 7am UTC

Postgresql will be unavailable for the duration of the upgrade.

We will update this post as the upgrades progress.

[May  5 03:09 UTC] The upgrade on Web22 is over.

[May  5 04:46 UTC] The upgrade on web223 is over while Web228 is still going on due to the large amount of data.

[May  5 05:20 UTC] The upgrade on Web228 is over.

[May  5 06:50 UTC] The upgrade on web23 is over.

-
-

[Done]Emergency maintenance on Web223, 23rd April, 2013.

Posted in Downtime by

Web223 filesystem shows errors. We’re currently taking the server down to run a FSCK. We will update this post as maintenance progresses.

2013-04-23 10:40 UTC FSCK finished and the server is back at operational status.

-
-

[Fixed] Web223 down

Posted in Downtime by

Web223 did not come back after a reboot. We are investigating the issue.

[Apr 15 01:51 UTC]  The server has a failed disk and  we are working with the datacenter to fix that before taking any further steps.

[Apr 15 02:09 UTC]  The disk has been replaced and the RAID is rebuilding but /home has FS errors too so we are fscking it.

[Apr 15 03:22 UTC]   We have stopped the fsck since it was really slow due to the RAID rebuild. We are letting the RAID rebuild before continuing with it.

[Apr 15 03:22 UTC]   We have stopped the fsck since it was really slow due to the RAID rebuild. We are letting the RAID rebuild before continuing with it.

[Apr 15 04:13 UTC] The RAID rebuild was going really slow so we have asked the datacenter to update the RAID firmware to see if that speeds up the rebuild.

[Apr 15 06:39 UTC] We have lowered the rebuild priority so that we can complete the fsck, it is at 13% now.

[Apr 15 08:23 UTC] The server was unintentionally rebooted by the datacenter while the fsck was running as they had to replace the drive since it stopped rebuilding. We have started the fsck again now.

[Apr 15 08:52 UTC] The fsck is at 64% now.

[Apr 15 10:39 UTC] The fsck on the disk didn’t succeed so we are setting up a new server now and restoring data to that server now.

[Apr 15 13:48 UTC] The new server is setup and we are restoring the data now.

[Apr 15 15:10 UTC] The data restoration is at about 25%. Note that the server’s main IP is changing from 62.212.65.160 to 5.153.9.62. If you’re using WebFaction’s DNS servers for your domains you don’t need to do anything but if you’re using third-party DNS servers then you will have to update your DNS records.

[Apr 15 18:42 UTC] Most data is restored and we’re working on postgresql databases.

[Apr 16 00:29 UTC] We’re still in the process of restoring data to the machine.

[Apr 16 10:01 UTC] The data has now been restored. If you notice any problems with your account open a support ticket and we’ll look into it asap.

-
-

[Done]Emergency maintenance on Web223, June 5th 2012.

Posted in Downtime by

Web223 had one disk failing. After we replaced the failing drive, the filesystem needed a fsck. This is now at 70% done. We will update this post as maintenance progresses.

2012-06-05 15:32 UTC: The RAID controller is showing further issues which we’re now investigating. We will update this post as the situation progresses.

2012-06-05 18:59 UTC: Web223 is back online. The disk array is rebuilding, so load is somewhat elevated at this time, but customer sites should be working (albeit a bit slowly). We’ll update this post when the rebuild is complete and load is normal.

2012-06-05 20:41 UTC: Web223’s RAID array is not rebuilding as normal. We’ll take the server offline again to update/change the RAID controller and firmware. We’ll update this post when the server goes offline.

2012-06-05 23:46 UTC:  We’ll take the server offline in the next few minutes. We’ll update this post as maintenance progresses.

2012-06-05 00:09 UTC: The server is back online and the RAID is rebuilding properly. We’ll keep this post open until the RAID is rebuilt.

2012-06-05 00:26 UTC: The server’s RAID is now rebuilt and the server is working normally at this time.

-
-