[monitoring] Emergency network maintenance affecting several EU servers

Posted in Downtime by

2017-04-27 00:20 UTC: at the present time, emergency network maintenance in our EU datacenter has caused several EU servers to go offline. We’ll update this post as more information becomes available.

2017-04-27 00:29 UTC: connectivity has been restored, we’ll continue to monitor.

-
-

[Fixed] web567 inaccessible

Posted in Downtime by

The server is currently not accessible. We are looking into this and will update as soon as we have further details.

2017-04-26 00:04 UTC: The servers is back online.

-
-

[Fixed] Web560 inaccessible

Posted in Downtime by

2017-04-25 1700 UTC: We are investigating the issue and will update the post as soon as we have further details.

2017-04-25 1730 UTC: The server is back online and serving traffic.

-
-

[Fixed] Web474 inaccessible

Posted in Downtime by

The server is currently not accessible. We are looking into this and will update as soon as we have more details.

2017-04-21 17:37 UTC: The servers is back online.

-
-

[Done] Emergency maintenance on Mailbox10, April 16, 2017

Posted in Downtime by

Mailbox10 will be taken down for an HDD swap and possibly RAID controller replacement,
Sunday April 16th, 2017 between 1:00 UTC and 5:00 UTC.

Sun Apr 16 01:15:32 UTC 2017: The server has been taken offline for maintenance

Sun Apr 16 02:15:32 UTC 2017: The failing drive has been replaced and the RAID is rebuilding (currently 3%).

Sun Apr 16 02:45:34 UTC 2017: The rebuild has failed, which indicates a faulty RAID controller, which is now being replaced.

Sun Apr 16 03:29:52 UTC 2017: The RAID controller has been replaced; now doing build/verify and rebuild.

Sun Apr 16 06:22:50 UTC 2017: The RAID controller initial build/verify is done and new HDD is rebuilding (currently 15%).

Sun Apr 16 07:00:51 UTC 2017: HDD is rebuilding (currently 35%).

Sun Apr 16 08:22:08 UTC 2017: HDD is rebuilding (currently 80%).

Sun Apr 16 08:52:54 UTC 2017: The rebuild has completed and the server is back online.

-
-

[Fixed] Read-only filesystem on Web429

Posted in Downtime by

2017-04-08 00:44 UTC: The filesystem on Web429 is currently read-only. We’re working to restore service at this time and will update this post with more information as it becomes available.

2017-04-08 00:45 UTC: We’re taking the machine down for a filesystem check.

2017-04-08 01:07 UTC: The filesystem check is complete. Some inconsistencies in the filesystem journal were repaired. The server is back online and responding normally at this time.

 

-
-

[Fixed] Incoming mail service disruption

Posted in Problems by

2017-04-03 19:03 UTC: At the present time, some of our mail.webfaction.com IMAP/POP proxies are not functioning. We’re working to resolve this and will update this post as soon as more information is available.

2017-04-03 20:14 UTC: The problem was caused by a fault in our monitoring systems. The fault has been corrected and IMAP/POP services are functioning normally.

-
-

[Completed] Emergency maintenance on web474, April 02, 2017

Posted in Downtime by

Web474 will be taken down for hard disk replacement on Sunday April 2nd, 2017 between 04:00 UTC and 08:00 UTC.

We will update this post as maintenance progresses.

Sun Apr  2 04:18:02 UTC 2017: The server has been taken offline for maintenance.

Sun Apr  2 05:39:44 UTC 2017: The first of two hard disks has been replaced and the RAID is rebuilding (25% currently)

Sun Apr  2 07:34:41 UTC 2017: The RAID has finished rebuilding after the first disk replacement, and the second disk is now being replaced.

Sun Apr  2 07:45:17 UTC 2017: The server is back online. The second disk was replaced and the RAID is rebuilding, which will decrease performance until it finishes.

Sun Apr  2 12:14:17 UTC 2017: All drives in the RAID have finished rebuilding. Maintenance is now complete.

-
-

[Completed] Emergency network maintenance in Singapore

Posted in Maintenance by

Between 16:00 and 18:00 UTC today (24 March 2017) a network module in our Singapore datacenter will be rebooted.

The following servers are serviced by this network module: web321, web322, web323, web339, web360, web361, web362, web375, web379, web390, web409, web422, web423, web424, web429, web438, web442, web463, web474, web486, web490, web505, web508, web517

However, no downtime or loss of connectivity is expected.

We’ll update this post when the maintenance is complete.

2017-03-24 17:07 UTC: The maintenance is complete.

-
-

[Fixed] Web346 offline

Posted in Downtime by

2017-03-16 20:01 UTC: The networking services on Web346 are currently down and the machine is offline. We’re working to restore service at this time and will update this post when more information is available.

2017-03-16 22:06 UTC: Web346 is back online after our upstream network provider reconfigured a switch. We’ll continue to monitor for signs of trouble.

2017-03-16 22:26 UTC: Web346 has gone offline again. We’re working to restore service at this time and will update this post when more information is available.

2017-03-16 23:25 UTC: We’re still investigating the connectivity issue.

2017-03-17 00:24 UTC: We’re still investigating the connectivity issue.

2017-03-17 01:44 UTC: We’ve switched Web346 to a different OS kernel and it seem to be stable now. We’ll continue to monitor.

2017-03-17 15:43 UTC: Several hours ago, we determined that Web346 was being attacked. At that time, we took steps to mitigate the attack, and the server has been stable since then. We’ll continue to monitor.

2017-03-24 15:22 UTC: We’ve not seen any further issues over the past several days.

-
-