[fixed] Partial incoming mail service disruption

Posted in Problems by

2017-06-20 20:44 UTC: A networking issue in one of our US data centers is affecting incoming mail service and webmail logins for some customers. We’re working to restore service and will update this post when more information is available.

2017-06-20 20:59 UTC: Service has been restored – we’ll continue to monitor.

2017-06-21 14:36 UTC: There have been no further disruptions, and the data center confirms that the networking issue is resolved.

-
-

[fixed] Web576 offline

Posted in Downtime by

2017-06-20 01:41 UTC: Web576 is currently offline due to a DDoS attack. We’re working to restore service and will update this post when more information is available.

2017-06-20 02:09 UTC: We’ve mitigated the attack upstream. The server appears to be stable, but the mitigation may affect UDP traffic. Please contact WebFaction support if you’re not able to access the server. We’ll continue to monitor.

2017-06-20 14:33 UTC: At the present time, Web576 is showing very high packet loss. We’re investigating at this time.

2017-06-20 17:48 UTC: Web576 is stable at this time. We’ll continue to monitor.

2017-06-21 14:37 UTC:  The attack has resumed, and we’re working to mitigate it.

2017-06-21 18:21 UTC: We’ve possibly identified the intended target of the attack and have isolated them from other customers. The server is stable at this time and we’ll continue to monitor.

2017-06-22 18:23 UTC: The server has been stable for the past 24 hours. We’ll continue to monitor.

2017-06-25 15:23 UTC: The server has been stable for over 72 hours, and there has been no further sign of any attack.

-
-

[Monitoring] Control panel, API, and website down

Posted in Downtime by

2017-06-05 19:36 UTC: our control panel server is currently not responding. This also affects the control panel API and www.webfaction.com. We’re working to restore service and will update this post when more info is available.

2017-06-05 19:47 UTC: the panel, API, and www.webfaction.com are back online. We’re monitoring for further issues.

-
-

[fixed] MySQL down on Web231

Posted in Problems by

2017-05-26 00:59 UTC: MySQL on Web231 is currently down. We’re working to restore service and will update this post when more info is available.

2017-05-26 01:13 UTC: We’re dumping and reloading all MySQL databases on Web231. The reload is halfway complete.

2017-05-26 01:25 UTC: All MySQL databases on Web231 have been restored and the MySQL service is now operational.

-
-

[Fixed] Multiple US servers offline

Posted in Problems by

2017-05-17 22:01 UTC: Several US shared and cloud servers in our St. Louis data center are currently offline We’ll update this post as soon as more info is available.

2017-05-17 22:07 UTC: The following servers are affected:

  • Shared: web537, web538, web539, web540, web541, web542, web543, web544, web545, web546, web547, web548, web549, web550, web551, web552, web553, web554, web555, web556, web557, web559, web560, web563, web590, web591, web592, web593, web594, web595, web596, web597, web598
  • Cloud: wf-207-38-86-39, wf-207-38-86-40, wf-207-38-86-43, wf-207-38-86-47, wf-207-38-86-48, wf-207-38-86-49, wf-207-38-86-73, wf-207-38-86-75, wf-207-38-86-76, wf-207-38-86-78, wf-207-38-86-79, wf-207-38-86-85, wf-207-38-86-88, wf-207-38-86-90, wf-207-38-86-92, wf-207-38-86-93, wf-207-38-86-94, wf-207-38-86-98, wf-207-38-86-102, wf-207-38-86-103, wf-207-38-86-104, wf-207-38-86-105, wf-207-38-86-106, wf-207-38-86-107, wf-207-38-92-110, wf-207-38-86-110, wf-207-38-86-111, wf-207-38-86-112, wf-207-38-86-114, wf-207-38-86-115, wf-207-38-86-116, wf-207-38-86-117, wf-207-38-86-118, wf-207-38-86-120, wf-207-38-86-121, wf-207-38-86-122, wf-207-38-86-123, wf-207-38-86-124, wf-207-38-86-125, wf-207-38-86-126, wf-207-38-86-127, wf-207-38-86-128, wf-207-38-86-129, wf-207-38-86-130, wf-207-38-86-131, wf-207-38-86-132, wf-207-38-86-134, wf-207-38-86-137, wf-207-38-86-139, wf-207-38-86-77, wf-207-38-86-135, wf-207-38-86-138, wf-207-38-86-140, wf-207-38-86-145, wf-207-38-86-147, wf-207-38-86-149, wf-207-38-86-150, wf-207-38-86-159, wf-207-38-86-160, wf-207-38-86-164, wf-207-38-86-165, wf-207-38-86-166, wf-207-38-86-167, wf-207-38-86-168, wf-207-38-86-170, wf-207-38-86-171, wf-207-38-86-175, wf-207-38-86-178, wf-207-38-86-180, wf-207-38-86-191, wf-207-38-86-193, wf-207-38-86-194, wf-207-38-86-195, wf-207-38-86-196, wf-207-38-86-197, wf-207-38-86-198, wf-207-38-86-199, wf-207-38-92-219, wf-207-38-92-220, wf-207-38-92-221, wf-207-38-92-222, wf-207-38-92-223, wf-207-38-92-224

2017-05-17 23:09 UTC: We’re still working with the data center to identify and resolve this issue. We’ll update this post as soon as more information is available.

2017-05-17 23:20 UTC: web537, web538, web539, web540, web541, web542, web543, web544, web545, web546, web547, web548, web549, web550, web551, web552, web553, web554, web555, web556, web557, web559, web560, web563, web590, web591, web592, web593, web594, web595, web596, web597, web598 are all back online.

2017-05-17 23:40 UTC: All US servers are back online at this time. We will continue to monitor and update this post if there are further developments.

-
-

Network maintenance on May 22, 24, and 25 – minimal downtime expected

Posted in Maintenance by

Our upstream provider will be performing emergency maintenance on three separate routers on May 22, 24, and 25.

A three hour maintenance window has been scheduled for each router. The total expected downtime within each window should not exceed three minutes.

The maintenance windows and affected servers are:

  • 22-May-2017 22:00 UTC through 23-May-2017 01:00 UTC: dweb186, web209, web210, web211, web217, web218, web223, web224, web225, web303, web304, web305, web306, web314, web316, web317, web327, web331, web332, web333, web334, web357, web358, web359, web364, web383, web392, web395, web416, web421, web435, web448, web457, web459, web464, web480, web483, web488
  • 24-May-2017 05:00 UTC through 08:00 UTC: ns1, web22, web23, web39, web40, web69, web70, web72, web74, web75, web119, web122, web183, web200, web213, web228, web230, web231, web233, web234, web235, web242, web248, web249, web338, web391, web397, web413, web453, web484, web487
  • 25-May-2017 05:00 UTC through 08:00 UTC: mail, mx7, mx8, web31, web42, web48, web155, web174, web178, web186, web226, web237, web241, web351, web354, web355, web356, web363, web365, web366, web367, web372, web378, web382, web387, web408, web450, web492, webmail
-
-

[Fixed] Web457 offline

Posted in Downtime by

2017-05-05 15:28 UTC: Web457 is offline due to an IP block from our upstream provider. We’re working with them to restore service and will update this post when more info is available.

2017-05-05 15:37 UTC: The IP block is lifted and Web457 is back online.

-
-

[Done] Emergency network maintenance affecting several EU servers

Posted in Downtime by

2017-04-27 00:20 UTC: At the present time, emergency network maintenance in our EU datacenter has caused several EU servers to go offline. We’ll update this post as more information becomes available.

2017-04-27 00:29 UTC: Connectivity has been restored, we’ll continue to monitor.

-
-

[Fixed] Read-only filesystem on Web429

Posted in Downtime by

2017-04-08 00:44 UTC: The filesystem on Web429 is currently read-only. We’re working to restore service at this time and will update this post with more information as it becomes available.

2017-04-08 00:45 UTC: We’re taking the machine down for a filesystem check.

2017-04-08 01:07 UTC: The filesystem check is complete. Some inconsistencies in the filesystem journal were repaired. The server is back online and responding normally at this time.

 

-
-

[Fixed] Incoming mail service disruption

Posted in Problems by

2017-04-03 19:03 UTC: At the present time, some of our mail.webfaction.com IMAP/POP proxies are not functioning. We’re working to resolve this and will update this post as soon as more information is available.

2017-04-03 20:14 UTC: The problem was caused by a fault in our monitoring systems. The fault has been corrected and IMAP/POP services are functioning normally.

-
-