[fixed] Delayed payment transactions

Posted in Problems by

2017-11-10 19:12 UTC: For the past few hours, we have been experiencing delays between the time a customer makes a payment and the time that the payment is posted to the customer’s account. The delay is occurring with our billing processor. If you make a payment and do not see it posted to your account immediately, please do not make an additional payment. The original payment should be posted within a few hours. We’ll update this post when the issue is resolved. Please contact WebFaction support if you have any questions or concerns.

2017-11-10 21:19 UTC: The issue has been resolved.

-
-

[Fixed] Several cloud servers offline

Posted in Downtime by

2017-10-06 17:13 UTC: Several of our cloud servers are currently offline due to an unknown issue. We’re investigating at this time and will update this post when more information is available. The affected servers are:

  • wf-173-255-197-72
  • wf-173-255-203-84
  • wf-198-58-100-37
  • wf-198-58-120-80
  • wf-198-58-98-119
  • wf-207-38-86-187
  • wf-207-38-86-192
  • wf-207-38-92-115
  • wf-207-38-92-14
  • wf-207-38-92-176
  • wf-207-38-92-182
  • wf-207-38-92-183
  • wf-207-38-92-184
  • wf-207-38-92-215
  • wf-207-38-92-244
  • wf-207-38-92-55
  • wf-207-38-92-93
  • wf-207-38-92-94
  • wf-207-38-93-134
  • wf-207-38-93-174
  • wf-207-38-93-187
  • wf-207-38-93-252
  • wf-207-38-93-253
  • wf-207-38-93-254
  • wf-207-38-94-111
  • wf-207-38-94-206
  • wf-207-38-94-230
  • wf-207-38-94-235
  • wf-207-38-94-251
  • wf-23-239-25-54
  • wf-45-33-113-161
  • wf-45-33-14-240
  • wf-45-33-18-166
  • wf-45-33-24-216
  • wf-45-33-7-73
  • wf-45-56-122-41
  • wf-45-56-123-50
  • wf-45-56-77-37
  • wf-45-79-0-162
  • wf-45-79-1-12
  • wf-45-79-10-158
  • wf-45-79-4-136
  • wf-45-79-7-111
  • wf-66-228-50-12
  • wf-72-14-182-215
  • wf-96-126-117-174

2017-10-06 17:32 UTC: Several cloud servers are back online after a reboot of the host hardware. The servers down at this time are:

  • wf-173-255-197-72
  • wf-173-255-203-84
  • wf-198-58-100-37
  • wf-198-58-120-80
  • wf-198-58-98-119
  • wf-23-239-25-54
  • wf-45-33-113-161
  • wf-45-33-14-240
  • wf-45-33-18-166
  • wf-45-33-24-216
  • wf-45-33-7-73
  • wf-45-56-122-41
  • wf-45-56-123-50
  • wf-45-56-77-37
  • wf-45-79-0-162
  • wf-45-79-1-12
  • wf-45-79-10-158
  • wf-45-79-4-136
  • wf-45-79-7-111
  • wf-66-228-50-12
  • wf-72-14-182-215
  • wf-96-126-117-174

2017-10-06 18:32 UTC: All of the affected servers are back online.

2017-10-11 12:59 UTC: We are taking the machine down for emergency maintenance to do a chassis swap.

2017-10-11 15:48 UTC: The server is back online.

-
-

[on hold] New outgoing email address restriction coming 1 September 2017

Posted in Service change by

Effective on 1 September 2017, our SMTP server will reject any outgoing message that uses a sender address in the form of “user@host.webfaction.com” (for example, “bob@web555.webfaction.com”).

This change primarily affects customers who are using sendmail (including the PHP mail() function) to send messages without explicitly setting a sender address.

If you’re not sure if you will be affected by this change, then send yourself a message from the potentially-affected website, script, or other application, and then inspect the headers of the received message to ensure that the “From” header is a valid address not in the form of “user@host.webfaction.com”, and does not use any of the domains listed in our email limitations and restrictions.

If you have any questions or concerns regarring this change, please open a support ticket and our team will be happy to assist.

2017-09-01 17:09 UTC: The new restrictions are now in effect.

2017-09-17 18:24 UTC: This change was disrupting outgoing email for several customers’ WordPress sites, so we’ve rolled it back and will re-deploy when that issue has been resolved.

-
-

[monitoring] SMTP blocked/throttled by Microsoft

Posted in Problems by

2017-07-27 21:16 UTC: Microsoft is currently blocking and/or throttling mail sent via our SMTP to hotmail.com, outlook.com, msn.com, and other email domains hosted by Microsoft. We’re working with Microsoft to resolve this issue and will update this post when more information is available.

2017-08-02 18:02 UTC: We’ve set up some downstream SMTP relays to help with delivery while we continue to work to get the main SMTP IP address unblocked.

2017-08-03 16:54 UTC: Mail sent via our SMTP to hotmail.com, outlook.com, msn.com, and other email domains hosted by Microsoft is no longer being blocked or throttled. We’ll continue to monitor.

-
-

[fixed] POP/IMAP service disruption

Posted in Problems by

2017-07-25 22:21 UTC: a filesystem issue on one of our POP/IMAP proxies is disrupting mail service for some customers. We’re working to restore service and will update this post when more info is available.

2017-07-25 23:09 UTC: The faulty mail proxy is back online. We’ll continue to monitor.

2017-07-27 21:15 UTC: We’ve seen no further issues with the POP/IMAP proxies.

-
-

[fixed] Partial incoming mail service disruption

Posted in Problems by

2017-06-20 20:44 UTC: A networking issue in one of our US data centers is affecting incoming mail service and webmail logins for some customers. We’re working to restore service and will update this post when more information is available.

2017-06-20 20:59 UTC: Service has been restored – we’ll continue to monitor.

2017-06-21 14:36 UTC: There have been no further disruptions, and the data center confirms that the networking issue is resolved.

-
-

[fixed] Web576 offline

Posted in Downtime by

2017-06-20 01:41 UTC: Web576 is currently offline due to a DDoS attack. We’re working to restore service and will update this post when more information is available.

2017-06-20 02:09 UTC: We’ve mitigated the attack upstream. The server appears to be stable, but the mitigation may affect UDP traffic. Please contact WebFaction support if you’re not able to access the server. We’ll continue to monitor.

2017-06-20 14:33 UTC: At the present time, Web576 is showing very high packet loss. We’re investigating at this time.

2017-06-20 17:48 UTC: Web576 is stable at this time. We’ll continue to monitor.

2017-06-21 14:37 UTC:  The attack has resumed, and we’re working to mitigate it.

2017-06-21 18:21 UTC: We’ve possibly identified the intended target of the attack and have isolated them from other customers. The server is stable at this time and we’ll continue to monitor.

2017-06-22 18:23 UTC: The server has been stable for the past 24 hours. We’ll continue to monitor.

2017-06-25 15:23 UTC: The server has been stable for over 72 hours, and there has been no further sign of any attack.

-
-

[Monitoring] Control panel, API, and website down

Posted in Downtime by

2017-06-05 19:36 UTC: our control panel server is currently not responding. This also affects the control panel API and www.webfaction.com. We’re working to restore service and will update this post when more info is available.

2017-06-05 19:47 UTC: the panel, API, and www.webfaction.com are back online. We’re monitoring for further issues.

-
-

[fixed] MySQL down on Web231

Posted in Problems by

2017-05-26 00:59 UTC: MySQL on Web231 is currently down. We’re working to restore service and will update this post when more info is available.

2017-05-26 01:13 UTC: We’re dumping and reloading all MySQL databases on Web231. The reload is halfway complete.

2017-05-26 01:25 UTC: All MySQL databases on Web231 have been restored and the MySQL service is now operational.

-
-

[Fixed] Multiple US servers offline

Posted in Problems by

2017-05-17 22:01 UTC: Several US shared and cloud servers in our St. Louis data center are currently offline We’ll update this post as soon as more info is available.

2017-05-17 22:07 UTC: The following servers are affected:

  • Shared: web537, web538, web539, web540, web541, web542, web543, web544, web545, web546, web547, web548, web549, web550, web551, web552, web553, web554, web555, web556, web557, web559, web560, web563, web590, web591, web592, web593, web594, web595, web596, web597, web598
  • Cloud: wf-207-38-86-39, wf-207-38-86-40, wf-207-38-86-43, wf-207-38-86-47, wf-207-38-86-48, wf-207-38-86-49, wf-207-38-86-73, wf-207-38-86-75, wf-207-38-86-76, wf-207-38-86-78, wf-207-38-86-79, wf-207-38-86-85, wf-207-38-86-88, wf-207-38-86-90, wf-207-38-86-92, wf-207-38-86-93, wf-207-38-86-94, wf-207-38-86-98, wf-207-38-86-102, wf-207-38-86-103, wf-207-38-86-104, wf-207-38-86-105, wf-207-38-86-106, wf-207-38-86-107, wf-207-38-92-110, wf-207-38-86-110, wf-207-38-86-111, wf-207-38-86-112, wf-207-38-86-114, wf-207-38-86-115, wf-207-38-86-116, wf-207-38-86-117, wf-207-38-86-118, wf-207-38-86-120, wf-207-38-86-121, wf-207-38-86-122, wf-207-38-86-123, wf-207-38-86-124, wf-207-38-86-125, wf-207-38-86-126, wf-207-38-86-127, wf-207-38-86-128, wf-207-38-86-129, wf-207-38-86-130, wf-207-38-86-131, wf-207-38-86-132, wf-207-38-86-134, wf-207-38-86-137, wf-207-38-86-139, wf-207-38-86-77, wf-207-38-86-135, wf-207-38-86-138, wf-207-38-86-140, wf-207-38-86-145, wf-207-38-86-147, wf-207-38-86-149, wf-207-38-86-150, wf-207-38-86-159, wf-207-38-86-160, wf-207-38-86-164, wf-207-38-86-165, wf-207-38-86-166, wf-207-38-86-167, wf-207-38-86-168, wf-207-38-86-170, wf-207-38-86-171, wf-207-38-86-175, wf-207-38-86-178, wf-207-38-86-180, wf-207-38-86-191, wf-207-38-86-193, wf-207-38-86-194, wf-207-38-86-195, wf-207-38-86-196, wf-207-38-86-197, wf-207-38-86-198, wf-207-38-86-199, wf-207-38-92-219, wf-207-38-92-220, wf-207-38-92-221, wf-207-38-92-222, wf-207-38-92-223, wf-207-38-92-224

2017-05-17 23:09 UTC: We’re still working with the data center to identify and resolve this issue. We’ll update this post as soon as more information is available.

2017-05-17 23:20 UTC: web537, web538, web539, web540, web541, web542, web543, web544, web545, web546, web547, web548, web549, web550, web551, web552, web553, web554, web555, web556, web557, web559, web560, web563, web590, web591, web592, web593, web594, web595, web596, web597, web598 are all back online.

2017-05-17 23:40 UTC: All US servers are back online at this time. We will continue to monitor and update this post if there are further developments.

-
-