CPU Vulnerabilities: Meltdown & Spectre

Posted in Uncategorized by

Researchers disclosed a security vulnerability affecting side-channel analysis of speculative execution on modern computer processors (CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754). This issue affects home computers, phones, tablets and servers. As such, nearly everyone who uses such a device is affected. Our team is working on mitigating this, but we expect to have to reboot every machine shortly in order to protect your data.

We are looking to reboot every machine in our fleet at non-business hours over the next few days, we will soon contact you to let you know of the emergency maintenance window we have allocated for your machine.

We will keep you updated on this serious issue and give you as much notice for any emergency reboots as we possibly can.

References:
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754

We will keep this thread updated and apologize in advance for the inconvenience. Should you have any questions about this maintenance, feel free to open a support ticket and our team will be happy to help.

2018-01-05 17:28 UTC: All Singapore shared servers have been updated and rebooted.

2018-01-05 22:44 UTC: All EU shared and dedicated servers have been updated and rebooted.

2018-01-06 13:56 UTC: All USA shared and dedicated servers have been updated and rebooted.

-
-

[fixed] Delayed payment transactions

Posted in Problems by

2017-11-10 19:12 UTC: For the past few hours, we have been experiencing delays between the time a customer makes a payment and the time that the payment is posted to the customer’s account. The delay is occurring with our billing processor. If you make a payment and do not see it posted to your account immediately, please do not make an additional payment. The original payment should be posted within a few hours. We’ll update this post when the issue is resolved. Please contact WebFaction support if you have any questions or concerns.

2017-11-10 21:19 UTC: The issue has been resolved.

-
-

[Fixed] Several cloud servers offline

Posted in Downtime by

2017-10-06 17:13 UTC: Several of our cloud servers are currently offline due to an unknown issue. We’re investigating at this time and will update this post when more information is available. The affected servers are:

  • wf-173-255-197-72
  • wf-173-255-203-84
  • wf-198-58-100-37
  • wf-198-58-120-80
  • wf-198-58-98-119
  • wf-207-38-86-187
  • wf-207-38-86-192
  • wf-207-38-92-115
  • wf-207-38-92-14
  • wf-207-38-92-176
  • wf-207-38-92-182
  • wf-207-38-92-183
  • wf-207-38-92-184
  • wf-207-38-92-215
  • wf-207-38-92-244
  • wf-207-38-92-55
  • wf-207-38-92-93
  • wf-207-38-92-94
  • wf-207-38-93-134
  • wf-207-38-93-174
  • wf-207-38-93-187
  • wf-207-38-93-252
  • wf-207-38-93-253
  • wf-207-38-93-254
  • wf-207-38-94-111
  • wf-207-38-94-206
  • wf-207-38-94-230
  • wf-207-38-94-235
  • wf-207-38-94-251
  • wf-23-239-25-54
  • wf-45-33-113-161
  • wf-45-33-14-240
  • wf-45-33-18-166
  • wf-45-33-24-216
  • wf-45-33-7-73
  • wf-45-56-122-41
  • wf-45-56-123-50
  • wf-45-56-77-37
  • wf-45-79-0-162
  • wf-45-79-1-12
  • wf-45-79-10-158
  • wf-45-79-4-136
  • wf-45-79-7-111
  • wf-66-228-50-12
  • wf-72-14-182-215
  • wf-96-126-117-174

2017-10-06 17:32 UTC: Several cloud servers are back online after a reboot of the host hardware. The servers down at this time are:

  • wf-173-255-197-72
  • wf-173-255-203-84
  • wf-198-58-100-37
  • wf-198-58-120-80
  • wf-198-58-98-119
  • wf-23-239-25-54
  • wf-45-33-113-161
  • wf-45-33-14-240
  • wf-45-33-18-166
  • wf-45-33-24-216
  • wf-45-33-7-73
  • wf-45-56-122-41
  • wf-45-56-123-50
  • wf-45-56-77-37
  • wf-45-79-0-162
  • wf-45-79-1-12
  • wf-45-79-10-158
  • wf-45-79-4-136
  • wf-45-79-7-111
  • wf-66-228-50-12
  • wf-72-14-182-215
  • wf-96-126-117-174

2017-10-06 18:32 UTC: All of the affected servers are back online.

2017-10-11 12:59 UTC: We are taking the machine down for emergency maintenance to do a chassis swap.

2017-10-11 15:48 UTC: The server is back online.

-
-

[Done] Control Panel and API maintenance Monday, September 4th 2017

Posted in Uncategorized by

On Monday, 4 September 2017  between 11:00 and 12:00 UTC, we will perform maintenance on our Control Panel.

During this time, the Control Panel and API will be temporarily unavailable. We apologize in advance for the inconvenience. Should you have any questions about this maintenance, feel free to open a support ticket and our support team will be happy to help.

2017-09-04 12:01 UTC: We are starting the maintenance.

2017-09-04 13:49 UTC: The maintenance is now over. Thank you for your patience, and if there are any issues please feel free to contact support by opening a ticket.

-
-

[Done] Emergency maintenance on Web564

Posted in Uncategorized by

We are taking down Web564 to replace a failed memory module. Downtime should be 15 minutes or less.

2017-08-20 16:13 UTC: Shutting down the machine.

2017-08-20 16:25 UTC: The memory has been replaced and server is back online.

-
-

[on hold] New outgoing email address restriction coming 1 September 2017

Posted in Service change by

Effective on 1 September 2017, our SMTP server will reject any outgoing message that uses a sender address in the form of “user@host.webfaction.com” (for example, “bob@web555.webfaction.com”).

This change primarily affects customers who are using sendmail (including the PHP mail() function) to send messages without explicitly setting a sender address.

If you’re not sure if you will be affected by this change, then send yourself a message from the potentially-affected website, script, or other application, and then inspect the headers of the received message to ensure that the “From” header is a valid address not in the form of “user@host.webfaction.com”, and does not use any of the domains listed in our email limitations and restrictions.

If you have any questions or concerns regarring this change, please open a support ticket and our team will be happy to assist.

2017-09-01 17:09 UTC: The new restrictions are now in effect.

2017-09-17 18:24 UTC: This change was disrupting outgoing email for several customers’ WordPress sites, so we’ve rolled it back and will re-deploy when that issue has been resolved.

-
-

[monitoring] SMTP blocked/throttled by Microsoft

Posted in Problems by

2017-07-27 21:16 UTC: Microsoft is currently blocking and/or throttling mail sent via our SMTP to hotmail.com, outlook.com, msn.com, and other email domains hosted by Microsoft. We’re working with Microsoft to resolve this issue and will update this post when more information is available.

2017-08-02 18:02 UTC: We’ve set up some downstream SMTP relays to help with delivery while we continue to work to get the main SMTP IP address unblocked.

2017-08-03 16:54 UTC: Mail sent via our SMTP to hotmail.com, outlook.com, msn.com, and other email domains hosted by Microsoft is no longer being blocked or throttled. We’ll continue to monitor.

-
-

[fixed] POP/IMAP service disruption

Posted in Problems by

2017-07-25 22:21 UTC: a filesystem issue on one of our POP/IMAP proxies is disrupting mail service for some customers. We’re working to restore service and will update this post when more info is available.

2017-07-25 23:09 UTC: The faulty mail proxy is back online. We’ll continue to monitor.

2017-07-27 21:15 UTC: We’ve seen no further issues with the POP/IMAP proxies.

-
-

[Done] Scheduled maintenance on Sunday, July 16, 2017 on Web599

Posted in Uncategorized by

We will be performing power maintenance on Web599 between 23:00 UTC July 16, 2017 and 00:00 UTC July 17, 2017.

Connectivity to the server will be interrupted for up to one hour during the maintenance window. We apologize in advance for the inconvenience. Should you have any questions about this maintenance, feel free to open a support ticket and our support team will be happy to help.

2017-07-17 00:08 UTC: The maintenance has begun.

2017-07-17 00:21 UTC: The maintenance is complete.

-
-

[fixed] Web523 offline

Posted in Downtime by

We are currently investigating the issue and will update as soon as we have further details.

2017-06-30 18:37 UTC: Web523 is currently in a rescue environment and our data center team are troubleshooting.

2017-06-30 20:53 UTC: Web523 had a corrupted kernel. The problem has been resolved and the server is now back online.

-
-