[Fixed] Several US servers inaccessible

Posted in Downtime by

We’ve just got an alert about the following US based servers being inaccessible.

We’re investigating at this time and will update this post when more information is available.

The affected shared servers are: web500, web501, web518, web521, web524, web590, web591, web592, web594, web595, web596, web597, web598, web599, web600, web601, web602, web603, web608, web609, web610, web611, web612, web613, web614, web615, web616, web617, web618, web619, web620, web621, web622, web623, web624, web625, web626, web627, web628, web629, web630, web631, web632, web634.

The affected cloud instances are: wf-148-72-160-10, wf-148-72-160-103, wf-148-72-160-104, wf-148-72-160-105, wf-148-72-160-106, wf-148-72-160-107, wf-148-72-160-108, wf-148-72-160-109, wf-148-72-160-110, wf-148-72-160-111, wf-148-72-160-114, wf-148-72-160-115, wf-148-72-160-117, wf-148-72-160-118, wf-148-72-160-119, wf-148-72-160-12, wf-148-72-160-120, wf-148-72-160-13, wf-148-72-160-14, wf-148-72-160-16, wf-148-72-160-17, wf-148-72-160-5, wf-148-72-160-6, wf-148-72-160-7, wf-148-72-160-8, wf-148-72-160-9, wf-207-38-86-102, wf-207-38-86-125, wf-207-38-86-135, wf-207-38-86-138, wf-207-38-86-140, wf-207-38-86-145, wf-207-38-86-147, wf-207-38-86-149, wf-207-38-86-150, wf-207-38-86-159, wf-207-38-86-160, wf-207-38-86-164, wf-207-38-86-165, wf-207-38-86-166, wf-207-38-86-167, wf-207-38-86-168, wf-207-38-86-170, wf-207-38-86-171, wf-207-38-86-175, wf-207-38-86-187, wf-207-38-86-192, wf-207-38-86-194, wf-207-38-86-67, wf-207-38-86-68, wf-207-38-86-77, wf-207-38-92-106, wf-207-38-92-115, wf-207-38-92-122, wf-207-38-92-123, wf-207-38-92-124, wf-207-38-92-125, wf-207-38-92-135, wf-207-38-92-14, wf-207-38-92-144, wf-207-38-92-149, wf-207-38-92-176, wf-207-38-92-18, wf-207-38-92-182, wf-207-38-92-183, wf-207-38-92-184, wf-207-38-92-215, wf-207-38-92-244, wf-207-38-92-5, wf-207-38-92-55, wf-207-38-92-6, wf-207-38-92-61, wf-207-38-92-7, wf-207-38-92-93, wf-207-38-92-94, wf-207-38-92-95, wf-207-38-93-134, wf-207-38-93-174, wf-207-38-93-187, wf-207-38-93-216, wf-207-38-93-252, wf-207-38-93-253, wf-207-38-93-254, wf-207-38-93-35, wf-207-38-93-36, wf-207-38-93-93, wf-207-38-94-108, wf-207-38-94-111, wf-207-38-94-140, wf-207-38-94-159, wf-207-38-94-181, wf-207-38-94-206, wf-207-38-94-21, wf-207-38-94-22, wf-207-38-94-224, wf-207-38-94-230, wf-207-38-94-235, wf-207-38-94-241, wf-207-38-94-251, wf-207-38-94-69, wf-207-38-94-70, wf-207-38-94-78, wf-207-38-94-91

2018-03-03 16:19 UTC We’ve identified the cause as a power outage. Servers are beginning to come back online now. We’ll update this post if anything changes.

2018-03-03 17:14 UTC The power outage was caused by a cooling failure. This is being worked on now. We’ll update this post when more information becomes available.

2018-03-03 18:50 UTC The issue is resolved and all servers have been restarted. We’ll continue to monitor. If you see any issues, please open a support ticket.

-
-

Unexpected downtime for Web463.

Posted in Uncategorized by

We had a brief period of downtime while we restarted the hardware. Web463 should be back online.

-
-

[Fixed] Webmail problems

Posted in Problems by

There is currently an issue with the Webmail interface at https://webmail.webfaction.com which is resulting in sporadic 500 Internal Server Errors for some email users. We are looking into the problem and hope to have normal service restored soon. We will update this entry as soon as we have more information.

2018-03-02 14:50 UTC: The root cause of the errors has been fixed. Thank you for your patience, and if there are any issues please feel free to contact support by opening a ticket.

-
-

[Done] Emergency maintenance on Web572 on Sunday, March 4, 2018

Posted in Uncategorized by

Web572 will be taken down to replace a failing memory module Sunday, March 4, 2018 between 00:00 and 01:00. The expected downtime should be 15 minutes or less.

2018-03-04 00:58 UTC: The maintenance is complete and the server is back online.

-
-

[Fixed] Delays in processing payments

Posted in Problems by

We are currently experiencing a temporary issue with our payment pre-processor.

Due to an issue on their side we are not receiving payment notifications in a timely manner, thus any payments, while received, are not instantly recorded to our control panel’s Payment Transaction History page.

This issue started today at 05:16 UTC and after communicating with them they said that it should be fixed by 14:30 UTC.

When they do resolve this issue we’ll go through all the payments made today, double check the transactions and add them to our panel.

If you did make a payment during that time but it is missing, feel free to raise a ticket and we’ll look into this.

2018-02-19 15:30 UTC: Our payment pre-processor have gotten back to us, letting us know that everything is fixed on their side and we are seeing payments going through successfully in the last hour. We are not double checking all transactions that happened during the window and adding any missing one to our panel.

2018-02-19 17:00 UTC: All transactions should be visible in our control panel now.

-
-

[Fixed] Web530 inaccessible

Posted in Downtime by

Web530 is currently not accessible. We are working to restore services as soon as possible.

2018-02-07 16:51 UTC: We are still working with our data center to bring the machine back online as soon as possible.

2018-02-07 20:04 UTC: The servers is back online.

2018-02-08 07:39 UTC: The server is experiencing network issues again and is inaccessible. We’re looking into this issue and will update this post as soon as we have more info.

2018-02-08 10:12 UTC: The server is up and running and looks stable. We’ll be monitoring it for a while before marking this issue as resolved. Please let us know if you notice any further issues from your end.

2018-02-08 10:49 UTC: Some customers have reported issues accessing the server via IPv4. We are looking into this. Customers using IPv6 should be able to access the server.

2018-02-08 14:30 UTC: After monitoring the server for some time, all services look stable so we’ll mark this issue as fixed.

-
-

[fixed] Web571 inaccessible

Posted in Downtime by

Web571 is currently not accessible. We are looking into the issue and will resolve as quickly as possible.

2018-01-07 08:45 UTC: The issue has been resolved with a reboot and the server should be responding to requests.

-
-

[Fixed] Web623 inaccessible

Posted in Downtime by

Web623 didn’t came back after our recent reboot to patch the ‘Meltdown’ and ‘Spectre’ security vulnerabilites.

We are working with our datacenter technicians to resolve this issue and get the server up and running again as quickly as possible.

2018-01-06 07:30 UTC: The server is back online.

-
-

[Fixed] Control panel, API, and website down

Posted in Uncategorized by

Our control panel server is currently not responding. This also affects the control panel API and www.webfaction.com. We’re working to restore service and will update this post when more info is available.

2018-01-06 07:04 UTC: The Control panel, API, and website are back online.

-
-

CPU Vulnerabilities: Meltdown & Spectre

Posted in Uncategorized by

Researchers disclosed a security vulnerability affecting side-channel analysis of speculative execution on modern computer processors (CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754). This issue affects home computers, phones, tablets and servers. As such, nearly everyone who uses such a device is affected. Our team is working on mitigating this, but we expect to have to reboot every machine shortly in order to protect your data.

We are looking to reboot every machine in our fleet at non-business hours over the next few days, we will soon contact you to let you know of the emergency maintenance window we have allocated for your machine.

We will keep you updated on this serious issue and give you as much notice for any emergency reboots as we possibly can.

References:
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754

We will keep this thread updated and apologize in advance for the inconvenience. Should you have any questions about this maintenance, feel free to open a support ticket and our team will be happy to help.

2018-01-05 17:28 UTC: All Singapore shared servers have been updated and rebooted.

2018-01-05 22:44 UTC: All EU shared and dedicated servers have been updated and rebooted.

2018-01-06 13:56 UTC: All USA shared and dedicated servers have been updated and rebooted.

-
-