[Fixed] Several cloud servers offline

Posted in Downtime by

2017-10-06 17:13 UTC: Several of our cloud servers are currently offline due to an unknown issue. We’re investigating at this time and will update this post when more information is available. The affected servers are:

  • wf-173-255-197-72
  • wf-173-255-203-84
  • wf-198-58-100-37
  • wf-198-58-120-80
  • wf-198-58-98-119
  • wf-207-38-86-187
  • wf-207-38-86-192
  • wf-207-38-92-115
  • wf-207-38-92-14
  • wf-207-38-92-176
  • wf-207-38-92-182
  • wf-207-38-92-183
  • wf-207-38-92-184
  • wf-207-38-92-215
  • wf-207-38-92-244
  • wf-207-38-92-55
  • wf-207-38-92-93
  • wf-207-38-92-94
  • wf-207-38-93-134
  • wf-207-38-93-174
  • wf-207-38-93-187
  • wf-207-38-93-252
  • wf-207-38-93-253
  • wf-207-38-93-254
  • wf-207-38-94-111
  • wf-207-38-94-206
  • wf-207-38-94-230
  • wf-207-38-94-235
  • wf-207-38-94-251
  • wf-23-239-25-54
  • wf-45-33-113-161
  • wf-45-33-14-240
  • wf-45-33-18-166
  • wf-45-33-24-216
  • wf-45-33-7-73
  • wf-45-56-122-41
  • wf-45-56-123-50
  • wf-45-56-77-37
  • wf-45-79-0-162
  • wf-45-79-1-12
  • wf-45-79-10-158
  • wf-45-79-4-136
  • wf-45-79-7-111
  • wf-66-228-50-12
  • wf-72-14-182-215
  • wf-96-126-117-174

2017-10-06 17:32 UTC: Several cloud servers are back online after a reboot of the host hardware. The servers down at this time are:

  • wf-173-255-197-72
  • wf-173-255-203-84
  • wf-198-58-100-37
  • wf-198-58-120-80
  • wf-198-58-98-119
  • wf-23-239-25-54
  • wf-45-33-113-161
  • wf-45-33-14-240
  • wf-45-33-18-166
  • wf-45-33-24-216
  • wf-45-33-7-73
  • wf-45-56-122-41
  • wf-45-56-123-50
  • wf-45-56-77-37
  • wf-45-79-0-162
  • wf-45-79-1-12
  • wf-45-79-10-158
  • wf-45-79-4-136
  • wf-45-79-7-111
  • wf-66-228-50-12
  • wf-72-14-182-215
  • wf-96-126-117-174

2017-10-06 18:32 UTC: All of the affected servers are back online.

2017-10-11 12:59 UTC: We are taking the machine down for emergency maintenance to do a chassis swap.

2017-10-11 15:48 UTC: The server is back online.

-
-

[fixed] web607 inaccessible

Posted in Downtime by

The server is currently not responding. We are investigating the issue and will post updates as soon as we have further details.

2017-09-07 18:35 UTC: There is an issue with the RAID controller. We are working to have this resolved as quickly as possible.

2017-09-07 19:15 UTC: The server is currently back online. However, we are still investigating the issue further.

2017-09-07 20:57 UTC: There have been no further issues since the server reboot.

-
-

[fixed] web527 inaccessible

Posted in Downtime by

Web527 did not come back after a reboot. We are looking into this and will resolve as quickly as possible.

2017-08-15 13:48 UTC: The server is back online. The issue was with the installed kernel. It was changed and the server rebooted.

-
-

[monintoring] web550 inaccessible

Posted in Downtime by

The server is currently inaccessible. We are investigating the issue and will update as soon as we have more details.

2017-07-12 00:33 UTC: Web550 is back online at this time. The problem appears to have been an DDoS attack targeting Web550. The attack has subsided. We’ll continue to monitor.

2017-07-12 15:32 UTC: Web550 was rebooted 90 minutes ago to attempt to resolve an issue with IPv6 connectivity. The server is back online, but IPv6 connectivity is down.

-
-

[fixed] Web523 offline

Posted in Downtime by

We are currently investigating the issue and will update as soon as we have further details.

2017-06-30 18:37 UTC: Web523 is currently in a rescue environment and our data center team are troubleshooting.

2017-06-30 20:53 UTC: Web523 had a corrupted kernel. The problem has been resolved and the server is now back online.

-
-

[Fixed] Web416 inaccessible

Posted in Downtime by

2017-06-27 13:10 UTC: The server is currently not responding. We are investigating the issue and will update this post as soon as we have further details.

2017-06-27 13:24 UTC: The server is back online.

-
-

[fixed] web530 and web561 offline

Posted in Downtime by

We are currently investigating the issue and will update as soon as we have further details.

2017-06-22 19:06 UTC: The servers are accessible again. There was a network issue that caused the temporary interruption.

-
-

[fixed] Web576 offline

Posted in Downtime by

2017-06-20 01:41 UTC: Web576 is currently offline due to a DDoS attack. We’re working to restore service and will update this post when more information is available.

2017-06-20 02:09 UTC: We’ve mitigated the attack upstream. The server appears to be stable, but the mitigation may affect UDP traffic. Please contact WebFaction support if you’re not able to access the server. We’ll continue to monitor.

2017-06-20 14:33 UTC: At the present time, Web576 is showing very high packet loss. We’re investigating at this time.

2017-06-20 17:48 UTC: Web576 is stable at this time. We’ll continue to monitor.

2017-06-21 14:37 UTC:  The attack has resumed, and we’re working to mitigate it.

2017-06-21 18:21 UTC: We’ve possibly identified the intended target of the attack and have isolated them from other customers. The server is stable at this time and we’ll continue to monitor.

2017-06-22 18:23 UTC: The server has been stable for the past 24 hours. We’ll continue to monitor.

2017-06-25 15:23 UTC: The server has been stable for over 72 hours, and there has been no further sign of any attack.

-
-

[Rescheduled] Scheduled maintenance on Sunday, June 18, 2017

Posted in Downtime by

We will be performing a power maintenance on the following servers at 16:00 UTC June 18, 2017 between and 4:00 UTC June 19, 2017:

wf-207-38-86-39
wf-207-38-86-40
wf-207-38-86-43
wf-207-38-86-47
wf-207-38-86-48
wf-207-38-86-49
wf-207-38-86-73
wf-207-38-86-75
wf-207-38-86-76
wf-207-38-86-78
wf-207-38-86-79
wf-207-38-86-85
wf-207-38-86-88
wf-207-38-86-90
wf-207-38-86-92
wf-207-38-86-93
wf-207-38-86-94
wf-207-38-86-98
wf-207-38-86-102
wf-207-38-86-103
wf-207-38-86-104
wf-207-38-86-105
wf-207-38-86-106
wf-207-38-86-107
wf-207-38-92-110
wf-207-38-86-178
wf-207-38-86-180
wf-207-38-86-191
wf-207-38-86-193
wf-207-38-86-194
wf-207-38-86-195
wf-207-38-86-196
wf-207-38-86-197
wf-207-38-86-198
wf-207-38-86-199
wf-207-38-93-249
wf-207-38-86-77
wf-207-38-86-135
wf-207-38-86-138
wf-207-38-86-140
wf-207-38-86-145
wf-207-38-86-147
wf-207-38-86-149
wf-207-38-86-150
wf-207-38-86-159
wf-207-38-86-160
wf-207-38-86-164
wf-207-38-86-165
wf-207-38-86-166
wf-207-38-86-167
wf-207-38-86-168
wf-207-38-86-170
wf-207-38-86-171
wf-207-38-86-175
web590
web591
web592
web593
web594
web595
web596
web597
web598
web600
web601
web602
web603

Connectivity to the servers will be interrupted for up to one hour during the maintenance window. We apologize in advance for the inconvenience. Should you have any questions about this maintenance, feel free to open a support ticket and our support team will be happy to help.

2017-06-19 16:20 UTC:  The maintenance had to be rescheduled and we’ll post the new schedule when it is ready.

2017-06-19 22:47 UTC: Rescheduled to Sunday, June 25, 2017

-
-

[Monitoring] Control panel, API, and website down

Posted in Downtime by

2017-06-05 19:36 UTC: our control panel server is currently not responding. This also affects the control panel API and www.webfaction.com. We’re working to restore service and will update this post when more info is available.

2017-06-05 19:47 UTC: the panel, API, and www.webfaction.com are back online. We’re monitoring for further issues.

-
-