[Done] Scheduled maintenance on Web40, July 30th, 2014.

Posted in Scheduled downtime by

Web40 will be taken down for a HDD and cables swap Wednesday July 30th 2014 at 12:00 UTC. We will update this post as maintenance will progress.

2014-07-30 12:32 UTC The HDD+cables have been swapped, the server is back to operational status.

-
-

[Done] Scheduled maintenance on Web412, July 30th, 2014.

Posted in Scheduled downtime by

Web412 will be taken down for swapping a failing disk Wednesday July 30th 2014, between 07:00 UTC and 10:00 UTC. We will update this post as maintenane will progress.

2014-07-30 08:11 The HDD+cables have been swapped, the server is back at operational status.

-
-

[Fixed] Web306 inaccessible

Posted in Downtime by

Web306 is currently not accessible. We are investigating the issue and will update this post as soon as we have more information.

-
-

[Done]Scheduled maintenance on Web429, July 25th, 2014.

Posted in Scheduled downtime by

Web429 will be taken down for a HDD+Cables swap Friday July 25th 2014, between 07:00 UTC and 09:00 UTC. We will update this post as maintenance will progress.

2014-07-25 08:34 UTC The replacement has been completed, the server is back at operational status.

-
-

[Done] DDOS attack on Web396

Posted in Problems by

Web396 is under a DDOS attack and its main IP (108.168.242.146) is currently null-routed.

We have set-up some proxy servers in front of Web396 and have emailed all customers to tell them to point their sites to the proxies.

Sites that are pointing to the proxies appear to be working fine.

We will update this post once the attack has subsided.

Update July 22th, 18:06 UTC: We have identified the target of the attack and have moved to to separate servers. At this point the attack on Web396 appears to be over and we have updated our DNS servers to point all the domains back to Web396. If you’re using external DNS servers you will need to update them to point your domains back to Web396

-
-

[Fixed] DDOS attacks on multiple servers

Posted in Downtime by

Currently we are investigating intermittent network connectivity with the following servers.

Web344
Web396
Web399
Web400
Web401
Web405
Web407
Web410
Web418
Web419
Dweb182
mx9
Mailbox10

We are working with our upstream provider and will update when we have more information.

2014-07-13 18:40 These servers are under a DDoS attack. Our upstream provider has placed these servers under Guard protection.

2014-07-14 04:08 DDoS attacks continue; updated affected server list. All servers with IP 108.168.242.* are affected, as the DDoS attack is indiscriminately directed at the entire subnet, not any server or site in particular. Cisco Guard protection is not very effective for large-scale DDoS attacks, so downtime continues.

2014-07-14 17:02 The DDOS attack is still ongoing although it seems to affect fewer servers at the moment. We are continuing to do our best to mitigate the impact. We have also identified a potential target of the attack and are moving the target to separate servers.

2014-07-14 02:23 UTC: The attack is ongoing.

2014-07-15 17:45 UTC: At this time, the DDoS attack is still ongoing. Our upstream provider is mitigating about 4Gbps of malicious incoming traffic, which has improved connectivity overall, but some legitimate traffic is unavoidably filtered as well. We’ll update this post when we have more information.

2014-07-16 00:41 UTC: At this time, the DDoS attack is ongoing. About 3.8 Gbps of malicious traffic is being filtered. Mail delays caused by the attack have been mostly resolved for now. We’ll update this post when we have more information.

2014-07-16 10:04 UTC: At this time, the DDoS attack is still ongoing. Mail delays caused by the attack may still be occurring. We’ll update this post when we have more information.

2014-07-16 12:41 UTC: Mitigation was removed for a short time which caused some mail delays. We have started mitigating the attack again and mail delays have been resolved for now.

2014-07-16 16:50 UTC: The attack was overwhelming our upstream provider’s DDoS mitigation. The bulk of the traffic was targeting 108.168.242.146 (web396), so our provider has null-routed that IP address. We’re working with them to restore service. The other servers affected by the attack are operating normally at this time.

2014-07-16 17:00 UTC: The null-route on web396 has been lifted. The attack is ongoing, but all of the servers listed above are still behind the DDoS mitigation system and appear to be operating normally at this time. We’ll update this post when we have more information.

2014-07-16 23:00 UTC: Due to the size of the attack on web396, we have split the domains on the machine over several proxy servers to get the sites back online. If you experience any issues with your sites open a ticket and we’ll look into it asap.

2014-07-17 17:29 UTC: The attack is ongoing. Our upstream provider has null-routed web396 again, and the DDoS mitigation system is currently inactive. We’re working to restore service at this time.

2014-07-17 1745 UTC: Unfortunately the hardware DDOS protection from our datacenter isn’t able to mitigate an attack that big. We have set up some proxies in front of Web396 and we have pointed all domains on Web396 to these proxies. We have also emailed all customers on Web396 to let them know what their new IPs are. At this point sites that are pointing to the proxies appear to be working fine. If you’re using external DNS servers you will need to update your DNS records yourself.

2014-07-18 02:23 UTC: At this time, almost all of the malicious traffic is going to the proxies in front of Web396. We’re analyzing the traffic to the proxy IPs to identify the sites that are actually being targeted by the attack. At this time, Web396 is performing poorly, but the other servers affected by the attack, including mail servers, appear to be working normally. We’ll update this post when we have more information.

2014-07-18 07:04 UTC: Malicious traffic appears to have subsided for the time being. Please wait for further updates while we verify network status.

2014-07-18 08:06 UTC: Traffic mitigation is still in effect. We’ll update this post when we have more information.

2014-07-18 19:00 UTC: The attack appears to be over (apart from Web396 which has a separate statusblog entry: http://statusblog.webfaction.com/2014/07/18/ddos-attack-on-web396/). We would like to apologize for the extended service degradation that the attack has caused. Our datacenter does have hardware DDOS protection but it wasn’t able to mitigate that attack. We ended up building a custom-made DDOS mitigation system and it appears to be working well. Assuming that the new system keeps working well for future attacks these attacks should have a much smaller impact on our service.

-
-

[Fixed] DDoS on Web408

Posted in Downtime by

The server is currently under a heavy DDoS attack. Mitigation has been enabled to help with the issue. We will continue to monitor and will update when we have more information.

2014-07-13 11:26 UTC: The server is back online and functioning normally.

2014-07-13 16:00 UTC: The DDoS attack on the server has resumed. Working on mitigation to restore service at this time.

2014-07-13 16:50 UTC: Mitigation has been fully implemented on the server. Some legitimate traffic may be impacted.

2014-07-15 17:30 UTC: The attack on Web408 has subsided and traffic to the machine is no longer being mitigated.

-
-

[closed] DDoS on Web396

Posted in Downtime by

The server is currently under a heavy DDoS attack. Mitigation has been enabled to help with the issue. We will continue to monitor and will update when we have more information. 2014-07-12 07:30 UTC: The DDoS has subsided and the server is back online. 2014-07-12 10:05 UTC: The DDoS is intermittent, there may be further issues pending further updates.

2014-07-13 1830 the DDoS attack has resumed. We are working with our upstream provider to mitigate the attack.

2014-07-15 17:43 UTC: this issue now being tracked on a separate post.

-
-

[Fixed] DDoS on Web456

Posted in Downtime by

The server is currently under a heavy DDoS attack. Mitigation has been enabled to help with the issue. We will continue to monitor and will update when we have more information.

2014-07-13 11:26 UTC: The server is back online and functioning normally.

-
-

[Fixed] Emergency maintenance on Web441

Posted in Downtime by

An issue is preventing the machine from booting. We are working to restore service as soon as possible.  We will post updates as we have them.

Update: The issue occurred during a kernel upgrade on the server and it is now fixed.

-
-