Currently we are investigating intermittent network connectivity with the following servers.

Web344
Web396
Web399
Web400
Web401
Web405
Web407
Web410
Web418
Web419
Dweb182
mx9
Mailbox10

We are working with our upstream provider and will update when we have more information.

2014-07-13 18:40 These servers are under a DDoS attack. Our upstream provider has placed these servers under Guard protection.

2014-07-14 04:08 DDoS attacks continue; updated affected server list. All servers with IP 108.168.242.* are affected, as the DDoS attack is indiscriminately directed at the entire subnet, not any server or site in particular. Cisco Guard protection is not very effective for large-scale DDoS attacks, so downtime continues.

2014-07-14 17:02 The DDOS attack is still ongoing although it seems to affect fewer servers at the moment. We are continuing to do our best to mitigate the impact. We have also identified a potential target of the attack and are moving the target to separate servers.

2014-07-14 02:23 UTC: The attack is ongoing.

2014-07-15 17:45 UTC: At this time, the DDoS attack is still ongoing. Our upstream provider is mitigating about 4Gbps of malicious incoming traffic, which has improved connectivity overall, but some legitimate traffic is unavoidably filtered as well. We’ll update this post when we have more information.

2014-07-16 00:41 UTC: At this time, the DDoS attack is ongoing. About 3.8 Gbps of malicious traffic is being filtered. Mail delays caused by the attack have been mostly resolved for now. We’ll update this post when we have more information.

2014-07-16 10:04 UTC: At this time, the DDoS attack is still ongoing. Mail delays caused by the attack may still be occurring. We’ll update this post when we have more information.

2014-07-16 12:41 UTC: Mitigation was removed for a short time which caused some mail delays. We have started mitigating the attack again and mail delays have been resolved for now.

2014-07-16 16:50 UTC: The attack was overwhelming our upstream provider’s DDoS mitigation. The bulk of the traffic was targeting 108.168.242.146 (web396), so our provider has null-routed that IP address. We’re working with them to restore service. The other servers affected by the attack are operating normally at this time.

2014-07-16 17:00 UTC: The null-route on web396 has been lifted. The attack is ongoing, but all of the servers listed above are still behind the DDoS mitigation system and appear to be operating normally at this time. We’ll update this post when we have more information.

2014-07-16 23:00 UTC: Due to the size of the attack on web396, we have split the domains on the machine over several proxy servers to get the sites back online. If you experience any issues with your sites open a ticket and we’ll look into it asap.

2014-07-17 17:29 UTC: The attack is ongoing. Our upstream provider has null-routed web396 again, and the DDoS mitigation system is currently inactive. We’re working to restore service at this time.

2014-07-17 1745 UTC: Unfortunately the hardware DDOS protection from our datacenter isn’t able to mitigate an attack that big. We have set up some proxies in front of Web396 and we have pointed all domains on Web396 to these proxies. We have also emailed all customers on Web396 to let them know what their new IPs are. At this point sites that are pointing to the proxies appear to be working fine. If you’re using external DNS servers you will need to update your DNS records yourself.

2014-07-18 02:23 UTC: At this time, almost all of the malicious traffic is going to the proxies in front of Web396. We’re analyzing the traffic to the proxy IPs to identify the sites that are actually being targeted by the attack. At this time, Web396 is performing poorly, but the other servers affected by the attack, including mail servers, appear to be working normally. We’ll update this post when we have more information.

2014-07-18 07:04 UTC: Malicious traffic appears to have subsided for the time being. Please wait for further updates while we verify network status.

2014-07-18 08:06 UTC: Traffic mitigation is still in effect. We’ll update this post when we have more information.

2014-07-18 19:00 UTC: The attack appears to be over (apart from Web396 which has a separate statusblog entry: http://statusblog.webfaction.com/2014/07/18/ddos-attack-on-web396/). We would like to apologize for the extended service degradation that the attack has caused. Our datacenter does have hardware DDOS protection but it wasn’t able to mitigate that attack. We ended up building a custom-made DDOS mitigation system and it appears to be working well. Assuming that the new system keeps working well for future attacks these attacks should have a much smaller impact on our service.