Due to a Denial of Service Network Attack, we are migrating Web241 to a data center with better DDOS mitigation. The server’s main IP address is being changed from 188.8.131.52 to 184.108.40.206 .
2015-10-04 13:50 UTC: The server is back in operational status. We continue to monitor
2015-10-04 14:45 UTC: The server has had no issues since coming back online
2015-10-04 18:30 UTC: The server has gone offline again and we are working with the data center to determine root cause.
2015-10-04 22:30 UTC: The server is back online and we will continue to monitor.
2015-10-05 03:30 UTC: The server is experiencing a DDOS attack again. We are migrating the server to a different datacenter that has better DDOS mitigation. We will update this post with more information.
2015-10-05 11:30 UTC: Home directories starting with the letters ‘a’ through ‘l’ have been transferred to the new server.
2015-10-05 13:50 UTC: Home directories starting with the letters ‘a’ through ‘r’ have been transferred to the new server.
2015-10-05 20:34 UTC: The new server IPs are now all online. Proxies are running on the old server to forward traffic to the new IP – however, the old server’s main IP is currently null-routed due to the DDOS. The initial home directory transfer has completed, and a second pass rsync is now complete up through home directories starting with the letter “r”.
2015-10-05 22:21 UTC: The DDOS followed us to the new server IP, which caused our upstream provider to null-route the IP. We’re in the process of activating the DDOS mitigation system to restore service.
2015-10-05 23:36 UTC: DDOS mitigation is active on web241 and the server should responding normally for most customers. Some customers may experience intermittent connection problems while the DDOS mitigation system is active.
2015-10-06 00:31 UTC: The attack has overwhelmed the mitigation system and the main server IP 220.127.116.11 has been null-routed. We’re working to restore service at this time.
2015-10-07 01:52 UTC: Apologies for the lack of updates. We’ve identified the target of the attack and have moved them to an isolated IP address. The attack should follow them within the next hour or so, at which point service on Web241 should stabilize.
2015-10-07 21:33 UTC: The attack has subsided and most customer domains have been moved back to the main server IP. If you’re using third-party DNS, you’ll need to point your domains back at the main server IP manually. Detailed instructions have been mailed to all customers.
On each of these days, the maintenance will take place between 7:30 am and 9:30 am EDT, and will require a network switch reboot that will result in a loss of network connectivity for up to 30 minutes (but probably less). We’ll update this post as the maintenance progresses.