[Done] Web241 Emergency Migration

Posted in Downtime by

Due to a Denial of Service Network Attack, we are migrating Web241 to a data center with better DDOS mitigation.  The server’s main IP address is being changed from 108.59.11.115 to 75.126.137.87 .

2015-10-04 13:50 UTC: The server is back in operational status. We continue to monitor

2015-10-04 14:45 UTC: The server has had no issues since coming back online

2015-10-04 18:30 UTC: The server has gone offline again and we are working with the data center to determine root cause.

2015-10-04 22:30 UTC: The server is back online and we will continue to monitor.

2015-10-05 03:30 UTC: The server is experiencing a DDOS attack again.  We are migrating the server to a different datacenter that has better DDOS mitigation.  We will update this post with more information.

2015-10-05 11:30 UTC: Home directories starting with the letters ‘a’ through ‘l’ have been transferred to the new server.

2015-10-05 13:50 UTC: Home directories starting with the letters ‘a’ through ‘r’ have been transferred to the new server.

2015-10-05 20:34 UTC: The new server IPs are now all online. Proxies are running on the old server to forward traffic to the new IP – however, the old server’s main IP is currently null-routed due to the DDOS. The initial home directory transfer has completed, and a second pass rsync is now complete up through home directories starting with the letter “r”.

2015-10-05 22:21 UTC: The DDOS followed us to the new server IP, which caused our upstream provider to null-route the IP. We’re in the process of activating the DDOS mitigation system to restore service.

2015-10-05 23:36 UTC: DDOS mitigation is active on web241 and the server should responding normally for most customers. Some customers may experience intermittent connection problems while the DDOS mitigation system is active.

2015-10-06 00:31 UTC: The attack has overwhelmed the mitigation system and the main server IP 75.126.137.87 has been null-routed. We’re working to restore service at this time.

2015-10-07 01:52 UTC: Apologies for the lack of updates. We’ve identified the target of the attack and have moved them to an isolated IP address. The attack should follow them within the next hour or so, at which point service on Web241 should stabilize.

2015-10-07 21:33 UTC: The attack has subsided and most customer domains have been moved back to the main server IP. If you’re using third-party DNS, you’ll need to point your domains back at the main server IP manually. Detailed instructions have been mailed to all customers.

-
-

[Done] Network maintenance July 8-10 2014

Posted in Scheduled downtime by

On July 8th through July 10th, our upstream provider will be performing network maintenance that will briefly interrupt network connectivity for several servers:

  • July 8:  Dweb120, Dweb121
  • July 9: Dweb127
  • July 10: Web232, Web233, Web234, Web236, Web238, Web239, Web240, Web241, Web310, Web311

On each of these days, the maintenance will take place between 7:30 am and 9:30 am EDT, and will require a network switch reboot that will result in a loss of network connectivity for up to 30 minutes (but probably less). We’ll update this post as the maintenance progresses.

 

2014-07-10 This maintenance was completed.

-
-

[Fixed] Network routing issues affecting several US servers

Posted in Downtime by

Several of our US servers are currently experiencing network routing issues. Affected servers may include:

Web239, Web241, Web240, Web238, Web310, Web311, Web312, Web313, Web236, Web234, Web232, Web233, Web235

We are investigating the issue currently.

2013-01-13 20:43 UTC A backup switch has been put in place to restore service while further troubleshooting continues on the primary switch that experienced issues.

2013-01-13 22:31 UTC We are currently replacing the switch.

2013-01-13 22:59 The switch has been replaced and all servers are back online.

-
-

[Fixed] Intermittent network routing issues affecting several US servers

Posted in Problems by

Several of our US servers are currently experiencing intermittent network routing issues. Affected servers may include: Dweb100 Dweb101 Dweb102 Dweb104 Dweb105 Dweb110 Dweb111 Dweb112 Dweb113 Dweb114 Dweb115 Dweb116 Dweb117 Dweb118 Dweb119 Dweb120 Dweb121 Dweb122 Dweb123 Dweb124 Dweb125 Dweb126 Dweb127 Dweb128 Dweb129 Dweb130 Dweb133 Dweb134 Dweb135 Dweb137 Dweb140 Dweb141 Dweb142 Dweb143 Dweb144 Dweb145 Dweb146 Dweb147 Dweb149 Dweb150 Dweb151 Dweb152 Dweb153 Dweb154 Dweb158 Dweb160 Dweb161 Dweb162 Dweb163 Dweb164 Dweb91 Dweb92 Dweb93 Dweb94 Dweb95 Dweb96 Dweb97 Mailbox8 Web102 Web105 Web106 Web108 Web11 Web114 Web117 Web119 Web12 Web122 Web126 Web129 Web143 Web148 Web15 Web151 Web155 Web162 Web174 Web175 Web178 Web180 Web182 Web183 Web186 Web187 Web198 Web199 Web200 Web213 Web219 Web220 Web226 Web227 Web228 Web229 Web230 Web231 Web232 Web233 Web234 Web235 Web236 Web237 Web238 Web239 Web24 Web240 Web241 Web243 Web244 Web245 Web246 Web247 Web25 Web27 Web28 Web30 Web300 Web301 Web302 Web307 Web308 Web309 Web31 Web310 Web311 Web312 Web313 Web318 Web319 Web320 Web324 Web328 Web329 Web330 Web335 Web336 Web337 Web338 Web34 Web341 Web342 Web343 Web344 Web345 Web346 Web347 Web348 Web349 Web35 Web37 Web39 Web4 Web40 Web42 Web48 Web49 Web5 Web55 Web57 Web65 Web69 Web70 Web72 Web74 Web75 Web80 Web83 Web91 Web95 Web99

We’re working to resolve this issue and will update this post when we have more information.

2012-10-15 6:07 UTC: The problem was an issue with an upstream network carrier and has been resolved.

-
-