The following servers are currently offline:
Dweb101, Dweb104, Dweb110, Dweb111, Dweb112, Dweb114, Dweb115, Dweb116, Dweb117, Dweb121, Dweb122, Dweb124, Dweb125, Dweb126, Dweb129, Dweb130, Dweb133, Dweb134, Dweb135, Dweb137, Dweb140, Dweb141, Dweb142, Dweb144, Dweb145, Dweb146, Dweb149, Dweb150, Dweb151, Dweb153, Dweb154, Dweb160, Dweb162, Dweb163, Dweb164, Dweb165, Dweb166, Dweb167, Dweb173, Dweb176, Dweb178, Dweb184, Dweb185, Dweb187, Dweb188, Dweb189, Dweb194, Dweb195, Dweb196, Dweb197, Dweb202, Dweb203, Dweb204, Dweb205, Dweb95, Mailbox8, Web102, Web105, Web106, Web108, Web114, Web117, Web126, Web129, Web148, Web162, Web175, Web180, Web182, Web187, Web198, Web219, Web220, Web227, Web229, Web232, Web236, Web238, Web24, Web240, Web243, Web244, Web247, Web25, Web27, Web28, Web30, Web300, Web308, Web311, Web318, Web319, Web320, Web328, Web330, Web342, Web343, Web345, Web347, Web35, Web49, Web55, Web80, Web91
This may be related to scheduled network maintenance by our upstream provider, but that was only expected to impact a few servers. We’re awaiting confirmation from our upstream provider and will update this post when we have more information.
2016-02-23 19:14 UTC: The problem appears to have been a power disturbance in LeaseWeb’s US data center. The machines listed above are coming back online at this time.
2016-02-23 20:34 UTC: The following servers are still offline at this time:Web25, Web106,Web182,Web300, Web308, Web343, Web347 – all of the other servers are back online. We’re working to restore service for the remaining machines and will update this post when we have more information.
2016-02-23 22:00 UTC:
- Web343 is back online, but its MySQL service is down – we’re presently working to restore it.
- Web106 and Web182 are showing signs of filesystem corruption. We’re running tests to confirm.
- Web25, Web300, Web308, and Web347 are still being investigated.
2016-02-23 22:18 UTC: Web182 is back online.
2016-02-23 22:29 UTC: The MySQL service on Web343 is now restored.
2016-02-23 22:30 UTC: We’re running a filesystem check on Web25.
2016-02-23 22:42 UTC: Web182 is offline again.
2016-02-23 22:50 UTC: Web182 and Web347 are both back online. The machines that are still down are Web25, Web106, Web300, and Web308.
2016-02-23 23:24 UTC: Web25 is back online.
2016-02-24 04:03 UTC: Dweb142 is offline again at this time.
2016-02-24 06:14 UTC: Some servers that have returned online are experiencing issues impacting our services. Our system administrators are working to restore these servers to full operation. Please check back here for further updates.
2016-02-24 12:35 UTC: At this point all servers are working normally except for Web106 and Web300. We expect to have these last two servers back online shortly.
2016-02-24 17:29 UTC: Web300 is back online.
2016-02-24 20:21 UTC: At this time, Web106 seems to be in good health, but we’re unable to bring up its network interfaces. We’re presently working to resolve that, and are bringing up a backup machine as a final contingency in the event that we’re unable to get Web106 back online.
2016-02-25 00:19 UTC: Web106 is back online.