Ongoing DNS issues

Posted in Downtime by

We’re currently troubleshooting a series of DNS problems that may be disrupting service for customers on several servers. We’ll update this post when we have more information.

2014-10-30 16:22 UTC: We’ve rebooted web302 to test a possible fix.

2014-10-30 17:53 UTC: A permanent solution to this issue is forthcoming. In the meantime, we’ve made a temporary change to our resolving DNS configuration to prevent further disruptions of service.

-
-

[FIxed] DNS resolution problems

Posted in Downtime by

In the past 24 hours we experienced a disruption in our resolving DNS service. The problem has been corrected, but there have been some lingering issues that may have caused PHP applications on Web119 and lower to be unable to resolve hostnames via DNS.

We’ve restarted the Apache service on those machines, which appears to have resolved the problem. If you’re still experiencing DNS resolution problems in your applications (any type of app on any server) then please open a support ticket and we’ll look into it ASAP.

-
-

[Done] DDoS attack on Web404

Posted in Downtime by

The server is currently under a heavy DDoS attack. We are working with our network engineers to resolve the issue.

2014-10-10 06:06 UTC: Web404 is back online.

2014-10-10 12:30 UTC:  The server has been placed under our mitigation system which has improved the situation, but packet loss and other network problems are still to be expected. We will keep monitoring the server.

2014-10-10 15:30 UTC:  We have removed the mitigation system since the DDoS attack has subsided.

-
-

[Fixed] Web216 not accessible

Posted in Downtime by

The server has stopped responding. We are looking into this and will resolve as quickly as possible.

2014-10-07 22:00 UTC: The server is back up and responding. There was no indication why it had stopped responding. A reboot resolved the issue.

2014-10-24 15:15 UTC: Currently we are experiencing an issue in the rack where web216 resides, we are investigating this issue and will post an update as soon as more information is available.

2014-10-24 16:38 UTC: The power bar has been replaced and the sever is back online.

-
-

[Done] Scheduled maintenance on Web430, October 3nd, 2014.

Posted in Downtime by

Web430 will be taken down for a fsck Friday, October 3rd 2014 between 07:00 UTC and 12:00 UTC. We have scheduled a 5-hour window for this maintenance though the actual downtime is likely to be a lot less than that. We will update this post as maintenance will progress.

2014-10-03 07:19 UTC: We are now preparing to take this server offline for the filesystem check.

2014-10-03 08:06 UTC: Preparations are complete and the server has just now gone offline for maintenance.

2014-10-03 08:31 UTC: The filesystem has been repaired and the server is now being brought back online.

2014-10-03 08:36 UTC: The maintenance is over and the server is back online.

-
-

[Done] Scheduled maintenance on Web241, October 2nd, 2014.

Posted in Downtime by

Web241 will be taken down for a fsck Thursday, October 2nd 2014 between 07:00 UTC and 12:00 UTC. We have scheduled a 5-hour window for this maintenance though the actual downtime is likely to be a lot less than that. We will update this post as maintenance will progress.

2014-10-02 12:07 UTC: The server is currently able to be accessed via SSH and HTTP(S).  Our system administrators are still repairing the filesystem.  We will update this statusblog when the repair is complete.

2014-10-02 14:12 UTC: The maintenance is taking longer than planned and we are working as quickly as possible to get the server back online.

2014-10-02 15:18 UTC: The maintenance is over and the server is back online.

-
-