[Done]Scheduled maintenance on Web108, August 10th, 2012.

Posted in Scheduled downtime by

Web108 will be taken down for a remote card update Friday August 10th 2012 at 13:00 UTC. We will update this post as maintenance progresses.

2012-08-10 12:38 UTC We’ve been able to update it without the need of a reboot. The server never left the operational status.

-
-

[Done] Scheduled Maintenance on Web102, August 8th, 2012.

Posted in Scheduled downtime by

Web102 will be taken down for a remote management card update, Wednesday August 8th 2012, at 13:00 UTC. We will update this post as maintenance progresses.

Update: The server is now back online

-
-

[Done]web172 down – filesystem read-only

Posted in Downtime by

The filesystem on Web172 went read-only and we’re currently running a forced filesystem check. We’ll update this post as we have more information.

2012-08-07 18:00 UTC: the filesystem check is 32% complete.

2012-08-07 19:54 UTC: the filesystem check is 42% complete.

2012-08-07 21:01 UTC: The filesystem check is 51% complete.

2012-08-07 22:54 UTC: The filesystem check is at 52.2% complete.

2012-08-08 02:30 UTC: The filesystem check is at 54.8% complete.

2012-08-08 04:30 UTC: The filesystem check is at 56.3% complete.

2012-08-08 12:30 UTC: The filesystem check has restarted.

2012-08-08 18:27 UTC: The filesystem check is 55.5% complete.

2012-08-08 21:27 UTC: The filesystem check is 59.5% complete.

2012-08-09 02:10 UTC: The filesystem check has returned numerous errors that are making the check process slower than normal and doesn’t insure that the machine will come back with all of the data intact. To this end we’re going to reload the operating system on Web172 and restore the data from our backups. The OS is currently being reloaded.

2012-08-09 06:14 UTC: The operating system has been reinstalled and we’re currently installing our platform tools on the machine. Once this is finished we’ll proceed with restoring user data.

2012-08-09 07:33 UTC: Our platform tools have been installed and all user databases have been reloaded. We’re now restoring user data.

2012-08-09 09:12 UTC: The user data is still being restored to the machine.

2012-08-09 11:02 UTC: The user data is still being restored to the machine.

2012-08-09 14:21 UTC: The user data is still being restored to the machine.

2012-08-09 18:37 UTC: The server is now back online and functional.

-
-

[fixed] web186 down – filesystem read-only

Posted in Downtime by

The filesystem on Web186 went read-only and we’re currently running a forced filesystem check. We’ll update this post as we have more information.

2012-08-02 20:11UTC: fsck is complete and server is back online.

-
-

[Fixed] Web186, Web334, and Web220 experiencing DDoS attack

Posted in Downtime by

Web186, Web334, and Web220 are experiencing a distributed denial-of-service attack, which is currently preventing access to all sites hosted there. We are working to mitigate the effects of the attack and will update this post with our progress.

2012-08-01 05:24 UTC: Service has been restored to Web220 and Web334. If you’re not able to access either of those servers, please open a support ticket so that we can help you get back on. Work on Web186 is still in progress.

2012-08-02 7:30:26 UTC: Service has been restored on Web186 too.

-
-

[Fixed] Network routing issues affecting multiple servers for some locations

Posted in Problems by

A network routing issue is currently affecting access to multiple web and mail servers from some locations (i.e., it doesn’t affect everybody).

We presently don’t have a list of all of the affected machines available – if the issue is not resolved in the next several minutes, then we’ll get a list together and post it here, along with any other information pertaining to the issue.

2012-08-01 19:54 UTC: The issue was a routing issue with the Level3.com (Global Crossing) network external to our own network, so it potentially affected any user on any server, depending on the user’s network route to the server. The issue has been resolved, and connectivity has been restored.

-
-

[Done]Scheduled Maintenance on Web149, August 2nd, 2012.

Posted in Scheduled downtime by

The server will be taken down for a RAM swap Thursday August 2nd 2012 between 08:00 UTC and 11:00 UTC. We will update this post as maintenance progresses.

2012-08-02 08:20 UTC The faulty RAM has been swapped, the server is back at operational status

-
-