The filesystem on Web172 went read-only and we’re currently running a forced filesystem check. We’ll update this post as we have more information.
2012-08-07 18:00 UTC: the filesystem check is 32% complete.
2012-08-07 19:54 UTC: the filesystem check is 42% complete.
2012-08-07 21:01 UTC: The filesystem check is 51% complete.
2012-08-07 22:54 UTC: The filesystem check is at 52.2% complete.
2012-08-08 02:30 UTC: The filesystem check is at 54.8% complete.
2012-08-08 04:30 UTC: The filesystem check is at 56.3% complete.
2012-08-08 12:30 UTC: The filesystem check has restarted.
2012-08-08 18:27 UTC: The filesystem check is 55.5% complete.
2012-08-08 21:27 UTC: The filesystem check is 59.5% complete.
2012-08-09 02:10 UTC: The filesystem check has returned numerous errors that are making the check process slower than normal and doesn’t insure that the machine will come back with all of the data intact. To this end we’re going to reload the operating system on Web172 and restore the data from our backups. The OS is currently being reloaded.
2012-08-09 06:14 UTC: The operating system has been reinstalled and we’re currently installing our platform tools on the machine. Once this is finished we’ll proceed with restoring user data.
2012-08-09 07:33 UTC: Our platform tools have been installed and all user databases have been reloaded. We’re now restoring user data.
2012-08-09 09:12 UTC: The user data is still being restored to the machine.
2012-08-09 11:02 UTC: The user data is still being restored to the machine.
2012-08-09 14:21 UTC: The user data is still being restored to the machine.
2012-08-09 18:37 UTC: The server is now back online and functional.
Web186, Web334, and Web220 are experiencing a distributed denial-of-service attack, which is currently preventing access to all sites hosted there. We are working to mitigate the effects of the attack and will update this post with our progress.
2012-08-01 05:24 UTC: Service has been restored to Web220 and Web334. If you’re not able to access either of those servers, please open a support ticket so that we can help you get back on. Work on Web186 is still in progress.
2012-08-02 7:30:26 UTC: Service has been restored on Web186 too.
A network routing issue is currently affecting access to multiple web and mail servers from some locations (i.e., it doesn’t affect everybody).
We presently don’t have a list of all of the affected machines available – if the issue is not resolved in the next several minutes, then we’ll get a list together and post it here, along with any other information pertaining to the issue.
2012-08-01 19:54 UTC: The issue was a routing issue with the Level3.com (Global Crossing) network external to our own network, so it potentially affected any user on any server, depending on the user’s network route to the server. The issue has been resolved, and connectivity has been restored.