[Done]Scheduled maintenance on Web31, December 5th, 2012.

Posted in Scheduled downtime by

Web31 will be taken down for a fsck Wednesday December 5th 2012 between 09:00 UTC and 11:00 UTC. We will update this post as maintenance progresses.

2012-12-05 13:01 UTC Fsck completed. The server is back to operational status.

-
-

[Fixed] Emergency Maintenance on Web332.

Posted in Downtime by

Web332 is not detecting its disks. We are currently investigating the issue and working to bring the server online.

[ 2012-12-04 06:10 UTC]  We have identified the problem as a malfunctioning RAID controller which is being replaced now.

[2012-12-04 06:28 UTC] The server is back after the RAID controller was replaced.

-
-

[Fixed] web40 is unavailable

Posted in Downtime by

web40 is currently down. We are investigating the issue.

2012-12-03 22:57 UTC: The server is now back online and functioning normally.

-
-

[Fixed] web230 is unavailable

Posted in Downtime by

web230 is currently down. We are investigating the issue.

2012-12-01 22:08 UTC: We’re running fsck on damaged partitions now.

2012-12-01 23:47 UTC: We’re still running fsck on damaged partitions.

2012-12-02 01:13 UTC: The filesystem check has returned numerous errors that are making the check process slower than normal and doesn’t insure that the machine will come back with all of the data intact. To this end we’re going to reload the operating system on Web230 and restore the data from our backups. The OS is currently being reloaded.

2012-12-02 08:15 UTC: The server OS has been reloaded and we have restored the mysql and postgresql databases from our backup, the restoration for home directories is still going on.

2012-12-02 10:53 UTC: The home directories restoration is still going on.

2012-12-02 14:00 UTC: The home directories have been restored from the backups and the server is back.

-
-