Web106’s file system went read-only. We are currently running fsck to bring the server back up.

2012-07-25 15:39 UTC: Web106 is back up and functioning properly.

2012-07-25 16:58 UTC: The filesystem has gone read-only again. We’re working to resolve the issue.

2012-07-25 18:21 UTC: The filesystem check is complete, and we’re running other hardware diagnostic tests at this time.

2012-07-25 20:33 UTC: The filesystem continues to go into a read-only state, even after successful checks. Since the disks seem to be OK, we’re arranging for a full chassis swap at this time.

2012-07-25 22:34 UTC: The chassis has been swapped. The filesystem still contains errors after the chassis swap; we’re running a filesystem check.

2012-07-26 12:04 UTC: The filesystem check is still in progress, 45% complete.

2012-07-26 01:20 UTC: The filesystem check is still in progress, 80.5% complete.

2012-07-26 03:25 UTC: The file system check is finished and the file system seems to be stable. We’re now working to bring the server back up on the network.

2012-07-26 04:26 UTC: The server is now back online. We’ll continue to monitor the server closely to make sure that no additional filesystem, hardware, or network errors are left unresolved.

2012-07-26 07:13 UTC: The server is down again; we suspect drive failure in one of the RAID disks which is causing the read-only condition. We need to fsck, backup, and replace that drive.

2012-07-26 10:14:41 UTC: There were problems getting the server into the rescue mode but the fsck has started now.

2012-07-26 13:48 UTC FSCK completed, RAID firmware upgraded and now rebuilding. The server is back to operational status. We will keep monitoring this machine closely.