Web178 filesystem went read only, we’re about to take the server down to run a fsck on it. We will update this post as maintenance progresses.
2013-02-26 13:20 UTC: FSCK is now running, progress is at 22%
2013-02-26 13:47 UTC: FSCK is at 50%
2013-02-26 14:54 UTC: FSCK completed and the server is back to operational status.
2013-02-26 15:22 UTC: Filesystem went read only again, we’re currently investigating the issue and will report progresses on this post.
2013-02-26 16:07 UTC: The server is back at operational status.
2013-02-26 17:14 UTC: The filesystem has gone read-only again. We’re working to restore service and will update this post when we have more information.
2013-02-26 18:15 UTC: We are now running fsck on the machine again and it is at 57%.
2013-02-26 20:03 UTC: After the initial fsck finished the file system is still not returning a clean file system from our checks so we are running further file system checks.
2013-02-26 21:51 UTC: Our further checks have shown some filesystem inodes that were multiply referenced and need to be manually removed. We are working on this now.
2013-02-26 23:53 UTC: The inodes have been removed and we’re now running a final fsck on the file system to insure it is clean of all file system errors before booting.
2013-02-27 00:07 UTC: The server is now back online and functioning normally. We’ll continue to watch the server and its filesystem very closely to insure that all errors were resolved.