Network problems in the data center are currently causing degraded performance and intermittent outages on multiple servers. We’re looking into the problem and hope to have it resolved soon. We will update this post as the situation develops.
We will be investigating the cause of visual alarms on mail5.webfaction.com and web11.webfaction.com at 30 Dec 2009 0400 GMT-6.
Depending on the severity we might have to take the server down and if so it will be down between a few minutes and a few hours. We will update this ticket as soon as we have more information tomorrow.
We are looking into the issue and will update this post when we get more info.
Update 1: We rebooted the server and it is currently running fsck.
Update 2: fsck is currently on a second pass
[09:10 PM CST] Update: The HD appears to be failing so we are going to attempt to back up what we can and replace it.
[10:50 PM CST] Update: The HD backup is underway.
[1:25 AM CST] Update: The HD backup is still underway.
[3:13 AM CST] Update: We are now going to reinstall the server and restore the data.
[7:54 AM CST] Update: The restoration of the data is still ongoing.
[1:17 PM CST] Update: The restoration of the data is still ongoing but some network issues between datacenters are slowing down the restoration. If you want to setup your site on another server in the mean time just open a ticket and we’ll give you a free extra plan on another machine.
[3:52 PM CST] Update: The network issues between datacenters have been fixed and data restoration is now happening at full speed. Data restoration is about 50% complete.
[6 PM CST] Update: The server is now back online. Open a ticket if you notice any problem with your account. We would like to apologize for the extended downtime. We are working hard to improve our procedures and our setup to minimize downtime when file system corruptions happen.
Web44 is currently experiencing very high load as we are restoring one final very large database to that machine. The high load is causing slow performance and intermittent problems with SSH logins. We expect the restore to be complete between 5PM and 6PM US Central time, at which point normal service should be restored.
We’re aware that there are some missing Python modules on Web44. Once the DB restore is complete, we’ll look into that issue.
Update 1: Web44 is back online and is functioning normally.