[Done] Scheduled down time on all web servers.
http://statusblog.webfaction.com/scheduled-down-time-on-all-web-servers-ii is done. We did not end up having to reboot Web 2 through Web 20 or Dweb 1 through Dweb 18.
http://statusblog.webfaction.com/scheduled-down-time-on-all-web-servers-ii is done. We did not end up having to reboot Web 2 through Web 20 or Dweb 1 through Dweb 18.
As a part of http://statusblog.webfaction.com/scheduled-down-time-on-all-web-servers-ii Web 85 was scheduled to be down for 10 minutes, it has been down for 45 minutes. Service will be restored ASAP.
Update [01:45 PM CDT]: Service has been restored. Web 85 was not down, it was inaccessible due to a loose network cable.
See http://statusblog.webfaction.com/scheduled-down-time-on-all-web-servers-ii for more details.
Web17 stopped responding several minutes ago and is currently being rebooter. We should have service restored soon.
2009-08-27 11:12 – Web17 is back online. The problem appears to have been extremely high load due to a massive number of incoming FTP connections.
At 6pm GMT today we will take down Web61 to check the filesystem with fsck. We will update this post with our progress.
Update 1.10pm CDT time: fsck has now started
Update 3pm CDT time: the fsck run is now over and the server is back online
The filesystem on Web61 went into a read-only state at around 19:30 CDT. We rebooted the machine at that time and it has not yet come back online.
We’re working with the data center team to resolve this issue and hope to have service restored soon. We’ll update this post when we have more info.
2009-08-15 20:46 CDT – web61 is back online, but the filesystem problem remains. We are still investigating the problem.
2009-08-16 04:26 CDT – web61 is now back online and working. We will schedule a fsck run for later in the day.
The filesystem on Web8 went into a read-only state several minutes ago. We rebooted the server, and it is currently running a filesystem check. We hope to have service restored soon.
We are currently experiencing an issue with our “mail2script” feature (see http://blog.webfaction.com/mail2script) which is causing it to miss a small percentage of emails.
We will update this post as soon as we have a resolution.
On Sunday August 31 between 11:00 AM and 3:00 PM CDT we will be rebooting all web servers to deploy a kernel update. We do not expect any server to see more than 10 minutes of down time.
web41.webfaction.com is currently down. We are looking into the problem and hope to have service restored soon.
Update [011:41 AM CST]: Upon rebooting the server we were faced with a kernel panic. We are still working on restoring service.
Update [02:35 PM CST]: We are still working on restoring service.
Update [04:40 PM CST]: We are still working on restoring service. If you would like you can submit a support ticket and we will give you access to another server. If we are not able to restore service soon we will setup another server to take its place.
Update [08:35 PM CST]: Service has been restored. The problem was file system configuration related and has been fixed.