Several of the datacenters hosting our servers are having network issues and a large chunk of our servers is currently unreachable.
Update: the network issues are now fixed and all servers are back online.
Update 2009-05-13 10:38 CDT: the datacenters just experienced another brief network outage, but all servers are back online at this time.
The filesystem on Web71 went into a read-only state a few minutes ago. We are currently rebooting the server and will investigate the problem further when it is back online.
Update 2009-05-07 08:43 CDT – Web71 is back online.
We’re currently investigating a problem that is affecting sites hosted on Web54. We’ll update this entry when we have more information.
Update 2009-05-07 11:33 CDT – Web54 is back online. The problem was high load due to excessive CPU utilization by our backup script. The problem with the script has been resolved.
We are currently investigating an issue following an upgrade to CentOS-5.3. Services are down on several servers and we are currently working on restoring them. The servers are:
Update 2009-05-06 11:10 CDT: We are in the process of reinstalling some RPMs on the servers in order to bring them back online.
Update 2009-05-06 11.50 CDT: Two servers are back online (web36 and web52). We are continuing to work on the other servers.
Update 2009-05-06 12:39 CDT: Web32, Web35 and Web39 are back online. We are continuing to work on the other servers.
Update 2009-05-06 12:55 CDT: Web52 is down again.
Update 2009-05-06 13:23 CDT: Web25, Web27 and Web58 are back online. We are continuing to work on the other servers.
Update 2009-05-06 13:28 CDT: Web55 and Web68 are back online. We are continuing to work on the other servers.
Update 2009-05-06 14:06 CDT: Web38 and Web57 are back online. We are continuing to work on the other servers.
Update 2009-05-06 14:12 CDT: Web52 is back online. We are continuing to work on the other servers.
Update 2009-05-06 16:11 CDT: Web31, Web34 and Web46 are back online. We are continuing to work on the other servers.
Update 2009-05-06 16:48 CDT: Dweb40 and Web43 are back online. We are continuing to work on the other servers.
Update 2009-05-06 16:55 CDT: Web23 and Web53 are back online. We are continuing to work on the other servers.
Update 2009-05-06 17:04 CDT: Web33 and Web75 are back online. We are continuing to work on Web49 (the last server!).
Update 2009-05-06 17:26 CDT: Web49 is back online – that was the last server affected by the problem, so we’re calling this one fixed. Sorry for the trouble, folks!
Web73 is currently down while we investigate some filesystem errors. We’ll update the post as soon as we have more information.
Update (12.40pm GMT): The filesystem on the server is corrupted beyond recovery so we’re going to do an OS reload and restore the data from backup. We’ll update this post with our progress.
Update (3.30pm GMT): We have now moved the server onto new hardware (in case the filesystem errors were hardware-related) and we have started copying all the data from backup.
Update (5.30pm GMT): The server is now back up with new hardware and the data from yesterday’s backup. Note that the RSA host key has changed so your SSH client may display a warning about it.
One of the drives on web42 died and we are currently rebuilding the RAID with the new drive. We will update this post once the server is back online.
2009-04-24 12:26 CDT – the drive rebuild is complete and Web42 is back online.
Mail services on mail5.webfaction.com and webmail.webfaction.com are currently not working. We are looking into the problem and hope to have normal service restored soon. We will update this entry as we have more information.
2009-04-17 10:24 CDT – troubleshooting on mail5/webmail is still in progress.
2009-04-17 11:33 CDT – we’ve just pointed webmail.webfaction.com at a different server IP. Webmail users will be able to access webmail as soon as the DNS change propagates, but you will not have access to your usual webmail address book since it is located on mail5. If your mailbox resides on mail5, you still will not be able to access your mail. Troubleshooting on mail5 is still in progress.
2009-04-17 15:37 CDT – the problem on mail5 appears to be a failed OS upgrade. We are re-installing packages now.
2009-04-17 17:29 CDT – mail5 is back online and webmail.webfaction.com has been pointed back to mail5. All mail5 users should be able to access their mail now, but the server may be slow to respond for the next several hours as it catches up with today’s incoming mail.
Web67 is currently down while we investigate a potential problem on the filesystem. We will update this entry as we have more information.
2009-04-14 12:13 CDT – A filesystem repair is in progress on Web67. We hope to have service restored soon.
2009-04-14 12:42 CDT – The filesystem repair on Web67 is still in progress.
2009-04-14 12:44 CDT – The filesystem repair on Web67 completed successfully and the server is now online.
Our datacenter is experiencing DNS issues right now. If your app on our server needs to resolve some domain names it may be unable to do so. We’ll update this post as soon as we have more information.
Update: the DNS issues are now resolved.
Web42 is currently down while we investigate an audible alarm. We’ll update this post once we know more about the issue.
Update: The problem was a degraded RAID on the server. The server is now back online and the RAID is rebuilding in the background.