[Fixed] DNS problem affecting multiple servers

Posted in Problems by

A problem with one of our DNS servers briefly interrupted access to most of our servers. The issue has been resolved and all servers are now accessible.

-
-

[Fixed] Network problems affecting multiple servers

Posted in Problems by

Network problems in the data center are currently causing degraded performance and intermittent outages on multiple servers. We’re looking into the problem and hope to have it resolved soon. We will update this post as the situation develops.

Update (7.13pm CST): Servers currently affected are: Web32, Web12, Dweb21, Dweb28, Web14, Dweb52, Web24, Web54, Dweb19, Web13, Mail4.

Update (7.30pm CST): Servers are now back online.

-
-

[Fixed] Web51 is currently down

Posted in Downtime by

We are looking into the issue and will update this post when we get more info.

Update 1: We rebooted the server and it is currently running fsck.

Update 2: fsck is currently on a second pass

[09:10 PM CST] Update: The HD appears to be failing so we are going to attempt to back up what we can and replace it.

[10:50 PM CST] Update: The HD backup is underway.

[1:25 AM CST] Update: The HD backup is still underway.

[3:13 AM CST] Update: We are now going to reinstall the server and restore the data.

[7:54 AM CST] Update: The restoration of the data is still ongoing.

[1:17 PM CST] Update: The restoration of the data is still ongoing but some network issues between datacenters are slowing down the restoration. If you want to setup your site on another server in the mean time just open a ticket and we’ll give you a free extra plan on another machine.

[3:52 PM CST] Update: The network issues between datacenters have been fixed and data restoration is now happening at full speed. Data restoration is about 50% complete.

[6 PM CST] Update: The server is now back online. Open a ticket if you notice any problem with your account. We would like to apologize for the extended downtime. We are working hard to improve our procedures and our setup to minimize downtime when file system corruptions happen.

-
-

[Fixed] Slow performance, intermittent SSH and Python problems on Web44

Posted in Problems by

Web44 is currently experiencing very high load as we are restoring one final very large database to that machine. The high load is causing slow performance and intermittent problems with SSH logins. We expect the restore to be complete between 5PM and 6PM US Central time, at which point normal service should be restored.

We’re aware that there are some missing Python modules on Web44. Once the DB restore is complete, we’ll look into that issue.

Update 1: Web44 is back online and is functioning normally.

-
-

[Fixed] Network problems affecting multiple servers

Posted in Problems by

Network problems in the data center are currently causing degraded performance and intermittent outages on multiple servers. We’re looking into the problem and hope to have it resolved soon. We will update this post as the situation develops.

Update (7.46pm CST): the problem is now fixed.

-
-

[Fixed] Incoming email delay for some emails

Posted in Problems by

We are experiencing a large email queue on one of our incoming email servers and it is resulting in some delay for some incoming emails.

We are currently working on flushing the queue and we will update this post when the situation is back to normal.

Update 4.30pm GMT: We’re about halfway through the backlog of emails.

Update 9.30pm GMT: All the backlog has now been processed and email delivery is back to normal.

-
-

[Fixed] Web61 and Web89 load issues

Posted in Problems by

Web61 and Web89 are both experiencing some load issues, which is causing the server loads to spike periodically and the servers to become unresponsive for short periods of time.

The problems on the two servers are actually not related, and the types of loads on each of the servers is quite different. It is only a coincidence that both of them are having similar issues at the same time.

We are actively working on fixing this problem and we will update this post when the issues have been resolved.

Update: Web61 is now fixed.

Update: Web89 is now fixed.

-
-

Network issues on Web71 [fixed]

Posted in Problems by

Web71 is currently experiencing network issues. It should still be accessible via HTTP and FTP, but outgoing network connections may result in long delays or timeouts.

We will update this post when the issues have been resolved.

2009-09-24 19:58 CDT – The problem has now been fixed and the server rebooted.

-
-

Mail2script missing some emails

Posted in Problems by

We are currently experiencing an issue with our “mail2script” feature (see http://blog.webfaction.com/mail2script) which is causing it to miss a small percentage of emails.

We will update this post as soon as we have a resolution.

-
-