Web59 Down (fixed)

Posted in Downtime by

Web59 is currently inaccessible and we’re looking into it.

Update: The problem was a misconfiguration in the firewall and it is now fixed.

-
-

Emails rejected by hotmail (fixed)

Posted in Downtime by

E-mails sent through our e-mail platform are currently rejected by Hotmail. We are currently talking to the Hotmail team to get this resolved as soon as possible. We will update this post once it is fixed.

Update: We are still working with Hotmail and the ban should be lifted tomorrow at the latest.

Update (4.09am GMT): E-mails to Hotmail are now going through again.

-
-

Web 69 Down (fixed)

Posted in Downtime by

Web 69 is currently down. Its root partition went read only and rebooting it revealed an issue which we are currently working on resolving now. We will post updates as they are available.

It appears to be an issue with the RAID controller and we are currently replacing the hardware and restoring all data from backup.

2009-03-09 06:00 PST: Web69 is still down having suffered a serious RAID controller failure. We have recovered all of the data from the server and are currently working on restoring it to a new standby server which will replace web69.

2009-03-09 06:16 PST: Web69 is now back online with all its data. We decided to move the data onto a new server to give us more time to check the hardware on the failing server. We copied all the data from just before the crash so no data has been lost.

-
-

Mail5 and Webmail problems (fixed)

Posted in Downtime by

Mail services on mail5.webfaction.com and webmail.webfaction.com are currently not working. We are looking into the problem and hope to have normal service restored soon. We will update this entry as we have more information.

2009-02-20 13:46 CST The Mail5/Webmail server has a disk problem. Repairs are now in progress.

2009-02-20 14:17 Repairs on Mail5 are still in progress. We have pointed ‘webmail.webfaction.com’ to a different mail server, so as soon as that DNS change propagates you’ll be able to access the webmail system (unless your mailbox resides on mail5). Your existing webmail address book and preferences will not be available, since they are stored on the server that is currently having problems.

2009-02-20 14:48 Mail5 is back online. The webmail system is still running on the alternate server.

2009-02-20 15:04 webmail.webfaction.com is pointing at the original server, so webmail users should now have access to their address books and preferences.

-
-

Web37 Down (Fixed)

Posted in Downtime by

Web37 is currently down. We are investigating the problem at this time and will update this entry as we have more info.

2009-02-17 9:25 CST – Web37 is back online.

-
-

Web64 Down (Fixed)

Posted in Downtime by

The filesystem on Web64 went read-only several minutes ago. We’re currently rebooting the machine and will have normal service restored ASAP.

2009-02-13 16:33 CST – Web64 is back online. We may need to bring it down again in the near future for filesystem maintenance, but we’ll give advance notice if we do.

2009-02-13 18:02 CST – The filesystem on Web64 just went read-only again, so we’re going to go ahead and perform filesystem maintenance now. We’ll get the machine back online ASAP.

2009-02-13 18:43 CST – The filesystem check on Web64 is still in progress.

2009-02-13 19:13 – The filesystem check is complete and Web64 is back online.

-
-

Web4 Down (fixed)

Posted in Downtime by

Web4 is down with a file system issue. We’ll update this post as soon as it is back up.

2009-02-03 10:46 CST – The filesystem check on Web4 is still in progress. We hope to have service on Web4 restored soon.

2009-02-03 11:18 CST – Web4 is back online.

-
-

Web46 DOS’d (fixed)

Posted in Downtime by

Web46 is being DOS’d. We’re hoping to restore service asap and we’ll update this post when it’s done.

Update: we were able to stop the attack so web46 is now acting normally.

-
-

Problems on Krait, Taipan, Viper and Mamba (fixed)

Posted in Downtime by

We’ve had some issues on these servers since yesterday. Basically, a misconfiguration (2 characters to be exact) in our memory watchdog script caused it to kill some root processes it shouldn’t have. This means that we had some SSH issues, DNS issues and databases issues on these servers (basically, these services were dying regularly and getting restarted later).

Fortunately we’ve been able to track down the problem and everything should be back to normal now.

Remi.

-
-