Web64 Down (Fixed)

Posted in Downtime by

The filesystem on Web64 went read-only several minutes ago. We’re currently rebooting the machine and will have normal service restored ASAP.

2009-02-13 16:33 CST – Web64 is back online. We may need to bring it down again in the near future for filesystem maintenance, but we’ll give advance notice if we do.

2009-02-13 18:02 CST – The filesystem on Web64 just went read-only again, so we’re going to go ahead and perform filesystem maintenance now. We’ll get the machine back online ASAP.

2009-02-13 18:43 CST – The filesystem check on Web64 is still in progress.

2009-02-13 19:13 – The filesystem check is complete and Web64 is back online.

-
-

Web4 Down (fixed)

Posted in Downtime by

Web4 is down with a file system issue. We’ll update this post as soon as it is back up.

2009-02-03 10:46 CST – The filesystem check on Web4 is still in progress. We hope to have service on Web4 restored soon.

2009-02-03 11:18 CST – Web4 is back online.

-
-

Web46 DOS’d (fixed)

Posted in Downtime by

Web46 is being DOS’d. We’re hoping to restore service asap and we’ll update this post when it’s done.

Update: we were able to stop the attack so web46 is now acting normally.

-
-

Problems on Krait, Taipan, Viper and Mamba (fixed)

Posted in Downtime by

We’ve had some issues on these servers since yesterday. Basically, a misconfiguration (2 characters to be exact) in our memory watchdog script caused it to kill some root processes it shouldn’t have. This means that we had some SSH issues, DNS issues and databases issues on these servers (basically, these services were dying regularly and getting restarted later).

Fortunately we’ve been able to track down the problem and everything should be back to normal now.

Remi.

-
-