[Fixed] Slow performance, intermittent SSH and Python problems on Web44

Posted in Problems by

Web44 is currently experiencing very high load as we are restoring one final very large database to that machine. The high load is causing slow performance and intermittent problems with SSH logins. We expect the restore to be complete between 5PM and 6PM US Central time, at which point normal service should be restored.

We’re aware that there are some missing Python modules on Web44. Once the DB restore is complete, we’ll look into that issue.

Update 1: Web44 is back online and is functioning normally.

-
-

[Fixed] Web44 down

Posted in Downtime by

Web44 is currently down and we’re looking into it. We’ll update this post when we have more information

Update 1: The problem appears to be related to the filesystem. We are currently running fsck on the server.

Update 2: Unfortunately fsck didn’t fix the issue so we are going to re-install the machine and restore all data from backup

Update 3: We are still recovering all the data from our backup servers. We are using our latest backup which is from less than 24h before Web44 went down.

Update 4: The machine is now back to normal apart from a few large MySql database which are still being imported into MySql. Open a ticket is you notice any problem with your account.

-
-

[Done] Kernel upgrades on RedHat servers tomorrow

Posted in Scheduled downtime by

We will be upgrading the kernels on all RedHat Enterprise Linux 4 servers tomorrow between 3pm GMT and 7pm GMT.

The downtime on each server should only be a few minutes.

The servers are mamba, krait, web1 to web20, dweb3 to dweb14 and mail1 to mail4.

We’ll update this post once the work is done.

Update [12:58 PM CST]: Done

-
-

[Done] Scheduled downtime on Web104

Posted in Scheduled downtime by

We will be taking down Web104 tomorrow at 4am CST to replace a failing drive in the RAID array.

The downtime should only last a few minutes. We’ll update this post once the work is completed.

Update 11.50am GMT: The drive has been replace and the server is now back online

-
-

[Fixed] Web5 down

Posted in Downtime by

Web5 is currently down and we’re looking into the issue.

We’ll update this post when we have more information.

Update: All websites on the server are now working but there is still an issue with some people not being able to access the server via SSH. We’re working on fixing the issue.

2009-11-07 14:33 CST: SSH services should now be working for all users on Web5. Please open a support ticket if you have problems accessing Web5.

-
-

[Done] Scheduled downtime on Mail3

Posted in Scheduled downtime by

On November 18, 2009 we will be moving Mail3 to a new rack.

The downtime last be less than 1h and will happen between midnight and 5am CDT.

We will update this post when the maintenance is complete.

Update: the maintenance is now complete

-
-