[DONE] Emergency maintenance on Web34

Posted in Downtime by

Web34′s filesytem went read-only. We’re currently taking the server down to run a fsck. We will update this post as maintenance progresses.

2013-04-30 14:11 UTC: We are still running fsck on the server.

2013-04-30 15:04 UTC: We are still running fsck on the server.

2013-04-30-16:15 UTC: fsck has completed and the server is back at operational status.

-
-

[Fixed] Disk failure on Web126

Posted in Downtime by

One of the disks on Web126 is failing and driving up system load, causing degraded performance.

We’re scheduling an emergency disk swap and will update this post when we have more information.

2013-04-29 19:11 UTC: The failed disk on Web126 has been replaced and server load has returned to normal levels.

-
-

[Done]Emergency maintenance on Web4, April 30th, 2013.

Posted in Downtime by

Web4’s filesytem went read-only. We’re currently taking the server down to run a fsck. We will update this post as maintenance progresses.

2013-04-30 10:34 UTC fsck completed and the server is back at operational status.

-
-

[Done]Emergency maintenance on web338

Posted in Downtime by

Web338 has shown poor performance and stopped responding, so we have taken the server offline for a filesystem check and hardware inspection. We will update this post as progress completes.

2013-04-28 05:09 UTC: The fsck has finished and the server is back online, but there are still some issues which need to be addressed, so the server will be rebooted again once more soon.

2013-04-28 05:44 UTC: We are now rebooting the machine after some changes to bring it fully back online.

2013-04-28 06:49 UTC: The server is now online and functioning normally.

-
-

[Done]Scheduled webmail maintenance Tuesday, 30 April 2013 – Midnight UTC

Posted in Downtime by

We will be performing scheduled maintenance on webmail.webfaction.com on Tuesday, 30 April 2013 from Midnight UTC  to approximately 02:30 a.m. We will update this post as maintenance progresses.

-
-

[Done]Emergency maintenance on Web122

Posted in Downtime by

Web122 filesystem went read only. We’re currently taking the server down for a fsck. We will update this post as maintenance progresses.

2013-04-24 14:40 UTC fsck is at 50%

2013-04-24 15:12 fsck finished and the server is back to operational status.

Web122 filesystem showing errors. We’re currently taking the server down to run a FSCK. We will update this post as maintenance progresses.

2013-04-25 16:28 UTC started fsck.

2013-04-25 16:49 UTC: At this time we are pausing the fsck operation in order to preserve the most data possible on Web122. We’ve already begun preparing a spare server to become the new Web122. Once our preparation is complete we will move all data on the current Web122 to the new Web122.

2013-04-25 18:39 UTC: We have begun the migration of user $HOME directory data to the new server.

2013-04-25 20:33 UTC: We are still migrating user $HOME data to the machine.

2013-04-26 00:27 UTC: All MySQL and PostgreSQL databases have now been copied to the new server and both are online. We are approximately half way done transferring user $HOME data to the new server.

2013-04-26 04:12 UTC: We’re still transferring user $HOME data to the machine. We’ve transferred approximately 4/5ths of the data from the old server to the new one. All user authentication is working correctly.

2013-04-26 07:35 UTC: All user data has now been transferred to the new server. The server is online and functioning normally.

-
-

[Done]Emergency maintenance on Web223, 23rd April, 2013.

Posted in Downtime by

Web223 filesystem shows errors. We’re currently taking the server down to run a FSCK. We will update this post as maintenance progresses.

2013-04-23 10:40 UTC FSCK finished and the server is back at operational status.

-
-

[Done] PostgreSQL upgrade on Web200

Posted in Scheduled downtime by

We are going to upgrade Postgresql 8.3 / PostGIS 1.3.6 to Postgresql 8.4 / PostGIS 1.5 on Web200 between 5 and 7 am UTC today.

Postgresql will be unavailable for the duration of the upgrade.

We will update this post when the upgrade is done.

[Apr 22 05:56] The upgrade is done and PostgreSQL is back on the new version. If you notice any problems with your databases please open a support ticket and we’ll look into it asap.

-
-

[Fixed] Read-only filesystem on Web389

Posted in Downtime by

The filesystem on Web389 is currently read-only. We’re working to restore service and will update this post when we have more information.

Update: We ran fsck on the server. It completely quickly and the server is now back to normal.

-
-

[Fixed] Read-only filesystem on Web174, downtime

Posted in Downtime by

Web174 has just gone read-only and we’re taking it down to fix it.

[Apr 19 05:27 UTC] The fsck is at 20% now.
[Apr 19 06:14 UTC] The fsck is at 70% now.
[Apr 19 06:32 UTC] The server is back  and is OK now.
-
-