[Done] Postgres and SQLite upgrade on Web244

Posted in Downtime by

We are going to upgrade Postgresql 8.3 / PostGIS 1.3.6 to Postgresql 8.4 / PostGIS 1.5 on Web244

at July 6th 2013 between 2AM and 4AM UTC.

SQLite will be also be upgraded from 3.3.6 to 3.5.9

Postgresql will be unavailable for the duration of the upgrade.

We will update this post as the upgrades progress.

[Jul  6 02:43] The upgrade is done.


[Done]Filesystem Upgrade on Web209

Posted in Downtime by

Web209′s filesystem will be upgraded from ext3 to ext4 on Saturday 6th July 2013

between 5am to 9 am UTC.

We will be updating this post as the upgrade progresses.

[Jul  6 05:31] We have bought the server down for the upgrade.

[Jul  6 08:34] The filesystem upgrade and the filesystem check after it are done, the server is having some problems

booting so we are working on bringing it down to rescue mode to investigate that.

[July 6 09:12] The kernel does not seem to be booting the device as ext4 even though the conversion is complete so we

are converting the filesystem back to ext3 so that the system can boot properly.

[Jul  6 13:32] The server is still not booting and we are trying to get it to rescue to fix the problem with its booting.

[Jul  6 15:34] The server is still not booting so we are moving this server to a new machine. We will start migrating data for it shortly.

[Jul  6 16:09] The data migration has started, the new IP of the server is

2013-07-06 17:59 UTC:  The data migration is ongoing.

2013-07-06 19:07 UTC:  Approximately 15% of the data has been transferred.

2013-07-06 20:44 UTC:  Approximately 23% of the data has been transferred.

2013-07-06 22:27 UTC:  Approximately 32% of the data has been transferred.

2013-07-07 00:19 UTC:  Approximately 43% of the data has been transferred.

2013-07-07 03:24 UTC:  MySQL, Postgres and Cronjobs have been restored so the sites should be coming online as the data transfer continues.

2013-07-07 07:30 UTC:  Approximately 80% of the data has been transferred.

2013-07-07 09:47 UTC:  All the data has been transferred.


[fixed] Web404 emergency maintenance

Posted in Downtime by

Web404 has a bad memory module and has been taken down for it to be replaced.

2013-07-03 23:35 UTC: The disk array is being checked to confirm it’s health.

2013-07-04 01:03 UTC: The hardware has been replaced and the server is back online again.


Reminder: IP address change for ns1.webfaction.com

Posted in Downtime by

As announced last week, ns1.webfaction.com will point to instead of currently starting today.

If you own any domains and use our DNS servers then you may have given the previous IP addresses to your domain registrar. If that is the case then we recommend that you update the IP addresses in your registrar’s control panel as soon as possible (you don’t have to wait until tomorrow) . Since we are changing the IP addresses of just one of our four DNS server there is no danger of your website or emails being inaccessible while you make the change.

Some registrars only ask for the DNS servers names not their IP addresses. If that’s the case with your registrar then you don’t need to do anything.

You can find more information about our DNS servers here: http://docs.webfaction.com/user-guide/domains.html#pointing-your-domain-to-webfaction-s-servers


[Done]Emergency maintenance on Web23

Posted in Downtime by

Web23 is currently down for emergency maintenance. The server has been unstable over the last 24 hours, we’ve had all of the RAM in the machine swapped earlier. We’re now going to have a full chassis replaced.

2013-07-02 19:02 UTC:  The server is now back online pending the equipment for the chassis swap being ready.

2013-07-02 20:16 UTC: The server is now being taken offline for the chassis swap.

2013-07-02 21:47 UTC: The server is still offline while the chassis swap progresses.

2013-07-02 22:56 UTC: The server’s chassis has been replaced but the RAM in the new chassis is failing to properly boot. We are still working with the datacenter to get this machine back online.

2013-07-02 23:55 UTC: The server is now back online. We’re continuing to monitor the server closely to verify that the chassis swap has fixed the hardware problems.

2013-07-03 18:52 UTC: The hardware on the machine has all been replaced so now we are investigating software issues that may be causing the downtime.


[Fixed] Web155 down

Posted in Downtime by

Web155 went down during a kernel upgrade. We’re working to put the server back online asap.

Update 11:06 UTC: The server is back to normal