[Done] Postgres upgrade on Web229-Web238

Posted in Downtime by

We are going to upgrade Postgresql 8.3 / PostGIS 1.3.6 to Postgresql 8.4 / PostGIS 1.5 on the following servers in the

upgrade interval mentioned alongside.

Web229    May 24, 2am to 4am UTC

Web230    May 24, 2am to 4am UTC

Web231    May 24, 2am to 4am UTC

Web232    May 24, 3am to 5am UTC

Web233    May 24, 3am to 5am UTC

Web234    May 24, 3am to 5am UTC

Web235    May 24, 4am to 6am UTC

Web236    May 24, 4am to 6am UTC

Web237    May 24, 4am to 6am UTC

Web238    May 24, 4am to 6am UTC

Postgresql will be unavailable for the duration of the upgrade.

We will update this post as the upgrades progress.

[May 24 02:47] The upgrade on Web230 and Web231 is over.

[May 24 02:59] The upgrade on Web229 is over.

[May 24 03:38] The upgrade on Web232,Web233 and Web234 is over.

[May 24 04:39] The upgrade on Web235-Web238 is over.

-
-

[Done]Emergency maintenance on Web385

Posted in Downtime by

Web385 experienced a kernel panic a short while ago.  We have rebooted the server but it is showing filesystem errors so we are taking the server down to run a fsck.  We will update this post as maintenance progresses.

2013-05-20 08:28 UTC FSCK completed, the server is back to operational status.

-
-

[Done] Postgres upgrade on Web216-218,Web220 and Web224-227

Posted in Downtime by

We are going to upgrade Postgresql 8.3 / PostGIS 1.3.6 to Postgresql 8.4 / PostGIS 1.5 on the following servers in the

upgrade interval mentioned alongside.

Web216    May 21, 2am to 4am UTC

Web217    May 21, 2am to 4am UTC

Web218    May 21, 2am to 4am UTC

Web220    May 21, 3am to 5am UTC

Web224    May 21, 3am to 5am UTC

Web225    May 21, 3am to 5am UTC

Web226    May 21, 4am to 6am UTC

Web227    May 21, 4am to 6am UTC

Postgresql will be unavailable for the duration of the upgrade.

We will update this post as the upgrades progress.

[May 21 03:55] The upgrade on Web216-218,Web220,Web224 and Web225 has been completed.

[May 21 04:46] The upgrade on Web226 and Web227 is over too.

-
-

[Done]Emergency maintenance on Web5, May 17th, 2013.

Posted in Downtime by

Web5 filesystem shows inconsistencies. We’re taking the server down to run a fsck. We’ll update this post as maintenance progresses.

2013-05-17 Fsck completed and the server is back to operational status.

-
-

[Done]Emergency maintenance on Web223, May 14th, 2013.

Posted in Downtime by

Web223 filesystem is in read-only state. We’re currently taking the server down and run a fsck. We’ll update this post as maintenance progresses.

2013-05-14 10:49 UTC Fsck completed and the server is back to operational status.

-
-

[Done]Emergency maintenance on Web70

Posted in Downtime by

Web70′s filesytem went read-only. We’re currently taking the server down to run a fsck. We will update this post as maintenance progresses.

2013-05-13 14:34 UTC: We are still running filesystem checks on the server.

2013-05-13 16:28 UTC: We are still running a filesystem check on the server.

2013-05-13 18:13 UTC: The filesystem check has stalled. We’ve already begun preparations to migrate the data that is available to a new server and the data for all $HOME directories is currently being transferred to the new server.

2013-05-13 19:17 UTC: Approximately 10% of the data has been transferred from the old server to the new server.

2013-05-13 21:08 UTC: We have cancelled the transfer of data from the old server. We’ve found problems with the data integrity. We’re now pulling data from our backups.

2013-05-13 22:47 UTC: We are still restoring data from backups.

2013-05-14 00:10 UTC: We have restored approximately 32% of the data from backups.

2013-05-14 02:13 UTC: We have restored approximately 35% of the data from backups.

2013-05-14 05:22 UTC: All MySQL and PostgreSQL databases and all user cron jobs have been transferred to the new server. We’re still migrating $HOME data.

2013-05-14 06:38 UTC: We are still restoring data from backups.

2013-05-14 10:20 UTC: All data has been restored and the server is back online.

2013-05-14 13.15 UTC: Contrary to what was previously announced some data is still missing and we are still in the progress of recovering it.

2013-05-14 15:03 UTC: We are still restoring data from backups.

2013-05-14 16:46 UTC: All MySql and PostgreSQL databases have now been recovered. Most of the home directories have been recovered too. We’re still working on the last few ones.

2013-05-15 03:05 UTC: At this time we have recovered as much data as possible from the old server.

-
-

[Done] Postgres Upgrade on Web208-211,Web213 and Web219

Posted in Downtime by

We are going to upgrade Postgresql 8.3 / PostGIS 1.3.6 to Postgresql 8.4 / PostGIS 1.5 on the following servers in the

upgrade interval mentioned alongside.

Web208    May 16, 2am to 4am UTC

Web209    May 16, 2am to 4am UTC

Web210    May 16, 3am to 5am UTC

Web211    May 16, 3am to 5am UTC

Web213    May 16, 4am to 6am UTC

Web219    May 16, 4am to 6am UTC

Postgresql will be unavailable for the duration of the upgrade.

We will update this post as the upgrades progress.

[May 16 02:03] We have started the upgrade process on Web208 and Web209.

[May 16 03:33] The upgrade has finished on Web208,Web209,Web210 and Web211.

[May 16 05:15] The upgrade on Web213 and Web219 is over.

-
-

[Fixed] Mailbox 10 down

Posted in Downtime by

One of our customer mailbox servers (mailbox 10) is currently offline. Affected customers are not able to access their email via POP, IMAP, or Webmail at this time. We’re working to restore service and will update this post when we have more information.

2013-05-12 1400 UTC The server was manually rebooted by the datacenter and has returned to normal operation status.

-
-

[Fixed] Emergency maintenance on Web223, May 9th, 2013.

Posted in Downtime by

Web223 filesystem went read-only. We’re currently taking the server down to run a fsck. We will update this post as maintenance progresses.

[May  9 10:44 UTC]  The server is back after a quick fsck and is OK now.

-
-

[Done]Scheduled maintenance on help.webfaction.com, webmail.webfaction.com, and docs.webfaction.com Friday, 10 May 2013 – 01:30 – 03:00 a.m. UTC

Posted in Downtime by

On Friday, 10 May 2013 from 01:30 – 03:00 a.m. UTC we’ll be performing maintenance on some of our support systems. During this maintenance window https://webmail.webfaction.com, https://help.webfaction.com, and http://docs.webfaction.com/ may be offline or inaccessible. We will update this post as maintenance progresses.

2013-05-10 02:01 UTC: We are beginning the maintenance now.

2013-05-10 02:36 UTC: We are cancelling this maintenance and will reschedule.

-
-