[Fixed] Web344 connectivity issues

Posted in Downtime by

We are currently experiencing connectivity issues with the server. We are investigating the issue and will update this post when we have more information.

2013-05-05 1800 UTC: Full Connectivity to Web344 has been restored. No further connection issues are anticipated.

2013-05-06 01:08 UTC: The connectivity issues started again. We are investigating the issue and will update this post when we have more information.

2013-05-06 01:34 UTC: Full Connectivity to Web344 has been restored. Our sysadmin team will keep monitoring web344.

2013-05-06 03:28 UTC: Our sysadmins could not antecipate any connection issues.

-
-

[Done] Emergency maintenance on Web174.

Posted in Downtime by

The file system has gone read only on Web174. We are now taking it offline to perform a file system check.

2013-05-03 19:19 UTC: The file system check has finished and the server is now back online.

2013-05-03 19:29 UTC: The file system has gone read only again. We are preparing to migrate the data from this server to a new server so that after this move there is no additional downtime from file system problems on the current hardware.

2013-05-03 23:04 UTC: The new server has been set up and our platform installed. We’re now transferring data from the $HOME directories on the old server to the new server.

2013-05-04 01:58 UTC:  Approximately half of the data has been transferred to the new machine.

2013-05-04 04:26 UTC:  Approximately 63% of the data has been transferred to the new machine.

2013-05-04 05:53 UTC: Approximately 80% of the data has been transferred to the new machine.

2013-05-04 07:23 UTC: The rsync for /home is complete we are now moving MySQL/Postgres data to the new server.

2013-05-04 07:53 UTC: The databases are back and the server is fully migrated now.

-
-

[Done]Scheduled maintenance on Dweb49 April 6th, 2013.

Posted in Scheduled downtime by

Dweb49 will be taken down Monday April 6th 2013 between 07:00 UTC and 09:00 UTC for a disk drive swap. We will update this post as maintenance will progresses.

2013-05-06 08:23 UTC After swapping one disk in the array the server was not able to boot and we had to swap the chassis too. The server is now back at operational status.

-
-

[Done] Emergency filesystem check on Web48

Posted in Downtime by

Web48’s filesystem has gone read-only due corruption so we are taking it down into rescue mode to run a filesystem check.

[May  3 05:06] The fsck is at pass 1B now.

[May  3 05:45] The check is over and the server is back and OK now.

-
-

[Fixed] Web346 connectivity issues

Posted in Problems by

We are currently experiencing connectivity issues with the server.

The issue is being investigated.

We are taking the server down to run fsck on the drives.

2013-05-01 18:16 UTC: fsck is complete and the server is back online.

-
-

[Done] PostgreSQL upgrade on Web22,Web23,Web223 and Web228

Posted in Scheduled downtime by

We are going to upgrade Postgresql 8.3 / PostGIS 1.3.6 to Postgresql 8.4 / PostGIS 1.5 on the following servers in the

upgrade interval mentioned alongside.

Web22    May 5, 2am to 4am UTC

Web228  May 5, 3am to 5am UTC

Web223  May 5, 4am to 6am UTC

Web23    May 5, 5am to 7am UTC

Postgresql will be unavailable for the duration of the upgrade.

We will update this post as the upgrades progress.

[May  5 03:09 UTC] The upgrade on Web22 is over.

[May  5 04:46 UTC] The upgrade on web223 is over while Web228 is still going on due to the large amount of data.

[May  5 05:20 UTC] The upgrade on Web228 is over.

[May  5 06:50 UTC] The upgrade on web23 is over.

-
-