[Done] Scheduled maintenance work on Web26, Web94, and Web150 on Thursday October 21st

Posted in Scheduled downtime by

Web26, Web94, and Web150 will be taken down at 7am UTC on Thursday October 21st for maintenance. The downtime should be less than 1 hour.

Updates: We have rescheduled the maintenance on web150 for Monday 25th at 7am UTC.

[14:27 UTC] Web26 and web94 are both up and running.

[10/25 07:05 UTC] Web150 is up and running. All work is now complete.

-
-

Change to MySQL configuration on all servers

Posted in General by

On 31 October 2010, we are changing the location of the socket used by MySQL. The current location is /tmp/mysql.sock – the new location will be /var/lib/mysql/mysql.sock.

This change should be transparent to the majority of customers, but if you’re using a specific socket parameter in your application’s database configuration, you will need to update it to point to the new location.

-
-

[Done] Scheduled maintenance work on Web150 on Monday October 18th

Posted in Downtime by

This server will be taken down on Monday October 18th for maintenance. The downtime should be about 1h. Web150 will be taken down at 8pm UTC.

[20:05 UTC] Update: the work is now complete and the server is back to normal.

-
-

[Done] Scheduled maintenance work on Web111 and Web150 on Friday, October 15th

Posted in Downtime by

These two servers will be taken down on Friday, October 15th to fix a problem with the RAID controller. The downtime should be less than 1hr. Web150 will be taken down at 0700UTC while Web111 will be taken down at 0730UTC.

[08:10 UTC] Update: Web111 has been rebooted, RAID disk is rebuilding. The server is accepting network requests.

[08:10 UTC] Update: Web150 has been rebooted, RAID disk is rebuilding. The server is accepting network requests.

[08:16 UTC] Update: the work is now complete and the servers are back to normal.

-
-

[Done] Scheduled maintenance work on Web29, Web43, and Web45 on Sunday October 10th

Posted in Downtime by

These three servers will be taken down on Sunday October 10th  to fix a problem with the RAID controller. The downtime should be less than 1h. All three servers will be taken down at 6pm UTC.

Update: The maintenance work is now completed on all three servers.

-
-

[Fixed] Secondary drive down on web11

Posted in Problems by

The secondary drive (holding /home2) is no longer being recognized on web11. We’re working to restore it asap.

[16:45 UTC] Update: The server is now back to normal

-
-

[Fixed] Scheduled maintenance work on Web9 on Tuesday, October 5

Posted in Scheduled downtime by

Web9 will be taken down at 11:00 AM UTC on Tuesday, October 5 to repair its RAID controller. Expected downtime should be less than one hour. We will update this post as the maintenance progresses.

[12:50 UTC] Update: The repair will take a little longer than expected, we’ll update this post as soon as it’s complete.

[14.11 UTC] Update: Repairing the RAID didn’t work so at this point we’re going to put a new RAID array in the server and copy the existing data onto the new RAID array.

[18.50 UTC] Update: We are still recovering data from old RAID.

[01.15 UTC] Update: We are still recovering data from old RAID.

[04:25 UTC] Update: We are still recovering data from old RAID.

[08:25 UTC] Update: All the data and the sites that were on the primary drive have been recovered and are up and running. We are now recovering the data that was on the secondary drive

[12:24 UTC] Update: A network issue with the datacenter where our backups servers are located is currently slowing down the recovering of the second drive. We’ll resume the recovery as soon as the network issue is resolved.

[15.47 UTC] Update: The network issue with the datacenter where our backups are located is resolved and we are currently restoring the data from the second drive.

[16:57 UTC] Update: Unfortunately the datacenter with our backup servers is experiencing intermittent network issues again and this is preventing us from recovering the backup data at a decent speed. We’re doing everything we can to fix the network issues with the datacenter.

[19:34 UTC] Update: The network issue with the datacenter hosting our backup servers is now fully resolve and we are now restoring data from the secondary drive at full speed.

[23:46 UTC] Update: All the data has now been recovered and the server is back to normal. Open a ticket if you notice any problem with your site. Apologies for the extended downtime due to a combination of RAID failure and network issue at the datacenter.

-
-

[Done] Scheduled maintenance work on Web50, Web56, Web65 and Web83 on Sunday October 3rd

Posted in Scheduled downtime by
These four servers will be taken down on Sunday October 3rd  to fix a problem with the RAID controller. The downtime should be less than 1h. Web50 will be taken down at 5pm UTC, Web56 at 5.30pm  UTC, Web65 at 5pm UTC and Web83 at 6pm UTC.
[17:20 UTC] Update: We are bringing Web50 down.
[17:55 UTC] Update: Web50 is up and running again.
Update: All four servers have now been fixed
-
-