On 31 October 2010, we are changing the location of the socket used by MySQL. The current location is /tmp/mysql.sock – the new location will be /var/lib/mysql/mysql.sock.
This change should be transparent to the majority of customers, but if you’re using a specific socket parameter in your application’s database configuration, you will need to update it to point to the new location.
These two servers will be taken down on Friday, October 15th to fix a problem with the RAID controller. The downtime should be less than 1hr. Web150 will be taken down at 0700UTC while Web111 will be taken down at 0730UTC.
[08:10 UTC] Update: Web111 has been rebooted, RAID disk is rebuilding. The server is accepting network requests.
[08:10 UTC] Update: Web150 has been rebooted, RAID disk is rebuilding. The server is accepting network requests.
[08:16 UTC] Update: the work is now complete and the servers are back to normal.
Web9 will be taken down at 11:00 AM UTC on Tuesday, October 5 to repair its RAID controller. Expected downtime should be less than one hour. We will update this post as the maintenance progresses.
[12:50 UTC] Update: The repair will take a little longer than expected, we’ll update this post as soon as it’s complete.
[14.11 UTC] Update: Repairing the RAID didn’t work so at this point we’re going to put a new RAID array in the server and copy the existing data onto the new RAID array.
[18.50 UTC] Update: We are still recovering data from old RAID.
[01.15 UTC] Update: We are still recovering data from old RAID.
[04:25 UTC] Update: We are still recovering data from old RAID.
[08:25 UTC] Update: All the data and the sites that were on the primary drive have been recovered and are up and running. We are now recovering the data that was on the secondary drive
[12:24 UTC] Update: A network issue with the datacenter where our backups servers are located is currently slowing down the recovering of the second drive. We’ll resume the recovery as soon as the network issue is resolved.
[15.47 UTC] Update: The network issue with the datacenter where our backups are located is resolved and we are currently restoring the data from the second drive.
[16:57 UTC] Update: Unfortunately the datacenter with our backup servers is experiencing intermittent network issues again and this is preventing us from recovering the backup data at a decent speed. We’re doing everything we can to fix the network issues with the datacenter.
[19:34 UTC] Update: The network issue with the datacenter hosting our backup servers is now fully resolve and we are now restoring data from the secondary drive at full speed.
[23:46 UTC] Update: All the data has now been recovered and the server is back to normal. Open a ticket if you notice any problem with your site. Apologies for the extended downtime due to a combination of RAID failure and network issue at the datacenter.
These four servers will be taken down on Sunday October 3rd to fix a problem with the RAID controller. The downtime should be less than 1h. Web50 will be taken down at 5pm UTC, Web56 at 5.30pm UTC, Web65 at 5pm UTC and Web83 at 6pm UTC.
[17:20 UTC] Update: We are bringing Web50 down.
[17:55 UTC] Update: Web50 is up and running again.