[Done]Scheduled maintenance for Web76, Wednesday September 28th

Posted in Downtime by

Web76 will be taken down on Wednesday September 28th at 11:00AM UTC for a disk replacement. The downtime should be less than a hour. We will update this post as maintenance progresses.

[11.36AM UTC] Update: the disk has been replaced, Web76 is back to normal operation.

-
-

[Done]Scheduled maintenance for Web183, Saturday September 24

Posted in Downtime by

We will be performing a scheduled hardware change on Web183 Saturday, September 24 for approximately 1 hour during the 4 hour window of 0:00 CDT – 04:00 CDT. We will update this post as maintenance progresses.

Update: [0:30 CDT] – The server is now being taken offline for the hardware change.
Update: [01:10 CDT] – The server is back online and responding to requests.

-
-

[Done] Scheduled maintenance work on Web180 on Friday September 23rd, 2011

Posted in Downtime by

[Done] Web180 will be taken down on Friday September 23rd, 2011 at 07:00AM UTC for a disk replacement. Maintenance should be less than a hour.

[07.53 UTC] The disk has been replaced and the server is back to normal operational status.

-
-

[Done] Scheduled maintenance work PostgreSQL on web9 on Sunday October 9th, 2011

Posted in Downtime by

web9’s PostgreSQL instance will be offline for maintenance Sunday October 9th, 2011. Downtime for PostgreSQL is expected to be less than two hours.

All other services will remain unaffected.

[07:31 UTC]: The upgrade is complete and PostgreSQL is back online.

-
-

[Done] Scheduled maintenance work on web64 on Wednesday September 21st, 2011

Posted in Downtime by

web64 will be taken down on Wednesday, September 21st for a routine disk swap. Total downtime should be less than one hour.

[13:45 GMT]: After a delay in starting the maintenance, the disk has been swapped and the server is back online.

-
-

[Fixed] Network issues with Web216, Web217, and Web218

Posted in Downtime by

There is a network issue at our Amsterdam data center, resulting in Web216, Web217, and Web218 being unreachable for the moment. We are working with our upstream provider to resolve the problem as soon as possible.

Update: [13:12 UTC] – Web217 and Web218 are back up. We are still working on bringing up Web216 as well.

Update: [14:16 UTC] – Web216 is back up as well.

-
-

[Fixed] MySQL problems on Web193 – Emergency Maintenance

Posted in Problems by

We’re investigating a problem with MySQL on Web193 and will update this post when we have more information.

Update: [20:13 CDT] – Our investigation into the MySQL problems has led to some emergency maintenance on the server as we look for the root cause. We’ll update this post as we get more information.

Update: [20:58 CDT] – The server is now back online and responding to requests. We are continuing to monitor the situation closely to verify that the server stays stable.

Update: [21:06 CDT] – We’ve taken the server mostly offline again to verify the integrity of the MySQL databases.

Update: [22:52 CDT] – We’ve found and repaired the error. We discovered some of the MySQL databases had been corrupted. We’ve run a check all on the databases and restored most of the databases that had become corrupted. Due the nature of the database errors some databases couldn’t be backed up and weren’t restored. We’ve restored the databases that were corrupted and could be restored from our latest backups.

Update: [05:36 CDT] – The problems with MySQL on Web193 are back. We’ve temporarily disabled InnoDB in MySQL while we are working on fixing the problem.

Update: [11:51 CDT] – The InnoDB engine is back up but all┬áInnoDB tables are currently read-only. We are sill working on restoring full functionality.

Update: [00:30 CDT] – We are about to get the whole chassis for the server swapped by our data center.

Update: [07:06 CDT] – All of the new hardware is in place.┬áInnoDB tables are going offline for a few minutes, hopefully for the last time, so that any remaining corrupted data can be fixed.

Update: [07:11 CDT] – All MySQL tables are now back online, free of any corruption and fully writeable.

-
-

[Done] Scheduled maintenance work on multiple servers on Sunday September 18th, 2011

Posted in Scheduled downtime by

One of our datacenters will be performing maintenance work on its network infrastructure this Sunday between 05:00 UTC and 13:00 UTC.

The following servers will likely experience periods of high latency or packet loss and may experience complete loss of connectivity for brief moments during that period: Dweb74, Dweb75, Dweb76, Dweb77, Dweb78, Dweb79, Dweb81, Dweb84, Mail9, Mailbox6, Mailbox7, Web196, Web197, Web198, Web199, Web200, Web201, Web202, Web203, Web204, Web205, Web206, Web207, Web212, Web213, Web214, Web215, Web219, Web220, Web221, Web222, and the control panel.

-
-

Scheduled downtime for Web27 on Thursday, September 29

Posted in Downtime by

Web27 will be taken down on Thursday, September 29 for approximately 1 hour while the data center replaces the power strip that powers our Ethernet switch.

-
-

[fixed] Emergency maintenance on Web175

Posted in Downtime by

The filesystem on Web175 has gone into a read-only state. We are running an filesystem check and will provide progress updates.

2011-09-14 18:28 UTC – The filesystem check is 50% complete.

2011-09-14 21:17 UTC – The filesystem check is complete and all services are back online.

-
-