[fixed] Network routing issues

Posted in Problems by

Earlier today, a networking routing issue between Comcast and our data center prevented access to some of our servers for some users. The issue has been resolved.

-
-

[Done]Emergency maintenance on Web118

Posted in Downtime by

Our monitors have detected a hard drive failure on Web118. We will be replacing the failed drive shortly. We will update the post as soon as we have more information.

[Update] 19:10 UTC – We are now taking the server offline to replace the failed hard drive.
[Update] 19:51 UTC – The server is back online and responding to requests.

-
-

[Fixed]Web69 down

Posted in Downtime by

Currently Web69 is down and not reachable after failing to come back from a restart. We are working to bring the server back online.

2011-06-14 02:20 UTC: The server is now back online and responding to requests.

-
-

[Fixed]Web178 down

Posted in Downtime by

The filesystem on Web178 went into a read-only state several minutes ago. We rebooted the server but it has not come back up yet. We’re investigating this issue now and will update this post when we have more information.

2011-06-10 20:24 UTC – A drive has failed on Web178. The drive is currently being replaced.

2011-06-10 21:24 UTC – The drive has been replaced. A filesystem check is currently running.

2011-06-10 21:49 UTC – The filesystem check is still in progress.

2011-06-11 00:30 UTC – The filesystem check is still in progress. Web178 has a 2TB disk, so the check will take a considerable amount of time to complete.

2011-06-11 02:30 UTC – The filesystem check is still in progress.

2011-06-11 04:45 UTC – The filesystem check has finished and the server is back online and responding to requests.

-
-

[Done] Scheduled maintenance work on control panel on Monday

Posted in Scheduled downtime by

On Monday June 13th at 8am UTC we will take down our control panel for some maintenance work.
The expected downtime is less than three hours.
We will update this post when the work is complete.

Update: The maintenance work is now complete and only took 30 minutes.

-
-

[Fixed] Scheduled maintenance on Web193

Posted in Scheduled downtime by

Web193 will be taken down at 2:00 PM UTC on Friday, June 3rd 2011 to repair its RAID controller. Expected downtime should be less than one hour. We will update this post as the maintenance progresses.

2011-06-03 16:41 UTC: Web193 has been taken offline for repair.

2011-06-03 17:52 UTC: The RAID controller has been replaced and the server is currently running a filesystem check.

2011-06-03 18:33 UTC: Web193 is back online, but we’re still seeing some problems with the disk hardware. We’ll update this post when we have more information.

2011-06-03 20:44 UTC: Web193 has been taken offline for an emergency disk replacement. We’ll update this post when we have more information.

2011-06-03 23:11 UTC: Web193 is back online following replacement of the RAID controller and one disk. The MySQL service would not restart due to filesystem corruption that resulted from the faulty hardware, so we restored all MySQL databases from a backup made yesterday around 11:30 PM UTC. Please open a support ticket if you have any questions or notice any problems with your sites or data.

-
-