[Done] Emergency maintenance on web70

Posted in Downtime by

The file system on web70 has gone read-only, and we are working to restore service as soon as possible.  We will post updates as we have them.

2014-03-11 02:41 UTC: We have booted the server into a rescue environment and are running an fsck.  fsck progress currently at 25%

2014-03-11 02:53 UTC: The fsck is at 70%

2014-03-11 03:10 UTC: The fsck is complete and the system is back online.  We are checking for further problems before marking this as fixed.

2014-03-11 03:19 UTC: We have found no further problems on web70.

-
-

[Fixed] Web368 intermittent downtime

Posted in Downtime by

Web368 has become largely unresponsive and we are working urgently to bring it back fully online.

2014-03-03 22:56 UTC: The server is back and has been stable for some time now so we are marking this as resolved.  If you encounter any further problems please let us know via a support ticket.

-
-

[Fixed] Web208 and Web210 down

Posted in Downtime by

Web208 and Web210 are currently not reachable.  There appears to be a network issue with an upstream provider.  We are investigating urgently.

Update 2014-01-24 08:07 UTC: The network issue has been resolved and the servers are now accessible again.

Update 2014-01-24 09:01 UTC: The servers are unreachable again. We are working with our upstream provider to restore connectivity.

Update 2014-01-24 09:12 UTC: Connectivity to both servers has been restored. Our upstream provider discovered a problem with the network switch.
An emergency maintenance has been scheduled to take place at 18:00 UTC. The network engineers at the data center will be replacing the faulty switch.

2014-01-25 1838 UTC The data center has confirmed the network switch was replaced and the machines are back online.

-
-

[Fixed] Network switch failure affecting several US servers

Posted in Downtime by

The following dedicated servers are currently inaccessible due to a failed network switch:

Web30
Web35
Web37
Web55
Web57
Web69
Web80
Web108
Web126
Web186
Web324
Dweb117
Dweb118
Dweb125

The data center is working to replace the switch at this time. We will update this post when we have more information.

2014-01-16 17:11UTC: The network switch was replaced and the machines are back online.

-
-

[Fixed] Web75 unresponsive

Posted in Downtime by

Web75 is currently unresponsive.  We are looking into this as a matter of urgency.

2014-01-05 06:41 UTC: The server is experiencing disk problems, and we are migrating it to new hardware. The main IP address will change from 199.58.84.73 to 108.168.213.90.

Update 3pm UTC: We’re still working on migrating all data to the new hardware.

Update 5pm UTC: All MySQL and PostgreSQL databases have now been migrated to the new server. All accounts up to the letter “i” have been migrated to the new hardware and their websites have been re-enabled. We’re working on the rest of the accounts.

Update 9.45pm UTC: The home directory migration is at the letter “s”. We’re working on the rest of the accounts.

Update 3am UTC: All data has now been migrated to the new server. We will leave a proxy running on the old server for 10 days to redirect all HTTP and HTTPS traffic to the new server. If you experience any problems with your sites open a ticket and we’ll look into it asap.

-
-

[Fixed] MySQL problems on Web211

Posted in Downtime by

We are experiencing problems on MySQL on web211 at the moment.  Some users may encounter problems updating or deleting data or databases.  We are working on fixing this right now.

2013-12-14 07:11 UTC: We are importing all of the current MySQL data into a new MySQL instance which should resolve the issues.  The re-import is about halfway complete at this stage.

2013-12-14 19:00 UTC: All MySQL data has been restored.

-
-

[Fixed] MySQL down on Web336

Posted in Downtime by

We are experiencing problems with MySQL on web336 at the moment, so MySQL is not accessible at the moment.  We are working urgently to bring it back online.

2013-12-07 04:10 UTC: We found some file system errors which may be related to the root issue, so we are now in the process of taking an immediate backup of MySQL as a precaution.

2013-12-07 05:36 UTC: Data dump is still in progress.

2013-12-07 08:00 UTC: We’ve managed to fix the problems and MySQL and the server are back online and fully operational.

-
-

[Fixed] Network maintenance in Virginia affecting several servers

Posted in Downtime by

Network maintenance in one of our US data centers is affecting connectivity. We’re working to restore service and hope to have this issue resolved soon.

2013-11-15 13:45 UTC: The network maintenance at the data center is complete. All connectivity has been restored.

-
-

[Fixed] Network maintenance in Virginia affecting several servers

Posted in Downtime by

Network maintenance in one of our data centers is affecting connectivity. We’re working to restore service and hope to have this issue resolved soon.

1345 UTC All service has been restored at this time. We will continue monitoring the situation.

1640 UTC No further issues have occured and the data center reports the network maintenance is complete.

-
-

[Done]Emergency maintenance on Web337, Saturday 14 September 2013.

Posted in Downtime by

We have found numerous file system errors in the system’s log that seem to be causing performance issues, we’re now taking this machine offline to perform a file system check. We’ll update this post when we have more info.

2013-09-14 19:52 UTC: The file system check has finished its first pass and we are now verifying that the remainder of the file systems are clean.

2013-09-14 20:02 UTC:  The server is now back online and functioning normally.

-
-