[Done] Memory swap on web509, November 4th 2016

Posted in Maintenance by

Web509 will be brought down to replace some faulty memory modules on November 4th between 2PM and 5PM UTC. Expected downtime is less than 30 minutes.

We will update this post as maintenance will progress.

 

14:01 The maintenance has begun

14:11 The maintenance is complete and the server is back online

-
-

[monitoring] Intermittent timeouts on multiple servers

Posted in Problems by

For the past 48 hours, we’ve been seeing a series of intermittent “504 Gateway Timeout” errors on multiple servers.

We are actively troubleshooting this issue and will update this post when we have more information.

2016-11-02 14:50 UTC: We have deployed a fix on the affected machines, and we are monitoring the servers closely. Please contact us if you still see such issues.

-
-

[Done] Reboots on Centos 5 machines

Posted in Maintenance by

Over the next couple days we are going to reboot all of our CentOS 5 servers to pick up important updates.

-
-

[monitoring] Intermittent connectivity for Web319

Posted in Problems by

2016-10-31 21:22 UTC: Connectivity to Web319 is intermittent from various locations. We’re investigating the issue at this time and will update this post when we have more information.

2016-11-01 00:55 UTC: Connectivity to Web319 has been restored. We’ll continue to monitor.

2016-11-08 23:24 UTC: Connectivity to Web319 is intermittent is once again intermittent. We’re investigating the issue at this time and will update this post when we have more information.

2016-11-08 23:38 UTC: Connectivity to Web319 has been restored following a reboot. We’ll continue to monitor.

-
-

[in progress] Web162 inaccessible

Posted in Downtime by

2016-10-29 15:30 UTC: The server is currently not responding. We are investigating the issue and will update this post as soon as we have more information.

2016-10-29 16:54 UTC: The server is back online at this time and we are monitoring it.

2016-10-29 20:00 UTC: The server is not responding again and we are trouble shooting again.

2016-10-29 20:10 UTC: The server back in online and we continue to troubleshoot the root cause.

2016-10-31 19:56 UTC: We have taken the machine down for emergency maintenance.

2016-10-31 20:39 UTC: We are currently running a filesystem check.

2016-10-31 23:49 UTC: We’re unable to recover the filesystem. At this time, we’re provisioning a new server and will migrate all customer data to the new machine ASAP.

2016-11-01 8:21 UTC: The server has been provisioned and we are in the process of setting up.

2016-11-01 10:22 UTC: The server has been setup and all databases restored.

2016-11-01 13:22 UTC: User accounts up to letter “C” have been restored.

2016-11-02 14:56 UTC: We are down to the last few account to restore.

-
-

[Fixed] Web339 inaccessible

Posted in Downtime by

The server is currently not accessible. We are investigating the issue and will update as soon as we have more information.

2016-10-28 14:01 UTC: The server is back online again. One of the disks was replaced.

2016-10-31 17:51 UTC: The server has been taken down for emergency maintenance.

2016-10-31 22:48 UTC: We’re preparing to reload the OS on Web339 and will restore customer data from a backup made earlier today.

2016-11-01 02:41 UTC: The server has been reloaded and is in the process of being setup.

2016-11-01 06:23 UTC: The server has been setup and databases restored.

2016-11-01 13:46 UTC: User accounts have been restored up to letter “P”.

2016-11-01 20:35 UTC: All user accounts have been restored. If you have any issues, please open a support ticket.

-
-

[Done] Reboots to pick up updates

Posted in Maintenance by
Over  the next few hours we are going to reboot all of our CentOS servers to pick up important updates.
-
-

[Monitoring] Web341 inaccessible

Posted in Downtime by

2016-10-27 10:15 UTC: The server is currently not responding. We are investigating the issue and will update this post as soon as we have more information.

2016-10-27 10:40 UTC: The issue has been resolved and the server is fully operational again.

2016-10-27 11:05 UTC: The server has started having some connectivity issues again. We are looking into this and will post another update as soon as we have more information.

2016-10-27 11:35 UTC: The problem is due to an ongoing DDoS attack against the server. It has now been placed under our DDoS mitigation system and it is currently accessible, but some customers may experience degraded network connectivity. We’re continuing to monitor the situation closely.

-
-

[monitoring] Web527 offline

Posted in Downtime by

2016-10-23 15:15 UTC: Web527 is presently offline – we’re investigating the problem and will update this post as soon as we have more information.

2016-10-23 16:22 UTC: Web527 is back online. We’ll continue to monitor.

-
-

[Fixed] Web533 offline

Posted in Downtime by

2016-10-19 21:49 UTC: Web533 is presently offline – we’re investigating the problem and will update this post as soon as we have more information.

2016-10-19 22:27 UTC: Web533 is back online. We’ll continue to monitor.

-
-