[monitoring] Web346 offline

Posted in Downtime by

2017-03-16 20:01 UTC: The networking services on Web346 are currently down and the machine is offline. We’re working to restore service at this time and will update this post when more information is available.

2017-03-16 22:06 UTC: Web346 is back online after our upstream network provider reconfigured a switch. We’ll continue to monitor for signs of trouble.

2017-03-16 22:26 UTC: Web346 has gone offline again. We’re working to restore service at this time and will update this post when more information is available.

2017-03-16 23:25 UTC: We’re still investigating the connectivity issue.

2017-03-17 00:24 UTC: We’re still investigating the connectivity issue.

2017-03-17 01:44 UTC: We’ve switched Web346 to a different OS kernel and it seem to be stable now. We’ll continue to monitor.

2017-03-17 15:43 UTC: Several hours ago, we determined that Web346 was being attacked. At that time, we took steps to mitigate the attack, and the server has been stable since then. We’ll continue to monitor.

-
-

[Fixed] Control panel and API

Posted in Problems by

2017-03-10 22:35 UTC: Our control panel and API was under heavy load for the past 10 minutes and may have been slow or unresponsive for some customers. The problem has been resolved and the panel and API are responding normally at this time.

-
-

[Fixed] Web544 connectivity issues

Posted in Downtime by

2017-01-04 19:00 UTC: The front end nginx server is down on Web544. We’re actively investigating this, please stand by for further updates.

2017-01-04 20:00 UTC: The front-end nginx server is up and operational at this time. Normal service has been restored.

-
-

[Fixed] Web473 connectivity issues

Posted in Problems by

2016-12-15 10:05 UTC: Connectivity to Web473 is intermittent from various locations. We’re investigating the issue at this time and will update this post when we have more information.

2016-12-15 10:45 UTC: Our data center technicians reported that this issue is due to a failed network router which caused interface connections and routing protocols on the device to stop forwarding traffic. The network engineers are currently working on getting the router back online.

2016-12-15 11:40 UTC: The network router is now fixed and the connectivity to Web473 has been restored.

-
-

[Fixed] Web470 connectivity issues

Posted in Problems by

2016-12-15 10:05 UTC: Connectivity to Web470 is intermittent from various locations. We’re investigating the issue at this time and will update this post when we have more information.

2016-12-15 10:45 UTC: Our data center technicians reported that this issue is due to a failed network router which caused interface connections and routing protocols on the device to stop forwarding traffic. The network engineers are currently working on getting the router back online.

2016-12-15 11:40 UTC: The network router is now fixed and the connectivity to Web470 has been restored.

-
-

[Fixed] Web344 inaccessible

Posted in Problems by

2016-12-14 8:30 UTC: The server is currently not responding. We are investigating the issue and will update this post as soon as we have further details.

2016-12-14 8:42 UTC: The issue has been resolved and the server is back online.

-
-

[Fixed] Web522 databases inaccessible

Posted in Downtime by

Web522 is having a problem on MySQL, we’re investigating it.

2016-12-12 12:52 UTC: We find out an error on users table and we restore it from backups

-
-

[monitoring] Intermittent timeouts on multiple servers

Posted in Problems by

For the past 48 hours, we’ve been seeing a series of intermittent “504 Gateway Timeout” errors on multiple servers.

We are actively troubleshooting this issue and will update this post when we have more information.

2016-11-02 14:50 UTC: We have deployed a fix on the affected machines, and we are monitoring the servers closely. Please contact us if you still see such issues.

-
-

[monitoring] Intermittent connectivity for Web319

Posted in Problems by

2016-10-31 21:22 UTC: Connectivity to Web319 is intermittent from various locations. We’re investigating the issue at this time and will update this post when we have more information.

2016-11-01 00:55 UTC: Connectivity to Web319 has been restored. We’ll continue to monitor.

2016-11-08 23:24 UTC: Connectivity to Web319 is intermittent is once again intermittent. We’re investigating the issue at this time and will update this post when we have more information.

2016-11-08 23:38 UTC: Connectivity to Web319 has been restored following a reboot. We’ll continue to monitor.

-
-

[Fixed] Web339 inaccessible

Posted in Downtime by

The server is currently not accessible. We are investigating the issue and will update as soon as we have more information.

2016-10-28 14:01 UTC: The server is back online again. One of the disks was replaced.

2016-10-31 17:51 UTC: The server has been taken down for emergency maintenance.

2016-10-31 22:48 UTC: We’re preparing to reload the OS on Web339 and will restore customer data from a backup made earlier today.

2016-11-01 02:41 UTC: The server has been reloaded and is in the process of being setup.

2016-11-01 06:23 UTC: The server has been setup and databases restored.

2016-11-01 13:46 UTC: User accounts have been restored up to letter “P”.

2016-11-01 20:35 UTC: All user accounts have been restored. If you have any issues, please open a support ticket.

-
-