Emergency maintenance on Web512

Posted in Downtime by

2020-11-27 14:30 UTC: Web512 is back from an emergency reboot. We plan on taking the server down for an fsck to check the file system. Please watch this area for updates and progress on the incident. We appreciate your patience.

2020-11-27 16:40 UTC: We are taking down Web512 for the fsck and will update when we have more information.

2020-11-27 18:00 UTC: The fsck is ongoing and there is no ETA at this time when it will be complete or the server back online.

2020-11-27 18:18 UTC: The fsck is complete on Web512. The server has been returned to operational status and is serving traffic. Our system administrators will continue monitoring the server closely.

-
-

[Fixed] Server Connectivity Issues in US

Posted in Downtime by

We are currently experiencing connectivity issues in USA affecting some servers.

Our network team is working on resolving the issue as quickly as possible.

Please watch this area for updates and progress on the incident.

2020-08-09 13:45 Our data center technicians are looking into the issue on the following servers.

Web512 IP Address: 207.38.86.249
Web513 IP Address: 207.38.86.251
Web514 IP Address: 207.38.86.252
Web515 IP Address: 207.38.86.253
Web555 IP Address: 207.38.86.26
Web556 IP Address: 207.38.86.27
Web560 IP Address: 207.38.86.91
Web563 IP Address: 207.38.86.81

 

2020-08-09 16:15 UTC: Our onsite technicians confirm all servers are online. The networking team is working on root cause as to why traffic is not routing to the servers.

2020-08-09 17:05 UTC: We have isolated the root cause to a bad router. Our team is in the process of replacing the router.

2020-08-09 18:19 UTC: Switch as been replaced and all servers are back online.

-
-

[Fixed] Web540 inaccessible

Posted in Downtime by

Web540 is currently inaccessible. We are looking into the issue and will resolve it as quickly as possible.

2018-09-15 16:05 UTC: Web540 is back online.

-
-

[Fixed] Web560 inaccessible

Posted in Downtime by

2017-04-25 1700 UTC: We are investigating the issue and will update the post as soon as we have further details.

2017-04-25 1730 UTC: The server is back online and serving traffic.

-
-

[Fixed] Web491 inaccessible

Posted in Downtime by

We are investigating the issue and will update the post as soon as we have further details.

2017-02-02 18:52 UTC The issue has been resolved and the server is back online.

2017-02-03 14:30 UTC: The server is inaccessible again. We are working to resolve the issue as soon as possible.

2017-02-03 15:20 UTC: The server is back online.

2017-02-15 10:30 UTC: The server is inaccessible again. We are working to bring back the machine online as soon as possible.

2017-02-15 10:45 UTC: The server is back online.

-
-

[Fixed] Web526 inaccessible

Posted in Downtime by

2016-12-15 18:26 UTC: The server is currently not responding. We are investigating the issue and will update this post as soon as we have further details.

2016-12-15 18:34 UTC: The front end nginx server is up and operational at this time. Normal service has been restored. Our System Administrators will be monitoring to make sure that it starts normally.

-
-

[Fixed] Web162 inaccessible

Posted in Downtime by

2016-10-29 15:30 UTC: The server is currently not responding. We are investigating the issue and will update this post as soon as we have more information.

2016-10-29 16:54 UTC: The server is back online at this time and we are monitoring it.

2016-10-29 20:00 UTC: The server is not responding again and we are trouble shooting again.

2016-10-29 20:10 UTC: The server back in online and we continue to troubleshoot the root cause.

2016-10-31 19:56 UTC: We have taken the machine down for emergency maintenance.

2016-10-31 20:39 UTC: We are currently running a filesystem check.

2016-10-31 23:49 UTC: We’re unable to recover the filesystem. At this time, we’re provisioning a new server and will migrate all customer data to the new machine ASAP.

2016-11-01 8:21 UTC: The server has been provisioned and we are in the process of setting up.

2016-11-01 10:22 UTC: The server has been setup and all databases restored.

2016-11-01 13:22 UTC: User accounts up to letter “C” have been restored.

2016-11-02 14:56 UTC: We are down to the last few account to restore.

-
-

[Monitoring] Web527 offline

Posted in Downtime by

2016-10-23 15:15 UTC: Web527 is presently offline – we’re investigating the problem and will update this post as soon as we have more information.

2016-10-23 16:22 UTC: Web527 is back online. We’ll continue to monitor.

-
-

[Resolved] Web446 DDOS

Posted in Downtime by

2016-10-06 1903 UTC The server has been placed under DDoS mitigation. It is currently accessible, but some customers may experience degraded network connectivity. We continue to monitor the machine closely.

2016-10-06 22:29 UTC: The attack has subsided and the server is responding normally at this time. The mitigation system is still in place, and we’ll continue to monitor for signs of trouble.

2016-10-07 18:45 UTC: The server has been removed from the mitigation system.

-
-

[Fixed] Web517 Inaccessible

Posted in Downtime by

The server is currently inaccessible. We are investigating the issue and will update when we have more information.

2016-08-09 21:48 UTC: It appears that the RAM on the server is bad. We’re having it replaced at this time and will update as soon as possible.

2016-08-10 00:35 UTC: The server is back online.

-
-