[Monitoring] Web442 DDoS

Posted in Problems by

Web442 is currently inaccessible. We are investigating the issue; please stand by for updates.

2016-04-17 03:45 UTC: Our upstream provider have enabled DDOS protection for the server. Network traffic will be degraded during that time. We will remove the DDOS protection as soon as the attack subsides.

2016-04-18 03:25 UTC: The heavy traffic has subsided and mitigation has been removed. At this time the server is stable. We will monitor the server a little further.

2016-04-18 09:30 UTC: The attack has resumed and the server is currently null-routed. We are working to restore the services ASAP.

2016-04-18 13:35 UTC: We’ve managed to mitigate the attack. We are monitoring the server closely.

2016-04-18 14:05 UTC: Our mitigation system was not able to contain the severity of the attack. The server has been null-routed again.

2016-04-18 22:19 UTC: The null route has been lifted and service has been restored. We’ll continue to monitor.

2016-04-20 10:30 UTC: The attack has resumed and our upstream provider has enabled DDOS protection for the server. Degraded network traffic and intermittent connectivity issues are to be expected while DDOS mitigation is active. We are working to restore normal connectivity as soon as possible.

2016-04-22 10:00 UTC: The attack has subsided and normal connectivity has been restored. We’ll continue to monitor.

2016-04-25 04:21 UTC: The attack has resumed and we’re working to have the null route removed and DDoS mitigation services enabled.

2016-04-25 05:27 UTC: The null route was removed under an hour ago and the server should be accessible. Keep in mind there may be intermittent access for some due to the DDoS mitigation service in place.

2016-04-25 10:00 UTC: The connectivity to the server has been stable for several hours. The DDoS mitigation service is still in place and we continue to monitor the server.

2016-04-26 04:25 UTC: The attack has subsided and normal connectivity has been restored. We’ll continue to monitor.

-
-

[Fixed] Web491 inaccessible

Posted in Downtime by

Web491 is currently inaccessible. We are investigating the issue; please stand by for updates.

2016-04-16 11:00 UTC: Web491 is back online.

-
-

[Fixed] Web162 Offline

Posted in Downtime by

Web162 went offline about 30 minutes ago. We’re working to restore service at this time and will update this post when more information is available.

2016-04-16 @ 04:00 UTC: Our datacenter is investigating the configuration of the network router.

2016-04-16 @ 09:45 UTC: We are performing a filesystem check.

2016-04-16 @ 12:00 UTC: The server is back online.  If you experience any problems, please open a support ticket.

-
-

[Fixed] Web512 MySQL Maintenance

Posted in Uncategorized by

The shared MySQL databases on Web512 are showing some signs of corruption.  We are rebooting the server into a Rescue environment, checking the filesystem, and will perform a MySQL recovery.

We will update this post as more information becomes available.

2016-04-13 @ 05:50 UTC : The filesystem check is complete.  We are starting the MySQL recovery process.

2016-04-13 @ 09:10 UTC : The recovery is complete.  If you experience any problems with your website, please open a support ticket.

-
-

[Fixed] Filesystem problems on Web461

Posted in Problems by

A few minutes ago, Web461 stopped accepting SSH connection. We rebooted the machine, and were able to log in after doing so, but load is rising quickly and there are signs of problems with the disk array. We’re currently investigating and will update this post when we have more information.

2016-04-12 03:30 UTC:  The server is working, but MySQL has become corrupted.  We are in the process of backing-up and restoring the databases.

2016-04-12 08:30 UTC:  MySQL restores are still in progress.

2016-04-12 10:30 UTC:  The server has lost network connectivity.  Our technicians are diagnosing the problem.  Once network connectivity is restored we will be able to finish the MySQL restore process.

2016-04-12 13:15 UTC:  The server is back online, and we are proceeding once again with MySQL restoration.

2016-04-12 15:30 UTC:  Most MySQL databases have been restored.

2016-04-12 16:30 UTC:  All MySQL databases have been restored.  If you notice any problems with your website, please open a support ticket.

-
-

[Fixed] Web493 inaccessible

Posted in Downtime by

Web493 is currently inaccessible, we’re working with the datacenter to investigate and resolve the outage. Please stand by.

2016-04-11 04:02 UTC:  We’re in the process of rebooting the server to further diagnose the issue.

2016-04-11 05:21 UTC: We find out a faulty network link on server, we have replaced the faulty equipment and the server is up and running at the moment

 

-
-

[Monitoring] Web508 DDoS

Posted in Problems by

We are currently experiencing a DDoS attack on web508. Our upstream provider has enabled mitigation for this attack.

There are still some intermittent access issues until the attack subsides further.

We will update this post as soon as we have further information.

2016-04-08 04:40 UTC:  The attack has subsided and Web508 is back online.

2016-04-08 06:24 UTC:  The attack has resumed, we’re continuing to work towards restoring services. Sorry for the troubles, thank you for your patience.

2016-04-08 08:10 UTC: The attack has subsided and Web508 is back online. We are monitoring the server to ensure the proper operation of all services.

2016-04-08 09:40 UTC: The attack has resumed, we are working to restore the services ASAP.

2016-04-08 11:00 UTC: The nature of the attack has changed; we are still working to mitigate it.

2016-04-08 11:40 UTC: At this point the attack was mitigated. We are monitoring the server closely.

2016-04-08 18:45 UTC: Web508 has been stable for the past several hours. We’ll continue to monitor.

2016-04-09 03:57 UTC: Web508 is inaccessible at the moment, we’re working to restore it once more. Please stand by.

2016-04-09 06:55 UTC: The DDoS has subsided, Web508 is now accessible and for the moment stable. We’re continuing to monitor it while looking at other avenues to help further mitigate any future attacks.

2016-04-09 09:30 UTC: The attack is back. We are working to mitigate it.

2016-04-09 10:10 UTC: We managed to mitigate the attack. The server is now operational, but the attack is still ongoing, so we’ll continue to monitor the server.

2016-04-09 14:30 UTC: Web508 has been stable for the past several hours. We’ll continue to monitor.

2016-04-11 02:22 UTC: The attack on Web508 seems to have stopped but we will continue to monitor the server for a little longer to be sure. Thank you for your patience.

2016-04-12 16:03 UTC: The attack on Web508 has resumed and the server is currently null-routed. We’re working to mitigate the attack and restore service at this time.

2016-04-12 16:35 UTC: Connectivity to Web508 has been restored. We’ll continue to monitor.

2016-04-13 23:29 UTC: Web508 has been stable for over 24 hours.

2016-04-15 18:06 UTC: The attack on Web508 has resumed and the server is currently null-routed. We’re working to mitigate the attack and restore service at this time.

2016-04-15 18:52 UTC: Connectivity to Web508 has been restored. We’ll continue to monitor.

2016-04-18 14:10 UTC: The attack on Web508 has resumed and the server has been placed in DDoS mitigation.  The server is now operational, but the attack is still ongoing, so we’ll continue to monitor the server. Some traffic may be blocked or a delayed as part of the mitigation.

-
-

[Fixed] Web404 DDoS

Posted in Uncategorized by

We are currently experiencing a DDoS attack on web404. Our upstream provider has enabled mitigation for this attack.

There are still some intermittent access issues until the attack subsides further.

We will update this post as soon as we have further information.

2016-04-02 04:53 UTC: Cisco Guard is still enabled to help reduce the impact. We are continuing to monitor.

2016-04-02 12:20 UTC: The attack was overwhelming our upstream provider’s DDoS mitigation system, so they were forced to null-route the server’s IP address. We’re working with them to restore service as soon as possible.

2016-04-02 13:00 UTC: The DDoS attack has subsided and the null route has been removed. We are continuing to monitor.

2016-04-02 23:35 UTC: The DDoS attack has resumed and the server IP has been null-routed again. Apologies for the troubles while we work to restore the server.

2016-04-03 02:03 UTC: The null-route on the server IP has been removed. The server is accessible with mitigation services enabled to help prevent further downtime. This however can cause intermittent access for some while the DDoS is in effect. We’ll continue to monitor the server. Thank you for your patience.

2016-04-04 17:07 UTC: Web404 has been stable for over 24 hours.

2016-04-04 18:10 UTC: The attack has resumed and Web404 has been null-routed again. We’re working to restore service at this time.

2016-04-04 18:47 UTC: The attack as subsided and the null-route has been lifted.

-
-

[Fixed] Web491 inaccessible

Posted in Downtime by

Web491 is currently inaccessible. We are investigating the issue, please stand by for updates.

2016-03-30 08:10 UTC: Web491 is back online.

-
-

[Fixed] Web524 Emergency RAID Maintenance

Posted in Uncategorized by

Web524 is being taken down to check the RAID controller. Please stand by for updates.

2016-03-24 21:24 UTC: Web524 is back online.

-
-