[Fixed] Webmail server offline

Posted in Problems by

Our webmail server is currently offline. We’re working to restore service and will update this post as soon as more information is available.

2016-06-22 00:55:04 GMT: The webmail server is back online.

-
-

[Fixed] Web519 inaccessible

Posted in Downtime by

There is an issue with the network card in the server.

We are investigating the issue with our upstream provider.

2016-06-21 22:52 UTC: We have identified the problem and are working with the datacenter to resolve the issue.

2016-06-21 23:38 UTC: The network issue has been resolve and the server is back online.

 

-
-

Web456 Inaccessible (DDoS)

Posted in Downtime by

The server is currently inaccessible. We are investigating the issue.

2016-06-18 09:50 UTC: The public IP of the server was null-routed by our upstream because of a large-scale DDoS. We are working with them  to restore the service.

2016-06-18 11:10 UTC: The server has been placed under DDoS mitigation. It is currently accessible, but some customers may experience degraded network connectivity. We continue to monitor the machine closely.

2016-06-18 13:30 UTC: The server is inaccessible again. We are investigating the issue.

2016-06-18 14:00 UTC: The server is accessible again. We continue to monitor.

2016-06-19 21:12 UTC: The server has remained stable, we’ll consider this issue closed.

2016-06-20 12:01 UTC: The server is inaccessible again. We are investigating the issue.

2016-06-20 14:06 UTC: The server has been placed under DDoS mitigation. It is currently accessible, but some customers may experience degraded network connectivity. We’ll continue to monitor the machine.

2016-06-23 16:14 UTC: The server is again being hit with a DDoS attack.  We are working with the datacenter to get the server accessible again.

2016-06-24 10:00 UTC: The server has been under DDoS mitigation for some time. It is currently accessible, but some customers may experience degraded network connectivity.

2016-06-25 11:00 UTC: The DDoS is still going on. The server is using the DDoS mitigation system, and the degraded network connectivity for some customers on the machine may continue.

2016-06-25 14:00 UTC: The scale of DDoS increased, and the server is inaccessible again. We are investigating the issue.

2016-06-26 11:30 UTC: The server is currently accessible. We’ll continue to monitor the machine.

2016-06-26 13:40 UTC: The server is again being hit with a DDoS attack and it has been placed under DDoS mitigation. It is currently accessible, but some customers may experience degraded network connectivity. We continue to monitor the machine closely.

2016-06-27 08:00 UTC:  We are in the process of setting up an alternative DDoS protection system. Customer sites may produce a ‘Site not configured’ page until we are done. We will update the server status as soon as we are done.

2016-06-27 09:00 UTC:  The new system is  in place. The sites on the server are accessible. If you experience any problems, please post a support ticket.

2016-07-06 15:15 UTC: The server is still being hit with a DDoS attack. However, mitigation is in place to keep the server accessible.

-
-

[Fixed] Web458 Mysql Server Not Operational

Posted in Downtime by

The MySQL server on Web458 is down. We’re working to restore the MySQL service on Web458 at this time, and will update this post when we have more information.

1730 UTC  The MySQL InnoDB store is corrupted. We’re attempting a dump/reload at this time.

1815 UTC All databases have been restored.

1920 UTC The Mysql server has been returned to Operational status. We will continue to monitor, but please open a ticket if you are still seeing an issue.

-
-

[Fixed] Web75 inaccessible

Posted in Downtime by

The server is currently not responding. We are looking into the issue and will post any updates as soon as possible.

2016-06-13 21:50 UTC: We are still working with our upstream provider to get the server back online.

2016-06-14 00:40 UTC: We have identified the RAID issue and are rebuilding in rescue mode to prevent any data loss.

2016-06-14 03:29 UTC: One of the failed drives in the RAID array has finished rebuilding, we are replacing the other and letting it rebuild before bringing back online.

2016-06-14 9:30 UTC: Some additional issues with the RAID setup on the server were discovered after our previous update. We have already resolved them and now we are waiting for the second drive in the RAID array to finish rebuilding. The process is currently at 20%.

2016-06-14 9:50 UTC: The process of rebuilding the second drive is currently at 40%.

2016-06-14 10:20 UTC: The process of rebuilding the second drive is currently at 60%.

2016-06-14 10:45 UTC: The process of rebuilding the second drive is currently at 76%.

2016-06-14 11:05 UTC: The process of rebuilding the second drive is currently at 88%.

2016-06-14 13:22 UTC: The servers is back online.

2016-06-14 19:20 UTC: The server went read-only, we are working on a solution.

2016-06-14 22:27 UTC: We are still working on a solution to prevent data loss.

2016-06-15 04:21 UTC: The RAID was beyond the state of recovery, we are replacing the server and will restore from backup.

2016-06-15 06:50 UTC: The server is setup and online. We have restored all databases and home directories starting with the letters ‘a’ through ‘d’.

2016-06-15 08:25 UTC: All home directories up to ‘f’ have been restored.

2016-06-15 09:25 UTC: All home directories up to ‘g’ have been restored.

2016-06-15 10:25 UTC: All home directories up to ‘i’ have been restored.

2016-06-15 14:46 UTC: All home directories up to ‘l’ have been restored.

2016-06-15 17:18 UTC: All home directories up to ‘p’ have been restored.

2016-06-15 20:17 UTC: All home directories up to ‘s’ have been restored.

2016-06-15 22:21 UTC: All accounts have been restored from backup. If you experience any issues with your sites, open a ticket and we’ll look into it ASAP.

-
-

[resolved] web162 inaccessible

Posted in Downtime by

We are currently investigating the issue and will post an update as soon as we have further information.

2016-06-11 16:32 UTC The server has been rebooted and the server is back online.

-
-

[resolved] web529 inaccessible

Posted in Downtime by

We are currently investigating the issue and will post an update as soon as we have further information.

2016-06-09 17:48 UTC We are in the process of swapping the RAM in the server to attempt to bring it back online. We will update again as soon as there is more details.

2016-06-09 17:56 UTC The RAM has been swapped and the server is back online.

-
-

[resolved] Web49 inaccessible

Posted in Downtime by

We are currently investigating the issue and will update as soon as we have further details.

2016-06-02 16:00UTC: The server is back online.

2016-06-03 05:39UTC: The server has become inaccessible, we are working towards a resolution and will make updates here as soon as we can. Thank you for your patience.

2016-06-03 07:02UTC: The server has returned to normal operation, please send us a support ticket if there are believed to be any other issues.

-
-