[Fixed] Web323 Inaccessible

Posted in Downtime by

Web323 is currently inaccessible and we are investigating as a matter of urgency – we’ll update this post when we have more information.

2012-09-16 20:06 UTC: Web323 is back online.

-
-

[Fixed] Control panel, API, and www down

Posted in Problems by

Our control panel, control panel API, and public sites are currently down. We’re currently working to restore service and will update this post when we have more information.

2012-09-14 18:25 UTC: We’re still working to restore service. It may be a couple more hours, but we’re hoping it will be sooner than that.

2012-09-14 19:20 UTC: The control panel, API, and www.webfaction.com are back online.  We took down the sites to investigate a potential security breach and only put them back online once we were satisfied that no data was compromised.

-
-

[Fixed] MySQL down on Web344

Posted in Problems by

The MySQL service is currently down on Web344. We’re working to restore service and will update this post when we have more information.

2012-09-10 23:23 UTC: The MySQL service is up in a partial read-only state at this time (InnoDB data cannot be modified).

2012-09-11 00:04 UTC: We’re backing up all MySQL data at this time and will restore it to a clean MySQL installation.

2012-09-11 01:49 UTC: We’re about 50% through restoring the MySQL data.

2012-09-11 02:18 UTC: All MySQL data has been restored and the MySQL service is operating normally.

-
-

[Fixed] Web348 is inaccessible, other servers affected as well, see post

Posted in Downtime by

The server is under a denial of service attack.

We are working on mitigating it and getting the server back up as soon as possible. A permanent solution is underway.

2012-09-03 10:37 UTC: The server is back for than 30 minutes now and the attack is over.

2012-09-03 11:42 UTC: The attack has started again. We are working to get the server back.

2012-09-03 13:59 UTC: The server is back for than one hour now.

2012-09-03 20:47 UTC: Experiencing network issues. We are investigating.

2012-09-03 23:07 UTC: The server was being DDoS’d, but the attack as subsided now. The server is accessible again.

2012-09-04 01:43 UTC: The server is back offline from another attack. The following servers are also affected as well: web349, web347, web346, web345, web247, web246

2012-09-04 11:50 UTC: The server has been continuously accessible for more than an hour now and we believe the volume based attack has stopped. We are still keeping an eye on the traffic in case it re-occurs.

2012-09-04 14:16 UTC:  The server is back offline from another attack. We are working with our datacenter provider to resolve this.

2012-09-04 21:03 UTC: Web348 is back online at this time.

2012-09-04 23:21 UTC: We’re seeing no further DDOS activity on web348, and we’ve taken measures to minimize the impact of an attack, should one happen again.

-
-