[Fixed] Issue with MySQL on Web311

Posted in Downtime by

The MySQL service on web311 is currently down. We’re working to restore service and will update this post when we have more information.

2014-05-15 13:42 UTC: We discovered an issue at filesystem level of database and we are dumping all data from it.

2014-05-15 14:27 UTC: We finished dumping the data and we are reloading data into a new mysql instance

2014-05-15 15:35 UTC: We are still reloading data on server

2014-05-15 16:35 UTC: We are still reloading data on server

2014-05-15 17:02 UTC: All databases have been restored and MySQL is back online. If you have any problems, please open a support ticket.

-
-

[Fixed] Web218 DDoS

Posted in Downtime by

The server is still under a heavy DDoS attack. Mitigation has been put in place to try and help with the issue.

We will update this post as soon as we have more information.

2014-05-13 16:45 UTC: The DDoS has subsided and the server is accessible again. We are continuing to monitor.

2014-05-13 19:28 UTC: The server has been stable for about the last 2.5 hours. We will continue to monitor.

-
-

[Fixed] Problems on mailbox7

Posted in Problems by

Our mailbox7 email storage server is currently under heavy load. We’re working to resolve this issue and will update this post as soon as we have more information.

2014-05-12 19:41 UTC: Mailbox7 is being rebooted to pick up a new kernel

2014-05-12 00:33 UTC: Mailbox7 has been stable for the past couple of hours. If you have trouble accessing your mail, then please open a support ticket.

-
-

[Done] Emergency migration of Web218, IP address change

Posted in Service change by

Due to a DDOS attack against Web218 we are doing an emergency migration of the server to a datacenter with much better DDOS protection.

The server’s main IP will change from 85.17.88.173 to 81.95.157.163.

If you’re using WebFaction’s DNS servers for all your domains then you don’t need to do anything.

If you’re using non-WebFaction DNS servers for some of your domains then you will have to update their A records yourself. We will leave a proxy running on the old server for 5 days to redirect all HTTP and HTTPS traffic to the new server.

We’ll update this post as the work progresses.

Update 11.10pm UTC: At this point all mysql and postgresql databases have been migrated to the new server. MySql, PostgreSQL, Nginx and Apache have been started on the new server. The first few home directories have been migrated to the new server and their sites are back online. We’re continuing to migrate the other home directories in alphabetical order.

Update 6.30am UTC: Home directories up to the letter “j” have been copied to the new server. We’re continuing to copy the rest of the home directories.

Update 4.30pm UTC: Home directories up to the letter “r” have been copied to the new server. We’re continuing to copy the rest of the home directories.

Update 2014-05-12 00:34 UTC: The migration is now over. If you experience any issues with your sites open a ticket and we’ll look into it asap. We have setup a proxy on the old server to redirect all HTTP and HTTPS traffic to the new server. The proxy will stay for 5 days. The migration took much longer than usual due to an ongoing DDOS attack on the server.

-
-

[Done] Web216 and Web218

Posted in Downtime by

Web216 and Web218 based in Amsterdam are experiencing network routing issues, which greatly affects network connectivity. Heavy packet loss and other network problems are to be expected.

We are working with our datacenter on resolving the problem and we will update this post as soon as we have more information on the subject.

1820 UTC service to Web216 has been restored.

1910 UTC Web 218 is undergoing a DDoS attack. This was causing the heavy packet loss on Web216. Currently we are trying to mitigate the attack on Web218. We will update this post when we have more information.

Update: we’re closing the entry and we’ll open one just for Web218

 

-
-

[Fixed] DDOS attack on Web427

Posted in Downtime by

Web427 is experiencing network routing issues, which greatly affects its network connectivity. Heavy packet loss and other network problems are to be expected.

We are working with our datacenter on resolving the problem and we will update this post as soon as we have more information on the subject.

2014-05-10 1817 UTC there is a DDOS Attack in progress on Web427. We have placed the server in our network protection system. The situation has improved but customers may still experience packet loss and other network problems. We will continue to monitor.

 

-
-

[Fixed] DDoS attack against Web331

Posted in Downtime by

Web331 is under a heavy DDoS attack at the moment, which is affecting its network connectivity.

We are working on mitigating the effects of the attack and restoring the normal operation of all services as soon as possible.

2014-05-09 11:10 UTC: Our datacenter engineers have enabled DDoS protection to mitigate the attack. The situation has improved but customers may still experience packet loss and other network problems. We will continue to monitor.

2014-05-11 17:00: UTC The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-12 18:32 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-13 09:50 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-15 10:50 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-16 14:47 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

-
-

[Done] Scheduled maintenance on smtp.webfaction.com on Satuday, May 31st at 9am UTC

Posted in Scheduled downtime by

On Saturday, May 31st at 9am UTC smtp.webfaction.com will be down for about 30 minutes for some maintenance work.

We’ll update this post as the maintenance progresses.

Update May 31st, 9.20am UTC: The maintenance work is finished

-
-

[Done]Scheduled maintenance on Web360, May 7th, 2014.

Posted in Downtime by

Web360 will be taken down for a faulty drive and cables replacement, Wednesday,  May 7th 2014, between 07:00 UTC and 10:00 UTC. We will update this post as maintenance will progress.

2014-05-07 09:18 UTC The maintenance is starting now.

2014-05-07 11:44 UTC The server is back to operational status.

-
-

[Update] Issue with DDOS attack against web331

Posted in Downtime by

DDOS Protection has been enabled on Web331 while under investigation. The server is accessible but you may still experience network problems in the time being.

We are working with our datacenter on resolving the problem swiftly.

-
-