[Fixed] DDoS attack against Web331

Posted in Downtime by

Web331 is under a heavy DDoS attack at the moment, which is affecting its network connectivity.

We are working on mitigating the effects of the attack and restoring the normal operation of all services as soon as possible.

2014-05-09 11:10 UTC: Our datacenter engineers have enabled DDoS protection to mitigate the attack. The situation has improved but customers may still experience packet loss and other network problems. We will continue to monitor.

2014-05-11 17:00: UTC The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-12 18:32 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-13 09:50 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-15 10:50 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

2014-05-16 14:47 UTC: The attack on the server is still ongoing and Web331 remains unders DDoS protection. We will continue to monitor.

-
-

[Done] Emergency maintenance on Web368

Posted in Downtime by

We are currently doing an emergency maintenance on Web368. We are working urgently to bring all services back online ASAP.

2013-03-05 10:30 UTC:  The normal operation of the server has been restored. All services are back online.

-
-

[Fixed] Emergency maintenance on Web211

Posted in Downtime by

Web211 is currently offline. We’re working to restore service at this time and will update this post as soon as we have more information.

2013-12-10 16:36 UTC: Web211 is back online.

-
-

[Done] Emergency Maintenance on Web245

Posted in Downtime by

The filesystem on Web245 went read-only. We are booting the server into rescue mode to run a filesystem check.  We will update this post as maintenance progresses.

2013-11-20 16:45 UTC: The filesystem check is 25% through its second pass.

2013-11-20 17:39 UTC: The filesystem check is complete and Web245 is back online. Some filesystem problems were detected and repaired. If you notice any problems with your sites or data, then please open a support ticket for assistance.

-
-

[Done] Emergency Maintenance on Web178

Posted in Downtime by

The filesystem on Web178 suddenly went read-only. We are booting the server into rescue mode to run a filesystem check.  We will update this post as maintenance progresses.

Update 16:06 UTC: The filesystem has some corruption so we are migrating the server to new hardware. Note that the main IP address will change from 108.59.4.70 to 75.126.149.9 so if you are using non-DNS servers for some of your domains you will have to update them to point your domains at the new IP.

Update 17:12 UTC: All MySql and PostgreSQL databases have been migrated to the new server as well as the first few home directories. All services have been started on the new server. We are working on migrating the rest of the home directories.

Update 22:10 UTC: All home directories starting with the letter a or b have been migrated. We’re working on the rest of the home directories.

Update 06:00 UTC: All home directories up to the letter p have been migrated. We’re working on the rest of the home directories.

Update 16:47 UTC: All home directories up to the letter t have been migrated. We’re working on the rest of the home directories.

Update 08:17 UTC: The migration is now finished. If you notice any problems with your account open a ticket and our support team will look into it asap.

-
-

[Fixed] Network maintenance in Virginia affecting several servers

Posted in Downtime by

Network maintenance in one of our US data centers is affecting connectivity. We’re working to restore service and hope to have this issue resolved soon.

2013-11-15 13:45 UTC: The network maintenance at the data center is complete. All connectivity has been restored.

-
-

[Done]Emergency Maintenance on Web226

Posted in Downtime by

Web226 stopped responding and did not come back as expected following a reboot.  We are booting the server into a rescue environment to perform a filesystem check.  We will update this post as maintenance progresses.

2013-11-14 10:22 UTC: Maintenance is complete and the server is back online.

2013-11-14 11:03 The filesystem went read only again. At this point we are taking the server offline and we are going to migrate it to new hardware. Note that the main IP address of the server will change from 108.59.9.167 to 75.126.113.173. If you’re using non-WebFaction DNS servers for some of your domains you will have to update them to point at the new IP.

2013-11-14 13:30: We have started copying the data to the new hardware

2013-11-14 17:50: At this point all MySql and PostgreSQL databases have been migrated to the new server. All home directories up to the letter h have been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-14 21:00: All home directories up to the letter l have been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-15 08:00: The migration stopped for a while overnight and had to be restarted. All home directories up to the letter p have been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-15 16:00: All home directories up to the letter t have been been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-15 20:00: The migration is now over. If you experience any issues with your site open a ticket and we’ll look into it asap.

-
-

[Done]Emergency maintenance on Web309

Posted in Downtime by

Web309 filesystem is showing errors. We’re taking the server down to run a fsck.  We will update this blog post as maintenance progresses.

2013-09-18 15:49 UTC While checking the status of server’s filesystem we found hardware issues we’re currently investigating.

2013-09-18 16:05 UTC Hardware issue solved, fsck is now running.

2013-09-18 16:48 UTC:  We are 46% of the way through on the first pass of the file system check.

2013-09-18 17:49 UTC: The file system check is now at 54.7% of the first pass.

2013-09-18 19:38 UTC: The file system check failed to return a clean file system. At this time we’ve decided to migrate the good data from the old machine to a new server that is in the process of being set up.

2013-09-18 21:23 UTC:  The data from the old server is now being transferred to the new server.

2013-09-19 00:58 UTC: We’re still transferring the data from the old server to the new server.

2013-09-19 05:46 UTC: MySQL, Postgresql and Cronjobs have been restored already so sites should be coming back up as the home directory restore progresses, it is at 30% now. Please file a support ticket if you notice any problems with the restored data.

2013-09-19 10:02 UTC: 53% of the data has been  transferred, please note the data is being restored alphabetically and it is upto “J” now, if your username comes very late please let us know by filing a support ticket to jump ahead in the queue.

2013-09-19 15:14 UTC: Approximately 92% of the data has been transferred and it is up to “P” now.

2013-09-19 18:19 UTC:  The transfer of data from the old server has finished.

-
-

[Fixed] Web364 down

Posted in Downtime by

Web364 has suddenly gone down and we are currently working to bring it back online ASAP.

2013-08-25 16:53 UTC: Web364 is back and operating normally.

-
-

[Done] Emergency maintenance on Web247

Posted in Downtime by

Web247 filesystem is showing inconsistencies. We’re taking the server down to run a fsck. We’ll update this post as the maintenance progresses.

2013-08-10 11:59 UTC: The fsck first pass is complete, second pass underway.
2013-08-10 12:34 UTC: Second pass at 60%.
2013-08-10 13:48 UTC: Second pass complete, scan 1C is underway.
2013-08-10 14:46 UTC: Fsck completed and the server is back at operational status.

-
-