[Done] Migration of Web335 to faster hardware on December 4th, IP address change

Posted in Service change by

Due to performance issues on Web335 we are going to migrate it to faster hardware on December 4th between 10am and 12pm UTC.

The server’s main IP will change from 108.59.2.69 to 108.168.205.70

If you’re using WebFaction’s DNS servers for all your domains then you don’t need to do anything.

If you’re using non-WebFaction DNS servers for some of your domains then you will have to update their A records yourself.

We’ll update this post as the work progresses.

Update Dec 4th 10am UTC: The migration is starting now

Update 12:30pm UTC: The migration is now over. If you experience any issues with your sites open a ticket and we’ll look into it asap

Update 1:25pm UTC: We have setup a proxy on the old server to redirect all HTTP and HTTPS traffic to the new server. The proxy will stay for one week.

-
-

[Cancelled] Migration of Web242 to faster hardware on December 4th, IP address change

Posted in Service change by

Due to performance issues on Web242 we are going to migrate it to faster hardware on December 4th between 10am and 12pm UTC.

The server’s main IP will change from 108.168.213.67 to 75.126.149.10

If you’re using WebFaction’s DNS servers for all your domains then you don’t need to do anything.

If you’re using non-WebFaction DNS servers for some of your domains then you will have to update their A records yourself.

We’ll update this post as the work progresses

Update: This post was written in error. No migration will happen on web242.

-
-

[Fixed] DDOS attack against Web199

Posted in Downtime by

Web199 is experiencing a DDOS at this moment, which greatly affects its network connectivity. Packet loss and other network problems are to be expected.

We are working with our datacenter on resolving the problem and we will update this post as soon as we have more information on the subject.

2013-11-25 16:39 UTC: The attack subsided, and connectivity is restored. We are carrying out an investigation along with our datacenter.

2013-11-25 17:10 UTC: The datacenter engineers reported the attack is over, so no more problems should be present, or arise. If you still experience issues connecting to the server then please contact our support team for assistance.

-
-

Emergency Maintenance for Mail5

Posted in Downtime by

Mail5 will be taken down shortly for an emergency hard drive replacement. We’ll update this post as maintenance progresses.

2013-11-24 1800 UTC the failing hard drive has been replaced, however the datacenter is replacing the RAID Controller now.

2013-11-24 21:54 UTC We are making a complete backup of the machine just in case the degraded RAID array causes data loss from the new disk.

2013-11-24 02:31 UTC We are still copying data from the degraded RAID array to the new one.

2013-11-24 06:12 UTC Data transfer still in progress.

2013-11-25 12:04 UTC The machine has been rebooted and RAID is rebuilding. The server is at operational status.

-
-

[Fixed] MySQL/Filesystem problems on Web312

Posted in Downtime by

The MySQL service on Web312 is currently down. We’re working to restore service and will update this post as soon as we have more information.

2013-11-21 20:50 UTC: Several corrupt databases are interfering with the restoration of the MySQL service. We’re working to isolate those databases to restore the service.

2013-11-21 22:15 UTC: We’re checking the Web312 filesystem at this moment because its filesystem became corrupt when restoring databases.

2013-11-21 23:12 UTC: The filesystem was too damaged to be recovered. We’re preparing a new server to restore backups.

013-11-22 00:50 UTC: We are still setting up the new server. We will update this post as soon as we have more information.

2013-11-22 02:11 UTC: The new server is up and running and data is being restored.

2013-11-22 05:07 UTC: PostgreSQL and MySQL Databases were restored. Home data is still being restored.

2013-11-22 13:53 UTC: Most home directories have been restored. We’re working on restoring the remaining data.

2013-11-23 09:30 UTC: 95% of home directories have now been restored. If your home directory is still missing data please open a support ticket.

2013-11-23 11:30 UTC: All home directories have now been restored. If you’re still experiencing issues please contact support.

-
-

[Done] Emergency Maintenance on Web245

Posted in Downtime by

The filesystem on Web245 went read-only. We are booting the server into rescue mode to run a filesystem check.  We will update this post as maintenance progresses.

2013-11-20 16:45 UTC: The filesystem check is 25% through its second pass.

2013-11-20 17:39 UTC: The filesystem check is complete and Web245 is back online. Some filesystem problems were detected and repaired. If you notice any problems with your sites or data, then please open a support ticket for assistance.

-
-

[Done] Emergency Maintenance on Web178

Posted in Downtime by

The filesystem on Web178 suddenly went read-only. We are booting the server into rescue mode to run a filesystem check.  We will update this post as maintenance progresses.

Update 16:06 UTC: The filesystem has some corruption so we are migrating the server to new hardware. Note that the main IP address will change from 108.59.4.70 to 75.126.149.9 so if you are using non-DNS servers for some of your domains you will have to update them to point your domains at the new IP.

Update 17:12 UTC: All MySql and PostgreSQL databases have been migrated to the new server as well as the first few home directories. All services have been started on the new server. We are working on migrating the rest of the home directories.

Update 22:10 UTC: All home directories starting with the letter a or b have been migrated. We’re working on the rest of the home directories.

Update 06:00 UTC: All home directories up to the letter p have been migrated. We’re working on the rest of the home directories.

Update 16:47 UTC: All home directories up to the letter t have been migrated. We’re working on the rest of the home directories.

Update 08:17 UTC: The migration is now finished. If you notice any problems with your account open a ticket and our support team will look into it asap.

-
-

[Fixed] Network maintenance in Virginia affecting several servers

Posted in Downtime by

Network maintenance in one of our US data centers is affecting connectivity. We’re working to restore service and hope to have this issue resolved soon.

2013-11-15 13:45 UTC: The network maintenance at the data center is complete. All connectivity has been restored.

-
-

[Fixed] Network maintenance in Virginia affecting several servers

Posted in Downtime by

Network maintenance in one of our data centers is affecting connectivity. We’re working to restore service and hope to have this issue resolved soon.

1345 UTC All service has been restored at this time. We will continue monitoring the situation.

1640 UTC No further issues have occured and the data center reports the network maintenance is complete.

-
-

[Done]Emergency Maintenance on Web226

Posted in Downtime by

Web226 stopped responding and did not come back as expected following a reboot.  We are booting the server into a rescue environment to perform a filesystem check.  We will update this post as maintenance progresses.

2013-11-14 10:22 UTC: Maintenance is complete and the server is back online.

2013-11-14 11:03 The filesystem went read only again. At this point we are taking the server offline and we are going to migrate it to new hardware. Note that the main IP address of the server will change from 108.59.9.167 to 75.126.113.173. If you’re using non-WebFaction DNS servers for some of your domains you will have to update them to point at the new IP.

2013-11-14 13:30: We have started copying the data to the new hardware

2013-11-14 17:50: At this point all MySql and PostgreSQL databases have been migrated to the new server. All home directories up to the letter h have been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-14 21:00: All home directories up to the letter l have been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-15 08:00: The migration stopped for a while overnight and had to be restarted. All home directories up to the letter p have been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-15 16:00: All home directories up to the letter t have been been migrated and their sites are running. We are working on restoring the other home directories.

2013-11-15 20:00: The migration is now over. If you experience any issues with your site open a ticket and we’ll look into it asap.

-
-