[Done] Filesystem Upgrade on Web248

Posted in Downtime by

Web248’s filesystem will be upgraded from ext3 to ext4 on Wednesday 3rd July 2013

between 5am to 9 am UTC.

We will be updating this post as the upgrade progresses.

[Jul  3 05:19] The upgrade is over and we are running a file system check now to make sure it is OK now. It is at 25%

at the moment.

[Jul  3 06:20] The filesystem check is over and the server is back.

-
-

[Done]Emergency msyql maintenance on web178

Posted in Downtime by

Web178 has been taken offline for emergency maintenance on mysql.

2013-06-28 23:41 UTC:  We are taking the server offline to perform a file system check in order to verify the MySQL problems are not being caused by file system corruption. We will update this post as the maintenance continues.

2013-06-29 01:23 UTC: The file system check has finished and the MySQL data corruption was not fixed as part of the check. We are continuing to investigate recovery options.

2013-06-29 02:31 UTC: We are now attempting a full export of any databases that do not have corrupted data. During this process the Nginx web server may be offline to stop automated queries from firing and possibly accessing corrupted tables causing the MySQL server to crash.

2013-06-29 03:31 UTC: The MySQL server is now back online and fully functional.

-
-

Outgoing SMTP server to discard mails using well-known domains in senders

Posted in Downtime by

Effective 11 July 2013, our outgoing SMTP servers will stop relaying outgoing messages which use well-known domains like gmail.com,yahoo.com, aol.com etc in the sender address. Messages sent using such domains in the sender address will be silently discarded.

This is being done as most of these domains use SPF and DKIM, hence relaying mails for them decreases our SMTP server’s reputation and increases chances of legitimate mails getting misidentified as spam.

If you have any questions or concerns regarding this change, please open a support ticket via our control panel to let us know.

[Jun 29] We have extended the deadline by which we will be implementing this to 11th July 2013.

-
-

IP address change for ns1.webfaction.com next Wednesday, 3 July 2013

Posted in Downtime by

From Wednesday July 3rd ns1.webfaction.com will point to 108.168.213.81 instead of 198.61.215.225 currently.

If you own any domains and use our DNS servers then you may have given the previous IP addresses to your domain registrar. If that is the case then we recommend that you update the IP addresses in your registrar’s control panel between June 30th and July 3rd . Since we are changing the IP addresses of just one of our four DNS server there is no danger of your website or emails being inaccessible while you make the change.

Some registrars only ask for the DNS servers names not their IP addresses. If that’s the case with your registrar then you don’t need to do anything.

You can find more information about our DNS servers here: http://docs.webfaction.com/user-guide/domains.html#pointing-your-domain-to-webfaction-s-servers

-
-

[Fixed]DDoS on Web391

Posted in Downtime by

Web391 server is experiencing a massive DDoS attack which our DDoS protection system is having trouble mitigating.

We’re doing everything we can to restore service at this time, and will update this post as soon as we have more information.

2013-06-25 19:35 UTC:  The server is now back online as the traffic has dropped to a level where the DDoS protection system can mitigate it. We are continuing to monitor the situation closely.

2013-06-25 22:02 UTC: The DDoS protection system is still in effect and blocking large amounts of traffic. We’re continuing to monitor the server and network quickly.

2013-06-26 06:37 UTC: The DDoS protection system is still in effect but the DDoS seems to have abated.

-
-

[Done] Filesystem Upgrade on Web22

Posted in Downtime by

Web22′s filesystem will be upgraded from ext3 to ext4 on Thursday 27th June 2013

between 5am to 9 am UTC.

We will be updating this post as the upgrade progresses.

[Jun 27 05:01] We have put the server into rescue mode to start the upgrade.

[Jun 27 05:22] The upgrade is done and we are running a filesystem check now.

[Jun 27 05:59] The check is at 82% now.

[Jun 27 06:17] The server is back and OK now.

-
-

[Completed]Scheduled Web31 migration

Posted in Downtime by

As scheduled we are starting the migration of Web31 to new hardware. During the migration services on the machine will be unavailable. We will update this post once the migration is over.

2013-06-25 01:16 UTC: The migration is still in process.

2013-06-25 04:01 UTC: The scheduled maintenance window has now concluded but the server is not back online and fully functional yet. We are working to insure that the server is functioning properly before allowing any users log ins.

2013-06-25 04:26 UTC: The server is now online and working normally.

-
-

[Fixed] Emergency maintenance on Web330

Posted in Downtime by

We have taken the server offline for emergency maintenance.

We will update as soon as we have more information.

2013-06-22 00:00 UTC: Web330 is back online

-
-

[Done] Scheduled maintenance on Mail5 on June 22th, 2013

Posted in Scheduled downtime by

Mail5 will be taken down to swap a failing hard drive on Saturday June 23rd between 09:00UTC and 12:00UTC. We will update this blog post as maintenance progresses.

2013-06-22 10:57: The disk has been swapped and the server is back online. There may be a slight degradation in performance as the RAID rebuilds.

-
-

[Postponed] Scheduled maintenance on Mail5, June 20th, 2013.

Posted in Scheduled downtime by

Mail5 will be taken down to swap a failing disk drive Thursday 20th June, 2013 between 07:00UTC and 10:00UTC. We will update this blog post as maintenance will progresses.

2013-06-20 11:26 UTC The disk has been swapped but the rebuild process has not started. We’re currently investigating the RAID adapter for possible issues.

2013-06-20 14:36 UTC We had to put the original drive back in place to let the RAID complete a rebuild and make the RAID ready to have a new swapped disk. The next scheduled maintenance will be at Friday 21st June 2013 between 07:00UTC and 10:00 for taking down the server and replace the drive. All services are back online but performance may be slightly degraded while the RAID rebuilds.

Update: This maintenance has been postponed. We’ll make a new post to announce the new date and time.

-
-