[Done] Scheduled down time on Sunday.

Posted in General by

On Sunday at…

  • …4 PM GMT we are going to replace Web 1’s /dev/sdc with a 300 GB HDD. Expected down time is an hour.
  • …5 PM GMT we are going to reboot Dweb 30, Dweb 57, Dweb 58, Dweb 59, Mail 2, and Mail 3 to install a kernel update. Expected down time is 15 minutes.

Update [4:45 PM GMT]: Due to an unexpected delay Web 1 was just taken down. We apologize for the inconvenience.
Update [6:10 PM GMT]: Updates:

  • At the rate data is rsyncing Web 1 will likely be down for another 2 and a half hours.
  • Dweb 30, Dweb 57, Dweb 58, Dweb 59, Mail 2, and Mail 3 have been rebooted.

Update [7:00 PM GMT]: At the rate data is rsyncing Web 1 will likely be down for another hour and 15 minutes.
Update [8:40 PM GMT]: The rsyncing is complete.
Update [10:00 PM GMT]: Done

-
-

[Fixed] DNS problem affecting multiple servers

Posted in Problems by

A problem with one of our DNS servers briefly interrupted access to most of our servers. The issue has been resolved and all servers are now accessible.

-
-

[Fixed] Network problems affecting multiple servers

Posted in Problems by

Network problems in the data center are currently causing degraded performance and intermittent outages on multiple servers. We’re looking into the problem and hope to have it resolved soon. We will update this post as the situation develops.

Update (7.13pm CST): Servers currently affected are: Web32, Web12, Dweb21, Dweb28, Web14, Dweb52, Web24, Web54, Dweb19, Web13, Mail4.

Update (7.30pm CST): Servers are now back online.

-
-

[Done] Scheduled work on Mail5 and Web11

Posted in Scheduled downtime by

We will be investigating the cause of visual alarms on mail5.webfaction.com and web11.webfaction.com at 30 Dec 2009 0400 GMT-6.

Depending on the severity we might have to take the server down and if so it will be down between a few minutes and a few hours. We will update this ticket as soon as we have more information tomorrow.

-
-

[Reminder] Kernel upgrade tomorrow.

Posted in Scheduled downtime by

See http://statusblog.webfaction.com/2009/12/14/kernel-upgrade/.

-
-

[Done] Kernel Upgrade

Posted in Scheduled downtime by

On Sunday December 20 at 11 AM CST we will begin rebooting the following servers to install a kernel upgrade:

  • Dweb19
  • Dweb21
  • Dweb23
  • Dweb26-dweb30
  • Dweb32
  • Dweb33
  • Dweb39
  • Dweb42-dweb47
  • Dweb49-dweb54
  • Mail1
  • Mail5-Mail8
  • Mailbox1-Mailbox4
  • Mamba
  • panel.webfaction.com
  • Taipan
  • Viper
  • Web21-Web34
  • Web36-Web43
  • Web46-Web48
  • Web50
  • Web52
  • Web54
  • Web56-Web66
  • Web68
  • Web69
  • Web71-Web83
  • Web85-Web89
  • Web91-Web100
  • Web102-Web108
  • Web110
  • We expect to be done by 7 PM CST and we do not expect any server to be down for more than 10 minutes.

    Update [06:35 PM CST]: Done

    -
    -

    [Fixed] Web51 is currently down

    Posted in Downtime by

    We are looking into the issue and will update this post when we get more info.

    Update 1: We rebooted the server and it is currently running fsck.

    Update 2: fsck is currently on a second pass

    [09:10 PM CST] Update: The HD appears to be failing so we are going to attempt to back up what we can and replace it.

    [10:50 PM CST] Update: The HD backup is underway.

    [1:25 AM CST] Update: The HD backup is still underway.

    [3:13 AM CST] Update: We are now going to reinstall the server and restore the data.

    [7:54 AM CST] Update: The restoration of the data is still ongoing.

    [1:17 PM CST] Update: The restoration of the data is still ongoing but some network issues between datacenters are slowing down the restoration. If you want to setup your site on another server in the mean time just open a ticket and we’ll give you a free extra plan on another machine.

    [3:52 PM CST] Update: The network issues between datacenters have been fixed and data restoration is now happening at full speed. Data restoration is about 50% complete.

    [6 PM CST] Update: The server is now back online. Open a ticket if you notice any problem with your account. We would like to apologize for the extended downtime. We are working hard to improve our procedures and our setup to minimize downtime when file system corruptions happen.

    -
    -