[Done] Scheduled maintenance work on June 13th, 2010

Posted in Scheduled downtime by

On Sunday June 13, 2010 at 4pm UTC the datacenter will be performing some upgrades to the network. We expect less than 20mins of downtime between 5am and 10am UTC on the following servers:

dweb32, dweb45, dweb49, dweb54, dweb57, dweb60, dweb63-dweb67, mail1, mail6, mamba, web1-web4, web15, web17, web19, web38, web44-web47, web49, web51, web55-web58, web60-web83, web134, web135, web136, web141

-
-

[Done] Scheduled maintenance work on May 30, 2010

Posted in Scheduled downtime by

On Sunday May 30, 2010 at 4pm UTC we will be upgrading the switches on two of our racks. We expect less than 30mins of downtime on the following servers:

dweb13, dweb14, dweb19, dweb21, dweb28, dweb29, dweb30, dweb33, dweb39, dweb52, dweb53, dweb59, dweb62, mail4, mail5, mail9, web101, web8, web11, web12-web14, web140, web16, web18, web20-web26, web32, web36, web39, web52, web54, web59, web103-web133

Update May 30, 2010 8:50pm UTC: The switch upgrades are complete and all servers are all available.

-
-

[Done] Scheduled maintenance work on Web6 on May 31st

Posted in Scheduled downtime by

On May 31st, 2010 at 10am UTC we will be adding some new hard drives to Web6. The server should be down for less than 20mins.

Update May 31, 2010 10:40pm UTC: The upgrade has been completed and the server is back online.

-
-

[Done] Scheduled maintenance work on June 6, 2010

Posted in Scheduled downtime by

On June 6, 2010 the datacenter will be adding additional capacity to some routers.
We anticipate 20 mins or downtime or less between 5am and 10am UTC on the following servers:

dweb23, dweb26, dweb27, dweb46, dweb47, dweb80
krait
mail2
web5-web7, web28-web31, web33-web35, web37, web40-web42, web48, web50, web53

We also anticipate a short downtime on our control panel.

-
-

[Fixed] Web55 inaccessible

Posted in Downtime by

Web55 is inaccessible. Ping is OK, but SSH and all web requests time out. The machine is being rebooted.

Edit: The reboot succeeded without complications, and the server is now up.

-
-

[Resolved] Web51 inaccessible

Posted in Downtime by

web51.webfaction.com is down, and we are currently working on resolving the issue.

Details:
Apache on Web51 failed to bind to ports 80/443. The server is in the process of being rebooted.

Edit: Web51 functionality restored after reboot completed.

-
-

[Fixed]Multiple Servers: Outage on mulitple servers.

Posted in Downtime by

We are currently investigating a loss of connectivity to multiple servers. We will update this post when we have more information.

[Update 8:02 UTC]: Servers are now coming back online. We will update when all of the servers are back online.

[Update 8:20 UTC]: All servers are now back online and fully functional.

-
-

[Fixed] Slow DNS lookups on web127

Posted in Downtime by

web127 is currently experiencing some DNS-related speed issues. This is causing some services such as MySQL and PostgreSQL to be much slower than usual.

We’re working on this now and will have it resolved ASAP.

[Update May 21, 2010 – 12:50 UTC] This has now been fixed – all DNS issues now appear to be resolved.

-
-

Router upgrades for DDOS protection May 30th

Posted in Downtime by

On May 30, 2010, between 6:01am and 12:00am UTC our service provider will be upgrading their routers in both Houston datacenters (hstnx1 and hstnx2) to enhance DDOS resilience. This should not cause any outages as all the work will be on redundant hardware . However customers may experience increased latency or sub optimal routing for brief spells during this window.

-
-

[Fixed] web55 is currently down

Posted in Downtime by

The main Apache server for web55 as well as the ssh server are currently down, which is preventing all traffic to the server. We are looking into this and will update this post as more information is uncovered.

We contacted the datacenter and they power cycle the machine. All services are now back and operational.

-
-