On Sunday June 13, 2010 at 4pm UTC the datacenter will be performing some upgrades to the network. We expect less than 20mins of downtime between 5am and 10am UTC on the following servers:
dweb32, dweb45, dweb49, dweb54, dweb57, dweb60, dweb63-dweb67, mail1, mail6, mamba, web1-web4, web15, web17, web19, web38, web44-web47, web49, web51, web55-web58, web60-web83, web134, web135, web136, web141
On Sunday May 30, 2010 at 4pm UTC we will be upgrading the switches on two of our racks. We expect less than 30mins of downtime on the following servers:
dweb13, dweb14, dweb19, dweb21, dweb28, dweb29, dweb30, dweb33, dweb39, dweb52, dweb53, dweb59, dweb62, mail4, mail5, mail9, web101, web8, web11, web12-web14, web140, web16, web18, web20-web26, web32, web36, web39, web52, web54, web59, web103-web133
Update May 30, 2010 8:50pm UTC: The switch upgrades are complete and all servers are all available.
On May 31st, 2010 at 10am UTC we will be adding some new hard drives to Web6. The server should be down for less than 20mins.
Update May 31, 2010 10:40pm UTC: The upgrade has been completed and the server is back online.
On June 6, 2010 the datacenter will be adding additional capacity to some routers.
We anticipate 20 mins or downtime or less between 5am and 10am UTC on the following servers:
dweb23, dweb26, dweb27, dweb46, dweb47, dweb80
krait
mail2
web5-web7, web28-web31, web33-web35, web37, web40-web42, web48, web50, web53
We also anticipate a short downtime on our control panel.
Web55 is inaccessible. Ping is OK, but SSH and all web requests time out. The machine is being rebooted.
Edit: The reboot succeeded without complications, and the server is now up.
web51.webfaction.com is down, and we are currently working on resolving the issue.
Details:
Apache on Web51 failed to bind to ports 80/443. The server is in the process of being rebooted.
Edit: Web51 functionality restored after reboot completed.
We are currently investigating a loss of connectivity to multiple servers. We will update this post when we have more information.
[Update 8:02 UTC]: Servers are now coming back online. We will update when all of the servers are back online.
[Update 8:20 UTC]: All servers are now back online and fully functional.
web127 is currently experiencing some DNS-related speed issues. This is causing some services such as MySQL and PostgreSQL to be much slower than usual.
We’re working on this now and will have it resolved ASAP.
[Update May 21, 2010 – 12:50 UTC] This has now been fixed – all DNS issues now appear to be resolved.
On May 30, 2010, between 6:01am and 12:00am UTC our service provider will be upgrading their routers in both Houston datacenters (hstnx1 and hstnx2) to enhance DDOS resilience. This should not cause any outages as all the work will be on redundant hardware . However customers may experience increased latency or sub optimal routing for brief spells during this window.
The main Apache server for web55 as well as the ssh server are currently down, which is preventing all traffic to the server. We are looking into this and will update this post as more information is uncovered.
We contacted the datacenter and they power cycle the machine. All services are now back and operational.