[Fixed] Web89 down

Posted in Downtime by

Web89 is currently down. We’re investigating the cause and hope to have service restored soon.

[UPDATE] We are still working with our datacenter to bring the server back online, currently there is no ETA. The issue is network related.

[UPDATE 19:52 GMT] All analysis of the server itself show no defect so far. The issue still seems to be within the network. There is still no ETA on a fix.

[UPDATE 21:20 GMT] Datacenter personnel are still investigating the issue. They are currently booting the system to a live CD. There is still no ETA on a fix.

[UPDATE 22:19 GMT] Web89 is back up and serving content. If you are still experiencing issues please let us know so we can resolve them. Datacenter personal are still unsure as the root cause of the issue.

-
-

[Fixed] Web90 down.

Posted in Downtime by

Web90 is currently down. We’re investigating the cause and hope to have service restored soon.

Update [05:00 GMT]: The problem appears to be file-system related. We’re running an FSCK on the server now.
Update [05:35 GMT]: The automatic FSCK failed. We are now running a manual FSCK on the server.
Update [06:35 GMT]: The manual FSCK is at 38% and still running.
Update [07:05 GMT]: The first FSCK has passed. We are currently running as second FSCK and it is at 86%.
Update [07:37 GMT]: The server is back online and serving requests.

-
-

[Done] Scheduled downtime on Web62 on Sunday

Posted in Scheduled downtime by

We will be taking down Web62 on Sunday Jan 31st at 4pm GMT to add some new drives to the server. We expect the work to take around 30mins and we will update this post when the work is complete.

Update 5.45pm GMT: the work is now complete

-
-

[Fixed] Web77 Slow performance, intermittent 502 errors

Posted in Downtime by

We are looking into the issue and will update this post when we have more information.

Update/Fixed [02:30 GMT]: The cause of the problems were enormous amounts of traffic being sent to the server. We have isolated and blocked the IPs responsible for the increased traffic and the server is working normally.

-
-

[Done] Schedule work on Web38 on Jan 28th 2010

Posted in Scheduled downtime by

We will be investigating the cause of visual alarms on web38.webfaction.com on Jan 28th, 2010 at 6am GMT.

Depending on the severity we might have to take the server down and if so it will be down between a few minutes and a few hours. We will update this ticket as soon as we have more information tomorrow

Update: Web38 was taken down for 5 minutes, and it is now fully operational again.

-
-

[Done] Schedule work on Web11 tomorrow

Posted in Scheduled downtime by

We will be investigating the cause of visual alarms on web11.webfaction.com on Jan 17th, 2010 at 4pm GMT.

Depending on the severity we might have to take the server down and if so it will be down between a few minutes and a few hours. We will update this ticket as soon as we have more information tomorrow

Update (6.20pm GMT): We replaced one of the drives in the RAID and the server is now back online.

-
-

[Fixed] Web22 down

Posted in Downtime by

Web22 is currently down. We’re investigating the cause and hope to have service restored soon.

Update [10:20 GMT]: The problem appears to be file-system related. We’re running an FSCK on the server now.

Update [13:50 GMT]: FSCK still running.

Update [17:15 GMT]: FSCK still running. We’ll update ASAP.

Update [20:06 GMT]: Unfortunately fsck didn’t fix the errors on the filesystem so at this point we are going to re-install the server and restore all data from backup.

Update [21:28 GMT]: We are currently restoring files on web22.

Update [22:26 GMT]: We are still restoring files on web22.

Update [12:53 GMT]: We are restoring user files on web22.

Update [02:10 GMT]: We are still restoring user files on web22.

Update [02:53 GMT]: We are still restoring user files on web22.

Update [02:53 GMT]: Web22 is up and we are verifying files.

Update [05:30 GMT]: Web22 is up and fully functional.

-
-

[Fixed] Web48 down.

Posted in Downtime by

Web48 is currently down. We’re investigating the cause and hope to have service restored soon.

Update [07:20 GMT]: Web48 is back online and the problem has been resolved. The problem appears to have been bad memory which was replaced. The server is working normally at this time.

-
-

[Fixed] Web40 down

Posted in Downtime by

Web40 is currently down. We’re investigating the cause and hope to have service restored soon.

Update [21:22 GMT]: Web40 is back online. The problem appears to have been an extremely high spike in system load. The server is working normally at this time.

-
-

[Fixed] Web1 undergoing emergency maintenance

Posted in Downtime by

Web1 is down for emergency software maintenance. We are working to resolve the issues and bring the machine back up.

Update [12:45 GMT]: Web1 is fully operational again. The problem was a misconfiguration in the boot manager.

-
-