[Done] Emergency maintenance on Mailbox10, April 16, 2017

Posted in Downtime by

Mailbox10 will be taken down for an HDD swap and possibly RAID controller replacement,
Sunday April 16th, 2017 between 1:00 UTC and 5:00 UTC.

Sun Apr 16 01:15:32 UTC 2017: The server has been taken offline for maintenance

Sun Apr 16 02:15:32 UTC 2017: The failing drive has been replaced and the RAID is rebuilding (currently 3%).

Sun Apr 16 02:45:34 UTC 2017: The rebuild has failed, which indicates a faulty RAID controller, which is now being replaced.

Sun Apr 16 03:29:52 UTC 2017: The RAID controller has been replaced; now doing build/verify and rebuild.

Sun Apr 16 06:22:50 UTC 2017: The RAID controller initial build/verify is done and new HDD is rebuilding (currently 15%).

Sun Apr 16 07:00:51 UTC 2017: HDD is rebuilding (currently 35%).

Sun Apr 16 08:22:08 UTC 2017: HDD is rebuilding (currently 80%).

Sun Apr 16 08:52:54 UTC 2017: The rebuild has completed and the server is back online.

-
-

[Completed] Emergency maintenance on web474, April 02, 2017

Posted in Downtime by

Web474 will be taken down for hard disk replacement on Sunday April 2nd, 2017 between 04:00 UTC and 08:00 UTC.

We will update this post as maintenance progresses.

Sun Apr  2 04:18:02 UTC 2017: The server has been taken offline for maintenance.

Sun Apr  2 05:39:44 UTC 2017: The first of two hard disks has been replaced and the RAID is rebuilding (25% currently)

Sun Apr  2 07:34:41 UTC 2017: The RAID has finished rebuilding after the first disk replacement, and the second disk is now being replaced.

Sun Apr  2 07:45:17 UTC 2017: The server is back online. The second disk was replaced and the RAID is rebuilding, which will decrease performance until it finishes.

Sun Apr  2 12:14:17 UTC 2017: All drives in the RAID have finished rebuilding. Maintenance is now complete.

-
-

[Fixed] Web366 unreachable

Posted in Uncategorized by

web366.webfaction.com is unreachable by the external network, and initial evidence suggests possible router failure. Our system administrators are looking into it.

[2016-05-23] The router issue has been addressed and the server is back online.

-
-

[Fixed] Web409 read-only filesystem

Posted in Downtime by

Web409 is down with a read-only filesystem. We are currently looking into the issue and will resolve it as quickly as possible.

[Sun Oct  4 02:00:34 UTC 2015] The machine has been taken offline for a filesystem check

[Sun Oct  4 02:45:42 UTC 2015] The machine is back online

-
-

[Fixed]DDoS on Web371

Posted in Downtime by

Web371 server is experiencing a massive DDoS attack which our DDoS protection system is having trouble mitigating.

We’re doing everything we can to restore service at this time, and will update this post as progress is made.

Update 10:08 am: Our DDoS protection is now doing a much better job at mitigating the attack. All sites are now up but performance is slightly degraded.

Update 1pm: The attack has stopped and the server is back to normal

-
-

[Done]Emergency maintenance on web338

Posted in Downtime by

Web338 has shown poor performance and stopped responding, so we have taken the server offline for a filesystem check and hardware inspection. We will update this post as progress completes.

2013-04-28 05:09 UTC: The fsck has finished and the server is back online, but there are still some issues which need to be addressed, so the server will be rebooted again once more soon.

2013-04-28 05:44 UTC: We are now rebooting the machine after some changes to bring it fully back online.

2013-04-28 06:49 UTC: The server is now online and functioning normally.

-
-