[Fixed] Web307, Web308, Web360 inaccessible

Posted in Downtime by

Web307, Web308, Web360 are currently running, but are not accessible via the network. We’re working to resolve this and will update this post as soon as we have more information.

2016-02-24 21:26 UTC: Web308 is back online.

2016-02-24 21:30 UTC: Web307 is back online.

2016-02-24 21:32 UTC: Web360 is back online.

The problem was caused by a misconfigured firewall on all three servers. The problem has been corrected.

-
-

[Fixed] Multiple servers offline

Posted in Downtime by

The following servers are currently offline:

Dweb101, Dweb104, Dweb110, Dweb111, Dweb112, Dweb114, Dweb115, Dweb116, Dweb117, Dweb121, Dweb122, Dweb124, Dweb125, Dweb126, Dweb129, Dweb130, Dweb133, Dweb134, Dweb135, Dweb137, Dweb140, Dweb141, Dweb142, Dweb144, Dweb145, Dweb146, Dweb149, Dweb150, Dweb151, Dweb153, Dweb154, Dweb160, Dweb162, Dweb163, Dweb164, Dweb165, Dweb166, Dweb167, Dweb173, Dweb176, Dweb178, Dweb184, Dweb185, Dweb187, Dweb188, Dweb189, Dweb194, Dweb195, Dweb196, Dweb197, Dweb202, Dweb203, Dweb204, Dweb205, Dweb95, Mailbox8, Web102, Web105, Web106, Web108, Web114, Web117, Web126, Web129, Web148, Web162, Web175, Web180, Web182, Web187, Web198, Web219, Web220, Web227, Web229, Web232, Web236, Web238, Web24, Web240, Web243, Web244, Web247, Web25, Web27, Web28, Web30, Web300, Web308, Web311, Web318, Web319, Web320, Web328, Web330, Web342, Web343, Web345, Web347, Web35, Web49, Web55, Web80, Web91

This may be related to scheduled network maintenance by our upstream provider, but that was only expected to impact a few servers. We’re awaiting confirmation from our upstream provider and will update this post when we have more information.

2016-02-23 19:14 UTC: The problem appears to have been a power disturbance in LeaseWeb’s US data center. The machines listed above are coming back online at this time.

2016-02-23 20:34 UTC: The following servers are still offline at this time:Web25, Web106,Web182,Web300, Web308, Web343, Web347 – all of the other servers are back online. We’re working to restore service for the remaining machines and will update this post when we have more information.

2016-02-23 22:00 UTC:

  • Web343 is back online, but its MySQL service is down – we’re presently working to restore it.
  • Web106 and Web182 are showing signs of filesystem corruption. We’re running tests to confirm.
  • Web25, Web300, Web308, and Web347 are still being investigated.

2016-02-23 22:18 UTC: Web182 is back online.

2016-02-23 22:29 UTC: The MySQL service on Web343 is now restored.

2016-02-23 22:30 UTC: We’re running a filesystem check on Web25.

2016-02-23 22:42 UTC: Web182 is offline again.

2016-02-23 22:50 UTC: Web182 and Web347 are both back online. The machines that are still down are Web25, Web106, Web300, and Web308.

2016-02-23 23:24 UTC: Web25 is back online.

2016-02-24 04:03 UTC: Dweb142 is offline again at this time.

2016-02-24 06:14 UTC: Some servers that have returned online are experiencing issues impacting our services. Our system administrators are working to restore these servers to full operation. Please check back here for further updates.

2016-02-24 12:35 UTC: At this point all servers are working normally except for Web106 and Web300. We expect to have these last two servers back online shortly.

2016-02-24 17:29 UTC: Web300 is back online.

2016-02-24 20:21 UTC: At this time, Web106 seems to be in good health, but we’re unable to bring up its network interfaces. We’re presently working to resolve that, and are bringing up a backup machine as a final contingency in the event that we’re unable to get Web106 back online.

2016-02-25 00:19 UTC: Web106 is back online.

-
-

[Fixed] MySQL down on Web343

Posted in Downtime by

In the past hour, three servers (Web243, Web342, and Web343) were unexpectedly restarted due to a power loss.

All three machines are back online, but the MySQL service on Web343 is currently down.

We’re working to restore the MySQL service on Web343 at this time, and will update this post when we have more information.

2016-02-19 18:21 UTC: The MySQL InnoDB store on Web343 is corrupted. We’re attempting a dump/reload at this time.

2016-02-19 19:22 UTC: The MySQL reload is ~50% complete.

2016-02-19 20:12 UTC: The MySQL data is reloaded, and we are troubleshooting an authentication issue at this time.

2016-02-19 21:01 UTC: The MySQL service on Web343 has been restored.

-
-

[Done] Network maintenance on 17 February 2016

Posted in Scheduled downtime by

On Wednesday, February 17th, 12:00 and 14:00 UTC our upstream provider will be performing essential switch maintenance in the racks that house the following servers:

  • Dweb129
  • Dweb130
  • Dweb135
  • Mailbox8
  • Web24
  • Web25
  • Web227
  • Web229
  • Web342
  • Web343

During this time frame, the above servers will be unreachable for 30 minutes.

We’ll update this post as the maintenance progresses.

2016-02-17 14:00 UTC The maintenance is complete.

-
-

Datacenter switch maintenance

Posted in Uncategorized by

Our upstream provider is performing switch maintenance in the Manassas datacenter on February 23th 2016, 12:00-14:00UTC

The following servers will be unavailable for up to 30 minutes.

dweb149, dweb163, dweb150, dweb151

-
-

[Fixed] MySQL down on Web482

Posted in Downtime by

The shared MySQL service on Web482 is currently down. We’re working to restore service at this time. We’ll update this post when more information is available.

Update 13:40 UTC: MySQL is now back to normal. The problem happened when a faulty disk caused the RAID array to become degraded and a MySQL database became corrupted. We’ve repaired the database and swapped the faulty disk and MySQL is now functioning normally.

-
-

[Done]Scheduled maintenance on webmail.webfaction.com, February 10th, 2016.

Posted in Scheduled downtime by

webmail.webfaction.com will be taken down for upgrade on Wednesday, February 10th 2016 between 08:00 UTC and 12:00 UTC. The maintenance should be less than 2 hours. We will update this post as maintenance progresses.

2016-02-10 08:46 UTC The upgrade has been completed.

-
-

[Fixed] Web506 inaccessible

Posted in Downtime by

The server is currently inaccessible. We are investigating the issue.

2016-02-04 06:51 UTC: Apologies for the delay, we’ve been working directly with the datacenter technicians to determine the cause and work towards restoring service to the server.

2016-02-04 07:01 UTC: Services have been restored to normal capacity, Web506 is online. It experienced a hardware issue related to the upstream network, which has been repaired. Please open a support ticket if there are any unresolved issues and we will look into it right away. Thank you for your patience.

2016-02-04 07:41 UTC: There appears to be some re-occurring network disturbances, so we’re re-opening this issue while we take further steps with the datacenter towards  a resolution. Thank you for your patience.

2016-02-04 10:40 UTC: Our datacenter engineers are still working to resolve the network issue. We will update this post as soon as new information arrives.

2016-02-04 11:40 UTC: The network connectivity has been restored through an alternative network interface.  We are still working to get the primary interface online, but things are stable at the moment.

2016-02-04 13:00 UTC: All issues with the network have been resolved.

2016-02-04 18:01 UTC: Web506 has lost connectivity again. We’re working to restore service and will update this post as soon as more information is available.

2016-02-04 20:11 UTC: Our data center team suspects motherboard failure. We’re scheduling a chassis swap at this time.

2016-02-04 20:27 UTC: We’re taking the server down for a chassis swap at this time.

2016-02-05 00:34 UTC: Apologies for the delays. Our datacenter team ran into some issues re-configuring the network interfaces for the new chassis. They are working to get the server up as quickly as possible.

2016-02-05 02:22 UTC: Public access has been restored, though please stand by for further updates confirming resolution on the situation.

2016-02-05 05:18 UTC: The server has been stable for a couple hours now, we’re marking the issue as resolved but will monitor the situation. If you experience any issues you believe to be unresolved, please open a support ticket and we will look into it right away. Thank you again for your patience.

2016-02-05 06:31 UTC: There has been a detectable outage on the server again, we’re once more in communication with our datacenter. Please stand by for further updates.

2016-02-05 09:44 UTC: Currently, the connectivity has been restored, but our data center engineers are still working to resolve the issue permanently. Please stand by for further updates.

2016-02-05 13:00  UTC: The connectivity has been stable in the past hours. We are still monitoring the server closely and working with the data center team towards a permanent resolution.

2016-02-05 14:00  UTC: The network connectivity issue has been resolved permanently.

-
-

[Fixed] Web489 inaccessible

Posted in Downtime by

The server is currently inaccessible. We are investigating the issue.

2016-02-03 23:27 UTC: Web489 is online, we’re still monitoring the server. Please stand by for further updates.

2016-02-04 01:10 UTC: Web489 has been stable for some time now. Please open a support ticket if there are any other issues.

 

-
-

Data center maintenance

Posted in Scheduled downtime by

Our upstream provider is performing maintenance in their Manassas data center on February 16th between 02:00 and 04:00 UTC.

The following servers will be unavailable for 30 minutes.

web24, web342, web343, web229, mailbox8, dweb129, dweb130, web227, web24, web25, dweb135

 

-
-