[Fixed] Multiple US servers offline

Posted in Problems by

2017-05-17 22:01 UTC: Several US shared and cloud servers in our St. Louis data center are currently offline We’ll update this post as soon as more info is available.

2017-05-17 22:07 UTC: The following servers are affected:

  • Shared: web537, web538, web539, web540, web541, web542, web543, web544, web545, web546, web547, web548, web549, web550, web551, web552, web553, web554, web555, web556, web557, web559, web560, web563, web590, web591, web592, web593, web594, web595, web596, web597, web598
  • Cloud: wf-207-38-86-39, wf-207-38-86-40, wf-207-38-86-43, wf-207-38-86-47, wf-207-38-86-48, wf-207-38-86-49, wf-207-38-86-73, wf-207-38-86-75, wf-207-38-86-76, wf-207-38-86-78, wf-207-38-86-79, wf-207-38-86-85, wf-207-38-86-88, wf-207-38-86-90, wf-207-38-86-92, wf-207-38-86-93, wf-207-38-86-94, wf-207-38-86-98, wf-207-38-86-102, wf-207-38-86-103, wf-207-38-86-104, wf-207-38-86-105, wf-207-38-86-106, wf-207-38-86-107, wf-207-38-92-110, wf-207-38-86-110, wf-207-38-86-111, wf-207-38-86-112, wf-207-38-86-114, wf-207-38-86-115, wf-207-38-86-116, wf-207-38-86-117, wf-207-38-86-118, wf-207-38-86-120, wf-207-38-86-121, wf-207-38-86-122, wf-207-38-86-123, wf-207-38-86-124, wf-207-38-86-125, wf-207-38-86-126, wf-207-38-86-127, wf-207-38-86-128, wf-207-38-86-129, wf-207-38-86-130, wf-207-38-86-131, wf-207-38-86-132, wf-207-38-86-134, wf-207-38-86-137, wf-207-38-86-139, wf-207-38-86-77, wf-207-38-86-135, wf-207-38-86-138, wf-207-38-86-140, wf-207-38-86-145, wf-207-38-86-147, wf-207-38-86-149, wf-207-38-86-150, wf-207-38-86-159, wf-207-38-86-160, wf-207-38-86-164, wf-207-38-86-165, wf-207-38-86-166, wf-207-38-86-167, wf-207-38-86-168, wf-207-38-86-170, wf-207-38-86-171, wf-207-38-86-175, wf-207-38-86-178, wf-207-38-86-180, wf-207-38-86-191, wf-207-38-86-193, wf-207-38-86-194, wf-207-38-86-195, wf-207-38-86-196, wf-207-38-86-197, wf-207-38-86-198, wf-207-38-86-199, wf-207-38-92-219, wf-207-38-92-220, wf-207-38-92-221, wf-207-38-92-222, wf-207-38-92-223, wf-207-38-92-224

2017-05-17 23:09 UTC: We’re still working with the data center to identify and resolve this issue. We’ll update this post as soon as more information is available.

2017-05-17 23:20 UTC: web537, web538, web539, web540, web541, web542, web543, web544, web545, web546, web547, web548, web549, web550, web551, web552, web553, web554, web555, web556, web557, web559, web560, web563, web590, web591, web592, web593, web594, web595, web596, web597, web598 are all back online.

2017-05-17 23:40 UTC: All US servers are back online at this time. We will continue to monitor and update this post if there are further developments.

-
-

Network maintenance on May 30th – minimal downtime expected

Posted in Maintenance by

Our upstream provider will be performing emergency maintenance one of their routers on May 30th starting at 08:00 UTC.

A three hour maintenance window has been scheduled. The total expected downtime should not exceed three minutes.

The affected servers are:

Mailbox12, Mailbox13, Mailbox14, Mailbox15, Mailbox16

 

-
-

[fixed] web513 inaccessible

Posted in Downtime by

The server is currently not accessible. We are looking into this and will update as soon as we have further details.

2017-05-15 14:59 UTC: We are still investigating the issue.

2017-05-15 17:20 UTC: Our datacenter technicians are working directly to power on the server.

2017-05-15 20:17 UTC: The server is accessible again.

-
-

Network maintenance on May 22, 24, and 25 – minimal downtime expected

Posted in Maintenance by

Our upstream provider will be performing emergency maintenance on three separate routers on May 22, 24, and 25.

A three hour maintenance window has been scheduled for each router. The total expected downtime within each window should not exceed three minutes.

The maintenance windows and affected servers are:

  • 22-May-2017 22:00 UTC through 23-May-2017 01:00 UTC: dweb186, web209, web210, web211, web217, web218, web223, web224, web225, web303, web304, web305, web306, web314, web316, web317, web327, web331, web332, web333, web334, web357, web358, web359, web364, web383, web392, web395, web416, web421, web435, web448, web457, web459, web464, web480, web483, web488
  • 24-May-2017 05:00 UTC through 08:00 UTC: ns1, web22, web23, web39, web40, web69, web70, web72, web74, web75, web119, web122, web183, web200, web213, web228, web230, web231, web233, web234, web235, web242, web248, web249, web338, web391, web397, web413, web453, web484, web487
  • 25-May-2017 05:00 UTC through 08:00 UTC: mail, mx7, mx8, web31, web42, web48, web155, web174, web178, web186, web226, web237, web241, web351, web354, web355, web356, web363, web365, web366, web367, web372, web378, web382, web387, web408, web450, web492, webmail
-
-

[Fixed] Web457 offline

Posted in Downtime by

2017-05-05 15:28 UTC: Web457 is offline due to an IP block from our upstream provider. We’re working with them to restore service and will update this post when more info is available.

2017-05-05 15:37 UTC: The IP block is lifted and Web457 is back online.

-
-

[Done] Emergency network maintenance affecting several EU servers

Posted in Downtime by

2017-04-27 00:20 UTC: At the present time, emergency network maintenance in our EU datacenter has caused several EU servers to go offline. We’ll update this post as more information becomes available.

2017-04-27 00:29 UTC: Connectivity has been restored, we’ll continue to monitor.

-
-

[Fixed] web567 inaccessible

Posted in Downtime by

The server is currently not accessible. We are looking into this and will update as soon as we have further details.

2017-04-26 00:04 UTC: The servers is back online.

-
-

[Fixed] Web560 inaccessible

Posted in Downtime by

2017-04-25 1700 UTC: We are investigating the issue and will update the post as soon as we have further details.

2017-04-25 1730 UTC: The server is back online and serving traffic.

-
-

[Fixed] Web474 inaccessible

Posted in Downtime by

The server is currently not accessible. We are looking into this and will update as soon as we have more details.

2017-04-21 17:37 UTC: The servers is back online.

-
-

[Done] Emergency maintenance on Mailbox10, April 16, 2017

Posted in Downtime by

Mailbox10 will be taken down for an HDD swap and possibly RAID controller replacement,
Sunday April 16th, 2017 between 1:00 UTC and 5:00 UTC.

Sun Apr 16 01:15:32 UTC 2017: The server has been taken offline for maintenance

Sun Apr 16 02:15:32 UTC 2017: The failing drive has been replaced and the RAID is rebuilding (currently 3%).

Sun Apr 16 02:45:34 UTC 2017: The rebuild has failed, which indicates a faulty RAID controller, which is now being replaced.

Sun Apr 16 03:29:52 UTC 2017: The RAID controller has been replaced; now doing build/verify and rebuild.

Sun Apr 16 06:22:50 UTC 2017: The RAID controller initial build/verify is done and new HDD is rebuilding (currently 15%).

Sun Apr 16 07:00:51 UTC 2017: HDD is rebuilding (currently 35%).

Sun Apr 16 08:22:08 UTC 2017: HDD is rebuilding (currently 80%).

Sun Apr 16 08:52:54 UTC 2017: The rebuild has completed and the server is back online.

-
-