Network maintenance on May 30th – minimal downtime expected

Posted in Maintenance by

Our upstream provider will be performing emergency maintenance one of their routers on May 30th starting at 08:00 UTC.

A three hour maintenance window has been scheduled. The total expected downtime should not exceed three minutes.

The affected servers are:

Mailbox12, Mailbox13, Mailbox14, Mailbox15, Mailbox16

 

-
-

Network maintenance on May 22, 24, and 25 – minimal downtime expected

Posted in Maintenance by

Our upstream provider will be performing emergency maintenance on three separate routers on May 22, 24, and 25.

A three hour maintenance window has been scheduled for each router. The total expected downtime within each window should not exceed three minutes.

The maintenance windows and affected servers are:

  • 22-May-2017 22:00 UTC through 23-May-2017 01:00 UTC: dweb186, web209, web210, web211, web217, web218, web223, web224, web225, web303, web304, web305, web306, web314, web316, web317, web327, web331, web332, web333, web334, web357, web358, web359, web364, web383, web392, web395, web416, web421, web435, web448, web457, web459, web464, web480, web483, web488
  • 24-May-2017 05:00 UTC through 08:00 UTC: ns1, web22, web23, web39, web40, web69, web70, web72, web74, web75, web119, web122, web183, web200, web213, web228, web230, web231, web233, web234, web235, web242, web248, web249, web338, web391, web397, web413, web453, web484, web487
  • 25-May-2017 05:00 UTC through 08:00 UTC: mail, mx7, mx8, web31, web42, web48, web155, web174, web178, web186, web226, web237, web241, web351, web354, web355, web356, web363, web365, web366, web367, web372, web378, web382, web387, web408, web450, web492, webmail
-
-

[Done] Emergency network maintenance affecting several EU servers

Posted in Downtime by

2017-04-27 00:20 UTC: At the present time, emergency network maintenance in our EU datacenter has caused several EU servers to go offline. We’ll update this post as more information becomes available.

2017-04-27 00:29 UTC: Connectivity has been restored, we’ll continue to monitor.

-
-

[Done] Emergency maintenance on Mailbox10, April 16, 2017

Posted in Downtime by

Mailbox10 will be taken down for an HDD swap and possibly RAID controller replacement,
Sunday April 16th, 2017 between 1:00 UTC and 5:00 UTC.

Sun Apr 16 01:15:32 UTC 2017: The server has been taken offline for maintenance

Sun Apr 16 02:15:32 UTC 2017: The failing drive has been replaced and the RAID is rebuilding (currently 3%).

Sun Apr 16 02:45:34 UTC 2017: The rebuild has failed, which indicates a faulty RAID controller, which is now being replaced.

Sun Apr 16 03:29:52 UTC 2017: The RAID controller has been replaced; now doing build/verify and rebuild.

Sun Apr 16 06:22:50 UTC 2017: The RAID controller initial build/verify is done and new HDD is rebuilding (currently 15%).

Sun Apr 16 07:00:51 UTC 2017: HDD is rebuilding (currently 35%).

Sun Apr 16 08:22:08 UTC 2017: HDD is rebuilding (currently 80%).

Sun Apr 16 08:52:54 UTC 2017: The rebuild has completed and the server is back online.

-
-

[Completed] Emergency maintenance on web474, April 02, 2017

Posted in Downtime by

Web474 will be taken down for hard disk replacement on Sunday April 2nd, 2017 between 04:00 UTC and 08:00 UTC.

We will update this post as maintenance progresses.

Sun Apr  2 04:18:02 UTC 2017: The server has been taken offline for maintenance.

Sun Apr  2 05:39:44 UTC 2017: The first of two hard disks has been replaced and the RAID is rebuilding (25% currently)

Sun Apr  2 07:34:41 UTC 2017: The RAID has finished rebuilding after the first disk replacement, and the second disk is now being replaced.

Sun Apr  2 07:45:17 UTC 2017: The server is back online. The second disk was replaced and the RAID is rebuilding, which will decrease performance until it finishes.

Sun Apr  2 12:14:17 UTC 2017: All drives in the RAID have finished rebuilding. Maintenance is now complete.

-
-

[Completed] Emergency network maintenance in Singapore

Posted in Maintenance by

Between 16:00 and 18:00 UTC today (24 March 2017) a network module in our Singapore datacenter will be rebooted.

The following servers are serviced by this network module: web321, web322, web323, web339, web360, web361, web362, web375, web379, web390, web409, web422, web423, web424, web429, web438, web442, web463, web474, web486, web490, web505, web508, web517

However, no downtime or loss of connectivity is expected.

We’ll update this post when the maintenance is complete.

2017-03-24 17:07 UTC: The maintenance is complete.

-
-

[Fixed] MySQL issue on Web347

Posted in Maintenance by

MySQL is down on web347 while we run a repair operation. We’ll update this post once we have more information.

2016-01-04 18:14 UTC MySQL is now operational and the server is online.

-
-

[Complete] Control panel and API maintenance reboots

Posted in Maintenance by

2016-12-05 17:13 UTC: We will be rebooting our control panel and API services in the next hour to accomplish some maintenance tasks. The services should be down for know more than a few minutes. We’ll update this post when the maintenance is complete.

2016-12-05 18:51 UTC: The maintenance is complete.

-
-

[Done] Reboots on Centos 6 servers

Posted in Maintenance by

Over the next few days we are going to reboot all of our CentOS 6 servers to pick up important updates.

-
-

[Done] Memory swap on web509, November 4th 2016

Posted in Maintenance by

Web509 will be brought down to replace some faulty memory modules on November 4th between 2PM and 5PM UTC. Expected downtime is less than 30 minutes.

We will update this post as maintenance will progress.

 

14:01 The maintenance has begun

14:11 The maintenance is complete and the server is back online

-
-