[Done] Scheduled maintenance on multiple servers (18,19,20 Dec 2012)

Posted in Scheduled downtime by

[PLEASE NOTE: There has been an error with the date for the migration for the 21st – this is actually happening on the 20th.  We apologise for any inconvenience and are working to get these servers back online ASAP]

Our upstream service provider will be performing maintenance to their infrastructure. This maintenance includes moving servers from the current facility to a new state-of-the-art facility. Apart from the servers being physically moved nothing else will change (in particular the server’s IP addresses will remain the same).
Servers will be powered off before the move and powered back on after the move. The anticipated downtime for each server is between 30 and 60 minutes.

We will update this blog post as maintenance progresses.

Here is the list of servers that will be affected:

18 Dec 2012 between 04:00 and 12:00 UTC:

  • Web5
  • Web11
  • Web12
  • Web15
  • Web24
  • Web25
  • Web30
  • Web31
  • Web34
  • Web35
  • Web37
  • Web39
  • Web40
  • Web42
  • Web44
  • Web48
  • Web49
  • Web55
  • Web57
  • Web69
  • Web72
  • Web75
  • Web80
  • Web83
  • Web91
  • Web95
  • Web99
  • Web102
  • Web105
  • Web106
  • Web108
  • Web117
  • Web119
  • Web126
  • Web143
  • Web148
  • Web151
  • Web155
  • Web162
  • Web175
  • Web178
  • Web180
  • Web182
  • Web183
  • Web186
  • Web187
  • Web198
  • Web219
  • Web220
  • Web227
  • Web228
  • Web229
  • Web230
  • Web231
  • Web243
  • Web318
  • Web319
  • Web320
  • Web324
  • Web335
  • Web336
  • Web337
  • Web338
  • Web341
  • Web342
  • Web343
  • Web344

19 Dec 2012 between 04:00 and 12:00 UTC:

  • Web4
  • Web27
  • Web28
  • Web65
  • Web70
  • Web74
  • Web114
  • Web122
  • Web129
  • Web174
  • Web199
  • Web200
  • Web213
  • Web226
  • Web232
  • Web233
  • Web234
  • Web235
  • Web236
  • Web237
  • Web238
  • Web239
  • Web240
  • Web241
  • Web244
  • Web300
  • Web301
  • Web302
  • Web307
  • Web308
  • Web309
  • Web310
  • Web311
  • Web312
  • Web313
  • Web328
  • Web329
  • Web330

21 20 Dec 2012 between 04:00 and 12:00 UTC:

  • Web245
  • Web246
  • Web247
  • Web345
  • Web346
  • Web347
  • Web348
  • Web349

[Fixed] Intermittent network routing issues affecting several US servers

Posted in Problems by

Several of our US servers are currently experiencing intermittent network routing issues. Affected servers may include:

Dweb100, Dweb101, Dweb102, Dweb104, Dweb110, Dweb111, Dweb112, Dweb113, Dweb114, Dweb115, Dweb116, Dweb117 ,Dweb118 ,Dweb119, Dweb121 ,Dweb122, Dweb123, Dweb124, web125, Dweb126, Dweb127, Dweb128, Dweb129, Dweb130, Dweb133, Dweb134, Dweb135, Dweb137, Dweb140, Dweb141, Dweb142, Dweb143, Dweb144, Dweb145, Dweb146, Dweb147, web149, Dweb150, Dweb151, Dweb152, Dweb153, Dweb154, Dweb158,  Dweb160, Dweb161, Dweb162, Dweb163, Dweb164, Dweb165, Dweb94, Dweb95, Dweb96, Dweb97, Web4, Web5, Web11, Web12, Web15, Web24, Web25, Web27, Web28, Web30, Web31, Web34, Web35, Web37, Web39, Web40, Web42, Web48, Web49, Web55, Web57, Web65, Web69, Web70, Web72, Web74, Web75, Web80, Web83, Web91, Web95, Web99, Web102, Web105, Web106, Web108, Web114, Web117, Web119, Web122, Web126, Web129, Web143, Web148, Web151, Web155, Web162, Web17, Web175, Web178, Web180, Web182, Web183, Web186, Web187, Web198, Web199, Web200, Web213, Web219, Web220, Web226, Web227, Web228, Web229, Web230, Web231, Web232, Web33, Web234, Web235, Web236, Web237, Web238, Web239, Web240, Web241, Web243, Web244, Web245, Web246, Web247, Web300, Web301, Web302, Web307, Web308, Web309, Web310, Web311, Web312 ,Web313, Web318, Web319, Web320, Web324, Web328, Web329, Web330, Web335, Web336, Web337, Web338, Web341, Web342, Web343, Web344, Web345, Web346, Web3,7, Web348, Web349

2012-11-26 05:18 UTC: Engineers are onsite replacing the faulty hardware.

2012-11-26 05:22 UTC: Note that the WebFaction control panel is also being intermittently affected by this problem.

2012-11-26 05:55 UTC: The faulty hardware has been replaced and all services are coming back online.  We are monitoring closely to ensure there are no further issues.

2012-11-26 06:55 UTC: Systems appear stable, so we are declaring this issue resolved.


[Done]Scheduled maintenance on Web332, Saturday 24 November 2012.

Posted in Downtime by

Web332 will be taken offline at approximately 23:00 UTC for a routine RAID firmware upgrade. We don’t anticipate the maintenance will last longer than 1 hour and we’ll keep this post updated as the maintenance progresses.

2012-11-24 22:12 UTC: The server has now gone down for maintenance. We’re tracking the progress and will update as we know more.

2012-11-24 23:11 UTC: The upgrade of the RAID firmware has failed. Currently the server is back online and working, however, we are preparing to do a chassis swap in order to have an up to date RAID firmware. This will insure that no data is lost in the event that the RAID card on this machine begins to fail. We will update this post when the server goes offline for the replacement.

2012-11-25 00:41 UTC: We are now taking the machine offline to replace the chassis.

2012-11-25 02:08 UTC: The machine is now back on line with a new chassis and things are working normally. The RAID firmware has been confirmed to be upgraded to the latest version and things are stable.


[Done]Emergency maintenance on Web28, 14th November, 2012.

Posted in Downtime by

Web28 filesystem went in R/O mode. We’re taking down the server to run a fsck. We will update this post as maintenance progresses.

2012-11-14 16:32 UTC the fsck finished execution. The server is back at operational status.


[Done]Scheduled maintenance on Web364, November 14th, 2012.

Posted in Scheduled downtime by

Web364 will be taken down for a firmware upgrade Wednesday November 14th betweeen 10:00 UTC and 11:00 UTC. We will update this post as maintenance progresses.

2012-11-14 10:43 UTC The server is back at operational status.


[Fixed] Mailbox3 down

Posted in Downtime by

Mailbox3 is currently unresponsive, so customers with mailboxes on that server will be unable to connect.  We are investigating as a matter of urgency.

2012-11-08 02:03 UTC: Mailbox3 is back online.


[Done]Emergency maintenance on Web326 Nov 7th, 2012.

Posted in Downtime by

Web326 became unresponsive. We’re currently investigating and will update this post when we have more information.

2012-11-07 13:03 UTC We fixed the issue and the server is back to operational status.

2012-11-07 14:05 Filesystem went read only, we’re currently running an fsck on it. We will update this post when we have more information.

2012-11-07 15:11 UTC fsck finished and the server is back at normal status.


[Done] Emergency maintenance on Web224, November 1st, 2012.

Posted in Downtime by

Web224 went down unexpectedly, we’re currently investigating and we will report progresses updating this post.

2012-11-01 13:21 UTC: We’re running fsck on damaged partitions now.

2012-11-01 15:04 UTC: We are replacing a disk on server and reloading OS.

2012-11-01 17:00 UTC: We reloaded OS on server and we’re currently setting it up.

2012-11-01 19:42 UTC: Web224 is back online. The customer /home partition was not affected by the disk issues, but we did need to restore MySQL databases, PostgreSQL databases, and cron jobs from our backups. If you notice any problems with these items, please open a support ticket so that we can look into it.

2012-11-01 20:01 UTC: We’ve received several reports of missing MySQL databases, so we are currently restoring those again.

2012-11-01 20:06 UTC: We restored all missing MySQL databases.