[Complete]Scheduled Web48 migration

Posted in Downtime by

As scheduled we are starting the migration of Web48 to new hardware. During the migration services on the machine will be unavailable. We will update this post once the migration is over.

2012-02-17 16:10 UTC: The migration is finished and the server is back online.

-
-

[Complete]Scheduled Web91 migration

Posted in Downtime by

As scheduled we are starting the migration of Web91 to new hardware. During the migration services on the machine will be unavailable. We will update this post once the migration is over.

2012-02-16 17:41 UTC: The migration is finished and the server is back online and fully functional.

-
-

[Complete]Scheduled Web83 migration

Posted in Downtime by

As scheduled we are starting the migration of Web83 to new hardware. During the migration services on the machine will be unavailable. We will update this post once the migration is over.

2012-02-16 16:05 UTC: The migration has finished and the server is now back online.

-
-

[Complete]Scheduled Web42 migration

Posted in Downtime by

As scheduled we are starting the migration of Web42 to new hardware. During the migration services on the machine will be unavailable. We will update this post once the migration is over.

2012-02-15 18:04 UTC: We’ve found an irregularity in the file system on the old server that has slowed down the migration considerably. We may run over the scheduled time window but we will update this post as more information is available.

2012-02-15 20:32 UTC: We are now past the scheduled migration window and the migration is still going. This post will be updated when we have more information.

2012-02-15 22:25 UTC: The migration process is still running. The file system errors seemed to have slowed the migration process more than we anticipated. We are monitoring the process carefully to insure that there is no data loss.

2012-02-15 01:17 UTC: The migration process is still running.

2012-02-15 02:25 UTC: The user data has been migrated to the new server and we’re verifying it’s integrity now.

2012-02-15 03:05 UTC: User data has been verified and we’re now processing the MySQL and PostgreSQL databases.

2012-02-15 04:09 UTC: We are currently verifying the integrity of the databases.

2012-02-15 04:26 UTC: The server is now back online and responding normally. We will continue to monitor the server closely to verify the file system problems on the old server have no effect on the new server.

2012-02-16 14:11 UTC: The server hasn’t shown any cause for concern. We’ll continue to monitor it normally.

-
-

[Complete]Scheduled Web95 migration

Posted in Downtime by

As scheduled we are starting the migration of Web95 to new hardware. During the migration services on the machine will be unavailable. We will update this post once the migration is over.

2012-02-15 19:08 UTC: We are currently at the end of the scheduled migration window but the server is not yet online. We will update this post as the migration progresses.

2012-02-15 20:01 UTC: Web95 is back online and working normally.

-
-

[Done] Reboots on various servers on February 14th 2012

Posted in Downtime by

On February 14th we will be rebooting the following servers for routine kernel updates at various times throughout the day:

  • web38
  • web44
  • web46
  • web56
  • web101
  • web103
  • web122
  • web127
  • web130
  • web134
  • web142
  • web154
  • web164
  • web165
  • web175
  • web182
  • web196
  • web226

Downtime on each server is expected to be less than 20 minutes. We will update this post as maintenance progresses.

2012-02-14 08:30 UTC: All servers have been rebooted and are back online.

-
-

[Done] Reboots on various servers on February 13th 2012

Posted in Downtime by

On February 13th we will be rebooting the following servers for routine kernel updates at various times throughout the day:

  • web40
  • web45
  • web64
  • web66
  • web76
  • web108
  • web110
  • web124
  • web131
  • web135
  • web136
  • web139
  • web141
  • web144
  • web148
  • web149
  • web150
  • web158
  • web185
  • web207
  • web214
  • web216
  • web217
  • web218
  • web219
  • web220
  • web221
  • web223
  • web224
  • web225

Downtime on each server is expected to be less than 20 minutes. We will update this post as maintenance progresses.

2012-02-13 10:00 UTC All servers except web216 are back online. We’re working on bringing web216 back now.
2012-02-13 10:10 UTC Web216 has suffered a hardware failure. The failing components are being replaced.
2012-02-13 10:15 UTC Web216 is back online.

-
-

[Done] Reboots on various servers on February 12th 2012

Posted in Downtime by

On February 12th we will be rebooting the following servers for routine kernel updates at various times throughout the day:

  • web31
  • web43
  • web61
  • web68
  • web69
  • web78
  • web79
  • web80
  • web81
  • web83
  • web96
  • web98
  • web104
  • web105
  • web107
  • web118
  • web133
  • web137
  • web162
  • web172
  • web179
  • web188
  • web194
  • web199
  • web201
  • web202
  • web203
  • web204
  • web206
  • web208
  • web209
  • web210
  • web211
  • web212
  • web215

Downtime on each server is expected to be less than 20 minutes. We will update this post as maintenance progresses.

2012-02-12 10:15 UTC All servers except web215 are back online. We’re working on bringing web215 back now.
2012-02-12 11:45 UTC web215 is back online.

-
-

[Done]Scheduled maintenance on Web119, Fabruary 10th.

Posted in Scheduled downtime by

Web119 will be taken offline for a RAID firmware upgrade at 12:00 UTC Friday 10th February 2012. We’ll update this post as maintenance progresses.

2012-02-10 13:25 UTC The firmware has been upgraded but RAID status is still degraded. We’re looking into this issue and possibly swap the RAID card if necessary.
2012-02-10 15:47 UTC RAID card replaced and resyncing. The server is back at operational status.

-
-

[Done]Scheduled maintenance on Web168, February 10th

Posted in Scheduled downtime by

Web168 will be taken offline for a disk replacement at Friday 10th, February 2012 at 11:00 UTC for about 1 hour. We’ll update this post as maintenance progresses.

2012-02-10 12.33 UTC The disk has been replaced and currently rebuilding. The server is back to operational status.

-
-