We are currently investigating an issue following an upgrade to CentOS-5.3. Services are down on several servers and we are currently working on restoring them. The servers are:

  • web23
  • web25
  • web27
  • web31
  • web32
  • web33
  • web34
  • web35
  • web36
  • web38
  • web39
  • web43
  • web46
  • web49
  • web52
  • web53
  • web55
  • web57
  • web58
  • web68
  • web75
  • dweb40

Update 2009-05-06 11:10 CDT: We are in the process of reinstalling some RPMs on the servers in order to bring them back online.

Update 2009-05-06 11.50 CDT: Two servers are back online (web36 and web52). We are continuing to work on the other servers.

Update 2009-05-06 12:39 CDT: Web32, Web35 and Web39 are back online. We are continuing to work on the other servers.

Update 2009-05-06 12:55 CDT: Web52 is down again.

Update 2009-05-06 13:23 CDT: Web25, Web27 and Web58 are back online. We are continuing to work on the other servers.

Update 2009-05-06 13:28 CDT: Web55 and Web68 are back online. We are continuing to work on the other servers.

Update 2009-05-06 14:06 CDT: Web38 and Web57 are back online. We are continuing to work on the other servers.

Update 2009-05-06 14:12 CDT: Web52 is back online. We are continuing to work on the other servers.

Update 2009-05-06 16:11 CDT: Web31, Web34 and Web46 are back online. We are continuing to work on the other servers.

Update 2009-05-06 16:48 CDT: Dweb40 and Web43 are back online. We are continuing to work on the other servers.

Update 2009-05-06 16:55 CDT: Web23 and Web53 are back online. We are continuing to work on the other servers.

Update 2009-05-06 17:04 CDT: Web33 and Web75 are back online. We are continuing to work on Web49 (the last server!).

Update 2009-05-06 17:26 CDT: Web49 is back online – that was the last server affected by the problem, so we’re calling this one fixed. Sorry for the trouble, folks!