[Fixed] Emergency filesystem check on web237 since it went read-only

Posted in Downtime by

Web237’s disk went read-only shortly after the scheduled maintenance so we have taken it into rescue for  an immediate fsck.

We will update this post as the fsck progresses.

[2012-12-19 10:40 UTC] The fsck is upto 10% now.

[2012-12-19 13:06 UTC] The fsck is still running

[2012-12-19 15:58 UTC] fsck is at pass 1C.

[2012-12-19 18:12 UTC] Unfortunately fsck failed to complete. At this point we are going to copy the data to a spare server. We will update this post as the copy progresses.

[2012-12-19 20:06 UTC] The copy of the data to a new server is at 15%

[2012-12-19 21:18 UTC] The copy of the data to a new server is at 25%

[2012-12-19 22:59 UTC] The copy of the data to a new server is still ongoing

[2012-12-20 00:50 UTC] The copy of the data to a new server is at 35%

2012-12-20 02:26 UTC: The copy of the data to a new server is at 45%

2012-12-20 03:14 UTC: The copy of the data to a new server is at 50%

2012-12-20 05:02 UTC: The copy of the data to a new server is at 60%

2012-12-20 09:38 UTC: The copy of the data is still going on.

2012-12-20 10:35 UTC: After hanging for a while the copy is now going at full speed again. It’s at 70%

2012-12-20 11:33 UTC: We are going to start re-enabling websites for accounts whose data has been copied onto the new server. If you would like your websites to be re-enabled as a higher priority open a support ticket.

2012-12-20 13:37 UTC: Some sites are re-enabled on the new Web237 machine and we’re working through the other ones. Please note that the machine’s main IP changed from 108.59.11.111 to 75.126.24.82 so if you’re using external DNS servers you will have to update your DNS records. If you’re using WebFaction’s DNS servers you don’t need to do anything.

2012-12-20 16:49 UTC: Most websites have new been re-enabled. We’re working on re-enabling the last few ones.

2012-12-20 22:19 UTC: All websites have now been re-enabled for several hours. We’re marking this issue as fixed and we’ll email a post-mortem to all Web237 users tomorrow.

-
-

[Fixed] Intermittent network routing issues affecting several US servers

Posted in Problems by

Several of our US servers are currently experiencing intermittent network routing issues. Affected servers may include: Dweb100 Dweb101 Dweb102 Dweb104 Dweb105 Dweb110 Dweb111 Dweb112 Dweb113 Dweb114 Dweb115 Dweb116 Dweb117 Dweb118 Dweb119 Dweb120 Dweb121 Dweb122 Dweb123 Dweb124 Dweb125 Dweb126 Dweb127 Dweb128 Dweb129 Dweb130 Dweb133 Dweb134 Dweb135 Dweb137 Dweb140 Dweb141 Dweb142 Dweb143 Dweb144 Dweb145 Dweb146 Dweb147 Dweb149 Dweb150 Dweb151 Dweb152 Dweb153 Dweb154 Dweb158 Dweb160 Dweb161 Dweb162 Dweb163 Dweb164 Dweb91 Dweb92 Dweb93 Dweb94 Dweb95 Dweb96 Dweb97 Mailbox8 Web102 Web105 Web106 Web108 Web11 Web114 Web117 Web119 Web12 Web122 Web126 Web129 Web143 Web148 Web15 Web151 Web155 Web162 Web174 Web175 Web178 Web180 Web182 Web183 Web186 Web187 Web198 Web199 Web200 Web213 Web219 Web220 Web226 Web227 Web228 Web229 Web230 Web231 Web232 Web233 Web234 Web235 Web236 Web237 Web238 Web239 Web24 Web240 Web241 Web243 Web244 Web245 Web246 Web247 Web25 Web27 Web28 Web30 Web300 Web301 Web302 Web307 Web308 Web309 Web31 Web310 Web311 Web312 Web313 Web318 Web319 Web320 Web324 Web328 Web329 Web330 Web335 Web336 Web337 Web338 Web34 Web341 Web342 Web343 Web344 Web345 Web346 Web347 Web348 Web349 Web35 Web37 Web39 Web4 Web40 Web42 Web48 Web49 Web5 Web55 Web57 Web65 Web69 Web70 Web72 Web74 Web75 Web80 Web83 Web91 Web95 Web99

We’re working to resolve this issue and will update this post when we have more information.

2012-10-15 6:07 UTC: The problem was an issue with an upstream network carrier and has been resolved.

-
-

[Fixed]Emergency maintenance on Web237, July 14th 2012.

Posted in Downtime by

Web237 is currently down for emergency maintenance. We’ll continue to update this post with additional information as it is available.

[Wed Jul  14 19:08:33 UTC 2012] We are currently running fsck and it is at 63% now.

[Wed Jul 14 23.16 UTC 2012] The server is now back online. If you experience any problem with your website open a support ticket and we’ll look into it asap.

-
-