[Done] Emergency Maintenance on Web178

Posted in Downtime by

The filesystem on Web178 suddenly went read-only. We are booting the server into rescue mode to run a filesystem check.  We will update this post as maintenance progresses.

Update 16:06 UTC: The filesystem has some corruption so we are migrating the server to new hardware. Note that the main IP address will change from 108.59.4.70 to 75.126.149.9 so if you are using non-DNS servers for some of your domains you will have to update them to point your domains at the new IP.

Update 17:12 UTC: All MySql and PostgreSQL databases have been migrated to the new server as well as the first few home directories. All services have been started on the new server. We are working on migrating the rest of the home directories.

Update 22:10 UTC: All home directories starting with the letter a or b have been migrated. We’re working on the rest of the home directories.

Update 06:00 UTC: All home directories up to the letter p have been migrated. We’re working on the rest of the home directories.

Update 16:47 UTC: All home directories up to the letter t have been migrated. We’re working on the rest of the home directories.

Update 08:17 UTC: The migration is now finished. If you notice any problems with your account open a ticket and our support team will look into it asap.

-
-

[Done]Emergency maintenance on Web178, 26 February 2013

Posted in Downtime by

Web178 filesystem went read only, we’re about to take the server down to run a fsck on it. We will update this post as maintenance progresses.

2013-02-26 13:20 UTC: FSCK is now running, progress is at 22%

2013-02-26 13:47 UTC: FSCK is at 50%

2013-02-26 14:54 UTC: FSCK completed and the server is back to operational status.

2013-02-26 15:22 UTC: Filesystem went read only again, we’re currently investigating the issue and will report progresses on this post.

2013-02-26 16:07 UTC: The server is back at operational status.

2013-02-26 17:14 UTC: The filesystem has gone read-only again. We’re working to restore service and will update this post when we have more information.

2013-02-26 18:15 UTC: We are now running fsck on the machine again and it is at 57%.

2013-02-26 20:03 UTC: After the initial fsck finished the file system is still not returning a clean file system from our checks so we are running further file system checks.

2013-02-26 21:51 UTC: Our further checks have shown some filesystem inodes that were multiply referenced and need to be manually removed. We are working on this now.

2013-02-26 23:53 UTC: The inodes have been removed and we’re now running a final fsck on the file system to insure it is clean of all file system errors before booting.

2013-02-27 00:07 UTC: The server is now back online and functioning normally. We’ll continue to watch the server and its filesystem very closely to insure that all errors were resolved.

-
-

[Fixed] Intermittent network routing issues affecting several US servers

Posted in Problems by

Several of our US servers are currently experiencing intermittent network routing issues. Affected servers may include: Dweb100 Dweb101 Dweb102 Dweb104 Dweb105 Dweb110 Dweb111 Dweb112 Dweb113 Dweb114 Dweb115 Dweb116 Dweb117 Dweb118 Dweb119 Dweb120 Dweb121 Dweb122 Dweb123 Dweb124 Dweb125 Dweb126 Dweb127 Dweb128 Dweb129 Dweb130 Dweb133 Dweb134 Dweb135 Dweb137 Dweb140 Dweb141 Dweb142 Dweb143 Dweb144 Dweb145 Dweb146 Dweb147 Dweb149 Dweb150 Dweb151 Dweb152 Dweb153 Dweb154 Dweb158 Dweb160 Dweb161 Dweb162 Dweb163 Dweb164 Dweb91 Dweb92 Dweb93 Dweb94 Dweb95 Dweb96 Dweb97 Mailbox8 Web102 Web105 Web106 Web108 Web11 Web114 Web117 Web119 Web12 Web122 Web126 Web129 Web143 Web148 Web15 Web151 Web155 Web162 Web174 Web175 Web178 Web180 Web182 Web183 Web186 Web187 Web198 Web199 Web200 Web213 Web219 Web220 Web226 Web227 Web228 Web229 Web230 Web231 Web232 Web233 Web234 Web235 Web236 Web237 Web238 Web239 Web24 Web240 Web241 Web243 Web244 Web245 Web246 Web247 Web25 Web27 Web28 Web30 Web300 Web301 Web302 Web307 Web308 Web309 Web31 Web310 Web311 Web312 Web313 Web318 Web319 Web320 Web324 Web328 Web329 Web330 Web335 Web336 Web337 Web338 Web34 Web341 Web342 Web343 Web344 Web345 Web346 Web347 Web348 Web349 Web35 Web37 Web39 Web4 Web40 Web42 Web48 Web49 Web5 Web55 Web57 Web65 Web69 Web70 Web72 Web74 Web75 Web80 Web83 Web91 Web95 Web99

We’re working to resolve this issue and will update this post when we have more information.

2012-10-15 6:07 UTC: The problem was an issue with an upstream network carrier and has been resolved.

-
-

[Fixed]Web178 down

Posted in Downtime by

The filesystem on Web178 went into a read-only state several minutes ago. We rebooted the server but it has not come back up yet. We’re investigating this issue now and will update this post when we have more information.

2011-06-10 20:24 UTC – A drive has failed on Web178. The drive is currently being replaced.

2011-06-10 21:24 UTC – The drive has been replaced. A filesystem check is currently running.

2011-06-10 21:49 UTC – The filesystem check is still in progress.

2011-06-11 00:30 UTC – The filesystem check is still in progress. Web178 has a 2TB disk, so the check will take a considerable amount of time to complete.

2011-06-11 02:30 UTC – The filesystem check is still in progress.

2011-06-11 04:45 UTC – The filesystem check has finished and the server is back online and responding to requests.

-
-

[fixed] Web178 offline

Posted in Downtime by

The filesystem on Web178 went into a read-only state several minutes ago. The server has failed to come back online after being rebooted.

We’re currently investigating this issue and hope to have service restored shortly. We will update this post with more information as it becomes available.

2011-04-01 20:55 UTC – Web178 is back online. The downtime was due to the server prompting for a manual filesystem check after it was rebooted. We’ve bypassed that for now and will schedule a full filesystem check in the near future.

-
-