[Resolved] Networking Problem
Updated Sep 2 at 10:20 CDT (first posted Sep 1 at 13:55 CDT) by David Sissitka in Downtime - 0 comment(s)
The Planet appears to be having a networking problem that's making some servers inaccessible to some people. We've contacted The Planet re the problem and we'll post updates as we receive them.
Update (02:07 PM CDT): The Planet is currently working on their DNS servers to resolve an issue with intermittent nameserver resolution. They didn't have an ETA as to when this would be done for us.
Read the full article and comments
PHP setup update (done)
Updated Aug 15 at 02:37 CDT (first posted Aug 12 at 07:47 CDT) by Remi in General - 0 comment(s)
In the next few hours we're going to continue our improvements to our PHP setup:
- Fix for remote_addr: We're rolling out a patch that fixes the remote_addr value available to PHP scripts.
- New PHP setup on all servers: We're rolling out the new PHP setup to all remaining servers (dweb1 to dweb17, web1 to web20, krait and mamba).
We'll update this post when these updates are finished.
Update: All servers are now running the new PHP setup. We're going to roll out the remote_addr patch now.
Update: All servers now have the remote_addr patch.
Read the full article and comments
Web4 rebooting (Fixed)
Posted Aug 6 at 08:20 CDT by Sean in Downtime, Problems - 0 comment(s)
The filesystem on Web4 went read-only a few moments ago. We are rebooting that server now and hope to have normal service restored soon.
Update 2008-08-06 08:33: Web4 is back online.
Read the full article and comments
Help system and Mail3 down (fixed)
Posted Aug 4 at 11:56 CDT by Sean in Downtime, Problems - 0 comment(s)
The help system and mail3 are currently down. We're currently rebooting that server and hope to have normal service restored soon.
Update: 2008-08-04 12:01 - Mail3 is back online.
Update: 2008-08-04 15:22 - IMAP and SSH stopped responding on mail3, so we had to reboot it once more.
Read the full article and comments
Web35 Down (Fixed)
Posted Aug 2 at 20:52 CDT by Richard in Downtime, Problems - 0 comment(s)
The filesystem on Web35 started having problems a few minutes ago. We are rebooting Web35 now, and will perform a filesystem check before bringing the server back online. We'll update this entry when we have new information.
Update 2008-08-02 21:01: The server is back up now.
Read the full article and comments
Mail server issues on Mail5 (Fixed)
Updated Jul 31 at 10:03 CDT (first posted Jul 30 at 11:24 CDT) by Sean in Problems - 0 comment(s)
Incoming email is currently delayed on Mail5. We're currently working to correct this issue and will update this entry when the problem has been resolved.
Update 2008-07-30 15:00 - we are still working to resolve this issue, thanks for your patience.
Update 2008-07-30 15:53 - the issue on Mail5 has been resolved and all backlogged mail has been delivered.
Update 2008-07-31 09:21 - This problem was caused by a huge backscatter attack. One of the domains we host was used in the return-path address by a spammer, which meant that all of their undeliverable email (and if you're a spammer there's a lot of it) got sent to us. This is not particularly unusual in itself, all spammers use forged headers so that innocent bystanders get there undeliverable email. What was unusual was the size of the attack. The hundreds of thousands of emails we received flooded our mail queue and lead to significant delays in delivering legitimate email. To restore normal service we implemented some mail filtering rules to prevent any more of the backscatter getting into the queue and then cleaned up the junk that had already got into it. We are currently investigating how we can catch and stop these kinds of attacks earlier in future in order to prevent this kind of delay happening again.
Read the full article and comments
Web13 down (fixed)
Posted Jul 28 at 04:44 CDT by Remi in Downtime - 0 comment(s)
Web13 is currently down for an unknown reason. We'll update this post once we have more information.
Update: Web13 is now back online. The problem was caused by a misconfiguration of the network interface on the machine.
Read the full article and comments
New PHP setup (done)
Updated Aug 12 at 07:45 CDT (first posted Jul 24 at 07:59 CDT) by Remi in General - 0 comment(s)
We're gradually rolling out a new PHP setup to all of our servers. This new setup fixes a security issue.
With the new setup PHP scripts are executed with mod_php as the user that the file belongs to. This means that there are no more permission issues and no more issues with files uploaded through PHP being owned by Apache.
Note that a small percentage of sites might need a few changes to keep working the same way with the new setup. Here are some changes that we will do to this small percentage of sites:
- If you are setting PHP options in a .htacess file (for instance: "php_flag register_globals on") we will enclose them in an "IfModule" tag like this: "<IfModule php5_module>php_flag register_globals on</IfModule>"
- If you are serving PHP files with ".html" extensions, the AddType directive will not work any more. Please see our KB article for an alternate solution.
- If you have some uploaded files that are owned by apache we will change the file owner back to you.
- PHP currently reports REMOTE_ADDR as 127.0.0.1 due to the proxy configuration. We expect to roll out a fix for this on Friday August 8, 2008.
So far, the following servers have the new PHP setup: dweb18 to dweb25, web21 to web28, web41 to web43
We will update this post with the rollout schedule for other servers.
Update (31 July, 9.30am GMT): Over the next few hours the new PHP setup will be deployed to the following servers: web29 to web40, web44, web45, dweb26 to dweb33.
Update (31 July, 18.10pm GMT): The following servers now use the new PHP setup: web21 and above, dweb18 and above.
Read the full article and comments
Network problem on Web3 (Fixed)
Updated Jul 30 at 11:27 CDT (first posted Jun 20 at 09:01 CDT) by Remi in Problems - 0 comment(s)
Web3 is currently experiencing some network issues and connection from certain places is either very slow or non existent. We're currently looking into the issue.
Update: the network issue has been resolved.
Read the full article and comments
Rebooting Web4 (done)
Updated Jun 11 at 05:49 CDT (first posted Jun 10 at 13:52 CDT) by Sean in Problems - 0 comment(s)
We're rebooting Web4 right now to switch to a new kernel - this should improve the load situation.
Update: Web4 is back online.
Read the full article and comments