PHP setup update (done)

Updated Aug 15 at 02:37 CDT (first posted Aug 12 at 07:47 CDT) by Remi in General  - 0 comment(s)

In the next few hours we're going to continue our improvements to our PHP setup:

• Fix for remote_addr: We're rolling out a patch that fixes the remote_addr value available to PHP scripts.
• New PHP setup on all servers: We're rolling out the new PHP setup to all remaining servers (dweb1 to dweb17, web1 to web20, krait and mamba).

We'll update this post when these updates are finished.

Update: All servers are now running the new PHP setup. We're going to roll out the remote_addr patch now.

Update: All servers now have the remote_addr patch.

Read the full article and comments

New PHP setup (done)

Updated Aug 12 at 07:45 CDT (first posted Jul 24 at 07:59 CDT) by Remi in General  - 0 comment(s)

We're gradually rolling out a new PHP setup to all of our servers. This new setup fixes a security issue.

With the new setup PHP scripts are executed with mod_php as the user that the file belongs to. This means that there are no more permission issues and no more issues with files uploaded through PHP being owned by Apache.

Note that a small percentage of sites might need a few changes to keep working the same way with the new setup. Here are some changes that we will do to this small percentage of sites:

• If you are setting PHP options in a .htacess file (for instance: "php_flag register_globals on") we will enclose them in an "IfModule" tag like this: "<IfModule php5_module>php_flag register_globals on</IfModule>"
• If you are serving PHP files with ".html" extensions, the AddType directive will not work any more. Please see our KB article for an alternate solution.
• If you have some uploaded files that are owned by apache we will change the file owner back to you.
• PHP currently reports REMOTE_ADDR as 127.0.0.1 due to the proxy configuration. We expect to roll out a fix for this on Friday August 8, 2008.

So far, the following servers have the new PHP setup: dweb18 to dweb25, web21 to web28, web41 to web43

We will update this post with the rollout schedule for other servers.

Update (31 July, 9.30am GMT): Over the next few hours the new PHP setup will be deployed to the following servers: web29 to web40, web44, web45, dweb26 to dweb33.

Update (31 July, 18.10pm GMT): The following servers now use the new PHP setup: web21 and above, dweb18 and above.

Read the full article and comments

MySql upgrades

Posted Apr 4 at 06:11 CDT by Remi in General  - 0 comment(s)

We're going to upgrade MySql on the following servers: Dweb1 to Dweb9 and Web1 to Web20.

The upgrade fixes some security flaws in MySql and MySql will only be down for a few seconds on each server to restart it.

Read the full article and comments

Local root exploit patched

Posted Feb 11 at 08:57 CDT by Remi in General  - 0 comment(s)

A local root exploit was announced last night (see http://it.slashdot.org/it/08/02/10/2011257.shtml). All of our CentOS5 servers were vulnerable. The exploit allowed any user of the system to become root.

We have patched all of our CentOS5 servers and they are no longer vulnerable to this exploit. In the process we had to reboot 3 of these servers to change the kernel versions: Web22, Web26 and Web27

Read the full article and comments

Control panel slowness (fixed)

Updated Jun 11 at 05:47 CDT (first posted Mar 9 at 13:22 CDT) by Remi in General  - 0 comment(s)

You might have noticed that the control panel has been quite slow lately.

This is because we have outgrown a single server for the central database (that poor central database is hammered by people using the control panel and by all the servers connecting to it to get their configuration).

Over the next few days we are going to get more servers and spread the load of the database over multiple servers, so things should hopefully get snappy again...

In case you're interested, we're using Postgres for our central database and we're going to use Slony to replicate the database over multiple servers.

Update: It should be much faster now.

Remi. Read the full article and comments

Welcome

Updated Aug 28 at 00:30 CDT (first posted Aug 27 at 10:46 CDT) by Remi in General  - 0 comment(s)

Welcome to our new status blog.

This is where we will keep you informed on what's happening on our servers:

• Scheduled downtime
• Upgrades
• Problems (yes, we're only human)

Remi. Read the full article and comments