mod_cgi, fork, and threaded MPMs

[Paul Querna <ch...@force-elite.com>]

Last week bloglines.com upgraded from 2.0.x to 2.2.x.  At the same time, 
I switched us from mod_cgid to mod_cgi.

mod_cgid has some problems if its path to the cgisock changes at any 
time, it really needs to call realpath() on the cgisock path, to avoid 
issues with how we distribute releases. (We swap some symlinks around, 
but as soon as these were changed, it would cause every CGI request to 
return a 503...)

I decided to instead to just switch to mod_cgi.  Fork() on a threaded 
process according to POSIX is supposed to be thread local.  And Linux 
wasn't supposed to be stupid about it, unlike Solaris.  So, running 
mod_cgi on the worker MPM should of been safe.

However, we have started getting reports of a problem.  Some users 
cookies are getting mixed up.  We use a cookie to determine which 
language to display to the user, and randomly, some users would get put 
into the wrong language.  We had a couple reports, and assumed at first 
it was user error, but then it happened to me... And I was damn sure I 
hadn't changed my language.

And we looked deeper. None of this code had changed. Not for a long long 
long time. The only things that changed was the upgrade to 2.2, and 
switching to mod_cgi.  We haven't been able to reproduce this in any 
test environment.

The OS is Linux, RHEL 4, update 2, with the 2.6.9-22.ELsmp kernel, NPTL 
enabled, and the Worker MPM.

I believe the cause is a race condition with fork.  In a highly threaded 
process, with perhaps nearly all of them calling fork(), there is a bug 
somewhere low level in glibc/nptl.  Sometimes you get the wrong data 
back, or your execv run later has the wrong environ.

So, we are in the process of switching back to mod_cgid for now.  Does 
anyone else have any ideas, or seen this type of problem before?

-Paul

Re: mod_cgi, fork, and threaded MPMs

[Jeff Trawick <tr...@gmail.com>]

On 1/14/06, Paul Querna <ch...@force-elite.com> wrote:
> Last week bloglines.com upgraded from 2.0.x to 2.2.x.  At the same time,
> I switched us from mod_cgid to mod_cgi.
>
> mod_cgid has some problems if its path to the cgisock changes at any
> time, it really needs to call realpath() on the cgisock path, to avoid
> issues with how we distribute releases. (We swap some symlinks around,
> but as soon as these were changed, it would cause every CGI request to
> return a 503...)
>
> I decided to instead to just switch to mod_cgi.  Fork() on a threaded
> process according to POSIX is supposed to be thread local.  And Linux
> wasn't supposed to be stupid about it, unlike Solaris.  So, running
> mod_cgi on the worker MPM should of been safe.

What happens with mutexes?  They get forked (including ownership), and
there is no code in APR/httpd to make sure the ownership of mutexes
makes sense in the forked child (the kind of stuff that is done in
pthread_atfork handlers).

What happens with pipes created to communicate with CGI children? 
Pipe handles accidently get inherited by child processes created by
other threads when then can hold them open unknowingly preventing EOF
from being seen from the original CGI at the proper time, right?

Cookie-going-to-wrong-user could be caused by file descriptor handling
glitch like a double close, but it would have to be in code that
Windows doesn't use (assuming there are enough Windows CGI setups). 
"close(9); ... close(9);" in prefork is sloppiness seen when you do
strace, but with worker it gets a CVE number.