Instability at startup for Apache2/mod_perl2 using worker MPM

[David Scott <ds...@earthlink.net>]

OK this is my third message today, hopefully it will be the last.

I've built Apache 2.2.3 and mod_perl 2.0.3 with libapreq 2.08 and the 
worker MPM.  There appears to be some kind of race condition at startup 
that prevents the server from coming up.  This only happens once in a 
while, and can be fixed by some minor configuration or code change (ie, 
adding a 'print STDERR "foobar\n"' to one of my Perl modules can cause 
the problem to occur; moving the print statement by one line can fix it).

There are two manifestations of this problem: either

(1) The first httpd process goes to 99% CPU usage and stays there 
indefinitely, never doing a restart; or
(2) The restart occurs and the new parent process forks *one* child and 
then goes to 99% CPU usage indefinitely, never completing the startup 
sequence.

I've been porting a fairly complex pure-Perl legacy application from 
mod_perl 1 to mod_perl 2.  The reason we're trying this is because 
memory requirements in MP1 are horrendous and child process death is 
extremely expensive.  The application appears to work fine once the 
Apache2 server starts.

In addition, it's not unusual for the mod_perl 1 server to take a long 
time to start up or shut down.

I suspect that there is some nasty interaction between my Perl modules 
(of which there are many).  Does anyone have suggestions on how I can 
track down the problem?  Apache2::Status is useless until the server 
starts, obviously.

Here is my system information:

Apache/2.2.3 (Unix) mod_apreq2-20051231/2.6.0 mod_perl/2.0.3 Perl/v5.8.4
configured -- resuming normal operations
[Thu Nov 30 14:15:39 2006] [info] Server built: Nov 29 2006 17:32:31

Also, here is perl -V:

Summary of my perl5 (revision 5 version 8 subversion 4) configuration:
   Platform:
     osname=linux, osvers=2.6.15.4, archname=i386-linux-thread-multi
     uname='linux ninsei 2.6.15.4 #1 smp preempt mon feb 20 09:48:53 pst
2006 i686 gnulinux '
     config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
-Dcccdlflags=-fPIC -Darchname=i386-linux -Dprefix=/usr
-Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8
-Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl/5.8.4
-Dsitearch=/usr/local/lib/perl/5.8.4 -Dman1dir=/usr/share/man/man1
-Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1
-Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm
-Duseshrplib -Dlibperl=libperl.so.5.8.4 -Dd_dosuid -des'
     hint=recommended, useposix=true, d_sigaction=define
     usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
     useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
     use64bitint=undef use64bitall=undef uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DDEBIAN -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
     optimize='-O2',
     cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN
-fno-strict-aliasing -I/usr/local/include'
     ccversion='', gccversion='3.3.5 (Debian 1:3.3.5-13)', gccosandvers=''
     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
     d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
     ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
     alignbytes=4, prototype=define
   Linker and Libraries:
     ld='cc', ldflags =' -L/usr/local/lib'
     libpth=/usr/local/lib /lib /usr/lib
     libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
     perllibs=-ldl -lm -lpthread -lc -lcrypt
     libc=/lib/libc-2.3.2.so, so=so, useshrplib=true,
libperl=libperl.so.5.8.4
     gnulibc_version='2.3.2'
   Dynamic Linking:
     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
     cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl):
   Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
   Built under linux
   Compiled at Mar 23 2006 21:49:08
   @INC:
     /etc/perl
     /usr/local/lib/perl/5.8.4
     /usr/local/share/perl/5.8.4
     /usr/lib/perl5
     /usr/share/perl5
     /usr/lib/perl/5.8
     /usr/share/perl/5.8
     /usr/local/lib/site_perl
     .

Thanks again.

David

Re: Instability at startup for Apache2/mod_perl2 using worker MPM

[Robert Landrum <rl...@aol.net>]

Perrin Harkins wrote:
> David Scott wrote:
>> I've built Apache 2.2.3 and mod_perl 2.0.3 with libapreq 2.08 and the 
>> worker MPM.  There appears to be some kind of race condition at 
>> startup that prevents the server from coming up.  This only happens 
>> once in a while, and can be fixed by some minor configuration or code 
>> change (ie, adding a 'print STDERR "foobar\n"' to one of my Perl 
>> modules can cause the problem to occur; moving the print statement by 
>> one line can fix it).
> 
> This sounds like a possible issue with your code and threaded perl.  It 
> doesn't ring a bell as a general mod_perl issue.
> 

Interesting problem.  I remember reading about an issue with threading 
(not specific to apache or mod_perl) that occured when file descriptors 
were being modified while there were contents in the buffer.

Just for giggles, try

{
local $|=1;
use MyModules;
...
}

Rob

Re: Instability at startup for Apache2/mod_perl2 using worker MPM

[Perrin Harkins <pe...@elem.com>]

David Scott wrote:
> I've built Apache 2.2.3 and mod_perl 2.0.3 with libapreq 2.08 and the 
> worker MPM.  There appears to be some kind of race condition at startup 
> that prevents the server from coming up.  This only happens once in a 
> while, and can be fixed by some minor configuration or code change (ie, 
> adding a 'print STDERR "foobar\n"' to one of my Perl modules can cause 
> the problem to occur; moving the print statement by one line can fix it).

This sounds like a possible issue with your code and threaded perl.  It 
doesn't ring a bell as a general mod_perl issue.

> I've been porting a fairly complex pure-Perl legacy application from 
> mod_perl 1 to mod_perl 2.  The reason we're trying this is because 
> memory requirements in MP1 are horrendous and child process death is 
> extremely expensive.

Do you realize that it will take MORE memory when run with threads?  If 
you want to keep memory down, use the prefork MPM.  See the list 
archives for discussion about memory and Perl threads.  Also, threads 
slow down startup since a lot of work has to be done when spawning them 
to copy all the data structures for each one.

> In addition, it's not unusual for the mod_perl 1 server to take a long 
> time to start up or shut down.

Slow startup on mod_perl 1 usually means you are compiling a lot of code 
or doing some other time-consuming thing in your startup code, like 
maybe talking to a database.

> Does anyone have suggestions on how I can 
> track down the problem?

I'd start by switching to prefork and seeing if it's still there.  If it 
is, the most reliable method is to take things out until the problem 
goes away.  It's not instant gratification, but it works.

- Perrin