You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Mark Stosberg <ma...@summersault.com> on 2004/10/04 21:13:25 UTC
[mp1] text-book forking code still producing zombies
Hello,
I've been enjoying the performance benefits of mod_perl, and appreciate
the help of the contributors here.
Now I'm investigating why our production code is producing zombies. I'll
post the key parts of my code below. First, some observations:
1. Zombies appear to be created only in a minority of the times the fork
is made.
2. Restarting Apache seems to kill the zombies. I was under the
impression that with a "true" zombie, an OS restart was required
to kill zombies.
Here's my code structure. It comes from a CGI::Application-based project using
Apache::Registry.
$SIG{CHLD} = 'IGNORE';
defined ($childpid = fork) or return $self->error(
title => 'Technical Failure. ',
msg => 'The search could not be completed due to a technical failure. ',
);
unless ($childpid) {
open STDIN, "</dev/null";
open STDOUT, ">/dev/null";
# break out a new DBI connection for use by the child
my $DBH = $self->connect_db(undef, { dbi_connect_method => 'connect' });
$self->param('dbh', $DBH);
eval {
&my_long_running_operation;
};
carp $@ if $@;
CORE::exit(0);
}
############################
I'm curious what possible causes are, or better alternatives for structuring
the forking code.
Below are my environment details. Thanks!
Mark
######
modperl/1.29
Apache 1.3.31
$ perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 2) configuration:
Platform:
osname=freebsd, osvers=4.10-release, archname=i386-freebsd
uname='freebsd my.secretservername.com 4.10-release freebsd 4.10-release #0: tue may 25 22:47:12 gmt 2004 root@perseus.cse.buffalo.edu:usrobjusrsrcsysgeneric i386 '
config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.8.2/mach -Dprivlib=/usr/local/lib/perl5/5.8.2 -Dman3dir=/usr/local/lib/perl5/5.8.2/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.8.2/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.8.2 -Dscriptdir=/usr/local/bin -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Doptimize=-O -pipe -Duseshrplib -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.2/BSDPAN" -Ud_dosuid -Ui_gdbm -Dusethreads=n -Dusemymalloc=y'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.2/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -I/usr/local/include',
optimize='-O -pipe ',
cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.2/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -I/usr/local/include'
ccversion='', gccversion='2.95.4 20020320 [FreeBSD]', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags ='-Wl,-E -L/usr/local/lib'
libpth=/usr/lib /usr/local/lib
libs=-lm -lcrypt -lutil -lc
perllibs=-lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-R/usr/local/lib/perl5/5.8.2/mach/CORE'
cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under freebsd
Compiled at Aug 15 2004 09:43:23
@INC:
/usr/local/lib/perl5/site_perl/5.8.2/mach
/usr/local/lib/perl5/site_perl/5.8.2
/usr/local/lib/perl5/site_perl
/usr/local/lib/perl5/5.8.2/BSDPAN
/usr/local/lib/perl5/5.8.2/mach
/usr/local/lib/perl5/5.8.2
.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [SumsaultRT #1249][mp1] text-book forking code still producing zombies
Posted by Mark Stosberg <ma...@summersault.com>.
On 2004-10-06, Mark Stosberg <ma...@summersault.com> wrote:
>
>>> Could setting MaxRequestsPerChild be a meaningful solution here? It
>>> would seem that this would cause the Apache child processes to
>>> periodically get re-created, killing related zombies in the process.
>>
>> If you aren't using MaxRequestsPerChild or something like
>> Apache::SizeLimit, you definitely should be.
>
> Great, We'll start using that.
Setting MaxRequestsPerChild to 100 seemed to be a sufficient solution in
this case. We are now seeing a zombie count that hovers between 20 to
30, very acceptable number for us.
Thanks to Stas and Perrin for your help!
Mark
--
http://mark.stosberg.com/
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [SumsaultRT #1249][mp1] text-book forking code still producing zombies
Posted by Mark Stosberg <ma...@summersault.com>.
On 2004-10-06, Perrin Harkins <pe...@elem.com> wrote:
> On Wed, 2004-10-06 at 16:56, Mark Stosberg wrote:
>> At your recommended, I installed strace and ran it on the hung process.
>> It doesn't work.
>
> Hmm, maybe it's time to add some logging to your forked processes to
> find out where they are getting stuck.
We did add some logging and found the child is getting all the way to
the end of code block-- it logs just before calling "CORE::die". I just
double-checked the logs and it looks we are always getting to
"CORE::die", but zombies don't seem to happen every time.
I also see the zombie count fluctuate a little bit. Some times it goes
down by just 1 or 2 on it's own.
>> Could setting MaxRequestsPerChild be a meaningful solution here? It
>> would seem that this would cause the Apache child processes to
>> periodically get re-created, killing related zombies in the process.
>
> If you aren't using MaxRequestsPerChild or something like
> Apache::SizeLimit, you definitely should be.
Great, We'll start using that.
> By the way, I'm loving your CGI::Application::ValidateRM (and
> Data::FormValidator) module.
Great! If you don't follow the CGI::Application list, you may be
interested to know there are several other "plugins" like ValidateRM
that have appeared recently:
http://search.cpan.org/search?query=CGI%3A%3AApplication%3A%3APlugin&mode=module
I'll soon release an update to the "DBH" plugin that supports multiple
database handles.
Mark
--
. . . . . . . . . . . . . . . . . . . . . . . . . . .
Mark Stosberg Principal Developer
mark@summersault.com Summersault, LLC
765-939-9301 ext 202 database driven websites
. . . . . http://www.summersault.com/ . . . . . . . .
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [SumsaultRT #1249][mp1] text-book forking code still producing
zombies
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2004-10-06 at 16:56, Mark Stosberg wrote:
> At your recommended, I installed strace and ran it on the hung process.
> It doesn't work.
Hmm, maybe it's time to add some logging to your forked processes to
find out where they are getting stuck.
> Could setting MaxRequestsPerChild be a meaningful solution here? It
> would seem that this would cause the Apache child processes to
> periodically get re-created, killing related zombies in the process.
If you aren't using MaxRequestsPerChild or something like
Apache::SizeLimit, you definitely should be.
By the way, I'm loving your CGI::Application::ValidateRM (and
Data::FormValidator) module.
- Perrin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [SumsaultRT #1249][mp1] text-book forking code still producing zombies
Posted by Mark Stosberg <ma...@summersault.com>.
Stas --
On 2004-10-05, Stas Bekman <st...@stason.org> wrote:
>
> May be it's just not working on your platform. Have you tried the other
> suggested solutions at:
> http://perl.apache.org/docs/1.0/guide/performance.html#Avoiding_Zombie_Processes
Thank you very much for the response. In the past we have tried a number
of other possibilities, based on part on this helpful documentation.
What we have now seems to work best, but still produces these zombie
like processes once in a while. This is a fairly busy website, so this
amounts to be about 500 or more per day.
> Also did you try to strace the hanging processes, to see what are they
> hanging on? (i don't remember if one can strace a zombie).
At your recommended, I installed strace and ran it on the hung process.
It doesn't work.
I think it's a salient point that the zombies go away with an Apache
restart. According to "ps" (and 'strace'), these are zombie processes,
but apparently they the still have some tie to their parent process.
Could setting MaxRequestsPerChild be a meaningful solution here? It
would seem that this would cause the Apache child processes to
periodically get re-created, killing related zombies in the process.
Thanks again. You're helping http://www.1-800-save-a-pet.com/ which
recently re-launched with Apache/mod_perl after using AOLserver with
TCL in its last incarnation.
Mark
--
. . . . . . . . . . . . . . . . . . . . . . . . . . .
Mark Stosberg Principal Developer
mark@summersault.com Summersault, LLC
765-939-9301 ext 202 database driven websites
. . . . . http://www.summersault.com/ . . . . . . . .
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [mp1] text-book forking code still producing zombies
Posted by Stas Bekman <st...@stason.org>.
Mark Stosberg wrote:
> Hello,
>
> I've been enjoying the performance benefits of mod_perl, and appreciate
> the help of the contributors here.
>
> Now I'm investigating why our production code is producing zombies. I'll
> post the key parts of my code below. First, some observations:
>
> 1. Zombies appear to be created only in a minority of the times the fork
> is made.
>
> 2. Restarting Apache seems to kill the zombies. I was under the
> impression that with a "true" zombie, an OS restart was required
> to kill zombies.
>
> Here's my code structure. It comes from a CGI::Application-based project using
> Apache::Registry.
>
> $SIG{CHLD} = 'IGNORE';
> defined ($childpid = fork) or return $self->error(
> title => 'Technical Failure. ',
> msg => 'The search could not be completed due to a technical failure. ',
> );
>
> unless ($childpid) {
> open STDIN, "</dev/null";
> open STDOUT, ">/dev/null";
>
> # break out a new DBI connection for use by the child
> my $DBH = $self->connect_db(undef, { dbi_connect_method => 'connect' });
> $self->param('dbh', $DBH);
>
> eval {
> &my_long_running_operation;
> };
> carp $@ if $@;
>
> CORE::exit(0);
> }
May be it's just not working on your platform. Have you tried the other
suggested solutions at:
http://perl.apache.org/docs/1.0/guide/performance.html#Avoiding_Zombie_Processes
Also did you try to strace the hanging processes, to see what are they
hanging on? (i don't remember if one can strace a zombie).
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html