You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "Paul J. Reder" <re...@raleigh.ibm.com> on 2001/02/27 22:38:00 UTC
Problem found in perform_idle_server_maintenance on prefork.
I ran a particularly abusive set of tests which Apache held up beautifully
through, but found a problem in the end.
After the run, the number of children did not return all the way back to the
steady state number. I have been able to reproduce the problem.
What seems to happen is that perform_idle_server_maintenance always picks the
highest child pid value to kill. In the problematic case the highest child is
just starting. Apache tries to kill this child (kill SIGWINCH). The child
tries to die but ends up in the following state:
#0 0x400e458b in __sigsuspend (set=0xbffff8ec) at ../sysdeps/unix/sysv/linux/sigsuspend.c:48
#1 0x400bd211 in __pthread_lock (lock=0x80c8b1c, self=0x400c36c0) at restart.h:32
#2 0x400ba99a in __pthread_mutex_lock (mutex=0x80c8b0c) at mutex.c:84
#3 0x80b1b56 in apr_unix_lock_intra (lock=0x80c8aec) at intraproc.c:120
#4 0x80af22c in apr_lock_acquire (lock=0x80c8aec) at locks.c:113
#5 0x80a0e1b in apr_pool_destroy (a=0x80eebec) at apr_pools.c:761
#6 0x8069d97 in clean_child_exit (code=0) at prefork.c:233
#7 0x8069f82 in just_die (sig=28) at prefork.c:354
#8 0x400bc522 in pthread_sighandler (signo=28, ctx={gs = 0, __gsh = 0, fs = 0, __fsh = 0, es = 43, __esh = 0, ds = 43,
__dsh = 0, edi = 3221224724, esi = 1073784336, ebp = 3221224232, esp = 3221224220, ebx = 197, edx = 135416640,
ecx = 135424848, eax = 135416640, trapno = 16, err = 0, eip = 134875327, cs = 35, __csh = 0, eflags = 66183,
esp_at_signal = 3221224220, ss = 43, __ssh = 0, fpstate = 0x0, oldmask = 2147483648, cr2 = 0}) at signals.c:91
#9 0x400e4408 in __restore () at ../sysdeps/unix/sysv/linux/i386/sigaction.c:127
#10 0x80a0985 in apr_pool_sub_make (p=0x80eebec, apr_abort=0) at apr_pools.c:481
#11 0x80a0a48 in apr_pool_create (newcont=0xbffffb78, cont=0x80eebec) at apr_pools.c:545
#12 0x806a447 in child_main (child_num_arg=196) at prefork.c:543
#13 0x806a9ad in make_child (s=0x80cefd4, slot=196) at prefork.c:869
#14 0x806ac71 in perform_idle_server_maintenance () at prefork.c:1010
#15 0x806b048 in ap_mpm_run (_pconf=0x80ceaec, plog=0x80eabcc, s=0x80cefd4) at prefork.c:1180
#16 0x807073c in main (argc=3, argv=0xbffffd14) at main.c:428
Apparently the child was in the midst of creating the pool when it got the interrupt. The
cleanup can't get the lock so the child is deadlocked with itself. Apache tries to kill this
child every second and fails. None of the other kids get cleaned up. The scoreboard lists this
child in the Starting state.
If I kill this child, then idle maintenance continues as normal killing kids down to the steady
state number.
Possible fixes:
---------------
1) Don't allow perform_idle_server_maintenance to clean up children in the Starting state.
Kill the highest numbered child not in the S state.
2) Look into closing the timing window that allows this to happen. Continue allowing state
S children to be killed.
3) Keep track of the last pid killed. If it is the same pid N times in a row, change its
state to Zombie and kill the next pid down. Log the presence of a zombie for debugging
purposes. Don't try to kill Zombies.
4) Something else more appropriate?
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: Problem found in perform_idle_server_maintenance on prefork.
Posted by Martin Kraemer <Ma...@Fujitsu-Siemens.com>.
On Tue, Feb 27, 2001 at 04:38:00PM -0500, Paul J. Reder wrote:
> Apaache tries to kill this child (kill SIGWINCH). The child
> tries to die but ends up in the following state:
Congratulations! Excellent analysis! Yes, I often had httpd-2.x children
hanging, especially after restarts or shutdowns. Your finding will
(hopefully soon) solve that.
Martin
--
<Ma...@Fujitsu-Siemens.com> | Fujitsu Siemens
<ma...@apache.org> | 81730 Munich, Germany
Re: Problem found in perform_idle_server_maintenance on prefork.
Posted by rb...@covalent.net.
On Wed, 28 Feb 2001, Ben Laurie wrote:
> Greg Stein wrote:
> >
> > On Wed, Feb 28, 2001 at 08:24:59PM +0000, Ben Laurie wrote:
> > > dean gaudet wrote:
> > > > and actually that graceful die flag might as well live in the scoreboard.
> > > > i can't remember why the graceful die flag doesn't live in the scoreboard
> > > > in 1.3... maybe i just never thought of it. hrm. i bet i just didn't
> > > > bother to move it there because 1.3 code had to be signal aware anyhow.
> > > > you may want to move this to the scoreboard in 2.0 to eliminate the
> > > > SIGWINCH in the children, it'll mean fewer potential problems with
> > > > non-signal aware 3rd party code.
> > >
> > > IIRC, I originally put the graceful die flag in the scoreboard. The
> > > issue was that if your server wasn't heavily loaded, many child
> > > processes didn't ever wake up to discover they should die. So you had to
> > > signal to interrupt the accept() they were blocked in.
> >
> > Hmm. Do we still block on accept()? Or do we block on select()? If the
> > latter, then we can wake them up with the "pipe of death" thingy.
>
> Dunno, but I can't think why we couldn't block on select() if we don't
> already (except I'll wager there's some darn braindead system out there
> that won't let us).
>
> BTW, its coming back to me - it was even more elegant - in the
> scoreboard was a "current generation" number - if a child's own
> generation was not equal to the current generation, then it died.
That generation is still there. :-)
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: Problem found in perform_idle_server_maintenance on prefork.
Posted by Ben Laurie <be...@algroup.co.uk>.
Greg Stein wrote:
>
> On Wed, Feb 28, 2001 at 08:24:59PM +0000, Ben Laurie wrote:
> > dean gaudet wrote:
> > > and actually that graceful die flag might as well live in the scoreboard.
> > > i can't remember why the graceful die flag doesn't live in the scoreboard
> > > in 1.3... maybe i just never thought of it. hrm. i bet i just didn't
> > > bother to move it there because 1.3 code had to be signal aware anyhow.
> > > you may want to move this to the scoreboard in 2.0 to eliminate the
> > > SIGWINCH in the children, it'll mean fewer potential problems with
> > > non-signal aware 3rd party code.
> >
> > IIRC, I originally put the graceful die flag in the scoreboard. The
> > issue was that if your server wasn't heavily loaded, many child
> > processes didn't ever wake up to discover they should die. So you had to
> > signal to interrupt the accept() they were blocked in.
>
> Hmm. Do we still block on accept()? Or do we block on select()? If the
> latter, then we can wake them up with the "pipe of death" thingy.
Dunno, but I can't think why we couldn't block on select() if we don't
already (except I'll wager there's some darn braindead system out there
that won't let us).
BTW, its coming back to me - it was even more elegant - in the
scoreboard was a "current generation" number - if a child's own
generation was not equal to the current generation, then it died.
Cheers,
Ben.
--
http://www.apache-ssl.org/ben.html
"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff
ApacheCon 2001! http://ApacheCon.com/
Re: Problem found in perform_idle_server_maintenance on prefork.
Posted by Greg Stein <gs...@lyra.org>.
On Wed, Feb 28, 2001 at 08:24:59PM +0000, Ben Laurie wrote:
> dean gaudet wrote:
> > and actually that graceful die flag might as well live in the scoreboard.
> > i can't remember why the graceful die flag doesn't live in the scoreboard
> > in 1.3... maybe i just never thought of it. hrm. i bet i just didn't
> > bother to move it there because 1.3 code had to be signal aware anyhow.
> > you may want to move this to the scoreboard in 2.0 to eliminate the
> > SIGWINCH in the children, it'll mean fewer potential problems with
> > non-signal aware 3rd party code.
>
> IIRC, I originally put the graceful die flag in the scoreboard. The
> issue was that if your server wasn't heavily loaded, many child
> processes didn't ever wake up to discover they should die. So you had to
> signal to interrupt the accept() they were blocked in.
Hmm. Do we still block on accept()? Or do we block on select()? If the
latter, then we can wake them up with the "pipe of death" thingy.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: Problem found in perform_idle_server_maintenance on prefork.
Posted by Ben Laurie <be...@algroup.co.uk>.
dean gaudet wrote:
> and actually that graceful die flag might as well live in the scoreboard.
> i can't remember why the graceful die flag doesn't live in the scoreboard
> in 1.3... maybe i just never thought of it. hrm. i bet i just didn't
> bother to move it there because 1.3 code had to be signal aware anyhow.
> you may want to move this to the scoreboard in 2.0 to eliminate the
> SIGWINCH in the children, it'll mean fewer potential problems with
> non-signal aware 3rd party code.
IIRC, I originally put the graceful die flag in the scoreboard. The
issue was that if your server wasn't heavily loaded, many child
processes didn't ever wake up to discover they should die. So you had to
signal to interrupt the accept() they were blocked in.
Cheers,
Ben.
--
http://www.apache-ssl.org/ben.html
"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff
ApacheCon 2001! http://ApacheCon.com/
Re: Problem found in perform_idle_server_maintenance on prefork.
Posted by Bill Stoddard <bi...@wstoddard.com>.
----- > i'm pretty sure SIGWINCH is just the SIGUSR1 replacement right? it
tells
> the child to die as soon as is convenient. so the process really only
> needs to die off after it's fully initialised. if so then all of the
> above is avoided by using signal handlers in the only way they're safe to
> use: to tweak a global variable. (1.3's USR1 handler is almost that
> simple.)
Jeff Trawick made this same point yesterday in a hall talk.
>
> and actually that graceful die flag might as well live in the scoreboard.
> i can't remember why the graceful die flag doesn't live in the scoreboard
> in 1.3... maybe i just never thought of it. hrm. i bet i just didn't
> bother to move it there because 1.3 code had to be signal aware anyhow.
> you may want to move this to the scoreboard in 2.0 to eliminate the
> SIGWINCH in the children, it'll mean fewer potential problems with
> non-signal aware 3rd party code.
>
+1
Bill
Re: Problem found in perform_idle_server_maintenance on prefork.
Posted by dean gaudet <dg...@arctic.org>.
yay asynchronous notification! it sure sucks :)
On Tue, 27 Feb 2001, Paul J. Reder wrote:
> 1) Don't allow perform_idle_server_maintenance to clean up children in the Starting state.
> Kill the highest numbered child not in the S state.
> 2) Look into closing the timing window that allows this to happen. Continue allowing state
> S children to be killed.
> 3) Keep track of the last pid killed. If it is the same pid N times in a row, change its
> state to Zombie and kill the next pid down. Log the presence of a zombie for debugging
> purposes. Don't try to kill Zombies.
> 4) Something else more appropriate?
1 and 3 just hide the problems... you could have trouble any time SIGWINCH
arrives during the alloc_mutex critical sections.
a more accurate fix would be something like this:
- add apr_lock_try() which behaves like pthread_mutex_trylock() does
- use apr_lock_try() in the signal handler.
if it succeeds, continue with cleanup
if it fails, then set a global indicating deferred cleanup
also block SIGWINCH before returning from the sighandler
- after every instance of apr_lock_release(alloc_mutex), test the global
and if deferred cleanup is indicated then proceed with the deferred
cleanup
but that alone is just the tip of the iceberg.
take a look at ap_popenf() in apache-1.3. notice that it starts with
ap_block_alarms().
ok now go look at apr_file_open() in apache-2.0. notice that it opens a
file and then registers the cleanup. notice how if SIGWINCH arrives at
the wrong time, the cleanup won't have been registered.
big deal it's just a file... i pretty much agree on this resource. but
i'm pointing it out more as a fundamental problem. essentially anything
which doesn't have access to the alloc_mutex can't guarantee atomicity
with respect to SIGWINCH. might as well not have cleanups at all, or just
not expect them to be called when the process is killed.
(alloc_mutex isn't the only way to solve it, you can do just what 1.3's
ap_block_alarms do.)
i'm pretty sure SIGWINCH is just the SIGUSR1 replacement right? it tells
the child to die as soon as is convenient. so the process really only
needs to die off after it's fully initialised. if so then all of the
above is avoided by using signal handlers in the only way they're safe to
use: to tweak a global variable. (1.3's USR1 handler is almost that
simple.)
and actually that graceful die flag might as well live in the scoreboard.
i can't remember why the graceful die flag doesn't live in the scoreboard
in 1.3... maybe i just never thought of it. hrm. i bet i just didn't
bother to move it there because 1.3 code had to be signal aware anyhow.
you may want to move this to the scoreboard in 2.0 to eliminate the
SIGWINCH in the children, it'll mean fewer potential problems with
non-signal aware 3rd party code.
almost all of these types of race conditions in 1.2 were discovered by
doing "while 1; kill -HUP" or kill -USR1 at the same time the server had a
load running against it, and i was browsing pages. (the same test helped
shake out ancient linux SMP kernel bugs too.)
-dean