accept mutex failure causes fork bomb

[Greg Ames <am...@gmail.com>]

I'm trying to debug a problem where apparently the accept mutex went bad on
a z/OS system running the worker MPM.  I'm guessing that some memory that we
use for the semaphore got clobbered but don't have proof yet.  The error log
looks like:

[Mon Sep 07 08:01:59 2009] [emerg] (121)EDC5121I Invalid argument.:
apr_proc_mutex_unlock failed. Attempting to shutdown process gracefully.
[Mon Sep 07 08:02:01 2009] [emerg] (121)EDC5121I Invalid argument.:
apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Mon Sep 07 08:02:02 2009] [emerg] (121)EDC5121I Invalid argument.:
apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[Mon Sep 07 08:02:02 2009] [emerg] (121)EDC5121I Invalid argument.:
apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[...]

The rest of the error log is filled with lock failures.  Looking at the time
stamps, you can see that perform_idle_server_maintenance went into
exponential expansion, maxing out at about 24 lock failures per second.
Unfortunately the fork()s were faster than z/OS could terminate the
processes that had detected the mutex problem, so after forking 978 httpd
children, the system ran out of real memory and had to be IPLed.

One of my colleagues asked why ServerLimit 64 didn't stop the fork bomb.
Good question.  The reason is that the error path calls signal_threads()
which causes the child to exit gracefully.  The listener thread sets
ps->quiescing on the way out, which allows the "squatting" logic in
perform_idle_server_maintenance to take over the scoreboard slot before the
previous process has completely exited, bypassing the ServerLimit throttle.

This raises several ideas for improvement:

* Should we do clean_child_exit(APEXIT_CHILDSICK or CHILDFATAL) for this
error?  We have a previous fix to detect accept mutex failures during
restarts and tone down the error messages.  I don't recall seeing any false
error messages after that fix.  We could also use requests_this_child to
detect if this process has ever successfully served a request, and only do
the clean_child_exit if it hasn't.

* Should we yank the squatting logic?  I think it is doing us more harm than
good.  IIRC it was put in to make the server respond faster when the
workload is spikey.  A more robust solution may be to set Min and
MaxSpareThreads farther apart and allow ServerLimit to be enforced
unconditionally.  disclaimer: I created ps->quiescing, so I was an
accomplice.

* Does it make sense to fork more than MaxSpareThreads worth of child
processes at a time?  MaxSpareThreads was 75 in this case, but we tried to
fork at least 600 threads (same as MaxClients) worth of child processes in
one pass of perform_idle_server_maintenance.

This applies to worker and event; some of it may also apply to prefork.  I'd
appreciate thoughts and suggestions before committing anything.

Thanks,
Greg

Re: accept mutex failure causes fork bomb

[Jeff Trawick <tr...@gmail.com>]

On Mon, Sep 14, 2009 at 4:27 PM, Greg Ames <am...@gmail.com> wrote:

> * Should we yank the squatting logic?  I think it is doing us more harm
> than good.  IIRC it was put in to make the server respond faster when the
> workload is spikey.


It finally occurred to me what you meant by spikey: triggering
MaxSpareThreads (duh).

The other conditions with squatting are graceful restart and
MaxRequestsPerChild.


>   A more robust solution may be to set Min and MaxSpareThreads farther
> apart and allow ServerLimit to be enforced unconditionally.



and not use graceful restart or MaxRequestsPerChild and hope you don't get
any child-fatal errors while long-running requests are being handled

and/or set ServerLimit "enough higher" than MaxClients / ThreadsPerChild to
allow ServerLimit to be enforced unconditionally even while some number of
processes hang around while a few dangling requests complete

--/--

slightly?-over-generalization:

The real problem is that processes are so expensive on some platforms in
terms of real memory or swap space or something else that real pain is
inevitable when you have more than a few processes only handling a small
number of requests.

The use of an array for the scoreboard exacerbates this because it
artificially limits the number of requests that can be handled in new
(squatting) processes.

Re: accept mutex failure causes fork bomb

[Greg Ames <am...@gmail.com>]

On Tue, Sep 15, 2009 at 8:07 AM, Jeff Trawick <tr...@gmail.com> wrote:

> On Mon, Sep 14, 2009 at 4:27 PM, Greg Ames <am...@gmail.com> wrote:
>
>> I'm trying to debug a problem where apparently the accept mutex went bad
>> on a z/OS system running the worker MPM.  I'm guessing that some memory that
>> we use for the semaphore got clobbered but don't have proof yet.  The error
>> log looks like:
>>
>> [Mon Sep 07 08:01:59 2009] [emerg] (121)EDC5121I Invalid argument.:
>> apr_proc_mutex_unlock failed. Attempting to shutdown process gracefully.
>>
>
> Could it be some system limit exceeded, like max waiters on a mutex?  (IIRC
> some OSs require tuning of SysV semaphores.  (Older Solaris comes to mind.))
>

it's possible.  but the very first error was for unlock, so the lock must
have worked.  I would think most of the SysV tuning stuff is more critical
for getting a lock than releasing it.


> * Should we do clean_child_exit(APEXIT_CHILDSICK or CHILDFATAL) for this
>> error?  We have a previous fix to detect accept mutex failures during
>> restarts and tone down the error messages.  I don't recall seeing any false
>> error messages after that fix.  We could also use requests_this_child to
>> detect if this process has ever successfully served a request, and only do
>> the clean_child_exit if it hasn't.
>>
>
> So nasty failures prior to successfully accepting a connection bypass
> squatting?  Good.
>

yep, that's the idea.  any of the exit() calls should bypass setting
ps->quiescing and disable squatting.

>
> CHILDSICK or CHILDFATAL in that case?  In this example it probably wasn't
> going to get any better.  However, I think it is reasonably likely that
> child process n+1 encounters some sort of resource limit, so CHILDSICK seems
> better.
>

I agree.  we have existing logic in the parent which will shut down the
server if it can't find any healthy worker threads.


>  * Should we yank the squatting logic?  I think it is doing us more harm
> than good.  IIRC it was put in to make the server respond faster when the
> workload is spikey.  A more robust solution may be to set Min and
> MaxSpareThreads farther apart and allow ServerLimit to be enforced
> unconditionally.  disclaimer: I created ps->quiescing, so I was an
> accomplice.
>
> My understanding is that squatting is required to deal with long-running
> requests that keep a child trying to exit from actually exiting, thus tying
> up a scoreboard process indefinitely.
>

you are right.  thanks for the reminder.

>
> Is it reasonable to have up to MaxClients worth of squatting like we have
> now?  (I think that is what we allow.)  No, I don't think so.
>

I think it's worse than that.  it's more like (MaxClients - threads in fully
active processes) / (avg. number of threads that p_i_s_m thinks are healthy
per quiescing process).  so if we have an average of one thread per
quiescing process downloading a DVD over dialup, you are exactly right.  if
we have zero healthy threads in quiescing processes because the worker
threads have exited but the process isn't completely gone, the denominator
can get pretty small and there isn't any limit on squatting.


> Should we axe squatting, respect ServerLimit, and thus make the admin raise
> ServerLimit
>

or make the admin increase the difference between MinSpareThreads and
MaxSpareThreads to reduce the oscillations between shutting down worker
processes and forking more.


> to accommodate exiting processes which are handling long-running requests
> (and waste a bit of shared memory at the same time)?  Maybe ;)  That change
> seems a bit drastic, but I'm probably just scared of another long period of
> time before I halfway understand how it behaves in the real world.
>

yeah this is a sensitive and difficult to grok piece of code.  I hope
anything we do here simplifies it.  I don't want to have to write the doc
for how ServerLimit and squatting interact.

Greg

Re: accept mutex failure causes fork bomb

[Jeff Trawick <tr...@gmail.com>]

On Mon, Sep 14, 2009 at 4:27 PM, Greg Ames <am...@gmail.com> wrote:

> I'm trying to debug a problem where apparently the accept mutex went bad on
> a z/OS system running the worker MPM.  I'm guessing that some memory that we
> use for the semaphore got clobbered but don't have proof yet.  The error log
> looks like:
>
> [Mon Sep 07 08:01:59 2009] [emerg] (121)EDC5121I Invalid argument.:
> apr_proc_mutex_unlock failed. Attempting to shutdown process gracefully.
>

Could it be some system limit exceeded, like max waiters on a mutex?  (IIRC
some OSs require tuning of SysV semaphores.  (Older Solaris comes to mind.))


> [Mon Sep 07 08:02:01 2009] [emerg] (121)EDC5121I Invalid argument.:
> apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
> [Mon Sep 07 08:02:02 2009] [emerg] (121)EDC5121I Invalid argument.:
> apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
> [Mon Sep 07 08:02:02 2009] [emerg] (121)EDC5121I Invalid argument.:
> apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
> [...]
>
> The rest of the error log is filled with lock failures.  Looking at the
> time stamps, you can see that perform_idle_server_maintenance went into
> exponential expansion, maxing out at about 24 lock failures per second.
> Unfortunately the fork()s were faster than z/OS could terminate the
> processes that had detected the mutex problem, so after forking 978 httpd
> children, the system ran out of real memory and had to be IPLed.
>
> One of my colleagues asked why ServerLimit 64 didn't stop the fork bomb.
> Good question.  The reason is that the error path calls signal_threads()
> which causes the child to exit gracefully.  The listener thread sets
> ps->quiescing on the way out, which allows the "squatting" logic in
> perform_idle_server_maintenance to take over the scoreboard slot before the
> previous process has completely exited, bypassing the ServerLimit throttle.
>
> This raises several ideas for improvement:
>
> * Should we do clean_child_exit(APEXIT_CHILDSICK or CHILDFATAL) for this
> error?  We have a previous fix to detect accept mutex failures during
> restarts and tone down the error messages.  I don't recall seeing any false
> error messages after that fix.  We could also use requests_this_child to
> detect if this process has ever successfully served a request, and only do
> the clean_child_exit if it hasn't.
>

So nasty failures prior to successfully accepting a connection bypass
squatting?  Good.

CHILDSICK or CHILDFATAL in that case?  In this example it probably wasn't
going to get any better.  However, I think it is reasonably likely that
child process n+1 encounters some sort of resource limit, so CHILDSICK seems
better.


>
> * Should we yank the squatting logic?  I think it is doing us more harm
> than good.  IIRC it was put in to make the server respond faster when the
> workload is spikey.  A more robust solution may be to set Min and
> MaxSpareThreads farther apart and allow ServerLimit to be enforced
> unconditionally.  disclaimer: I created ps->quiescing, so I was an
> accomplice.
>

My understanding is that squatting is required to deal with long-running
requests that keep a child trying to exit from actually exiting, thus tying
up a scoreboard process indefinitely.  (We've had the assumption that we
didn't want to drastically overallocate scoreboard to accommodate a bunch of
children with only a few threads handling requests.)

Is it reasonable to have up to MaxClients worth of squatting like we have
now?  (I think that is what we allow.)  No, I don't think so.

Should we axe squatting, respect ServerLimit, and thus make the admin raise
ServerLimit to accommodate exiting processes which are handling long-running
requests (and waste a bit of shared memory at the same time)?  Maybe ;)
 That change seems a bit drastic, but I'm probably just scared of another
long period of time before I halfway understand how it behaves in the real
world.



> * Does it make sense to fork more than MaxSpareThreads worth of child
> processes at a time?  MaxSpareThreads was 75 in this case, but we tried to
> fork at least 600 threads (same as MaxClients) worth of child processes in
> one pass of perform_idle_server_maintenance.
>

That's a good idea.


>
> This applies to worker and event; some of it may also apply to prefork.
> I'd appreciate thoughts and suggestions before committing anything.
>
> Thanks,
> Greg
>



-- 
Born in Roswell... married an alien...