You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Alexander Lazic <al...@none.at> on 2006/04/09 00:10:21 UTC
Bug in 2.0.56-dev
Hi,
on 05-04-2006 i have get the latest snapshots from:
http://cvs.apache.org/snapshots/
httpd-2.0.x_20060405102636.tar.gz
apr-0.9.x_20060405102142.tar.gz
apr-util-0.9.x_20060405102200.tar.gz
Build it on solaris 10 x86 with gcc.
./configure --prefix=/web/apache-2.0.56 --with-mpm=worker --enable-so \
--enable-rewrite --disable-userdir --disable-imap --disable-cgi \
--disable-cgid --disable-asis --enable-http --enable-ssl\
--enable-headers --enable-expires --enable-deflate \
--disable-charset-lite --enable-rule=SHARED_CORE \
CFLAGS='-DSHARED_CORE -DUSE_SSL' CPPFLAGS='-DSHARED_CORE -DUSE_SSL' \
--with-ssl=/usr/local/ssl --enable-proxy --enable-proxy-http
After 'make install' i started apache, then some seconds later i got the
message '...MaxClients reached...' but there was no entry in the access
log, and nobody have make a request to this server.
What i have also see httpd forks StartServers-processes.
Is this a known bug?!
regards
Alex
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Jeff Trawick <tr...@gmail.com>.
On 5/2/06, Chris Darroch <ch...@pearsoncmg.com> wrote:
> If you can bear with me for a day or two more, I should have
> a collection of patches ready. These tackle the issue by
> tracking the start and listener threads in a nice new spot in
> the scoreboard, and also clean up various issues and bugs relating
> to fork(), ThreadLimit, ServerLimit, MaxClients, etc. They also
> tackle some issues raised in various XXX comments and clean up
> some stale cruft in scoreboard.h.
Note that perform_idle_server_maintenance() can't depend on the child
process to put stuff in the scoreboard in order to avoid a fork bomb.
Assume fork() stalls for 30 seconds in the child before returning to
our code, and make sure there is no opportunity for ramping up the
number of processes while we wait for a child to get a chance to
initialize.
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Chris Darroch <ch...@pearsoncmg.com>.
Jeff Trawick wrote:
> On 5/1/06, Greg Ames <gr...@apache.org> wrote:
>
>> after more thought, there is a simpler patch that should do the job. the key to both of
>> these is how threads in SERVER_DEAD state with a pid in the scoreboard are treated. this
>> means that p_i_s_m forked on a previous timer pop but some thread never made it into
>> SERVER_STARTING state.
>>
>> the difference: this patch just counts those potential threads as idle, and allows
>> MinSpareThreads worth of processes to be forked before putting on the brakes. the
>> previous patch pauses the forking immediately when the strange situation is detected but
>> requires more code and a new variable.
>
> new patch is fine with me; I think we've lost our other interested
> parties on this thread anyway ;)
I'm still here! I had to be away from the keyboard most of last
week, so have only returned to this thread (pun intended, alas) recently.
I also found myself fixing a variety of other things that turned up
in the vicinity of this issue; patches forthcoming on those subjects.
This is indeed a very straightforward fix; I've been trying to
ponder its consequences overnight. The questions I've got (no
answers yet, just thought I'd ping so you know I'm here) would be
(a) whether you want to allow for SERVER_GRACEFUL as well to be
counted as idle, and (b) whether there's any reason to want to
not proceed through the if (any_dead_threads && ...) logic that
follows in p_i_s_m().
If you can bear with me for a day or two more, I should have
a collection of patches ready. These tackle the issue by
tracking the start and listener threads in a nice new spot in
the scoreboard, and also clean up various issues and bugs relating
to fork(), ThreadLimit, ServerLimit, MaxClients, etc. They also
tackle some issues raised in various XXX comments and clean up
some stale cruft in scoreboard.h.
Chris.
--
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263 E4DE C8E3 FA36 366A 375B
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Jeff Trawick <tr...@gmail.com>.
On 5/1/06, Greg Ames <gr...@apache.org> wrote:
> Jeff Trawick wrote:
>
> after more thought, there is a simpler patch that should do the job. the key to both of
> these is how threads in SERVER_DEAD state with a pid in the scoreboard are treated. this
> means that p_i_s_m forked on a previous timer pop but some thread never made it into
> SERVER_STARTING state.
>
> the difference: this patch just counts those potential threads as idle, and allows
> MinSpareThreads worth of processes to be forked before putting on the brakes. the
> previous patch pauses the forking immediately when the strange situation is detected but
> requires more code and a new variable.
new patch is fine with me; I think we've lost our other interested
parties on this thread anyway ;)
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Greg Ames <gr...@apache.org>.
Jeff Trawick wrote:
> I have been working with a user on one of these fork bomb scenarios
> and assumed it was the child_init hook. But after giving them a test
> fix that relies on a child setting scoreboard fields in child_main
> before child-init hooks run, and also adds some debugging traces
> related to calling child-init hooks, it is clear that their stall
> occurs BEFORE the child-init hook. Which leaves a stretch of fairly
> simple code.
>
> (Best theory is bad stuff happening in an atfork handler registred by
> a third-party module or some library it uses. But that's besides the
> point.)
after more thought, there is a simpler patch that should do the job. the key to both of
these is how threads in SERVER_DEAD state with a pid in the scoreboard are treated. this
means that p_i_s_m forked on a previous timer pop but some thread never made it into
SERVER_STARTING state.
the difference: this patch just counts those potential threads as idle, and allows
MinSpareThreads worth of processes to be forked before putting on the brakes. the
previous patch pauses the forking immediately when the strange situation is detected but
requires more code and a new variable. I'm leaning toward this one because it is simpler.
opinions?
Greg
--- server/mpm/worker/worker.c (revision 398659)
+++ server/mpm/worker/worker.c (working copy)
@@ -1422,7 +1422,7 @@
*/
if (ps->pid != 0) { /* XXX just set all_dead_threads in outer for
loop if no pid? not much else matters */
- if (status <= SERVER_READY && status != SERVER_DEAD &&
+ if (status <= SERVER_READY &&
!ps->quiescing &&
ps->generation == ap_my_generation) {
++idle_thread_count;
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Jeff Trawick <tr...@gmail.com>.
On 4/11/06, Chris Darroch <ch...@pearsoncmg.com> wrote:
>
> In my case, the problem relates to how long the child_init phase
> takes to execute. I can "tune" this by raising DBDMin (and DBDKeep)
> so that mod_dbd attempts to open increasingly large numbers of
> DB connections during child_init. With DBDMin set to 0 or 1,
> all is well; no funny behaviour. Up at DBDMin and DBDKeep at 3,
> that's when (for me) things go pear-shaped.
I have been working with a user on one of these fork bomb scenarios
and assumed it was the child_init hook. But after giving them a test
fix that relies on a child setting scoreboard fields in child_main
before child-init hooks run, and also adds some debugging traces
related to calling child-init hooks, it is clear that their stall
occurs BEFORE the child-init hook. Which leaves a stretch of fairly
simple code.
(Best theory is bad stuff happening in an atfork handler registred by
a third-party module or some library it uses. But that's besides the
point.)
Greg Ames and I chatted about this today and (98% Greg) came up with
the attached patch, which does not rely on the child storing anything
in the scoreboard in order to avoid the fork bomb.
I've played with it a bit on 2.0 but not trunk, which this patch is
for. But the code's pretty much the same ;)
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Chris Darroch <ch...@pearsoncmg.com>.
Colm:
>> The worker and event MPMs would use these to track their
>> non-worker threads; and the parent process for these MPMs could
>> monitor them as per option C to decide when the child process's
>> workers were ready to be counted.
>
> +1, I think this could be very useful, I came accross the same kind of
> problems when looking at handling the odd mod_cgid race conditions when
> doing graceful stops, but in the end didn't solve them (we still have
> slightly different death semantics between prefork and worker-like MPM's
> for cgi termination).
Great, I'll get started on this first thing tomorrow. I'm
away Thurs-Mon this week but with luck will have some patches
before I go for all these co-mingled issues.
>> Finally, a question without checking the code first ... I notice
>> that worker.c has the following, taken from prefork.c:
>>
>> /* fork didn't succeed. Fix the scoreboard or else
>> * it will say SERVER_STARTING forever and ever
>> */
>> ap_update_child_status_from_indexes(slot, 0, SERVER_DEAD, NULL);
>>
>> Is that right? Should it perhaps cycle through and set all
>> non-DEAD workers to DEAD? I'm thinking that in prefork,
>> threads_limit is set to 1, so mod_status only checks the first
>> thread in a slot, but here it'll check all of them.
>
> I'm not sure what you mean. As in, all potential workers globally? That
> child isn't going to have any actual workers, since child_main() never
> got run.
What I noticed is that the prefork MPM seems to consistently
use the first worker_score in the scoreboard for all its record-
keeping; that makes sense because its thread_limit is always 1.
So in make_child(), it sets that worker_score to SERVER_STARTING,
does the fork(), and if that fails, resets the worker_score
to SERVER_DEAD. No problems there.
However, the code seems to have been copied over into the
worker MPM, and that particular if-fork()-fails bit still does
the same business, even though other things have changed around
it. In particular, multiple worker_score structures are in
use for each child (ap_threads_per_child of them), and
make_child() doesn't begin by setting them all to SERVER_STARTING --
or even one of them -- so having it subsequently handle the
fork()-fails error condition by resetting the first one to
SERVER_DEAD seems incorrect.
Further, make_child() may be creating a child to replace
a gracefully exiting one, and if so, it shouldn't actually
touch any of the worker_score statuses. That's because what
start_threads() in the child does -- in the case where fork()
succeeds -- is to watch for threads in the old process that
have marked their status as SERVER_GRACEFUL and only then
initiate new workers to replace them. So in fact, if fork()
fails, such old workers may still be executing and their
status values may not yet be SERVER_GRACEFUL. As things stand,
if one of them happens to be have a scoreboard index of 0, its
value will get overwritten by make_child().
I think probably the right thing is for make_child() just
to do nothing here. That makes the same assumption the rest
of the code does, namely, that worker threads will reach their
normal termination and set their statuses to SERVER_DEAD or
SERVER_GRACEFUL; and if a child process does fail, then
the parent process's will notice in server_main_loop() and
set all of the child's threads' statuses to SERVER_DEAD.
Anyway, I'll include that change in my patchset for review;
I may of course have missed something important.
Chris.
--
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263 E4DE C8E3 FA36 366A 375B
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Sat, Apr 15, 2006 at 10:20:29AM -0400, Chris Darroch wrote:
> Re my option C, it also occurs to me that instead of squeezing
> the worker MPM's start and listener threads into extra, internal
> worker_score structures, it might be more appropriate to create a
> new section of the scoreboard with non_worker_score structures.
> These would just have pid, tid, and status fields only.
>
> The worker and event MPMs would use these to track their
> non-worker threads; and the parent process for these MPMs could
> monitor them as per option C to decide when the child process's
> workers were ready to be counted.
+1, I think this could be very useful, I came accross the same kind of
problems when looking at handling the odd mod_cgid race conditions when
doing graceful stops, but in the end didn't solve them (we still have
slightly different death semantics between prefork and worker-like MPM's
for cgi termination).
> Finally, a question without checking the code first ... I notice
> that worker.c has the following, taken from prefork.c:
>
> /* fork didn't succeed. Fix the scoreboard or else
> * it will say SERVER_STARTING forever and ever
> */
> ap_update_child_status_from_indexes(slot, 0, SERVER_DEAD, NULL);
>
> Is that right? Should it perhaps cycle through and set all
> non-DEAD workers to DEAD? I'm thinking that in prefork,
> threads_limit is set to 1, so mod_status only checks the first
> thread in a slot, but here it'll check all of them.
I'm not sure what you mean. As in, all potential workers globally? That
child isn't going to have any actual workers, since child_main() never
got run.
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Chris Darroch <ch...@pearsoncmg.com>.
Hi --
> for (i = 0; i < ap_threads_per_child; i++) {
> if (status != SERVER_GRACEFUL && status != SERVER_DEAD) {
> ap_update_child_status_from_indexes(slot, i, SERVER_INIT, NULL);
> }
> }
[snip]
> ... after make_child() does its check for != GRACEFUL and != DEAD ...
After heading back to bed I realized this should be, of course,
if (status == SERVER_GRACEFUL || status == SERVER_DEAD)
Never cut-'n'-paste when you're tired!
Re my option C, it also occurs to me that instead of squeezing
the worker MPM's start and listener threads into extra, internal
worker_score structures, it might be more appropriate to create a
new section of the scoreboard with non_worker_score structures.
These would just have pid, tid, and status fields only.
The worker and event MPMs would use these to track their
non-worker threads; and the parent process for these MPMs could
monitor them as per option C to decide when the child process's
workers were ready to be counted.
It would also solve an outstanding problem for me with a
module I maintain that uses its own private threads; there's
nowhere in the scoreboard that I know of where I can register
them. That might also intersect with this conversation:
http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=112902736905643&w=2
A possible set of calls for managing these might be:
num = ap_register_non_worker_thread(slot, tid, status)
ap_update_non_worker_status_from_indexes(slot, num, status)
ap_unregister_non_worker_thread(slot, num)
ret = ap_get_scoreboard_non_worker(slot, num, &status)
The child process registers its non-worker threads, making
sure that it registers one of them as STARTING immediately after
ap_reopen_scoreboard() so that it precedes ap_run_child_init().
This will be guaranteed to go into the [slot][0] structure.
It can then set that to READY once it's ready for its workers
to be counted individually.
Then the parent process can monitor that structure's status
using ap_get_scoreboard_non_worker(), and do the "wait till
all workers are running" thing in p_i_s_m(). Meanwhile
mod_status can call ap_get_scoreboard_non_worker until it
returns an error code and then it'll know it's checked all
the registered non-workers; these can be displayed in the report.
Finally, a question without checking the code first ... I notice
that worker.c has the following, taken from prefork.c:
/* fork didn't succeed. Fix the scoreboard or else
* it will say SERVER_STARTING forever and ever
*/
ap_update_child_status_from_indexes(slot, 0, SERVER_DEAD, NULL);
Is that right? Should it perhaps cycle through and set all
non-DEAD workers to DEAD? I'm thinking that in prefork,
threads_limit is set to 1, so mod_status only checks the first
thread in a slot, but here it'll check all of them.
Chris.
--
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263 E4DE C8E3 FA36 366A 375B
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Chris Darroch <ch...@pearsoncmg.com>.
Hi --
Someone tried to send me a fax in the middle of the night,
so I've been up for a while and I think I've realized there are
several subtle contention issues involved with any fix for this issue.
First of all, I should note that my initial patch in Bugzilla
has a flaw; it needs an else clause around most of the logic in
perform_idle_server_maintenance(). With that in place, it works
pretty well for me. However, there's a theoretical race condition
that I'll describe below.
To summarize where we stand, we're trying to avoid the situation
where the parent process creates a worker MPM child, and then before
the child gets a chance to run very far, the parent's
perform_idle_server_maintenance() routine (hereafter p_i_s_m(),
per Greg Ames) checks and sees no/few running worker threads
for that child and spawns another child.
Jeff Trawick suggested a process_score.mpm_state field used
much the way my patch uses it. The parent sets it to SERVER_STARTING
after creating the child in make_child(), and the child sets it
to SERVER_READY when some workers have been created. Then
p_i_s_m() can check it and, if not READY, bypass the count of
workers and assume "they'll be ready soon".
Greg Ames preferred the other approach I initially mentioned;
using the worker_score.status field only and adding a new
status value; let's call it SERVER_INIT. In this case,
either the parent or the child would perform this logic shortly
after the child was created:
for (i = 0; i < ap_threads_per_child; i++) {
if (status != SERVER_GRACEFUL && status != SERVER_DEAD) {
ap_update_child_status_from_indexes(slot, i, SERVER_INIT, NULL);
}
}
Then p_i_s_m() would accept this worker state as meaning "I'll
be ready soon".
Sticking with the latter approach, I'll try to explain the
contention issue I see. Fundamentally, it comes about because
normally only the parent changes process_score fields, while only
the child changes worker_score fields (synchronizing between
its threads and sometimes checking for an GRACEFUL or DEAD
status from another child's threads).
Suppose the child runs the SERVER_INIT setup code above
in child_main(), before ap_run_child_init(). It seems to me
that it's at least theoretically possible that the parent's
p_i_s_m() still gets scheduled first by the OS and concludes
that another child needs to be created.
Suppose instead that the parent does the setup code in
make_child(), right after successfully forking. Then p_i_s_m()
can't see un-INIT worker status values. However it's conceivable
that right after make_child() does its check for != GRACEFUL and
!= DEAD that the child's threads get scheduled and set the worker
status to READY; this then gets clobbered forever by the parent.
One option might be to use apr_atomic_cas32() routines on the
worker status fields -- perhaps only required in make_child(), even.
That might be viable, but I confess I can't see all the
ramifications right now, and those status fields are updated
in lots of places via ap_update_child_status_from_indexes().
Let's call this option A.
Reviewing the former approach, the same kinds of contention
exist because make_child() sets mpm_state to STARTING and
then child's start_threads() sets it to READY; if scheduling
causes them to be reversed, the child will always be treated
as having a full complement of "about to be ready" workers.
It might be straightforward to use apr_atomic_cas32() here
since we're the only users of the field. Let's call this
option B. Still, based on scoreboard.h, I think it makes sense
if the parent is the only one to edit process_score fields:
/* stuff which the parent generally writes and the children rarely read */
typedef struct process_score process_score;
Option C is somewhat more complex, but perhaps handles a
number of different issues. Both a process_score.mpm_state
and a worker_score.internal field are added.
The child process maintains two additional worker_score
structures at ap_threads_per_child and ap_threads_per_child+1,
these are marked internal and correspond to the listener
and start threads. They just set their status to STARTING,
READY, GRACEFUL, and DEAD as appropriate. No other changes
to the child process are needed, I believe.
The parent process sets mpm_state to STARTING in make_child(),
and follows the logic of option B, except that atomics aren't
needed. That's because in p_i_s_m() it checks the child's
start thread's internal worker_score structure, and if it is READY,
then it flips the mpm_state to READY. Otherwise, if mpm_state
is STARTING, it does the bypass logic and assumes that all
workers "will be ready soon".
This seems to me to have the advantage that only the parent
edits process_score and only the child edits worker_score, which
is conceptually clearer and reduces the ways contention issues
can creep in. It does introduce some minor changes to make
sure there's room in the scoreboard for the extra two internal
threads and to make mod_status not see them, perhaps by
checking the internal flag in ap_get_scoreboard_worker().
It might also lead the way to dealing with this comment about
how to list non-worker threads in the scoreboard:
/* What state should this child_main process be listed as in the
* scoreboard...?
...
* This state should be listed separately in the scoreboard, in some kind
* of process_status, not mixed in with the worker threads' status.
* "life_status" is almost right, but it's in the worker's structure, and
* the name could be clearer. gla
*/
I'm going to chew over these options this weekend and
hopefully have a patchset next week. Advice welcome!
Chris.
--
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263 E4DE C8E3 FA36 366A 375B
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Greg Ames <gr...@apache.org>.
Jeff Trawick wrote:
>>>There are problems accounting for child processes which are trying to
>>>initialize that result in the parent thinking it needs to create more
>>>children. The less harmful flavor is when it thinks (incorrectly) it
>>>is already at MaxClients and issues the "reached MaxClients" message.
working on a patch to improve the message.
>>>More disturbing is when MaxClients is very high and the parent keeps
>>>creating new children using exponential ramp-up. That can be very
>>>painful.
>> However, if the child processes are starting "slowly" because
>>ap_run_child_init() in child_main() is taking its time, then
>>start_threads() hasn't even been run yet, so the threads aren't
>>marked SERVER_STARTING -- they're just set to 0 as the default
>>value. But 0 == SERVER_DEAD, so the main process sees a lot
>>of dead worker threads and begins spawning new child processes,
>>up to MaxClients/ThreadsPerChild in the worst case.
>> I considered wedging another thread status into the
>>scoreboard, between SERVER_DEAD (the initial value) and
>>SERVER_STARTING. The make_child() would set all the thread
>>slots to this value and start_threads() would later flip them
>>to SERVER_STARTING after actually creating the worker threads.
that would be the cleanest fix IMO. I dislike having state information scattered all over
the place due to bad experiences with such in a previous job. whether or not it is
practical to have a single state field in a stable httpd release is another question.
>> That would have various ripple effects on other bits of
>>httpd, though, like mod_status and other MPMs, etc.sh
>
> In other words, breaks binary compatibility...
so the binary state values in the scoreboard are an API? if we never declared they
weren't an API maybe they are by default. but that really ties our hands. syncing
mod_status and the event MPM doesn't bother me much.
> Other modules should see the threads in SERVER_STARTING state anyway.
> IOW, I think we should set state to SERVER_STARTING before we do any
> potentially-lengthy work like running child-init hooks so that the
> state as seen from the outside makes sense. That also means resetting
> the state if something fails (e.g., pthread_create()).
>
> But that isn't needed for proper operation of the MPM, which is what
> we're after at the moment... But it would be great to be able to see
> from mod_status that a child is taking way too long in the
> SERVER_STARTING state.
yep, or whatever we call this state.
So instead
> I was considering adding something to process_score for this issue but
> I decided against it, hopefully for an bogus reason -- binary
> compatibility breakage.
>
> This isn't binary compatibility breakage since we provide
> ap_get_scoreboard_process() for modules to retrieve a process_score
> structure, and if fields get added to the end for the use of the MPM
> then no worries since we don't support modules creating their own
> process_score structures and stuffing them in the scoreboard.
>
> (confirmation from the crowd?)
>
> Instead of "unsigned char status" I'd prefer something like
>
> apr_int32_t mpm_state; /* internal state for MPM; meaning may change
> * in the future, so not for use
> by other modules
> */
adding something to the end of process score is probably the best solution for stable
releases. but in trunk I would prefer to see:
* the worker score state field be the one true state field for purposes of managing
processes/threads to simplify logic + developer understanding,
* the preceding marked as internal, subject to change,
* an appropriate mmn bump (not an expert here),
* the state values spread out some with some reserved values interspersed.
the latter would let us un-reserve a state value in the right range if we need it in the
future without as much potential disruption. the downside is that arrays indexed by state
values would grow. no big deal.
I volunteer to do this work in trunk if we have consensus that it's the right thing.
> If a particular MPM wants to store SERVER_STARTING/SERVER_DEAD/etc. then fine.
>
>
>> During this period, while the new child process is running
>>ap_run_child_init() and friends, perform_idle_server_maintenance()
>>just counts that child process's worker threads as all being
>>effectively in SERVER_STARTING mode. Once the process_score.status
>>field changes to SERVER_READY, perform_idle_server_maintenance()
>>begins to look at the individual thread status values.
>>
>> Any thoughts? The patch in Bugzilla doesn't address other
>>MPMs that might see the same behaviour (event, and maybe prefork?)
>>
>>http://issues.apache.org/bugzilla/show_bug.cgi?id=39275
> A gracefully exiting process has lost its process score field and
> gradually loses its worker_score fields as well. Gracefully exiting
> threads aren't counted as active or idle.
> I think this means we can create a new process to make up for
> gracefully exiting threads that we won't necessarily need once they
> finish and new threads in that process scoreboard slot take over.
> Unavoidable, since gracefully exiting threads can take forever.
right. not counting gracefully exiting threads as idle helps p_i_s_m start new processes
sooner.
Greg
Re: [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Jeff Trawick <tr...@gmail.com>.
On 4/11/06, Chris Darroch <ch...@pearsoncmg.com> wrote:
> Hi --
>
> Alexander Lazic wrote:
>
> >> After 'make install' i started apache, then some seconds later i got the
> >> message '...MaxClients reached...' but there was no entry in the access
> >> log, and nobody have make a request to this server.
>
> Jeff Trawick wrote:
>
> > There are problems accounting for child processes which are trying to
> > initialize that result in the parent thinking it needs to create more
> > children. The less harmful flavor is when it thinks (incorrectly) it
> > is already at MaxClients and issues the "reached MaxClients" message.
> > More disturbing is when MaxClients is very high and the parent keeps
> > creating new children using exponential ramp-up. That can be very
> > painful.
>
> I have been seeing something similar with 2.2.0 using the worker
> MPM, where with the following settings, I get over 10 child processes
> initializing immediately (e.g., up to 15), and then they drop back to
> 10. I see the "server reached MaxClients" message as well right
> after httpd startup, although nothing is connecting yet.
>
> <IfModule mpm_worker_module>
> StartServers 10
> MaxClients 150
> MinSpareThreads 25
> MaxSpareThreads 100
> ThreadsPerChild 10
> </IfModule>
>
> In my case, the problem relates to how long the child_init phase
> takes to execute. I can "tune" this by raising DBDMin (and DBDKeep)
> so that mod_dbd attempts to open increasingly large numbers of
> DB connections during child_init. With DBDMin set to 0 or 1,
> all is well; no funny behaviour. Up at DBDMin and DBDKeep at 3,
> that's when (for me) things go pear-shaped.
>
> In server/mpm/worker/worker.c, after make_child() creates a
> child process it immediately sets the scoreboard parent slot's pid
> value. The main process goes into server_main_loop() and begins
> executing perform_idle_server_maintenance() every second; this
> looks at any process with a non-zero pid in the scoreboard and
> assumes that any of its worker threads marked SERVER_DEAD are,
> in fact, dead.
>
> However, if the child processes are starting "slowly" because
> ap_run_child_init() in child_main() is taking its time, then
> start_threads() hasn't even been run yet, so the threads aren't
> marked SERVER_STARTING -- they're just set to 0 as the default
> value. But 0 == SERVER_DEAD, so the main process sees a lot
> of dead worker threads and begins spawning new child processes,
> up to MaxClients/ThreadsPerChild in the worst case. In this case,
> when no worker threads have started yet, but all possible child
> processes have been spawned (and are working through their
> child_init phases), then the following is true and the
> "server reached MaxClients" message is printed, even though
> the server hasn't started accepting connections yet:
>
> else if (idle_thread_count < min_spare_threads) {
> /* terminate the free list */
> if (free_length == 0) {
>
> I considered wedging another thread status into the
> scoreboard, between SERVER_DEAD (the initial value) and
> SERVER_STARTING. The make_child() would set all the thread
> slots to this value and start_threads() would later flip them
> to SERVER_STARTING after actually creating the worker threads.
>
> That would have various ripple effects on other bits of
> httpd, though, like mod_status and other MPMs, etc.
In other words, breaks binary compatibility...
Other modules should see the threads in SERVER_STARTING state anyway.
IOW, I think we should set state to SERVER_STARTING before we do any
potentially-lengthy work like running child-init hooks so that the
state as seen from the outside makes sense. That also means resetting
the state if something fails (e.g., pthread_create()).
But that isn't needed for proper operation of the MPM, which is what
we're after at the moment... But it would be great to be able to see
from mod_status that a child is taking way too long in the
SERVER_STARTING state.
> So instead
> I tried adding a status field to the process_score scoreboard
> structure, and making the following changes to worker.c such that
> this field is set by make_child to SERVER_STARTING and then
> changed to SERVER_READY once the start thread that runs
> start_threads() has done its initial work.
I was considering adding something to process_score for this issue but
I decided against it, hopefully for an bogus reason -- binary
compatibility breakage.
This isn't binary compatibility breakage since we provide
ap_get_scoreboard_process() for modules to retrieve a process_score
structure, and if fields get added to the end for the use of the MPM
then no worries since we don't support modules creating their own
process_score structures and stuffing them in the scoreboard.
(confirmation from the crowd?)
Instead of "unsigned char status" I'd prefer something like
apr_int32_t mpm_state; /* internal state for MPM; meaning may change
* in the future, so not for use
by other modules
*/
If a particular MPM wants to store SERVER_STARTING/SERVER_DEAD/etc. then fine.
> During this period, while the new child process is running
> ap_run_child_init() and friends, perform_idle_server_maintenance()
> just counts that child process's worker threads as all being
> effectively in SERVER_STARTING mode. Once the process_score.status
> field changes to SERVER_READY, perform_idle_server_maintenance()
> begins to look at the individual thread status values.
>
> Any thoughts? The patch in Bugzilla doesn't address other
> MPMs that might see the same behaviour (event, and maybe prefork?)
>
> http://issues.apache.org/bugzilla/show_bug.cgi?id=39275
>
> It also doesn't necessarily play ideally well with the fact that
> new child processes can gradually take over thread slots in
> the scoreboard from a gracefully exiting old process -- the
> count of idle threads for that process will be pegged (only
> by perform_idle_server_maintenance()) at ap_threads_per_child
> until the new process creates its first new worker thread.
> But, that may be just fine.... I'll keep poking around and
> testing and maybe a better idea will present itself.
A gracefully exiting process has lost its process score field and
gradually loses its worker_score fields as well. Gracefully exiting
threads aren't counted as active or idle.
I think this means we can create a new process to make up for
gracefully exiting threads that we won't necessarily need once they
finish and new threads in that process scoreboard slot take over.
Unavoidable, since gracefully exiting threads can take forever.
[PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Posted by Chris Darroch <ch...@pearsoncmg.com>.
Hi --
Alexander Lazic wrote:
>> After 'make install' i started apache, then some seconds later i got the
>> message '...MaxClients reached...' but there was no entry in the access
>> log, and nobody have make a request to this server.
Jeff Trawick wrote:
> There are problems accounting for child processes which are trying to
> initialize that result in the parent thinking it needs to create more
> children. The less harmful flavor is when it thinks (incorrectly) it
> is already at MaxClients and issues the "reached MaxClients" message.
> More disturbing is when MaxClients is very high and the parent keeps
> creating new children using exponential ramp-up. That can be very
> painful.
I have been seeing something similar with 2.2.0 using the worker
MPM, where with the following settings, I get over 10 child processes
initializing immediately (e.g., up to 15), and then they drop back to
10. I see the "server reached MaxClients" message as well right
after httpd startup, although nothing is connecting yet.
<IfModule mpm_worker_module>
StartServers 10
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 100
ThreadsPerChild 10
</IfModule>
In my case, the problem relates to how long the child_init phase
takes to execute. I can "tune" this by raising DBDMin (and DBDKeep)
so that mod_dbd attempts to open increasingly large numbers of
DB connections during child_init. With DBDMin set to 0 or 1,
all is well; no funny behaviour. Up at DBDMin and DBDKeep at 3,
that's when (for me) things go pear-shaped.
In server/mpm/worker/worker.c, after make_child() creates a
child process it immediately sets the scoreboard parent slot's pid
value. The main process goes into server_main_loop() and begins
executing perform_idle_server_maintenance() every second; this
looks at any process with a non-zero pid in the scoreboard and
assumes that any of its worker threads marked SERVER_DEAD are,
in fact, dead.
However, if the child processes are starting "slowly" because
ap_run_child_init() in child_main() is taking its time, then
start_threads() hasn't even been run yet, so the threads aren't
marked SERVER_STARTING -- they're just set to 0 as the default
value. But 0 == SERVER_DEAD, so the main process sees a lot
of dead worker threads and begins spawning new child processes,
up to MaxClients/ThreadsPerChild in the worst case. In this case,
when no worker threads have started yet, but all possible child
processes have been spawned (and are working through their
child_init phases), then the following is true and the
"server reached MaxClients" message is printed, even though
the server hasn't started accepting connections yet:
else if (idle_thread_count < min_spare_threads) {
/* terminate the free list */
if (free_length == 0) {
I considered wedging another thread status into the
scoreboard, between SERVER_DEAD (the initial value) and
SERVER_STARTING. The make_child() would set all the thread
slots to this value and start_threads() would later flip them
to SERVER_STARTING after actually creating the worker threads.
That would have various ripple effects on other bits of
httpd, though, like mod_status and other MPMs, etc. So instead
I tried adding a status field to the process_score scoreboard
structure, and making the following changes to worker.c such that
this field is set by make_child to SERVER_STARTING and then
changed to SERVER_READY once the start thread that runs
start_threads() has done its initial work.
During this period, while the new child process is running
ap_run_child_init() and friends, perform_idle_server_maintenance()
just counts that child process's worker threads as all being
effectively in SERVER_STARTING mode. Once the process_score.status
field changes to SERVER_READY, perform_idle_server_maintenance()
begins to look at the individual thread status values.
Any thoughts? The patch in Bugzilla doesn't address other
MPMs that might see the same behaviour (event, and maybe prefork?)
http://issues.apache.org/bugzilla/show_bug.cgi?id=39275
It also doesn't necessarily play ideally well with the fact that
new child processes can gradually take over thread slots in
the scoreboard from a gracefully exiting old process -- the
count of idle threads for that process will be pegged (only
by perform_idle_server_maintenance()) at ap_threads_per_child
until the new process creates its first new worker thread.
But, that may be just fine.... I'll keep poking around and
testing and maybe a better idea will present itself.
Chris.
Re: Bug in 2.0.56-dev
Posted by Jeff Trawick <tr...@gmail.com>.
On 4/10/06, Alexander Lazic <al...@none.at> wrote:
> Hi,
>
> On Mon 10.04.2006 09:52, Jeff Trawick wrote:
> >On 4/8/06, Alexander Lazic <al...@none.at> wrote:
> >>
> >> Is this a known bug?!
> >
> >y; I've been looking at this type of problem for a few days...
> >hopefully I can post a patch before long
>
> Thanx for your positive answer ;-)
>
> >Apache works fine except for that bogus message, right?
>
> Hm yes but the multipart-post Bug# 37145 seams to be still a problem :-(
fair enough; I should have asked about any ill behavior also
attributed to the bogus "reached MaxClients" message ;)
Re: Bug in 2.0.56-dev
Posted by Alexander Lazic <al...@none.at>.
Hi,
On Mon 10.04.2006 09:52, Jeff Trawick wrote:
>On 4/8/06, Alexander Lazic <al...@none.at> wrote:
>>
>> Is this a known bug?!
>
>y; I've been looking at this type of problem for a few days...
>hopefully I can post a patch before long
Thanx for your positive answer ;-)
>Apache works fine except for that bogus message, right?
Hm yes but the multipart-post Bug# 37145 seams to be still a problem :-(
For now i have switched to the 2.0.55 with the second patch and after 20
or 2 Uploads some headers are lost.
Regards
Alex
Re: Bug in 2.0.56-dev
Posted by Jeff Trawick <tr...@gmail.com>.
On 4/8/06, Alexander Lazic <al...@none.at> wrote:
> on 05-04-2006 i have get the latest snapshots from:
>
> http://cvs.apache.org/snapshots/
>
> httpd-2.0.x_20060405102636.tar.gz
> apr-0.9.x_20060405102142.tar.gz
> apr-util-0.9.x_20060405102200.tar.gz
>
> Build it on solaris 10 x86 with gcc.
>
> ./configure --prefix=/web/apache-2.0.56 --with-mpm=worker --enable-so \
> --enable-rewrite --disable-userdir --disable-imap --disable-cgi \
> --disable-cgid --disable-asis --enable-http --enable-ssl\
> --enable-headers --enable-expires --enable-deflate \
> --disable-charset-lite --enable-rule=SHARED_CORE \
> CFLAGS='-DSHARED_CORE -DUSE_SSL' CPPFLAGS='-DSHARED_CORE -DUSE_SSL' \
> --with-ssl=/usr/local/ssl --enable-proxy --enable-proxy-http
>
> After 'make install' i started apache, then some seconds later i got the
> message '...MaxClients reached...' but there was no entry in the access
> log, and nobody have make a request to this server.
>
> What i have also see httpd forks StartServers-processes.
>
> Is this a known bug?!
y; I've been looking at this type of problem for a few days...
hopefully I can post a patch before long
Apache works fine except for that bogus message, right?
There are problems accounting for child processes which are trying to
initialize that result in the parent thinking it needs to create more
children. The less harmful flavor is when it thinks (incorrectly) it
is already at MaxClients and issues the "reached MaxClients" message.
More disturbing is when MaxClients is very high and the parent keeps
creating new children using exponential ramp-up. That can be very
painful.