You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Justin <jb...@dslreports.com> on 2000/06/23 04:38:53 UTC
MaxChildRequests and modperl - issue
The backend/frontend setup works well, but for the following problem
that I think is a risk for loaded sites:
I set MaxChildRequests to 500, to effectively clean house every
now and again, as is recommended.. I have about 12 backend
modperl servers, and that handles the load fine. (about 250 front
ends).
What happens is, though, they all tick up towards MaxChildRequests,
and they pretty much start getting cloe to 500 over the period of
less than a minute. (about 10-20 minutes after a server restart).
What I think happens is the children die after their last request,
and apache does not kick off a new child straight away.. MinFree is
set to 2 .. as 12 becomes 11 becomes 10 becomes 9, my backend is
getting less and less powerful and more and more swamped. When
Apache does wakeup and spawn a new child, it takes many seconds
to interpret all the (large amount) of perl code and modules that
are not in the parent... up to 10 seconds now since its only getting
a fraction of the box.. this gives the unlucky user a dead browser.
Worse, the remaining alive servers are dieing faster now as they
handle more and more of the load, and rush towards 500 to contribute
to the same traffic jam of booting children.
The effect is essentially that what starts out as random child
death/restart collapses to all backend processes rebooting at the
same time, and an effectively dead server for about 20 seconds.
Any easy fixes to this? am I the first to find this? Am i missing
something obvious? Perhaps having my own ChildRequest counter and
dieing myself, but more randomly, is a better way?
thanks
-Justin
dslreports.com
Re: MaxChildRequests and modperl - issue
Posted by Vivek Khera <kh...@kciLink.com>.
>>>>> "J" == Justin <jb...@dslreports.com> writes:
J> and apache does not kick off a new child straight away.. MinFree is
J> set to 2 .. as 12 becomes 11 becomes 10 becomes 9, my backend is
J> getting less and less powerful and more and more swamped. When
Make your min-free servers higher, and set max-free servers a bit
higher than that. If it is taking 10 seconds to fire up a new one,
then you should have at least 10 seconds worth of spares for your
load.
The exact values of these parameters can only be gotten by statistical
analysis of your specific load. The apache status page generated from
within apache is a good help to see the max load and how often you got
there.
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: khera@kciLink.com Rockville, MD +1-301-545-6996
GPG & MIME spoken here http://www.khera.org/~vivek/
Re: MaxChildRequests and modperl - issue
Posted by Vivek Khera <kh...@kciLink.com>.
>>>>> "DH" == David Hodgkinson <da...@hodgkinson.org> writes:
DH> If you take a large script, throw it at Apache::Registry, then you'll
DH> be compiling the script every time the a diaghter respawns.
No, it doesn't happen when the child process is "spawned" (what is
this, OS/2? We "fork" around here!) nothing is recompiled. Your
Registry program will be compiled when it is requested. No sooner, no
later.
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: khera@kciLink.com Rockville, MD +1-301-545-6996
GPG & MIME spoken here http://www.khera.org/~vivek/
Re: MaxChildRequests and modperl - issue
Posted by David Hodgkinson <da...@hodgkinson.org>.
Stas Bekman <st...@stason.org> writes:
> > Stas Bekman <st...@stason.org> writes:
> >
> >
> > > I think it doesn't matter whether it's a script or a module.
> >
> > So if startup.pl contains:
> >
> > use CGI ;
> > use DBI ;
> > use My::Application ;
> >
> > Then there's no gain?
>
> Of course there is!
>
> I'm talking about a completely diffent thing. Let me repeat this again.
>
> Spawning of the child process has almost nothing to do with code. When the
> process is spawned it's memory pages are _shared_ with parents, and the
> child process has almost no pages on its own.
So, we have:
- initial apache startup:
compile stuff that's in startup.pl
- the Apache::Registry script gets run:
compile the Apache::Registry script[0]
memory pages get dirtied as we touch various bits of memory
connect to any resources[1]
[0] Which, ideally is just a stub.
[1] Subsequently these connections can be cached, say in code that
knows about Apache::Registry, or for free with Apache::DBI.
Are we nearer now?
I was having a block about pre-forking threads and running a script
for the first time, not trying to be deliberately obtuse.
--
Dave Hodgkinson, http://www.hodgkinson.org
Editor-in-chief, The Highway Star http://www.deep-purple.com
Apache, mod_perl, MySQL, Sybase hired gun for, well, hire
-----------------------------------------------------------------
Re: MaxChildRequests and modperl - issue
Posted by Stas Bekman <st...@stason.org>.
> Stas Bekman <st...@stason.org> writes:
>
>
> > I think it doesn't matter whether it's a script or a module.
>
> So if startup.pl contains:
>
> use CGI ;
> use DBI ;
> use My::Application ;
>
> Then there's no gain?
Of course there is!
I'm talking about a completely diffent thing. Let me repeat this again.
Spawning of the child process has almost nothing to do with code. When the
process is spawned it's memory pages are _shared_ with parents, and the
child process has almost no pages on its own.
And something new:
The only thing that can add an overhead to the time when the process is
ready to serve the first process is if there is any code that was
registered to run at the ChildInit phase. Therefore it's quite possible
that the new processes will get ready to serve in a much slower rate if
there is some delay imposed by the code in ChildInit phase. Think about
connect_on_init which might take a few seconds to complete for if there is
some problem. This call blocks the the desire of the mod_perl soldier to
go into the battle field :0
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: MaxChildRequests and modperl - issue
Posted by David Hodgkinson <da...@hodgkinson.org>.
Stas Bekman <st...@stason.org> writes:
> I think it doesn't matter whether it's a script or a module.
So if startup.pl contains:
use CGI ;
use DBI ;
use My::Application ;
Then there's no gain?
--
Dave Hodgkinson, http://www.hodgkinson.org
Editor-in-chief, The Highway Star http://www.deep-purple.com
Apache, mod_perl, MySQL, Sybase hired gun for, well, hire
-----------------------------------------------------------------
Re: MaxChildRequests and modperl - issue
Posted by Stas Bekman <st...@stason.org>.
On 23 Jun 2000, David Hodgkinson wrote:
>
> Stas Bekman <st...@stason.org> writes:
>
> > > If you take a large script, throw it at Apache::Registry, then you'll
> > > be compiling the script every time the a diaghter respawns.
> > >
> > > If you have your script largely preloaded when Apache starts spawning
> > > daughters, then you don't have that overhead.
> >
> > That's wrong. The script won't be compiled until it will be used. When a
> > process gets spawned, it doesn't know what scripts/handlers it's going to
> > use if they are not preloaded. And if they are, they are shared.
>
> I'm being unclear.
>
> When I say "script", I mean a small piece of stub code that makes
> calls into modules, that contain the bulk of the code.
>
> These modules are then loaded in startup.pl, and are therefore
> precompiled.
>
> I'd made the jump beyond a single, monolithic ".pl".
>
> Are we getting closer?
I think it doesn't matter whether it's a script or a module.
BTW, I read somewhere at Apache performance notes at apache.org that when
the load is high Apache would pre-spawn childs first 2, then 4, then 8
etc. As long as the MaxClients is bigger than the number of processes that
are running at the given moment.
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: MaxChildRequests and modperl - issue
Posted by David Hodgkinson <da...@hodgkinson.org>.
Stas Bekman <st...@stason.org> writes:
> > If you take a large script, throw it at Apache::Registry, then you'll
> > be compiling the script every time the a diaghter respawns.
> >
> > If you have your script largely preloaded when Apache starts spawning
> > daughters, then you don't have that overhead.
>
> That's wrong. The script won't be compiled until it will be used. When a
> process gets spawned, it doesn't know what scripts/handlers it's going to
> use if they are not preloaded. And if they are, they are shared.
I'm being unclear.
When I say "script", I mean a small piece of stub code that makes
calls into modules, that contain the bulk of the code.
These modules are then loaded in startup.pl, and are therefore
precompiled.
I'd made the jump beyond a single, monolithic ".pl".
Are we getting closer?
--
Dave Hodgkinson, http://www.hodgkinson.org
Editor-in-chief, The Highway Star http://www.deep-purple.com
Apache, mod_perl, MySQL, Sybase hired gun for, well, hire
-----------------------------------------------------------------
Re: MaxChildRequests and modperl - issue
Posted by Stas Bekman <st...@stason.org>.
On 23 Jun 2000, David Hodgkinson wrote:
>
> Stas Bekman <st...@stason.org> writes:
>
> > On 23 Jun 2000, David Hodgkinson wrote:
> >
> > >
> > > Justin <jb...@dslreports.com> writes:
> > >
> > > > What I think happens is the children die after their last request,
> > > > and apache does not kick off a new child straight away.. MinFree is
> > > > set to 2 .. as 12 becomes 11 becomes 10 becomes 9, my backend is
> > > > getting less and less powerful and more and more swamped. When
> > > > Apache does wakeup and spawn a new child, it takes many seconds
> > > > to interpret all the (large amount) of perl code and modules that
> > > > are not in the parent... up to 10 seconds now since its only getting
> > > > a fraction of the box.. this gives the unlucky user a dead browser.
> > > > Worse, the remaining alive servers are dieing faster now as they
> > > > handle more and more of the load, and rush towards 500 to contribute
> > > > to the same traffic jam of booting children.
> > >
> > > I'd worry about the cranking up time. It sounds like you need to
> > > preload your modules in the parent process.
> > >
> > > Stas - please correct me if I'm wrong here, but preloading modules in
> > > startup.pl (or whatever) _does_ make a difference to the spawning time
> > > of an Apache daughter?
> >
> > It makes no difference to the spawning time, since all the code is shared
> > at the beginning.
>
> Isn't that what I said?! :-)
It isn't. You've said: "It _does_ make a difference", whereas I said: "It
doesn't make any difference". Your question was about spawning times, not
about the overhead added during the first request.
> If you take a large script, throw it at Apache::Registry, then you'll
> be compiling the script every time the a diaghter respawns.
>
> If you have your script largely preloaded when Apache starts spawning
> daughters, then you don't have that overhead.
That's wrong. The script won't be compiled until it will be used. When a
process gets spawned, it doesn't know what scripts/handlers it's going to
use if they are not preloaded. And if they are, they are shared.
> > When a new process is spawn 99.9% of the memory pages
> > are shared with parent. When the pages are getting dirty (some data
> > modified) a copy-on-write happen which slows things down, but it has
> > nothing to do with child spawning. The slowdown happens during the
> > request's processing.
> >
> > If you don't preload your code, the first time a script need it the delay
> > of load and compilation will happen and there will be less memory shared.
>
>
> --
> Dave Hodgkinson, http://www.hodgkinson.org
> Editor-in-chief, The Highway Star http://www.deep-purple.com
> Apache, mod_perl, MySQL, Sybase hired gun for, well, hire
> -----------------------------------------------------------------
>
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: MaxChildRequests and modperl - issue
Posted by David Hodgkinson <da...@hodgkinson.org>.
Stas Bekman <st...@stason.org> writes:
> On 23 Jun 2000, David Hodgkinson wrote:
>
> >
> > Justin <jb...@dslreports.com> writes:
> >
> > > What I think happens is the children die after their last request,
> > > and apache does not kick off a new child straight away.. MinFree is
> > > set to 2 .. as 12 becomes 11 becomes 10 becomes 9, my backend is
> > > getting less and less powerful and more and more swamped. When
> > > Apache does wakeup and spawn a new child, it takes many seconds
> > > to interpret all the (large amount) of perl code and modules that
> > > are not in the parent... up to 10 seconds now since its only getting
> > > a fraction of the box.. this gives the unlucky user a dead browser.
> > > Worse, the remaining alive servers are dieing faster now as they
> > > handle more and more of the load, and rush towards 500 to contribute
> > > to the same traffic jam of booting children.
> >
> > I'd worry about the cranking up time. It sounds like you need to
> > preload your modules in the parent process.
> >
> > Stas - please correct me if I'm wrong here, but preloading modules in
> > startup.pl (or whatever) _does_ make a difference to the spawning time
> > of an Apache daughter?
>
> It makes no difference to the spawning time, since all the code is shared
> at the beginning.
Isn't that what I said?! :-)
If you take a large script, throw it at Apache::Registry, then you'll
be compiling the script every time the a diaghter respawns.
If you have your script largely preloaded when Apache starts spawning
daughters, then you don't have that overhead.
> When a new process is spawn 99.9% of the memory pages
> are shared with parent. When the pages are getting dirty (some data
> modified) a copy-on-write happen which slows things down, but it has
> nothing to do with child spawning. The slowdown happens during the
> request's processing.
>
> If you don't preload your code, the first time a script need it the delay
> of load and compilation will happen and there will be less memory shared.
--
Dave Hodgkinson, http://www.hodgkinson.org
Editor-in-chief, The Highway Star http://www.deep-purple.com
Apache, mod_perl, MySQL, Sybase hired gun for, well, hire
-----------------------------------------------------------------
Re: MaxChildRequests and modperl - issue
Posted by Stas Bekman <st...@stason.org>.
On 23 Jun 2000, David Hodgkinson wrote:
>
> Justin <jb...@dslreports.com> writes:
>
> > What I think happens is the children die after their last request,
> > and apache does not kick off a new child straight away.. MinFree is
> > set to 2 .. as 12 becomes 11 becomes 10 becomes 9, my backend is
> > getting less and less powerful and more and more swamped. When
> > Apache does wakeup and spawn a new child, it takes many seconds
> > to interpret all the (large amount) of perl code and modules that
> > are not in the parent... up to 10 seconds now since its only getting
> > a fraction of the box.. this gives the unlucky user a dead browser.
> > Worse, the remaining alive servers are dieing faster now as they
> > handle more and more of the load, and rush towards 500 to contribute
> > to the same traffic jam of booting children.
>
> I'd worry about the cranking up time. It sounds like you need to
> preload your modules in the parent process.
>
> Stas - please correct me if I'm wrong here, but preloading modules in
> startup.pl (or whatever) _does_ make a difference to the spawning time
> of an Apache daughter?
It makes no difference to the spawning time, since all the code is shared
at the beginning. When a new process is spawn 99.9% of the memory pages
are shared with parent. When the pages are getting dirty (some data
modified) a copy-on-write happen which slows things down, but it has
nothing to do with child spawning. The slowdown happens during the
request's processing.
If you don't preload your code, the first time a script need it the delay
of load and compilation will happen and there will be less memory shared.
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: MaxChildRequests and modperl - issue
Posted by David Hodgkinson <da...@hodgkinson.org>.
Justin <jb...@dslreports.com> writes:
> What I think happens is the children die after their last request,
> and apache does not kick off a new child straight away.. MinFree is
> set to 2 .. as 12 becomes 11 becomes 10 becomes 9, my backend is
> getting less and less powerful and more and more swamped. When
> Apache does wakeup and spawn a new child, it takes many seconds
> to interpret all the (large amount) of perl code and modules that
> are not in the parent... up to 10 seconds now since its only getting
> a fraction of the box.. this gives the unlucky user a dead browser.
> Worse, the remaining alive servers are dieing faster now as they
> handle more and more of the load, and rush towards 500 to contribute
> to the same traffic jam of booting children.
I'd worry about the cranking up time. It sounds like you need to
preload your modules in the parent process.
Stas - please correct me if I'm wrong here, but preloading modules in
startup.pl (or whatever) _does_ make a difference to the spawning time
of an Apache daughter?
--
Dave Hodgkinson, http://www.hodgkinson.org
Editor-in-chief, The Highway Star http://www.deep-purple.com
Apache, mod_perl, MySQL, Sybase hired gun for, well, hire
-----------------------------------------------------------------
Re: MaxChildRequests and modperl - issue
Posted by Perrin Harkins <pe...@primenet.com>.
On Thu, 22 Jun 2000, Justin wrote:
> What I think happens is the children die after their last request,
> and apache does not kick off a new child straight away.. MinFree is
> set to 2
Does it help if you crank MinSpareServers higher?
> Perhaps having my own ChildRequest counter and dieing myself, but more
> randomly, is a better way?
You could look into using Apache::SizeLimit or Apache::GTopLimit
instead. I do this and set MaxRequestsPerChild to 0 for unlimited. This
way your processes only die when necessary.
- Perrin
RE: MaxChildRequests and modperl - issue
Posted by Craig Vincent <2b...@home.com>.
>Any easy fixes to this? am I the first to find this? Am i missing
>something obvious? Perhaps having my own ChildRequest counter and
>dieing myself, but more randomly, is a better way?
If you're using mod_perl why not use the Apache::SizeLimit module instead of
the MaxChildRequests. AFAIK the only reason to have children kill
themselves and restart occassionally is to help offset any memory leaks in
the system...Apache::SizeLimit goes one step further...it will check a child
once every X requests seeing if the child is over a certain size in memory
(say 20M)...if it is...than it kills the child so the parent can spawn a
fresh one...otherwise it leaves the child as is....
Perhaps that would solve your problem? I use it at a check for children
over 10 megs every 50 requests and my servers run problemfree handling over
450 requests/sec each.
Sincerely,
Craig Vincent
Re: MaxChildRequests and modperl - issue
Posted by Blue <bl...@calico.gator.net>.
On Thu, 22 Jun 2000, Justin wrote:
Hi Justin,
> I set MaxChildRequests to 500, to effectively clean house every
> now and again, as is recommended.. I have about 12 backend
> modperl servers, and that handles the load fine. (about 250 front
> ends).
What is MaxClients set to? Also, do you have keepalives on, and if so,
what is the timeout on them set to?
Also, what version of Apache are you using?
--
Blue Lang Unix Systems Admin
QSP, Inc., 3200 Atlantic Ave, Ste 100, Raleigh, NC, 27604
Home: 919 835 1540 Work: 919 875 6994 Fax: 919 872 4015