dynamically change max client value

[David Burry <db...@tagnet.org>]

Recently there has been a little discussion about an API in apache for
controlling starts, stops, restarts, etc...

I have an idea that may help me solve a problem I've been having.  The
problem is in limiting the number of processes that will run on a machine to
somewhere below where the machine will keel over and die, while still being
close to the maximum the machine will handle.  The issue is depending on
what the majority of those processes are doing it changes the maximum number
a given machine can handle by a few orders of magnitude, so a multi-purpose
machine that serves, say, static content and cgi scripts (or other things
that vary greatly in machine resource usage) cannot be properly tuned for
maximum performance while guaranteeing the machine won't die under heavy
load.

The solution I've thought of is... what if Apache had an API that could be
used to say "no more processes, whatever you have NOW is the max!"  or
otherwise to dynamically raise or lower the max number (perhaps "oh there's
too many, reduce a bit")....  You see, an external monitoring system could
monitor cpu and memory and whatnot and dynamically adjust apache depending
on what it's doing.....  This kind of system could really increase the
stability of any large Apache server farm, and help keep large traffic
spikes from killing apache so bad that nobody gets served anything at all.

In fact this idea could be extended someday to dynamically change all sorts
of apache configuration things, but all I really need that I know of right
now is the max client value...

What do you all think of this idea?  Does this already exist perhaps?

Dave

Re: dynamically change max client value

[David Burry <db...@tagnet.org>]

Interesting comments, thanks.  You obviously speak from experience.

The idea I was having is that no matter how overloaded a machine becomes, it should never run so far out of resources that it dies, but there should be some kind of limit in place... I thought this was what MaxClients was for, however it varies too greatly what the optimum value is depending on what's getting hit so I don't know what to do about it...

Your idea of a reverse proxy in front (or module) turning away or redirecting some requests in a very light fashion deserves merit...  I know that Excite.com used to have a very light static home page they'd slap up during peak loads for this very reason, I'll investigate something like that for the solution to my problem instead of what I originally suggested, thanks.

Dave


At 12:01 PM 11/4/2002 -0800, Scott Hess wrote:
>Based on my experience, this wouldn't be a high-quality solution, it would
>be a hack.  I've seen very few cases where load spiked enough to be an
>issue, but was transient enough that a solution like this would work - and
>in those cases, plain old Unix multitasking normally suffices.
>
>What happens if you implement the solution anyhow is that you get a bunch
>of users stuck in the ListenBacklog.  So they'll wait a couple minutes
>before their request even starts.  If you have a deep backlog, requests
>just pile up so that the machine never gets its head above water.  In the
>worst case, clients will timeout while their request is in the backlog,
>but since you don't find that out until you send a response which writes
>out to the network, you can very easily do work that can never be
>delivered.  Beyond all that, the user experience simply _sucks_.
>
>[Yes, I've done what you suggest, just not using the implementation you
>suggest.  It's integrated into an existing custom module, you could also
>probably do it with a reverse proxy.  In the end, it was not a productive
>solution.]
>
>What I think you really want is a module that will intercept all requests,
>and send back "The server is really busy, try again in five minutes" if
>the server is too busy by some measure.  You generally want this to be a
>super-low-cost option, so that you can spin through requests very quickly.  
>Optimally, no externally-blockable pieces (no database connections, no
>locking filesystem access, etc).  One relatively simple option might be to
>use a Squid, and an URL redirector which implements the magic check.  If
>the machine is not busy, send through to the real server, if the machine
>is busy, redirect to an URL which will deliver your message.
>
>[Again, yes, I've done this in Apache1.3, but in code targetted to our
>custom modules.  You could certainly do it more generically, I just
>haven't had the need.  You might check mod_backhand.]
>
>Later,
>scott
>
>
>On Mon, 4 Nov 2002, David Burry wrote:
>> I realize that allowing _everything_ to be dynamically configured via
>> SNMP (or signal or something) would probably be too substantial of an
>> API change to be considered for the current code base, but it would be
>> nice to consider it for some future major revision of Apache....
>> 
>> And it would be more than just "nice" if at least the max client value
>> thing could be somehow worked into the current versions of Apache...  
>> There is a current very real and very large problem that could be solved
>> by this, not just a "nice to have" feature.  This is what I meant to
>> emphasize in my original email...
>> 
>> Dave
>> 
>> ----- Original Message -----
>> From: "Dirk-Willem van Gulik" <di...@webweaving.org>
>> To: <de...@httpd.apache.org>
>> Sent: Monday, November 04, 2002 9:35 AM
>> Subject: Re: dynamically change max client value
>> 
>> 
>> >
>> > In my ideal world every config directive would be able to advertize or
>> > register an optional 'has changed' hook. Which, if present, would be
>> > called in context whenever a value is somehow updated (through snmp, a
>> > configd, signal, wathever). If there is no such hook; the old -update- on
>> > graceful restart is the default (though it sure would be nice to have some
>> > values also advertize that they need a full shutdown and restart).
>> >
>> > Of course one could also argue for not just a put but also for a 'get'
>> > interface in context :-)
>> >
>> > Dw
>> >
>> > On Mon, 4 Nov 2002, David Burry wrote:
>> >
>> > > Recently there has been a little discussion about an API in apache for
>> > > controlling starts, stops, restarts, etc...
>> > >
>> > > I have an idea that may help me solve a problem I've been having.  The
>> > > problem is in limiting the number of processes that will run on a
>> machine to
>> > > somewhere below where the machine will keel over and die, while still
>> being
>> > > close to the maximum the machine will handle.  The issue is depending on
>> > > what the majority of those processes are doing it changes the maximum
>> number
>> > > a given machine can handle by a few orders of magnitude, so a
>> multi-purpose
>> > > machine that serves, say, static content and cgi scripts (or other
>> things
>> > > that vary greatly in machine resource usage) cannot be properly tuned
>> for
>> > > maximum performance while guaranteeing the machine won't die under heavy
>> > > load.
>> > >
>> > > The solution I've thought of is... what if Apache had an API that could
>> be
>> > > used to say "no more processes, whatever you have NOW is the max!"  or
>> > > otherwise to dynamically raise or lower the max number (perhaps "oh
>> there's
>> > > too many, reduce a bit")....  You see, an external monitoring system
>> could
>> > > monitor cpu and memory and whatnot and dynamically adjust apache
>> depending
>> > > on what it's doing.....  This kind of system could really increase the
>> > > stability of any large Apache server farm, and help keep large traffic
>> > > spikes from killing apache so bad that nobody gets served anything at
>> all.
>> > >
>> > > In fact this idea could be extended someday to dynamically change all
>> sorts
>> > > of apache configuration things, but all I really need that I know of
>> right
>> > > now is the max client value...
>> > >
>> > > What do you all think of this idea?  Does this already exist perhaps?
>> > >
>> > > Dave
>> > >
>> > >
>> >
>> 

Re: dynamically change max client value

[Scott Hess <sc...@avantgo.com>]

Based on my experience, this wouldn't be a high-quality solution, it would
be a hack.  I've seen very few cases where load spiked enough to be an
issue, but was transient enough that a solution like this would work - and
in those cases, plain old Unix multitasking normally suffices.

What happens if you implement the solution anyhow is that you get a bunch
of users stuck in the ListenBacklog.  So they'll wait a couple minutes
before their request even starts.  If you have a deep backlog, requests
just pile up so that the machine never gets its head above water.  In the
worst case, clients will timeout while their request is in the backlog,
but since you don't find that out until you send a response which writes
out to the network, you can very easily do work that can never be
delivered.  Beyond all that, the user experience simply _sucks_.

[Yes, I've done what you suggest, just not using the implementation you
suggest.  It's integrated into an existing custom module, you could also
probably do it with a reverse proxy.  In the end, it was not a productive
solution.]

What I think you really want is a module that will intercept all requests,
and send back "The server is really busy, try again in five minutes" if
the server is too busy by some measure.  You generally want this to be a
super-low-cost option, so that you can spin through requests very quickly.  
Optimally, no externally-blockable pieces (no database connections, no
locking filesystem access, etc).  One relatively simple option might be to
use a Squid, and an URL redirector which implements the magic check.  If
the machine is not busy, send through to the real server, if the machine
is busy, redirect to an URL which will deliver your message.

[Again, yes, I've done this in Apache1.3, but in code targetted to our
custom modules.  You could certainly do it more generically, I just
haven't had the need.  You might check mod_backhand.]

Later,
scott


On Mon, 4 Nov 2002, David Burry wrote:
> I realize that allowing _everything_ to be dynamically configured via
> SNMP (or signal or something) would probably be too substantial of an
> API change to be considered for the current code base, but it would be
> nice to consider it for some future major revision of Apache....
> 
> And it would be more than just "nice" if at least the max client value
> thing could be somehow worked into the current versions of Apache...  
> There is a current very real and very large problem that could be solved
> by this, not just a "nice to have" feature.  This is what I meant to
> emphasize in my original email...
> 
> Dave
> 
> ----- Original Message -----
> From: "Dirk-Willem van Gulik" <di...@webweaving.org>
> To: <de...@httpd.apache.org>
> Sent: Monday, November 04, 2002 9:35 AM
> Subject: Re: dynamically change max client value
> 
> 
> >
> > In my ideal world every config directive would be able to advertize or
> > register an optional 'has changed' hook. Which, if present, would be
> > called in context whenever a value is somehow updated (through snmp, a
> > configd, signal, wathever). If there is no such hook; the old -update- on
> > graceful restart is the default (though it sure would be nice to have some
> > values also advertize that they need a full shutdown and restart).
> >
> > Of course one could also argue for not just a put but also for a 'get'
> > interface in context :-)
> >
> > Dw
> >
> > On Mon, 4 Nov 2002, David Burry wrote:
> >
> > > Recently there has been a little discussion about an API in apache for
> > > controlling starts, stops, restarts, etc...
> > >
> > > I have an idea that may help me solve a problem I've been having.  The
> > > problem is in limiting the number of processes that will run on a
> machine to
> > > somewhere below where the machine will keel over and die, while still
> being
> > > close to the maximum the machine will handle.  The issue is depending on
> > > what the majority of those processes are doing it changes the maximum
> number
> > > a given machine can handle by a few orders of magnitude, so a
> multi-purpose
> > > machine that serves, say, static content and cgi scripts (or other
> things
> > > that vary greatly in machine resource usage) cannot be properly tuned
> for
> > > maximum performance while guaranteeing the machine won't die under heavy
> > > load.
> > >
> > > The solution I've thought of is... what if Apache had an API that could
> be
> > > used to say "no more processes, whatever you have NOW is the max!"  or
> > > otherwise to dynamically raise or lower the max number (perhaps "oh
> there's
> > > too many, reduce a bit")....  You see, an external monitoring system
> could
> > > monitor cpu and memory and whatnot and dynamically adjust apache
> depending
> > > on what it's doing.....  This kind of system could really increase the
> > > stability of any large Apache server farm, and help keep large traffic
> > > spikes from killing apache so bad that nobody gets served anything at
> all.
> > >
> > > In fact this idea could be extended someday to dynamically change all
> sorts
> > > of apache configuration things, but all I really need that I know of
> right
> > > now is the max client value...
> > >
> > > What do you all think of this idea?  Does this already exist perhaps?
> > >
> > > Dave
> > >
> > >
> >
> 

Re: dynamically change max client value

[David Burry <db...@tagnet.org>]

I realize that allowing _everything_ to be dynamically configured via SNMP
(or signal or something) would probably be too substantial of an API change
to be considered for the current code base, but it would be nice to consider
it for some future major revision of Apache....

And it would be more than just "nice" if at least the max client value thing
could be somehow worked into the current versions of Apache...  There is a
current very real and very large problem that could be solved by this, not
just a "nice to have" feature.  This is what I meant to emphasize in my
original email...

Dave

----- Original Message -----
From: "Dirk-Willem van Gulik" <di...@webweaving.org>
To: <de...@httpd.apache.org>
Sent: Monday, November 04, 2002 9:35 AM
Subject: Re: dynamically change max client value


>
> In my ideal world every config directive would be able to advertize or
> register an optional 'has changed' hook. Which, if present, would be
> called in context whenever a value is somehow updated (through snmp, a
> configd, signal, wathever). If there is no such hook; the old -update- on
> graceful restart is the default (though it sure would be nice to have some
> values also advertize that they need a full shutdown and restart).
>
> Of course one could also argue for not just a put but also for a 'get'
> interface in context :-)
>
> Dw
>
> On Mon, 4 Nov 2002, David Burry wrote:
>
> > Recently there has been a little discussion about an API in apache for
> > controlling starts, stops, restarts, etc...
> >
> > I have an idea that may help me solve a problem I've been having.  The
> > problem is in limiting the number of processes that will run on a
machine to
> > somewhere below where the machine will keel over and die, while still
being
> > close to the maximum the machine will handle.  The issue is depending on
> > what the majority of those processes are doing it changes the maximum
number
> > a given machine can handle by a few orders of magnitude, so a
multi-purpose
> > machine that serves, say, static content and cgi scripts (or other
things
> > that vary greatly in machine resource usage) cannot be properly tuned
for
> > maximum performance while guaranteeing the machine won't die under heavy
> > load.
> >
> > The solution I've thought of is... what if Apache had an API that could
be
> > used to say "no more processes, whatever you have NOW is the max!"  or
> > otherwise to dynamically raise or lower the max number (perhaps "oh
there's
> > too many, reduce a bit")....  You see, an external monitoring system
could
> > monitor cpu and memory and whatnot and dynamically adjust apache
depending
> > on what it's doing.....  This kind of system could really increase the
> > stability of any large Apache server farm, and help keep large traffic
> > spikes from killing apache so bad that nobody gets served anything at
all.
> >
> > In fact this idea could be extended someday to dynamically change all
sorts
> > of apache configuration things, but all I really need that I know of
right
> > now is the max client value...
> >
> > What do you all think of this idea?  Does this already exist perhaps?
> >
> > Dave
> >
> >
>

Re: dynamically change max client value

[Dirk-Willem van Gulik <di...@webweaving.org>]

In my ideal world every config directive would be able to advertize or
register an optional 'has changed' hook. Which, if present, would be
called in context whenever a value is somehow updated (through snmp, a
configd, signal, wathever). If there is no such hook; the old -update- on
graceful restart is the default (though it sure would be nice to have some
values also advertize that they need a full shutdown and restart).

Of course one could also argue for not just a put but also for a 'get'
interface in context :-)

Dw

On Mon, 4 Nov 2002, David Burry wrote:

> Recently there has been a little discussion about an API in apache for
> controlling starts, stops, restarts, etc...
>
> I have an idea that may help me solve a problem I've been having.  The
> problem is in limiting the number of processes that will run on a machine to
> somewhere below where the machine will keel over and die, while still being
> close to the maximum the machine will handle.  The issue is depending on
> what the majority of those processes are doing it changes the maximum number
> a given machine can handle by a few orders of magnitude, so a multi-purpose
> machine that serves, say, static content and cgi scripts (or other things
> that vary greatly in machine resource usage) cannot be properly tuned for
> maximum performance while guaranteeing the machine won't die under heavy
> load.
>
> The solution I've thought of is... what if Apache had an API that could be
> used to say "no more processes, whatever you have NOW is the max!"  or
> otherwise to dynamically raise or lower the max number (perhaps "oh there's
> too many, reduce a bit")....  You see, an external monitoring system could
> monitor cpu and memory and whatnot and dynamically adjust apache depending
> on what it's doing.....  This kind of system could really increase the
> stability of any large Apache server farm, and help keep large traffic
> spikes from killing apache so bad that nobody gets served anything at all.
>
> In fact this idea could be extended someday to dynamically change all sorts
> of apache configuration things, but all I really need that I know of right
> now is the max client value...
>
> What do you all think of this idea?  Does this already exist perhaps?
>
> Dave
>
>