Dynamically controlling process pool size

[Charles Randall <cr...@matchlogic.com>]

I've been examining Apache 1.2.4 under load on one of our production
systems. I've observed that the server is slow to respond when the
process pool is growing (request processing times multiply by 2-10x).

After looking at the code, it appears that the pool sizing logic is
something like this:

	Wait for a child process to die or for one second (whichever
comes first)
	then resize the pool if necessary

If my observations are correct, and MaxRequestsPerChild is large, then
server processes can only be spawned at a maximum rate of one per
second. This appears to be a reasonable trade-off for most systems.

I know that I can set MaxSpareServers and MinSpareServers to build up
enough "headroom" to absorb peaks. However, this isn't very optimal.

Ideally, the sizing of the process pool would be more tunable (at a
minimum, adding a "PollServerPoolSizeInterval"). 

I'm wondering if it would be feasible to isolate the code to "resize the
pool if necessary" for experimentation (the obvious response is "Well,
then. Go for it."). I'm thinking of doing things like taking into
account historical information for the last N minutes -- in other words,
don't shrink the pool if I expect another surge like I saw 5 minutes
ago.

Does anyone have any new ideas in this area?

Charles F. Randall
crandall@matchlogic.com
MatchLogic, Inc.

Re: Dynamically controlling process pool size

[Dean Gaudet <dg...@arctic.org>]

On Wed, 31 Dec 1997, Charles Randall wrote:

> I've been examining Apache 1.2.4 under load on one of our production
> systems. I've observed that the server is slow to respond when the
> process pool is growing (request processing times multiply by 2-10x).
> 
> After looking at the code, it appears that the pool sizing logic is
> something like this:
> 
> 	Wait for a child process to die or for one second (whichever
> comes first)
> 	then resize the pool if necessary
> 
> If my observations are correct, and MaxRequestsPerChild is large, then
> server processes can only be spawned at a maximum rate of one per
> second. This appears to be a reasonable trade-off for most systems.

Right.

> I know that I can set MaxSpareServers and MinSpareServers to build up
> enough "headroom" to absorb peaks. However, this isn't very optimal.

You should also set up StartServers high enough to fit the load, you get
those "for free" in that they don't take a second a piece to start. 

> Ideally, the sizing of the process pool would be more tunable (at a
> minimum, adding a "PollServerPoolSizeInterval"). 
> 
> I'm wondering if it would be feasible to isolate the code to "resize the
> pool if necessary" for experimentation (the obvious response is "Well,
> then. Go for it."). I'm thinking of doing things like taking into
> account historical information for the last N minutes -- in other words,
> don't shrink the pool if I expect another surge like I saw 5 minutes
> ago.

The place to look is the 1.3 code as others have mentioned.  If you look
at the perform_idle_server_maintenance() function in src/main/http_main.c
you'll find all the stuff you need to tweak and play with the spawning
policy.  I'd love to know if it helps your problem... it's the best I
could think of without learning lots of queueing theory.

The 1.3 behaviour is an exponential growth rate of spawning, with linear
removal of excess idle children.  The 1.2 behaviour is linear spawning,
and almost immediate removal of excess idle children... if you look at the
child_main() code you'll see each child scans the scoreboard after each
hit and decides to suicide or not.  So if the load drops off suddenly you
could have many children killing themselves.  None of that goes on in 1.3,
only the parent makes spawn/kill decisions based on the load. 

Dean

Re: Dynamically controlling process pool size

[Marc Slemko <ma...@worldgate.com>]

Start by looking at the 1.3 code.  It does exponential spawning:

  *) In the event that the server is starved for idle servers it will
     spawn 1, then 2, then 4, ..., then 32 servers each second,
     doubling each second.  It'll also give a warning in the errorlog
     since the most common reason for this is a poor StartServers
     setting.  The define MAX_SPAWN_RATE can be used to raise/lower
     the maximum.  [Dean Gaudet]

Yes, it can be quite interesting when you run into an excess of requests
for your server because that increases the time to serve request which
increases the backlog which increases... etc.  This helps it a good bit if
your machine has the resources.

On Wed, 31 Dec 1997, Charles Randall wrote:

> I've been examining Apache 1.2.4 under load on one of our production
> systems. I've observed that the server is slow to respond when the
> process pool is growing (request processing times multiply by 2-10x).
> 
> After looking at the code, it appears that the pool sizing logic is
> something like this:
> 
> 	Wait for a child process to die or for one second (whichever
> comes first)
> 	then resize the pool if necessary
> 
> If my observations are correct, and MaxRequestsPerChild is large, then
> server processes can only be spawned at a maximum rate of one per
> second. This appears to be a reasonable trade-off for most systems.
> 
> I know that I can set MaxSpareServers and MinSpareServers to build up
> enough "headroom" to absorb peaks. However, this isn't very optimal.
> 
> Ideally, the sizing of the process pool would be more tunable (at a
> minimum, adding a "PollServerPoolSizeInterval"). 
> 
> I'm wondering if it would be feasible to isolate the code to "resize the
> pool if necessary" for experimentation (the obvious response is "Well,
> then. Go for it."). I'm thinking of doing things like taking into
> account historical information for the last N minutes -- in other words,
> don't shrink the pool if I expect another surge like I saw 5 minutes
> ago.
> 
> Does anyone have any new ideas in this area?
> 
> Charles F. Randall
> crandall@matchlogic.com
> MatchLogic, Inc.
> 
>