Regarding worker MPM and queue_push/pop

["MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1)" <ma...@hp.com>]

Hi,
	I'm trying to run the SPECweb99 against Apache (on a 1-way box).

I noticed that if I start one server process with a large number of threads
(1000), the server goes into a heavily sleep state (with around 80 % idle
time).

A first guess is that I'm using SysV semaphores, and a semlock can bring
down the entire httpd to crawl. I'm re-compiling using pthread mutexes
whenever possible.

I took a gprof on the system, and noticed that httpd was sleeping on a
condition - the first guess ap_queue_pop () !!..(anything else ?). 
Question : Has anybody done some sort of profiling on the ap_queue_* stuff -
if so, can you please share the data ? 

I had another dumb question - (was this already considered and rejected ?)
instead of having the worker threads compete for the incoming connections
(using ap_queue_pop .. and hence mutex_lock), assign the connection to the
next free thread on a round-robin basis - if I'm not wrong, zeus does
something similar.

-Madhu

Re: Regarding worker MPM and queue_push/pop

[Aaron Bannert <aa...@clove.org>]

On Wed, Dec 03, 2003 at 11:38:25PM -0800, MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote:
> A first guess is that I'm using SysV semaphores, and a semlock can bring
> down the entire httpd to crawl. I'm re-compiling using pthread mutexes
> whenever possible.

Depending on the implementation in your libc/kernel, pthread may be
better suited for Apache. I have seen noticable improvements on Solaris
using pthread, for example.

> I took a gprof on the system, and noticed that httpd was sleeping on a
> condition - the first guess ap_queue_pop () !!..(anything else ?). 
> Question : Has anybody done some sort of profiling on the ap_queue_* stuff -
> if so, can you please share the data ? 
> 
> I had another dumb question - (was this already considered and rejected ?)
> instead of having the worker threads compete for the incoming connections
> (using ap_queue_pop .. and hence mutex_lock), assign the connection to the
> next free thread on a round-robin basis - if I'm not wrong, zeus does
> something similar.

We don't sleep on a mutex_lock, since that is not a good way to do
predictive scheduling and makes no guarantees about sharing, etc.
As you noticed above, when a thread is idle is waits on a condition
in the ap_queue. There is one thread feeding that queue, and when
it accept()s a connection is pushes it into that queue, effectively
triggering the condition on a single thread which then wakes up
and does its work.

I can't imagine why sleeping would consume 20% of your CPU when the
server is idle. Perhaps you should look into the scalability of your
thread library. On what kind of system are you seeing this problem
(hardware/OS/rev/etc)?

Are you seeing big load spikes when only a trickle of requests are coming
in?

-aaron

Re: Regarding worker MPM and queue_push/pop

[Cliff Woolley <jw...@virginia.edu>]

On Wed, 3 Dec 2003, MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote:

> instead of having the worker threads compete for the incoming connections
> (using ap_queue_pop .. and hence mutex_lock), assign the connection to the
> next free thread on a round-robin basis - if I'm not wrong, zeus does
> something similar.

Isn't that what the leader-follower MPM does?

--Cliff