[users@httpd] varying 2.0.55 worker MPM process/threads/memory behavior

[ed...@verizon.com]

We are attempting to upgrade our Apache web servers to 2.0.55.  We are 
running on several different platforms with different versions of Red 
Hat Linux.  We are using the worker module compiled in.

In our test env, on a 512 Meg RAM box, we can start with the desired 512 
threads.  Our config looks like this:
ServerLimit 1
ThreadLimit 512
StartServers 1
MaxClients 512
MinSpareThreads 1
MaxSpareThreads 512
ThreadsPerChild 512
MaxRequestsPerChild  0

These all show up with a ps -aef | grep httpd.  On this box, free is 
192420 with httpd stopped and 189000 with it running.  Not very much 
difference.

In UAT, on a 1 Gig RAM box, we get an error message "unable to create 
worker thread" after about 300 threads. 

Why would the threads require more memory when there is twice as much 
memory available? 
And how could the same config start OK on a smaller box? 

We can start with limits set at 256 instead of 512.  But only the root 
process and a couple other processes (listener and starter) show up with 
ps -aef.  We can get the others with ps -aefm.  So why are these threads 
treated like full processes on some boxes but threads on others?  Are 
there some OS level settings that affect this behavior?

One final bit of info, if that isn't enough to confuse you, is that we 
have another build with the worker module from version 2.0.50.  We can 
copy this build over to the UAT box and run it with limits set at 550. 

Any ideas as to what is going on?  I have examined the config logs from 
the various builds and don't see anything that would affect this 
behavior.  They all have -pthread. 

I did some testing and tracking of memory usage on another box but it 
doesn't seem to add up.  The starting point is this:
total=513208 used=479884   free=36008  shared=0

Starting with limits at 128, free drops to 33324.  This doesn't seem 
like very much.
with limits at 250, free drops to 28764  - not much more
But on this box we can't even run 256, we get the out of memory error 
after 253 threads. 

I obviously don't understand Apache memory requirements or usage patterns. 
 
When I get it running with128 and look at threads with top H they have 
RSS=4456 and SHARE=2412
When I get it running with 250 and look at threads with top H they have 
RSS=5708 and SHARE=2426

If someone could explain this or point me to docs that explain this I 
would greatly appreciate it.

Thanks,
Ed