mod_cern_meta/3366: I am getting this - (11)Resource temporarily unavailable: couldn't spawn child process - on very busy servers

[Kevin Goddard <dr...@draddog.com>]

>Number:         3366
>Category:       mod_cern_meta
>Synopsis:       I am getting this - (11)Resource temporarily unavailable: couldn't spawn child process - on very busy servers
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    apache
>State:          open
>Class:          sw-bug
>Submitter-Id:   apache
>Arrival-Date:   Sun Nov  8 16:10:00 PST 1998
>Last-Modified:
>Originator:     draddog@draddog.com
>Organization:
apache
>Release:        1.3.3
>Environment:
Here is the output from a Uname -a:
Linux server.domain.com 2.0.34 #9 Wed Nov 4 15:46:51 EST 1998 i686 unknown
The server is a Dell Poweredge 2300 with 256 MB of RAM and a Pentium II 400 processor.  I am having the exact same problem on three identical machines.  All running the same version of Linux with identical Hard ware configuration
>Description:
Okay I have seen this problem listed before, and I have applied every single fix listed.  The server is not listening to any port but 80.  Here is a list of a ulimit -a:
core file size (blocks)  1000000
data seg size (kbytes)   unlimited
file size (blocks)       unlimited
max memory size (kbytes) unlimited
stack size (kbytes)      8192
cpu time (seconds)       unlimited
max user processes       unlimited
pipe size (512 bytes)    8
open files               1024
virtual memory (kbytes)  2105343

As you can see, but the user processes and the open files is very high.  I am currently getting this error message on one machine which has 227 httpd processes running and 600 tcp connections to it.  I have the HARD_SERVER_LIMIT set to 1000.  The following info is in my config:
MaxClients 400
StartServers 100
MinSpareServers 50
MaxSpareServers 100
MaxRequestsPerChild 100000
The server works fine after restarting it for a certain lenght of time (less then 30 minutes) and then proceeds to generate (11)Resource temporarily unavailable: couldn't spawn child process for any CGi's that run on it.  Though we can restart the server every 15 or so minutes, I believe that Apache should run a bit better.  I have the same version running on 3 other machines that are a different hardware configuration (they have Pentium II 300's) that are running fine.  They all run anywhere between 200 to 1000 tcp connections at one time.
>How-To-Repeat:
You can go to http://www.draddog.com/sanctum/calcrank.html and enter 100 for both boxes and hit submit.  It may or may not error out.
>Fix:

>Audit-Trail:
>Unformatted:
[In order for any reply to be added to the PR database, ]
[you need to include <ap...@Apache.Org> in the Cc line ]
[and leave the subject line UNCHANGED.  This is not done]
[automatically because of the potential for mail loops. ]
[If you do not include this Cc, your reply may be ig-   ]
[nored unless you are responding to an explicit request ]
[from a developer.                                      ]
[Reply only with text; DO NOT SEND ATTACHMENTS!         ]