apache 2.2 can't keep up with apache 1.3

[William Taylor <wi...@corp.sonic.net>]

We have written a modules for apache that for certain reasons requires
one hit per child.
On apache 1.3 this worked fine with a minor tweak to
SCOREBOARD_MAINTENANCE_INTERVAL
setting it to 10000 from 1000000.

Note that this behavior can be tested without our module by setting
"MaxRequestsPerChild 1"

We have been running apache 1.3.41 with:
MinSpareServers 500
MaxSpareServers 600
StartServers 550

Using JMeter to send some requests if we request a single page using 3
threads, apache has no problem keeping up and
doesn't dip below 485 idle servers. With debug logging on you see it
jumping from spawning 8 to 16 to 32 children. We didn't
need to increase the MAX_SPAWN_RATE from the default of 32 for apache to
keep up just fine.

Output from apachectl status
   197 requests/sec - 28.6 kB/second - 148 B/request
   5 requests currently being processed, 492 idle servers
________________________________________________________________
__________________________.___._________________________________
________________________________._____________________________..
__.___._.W_WW..WW_______________________________________________
__________..__________________._..______________________________
________________________________________________________________
________________________________________________________________
____________________________________....._..._.___________..__..
........._________.____.........................................


Recently we decided we should update our module for apache 2.2 . However
we found the performance of apache 2.2 was falling short
compared to apache 1.3.

Running apache 2.2.22 with the same number of servers falls down pretty
quickly with the same load from JMeter.
Within a few seconds it drops down to 0 idle connections. In the
error_log you see apache constantly saying it's spawning
32 children but the number of children are really low.  Example log
entry "spawning 32 children, there are 2 idle, and 9 total children"
Occasionally it will start catching up but quickly drop back down.
Example log entry "spawning 32 children, there are 169 idle, and 180
total child"

apachectl status:
   27.8 requests/sec - 7.0 kB/second - 256 B/request
   13 requests currently being processed, 0 idle workers
 CCCWCWRSSS............................CCCCCC....................
 ................................................................


apachectl status when it starts catching back up:
  161 requests/sec - 11.4 kB/second - 72 B/request
   4 requests currently being processed, 477 idle workers
 ___________C____WR____________________._________________________
 ________________________________________________________________
 ________.______.._......_...C___________________________________


I have tried upping the MAX_SPAWN_RATE to 512 and starting off the
idle_spawn_rate at 128 and never letting
it fall below that. It seemed like things got marginally better but not
by much. I realize most people would think
we are crazy for running one hit per child but it's just something we
need to do. This has all been tested on the
same hardware also. I tried stracing and running gprof but nothing
seemed obvious. My best guess at this point
is apache is taking to long to close connections or there is some other
sleeps or something that I'm not seeing.

Any thoughts on what could be the issue would be most appreciated.