You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Colm MacCarthaigh <co...@stdlib.net> on 2004/01/07 22:31:13 UTC
[PATCH] raise MAX_SERVER_LIMIT
Not entirely serious, but today, we actually hit this, in production :)
The hardware, a dual 2Ghz Xeon with 12Gb RAM with Linux 2.6.1-rc2 coped,
and remained responsive. So 20,000 may no longer be outside the realms
of what administrators reasonably desire to have.
Index: server/mpm/prefork/prefork.c
===================================================================
RCS file: /home/cvspublic/httpd-2.0/server/mpm/prefork/prefork.c,v
retrieving revision 1.286
diff -u -u -r1.286 prefork.c
--- server/mpm/prefork/prefork.c 1 Jan 2004 13:26:25 -0000
1.286
+++ server/mpm/prefork/prefork.c 7 Jan 2004 21:24:55 -0000
@@ -123,7 +123,7 @@
* some sort of compile-time limit to help catch typos.
*/
#ifndef MAX_SERVER_LIMIT
-#define MAX_SERVER_LIMIT 20000
+#define MAX_SERVER_LIMIT 100000
#endif
#ifndef HARD_THREAD_LIMIT
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Bill Stoddard <bi...@wstoddard.com>.
Colm MacCarthaigh wrote:
> Not entirely serious, but today, we actually hit this, in production :)
> The hardware, a dual 2Ghz Xeon with 12Gb RAM with Linux 2.6.1-rc2 coped,
> and remained responsive. So 20,000 may no longer be outside the realms
> of what administrators reasonably desire to have.
>
> Index: server/mpm/prefork/prefork.c
> ===================================================================
> RCS file: /home/cvspublic/httpd-2.0/server/mpm/prefork/prefork.c,v
> retrieving revision 1.286
> diff -u -u -r1.286 prefork.c
> --- server/mpm/prefork/prefork.c 1 Jan 2004 13:26:25 -0000
> 1.286
> +++ server/mpm/prefork/prefork.c 7 Jan 2004 21:24:55 -0000
> @@ -123,7 +123,7 @@
> * some sort of compile-time limit to help catch typos.
> */
> #ifndef MAX_SERVER_LIMIT
> -#define MAX_SERVER_LIMIT 20000
> +#define MAX_SERVER_LIMIT 100000
> #endif
>
> #ifndef HARD_THREAD_LIMIT
Holy cow!
Bill
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Wed, Jan 28, 2004 at 10:40:54AM +0000, Colm MacCarthaigh wrote:
> On Tue, Jan 27, 2004 at 02:24:46PM -0500, Jeff Trawick wrote:
> > I'm testing with this patch currently (so far so good):
>
> Same here, I've applied the patch, and right now have 1 hours uptime,
> which is 12 times more than I've ever had with worker before.
> Looks like that was it. Where do I send the beer?
Still going well, no problems since the patch, but after all that,
I can finally benchmark worker Vs prefork properly:
Prefork:
Server Software: Apache/2.0.48
Server Hostname: ftp.heanet.ie
Server Port: 80
Document Path: /
Document Length: 2841 bytes
Concurrency Level: 40
Time taken for tests: 31.223 seconds
Complete requests: 10000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 29910000 bytes
HTML transferred: 28410000 bytes
Requests per second: 320.28 [#/sec] (mean)
Time per request: 124.89 [ms] (mean)
Time per request: 3.12 [ms] (mean, across all concurrent requests)
Transfer rate: 957.95 [Kbytes/sec] received
Connnection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.5 0 51
Processing: 17 124 192.2 54 1692
Waiting: 14 124 192.2 54 1691
Total: 17 124 192.3 55 1693
Percentage of the requests served within a certain time (ms)
50% 55
66% 67
75% 84
80% 108
90% 337
95% 547
98% 706
99% 861
100% 1693 (last request)
Worker:
Server Software: Apache/2.0.48
Server Hostname: ftp.heanet.ie
Server Port: 80
Document Path: /
Document Length: 2841 bytes
Concurrency Level: 40
Time taken for tests: 39.907 seconds
Complete requests: 10000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 29910000 bytes
HTML transferred: 28410000 bytes
Requests per second: 250.58 [#/sec] (mean)
Time per request: 159.63 [ms] (mean)
Time per request: 3.99 [ms] (mean, across all concurrent requests)
Transfer rate: 749.49 [Kbytes/sec] received
Connnection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.5 0 17
Processing: 1 159 187.8 90 1540
Waiting: 1 159 187.8 90 1540
Total: 1 159 187.8 90 1540
Percentage of the requests served within a certain time (ms)
50% 90
66% 124
75% 176
80% 206
90% 345
95% 590
98% 865
99% 968
100% 1540 (last request)
Oh well :)
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Jeff Trawick <tr...@attglobal.net>.
Colm MacCarthaigh wrote:
> On Tue, Jan 27, 2004 at 02:24:46PM -0500, Jeff Trawick wrote:
>
>>I'm testing with this patch currently (so far so good):
>
> Same here, I've applied the patch, and right now have 1 hours uptime,
> which is 12 times more than I've ever had with worker before.
> Looks like that was it. Where do I send the beer?
cool!
hold off on the beer for now... I couldn't go through a keg before it went
flat, and the cans and bottles don't really do it for me, draft device or not...
> Oh, and if someone is committing to worker.c, do me a favour and
> add a zero to MAX_SERVER_LIMIT there too ;)
yup; MAX_THREAD_LIMIT should be increased too, as I have come close enough to
20,000 ThreadsPerChild on AIX before to wonder if I could/would hit it... I'll
make a note to go through the Unix MPMs soon-ish and give some more breathing room
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Tue, Jan 27, 2004 at 02:24:46PM -0500, Jeff Trawick wrote:
> I'm testing with this patch currently (so far so good):
Same here, I've applied the patch, and right now have 1 hours uptime,
which is 12 times more than I've ever had with worker before.
Looks like that was it. Where do I send the beer?
Oh, and if someone is committing to worker.c, do me a favour and
add a zero to MAX_SERVER_LIMIT there too ;)
> Index: server/mpm/worker/worker.c
> ===================================================================
> RCS file: /home/cvs/httpd-2.0/server/mpm/worker/worker.c,v
> retrieving revision 1.145
> diff -u -r1.145 worker.c
> --- server/mpm/worker/worker.c 27 Jan 2004 15:19:58 -0000 1.145
> +++ server/mpm/worker/worker.c 27 Jan 2004 19:20:10 -0000
> @@ -1441,7 +1441,8 @@
> ++idle_thread_count;
> }
> }
> - if (any_dead_threads && totally_free_length < idle_spawn_rate
> + if (any_dead_threads && totally_free_length < idle_spawn_rate
> + && free_length < MAX_SPAWN_RATE
> && (!ps->pid /* no process in the slot */
> || ps->quiescing)) { /* or at least one is going
> away */
> if (all_dead_threads) {
>
>
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick wrote:
> worker MPM stack corruption in parent:
>
> int free_slots[MAX_SPAWN_RATE];
>
> ...
>
> /* great! we prefer these, because the new process can
> * start more threads sooner. So prioritize this slot
> * by putting it ahead of any slots with active threads.
> *
> * first, make room by moving a slot that's potentially
> still
> * in use to the end of the array
> */
> NEW CODE-> ap_assert(free_length < MAX_SPAWN_RATE);
> free_slots[free_length] =
> free_slots[totally_free_length];
>
> [Tue Jan 27 12:20:19 2004] [crit] [Tue Jan 27 12:20:19 2004] file
> worker.c, line 1590, assertion "free_length < MAX_SPAWN_RATE" failed
I'm testing with this patch currently (so far so good):
Index: server/mpm/worker/worker.c
===================================================================
RCS file: /home/cvs/httpd-2.0/server/mpm/worker/worker.c,v
retrieving revision 1.145
diff -u -r1.145 worker.c
--- server/mpm/worker/worker.c 27 Jan 2004 15:19:58 -0000 1.145
+++ server/mpm/worker/worker.c 27 Jan 2004 19:20:10 -0000
@@ -1441,7 +1441,8 @@
++idle_thread_count;
}
}
- if (any_dead_threads && totally_free_length < idle_spawn_rate
+ if (any_dead_threads && totally_free_length < idle_spawn_rate
+ && free_length < MAX_SPAWN_RATE
&& (!ps->pid /* no process in the slot */
|| ps->quiescing)) { /* or at least one is going away */
if (all_dead_threads) {
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Jeff Trawick <tr...@attglobal.net>.
worker MPM stack corruption in parent:
int free_slots[MAX_SPAWN_RATE];
...
/* great! we prefer these, because the new process can
* start more threads sooner. So prioritize this slot
* by putting it ahead of any slots with active threads.
*
* first, make room by moving a slot that's potentially
still
* in use to the end of the array
*/
NEW CODE-> ap_assert(free_length < MAX_SPAWN_RATE);
free_slots[free_length] =
free_slots[totally_free_length];
[Tue Jan 27 12:20:19 2004] [crit] [Tue Jan 27 12:20:19 2004] file
worker.c, line 1590, assertion "free_length < MAX_SPAWN_RATE" failed
worker MPM parent exits
Posted by Jeff Trawick <tr...@attglobal.net>.
Colm MacCarthaigh wrote:
> On Mon, Jan 26, 2004 at 10:05:33PM +0000, Colm MacCarthaigh wrote:
>
>>>disable the check for geteuid()==0 and see if you get backtrace?
>>>exception hook purposefully doesn't run as root (I assume your parent is
>>>running as root)
>>
>>No problem, first thing tomorrow :)
>>
>
>
> O.k., done that, and now have a useful backtrace (might we worth adding
> a #define to turn those checks off):
yeah... note that it looks like apache-1.3 is about to get the hook
too, but it is enabled via a directive in httpd.conf
(EnableExceptionHook {on|off}).. maybe a special keyword like AllUids
could be used to enable the hook for root without rebuilding the server...
> backtrace for signal 11 from process 0 (parent 7786, thread "pid" 7789)
> main() is at 808e124
> /usr/local/web/modules/mod_backtrace.so[0x550149bf]
> /usr/local/web/modules/mod_backtrace.so[0x55014a34]
> /usr/local/web/bin/httpd(ap_run_fatal_exception+0x30)[0x80947f8]
> /usr/local/web/bin/httpd[0x8094847]
> /usr/local/web/bin/httpd[0x8094880]
not enough entries in backtrace :) I guess that is a stack corruption
issue?
> Attaching gdb to the running process before it dies gets me
> nowhere:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000008 in ?? ()
yeah, been there done that seen that :(
I've been debugging similar problem with 2.0.47 for the last day or
so... took a while to happen upon your discussion within the innocuous
thread "[PATCH] raise MAX_SERVER_LIMIT" ;)
I don't have any hard data yet other than to say I see something roughly
similar under load and something is corrupted with stackframes
preventing cores or controlling parent under gdb or apparently any other
standard techniques from shedding any light on the issue. I'll be
adding some sort of tracing to find out more closely what the parent is
doing when it crashes.
Yesterday I had strace running on parent at the time and the last
syscall showed it killing a child via write("!"), then SIGSEGV.
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Mon, Jan 26, 2004 at 10:05:33PM +0000, Colm MacCarthaigh wrote:
> > disable the check for geteuid()==0 and see if you get backtrace?
> > exception hook purposefully doesn't run as root (I assume your parent is
> > running as root)
>
> No problem, first thing tomorrow :)
>
O.k., done that, and now have a useful backtrace (might we worth adding
a #define to turn those checks off):
backtrace for signal 11 from process 0 (parent 7786, thread "pid" 7789)
main() is at 808e124
/usr/local/web/modules/mod_backtrace.so[0x550149bf]
/usr/local/web/modules/mod_backtrace.so[0x55014a34]
/usr/local/web/bin/httpd(ap_run_fatal_exception+0x30)[0x80947f8]
/usr/local/web/bin/httpd[0x8094847]
/usr/local/web/bin/httpd[0x8094880]
/lib/libpthread.so.0[0x550daf54]
/lib/libc.so.6[0x551126b8]
end of backtrace from process 0
None of the addresses seem to translate to anything useful nm can
produce, I tried subtracting their offset, using only the high-order
bits, using only the low order bits, but it got me nowhere :/ A corefile
backtrace in gdb get me:
#0 0x0808856b in server_main_loop (remaining_children_to_start=Cannot
access memory at address 0xb)
at worker.c:1545
1545 ap_wait_or_timeout(&exitwhy, &status, &pid, pconf);
Attaching gdb to the running process before it dies gets me
nowhere:
Program received signal SIGSEGV, Segmentation fault.
0x00000008 in ?? ()
is all I get when it happens. No backtrace available :( I have however
managed to get the tcpdump of the period - which is 484Megs for about
5 minutes of runtime (it's a busy box!), if anyone wants it - mail
and I'll supply. In the meantime, I'm going to scour the code once
more, to see if there's anything I missed the last 10 times :)
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Mon, Jan 26, 2004 at 04:25:58PM -0500, Jeff Trawick wrote:
> >*sigh*, forensic_id didn't catch it,
>
> forensic_id is just for crash in child
I know, but I couldnt rule out a crash in the child being a root cause
... until now, it doesn't look like it's trigger by a particular URI
anyway.
> >backtrace didn't catch it,
> >whatkilledus didn't catch it, all tried individually.
>
> disable the check for geteuid()==0 and see if you get backtrace?
> exception hook purposefully doesn't run as root (I assume your parent is
> running as root)
No problem, first thing tomorrow :)
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Jeff Trawick <tr...@attglobal.net>.
Colm MacCarthaigh wrote:
> On Mon, Jan 26, 2004 at 06:28:03PM +0000, Colm MacCarthaigh wrote:
>
>>>I'd love to find out what's causing your worker failures. Are you using
>>>any thread-unsafe modules or libraries?
>>
>>Not to my knowledge, I wasn't planning to do this till later, but
>>I've bumped to 2.1, I'll try out the forensic_id and backtrace
>>modules right now, and see how that goes.
>
>
> *sigh*, forensic_id didn't catch it,
forensic_id is just for crash in child
> backtrace didn't catch it,
> whatkilledus didn't catch it, all tried individually.
disable the check for geteuid()==0 and see if you get backtrace?
exception hook purposefully doesn't run as root (I assume your parent is
running as root)
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Ben Laurie <be...@algroup.co.uk>.
Colm MacCarthaigh wrote:
> On Mon, Jan 26, 2004 at 06:28:03PM +0000, Colm MacCarthaigh wrote:
>
>>>I'd love to find out what's causing your worker failures. Are you using
>>>any thread-unsafe modules or libraries?
>>
>>Not to my knowledge, I wasn't planning to do this till later, but
>>I've bumped to 2.1, I'll try out the forensic_id and backtrace
>>modules right now, and see how that goes.
>
>
> *sigh*, forensic_id didn't catch it, backtrace didn't catch it,
> whatkilledus didn't catch it, all tried individually. The parent just
> dumps core; the children live on, serve their content and log their
> request and then drop off one by one. No uncomplete requests,
> no backtrace or other exception info thrown into any log.
>
> corefile is as useful as ever, unbacktracable. suggestions welcome!
mod_log_forensic?
--
http://www.apache-ssl.org/ben.html http://www.thebunker.net/
"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Glenn <gs...@gluelogic.com>.
On Mon, Jan 26, 2004 at 07:37:23PM +0000, Colm MacCarthaigh wrote:
> On Mon, Jan 26, 2004 at 06:28:03PM +0000, Colm MacCarthaigh wrote:
> > > I'd love to find out what's causing your worker failures. Are you using
> > > any thread-unsafe modules or libraries?
> >
> > Not to my knowledge, I wasn't planning to do this till later, but
> > I've bumped to 2.1, I'll try out the forensic_id and backtrace
> > modules right now, and see how that goes.
>
> *sigh*, forensic_id didn't catch it, backtrace didn't catch it,
> whatkilledus didn't catch it, all tried individually. The parent just
> dumps core; the children live on, serve their content and log their
> request and then drop off one by one. No uncomplete requests,
> no backtrace or other exception info thrown into any log.
>
> corefile is as useful as ever, unbacktracable. suggestions welcome!
Have you tried setting up a signal handler for SIGSEGV and calling
kill(getpid(), SIGSTOP);
in the signal handler? After attaching to the process with gdb, send
a CONT signal to the process from another terminal. It's worth a shot.
(Is the process dying from SIGSEGV or some other signal?
Does the core file tell you?)
Can you get a tcpdump of the traffic leading up to the crash?
(Yeah I know it would be a lot)
If you can get a tcpdump, and then can replay the traffic and
reproduce it, more of us can look at this.
Cheers,
Glenn
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Mon, Jan 26, 2004 at 06:28:03PM +0000, Colm MacCarthaigh wrote:
> > I'd love to find out what's causing your worker failures. Are you using
> > any thread-unsafe modules or libraries?
>
> Not to my knowledge, I wasn't planning to do this till later, but
> I've bumped to 2.1, I'll try out the forensic_id and backtrace
> modules right now, and see how that goes.
*sigh*, forensic_id didn't catch it, backtrace didn't catch it,
whatkilledus didn't catch it, all tried individually. The parent just
dumps core; the children live on, serve their content and log their
request and then drop off one by one. No uncomplete requests,
no backtrace or other exception info thrown into any log.
corefile is as useful as ever, unbacktracable. suggestions welcome!
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Mon, Jan 26, 2004 at 10:09:20AM -0800, Aaron Bannert wrote:
> On Thu, Jan 15, 2004 at 04:04:38PM +0000, Colm MacCarthaigh wrote:
> > There were other changes co-incidental to that, like going to 12Gb
> > of RAM, which certainly helped, so it's hard to narrow it down too
> > much.
>
> Ok with 18,000 or so child processes (all in the run queue) what does
> your load look like? Also, what kind of memory footprint are you seeing?
At the time, we were seeing a laod of between 8 and 15, varying like a
sawtooth waveform. It would climb and climb, there'd be a steady sharp
decrease and the cycle would star again. At one point I miscompiled in
the Linux pre-empt options into the Kernel, and that made things very
intresting. Load was much more radically in it's mood-swings then.
There would be points when it would slow to a crawl, and the ammount
of data we shipped was down - we only managed to peak at 200Mbit/sec
during the heaviest part of it. Our daily peak is about 380Mbit,
but hopefully we'll be more ready next time. I've managed to commision
the second server, and move the updates to it, see:
http://ftp.heanet.ie/about/
for an idea of the architecture. As for memory footprint, it wasn't
too bad, I actually put the system into 4Gb mode to avoid bounce-buffering
- something I hadn't fully mapped out yet. We were using all of the RAM,
but that's not unusual for us, we aggressively cache as much of the
filesystem as XFS lets us. All of the Apache instances added up to
about 165Mb of RAM.
> > I don't use worker because it still dumps an un-backtracable corefile
> > within about 5 minutes for me. I still have no idea why, though plenty
> > of corefiles. I havn't tried a serious analysis yet, becasue I've been
> > moving house, but I hope to get to it soon. Moving to worker would be
> > a good thing :)
>
> I'd love to find out what's causing your worker failures. Are you using
> any thread-unsafe modules or libraries?
Not to my knowledge, I wasn't planning to do this till later, but
I've bumped to 2.1, I'll try out the forensic_id and backtrace
modules right now, and see how that goes.
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Aaron Bannert <aa...@clove.org>.
On Thu, Jan 15, 2004 at 04:04:38PM +0000, Colm MacCarthaigh wrote:
> There were other changes co-incidental to that, like going to 12Gb
> of RAM, which certainly helped, so it's hard to narrow it down too
> much.
Ok with 18,000 or so child processes (all in the run queue) what does
your load look like? Also, what kind of memory footprint are you seeing?
> I don't use worker because it still dumps an un-backtracable corefile
> within about 5 minutes for me. I still have no idea why, though plenty
> of corefiles. I havn't tried a serious analysis yet, becasue I've been
> moving house, but I hope to get to it soon. Moving to worker would be
> a good thing :)
I'd love to find out what's causing your worker failures. Are you using
any thread-unsafe modules or libraries?
-aaron
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by Colm MacCarthaigh <co...@stdlib.net>.
On Thu, Jan 15, 2004 at 10:49:43AM -0500, gregames@apache.org wrote:
> >-#define MAX_SERVER_LIMIT 20000
> >+#define MAX_SERVER_LIMIT 100000
>
> dang!
>
> Committed a limit of 200000.
>
> A couple of observations:
>
> * I don't think you could do this with an early 2.4 kernel on i386 because
> of eating up kernel memory with LDTs, assuming APR thinks it can support
> threads. Not sure about current kernels from popular distros.
I'm running 2.6.1-mm2 now, and things are much much better, we got away
with it - just about with 2.6.1 vanilla, but -mm2 has improved the
stability a lot. Peaked at just over 18,000 with 2.6.1-mm2 so far and
it was a lot more bearable.
We were having major stability problems with 2.4, and as soon as we
went to 2.6 we found out why - we were getting a lot more client
requests than we thought, but they were queuing. We hit 20,000 within
2 days of going to 2.6.1-rc2.
There were other changes co-incidental to that, like going to 12Gb
of RAM, which certainly helped, so it's hard to narrow it down too
much.
> * Should I assume you tried worker but it uses too much CPU? If so, is it
> a small percentage more or really really bad?
I don't use worker because it still dumps an un-backtracable corefile
within about 5 minutes for me. I still have no idea why, though plenty
of corefiles. I havn't tried a serious analysis yet, becasue I've been
moving house, but I hope to get to it soon. Moving to worker would be
a good thing :)
If I get time, I'll compile the forensic logging module and see if
I can find if it has a request-specific trigger I can replicate for
testing. I have worker running for months on end on the exact same
platform with much lower yield rates, so I suspect the problem is
only being triggered by the sheer volume of requests.
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
Re: [PATCH] raise MAX_SERVER_LIMIT
Posted by gr...@apache.org.
Colm MacCarthaigh wrote:
> Not entirely serious, but today, we actually hit this, in production :)
> The hardware, a dual 2Ghz Xeon with 12Gb RAM with Linux 2.6.1-rc2 coped,
> and remained responsive. So 20,000 may no longer be outside the realms
> of what administrators reasonably desire to have.
> -#define MAX_SERVER_LIMIT 20000
> +#define MAX_SERVER_LIMIT 100000
dang!
Committed a limit of 200000.
A couple of observations:
* I don't think you could do this with an early 2.4 kernel on i386 because of
eating up kernel memory with LDTs, assuming APR thinks it can support threads.
Not sure about current kernels from popular distros.
* Should I assume you tried worker but it uses too much CPU? If so, is it a
small percentage more or really really bad?
Thanks,
Greg