You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Tani Hosokawa <un...@riverstyx.net> on 1999/06/21 06:00:50 UTC
some questions about threaded apache
I'm still experimenting with the threaded Apache, and I'm noticing some
odd behaviour. I was wondering if someone could clarify things for me.
MinSpareServers 20
MaxSpareServers 40
StartServers 10
MaxClients 23
ThreadsPerChild 64
Now, I assumed that MinSpareServers would be the number of processes, each
with a variable number of threads. So 1 and 2 would seem to be
reasonable. However, when I use those values I get (according to
server-status) 0 idle servers, and I do see a noticeably slower connection
(although once connected I'm quick, presumably because I'm reusing my
connection). When I increased those values to 20/40, it never kills any
threads, so I always have a total of 1454 threads (I chose those numbers
because I've limited NR_TASKS to 1500 per user). This doesn't really pose
much of a problem to me, but I would assume that other people would like
the number of threads/processes to decrease when load dies down :-)
As an aside, threaded Apache seems to handle load MUCH better than normal
Apache... 1.3.6 showing a consistent load of about 6-9 only has a load of
1.2 on threaded apache :)
---
tani hosokawa
river styx internet
Re: some questions about threaded apache
Posted by Tani Hosokawa <un...@riverstyx.net>.
On Sun, 20 Jun 1999, Manoj Kasichainula wrote:
> On Sun, Jun 20, 1999 at 09:00:50PM -0700, Tani Hosokawa wrote:
> > Now, I assumed that MinSpareServers would be the number of processes, each
> > with a variable number of threads.
>
> Nope. The server in the apache-apr directory has a variable number of
> processes and a fixed number of threads right now.
>
> > So 1 and 2 would seem to be reasonable.
>
> I would think so.
>
> > However, when I use those values I get (according to
> > server-status) 0 idle servers, and I do see a noticeably slower connection
> > (although once connected I'm quick, presumably because I'm reusing my
> > connection).
>
> That's weird. My very hasty tests don't show this problem. Which
> modules do you have loaded? How much slower a connection are we
> talking about here?
>
> How many concurrent connections is your server running at? Most of our
> testing before was with ApacheBench, and the number of concurrent
> connections went up to around 100 (though ThreadsPerChild was low
> too). So I'm guessing you've been the best tester so far.
I've got mod_rewrite compiled in, I removed mod_userdir, I added
-DDYNAMIC_MODULE_LIMIT=0, other than that it's stock source (with
HARD_THREAD_LIMIT and HARD_SERVER_LIMIT increased of course). I've tested
two servers, one with about 500 connections, the other was at about 1200
before I left (and it died at some point).
I'm talking about a 3-5 second wait before the connection actually
establishes. After that initial connection, the problem seems to pretty
much go away, until I try to connect from somewhere else.
> > When I increased those values to 20/40, it never kills any
> > threads, so I always have a total of 1454 threads (I chose those numbers
> > because I've limited NR_TASKS to 1500 per user).
>
> Hmmm, there are actually more than 64 threads in a 64-thread server.
> There's the sigwait thread and the Linuxthreads monitor thread. So you
> might hit NR_TASKS.
Well, there's virtually no CGI happening on this server, so I doubt it'd
eat up the remaining tasks...
> I'm curious if mod_status is lying or not. Do ps and pstree agree with
> what mod_status is telling you?
pstree seems to agree with what I'm seeing. The behaviour I saw before
seems to have evened out a little, but I've still got 1132 servers
running, with only about 500 actively serving requests. The server that
was working really hard (about 1200 active connections) seems to have
crashed somewhere, and I wasn't around to watch... :(
|-httpd-apr-+-buildstats
| |-2*[httpd-apr---httpd-apr---httpd-apr]
| |-httpd-apr---httpd-apr---2*[httpd-apr]
| `-20*[httpd-apr---httpd-apr---64*[httpd-apr]]
> Also, I did notice one problem a while back when I tried to bang hard
> on the the hybrid server with default user settings in Red Hat (256
> processes). When we hit that limit, the server hung in
> pthread_create. My guess is that the main thread sends a message to
> the monitor thread, which then chokes on the clone() call, and doesn't
> wake up the main thread. Hijinks then ensue.
I've seen similar behaviour under the preforked Apache. I get a bunch of
unkillable processes that don't do anything but hang on to the the bound
socket so I can't start another Apache, so I end up needing to reboot...
> > As an aside, threaded Apache seems to handle load MUCH better than normal
> > Apache... 1.3.6 showing a consistent load of about 6-9 only has a load of
> > 1.2 on threaded apache :)
> Cool. That was the intention. I was worried that the benefit of
> threading wouldn't be as great on Linux because processes and threads
> are so similar.
---
tani hosokawa
river styx internet
Re: some questions about threaded apache
Posted by Manoj Kasichainula <ma...@io.com>.
On Sun, Jun 20, 1999 at 09:00:50PM -0700, Tani Hosokawa wrote:
> Now, I assumed that MinSpareServers would be the number of processes, each
> with a variable number of threads.
Nope. The server in the apache-apr directory has a variable number of
processes and a fixed number of threads right now.
> So 1 and 2 would seem to be reasonable.
I would think so.
> However, when I use those values I get (according to
> server-status) 0 idle servers, and I do see a noticeably slower connection
> (although once connected I'm quick, presumably because I'm reusing my
> connection).
That's weird. My very hasty tests don't show this problem. Which
modules do you have loaded? How much slower a connection are we
talking about here?
How many concurrent connections is your server running at? Most of our
testing before was with ApacheBench, and the number of concurrent
connections went up to around 100 (though ThreadsPerChild was low
too). So I'm guessing you've been the best tester so far.
> When I increased those values to 20/40, it never kills any
> threads, so I always have a total of 1454 threads (I chose those numbers
> because I've limited NR_TASKS to 1500 per user).
Hmmm, there are actually more than 64 threads in a 64-thread server.
There's the sigwait thread and the Linuxthreads monitor thread. So you
might hit NR_TASKS.
I'm curious if mod_status is lying or not. Do ps and pstree agree with
what mod_status is telling you?
Also, I did notice one problem a while back when I tried to bang hard
on the the hybrid server with default user settings in Red Hat (256
processes). When we hit that limit, the server hung in
pthread_create. My guess is that the main thread sends a message to
the monitor thread, which then chokes on the clone() call, and doesn't
wake up the main thread. Hijinks then ensue.
> As an aside, threaded Apache seems to handle load MUCH better than normal
> Apache... 1.3.6 showing a consistent load of about 6-9 only has a load of
> 1.2 on threaded apache :)
Cool. That was the intention. I was worried that the benefit of
threading wouldn't be as great on Linux because processes and threads
are so similar.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"Firewalls don't know the difference between a virus, a Trojan Horse, or
Windows NT." -- An advertising brochure from DataLynx, Inc. ("SECURITY THROUGH
STRENGTH")
Re: some questions about threaded apache
Posted by Tani Hosokawa <un...@riverstyx.net>.
On Mon, 21 Jun 1999, Ryan Bloom wrote:
> In the current implementation, threads don't die on their own. The only
> time we spawn new threads, or kill old ones, is when a whole new process
> is created, or a whole procesw goes away. This may change in the future,
> but spawning and killing threads is something that most pthread libraries
> don't do well.
Hmm. Is there any way you could keep new thread creation to one process
until it's full, instead of just randomly selecting one? I assume that
behaviour is because there are several processes waiting on a socket and
the kernel hands it off to one of them at random, but if you could have
just one thread waiting, then passing it off, it'd be handy... otherwise,
idle processes will never be culled. I'd like to get that extra memory
back sometime to use for disk caching, etc, since at peak times the server
gets up to around 1300 active threads and at low time it gets down to 400
:)
> The 0 idle servers may be due to how we are computing the number of idle
> servers. Currently, it is based on the total number of threads in the
> server.
>
> And it's REALLY good to know the threaded Apache is able to handle a real
> world load better than the process based one. :)
Well, according to server-status right now, there are 911 requests being
processed, and ps reports that there are 953 httpd-apr processes running.
That's listing the main parent, 20 children, and that other thread that
doesn't actually process requests, so 912 threads. I'm guessing that
extra one is just because it got spawned while I was switching windows.
I'd assume that would mean there really are only 0 available threads. The
scoreboard doesn't show any threads that are just waiting for connections:
967 connections currently being processed, 0 idle servers
KKKKKKLWKKWKKWWWKKKWWKKKKKKWKWKKWKKKWWWKWKKWKKKWKKKWKKWKKWKKKKKK
...................................................W......K.....
KKKWKWKKKLKKKKLKKLKKKKKKKKKKWKWKKKKKWKKKWKKKKKKKKWKKKKWKKWKKWKRK
KKWKKKKLKKKKWKKKKKKWWKKKWKKKKKWKWWWKKKKKKKKKKKKWWWWKKKKKKKKWKKKK
KKKKKKKKKWKWKWKWKKKLKWWKKKKWKKKWKKWKKKKKKKKKKKWKKKKKKKKKKWKKKKKK
KKKKWKWKKKKKWKKKWKKKKKKKWWWKKKKKKKKKKWKKKKWKWKKKKWKKKKWKKKKKKWKW
WKKKKKKWKKKKWKWKKKKKWKWWKKWKWKKWKKKWKKWWKKKKKKKRKKKWKKKKKKKKKKKK
WKWKKWKKKLKKKKKKKKWKKKWKWKKKKKKKWWKKWKKKKKKKKKKKKWWKWKKWKWWKWWKK
...................W......W.....................................
KKKWWKWKWKKWKKKKWKKKWKWKWKKKKKKKKKKKKWWKKKWWRKKKWKWKKKKRKKKKKKKK
KKKWKWKKKWWKKWWKKWLKWKWKKKKWWWWKKKKKKKKLKKKKWKKKKWWKWKWKKKKWKKWK
KKKKKKKKKKKKKKWWKWWKKKWKKKKWWKKKWWKKKKKKKKKKKKKWKKKWKKKWKKKKKWKW
...........................W....................................
KWKKWKKKKKKKWWKWKKWKKKKKWWWWWWKLKKKKKKWKKKKKKKKKKKKLKKKKKKKKKWKW
KKKKKWWKWKLWWWKWWKKWKKKWKKKKKKKKWWWKKKKKKKKKWKKKKKWKKKKKWKLWWKKK
.....W..........................................................
KKKWKKWWKKWKKKWKKKKKKKWWWKKLWKWWWWKKKKKKKWWKKKKKKKKKKWWWWKKKKKKK
KKKKKWKWKKKWKKKWKKKKWKKKKKWWKKKKWKKKKKWKKKKKWWKKKKKKKKKKKKKWKKKK
............W...................................................
KKKKKKKKKKKLWKKWKKKKWKKKKKKWKKKKWWWKWKKKKWKWKKKWLKWKKWWKKWKKKKKW
................................................................
................................................................
................................................................
................................................................
So that means every connection from here on in is going to require a
spawning of a thread (which appears to take a fair bit of time sometimes).
Any suggestions on what I might do about this? There are a couple
processes kicking about there that only have one or two threads in them,
but something's gone awry here, I think.
---
tani hosokawa
river styx internet
Re: some questions about threaded apache
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
> I'm still experimenting with the threaded Apache, and I'm noticing some
> odd behaviour. I was wondering if someone could clarify things for me.
>
> MinSpareServers 20
> MaxSpareServers 40
> StartServers 10
> MaxClients 23
> ThreadsPerChild 64
>
> Now, I assumed that MinSpareServers would be the number of processes, each
> with a variable number of threads. So 1 and 2 would seem to be
> reasonable. However, when I use those values I get (according to
> server-status) 0 idle servers, and I do see a noticeably slower connection
> (although once connected I'm quick, presumably because I'm reusing my
> connection). When I increased those values to 20/40, it never kills any
> threads, so I always have a total of 1454 threads (I chose those numbers
> because I've limited NR_TASKS to 1500 per user). This doesn't really pose
> much of a problem to me, but I would assume that other people would like
> the number of threads/processes to decrease when load dies down :-)
In the current implementation, threads don't die on their own. The only
time we spawn new threads, or kill old ones, is when a whole new process
is created, or a whole procesw goes away. This may change in the future,
but spawning and killing threads is something that most pthread libraries
don't do well.
The 0 idle servers may be due to how we are computing the number of idle
servers. Currently, it is based on the total number of threads in the
server.
And it's REALLY good to know the threaded Apache is able to handle a real
world load better than the process based one. :)
Ryan
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.