You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Tani Hosokawa <un...@riverstyx.net> on 1999/06/21 10:55:56 UTC
threaded apache question
|-httpd-apr-+-buildstats
| |-11*[httpd-apr---httpd-apr---64*[httpd-apr]]
| |-httpd-apr---httpd-apr---3*[httpd-apr]
| |-6*[httpd-apr---httpd-apr---httpd-apr]
| |-httpd-apr---httpd-apr---5*[httpd-apr]
| `-httpd-apr---httpd-apr---4*[httpd-apr]
Manoj, you said that the number of threads is fixed, with a variable
number of processes? Why would I be seeing the above? I'm assuming those
processes with just a few threads showing up correspond to these lines in
the scoreboard:
77.3 requests/sec - 0.7 MB/second - 8.7 kB/request
900 connections currently being processed, 68 idle servers
.K...K......W....L..W.....W.K....K..........W...K....K.........W
.......K.W......................................................
..W.W.W.....W.....W....W......................................K.
.W..............................................................
.............................W........W.........................
..........W.....................................................
..............W.................................................
..W.KW.........W.W....K........W.....WWW.K....W.................
W...........W................K....K.............................
If I understand what you're saying correctly, each process should always
have 64 threads, with some just being inactive (marked by _, not .),
correct? Or does it take a while to ramp up to this? It seems kind of
silly to randomly sprinkle the connections through the various processes,
if there's a maximum number of spare processes set. None of the processes
can exit then, because there's always one or two threads running around in
that process, when those threads should be all living in one process...
---
tani hosokawa
river styx internet
Re: threaded apache question
Posted by Tani Hosokawa <un...@riverstyx.net>.
On Tue, 22 Jun 1999, Manoj Kasichainula wrote:
> > In all cases, when I tried to quit and disconnect from gdb it hung and I
> > had to ^Z, then kill explicitly. One was in the second state and had a
> > second thread in that processes, which I attempted to attach to but
> > couldn't... it just hung.
>
> I've had problems with gdb and threads before, at least with kernel &
> glibc 2.0.
>
> > > Also, what is MaxRequestsPerChild set to? I haven't seen any
> > > MaxRequestsPerChild bugs in the code lately, but some could still be
> > > lurking.
> >
> > 10000, so nothing ought to be going awry there...
>
> Hmmmmm...
>
> You said you had problems under 1.3 also? I'm not sure what to say if
> this is the same problem. If you change that 10000 to something like
> 10^7, does this occur less frequently?
I'll give that s try...
> Also, which kernel and glibc are on this machine?
>
> I realize I'm not being very helpful here. :)
2.2.10 and 2.2.9, it occurs on both I believe... glibc is whatever shipped
with RH5.2 (2.0.7 I think).
---
tani hosokawa
river styx internet
Re: threaded apache question
Posted by Manoj Kasichainula <ma...@io.com>.
On Mon, Jun 21, 1999 at 02:33:53PM -0700, Tani Hosokawa wrote:
> found one like this:
>
> #0 0x400e02de in __select ()
> #1 0x21 in ?? ()
> #2 0x806c57e in write_with_errors ()
> #3 0x806c64d in bcwrite ()
> #4 0x806cad5 in ap_bwrite ()
> #5 0x807b8d6 in ap_send_mmap ()
The "??" is probably sendwithtimeout. As you've described, it's
hanging on the socket.
> found several like this:
>
> #0 0x400e54f4 in __syscall_writev ()
> #1 0x401096cc in __DTOR_END__ ()
> #2 0x806c401 in writev_it_all ()
> #3 0x806c825 in large_write ()
> #4 0x806c8f1 in ap_bwrite ()
> #5 0x807b8d6 in ap_send_mmap ()
ditto
> In all cases, when I tried to quit and disconnect from gdb it hung and I
> had to ^Z, then kill explicitly. One was in the second state and had a
> second thread in that processes, which I attempted to attach to but
> couldn't... it just hung.
I've had problems with gdb and threads before, at least with kernel &
glibc 2.0.
> > Also, what is MaxRequestsPerChild set to? I haven't seen any
> > MaxRequestsPerChild bugs in the code lately, but some could still be
> > lurking.
>
> 10000, so nothing ought to be going awry there...
Hmmmmm...
You said you had problems under 1.3 also? I'm not sure what to say if
this is the same problem. If you change that 10000 to something like
10^7, does this occur less frequently?
Also, which kernel and glibc are on this machine?
I realize I'm not being very helpful here. :)
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"I'm sorry if this is not the right word, but I learned English on Usenet."
-- Miguel de Icaza, Linux Expo '98
Re: threaded apache question
Posted by Tani Hosokawa <un...@riverstyx.net>.
On Mon, 21 Jun 1999, Manoj Kasichainula wrote:
> On Mon, Jun 21, 1999 at 01:55:56AM -0700, Tani Hosokawa wrote:
> > .K...K......W....L..W.....W.K....K..........W...K....K.........W
> > .......K.W......................................................
> > ..W.W.W.....W.....W....W......................................K.
> > .W..............................................................
> > .............................W........W.........................
> > ..........W.....................................................
> > ..............W.................................................
> > ..W.KW.........W.W....K........W.....WWW.K....W.................
> > W...........W................K....K.............................
> >
> > If I understand what you're saying correctly, each process should always
> > have 64 threads, with some just being inactive (marked by _, not .),
> > correct?
>
> Correct.
>
> It looks like all of these processes is trying to die from hitting
> MaxRequestsPerChild (or did something initiate a graceful restart?),
> but not all of the threads are reaching the point where they learn
> it's time to die.
The server had been running for about half an hour at that point, but I
hadn't done anything to it.
> Could you attach a debugger to some of these errant threads to see
> what they're actually doing? Are they hung on a socket, like you were
> describing for 1.3?
found one like this:
#0 0x400e02de in __select ()
#1 0x21 in ?? ()
#2 0x806c57e in write_with_errors ()
#3 0x806c64d in bcwrite ()
#4 0x806cad5 in ap_bwrite ()
#5 0x807b8d6 in ap_send_mmap ()
#6 0x80743b5 in default_handler ()
#7 0x806db0e in ap_invoke_handler ()
#8 0x807ed11 in process_request_internal ()
#9 0x807ed7a in ap_process_request ()
#10 0x8076753 in process_socket ()
#11 0x80768d9 in worker_thread ()
#12 0x40069357 in pthread_start_thread (arg=0xb7fffea4) at manager.c:192
found several like this:
#0 0x400e54f4 in __syscall_writev ()
#1 0x401096cc in __DTOR_END__ ()
#2 0x806c401 in writev_it_all ()
#3 0x806c825 in large_write ()
#4 0x806c8f1 in ap_bwrite ()
#5 0x807b8d6 in ap_send_mmap ()
#6 0x807437c in default_handler ()
#7 0x806db0e in ap_invoke_handler ()
#8 0x807ed11 in process_request_internal ()
#9 0x807ed7a in ap_process_request ()
#10 0x8076753 in process_socket ()
#11 0x80768d9 in worker_thread ()
#12 0x40069357 in pthread_start_thread (arg=0xb95ffea4) at manager.c:192
In all cases, when I tried to quit and disconnect from gdb it hung and I
had to ^Z, then kill explicitly. One was in the second state and had a
second thread in that processes, which I attempted to attach to but
couldn't... it just hung.
> Also, what is MaxRequestsPerChild set to? I haven't seen any
> MaxRequestsPerChild bugs in the code lately, but some could still be
> lurking.
10000, so nothing ought to be going awry there...
---
tani hosokawa
river styx internet
Re: threaded apache question
Posted by Manoj Kasichainula <ma...@io.com>.
On Mon, Jun 21, 1999 at 01:55:56AM -0700, Tani Hosokawa wrote:
> .K...K......W....L..W.....W.K....K..........W...K....K.........W
> .......K.W......................................................
> ..W.W.W.....W.....W....W......................................K.
> .W..............................................................
> .............................W........W.........................
> ..........W.....................................................
> ..............W.................................................
> ..W.KW.........W.W....K........W.....WWW.K....W.................
> W...........W................K....K.............................
>
> If I understand what you're saying correctly, each process should always
> have 64 threads, with some just being inactive (marked by _, not .),
> correct?
Correct.
It looks like all of these processes is trying to die from hitting
MaxRequestsPerChild (or did something initiate a graceful restart?),
but not all of the threads are reaching the point where they learn
it's time to die.
Could you attach a debugger to some of these errant threads to see
what they're actually doing? Are they hung on a socket, like you were
describing for 1.3?
Also, what is MaxRequestsPerChild set to? I haven't seen any
MaxRequestsPerChild bugs in the code lately, but some could still be
lurking.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"The early bird who catches the worm works for someone who comes in late
and owns the worm farm." -- Travis McGee