You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Jeff Mohler <sp...@gmail.com> on 2006/09/29 06:31:49 UTC
[users@httpd] Debugging a runaway httpd process
I am suffering from runaway httpd processes, and I dont understand
enough about systems at this level, to understand why.
I asked my server "admin" (IE: Smart helper) to add PHP support and a
few other minor things to the environment, which has been stable since
2002.
Now, the httpd process randomly starts to run away with the whole machine.
I stumbled into ktrace, and Ive found some fnctl calls that go
nowhere, and tonight I caught it starting actually:
---
99590 httpd CALL gettimeofday(0xbfbff9b8,0)
99590 httpd RET gettimeofday 0
99590 httpd CALL kill(0x1ad8,0x1e)
99590 httpd RET kill 0
99590 httpd CALL wait4(0xffffffff,0xbfbffac8,0x1,0)
99590 httpd RET wait4 0
99590 httpd PSIG SIGCHLD caught handler=0x28bbc510 mask=0x0 code=0x0
99590 httpd CALL sigreturn(0xbfbff710)
99590 httpd RET sigreturn JUSTRETURN
99590 httpd CALL read(0x36,0x81bef20,0x80)
99590 httpd RET read -1 errno 35 Resource temporarily unavailable
99590 httpd CALL fcntl(0,0x4,0x4)
99590 httpd RET fcntl -1 errno 19 Operation not supported by device
99590 httpd CALL fcntl(0x1,0x4,0x5)
99590 httpd RET fcntl -1 errno 19 Operation not supported by device
99590 httpd CALL fcntl(0x2,0x4,0xd)
99590 httpd RET fcntl 0
99590 httpd CALL fcntl(0,0x4,0x4)
99590 httpd RET fcntl -1 errno 19 Operation not supported by device
99590 httpd CALL fcntl(0x1,0x4,0x5)
99590 httpd RET fcntl -1 errno 19 Operation not supported by device
---
and so on...
The system:
DNS1# httpd -v
Server version: Apache/1.3.37 (Unix)
Server built: Sep 4 2006 20:16:45
DNS1# uname -a
FreeBSD DNS1.vidnet.net 5.0-RELEASE FreeBSD 5.0-RELEASE #0: Thu Jan 16
22:16:53 GMT 2003
root@hollin.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC i386
My smart helper doesnt know what to do next, but Id like to learn a
few things and see whats going on in the heart of the beast.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: Debugging a runaway httpd process
Posted by Jeff Mohler <sp...@gmail.com>.
Re-sending, in the hope that someone can assist with this mystery
runaway httpd problem I am having.
Thanks again.
On 9/28/06, Jeff Mohler <sp...@gmail.com> wrote:
> I am suffering from runaway httpd processes, and I dont understand
> enough about systems at this level, to understand why.
>
> I asked my server "admin" (IE: Smart helper) to add PHP support and a
> few other minor things to the environment, which has been stable since
> 2002.
>
> Now, the httpd process randomly starts to run away with the whole machine.
>
> I stumbled into ktrace, and Ive found some fnctl calls that go
> nowhere, and tonight I caught it starting actually:
>
> ---
> 99590 httpd CALL gettimeofday(0xbfbff9b8,0)
> 99590 httpd RET gettimeofday 0
> 99590 httpd CALL kill(0x1ad8,0x1e)
> 99590 httpd RET kill 0
> 99590 httpd CALL wait4(0xffffffff,0xbfbffac8,0x1,0)
> 99590 httpd RET wait4 0
> 99590 httpd PSIG SIGCHLD caught handler=0x28bbc510 mask=0x0 code=0x0
> 99590 httpd CALL sigreturn(0xbfbff710)
> 99590 httpd RET sigreturn JUSTRETURN
> 99590 httpd CALL read(0x36,0x81bef20,0x80)
> 99590 httpd RET read -1 errno 35 Resource temporarily unavailable
> 99590 httpd CALL fcntl(0,0x4,0x4)
> 99590 httpd RET fcntl -1 errno 19 Operation not supported by device
> 99590 httpd CALL fcntl(0x1,0x4,0x5)
> 99590 httpd RET fcntl -1 errno 19 Operation not supported by device
> 99590 httpd CALL fcntl(0x2,0x4,0xd)
> 99590 httpd RET fcntl 0
> 99590 httpd CALL fcntl(0,0x4,0x4)
> 99590 httpd RET fcntl -1 errno 19 Operation not supported by device
> 99590 httpd CALL fcntl(0x1,0x4,0x5)
> 99590 httpd RET fcntl -1 errno 19 Operation not supported by device
> ---
>
> and so on...
>
> The system:
> DNS1# httpd -v
> Server version: Apache/1.3.37 (Unix)
> Server built: Sep 4 2006 20:16:45
> DNS1# uname -a
> FreeBSD DNS1.vidnet.net 5.0-RELEASE FreeBSD 5.0-RELEASE #0: Thu Jan 16
> 22:16:53 GMT 2003
> root@hollin.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC i386
>
>
> My smart helper doesnt know what to do next, but Id like to learn a
> few things and see whats going on in the heart of the beast.
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Re: Re: [users@httpd] Debugging a runaway httpd process
Posted by Jeff Mohler <sp...@gmail.com>.
I apologise, I re-sent the wrong Email with the wrong debug output.
Here is the proper output.
On 9/29/06, Jeff Mohler <sp...@gmail.com> wrote:
> Finally got one to run away..
>
> gdb output of the runaway gives:
>
> (gdb) where
> #0 0x28bbcec0 in _thread_sig_handle_pending () from /usr/lib/libc_r.so.5
> #1 0x28bbc780 in _thread_sig_handler () from /usr/lib/libc_r.so.5
> #2 0x28bbc981 in _thread_sig_handler () from /usr/lib/libc_r.so.5
> #3 0x28bbd9f8 in _thread_sig_wrapper () from /usr/lib/libc_r.so.5
> #4 0xbfbfd010 in ?? ()
> #5 0x28bbd3e0 in _thread_sig_handle_pending () from /usr/lib/libc_r.so.5
> #6 0x28bbcd3f in _thread_sig_check_pending () from /usr/lib/libc_r.so.5
> #7 0x28bc2c59 in _thread_kern_scheduler () from /usr/lib/libc_r.so.5
> (gdb)
>
>
> gdb of healthy processes gives different data:
> (gdb) where
> #0 0x2811d9b3 in read () from /usr/lib/libc.so.5
> #1 0x08051212 in ap_read ()
> #2 0x08051703 in read_with_errors ()
> #3 0x0805197a in ap_bgets ()
> #4 0x08060543 in ap_getline ()
> #5 0x080607b9 in read_request_line ()
> #6 0x08060e1d in ap_read_request ()
> #7 0x0805debb in child_main ()
> #8 0x0805e06a in make_child ()
> #9 0x0805e0f8 in startup_children ()
> #10 0x0805e726 in standalone_main ()
> #11 0x0805eef7 in main ()
> #12 0x0804e595 in _start ()
>
> When one httpd goes bad, others quickly follow as such:
>
> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
> 1397 www 129 0 18972K 12604K RUN 1:03 21.19% 21.19% httpd
> 1398 www 129 0 18972K 12604K RUN 1:01 21.09% 21.09% httpd
> 1329 www 129 0 18984K 12608K RUN 1:01 21.00% 21.00% httpd
> 1415 www 129 0 18984K 12608K RUN 0:15 20.65% 20.65% httpd
> 1423 smmsp -8 0 3720K 3192K biord 0:01 0.05% 0.05% sendmail
> 1258 www 96 0 18992K 12624K STOP 5:51 0.00% 0.00% httpd
> 403 mysql 96 0 45824K 21420K select 0:18 0.00% 0.00% mysqld
> 242 bind 96 0 5740K 5036K select 0:12 0.00% 0.00% named
> 1062 www 4 0 23740K 17580K sbwait 0:01 0.00% 0.00% httpd
> 446 root 96 0 18908K 12544K select 0:01 0.00% 0.00% httpd
>
>
> The newer runaway httpds show the same gdb output as well.
>
>
>
> On 9/29/06, Joshua Slive <jo...@slive.ca> wrote:
> > On 9/29/06, Jeff Mohler <sp...@gmail.com> wrote:
> > > I am suffering from runaway httpd processes, and I dont understand
> > > enough about systems at this level, to understand why.
> > >
> > > I asked my server "admin" (IE: Smart helper) to add PHP support and a
> > > few other minor things to the environment, which has been stable since
> > > 2002.
> > >
> > > Now, the httpd process randomly starts to run away with the whole machine.
> > >
> > > I stumbled into ktrace, and Ive found some fnctl calls that go
> > > nowhere, and tonight I caught it starting actually:
> >
> > A gdb backtrace might be more informative:
> > http://httpd.apache.org/dev/debugging.html#backtrace
> >
> > Joshua.
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > " from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> >
> >
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Re: [users@httpd] Debugging a runaway httpd process
Posted by Jeff Mohler <sp...@gmail.com>.
Finally got one to run away..
gdb output of the runaway gives:
(gdb) where
#0 0x28bbcec0 in _thread_sig_handle_pending () from /usr/lib/libc_r.so.5
#1 0x28bbc780 in _thread_sig_handler () from /usr/lib/libc_r.so.5
#2 0x28bbc981 in _thread_sig_handler () from /usr/lib/libc_r.so.5
#3 0x28bbd9f8 in _thread_sig_wrapper () from /usr/lib/libc_r.so.5
#4 0xbfbfd010 in ?? ()
#5 0x28bbd3e0 in _thread_sig_handle_pending () from /usr/lib/libc_r.so.5
#6 0x28bbcd3f in _thread_sig_check_pending () from /usr/lib/libc_r.so.5
#7 0x28bc2c59 in _thread_kern_scheduler () from /usr/lib/libc_r.so.5
(gdb)
gdb of healthy processes gives different data:
(gdb) where
#0 0x2811d9b3 in read () from /usr/lib/libc.so.5
#1 0x08051212 in ap_read ()
#2 0x08051703 in read_with_errors ()
#3 0x0805197a in ap_bgets ()
#4 0x08060543 in ap_getline ()
#5 0x080607b9 in read_request_line ()
#6 0x08060e1d in ap_read_request ()
#7 0x0805debb in child_main ()
#8 0x0805e06a in make_child ()
#9 0x0805e0f8 in startup_children ()
#10 0x0805e726 in standalone_main ()
#11 0x0805eef7 in main ()
#12 0x0804e595 in _start ()
When one httpd goes bad, others quickly follow as such:
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
1397 www 129 0 18972K 12604K RUN 1:03 21.19% 21.19% httpd
1398 www 129 0 18972K 12604K RUN 1:01 21.09% 21.09% httpd
1329 www 129 0 18984K 12608K RUN 1:01 21.00% 21.00% httpd
1415 www 129 0 18984K 12608K RUN 0:15 20.65% 20.65% httpd
1423 smmsp -8 0 3720K 3192K biord 0:01 0.05% 0.05% sendmail
1258 www 96 0 18992K 12624K STOP 5:51 0.00% 0.00% httpd
403 mysql 96 0 45824K 21420K select 0:18 0.00% 0.00% mysqld
242 bind 96 0 5740K 5036K select 0:12 0.00% 0.00% named
1062 www 4 0 23740K 17580K sbwait 0:01 0.00% 0.00% httpd
446 root 96 0 18908K 12544K select 0:01 0.00% 0.00% httpd
The newer runaway httpds show the same gdb output as well.
On 9/29/06, Joshua Slive <jo...@slive.ca> wrote:
> On 9/29/06, Jeff Mohler <sp...@gmail.com> wrote:
> > I am suffering from runaway httpd processes, and I dont understand
> > enough about systems at this level, to understand why.
> >
> > I asked my server "admin" (IE: Smart helper) to add PHP support and a
> > few other minor things to the environment, which has been stable since
> > 2002.
> >
> > Now, the httpd process randomly starts to run away with the whole machine.
> >
> > I stumbled into ktrace, and Ive found some fnctl calls that go
> > nowhere, and tonight I caught it starting actually:
>
> A gdb backtrace might be more informative:
> http://httpd.apache.org/dev/debugging.html#backtrace
>
> Joshua.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Re: [users@httpd] Debugging a runaway httpd process
Posted by Jeff Mohler <sp...@gmail.com>.
Thanks Joshua, I will dig into that today to see what I can find.
On 9/29/06, Joshua Slive <jo...@slive.ca> wrote:
> On 9/29/06, Jeff Mohler <sp...@gmail.com> wrote:
> > I am suffering from runaway httpd processes, and I dont understand
> > enough about systems at this level, to understand why.
> >
> > I asked my server "admin" (IE: Smart helper) to add PHP support and a
> > few other minor things to the environment, which has been stable since
> > 2002.
> >
> > Now, the httpd process randomly starts to run away with the whole machine.
> >
> > I stumbled into ktrace, and Ive found some fnctl calls that go
> > nowhere, and tonight I caught it starting actually:
>
> A gdb backtrace might be more informative:
> http://httpd.apache.org/dev/debugging.html#backtrace
>
> Joshua.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Debugging a runaway httpd process
Posted by Joshua Slive <jo...@slive.ca>.
On 9/29/06, Jeff Mohler <sp...@gmail.com> wrote:
> I am suffering from runaway httpd processes, and I dont understand
> enough about systems at this level, to understand why.
>
> I asked my server "admin" (IE: Smart helper) to add PHP support and a
> few other minor things to the environment, which has been stable since
> 2002.
>
> Now, the httpd process randomly starts to run away with the whole machine.
>
> I stumbled into ktrace, and Ive found some fnctl calls that go
> nowhere, and tonight I caught it starting actually:
A gdb backtrace might be more informative:
http://httpd.apache.org/dev/debugging.html#backtrace
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org