You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Paul A Houle <ph...@cornell.edu> on 2005/06/29 17:40:56 UTC
how do i debug a segfault?
This weekend we had the kind of experience with Apache httpd which
we expect from Microsoft IIS or Tomcat.
We're running a self-compiled 2.0.54 on RHEL 4 on x86_64 on a 4-way
machine.
Our server got kicked around midnight to rotate logs, but around
5AM we started getting a large volume (> 1 /sec) of messages like
[Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
Segmentation fault (11)
[Tue Jun 28 14:45:53 2005] [notice] child pid 28183 exit signal
Segmentation fault (11)
[Tue Jun 28 14:45:53 2005] [notice] child pid 28184 exit signal
Segmentation fault (11)
This server isn't very heavily loaded, it's lucky if it's getting
10000 hits/day at this point. The site still uses CGI extensively:
some CGIs worked just fine, but other CGIs failed with a 0 length
document, I think nothing in the log.
Kicking the server resolved the problem, at least for now.
It has ExtendedStatus on and my hunches are: (i) the problem is
x86_64 specific (haven't seen this on a heavily loaded x86 machine) and
(ii) the underlying problem is in server global state.
One obvious step is to set up monitoring of stderr (as has been
discussed) to page me and maybe auto-kick the server if this happens
again -- but I'd like to see a real fix.
Re: how do i debug a segfault?
Posted by Paul Querna <ch...@force-elite.com>.
Paul A Houle wrote:
> This weekend we had the kind of experience with Apache httpd which
> we expect from Microsoft IIS or Tomcat.
>
> We're running a self-compiled 2.0.54 on RHEL 4 on x86_64 on a 4-way
> machine.
>
> Our server got kicked around midnight to rotate logs, but around
> 5AM we started getting a large volume (> 1 /sec) of messages like
>
> [Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
> Segmentation fault (11)
> [Tue Jun 28 14:45:53 2005] [notice] child pid 28183 exit signal
> Segmentation fault (11)
> [Tue Jun 28 14:45:53 2005] [notice] child pid 28184 exit signal
> Segmentation fault (11)
>
> This server isn't very heavily loaded, it's lucky if it's getting
> 10000 hits/day at this point. The site still uses CGI extensively:
> some CGIs worked just fine, but other CGIs failed with a 0 length
> document, I think nothing in the log.
>
> Kicking the server resolved the problem, at least for now.
>
> It has ExtendedStatus on and my hunches are: (i) the problem is
> x86_64 specific (haven't seen this on a heavily loaded x86 machine)
> and (ii) the underlying problem is in server global state.
>
> One obvious step is to set up monitoring of stderr (as has been
> discussed) to page me and maybe auto-kick the server if this happens
> again -- but I'd like to see a real fix.
Setup CoreDumpDirectory:
http://httpd.apache.org/docs-2.0/mod/core.html#CoreDumpDirectory
Once you get a coredump, we can actually debug it.
Re: how do i debug a segfault?
Posted by "Akins, Brian" <Br...@turner.com>.
On 6/29/05 1:39 PM, "Paul A Houle" <ph...@cornell.edu> wrote:
> Akins, Brian wrote:
>
>> Sorry if I missed it, which mpm are you using?
>>
>>
>>
> prefork
>
For prefork, just follow the directions I gave and forget all the "thread"
stuff.
--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies
Re: how do i debug a segfault?
Posted by Paul A Houle <ph...@cornell.edu>.
Akins, Brian wrote:
>Sorry if I missed it, which mpm are you using?
>
>
>
prefork
Technology Forums
Posted by anshul <an...@vishwatech.com>.
Hi all,
We've our new Technology Forums started from India.
http://forums.mediasworks.com/
I invite you all to join our community.
http://forums.mediasworks.com/
Thanks & regards.
This mail is not a SPAM or solicitation badly.
This is a one-time notification news.
You won't recieve this message again.
Delete this message, if you don't like it.
----- Original Message -----
From: "William A. Rowe, Jr." <wr...@rowe-clan.net>
To: <do...@httpd.apache.org>
Cc: <de...@httpd.apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: how do i debug a segfault?
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 12:13 PM 6/29/2005, ed elliott wrote:
>If you think it is usefull I could do a version of this (and this page <http://httpd.apache.org/dev/debugging.html>http://httpd.apache.org/dev/debugging.html ) using the free "ms debugging tools for windows" for win32 users?
YES!!! I'm happy to commit that one - it's the entire point of our
distributing .pdb files for end users! Thank you for the offer.
We should probably move debugging.html into docs/manual/developer/
see http://httpd.apache.org/docs-2.1/developer/
Let's take this to docs@httpd.
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: how do i debug a segfault?
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 12:13 PM 6/29/2005, ed elliott wrote:
>If you think it is usefull I could do a version of this (and this page <http://httpd.apache.org/dev/debugging.html>http://httpd.apache.org/dev/debugging.html ) using the free "ms debugging tools for windows" for win32 users?
YES!!! I'm happy to commit that one - it's the entire point of our
distributing .pdb files for end users! Thank you for the offer.
We should probably move debugging.html into docs/manual/developer/
see http://httpd.apache.org/docs-2.1/developer/
Let's take this to docs@httpd.
Bill
Re: how do i debug a segfault?
Posted by ed elliott <ed...@yahoo.co.uk>.
cool I'll get onto it and send it on probably next week some time.
Paul Querna <ch...@force-elite.com> wrote:ed elliott wrote:
> If you think it is usefull I could do a version of this (and this page
> http://httpd.apache.org/dev/debugging.html ) using the free "ms
> debugging tools for windows" for win32 users?
>
> ed
Yes, that would be awesome.
-Paul
>
> */Joshua Slive /* wrote:
>
>
> Akins, Brian wrote:
> > On 6/29/05 11:40 AM, "Paul A Houle" wrote:
> >
> >
> >
> >>[Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
> >>Segmentation fault (11)
> >
> >
> > Sorry if I missed it, which mpm are you using?
> >
> > Basically before you start apache do:
> > ulimit -c unlimited
>
> [...]
>
> Would you mind adding something like that to
> http://httpd.apache.org/dev/debugging.html
>
> This question comes up all the time on the users list, and the
> current
> debugging page doesn't directly address the most standard situations.
>
> Joshua.
>
> ------------------------------------------------------------------------
> Yahoo! Messenger
>
> NEW - crystal clear PC to PC calling worldwide with voicemail
>
---------------------------------
How much free photo storage do you get? Store your holiday snaps for FREE with Yahoo! Photos. Get Yahoo! Photos
Re: how do i debug a segfault?
Posted by Paul Querna <ch...@force-elite.com>.
ed elliott wrote:
> If you think it is usefull I could do a version of this (and this page
> http://httpd.apache.org/dev/debugging.html ) using the free "ms
> debugging tools for windows" for win32 users?
>
> ed
Yes, that would be awesome.
-Paul
>
> */Joshua Slive <jo...@slive.ca>/* wrote:
>
>
> Akins, Brian wrote:
> > On 6/29/05 11:40 AM, "Paul A Houle" wrote:
> >
> >
> >
> >>[Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
> >>Segmentation fault (11)
> >
> >
> > Sorry if I missed it, which mpm are you using?
> >
> > Basically before you start apache do:
> > ulimit -c unlimited
>
> [...]
>
> Would you mind adding something like that to
> http://httpd.apache.org/dev/debugging.html
>
> This question comes up all the time on the users list, and the
> current
> debugging page doesn't directly address the most standard situations.
>
> Joshua.
>
> ------------------------------------------------------------------------
> Yahoo! Messenger
> <http://us.rd.yahoo.com/mail/uk/taglines/default/messenger/*http://uk.messenger.yahoo.com>
> NEW - crystal clear PC to PC calling worldwide with voicemail
> <http://us.rd.yahoo.com/mail/uk/taglines/default/messenger/*http://uk.beta.messenger.yahoo.com>
Re: how do i debug a segfault?
Posted by ed elliott <ed...@yahoo.co.uk>.
If you think it is usefull I could do a version of this (and this page http://httpd.apache.org/dev/debugging.html ) using the free "ms debugging tools for windows" for win32 users?
ed
Joshua Slive <jo...@slive.ca> wrote:
Akins, Brian wrote:
> On 6/29/05 11:40 AM, "Paul A Houle"
wrote:
>
>
>
>>[Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
>>Segmentation fault (11)
>
>
> Sorry if I missed it, which mpm are you using?
>
> Basically before you start apache do:
> ulimit -c unlimited
[...]
Would you mind adding something like that to
http://httpd.apache.org/dev/debugging.html
This question comes up all the time on the users list, and the current
debugging page doesn't directly address the most standard situations.
Joshua.
---------------------------------
Yahoo! Messenger NEW - crystal clear PC to PCcalling worldwide with voicemail
Re: how do i debug a segfault?
Posted by Joshua Slive <jo...@slive.ca>.
Akins, Brian wrote:
> On 6/29/05 11:40 AM, "Paul A Houle" <ph...@cornell.edu> wrote:
>
>
>
>>[Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
>>Segmentation fault (11)
>
>
> Sorry if I missed it, which mpm are you using?
>
> Basically before you start apache do:
> ulimit -c unlimited
[...]
Would you mind adding something like that to
http://httpd.apache.org/dev/debugging.html
This question comes up all the time on the users list, and the current
debugging page doesn't directly address the most standard situations.
Joshua.
Re: how do i debug a segfault?
Posted by "Akins, Brian" <Br...@turner.com>.
On 6/29/05 11:40 AM, "Paul A Houle" <ph...@cornell.edu> wrote:
> [Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
> Segmentation fault (11)
Sorry if I missed it, which mpm are you using?
Basically before you start apache do:
ulimit -c unlimited
Set CoreDumpDirectory to so directory writable by the web server user (not
root).
Start apache.
You should get a core file. Use GDB to look at it: assuming the core is in
tmp.
gdb --core=/tmp/core.<pid> /path/to/httpd
If you are using a threaded mpm, then use the following gdb command to track
down the offending thread:
thread apply all bt
Then switch to that thread:
thread <id>
--
Brian Akins
Lead Systems Engineer
CNN Internet Technologies