You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by justin <ju...@dslr.net> on 2004/03/10 00:11:01 UTC
how best to debug random SEGVs?
Our modperl servers with apache+modperl >= versions 1.3.28 and modperl
1.28, crash randomly with
[Tue Mar 9 18:02:17 2004] [notice] child pid 10940 exit signal Segmentation fault (11)
I'd say on the order of 1 in 100 requests or less. But after an hour,
the above lines collect in the error log, its the only exciting thing
happening to the servers.
The crash seems to happen somewhere after serving the page because the
access.log for the crashing process shows a last good request (no 500
error or anything). The last request processed before death is random,
there is no URL that triggers this segv. The age of the child before it
segvs also seems to be random.
I've tried rebuilding/reinstalling everything on both releases, and the
error persists. Everything else works ok otherwise. Earlier versions of
apache+modperl (running on earlier redhats) do not crash like this.
The box(es) are recent hardware, redhat9, otherwise well behaved.
For some reason I can no longer strace -p httpd children (get
attach/detach and the child dies), although I can strace -p other
demons. So I can't even follow a child in case it dies :( -- any reason
for this, at least?
How can I best debug this? perhaps capture the segv and do a stack trace
or write a core file? it only happens in production, or if it happens in
dev, then I need a production-like request rate anyway.
thanks
-justin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: how best to debug random SEGVs?
Posted by Stas Bekman <st...@stason.org>.
Ged Haywood wrote:
>>perl is now perl 5.8.0 with MAINT18379 patch
The real issue is often the fact that vendors ship perl releases that were
never released, i.e. taking some release and adding a bunch of patches to it.
so when you think you have 5.8.0 on RedHat most of time you don't, since
usually it's 5.8.0 + some patches. p5p never intended to deliver those
half-baked products to you.
Luckily with the current change in the perl release policy (release a new
version every 3 months or so), distros have little excuse not to give their
users real releases without adding any patches from the post-release. 5.8.4
will be released somewhere in April.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re[2]: how best to debug random SEGVs?
Posted by Ged Haywood <ge...@jubileegroup.co.uk>.
Hi there,
On Tue, 9 Mar 2004, justin wrote:
> For what it is worth and in case anyone googles for this issue in
> future, the segvs went away when I applied the latest kernel patches and
> glibc stuff for redhat9, and the latest perl install, from
> updates.redhat.com
>
> the kernel is now 2.4.20-30.9smp
>
> perl is now perl 5.8.0 with MAINT18379 patch
Please when reporting problems refer to the documentation on the
mod_perl Website which tells you what information to report. Stas
gave you a link. There is also advice about upgrading software.
In this case I think you could have saved yourself time simply by
telling us that you were using Perl 5.8.0 which is known to have a few
problems. I'd recommend upgrading to 5.8.3 when you can, although
obviously you'll want to be cautious in production systems. My
preference is to compile from source. If you leave both a new and an
old version of Perl on your system, make sure you know which one is
being used by your scripts. I'm not above building a completely new
system to make sure there are no old modules, libraries, binaries and
what-not left lying around to confuse things.
73,
Ged.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re[2]: how best to debug random SEGVs?
Posted by justin <ju...@dslr.net>.
For what it is worth and in case anyone googles for this issue in
future, the segvs went away when I applied the latest kernel patches and
glibc stuff for redhat9, and the latest perl install, from
updates.redhat.com
the kernel is now 2.4.20-30.9smp
perl is now perl 5.8.0 with MAINT18379 patch
I did not recompile modperl or apache after the OS upgrade.
strace on child processes started working again as well.
I think base redhat9 (shrike) with no updates is not quite trustworthy
for apache.
thanks for the explanation, though, it'll help in future.
-justin
SB> justin wrote:
>> Our modperl servers with apache+modperl >= versions 1.3.28 and modperl
>> 1.28, crash randomly with
>>
>> [Tue Mar 9 18:02:17 2004] [notice] child pid 10940 exit signal Segmentation fault (11)
SB> [...]
>> How can I best debug this? perhaps capture the segv and do a stack trace
>> or write a core file? it only happens in production, or if it happens in
>> dev, then I need a production-like request rate anyway.
SB> Configure the system to allow core files dumping as explained here:
SB> http://perl.apache.org/docs/2.0/devel/debug/c.html#Getting_the_core_File_Dumped
SB> once you have the core file, you can get the backtrace to find where it fails:
SB> http://perl.apache.org/docs/1.0/guide/help.html#How_to_Report_Problems
SB> or may be you will have luck with this module:
SB> http://search.cpan.org/dist/Debug-FaultAutoBT/
SB> __________________________________________________________________
SB> Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
SB> http://stason.org/ mod_perl Guide ---> http://perl.apache.org
SB> mailto:stas@stason.org http://use.perl.org http://apacheweek.com
SB> http://modperlbook.org http://apache.org http://ticketmaster.com
--
Best regards,
justin mailto:justin@dslr.net
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: how best to debug random SEGVs?
Posted by Stas Bekman <st...@stason.org>.
justin wrote:
> Our modperl servers with apache+modperl >= versions 1.3.28 and modperl
> 1.28, crash randomly with
>
> [Tue Mar 9 18:02:17 2004] [notice] child pid 10940 exit signal Segmentation fault (11)
[...]
> How can I best debug this? perhaps capture the segv and do a stack trace
> or write a core file? it only happens in production, or if it happens in
> dev, then I need a production-like request rate anyway.
Configure the system to allow core files dumping as explained here:
http://perl.apache.org/docs/2.0/devel/debug/c.html#Getting_the_core_File_Dumped
once you have the core file, you can get the backtrace to find where it fails:
http://perl.apache.org/docs/1.0/guide/help.html#How_to_Report_Problems
or may be you will have luck with this module:
http://search.cpan.org/dist/Debug-FaultAutoBT/
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html