You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Victor Tsang <ht...@gmail.com> on 2004/10/14 10:23:00 UTC
debuging run away httpd process
Hello everyone,
This could be a bit off topic, please kindly forgive me if it is so,
I am seeking for tips on debugging perl script that is running under
mod_perl.
We use mod_perl heavily here, and lately I notice some apache
children for some reason chew up large amount of cpu (indeed all cpu).
Using apache's server-status i can tell it is one of the new script
we have deployed that caused the probloem, but trouble is, the script
is big, where should I start looking.
What I usually do is to use strace, lsof to see if I can get some
hints as of what's the process is doing, but this time, running strace
on the httpd process simply return nothing, and lsof doesn't offer
much tips either. checking /proc/<processID>/fd/ I found the process
has opened some socket binding to some inode, I used netstat -nape to
try to find what those inode were binded to, but I couldn't find
anything.
I think my senior has nailed the problem by reviewing the source,
but I wonder if there's something I could have done more/better in
analysing the problem. Is the method/tool chain I'm using appropiate?
is there anything inside mod_perl I can turn on/off/tweak so I can
get more knowledge of running mod_process?
Any advice would be much appreicated.
Tor.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: debuging run away httpd process
Posted by victor <vi...@outblaze.com>.
>
>I've no idea what stability impact the profiler would have. It does impact CPU
>usage and memory consumption, it also generates potentially very large log
>files if it's running for any significant amount of time. You also have to
>shut the server down to finalise the profile logs (otherwise they tend to be
>difficult it not impossible to use). I don't have any stats on exactly how
>much the profiler overhead is (does anyone?)
>
>So I'd be very leary of using the profiler in production, especially if you
>have a high traffic site. If you have a situation that's not at all
>reproduceable in your testing environment, and you have a live environment
>where you can swap a webserver in and out of the configuration easily (or it
>doesn't matter if the server is down for a few minutes), then you might
>attempt it with caution, but you'd have to watch it - I certainly wouldn't
>leave it running by itself.
>
>Depending on your situation, the idea may give your collegues or manager fits.
>I'm working in situations where the server being down for five minutes
>wouldn't bother anyone much. That's probably a relatively rare occurance in a
>commerical situation.
>
>
>
They won't like the idea of taking anything down ^^, but if that would
help sovling the problem, taking down a single machine from the farm for
several minutes should be ok. I will give it a shot so next time if
anything similar show up, I will have some thing to study.
Many thanks for the tips Malcolm.
Tor.
--
<!---------------------------------------------
Victor
Development Engineer
Outblaze Ltd
---------------------------------------------->
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: debuging run away httpd process
Posted by Malcolm J Harwood <mj...@liminalflux.net>.
On Thursday 14 October 2004 10:49 pm, victor wrote:
> Hi malcolm,
> >This might be stating the obvious, but have you run it under a profiler? (
> >Devel::Profiler::Apache is what I've been using). If it's a problem with
> > the actual perl script, that should give you a good idea what part of the
> > script is eating CPU.
> I have that setup in the stage system, but not in the production.
> These unusal problem we have have never been seen in our staging
> platform, they are always kick started by condition we have never
> expected. Or would you suggest I install that in production as well?
I've no idea what stability impact the profiler would have. It does impact CPU
usage and memory consumption, it also generates potentially very large log
files if it's running for any significant amount of time. You also have to
shut the server down to finalise the profile logs (otherwise they tend to be
difficult it not impossible to use). I don't have any stats on exactly how
much the profiler overhead is (does anyone?)
So I'd be very leary of using the profiler in production, especially if you
have a high traffic site. If you have a situation that's not at all
reproduceable in your testing environment, and you have a live environment
where you can swap a webserver in and out of the configuration easily (or it
doesn't matter if the server is down for a few minutes), then you might
attempt it with caution, but you'd have to watch it - I certainly wouldn't
leave it running by itself.
Depending on your situation, the idea may give your collegues or manager fits.
I'm working in situations where the server being down for five minutes
wouldn't bother anyone much. That's probably a relatively rare occurance in a
commerical situation.
--
Everything that irritates us about others can lead us to an understanding
of ourselves.
- Carl Jung
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: debuging run away httpd process
Posted by victor <vi...@outblaze.com>.
Hi malcolm,
Malcolm J Harwood wrote:
>On Thursday 14 October 2004 04:23 am, Victor Tsang wrote:
>
>
>
>> We use mod_perl heavily here, and lately I notice some apache
>>children for some reason chew up large amount of cpu (indeed all cpu).
>> Using apache's server-status i can tell it is one of the new script
>>we have deployed that caused the probloem, but trouble is, the script
>>is big, where should I start looking.
>>
>>
>
>This might be stating the obvious, but have you run it under a profiler? (
>Devel::Profiler::Apache is what I've been using). If it's a problem with the
>actual perl script, that should give you a good idea what part of the script
>is eating CPU.
>
>
I have that setup in the stage system, but not in the production.
These unusal problem we have have never been seen in our staging
platform, they are always kick started by condition we have never
expected. Or would you suggest I install that in production as well?
Tor.
--
<!---------------------------------------------
Victor
Development Engineer
Outblaze Ltd
---------------------------------------------->
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: debuging run away httpd process
Posted by Malcolm J Harwood <mj...@liminalflux.net>.
On Thursday 14 October 2004 04:23 am, Victor Tsang wrote:
> We use mod_perl heavily here, and lately I notice some apache
> children for some reason chew up large amount of cpu (indeed all cpu).
> Using apache's server-status i can tell it is one of the new script
> we have deployed that caused the probloem, but trouble is, the script
> is big, where should I start looking.
This might be stating the obvious, but have you run it under a profiler? (
Devel::Profiler::Apache is what I've been using). If it's a problem with the
actual perl script, that should give you a good idea what part of the script
is eating CPU.
--
There are only two ways to live your life. One is as though nothing is a
miracle. The other is as if everything is.
- Albert Einstein
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html