Efficiently monitoring httpd children and signalling those using too much memory/cpu

[Jonathan Swartz <sw...@pobox.com>]

We use Apache/mod_perl 2 and occasionally get a child httpd process that spins out of control, either consuming ever-increasing amounts of memory or max cpu. Usually due to an infinite loop or other bug in a specific part of the site - this sort of thing happens.

I would like to monitor for such httpd children every second or so, and when finding one, send it a USR2 signal so it can dump its current Perl stack to our error logs.

(Note that we are using rlimit/BSD::Resource, but when a process exceeds its memory rlimit, it dies immediately with no good way to log or take action (see Getting stack trace from Perl "Out of memory" error). So AFAICT rlimit will have to be complemented by an external monitoring program.)

I see that monit is highly regarded, but I can't figure out any way to monitor and send a signal to an individual httpd process. It seems geared towards killing or restarting services like apache as a whole. Am I missing something?

If I have to write a Perl script that does this, is there anything that will efficiently monitor the process table? Proc::ProcessTable takes .02 secs of cpu to generate a single table, so running that every second seems unnecessarily expensive. Opening up a "/usr/bin/top -b -d 1 |" and parsing the results is my current best idea.

Obviously would rather not badly reinvent a wheel if possible.

Thanks!
Jon

Re: Segment Fault when creating a thread

[Fred Moyer <fr...@redhotpenguin.com>]

I'm not that knowledgeable on threads, but this guy managed to get the
working with mpm_worker a month or so ago:

http://www.gossamer-threads.com/lists/modperl/modperl/103420

That is still a rather old version of Perl, so I would suggest trying
5.10 or something newer.

On Tue, Aug 23, 2011 at 2:58 PM, Mike OK <mi...@acorg.com> wrote:
> Thanks for the tip Fred.  I just now upgraded to 2.0.5 but still get the
> same error.
>
> Mike
>
>
> ----- Original Message ----- From: "Fred Moyer" <fr...@redhotpenguin.com>
> To: "Mike OK" <mi...@acorg.com>
> Cc: "mod_perl list" <mo...@perl.apache.org>
> Sent: Tuesday, August 23, 2011 5:26 PM
> Subject: Re: Segment Fault when creating a thread
>
>
> Might want to try 2.0.5 which is about 6 months old.
>
> On Tue, Aug 23, 2011 at 2:23 PM, Mike OK <mi...@acorg.com> wrote:
>>
>> I must have a problem with my mod_perl set up. When I try to run this
>> script, I get a "child pid 451 exit signal Segmentation fault (11)". It
>> seems that creating the thread causes the error. This is the startup line
>> from the log file with the apache, mod_perl, and perl versions.
>>
>> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
>> normal operations
>>
>> The script that causes a failure is
>>
>> #!/usr/bin/perl -wT
>>
>> use strict;
>> use threads;
>> use threads::shared;
>>
>> my $temp = threads->new(\&foo,"mike");
>> exit;
>>
>> sub foo {
>> my $input = shift;
>> my $number = int(rand(10));
>> print "this went in the thread\n";
>> return($number);
>> }
>
>

Re: Segment Fault when creating a thread

[Mike OK <mi...@acorg.com>]

Thanks for the tip Fred.  I just now upgraded to 2.0.5 but still get the 
same error.

Mike


----- Original Message ----- 
From: "Fred Moyer" <fr...@redhotpenguin.com>
To: "Mike OK" <mi...@acorg.com>
Cc: "mod_perl list" <mo...@perl.apache.org>
Sent: Tuesday, August 23, 2011 5:26 PM
Subject: Re: Segment Fault when creating a thread


Might want to try 2.0.5 which is about 6 months old.

On Tue, Aug 23, 2011 at 2:23 PM, Mike OK <mi...@acorg.com> wrote:
> I must have a problem with my mod_perl set up. When I try to run this
> script, I get a "child pid 451 exit signal Segmentation fault (11)". It
> seems that creating the thread causes the error. This is the startup line
> from the log file with the apache, mod_perl, and perl versions.
>
> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
> normal operations
>
> The script that causes a failure is
>
> #!/usr/bin/perl -wT
>
> use strict;
> use threads;
> use threads::shared;
>
> my $temp = threads->new(\&foo,"mike");
> exit;
>
> sub foo {
> my $input = shift;
> my $number = int(rand(10));
> print "this went in the thread\n";
> return($number);
> } 

Re: Segment Fault when creating a thread

[Fred Moyer <fr...@redhotpenguin.com>]

Might want to try 2.0.5 which is about 6 months old.

On Tue, Aug 23, 2011 at 2:23 PM, Mike OK <mi...@acorg.com> wrote:
> I must have a problem with my mod_perl set up.  When I try to run this
> script, I get a "child pid 451 exit signal Segmentation fault (11)".  It
> seems that creating the thread causes the error. This is the startup line
> from the log file with the apache, mod_perl, and perl versions.
>
> Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming
> normal operations
>
> The script that causes a failure is
>
> #!/usr/bin/perl -wT
>
> use strict;
> use threads;
> use threads::shared;
>
> my $temp = threads->new(\&foo,"mike");
> exit;
>
> sub foo {
>    my $input = shift;
>    my $number = int(rand(10));
>    print "this went in the thread\n";
>    return($number);
> }

Segment Fault when creating a thread

[Mike OK <mi...@acorg.com>]

I must have a problem with my mod_perl set up.  When I try to run this script, I get a "child pid 451 exit signal Segmentation fault (11)".  It seems that creating the thread causes the error. This is the startup line from the log file with the apache, mod_perl, and perl versions.

Apache/2.2.11 (Unix) mod_perl/2.0.4 Perl/v5.8.5 configured -- resuming normal operations

The script that causes a failure is

#!/usr/bin/perl -wT

use strict;
use threads;
use threads::shared;

my $temp = threads->new(\&foo,"mike");
exit;

sub foo {
   my $input = shift;
   my $number = int(rand(10));
   print "this went in the thread\n";
   return($number);
}

Re: Efficiently monitoring httpd children and signalling those using too much memory/cpu

[Scott Gifford <sg...@suspectclass.com>]

On Mon, Aug 22, 2011 at 8:10 PM, Jonathan Swartz <sw...@pobox.com> wrote:

> We use Apache/mod_perl 2 and occasionally get a child httpd process that
> spins out of control, either consuming ever-increasing amounts of memory or
> max cpu. Usually due to an infinite loop or other bug in a specific part of
> the site - this sort of thing happens.
>
> I would like to monitor for such httpd children every second or so, and
> when finding one, send it a USR2 signal so it can dump its current Perl
> stack to our error logs.
>
A few ideas:

   - If your requests are typically short and the memory allocation uses
   enough CPU time, you could set a soft limit for CPU time then catch
   $SIG{XCPU} (you would also need to limit how many requests your child
   processes handle).  It worked for me in a quick test.
   - If the memory usage is significant, as a quick check you could look at
   the total free memory available on the system, and only if it falls below a
   threshold do a more complex check with Proc::ProcessTable.
   - If the runaway process causes the load average to go up, you could look
   at the lod average, and only if it rises above a threshold do a more complex
   check with Proc::ProcessTable.
   - If your requests are typically short, you could create a small watchdog
   server; a request would register its PID with the watchdog server, then
   unregister when it finishes.  If the watchdog sees a request register that
   does not complete within some time limit, it could send SIGUSR2.  I have
   used a solution like this in the past, and it is effective, if a bit
   cumbersome.
   - Apache::Scoreboard<http://search.cpan.org/~mjh/Apache-Scoreboard-2.09.2/>
can
   get you the PIDs of just the Apache processes, and some basic state
   information.  You might be able to use this to make your process table scan
   more efficient.  Maybe you could write a URL handler to do your checking
   and signaling using the scoreboard from within Apache, then load the URL
   periodically to trigger the test.

Hope this is helpful,

-----Scott.