You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Brian Hirt <bh...@mobygames.com> on 2004/05/06 04:11:56 UTC
Callback called exit.
I've been running across a problem lately where a child process
terminates because of an out of memory error. It prints Out of Memory
once, the the process sucks up all available cpu print "Callback called
exit." to the log file until it hit's it's 2GB max size.
I have some Apache::Resource limits set, and they probably need to be
raised, but the way the error is handled is not very graceful. I'd
expect the child to just terminate after reporting the first error
message. I'm not sure if this is a perl problem or a mod_perl
problem. I'd still like to figure out how to prevent the repeating
message from happening.
Anyway, I've been pulling my hair out trying to prevent this, and I've
finally figured out how to trap this. I have some suggestions for the
documentation, because the following url could use some help:
http://perl.apache.org/docs/1.0/guide/
troubleshooting.html#Callback_called_exit
=> "Note that Perl 5.005 and later have PERL_EMERGENCY_SBRK turned on
by default."
This is only true if perl was built to use it's own malloc. However,
"usemymalloc=y" is not the default for many systems because they assume
the OS version is probably a better implementation (which could be
true). However, when perl's internal malloc is used, none of the
suggestions for solving the out of memory problem or repeated Callback
called exit messages work.
=> "See Perl's INSTALL document for this item:"
This might have been true at one point. Newer versions of perl 5.6
and 5.8 have no reference to this option in the INSTALL document
=> "=item -DPERL_EMERGENCY_SBRK ..... "
a better quotation would be from perlvar.pod which states the crux of
the matter: " ..... Suppose that your Perl were compiled with
-DPERL_EMERGENCY_SBRK and used Perl's malloc ..."
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Stas Bekman <st...@stason.org>.
Glenn wrote:
[...]
>>http://perl.apache.org/docs/1.0/guide/troubleshooting.html#Callback_called_exit
>
>
> I've followed that advice and explicitly allocated memory into $^M.
> I have the following in my mod_perl_startup.pl, which I run from
> httpd.conf with PerlRequire /path/to/mod_perl_startup.pl
> If 64K is not enough for you, try increasing the allocation.
>
> Cheers,
> Glenn
>
> use strict;
>
> ## ----------
> ## ----------
>
> ## This section is similar in scope to Apache::Debug.
> ## Delivers a stack backtrace to the error log when perl code dies.
>
> ## Allocate 64K as an emergency memory pool for use in out of memory situation
> $^M = 0x00 x 65536;
>
> ## Little trick to initialize this routine here so that in the case of OOM,
> ## compiling this routine doesn't eat memory from the emergency memory pool $^M
> use CGI::Carp ();
> eval { CGI::Carp::confess('init') };
>
> ## Importing CGI::Carp sets $main::SIG{__DIE__} = \&CGI::Carp::die;
> ## Override that to additionally give a stack backtrace
> $main::SIG{__DIE__} = \&CGI::Carp::confess;
Brian, you may want to include Glenn's useful tips as well in the patch.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Stas Bekman <st...@stason.org>.
Brian Hirt wrote:
> I've typed up my suggestions to the troubleshooting doc, and
> incorporated glen's suggestions too. Stas wants me to post to the list
> to see if there are any comments / corrections.
>
> I wasn't sure if I should put a comment in about __DIE__ handlers and
> their use with evals, it seemed like that might be too general perl.
While true for the rest of the documentation, there is no limit to what you
can put it into the troubleshooting section, as long as it helps to
troubleshoot the issue :)
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Stas Bekman <st...@stason.org>.
Brian Hirt wrote:
> I've typed up my suggestions to the troubleshooting doc, and
> incorporated glen's suggestions too. Stas wants me to post to the list
> to see if there are any comments / corrections.
>
> I wasn't sure if I should put a comment in about __DIE__ handlers and
> their use with evals, it seemed like that might be too general perl.
Thanks Brian, very late but now committed.
> Index: src/docs/1.0/guide/troubleshooting.pod
> ===================================================================
> RCS file:
> /home/cvspublic/modperl-docs/src/docs/1.0/guide/troubleshooting.pod,v
> retrieving revision 1.28
> diff -u -r1.28 troubleshooting.pod
> --- src/docs/1.0/guide/troubleshooting.pod 5 May 2004 03:29:38
> -0000 1.28
> +++ src/docs/1.0/guide/troubleshooting.pod 6 May 2004 22:40:07 -0000
> @@ -589,27 +589,45 @@
>
> If something goes really wrong with your code, Perl may die with an
> "Out of memory!" message and/or "Callback called exit". Common causes
> of this
> -are never-ending loops, deep recursion, or calling an
> -undefined subroutine. Here's one way to catch the problem: See Perl's
> -INSTALL document for this item:
> +are never-ending loops, deep recursion, or calling an undefined
> subroutine.
>
> - =item -DPERL_EMERGENCY_SBRK
> +If you are using perl 5.005 or later, and perl is compiled to use it's
> own
> +malloc rutines, you can trap out of memory errors by setting aside an
> extra
> +memory pool in the special variable $^M. By default perl uses the
> operating
> +system malloc for many popular systems, so unless you build perl with
> +'usemymalloc=y' you probably wont be able to use $^M. To check if your
> mod_perl
> +was compiled to use perl's internal malloc(), stick the following code
> in a
> +handler and see if usemymalloc is set to 'y'
> +
> + use Config;
> +
> + print Config::myconfig();
> +
> +Here is an explanation of $^M from perlvar(i):
> +
> + $^M By default, running out of memory is an untrap-
> + pable, fatal error. However, if suitably built,
> + Perl can use the contents of $^M as an emergency
> + memory pool after die()ing. Suppose that your
> + Perl were compiled with -DPERL_EMERGENCY_SBRK and
> + used Perl's malloc. Then
> +
> + $^M = 'a' x (1 << 16);
> +
> + would allocate a 64K buffer for use in an emer-
> + gency. See the INSTALL file in the Perl distribu-
> + tion for information on how to enable this option.
> + To discourage casual use of this advanced feature,
> + there is no English long name for this variable.
>
> - If PERL_EMERGENCY_SBRK is defined, running out of memory need not be a
> - fatal error: a memory pool can allocated by assigning to the special
> - variable $^M. See perlvar(1) for more details.
> -
> -If you compile with that option and add 'C<use Apache::Debug level
> +If your perl installation supports $^M and you add 'C<use Apache::Debug
> level
> =E<gt> 4;>' to your Perl script, it will allocate the C<$^M> emergency
> pool and the C<$SIG{__DIE__}> handler will call C<Carp::confess>,
> giving you a stack trace which should reveal where the problem is.
> See the C<Apache::Resource> module for ways to control httpd
> processes.
>
> -Note that Perl 5.005 and later have C<PERL_EMERGENCY_SBRK> turned on
> -by default.
> -
> -The other trick is to have a startup script initialize
> +Another trick is to have a startup script initialize
> C<Carp::confess>, like so:
>
> use Carp ();
> @@ -617,6 +635,24 @@
>
> this way, when the real problem happens, C<Carp::confess> doesn't eat
> memory in the emergency pool (C<$^M>).
> +
> +Some other mod_perl users have reported that this works well for them:
> +
> + ## Allocate 64K as an emergency memory pool for use in out of
> memory situation
> + $^M = 0x00 x 65536;
> +
> + ## Little trick to initialize this routine here so that in the case
> of OOM,
> + ## compiling this routine doesn't eat memory from the emergency
> memory pool $^M
> + use CGI::Carp ();
> + eval { CGI::Carp::confess('init') };
> +
> + ## Importing CGI::Carp sets $main::SIG{__DIE__} = \&CGI::Carp::die;
> + ## Override that to additionally give a stack backtrace
> + $main::SIG{__DIE__} = \&CGI::Carp::confess;
> +
> +Discussion of $^M has come up on PerlMonks, and there is speculation
> that $^M is a
> +forgotten feature that's not well supported. See
> +http://perlmonks.org/index.pl?node_id=287850 for more information.
>
> =head2 server reached MaxClients setting, consider raising the
> MaxClients setting
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Brian Hirt <bh...@mobygames.com>.
I've typed up my suggestions to the troubleshooting doc, and
incorporated glen's suggestions too. Stas wants me to post to the
list to see if there are any comments / corrections.
I wasn't sure if I should put a comment in about __DIE__ handlers and
their use with evals, it seemed like that might be too general perl.
Index: src/docs/1.0/guide/troubleshooting.pod
===================================================================
RCS file:
/home/cvspublic/modperl-docs/src/docs/1.0/guide/troubleshooting.pod,v
retrieving revision 1.28
diff -u -r1.28 troubleshooting.pod
--- src/docs/1.0/guide/troubleshooting.pod 5 May 2004 03:29:38
-0000 1.28
+++ src/docs/1.0/guide/troubleshooting.pod 6 May 2004 22:40:07
-0000
@@ -589,27 +589,45 @@
If something goes really wrong with your code, Perl may die with an
"Out of memory!" message and/or "Callback called exit". Common causes
of this
-are never-ending loops, deep recursion, or calling an
-undefined subroutine. Here's one way to catch the problem: See Perl's
-INSTALL document for this item:
+are never-ending loops, deep recursion, or calling an undefined
subroutine.
- =item -DPERL_EMERGENCY_SBRK
+If you are using perl 5.005 or later, and perl is compiled to use
it's own
+malloc rutines, you can trap out of memory errors by setting aside an
extra
+memory pool in the special variable $^M. By default perl uses the
operating
+system malloc for many popular systems, so unless you build perl with
+'usemymalloc=y' you probably wont be able to use $^M. To check if
your mod_perl
+was compiled to use perl's internal malloc(), stick the following code
in a
+handler and see if usemymalloc is set to 'y'
+
+ use Config;
+
+ print Config::myconfig();
+
+Here is an explanation of $^M from perlvar(i):
+
+ $^M By default, running out of memory is an untrap-
+ pable, fatal error. However, if suitably built,
+ Perl can use the contents of $^M as an emergency
+ memory pool after die()ing. Suppose that your
+ Perl were compiled with -DPERL_EMERGENCY_SBRK and
+ used Perl's malloc. Then
+
+ $^M = 'a' x (1 << 16);
+
+ would allocate a 64K buffer for use in an emer-
+ gency. See the INSTALL file in the Perl distribu-
+ tion for information on how to enable this option.
+ To discourage casual use of this advanced feature,
+ there is no English long name for this variable.
- If PERL_EMERGENCY_SBRK is defined, running out of memory need not be
a
- fatal error: a memory pool can allocated by assigning to the special
- variable $^M. See perlvar(1) for more details.
-
-If you compile with that option and add 'C<use Apache::Debug level
+If your perl installation supports $^M and you add 'C<use
Apache::Debug level
=E<gt> 4;>' to your Perl script, it will allocate the C<$^M> emergency
pool and the C<$SIG{__DIE__}> handler will call C<Carp::confess>,
giving you a stack trace which should reveal where the problem is.
See the C<Apache::Resource> module for ways to control httpd
processes.
-Note that Perl 5.005 and later have C<PERL_EMERGENCY_SBRK> turned on
-by default.
-
-The other trick is to have a startup script initialize
+Another trick is to have a startup script initialize
C<Carp::confess>, like so:
use Carp ();
@@ -617,6 +635,24 @@
this way, when the real problem happens, C<Carp::confess> doesn't eat
memory in the emergency pool (C<$^M>).
+
+Some other mod_perl users have reported that this works well for them:
+
+ ## Allocate 64K as an emergency memory pool for use in out of
memory situation
+ $^M = 0x00 x 65536;
+
+ ## Little trick to initialize this routine here so that in the
case of OOM,
+ ## compiling this routine doesn't eat memory from the emergency
memory pool $^M
+ use CGI::Carp ();
+ eval { CGI::Carp::confess('init') };
+
+ ## Importing CGI::Carp sets $main::SIG{__DIE__} = \&CGI::Carp::die;
+ ## Override that to additionally give a stack backtrace
+ $main::SIG{__DIE__} = \&CGI::Carp::confess;
+
+Discussion of $^M has come up on PerlMonks, and there is speculation
that $^M is a
+forgotten feature that's not well supported. See
+http://perlmonks.org/index.pl?node_id=287850 for more information.
=head2 server reached MaxClients setting, consider raising the
MaxClients setting
regards,
Brian
On May 6, 2004, at 1:19 PM, Stas Bekman wrote:
> Brian Hirt wrote:
>> I too followed the advice too, but it did nothing but lead my down
>> the wrong path. The advice should be updated.
>> My point is that $^M does absolutely nothing unless you use perl's
>> malloc, which isn't true for most common perl installations these
>> days. compiling with PERL_EMERGENCY_SBRK doesn't help either
>> because it's the default if you usemymalloc, and useless if you
>> don't You MUST compile perl with "-Dusemymalloc=y". A simple
>> grep in the perl hints directory shows that many popular systems
>> such as linux, freebsd and openbsd default to the system malloc
>> which disables the functionality of $^M.
>> I'd simply like to see the documentation updated, and I'm happy to do
>> it. I know it would have saved me hours and hours of headaches.
>> The documentation as it stands now is misleading.
>> of course if you use perl's malloc, the advice helps.
>
> Doc patches are always welcome here. Please patch against the source
> pod.
> http://perl.apache.org/download/docs.html#Download
>
>> I'd still like to know why mod_perl can get into an infinite loop
>> writtitng "Callback called exit". In perl.c, when that happens
>> my_exit_jump(); is called which should presumably exit the process,
>> but somehow that doesn't happen and some sort of infinite loop
>> occurs outside of my code that fills the log of with gigibytes of
>> 'Callback called exit' messages.
>
> Normally that happens when perl gets its calls stack messed up and it
> starts to loop. I know I hit that myself while developing mp2 when I
> was trying to write my own version of die/exit/etc, which I quickly
> gave up. It is possible that there is a bug in perl, which gets
> triggered only in certain situations. If you can give p5p a
> reproducable case, I'm sure it'll be fixed.
>
> __________________________________________________________________
> Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
> http://stason.org/ mod_perl Guide ---> http://perl.apache.org
> mailto:stas@stason.org http://use.perl.org http://apacheweek.com
> http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Stas Bekman <st...@stason.org>.
Brian Hirt wrote:
> I too followed the advice too, but it did nothing but lead my down the
> wrong path. The advice should be updated.
>
> My point is that $^M does absolutely nothing unless you use perl's
> malloc, which isn't true for most common perl installations these days.
> compiling with PERL_EMERGENCY_SBRK doesn't help either because it's
> the default if you usemymalloc, and useless if you don't You MUST
> compile perl with "-Dusemymalloc=y". A simple grep in the perl hints
> directory shows that many popular systems such as linux, freebsd and
> openbsd default to the system malloc which disables the functionality
> of $^M.
>
> I'd simply like to see the documentation updated, and I'm happy to do
> it. I know it would have saved me hours and hours of headaches. The
> documentation as it stands now is misleading.
>
> of course if you use perl's malloc, the advice helps.
Doc patches are always welcome here. Please patch against the source pod.
http://perl.apache.org/download/docs.html#Download
> I'd still like to know why mod_perl can get into an infinite loop
> writtitng "Callback called exit". In perl.c, when that happens
> my_exit_jump(); is called which should presumably exit the process, but
> somehow that doesn't happen and some sort of infinite loop occurs
> outside of my code that fills the log of with gigibytes of 'Callback
> called exit' messages.
Normally that happens when perl gets its calls stack messed up and it starts
to loop. I know I hit that myself while developing mp2 when I was trying to
write my own version of die/exit/etc, which I quickly gave up. It is possible
that there is a bug in perl, which gets triggered only in certain situations.
If you can give p5p a reproducable case, I'm sure it'll be fixed.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Brian Hirt <bh...@mobygames.com>.
I too followed the advice too, but it did nothing but lead my down the
wrong path. The advice should be updated.
My point is that $^M does absolutely nothing unless you use perl's
malloc, which isn't true for most common perl installations these days.
compiling with PERL_EMERGENCY_SBRK doesn't help either because it's
the default if you usemymalloc, and useless if you don't You MUST
compile perl with "-Dusemymalloc=y". A simple grep in the perl hints
directory shows that many popular systems such as linux, freebsd and
openbsd default to the system malloc which disables the functionality
of $^M.
I'd simply like to see the documentation updated, and I'm happy to do
it. I know it would have saved me hours and hours of headaches. The
documentation as it stands now is misleading.
of course if you use perl's malloc, the advice helps.
I'd still like to know why mod_perl can get into an infinite loop
writtitng "Callback called exit". In perl.c, when that happens
my_exit_jump(); is called which should presumably exit the process, but
somehow that doesn't happen and some sort of infinite loop occurs
outside of my code that fills the log of with gigibytes of 'Callback
called exit' messages.
regards,
brian
On May 5, 2004, at 11:00 PM, Glenn wrote:
> On Wed, May 05, 2004 at 08:11:56PM -0600, Brian Hirt wrote:
>> I've been running across a problem lately where a child process
>> terminates because of an out of memory error. It prints Out of
>> Memory
>> once, the the process sucks up all available cpu print "Callback
>> called
>> exit." to the log file until it hit's it's 2GB max size.
>>
>> I have some Apache::Resource limits set, and they probably need to be
>> raised, but the way the error is handled is not very graceful. I'd
>> expect the child to just terminate after reporting the first error
>> message. I'm not sure if this is a perl problem or a mod_perl
>> problem. I'd still like to figure out how to prevent the repeating
>> message from happening.
>>
>> Anyway, I've been pulling my hair out trying to prevent this, and I've
>> finally figured out how to trap this. I have some suggestions for
>> the
>> documentation, because the following url could use some help:
>>
>> http://perl.apache.org/docs/1.0/guide/
>> troubleshooting.html#Callback_called_exit
>
> I've followed that advice and explicitly allocated memory into $^M.
> I have the following in my mod_perl_startup.pl, which I run from
> httpd.conf with PerlRequire /path/to/mod_perl_startup.pl
> If 64K is not enough for you, try increasing the allocation.
>
> Cheers,
> Glenn
>
> use strict;
>
> ## ----------
> ## ----------
>
> ## This section is similar in scope to Apache::Debug.
> ## Delivers a stack backtrace to the error log when perl code dies.
>
> ## Allocate 64K as an emergency memory pool for use in out of memory
> situation
> $^M = 0x00 x 65536;
>
> ## Little trick to initialize this routine here so that in the case of
> OOM,
> ## compiling this routine doesn't eat memory from the emergency memory
> pool $^M
> use CGI::Carp ();
> eval { CGI::Carp::confess('init') };
>
> ## Importing CGI::Carp sets $main::SIG{__DIE__} = \&CGI::Carp::die;
> ## Override that to additionally give a stack backtrace
> $main::SIG{__DIE__} = \&CGI::Carp::confess;
>
> --
> Report problems: http://perl.apache.org/bugs/
> Mail list info: http://perl.apache.org/maillist/modperl.html
> List etiquette: http://perl.apache.org/maillist/email-etiquette.html
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Glenn <gs...@gluelogic.com>.
On Wed, May 05, 2004 at 08:11:56PM -0600, Brian Hirt wrote:
> I've been running across a problem lately where a child process
> terminates because of an out of memory error. It prints Out of Memory
> once, the the process sucks up all available cpu print "Callback called
> exit." to the log file until it hit's it's 2GB max size.
>
> I have some Apache::Resource limits set, and they probably need to be
> raised, but the way the error is handled is not very graceful. I'd
> expect the child to just terminate after reporting the first error
> message. I'm not sure if this is a perl problem or a mod_perl
> problem. I'd still like to figure out how to prevent the repeating
> message from happening.
>
> Anyway, I've been pulling my hair out trying to prevent this, and I've
> finally figured out how to trap this. I have some suggestions for the
> documentation, because the following url could use some help:
>
> http://perl.apache.org/docs/1.0/guide/troubleshooting.html#Callback_called_exit
I've followed that advice and explicitly allocated memory into $^M.
I have the following in my mod_perl_startup.pl, which I run from
httpd.conf with PerlRequire /path/to/mod_perl_startup.pl
If 64K is not enough for you, try increasing the allocation.
Cheers,
Glenn
use strict;
## ----------
## ----------
## This section is similar in scope to Apache::Debug.
## Delivers a stack backtrace to the error log when perl code dies.
## Allocate 64K as an emergency memory pool for use in out of memory situation
$^M = 0x00 x 65536;
## Little trick to initialize this routine here so that in the case of OOM,
## compiling this routine doesn't eat memory from the emergency memory pool $^M
use CGI::Carp ();
eval { CGI::Carp::confess('init') };
## Importing CGI::Carp sets $main::SIG{__DIE__} = \&CGI::Carp::die;
## Override that to additionally give a stack backtrace
$main::SIG{__DIE__} = \&CGI::Carp::confess;
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Glenn <gs...@gluelogic.com>.
On Thu, May 06, 2004 at 10:55:14AM -0600, Brian Hirt wrote:
>
> On May 6, 2004, at 10:27 AM, Perrin Harkins wrote:
>
> >On Wed, 2004-05-05 at 22:11, Brian Hirt wrote:
> >>I've been running across a problem lately where a child process
> >>terminates because of an out of memory error. It prints Out of
> >>Memory
> >>once, the the process sucks up all available cpu print "Callback
> >>called
> >>exit." to the log file until it hit's it's 2GB max size.
> >
> >I'm just guessing here, but this is probably because apache is trying
> >to
> >spawn new processes, and they keep dying because there's no memory.
> >
>
> Thanks for the response, interesting insight into the history of $^M.
>
> When I've seen this happen, it's the same PID spewing the messages,
> there are no forkings going on. The system isn't actually out of
> memory, and there is plenty of it available for the parent httpd
> process to fork. The child process has an rlimit set which is why
> it's getting an out of memory error. I initially set the rlimit,
> because at one point in the past the ImageMagick module would every now
> and then go crazy and consume all available memory which would bring
> down everything.
Yes, thanks as well. I didn't know how ineffective that was, and am
glad I wasn't setting aside too much memory for it.
Brian, if you can trigger the OOM and "Callback called exit" loop,
would you try my example mod_perl_startup.pl and use this at the end:
## Importing CGI::Carp sets $main::SIG{__DIE__} = \&CGI::Carp::die;
## Override that to additionally give a stack backtrace
$main::SIG{__DIE__} = sub { undef $M; &CGI::Carp::confess; };
The 'undef $M' will mark the memory as unused (as long as nothing
else has a reference to it), and if Perl garbage collection kicks
in before the looping problem, then you might have some memory to
work with. I don't know the threshold offhand that Perl uses to
trigger freeing the memory back to the system when using the
system malloc, but a couple of MBs would most likely do it.
Of course, this assumes that the loop is occurring somewhere after
die() is called, and after this routine is called. Worth a shot...
Cheers,
Glenn
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Brian Hirt <bh...@mobygames.com>.
On May 6, 2004, at 10:27 AM, Perrin Harkins wrote:
> On Wed, 2004-05-05 at 22:11, Brian Hirt wrote:
>> I've been running across a problem lately where a child process
>> terminates because of an out of memory error. It prints Out of
>> Memory
>> once, the the process sucks up all available cpu print "Callback
>> called
>> exit." to the log file until it hit's it's 2GB max size.
>
> I'm just guessing here, but this is probably because apache is trying
> to
> spawn new processes, and they keep dying because there's no memory.
>
Thanks for the response, interesting insight into the history of $^M.
When I've seen this happen, it's the same PID spewing the messages,
there are no forkings going on. The system isn't actually out of
memory, and there is plenty of it available for the parent httpd
process to fork. The child process has an rlimit set which is why
it's getting an out of memory error. I initially set the rlimit,
because at one point in the past the ImageMagick module would every now
and then go crazy and consume all available memory which would bring
down everything.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: Callback called exit.
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2004-05-05 at 22:11, Brian Hirt wrote:
> I've been running across a problem lately where a child process
> terminates because of an out of memory error. It prints Out of Memory
> once, the the process sucks up all available cpu print "Callback called
> exit." to the log file until it hit's it's 2GB max size.
I'm just guessing here, but this is probably because apache is trying to
spawn new processes, and they keep dying because there's no memory.
> => "See Perl's INSTALL document for this item:"
>
> This might have been true at one point. Newer versions of perl 5.6
> and 5.8 have no reference to this option in the INSTALL document
There was discussion about this on PerlMonks:
http://perlmonks.org/index.pl?node_id=287850
The most informative quote was this one:
"$^M was an Ilya-thing, which he added in during his work on perl's
built-in memory allocation system. Nobody else ever really cared about
it so far as I remember from p5p, so it suffers some from both neglect
and unfinshedness. I remember in the 5.004/5.005 days that it used to
work, but I honestly don't remember how, nor whether it ever worked for
anyone other than Ilya."
- Perrin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html