You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by "Keith G. Murphy" <ke...@mindspring.com> on 2002/10/16 17:49:10 UTC
Memory leak on reload when the 'Pg' driver is preloaded
Since memory leaks seem to be the topic du jour, I wondered if anyone
else had seen this one:
When using a modular mod_perl, I get a huge leak if I preload the 'Pg'
driver in my startup perl script thus:
#!/usr/bin/perl
use strict;
use Apache::Status ();
use Apache::DBI ();
DBI->install_driver('Pg');
1; #return true value
The leak happens when you send SIGUSR1 to the parent. Every time you do
it, you get more memory eaten (multiple megabytes), both in the parent
and (new) children.
It happens with the stock apache-ssl that comes from Debian stable.
Could well be a Debian-specific problem.
Using Debian's static-mod_perled apache-perl eliminates the problem.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by Vivek Khera <kh...@kcilink.com>.
>>>>> "KGM" == Keith G Murphy <ke...@mindspring.com> writes:
KGM> When using a modular mod_perl, I get a huge leak if I preload the 'Pg'
KGM> driver in my startup perl script thus:
Hmmm. Interesting theory. I shall have to investigate it. I also
see a multi-megabyte memory leak in my app when DBD::Pg is preloaded,
but I didn't track it down to just that module. Does it happen for
you for any perl module that loads another XS compiled (ie, .so) file
in addition to the .pm file?
I'm on FreeBSD.
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: khera@kciLink.com Rockville, MD +1-240-453-8497
AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Keith G. Murphy" <ke...@mindspring.com>.
Juha-Mikko Ahonen wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wednesday 16 October 2002 20:25, Keith G. Murphy wrote:
>
>>By "should", do you mean that the problems with the loadable module
>>overall are so well-known that no one in his right mind should ever
>>use it?
>
>
> Yes. The problems with DSO mod_perl are well known. It works perfectly
> when build staticly into Apache, though. Install apache-perl package
> instead of apache and libapache-mod-perl. Everything works with it as
> it would on normal Apache. After migrating to apache-perl, remove the
> LoadModule for mod_perl from your Apache configuration.
I already have, months ago.
>
>
>>Might it be related to the Perl-malloc problem cited here:
>>
>>http://apache.perl.org/docs/1.0/guide/install.html#When_DSO_can_be_Us
>>ed
>>
>>If either of the above or true, seems like Debian needs to quit
>>distributing libapache-mod-perl.
>
>
> The problem is not Debian-related, so it won't justify removing the
> package from Debian.
Huh? I would think that would be a really good reason to yank it. If
it were Debian-related, the maintainer could fix it and keep it.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by Juha-Mikko Ahonen <ju...@mail.suomi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Wednesday 16 October 2002 20:25, Keith G. Murphy wrote:
> By "should", do you mean that the problems with the loadable module
> overall are so well-known that no one in his right mind should ever
> use it?
Yes. The problems with DSO mod_perl are well known. It works perfectly
when build staticly into Apache, though. Install apache-perl package
instead of apache and libapache-mod-perl. Everything works with it as
it would on normal Apache. After migrating to apache-perl, remove the
LoadModule for mod_perl from your Apache configuration.
> Might it be related to the Perl-malloc problem cited here:
>
> http://apache.perl.org/docs/1.0/guide/install.html#When_DSO_can_be_Us
>ed
>
> If either of the above or true, seems like Debian needs to quit
> distributing libapache-mod-perl.
The problem is not Debian-related, so it won't justify removing the
package from Debian.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE9raVtnksV4Ys/z5gRAt+TAJwOZXcdnVNoLWLza7NU6eOpCOB+dQCdEliU
VDvG6M+qI88/RN/jj7KsCC0=
=0/Bl
-----END PGP SIGNATURE-----
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Keith G. Murphy" <ke...@mindspring.com>.
Juha-Mikko Ahonen wrote:
> I looked into it with the following setup:
> apache 1.3.26-0woody1
> libapache-mod-perl 1.27-2
> postgresql 7.2.1-2woody2
>
> There was a Test.pm module handling all requests for /. It opened a
> connection to the database and fetched a couple of rows.
>
> With DBI->install_driver('Pg'), the Apache processes stayed at 8.4 MB. I
> bombarded the Apache with ApacheBench and multiple SIGUSR1 but I found
> no memory leaks. The process sizes did not change.
The interesting thing I see here is that you are using a different
libapache-mod-perl version. I'm using the one from stable. I hesitate
to install yours (from unstable), since apt-get wants to upgrade all of
perl (a whole *bunch* of shit), and I'm not keen on the idea of
permanently tracking unstable with something as complex as Perl and its
dependents. I mean, crap, it even wants to upgrade libc6. Last time I
tried, downgrading libc6 was not fun. I'm guessing you might find it
difficult to downgrade as well, since you probably have Perl 5.8. Any
ideas?
>
>
>
> I'm pretty tired at the moment, so there might have been something I
> mislooked. I'll give it another look tomorrow. At work, where I don't
> have any distractions :)
I appreciate you looking at this. But it's mainly for the maintainer at
this point...
>
> It's pretty odd, in my view, that the Apache processes which did not
> preload the Pg driver were smaller.
Why would that be odd? Seems intuitively right to me.
>
> Keith, could you post results to modperl list if people at debian-isp
> come to a different (or to the same) solution. I'm not subscribed to
> the list.
Will do, for sure.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by Juha-Mikko Ahonen <ju...@mail.suomi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
> OK, it gets weirder. The following script produces the leak. If I
> comment out the install_driver line, I get a big old segfault! Same
> if I comment out the Apache::DBI line in addition. This works with
> plain apache, or apache-ssl.
>
> #!/usr/bin/perl
> use strict;
> use Apache::Status ();
> use Apache::DBI ();
> DBI->install_driver('Pg');
> 1; #return true value
I looked into it with the following setup:
apache 1.3.26-0woody1
libapache-mod-perl 1.27-2
postgresql 7.2.1-2woody2
There was a Test.pm module handling all requests for /. It opened a
connection to the database and fetched a couple of rows.
With DBI->install_driver('Pg'), the Apache processes stayed at 8.4 MB. I
bombarded the Apache with ApacheBench and multiple SIGUSR1 but I found
no memory leaks. The process sizes did not change.
> If I comment it all out including the Apache::Status line, I get
> neither the segfault or the leak. (Well, small leak, a few bytes).
Without the install_driver() in place, everything worked as well. The
Apache processes stayed at near 6 MB. No segfaults. No problems.
> So the install_driver masks a bigger problem with Apache::Status?
> That's just too weird. But true, I swear.
>
> Is there anyone else out there with Debian stable w/
> apache+libapache-mod-perl who can try to reproduce this simple case?
> I'd really like either a confirmation or a counterexample. If the
> latter, I'd chalk it up to a weirdness on my system (and probably not
> even pursue it, since I've moved on to the static version). I'm
> pursuing it this far so Dan J. might get some useful feedback.
I'm pretty tired at the moment, so there might have been something I
mislooked. I'll give it another look tomorrow. At work, where I don't
have any distractions :)
It's pretty odd, in my view, that the Apache processes which did not
preload the Pg driver were smaller.
Keith, could you post results to modperl list if people at debian-isp
come to a different (or to the same) solution. I'm not subscribed to
the list.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE9rxEXnksV4Ys/z5gRAqlBAJ9xKgkCmCxsg8huo/C1N0uk2XKSowCgoCdr
FdyuRGCkZvDifk5fyqfOcdo=
=im3K
-----END PGP SIGNATURE-----
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Keith G. Murphy" <ke...@mindspring.com>.
[debian-isp readers, to recap, I'm trying to confirm a
memory-leak/segfault problem with Debian stable plus apache(-ssl) plus
libapache-mod-perl. The memory leak happens upon
/etc/init.d/apache(-ssl) reload. You can see my startup script and my
other comments below.]
Juha-Mikko Ahonen wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wednesday 16 October 2002 22:52, Keith G. Murphy wrote:
>
>>It's not like it was an obvious problem: I only got the DSO to leak
>>when loading the Pg driver. That's pretty obscure.
>
>
> Have you tried to connect() without loading the Pg driver first? E.g.
> $dbh = DBI->connect('dbi:Pg:dbname=foo;host=bar', 'user', 'pass');
>
> It would help eliminate if the leak is caused by your startup script. If
> it is, it may be the DBI module that's leaking memory.
OK, it gets weirder. The following script produces the leak. If I
comment out the install_driver line, I get a big old segfault! Same if
I comment out the Apache::DBI line in addition. This works with plain
apache, or apache-ssl.
#!/usr/bin/perl
use strict;
use Apache::Status ();
use Apache::DBI ();
DBI->install_driver('Pg');
1; #return true value
If I comment it all out including the Apache::Status line, I get neither
the segfault or the leak. (Well, small leak, a few bytes).
So the install_driver masks a bigger problem with Apache::Status?
That's just too weird. But true, I swear.
Is there anyone else out there with Debian stable w/
apache+libapache-mod-perl who can try to reproduce this simple case?
I'd really like either a confirmation or a counterexample. If the
latter, I'd chalk it up to a weirdness on my system (and probably not
even pursue it, since I've moved on to the static version). I'm
pursuing it this far so Dan J. might get some useful feedback.
I've submitted this to debian-isp as well; maybe someone there can try it.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by Juha-Mikko Ahonen <ju...@mail.suomi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Wednesday 16 October 2002 22:52, Keith G. Murphy wrote:
> It's not like it was an obvious problem: I only got the DSO to leak
> when loading the Pg driver. That's pretty obscure.
Have you tried to connect() without loading the Pg driver first? E.g.
$dbh = DBI->connect('dbi:Pg:dbname=foo;host=bar', 'user', 'pass');
It would help eliminate if the leak is caused by your startup script. If
it is, it may be the DBI module that's leaking memory.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE9rcZ5nksV4Ys/z5gRAjBQAJwKOUuu4B14yLe1msSNekI0mnPvdACdFMOe
TFLM0arS4L177Y+OuJMAUgM=
=GsvK
-----END PGP SIGNATURE-----
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Keith G. Murphy" <ke...@mindspring.com>.
Daniel Jacobowitz wrote:
> On Wed, Oct 16, 2002 at 02:01:33PM -0500, Keith G. Murphy wrote:
>>
>>My own bug report is now 47 days old, without apparent followup.
Hmmm, I probably should not have posted that. Sounds like a major whine.
>
>
> That's because I'm having an attack of real life. I haven't had time
> to look at my Debian bug reports in a month or so now.
>
> I don't know why the DSO is leaking in a case when the static module
> isn't, but it should be fixable; I just need to find five or six hours
> to sit down with it.
>
Thanks, Dan, this really wasn't meant to be a slam-the-developer
session. I really, *really* appreciate all the work you've done already.
Now I feel tacky. :-(
It's not like it was an obvious problem: I only got the DSO to leak when
loading the Pg driver. That's pretty obscure.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Keith G. Murphy" <ke...@mindspring.com>.
Ged Haywood wrote:
> Hi there,
>
> On Wed, 16 Oct 2002, Keith G. Murphy wrote:
>
>
>>>Significant improvements have been made in
>>>the reliability of mod_perl as DSO and nowadays there is much less
>>>discussion about it on this list.
>>
>>Are you sure it's not because 'most everyone has silently given up on it?
>
>
> Yes, I'm sure. Red Hat comes with Apache prebuilt for DSO I believe.
> I have worked on contracts for a number of organizations which use DSO
> mod_perl in high volume environments (hundreds of thousands of users).
>
Then I take your word for it. (My question was sincere, not
argumentative, by the way).
>
>>>There are still one or two dusty
>>>corners but in general thesedays I'd say try it.
>>
>>I had not expected Debian stable to be one of the dusty corners.
>
>
> What I meant was that some (perhaps slightly unusual in some way)
> configurations seem to give more problems than others. For example
> I think that servers using SSL have perhaps been prone to troubles,
> but it could just as easily be that the people who experienced the
> problems were not careful to follow what you might say are slightly
> more complex than usual build procedures.
>
Hmmm. In my case, I replicated the problem with Debian's plain vanilla
Apache as well, for the very purpose of simplification. So SSL does not
seem to be involved.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by Daniel Jacobowitz <dr...@false.org>.
On Wed, Oct 16, 2002 at 02:01:33PM -0500, Keith G. Murphy wrote:
> Ged Haywood wrote:
> >Hi there,
> >
> >On Wed, 16 Oct 2002, Keith G. Murphy wrote:
> >
> >
> >>do you mean that the problems with the loadable module overall are
> >>so well-known that no one in his right mind should ever use it?
> >
> >
> >It's not as bad as that. Significant improvements have been made in
> >the reliability of mod_perl as DSO and nowadays there is much less
> >discussion about it on this list.
>
> Are you sure it's not because 'most everyone has silently given up on it?
>
> >There are still one or two dusty
> >corners but in general thesedays I'd say try it.
>
> I had not expected Debian stable to be one of the dusty corners. But I
> did find, upon investigating the bug reports, that there were *very* old
> reports about memory leaks, etc., with libapache-mod-perl.
>
> My own bug report is now 47 days old, without apparent followup.
That's because I'm having an attack of real life. I haven't had time
to look at my Debian bug reports in a month or so now.
I don't know why the DSO is leaking in a case when the static module
isn't, but it should be fixable; I just need to find five or six hours
to sit down with it.
--
Daniel Jacobowitz
MontaVista Software Debian GNU/Linux Developer
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by Ged Haywood <ge...@www2.jubileegroup.co.uk>.
Hi there,
On Wed, 16 Oct 2002, Keith G. Murphy wrote:
> > Significant improvements have been made in
> > the reliability of mod_perl as DSO and nowadays there is much less
> > discussion about it on this list.
>
> Are you sure it's not because 'most everyone has silently given up on it?
Yes, I'm sure. Red Hat comes with Apache prebuilt for DSO I believe.
I have worked on contracts for a number of organizations which use DSO
mod_perl in high volume environments (hundreds of thousands of users).
> > There are still one or two dusty
> > corners but in general thesedays I'd say try it.
>
> I had not expected Debian stable to be one of the dusty corners.
What I meant was that some (perhaps slightly unusual in some way)
configurations seem to give more problems than others. For example
I think that servers using SSL have perhaps been prone to troubles,
but it could just as easily be that the people who experienced the
problems were not careful to follow what you might say are slightly
more complex than usual build procedures.
73,
Ged.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Keith G. Murphy" <ke...@mindspring.com>.
Ged Haywood wrote:
> Hi there,
>
> On Wed, 16 Oct 2002, Keith G. Murphy wrote:
>
>
>>do you mean that the problems with the loadable module overall are
>>so well-known that no one in his right mind should ever use it?
>
>
> It's not as bad as that. Significant improvements have been made in
> the reliability of mod_perl as DSO and nowadays there is much less
> discussion about it on this list.
Are you sure it's not because 'most everyone has silently given up on it?
> There are still one or two dusty
> corners but in general thesedays I'd say try it.
I had not expected Debian stable to be one of the dusty corners. But I
did find, upon investigating the bug reports, that there were *very* old
reports about memory leaks, etc., with libapache-mod-perl.
My own bug report is now 47 days old, without apparent followup.
> If it doesn't seem
> to give you problems then stay with it.
>
> If at first you don't succeed, try again. Then give up.
Actually, that is what I have done already, several months ago. Seeing
several reports of memory leak problems in the list made me think:
"Hmmm, wonder if someone else has seen this?"
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by Ged Haywood <ge...@www2.jubileegroup.co.uk>.
Hi there,
On Wed, 16 Oct 2002, Keith G. Murphy wrote:
> do you mean that the problems with the loadable module overall are
> so well-known that no one in his right mind should ever use it?
It's not as bad as that. Significant improvements have been made in
the reliability of mod_perl as DSO and nowadays there is much less
discussion about it on this list. There are still one or two dusty
corners but in general thesedays I'd say try it. If it doesn't seem
to give you problems then stay with it.
If at first you don't succeed, try again. Then give up. No sense
being a damned fool about it. (Somebody remind me who said that:)
73,
Ged.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Keith G. Murphy" <ke...@mindspring.com>.
Stathy G. Touloumis wrote:
>
>
>> Using Debian's static-mod_perled apache-perl eliminates the problem.
>
>
> Do you mean you are using the 'so' version that comes with Debian?
Yes, in the case that failed. The package is called 'libapache-mod-perl'.
>
> You
> should be using the static build of apache/mod_perl
>
You're telling me. :-(
By "should", do you mean that the problems with the loadable module
overall are so well-known that no one in his right mind should ever use
it?
Might it be related to the Perl-malloc problem cited here:
http://apache.perl.org/docs/1.0/guide/install.html#When_DSO_can_be_Used
If either of the above or true, seems like Debian needs to quit
distributing libapache-mod-perl.
Thanks.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Posted by "Stathy G. Touloumis" <st...@edisonaffiliates.com>.
>Using Debian's static-mod_perled apache-perl eliminates the problem.
Do you mean you are using the 'so' version that comes with Debian? You
should be using the static build of apache/mod_perl