You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Gregory Matthews <gr...@iwebtips.com> on 2002/05/20 05:14:52 UTC
Memory Leaks
I have a couple of questions regarding leaking memory in mod_perl:
1. What are the main culprits, in order of severity, of memory leaks, i.e.:
a. global variables (NOT lexically scoped via my)
b. ...
c. ...
2. When writing code from scratch (a new application), what is the best
way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On,
etc.. ?
I posted an earlier question regarding Apache::Leak which I am hoping to
receive some positive input. I would like to check my final product once
completed to ensure no leaks exist.
Thanks everyone.
Gregory
Re: Memory Leaks
Posted by "F. Xavier Noria" <fx...@isoco.com>.
On Mon, 20 May 2002 16:23:40 -0500
Gregory Matthews <gr...@iwebtips.com> wrote:
: Unfortunately, we only have one machine. If we did employ the cron job as
: a clean-up utility once per day, wouldn't the potential impact of a site
: being unavailable only be for a few seconds (until Apache restarted)?
... provided one is not using memory as unique storage for sessions or
some other data maybe?
-- fxn
Re: Memory Leaks
Posted by Stas Bekman <st...@stason.org>.
Gregory Matthews wrote:
> Does using the Apache::GTopLimit module have the same net effect as
> restarting the server itself by simply killing off the actual processes
> which are growing beyond the set threshold, and thereby causing new
> processes to be born?
It's not the exactly the same, since it won't pick up any changes in
Perl modules on the disk. And that's one of the main reasons for doing
restarts. Otherwise yes.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory Leaks
Posted by Gregory Matthews <gr...@iwebtips.com>.
Does using the Apache::GTopLimit module have the same net effect as
restarting the server itself by simply killing off the actual processes
which are growing beyond the set threshold, and thereby causing new
processes to be born?
Gregory
At 03:27 PM 5/20/2002 -0700, you wrote:
>On Mon, 20 May 2002, Perrin Harkins wrote:
>
> > Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since
> > it results in fewer unnecessary restarts. However, it's still a good
> > idea to restart periodically, because some of the shared memory seems to
> > become unshared over time no matter what you do, and restarting fixes that.
>
>that reminds me, i wrote a c version of Apache::GTopLimit 2+ years ago (at
>somepoint in the 5 months i worked at backflip.com), but never released
>it. if somebody wants to maintain/release it, the package is here:
>http://perl.apache.org/~dougm/mod_gtop-0.02.tar.gz
Re: Memory Leaks
Posted by Doug MacEachern <do...@covalent.net>.
On Mon, 20 May 2002, Perrin Harkins wrote:
> Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since
> it results in fewer unnecessary restarts. However, it's still a good
> idea to restart periodically, because some of the shared memory seems to
> become unshared over time no matter what you do, and restarting fixes that.
that reminds me, i wrote a c version of Apache::GTopLimit 2+ years ago (at
somepoint in the 5 months i worked at backflip.com), but never released
it. if somebody wants to maintain/release it, the package is here:
http://perl.apache.org/~dougm/mod_gtop-0.02.tar.gz
Re: Memory Leaks
Posted by Perrin Harkins <pe...@elem.com>.
Jason wrote:
> If you don't want to restart the server then don't.... do this instead, it should help prevent small leaks from being a problem.
> http://httpd.apache.org/docs-2.0/mod/mpm_common.html#maxrequestsperchild
Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since
it results in fewer unnecessary restarts. However, it's still a good
idea to restart periodically, because some of the shared memory seems to
become unshared over time no matter what you do, and restarting fixes that.
- Perrin
Re: Memory Leaks
Posted by Jason <jt...@cartmanager.net>.
If you don't want to restart the server then don't.... do this instead, it should help prevent small leaks from being a problem.
http://httpd.apache.org/docs-2.0/mod/mpm_common.html#maxrequestsperchild
----- Original Message -----
From: "Per Einar Ellefsen" <pe...@skynet.be>
To: "Gregory Matthews" <gr...@iwebtips.com>
Cc: <mo...@apache.org>
Sent: Monday, May 20, 2002 3:30 PM
Subject: Re: Memory Leaks
> At 23:23 20.05.2002, Gregory Matthews wrote:
> >Unfortunately, we only have one machine. If we did employ the cron job as
> >a clean-up utility once per day, wouldn't the potential impact of a site
> >being unavailable only be for a few seconds (until Apache restarted)?
>
> And if something goes wrong? You'd be having a server offline with noone
> knowing about it.
>
> >At 05:12 PM 5/20/2002 -0400, you wrote:
> >
> >>Like another suggestion, we have a cluster of machines and roll the
> >>restarts every hour. Each machine is offset but 10 minutes.
> >>
> >>Gregory Matthews writes:
> >> > I too thought of setting a cron job to restart the server once per day in
> >> > order to keep the memory "fresh".
> >> >
> >> > In a production environment, are there any downsides to doing this, i.e.,
> >> > server inaccessibility, etc..?
> >> >
> >> > Thanks.
> >> >
> >> > Gregory
> >> >
> >> > At 08:25 AM 5/20/2002 -0400, you wrote:
> >> >
> >> > >It is more an issue of it being worth tracking down a small memory
> >> > >leak vs a large memory leak. Our software still has some very small
> >> > >leaks, on the order of 10kv every hour... it would probably take us a
> >> > >month to track down and solve all these problems. I find it easier to
> >> > >restart the web servers daily.
> >> > >
> >> > >We did have some enourmous leaks as well, based on circular reference,
> >> > >and those ate up 1 GB of memory in about 30 minutes... It took us
> >> > >about three weeks to find it.
> >> > >
> >> > >Gregory Matthews writes:
> >> > > > So am I being overly paranoid concerning the "leak" potential of
> >> mod_perl
> >> > > > programming?
> >> > > >
> >> > > > If I start with "strict" code to begin with and try my best to
> >> stay away
> >> > > > from the problems you mentioned, then any potential memory leak/drain
> >> > > > issues will be avoided?
> >> > > >
> >> > > > Keep in mind, although my application is not designed to launch
> >> the space
> >> > > > shuttle, I do want it to be solid/stable/peformance-packed from the
> >> > > ground up.
> >> > > >
> >> > > > I will be also be using MySql with the Apache::DBI module.
> >> > > >
> >> > > > Thanks in advance.
> >> > > >
> >> > > > Gregory
> >> > > >
> >> > > >
> >> > > > At 11:34 PM 5/19/2002 -0400, you wrote:
> >> > > > > > I have a couple of questions regarding leaking memory in mod_perl:
> >> > > > > >
> >> > > > > > 1. What are the main culprits, in order of severity, of
> >> memory leaks,
> >> > > > >i.e.:
> >> > > > > >
> >> > > > > > a. global variables (NOT lexically scoped via my)
> >> > > > > > b. ...
> >> > > > > > c. ...
> >> > > > > >
> >> > > > > > 2. When writing code from scratch (a new application), what
> >> is the
> >> > > > >best
> >> > > > > > way to avoid creating leaks to begin with, i.e., use strict;,
> >> PerlWarn
> >> > > > >On,
> >> > > > > > etc.. ?
> >> > > > >
> >> > > > >There are actually not very many ways you can leak memory in Perl
> >> (and
> >> > > > >thus mod_perl). Most people confuse memory growth with memory
> >> leakage.
> >> > > > >If you want to know how to avoid memory growth, look at the
> >> performance
> >> > > > >tuning stuff in the Guide, like passing references, avoiding
> >> slurping of
> >> > > > >large files, controlling the buffering of DBI result sets, etc.
> >> > > > >
> >> > > > >Leaks are caused by circular references, the string form of eval (at
> >> > > > >least it used to leak a little), nested closures (sometimes created
> >> > > > >accidentally with the Error module), and one or two obscure syntax
> >> > > > >problems. I think one of them involved code like "my $x = 7 if $y;".
> >> > > > >Matt Sergeant got bitten by this in the early stages of AxKit
> >> > > > >development, and the details are in the mailing list archive.
> >> > > > >
> >> > > > >Global variables by themselves are not a source of leaks or
> >> growth. If
> >> > > > >you slurp a large file into a global, your process will grow, but the
> >> > > > >same is true for a lexical.
> >> > > > >
> >> > > > >- Perrin
> >> > > >
> >> > > >
> >> > >
> >> > >--
> >> > >C Wayne Huling <c....@pobox.com>
> >> >
> >> >
> >>
> >>--
> >>C Wayne Huling <c....@pobox.com>
> >
> >
>
> --
> Per Einar Ellefsen
> per.einar@skynet.be
>
Re: Memory Leaks
Posted by Perrin Harkins <pe...@elem.com>.
Per Einar Ellefsen wrote:
> And if something goes wrong? You'd be having a server offline with noone
> knowing about it.
You can easilly set up mon (http://www.kernel.org/software/mon/) to page
you if the server doesn't come back up within a certain amount of time.
- Perrin
Re: Memory Leaks
Posted by Per Einar Ellefsen <pe...@skynet.be>.
At 23:23 20.05.2002, Gregory Matthews wrote:
>Unfortunately, we only have one machine. If we did employ the cron job as
>a clean-up utility once per day, wouldn't the potential impact of a site
>being unavailable only be for a few seconds (until Apache restarted)?
And if something goes wrong? You'd be having a server offline with noone
knowing about it.
>At 05:12 PM 5/20/2002 -0400, you wrote:
>
>>Like another suggestion, we have a cluster of machines and roll the
>>restarts every hour. Each machine is offset but 10 minutes.
>>
>>Gregory Matthews writes:
>> > I too thought of setting a cron job to restart the server once per day in
>> > order to keep the memory "fresh".
>> >
>> > In a production environment, are there any downsides to doing this, i.e.,
>> > server inaccessibility, etc..?
>> >
>> > Thanks.
>> >
>> > Gregory
>> >
>> > At 08:25 AM 5/20/2002 -0400, you wrote:
>> >
>> > >It is more an issue of it being worth tracking down a small memory
>> > >leak vs a large memory leak. Our software still has some very small
>> > >leaks, on the order of 10kv every hour... it would probably take us a
>> > >month to track down and solve all these problems. I find it easier to
>> > >restart the web servers daily.
>> > >
>> > >We did have some enourmous leaks as well, based on circular reference,
>> > >and those ate up 1 GB of memory in about 30 minutes... It took us
>> > >about three weeks to find it.
>> > >
>> > >Gregory Matthews writes:
>> > > > So am I being overly paranoid concerning the "leak" potential of
>> mod_perl
>> > > > programming?
>> > > >
>> > > > If I start with "strict" code to begin with and try my best to
>> stay away
>> > > > from the problems you mentioned, then any potential memory leak/drain
>> > > > issues will be avoided?
>> > > >
>> > > > Keep in mind, although my application is not designed to launch
>> the space
>> > > > shuttle, I do want it to be solid/stable/peformance-packed from the
>> > > ground up.
>> > > >
>> > > > I will be also be using MySql with the Apache::DBI module.
>> > > >
>> > > > Thanks in advance.
>> > > >
>> > > > Gregory
>> > > >
>> > > >
>> > > > At 11:34 PM 5/19/2002 -0400, you wrote:
>> > > > > > I have a couple of questions regarding leaking memory in mod_perl:
>> > > > > >
>> > > > > > 1. What are the main culprits, in order of severity, of
>> memory leaks,
>> > > > >i.e.:
>> > > > > >
>> > > > > > a. global variables (NOT lexically scoped via my)
>> > > > > > b. ...
>> > > > > > c. ...
>> > > > > >
>> > > > > > 2. When writing code from scratch (a new application), what
>> is the
>> > > > >best
>> > > > > > way to avoid creating leaks to begin with, i.e., use strict;,
>> PerlWarn
>> > > > >On,
>> > > > > > etc.. ?
>> > > > >
>> > > > >There are actually not very many ways you can leak memory in Perl
>> (and
>> > > > >thus mod_perl). Most people confuse memory growth with memory
>> leakage.
>> > > > >If you want to know how to avoid memory growth, look at the
>> performance
>> > > > >tuning stuff in the Guide, like passing references, avoiding
>> slurping of
>> > > > >large files, controlling the buffering of DBI result sets, etc.
>> > > > >
>> > > > >Leaks are caused by circular references, the string form of eval (at
>> > > > >least it used to leak a little), nested closures (sometimes created
>> > > > >accidentally with the Error module), and one or two obscure syntax
>> > > > >problems. I think one of them involved code like "my $x = 7 if $y;".
>> > > > >Matt Sergeant got bitten by this in the early stages of AxKit
>> > > > >development, and the details are in the mailing list archive.
>> > > > >
>> > > > >Global variables by themselves are not a source of leaks or
>> growth. If
>> > > > >you slurp a large file into a global, your process will grow, but the
>> > > > >same is true for a lexical.
>> > > > >
>> > > > >- Perrin
>> > > >
>> > > >
>> > >
>> > >--
>> > >C Wayne Huling <c....@pobox.com>
>> >
>> >
>>
>>--
>>C Wayne Huling <c....@pobox.com>
>
>
--
Per Einar Ellefsen
per.einar@skynet.be
Re: Memory Leaks
Posted by Gregory Matthews <gr...@iwebtips.com>.
Unfortunately, we only have one machine. If we did employ the cron job as
a clean-up utility once per day, wouldn't the potential impact of a site
being unavailable only be for a few seconds (until Apache restarted)?
Gregory
At 05:12 PM 5/20/2002 -0400, you wrote:
>Like another suggestion, we have a cluster of machines and roll the
>restarts every hour. Each machine is offset but 10 minutes.
>
>Gregory Matthews writes:
> > I too thought of setting a cron job to restart the server once per day in
> > order to keep the memory "fresh".
> >
> > In a production environment, are there any downsides to doing this, i.e.,
> > server inaccessibility, etc..?
> >
> > Thanks.
> >
> > Gregory
> >
> > At 08:25 AM 5/20/2002 -0400, you wrote:
> >
> > >It is more an issue of it being worth tracking down a small memory
> > >leak vs a large memory leak. Our software still has some very small
> > >leaks, on the order of 10kv every hour... it would probably take us a
> > >month to track down and solve all these problems. I find it easier to
> > >restart the web servers daily.
> > >
> > >We did have some enourmous leaks as well, based on circular reference,
> > >and those ate up 1 GB of memory in about 30 minutes... It took us
> > >about three weeks to find it.
> > >
> > >Gregory Matthews writes:
> > > > So am I being overly paranoid concerning the "leak" potential of
> mod_perl
> > > > programming?
> > > >
> > > > If I start with "strict" code to begin with and try my best to stay
> away
> > > > from the problems you mentioned, then any potential memory leak/drain
> > > > issues will be avoided?
> > > >
> > > > Keep in mind, although my application is not designed to launch the
> space
> > > > shuttle, I do want it to be solid/stable/peformance-packed from the
> > > ground up.
> > > >
> > > > I will be also be using MySql with the Apache::DBI module.
> > > >
> > > > Thanks in advance.
> > > >
> > > > Gregory
> > > >
> > > >
> > > > At 11:34 PM 5/19/2002 -0400, you wrote:
> > > > > > I have a couple of questions regarding leaking memory in mod_perl:
> > > > > >
> > > > > > 1. What are the main culprits, in order of severity, of memory
> leaks,
> > > > >i.e.:
> > > > > >
> > > > > > a. global variables (NOT lexically scoped via my)
> > > > > > b. ...
> > > > > > c. ...
> > > > > >
> > > > > > 2. When writing code from scratch (a new application), what is the
> > > > >best
> > > > > > way to avoid creating leaks to begin with, i.e., use strict;,
> PerlWarn
> > > > >On,
> > > > > > etc.. ?
> > > > >
> > > > >There are actually not very many ways you can leak memory in Perl (and
> > > > >thus mod_perl). Most people confuse memory growth with memory
> leakage.
> > > > >If you want to know how to avoid memory growth, look at the
> performance
> > > > >tuning stuff in the Guide, like passing references, avoiding
> slurping of
> > > > >large files, controlling the buffering of DBI result sets, etc.
> > > > >
> > > > >Leaks are caused by circular references, the string form of eval (at
> > > > >least it used to leak a little), nested closures (sometimes created
> > > > >accidentally with the Error module), and one or two obscure syntax
> > > > >problems. I think one of them involved code like "my $x = 7 if $y;".
> > > > >Matt Sergeant got bitten by this in the early stages of AxKit
> > > > >development, and the details are in the mailing list archive.
> > > > >
> > > > >Global variables by themselves are not a source of leaks or
> growth. If
> > > > >you slurp a large file into a global, your process will grow, but the
> > > > >same is true for a lexical.
> > > > >
> > > > >- Perrin
> > > >
> > > >
> > >
> > >--
> > >C Wayne Huling <c....@pobox.com>
> >
> >
>
>--
>C Wayne Huling <c....@pobox.com>
Re: Memory Leaks
Posted by Stas Bekman <st...@stason.org>.
Per Einar Ellefsen wrote:
> At 23:54 20.05.2002, Allen Day wrote:
>
>> I've noticed that if I restart apache while I'm in the middle of a
>> download (MP3 stream), after the buffer in my MP3 player runs out, it
>> skips to the next track -- presumably because the connection was closed.
>>
>> This might cause a problem for you if your users are downloading big
>> files. They might have to restart from the beginning if they didn't
>> cache
>> the partial download somewhere.
>
>
> Hmm, if you are serving big files off of mod_perl, memory leaks are the
> least of your problems :)
Well, you can serve big files without reading them into a memory at
once. Why there would be memory leaks?
> That doesn't apply to Apache::MP3 of course,
> for which it's normal, but in no case should your mod_perl server be
> serving your big files.
The reason for not serving big files with mod_perl, is that you don't
want to tie heavy and snappy mod_perl servers to wait indefinitely for
the client to grab their data. If you have plenty of memory or you have
just a few clients (intranet?) that's just fine. This is all discussed here:
http://perl.apache.org/release/docs/1.0/guide/strategy.html#Adding_a_Proxy_Server_in_http_Accelerator_Mode
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory Leaks
Posted by Per Einar Ellefsen <pe...@skynet.be>.
At 00:45 21.05.2002, Issac Goldstand wrote:
>I'd like to try to disagree here. I have built several file-related
>webapps where I have implemented virtual filesystems which require special
>perl modules to access the files at all. mod_perl takes care of serving
>the requests. If I need a restart, then I can still safely use
>graceful. Admittedly there are times when something could very well get
>screwed up, but my solution to that is to develop a better front-end
>server with it's own buffer so that the back-end can swiftly serve the
>files leaving much more idle time (in comparison to directly connecting
>remote client to fileserver) for backend restarts if needed.
In the case that you need advanced logic such as that, I clearly agree with
both you and Allen. And a proxy server is very needed in such a case :)
>Per Einar Ellefsen wrote:
>
>>At 23:54 20.05.2002, Allen Day wrote:
>>
>>>I've noticed that if I restart apache while I'm in the middle of a
>>>download (MP3 stream), after the buffer in my MP3 player runs out, it
>>>skips to the next track -- presumably because the connection was closed.
>>>
>>>This might cause a problem for you if your users are downloading big
>>>files. They might have to restart from the beginning if they didn't cache
>>>the partial download somewhere.
>>
>>
>>Hmm, if you are serving big files off of mod_perl, memory leaks are the
>>least of your problems :) That doesn't apply to Apache::MP3 of course,
>>for which it's normal, but in no case should your mod_perl server be
>>serving your big files.
>>
>>>On Mon, 20 May 2002, Matt Sergeant wrote:
>>>
>>> > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote:
>>> > > I too thought of setting a cron job to restart the server once per
>>> day in
>>> > > order to keep the memory "fresh".
>>> > >
>>> > > In a production environment, are there any downsides to doing this,
>>> i.e.,
>>> > > server inaccessibility, etc..?
>>> >
>>> > It's very rare to have a site that can't cope with just a few seconds
>>> > downtime. Most users won't even notice, save for some slight delay in
>>> getting
>>> > their request through. Users tend to be pretty used to trying again
>>> in this
>>> > world of "reliable" computing.
>>
>>--
>>Per Einar Ellefsen
>>per.einar@skynet.be
Re: Memory Leaks
Posted by Issac Goldstand <ma...@beamartyr.net>.
I'd like to try to disagree here. I have built several file-related
webapps where I have implemented virtual filesystems which require
special perl modules to access the files at all. mod_perl takes care of
serving the requests. If I need a restart, then I can still safely use
graceful. Admittedly there are times when something could very well get
screwed up, but my solution to that is to develop a better front-end
server with it's own buffer so that the back-end can swiftly serve the
files leaving much more idle time (in comparison to directly connecting
remote client to fileserver) for backend restarts if needed.
Issac
Per Einar Ellefsen wrote:
> At 23:54 20.05.2002, Allen Day wrote:
>
>> I've noticed that if I restart apache while I'm in the middle of a
>> download (MP3 stream), after the buffer in my MP3 player runs out, it
>> skips to the next track -- presumably because the connection was closed.
>>
>> This might cause a problem for you if your users are downloading big
>> files. They might have to restart from the beginning if they didn't
>> cache
>> the partial download somewhere.
>
>
> Hmm, if you are serving big files off of mod_perl, memory leaks are
> the least of your problems :) That doesn't apply to Apache::MP3 of
> course, for which it's normal, but in no case should your mod_perl
> server be serving your big files.
>
>> On Mon, 20 May 2002, Matt Sergeant wrote:
>>
>> > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote:
>> > > I too thought of setting a cron job to restart the server once
>> per day in
>> > > order to keep the memory "fresh".
>> > >
>> > > In a production environment, are there any downsides to doing
>> this, i.e.,
>> > > server inaccessibility, etc..?
>> >
>> > It's very rare to have a site that can't cope with just a few seconds
>> > downtime. Most users won't even notice, save for some slight delay
>> in getting
>> > their request through. Users tend to be pretty used to trying again
>> in this
>> > world of "reliable" computing.
>
>
Re: Memory Leaks
Posted by Allen Day <al...@ucla.edu>.
I mentioned the connection closing as a drawback of restarting the server
-- it was slightly OT for the thread.
Yes, it is a subclass of Apache::MP3 that can stream video and audio.
There is an old version called Apache::Jukebox in the Apache::MP3 CVS at
namp.sourceforge.net in case anyone is interested.
-Allen
On Mon, 20 May 2002, Per Einar Ellefsen wrote:
> At 23:54 20.05.2002, Allen Day wrote:
> >I've noticed that if I restart apache while I'm in the middle of a
> >download (MP3 stream), after the buffer in my MP3 player runs out, it
> >skips to the next track -- presumably because the connection was closed.
> >
> >This might cause a problem for you if your users are downloading big
> >files. They might have to restart from the beginning if they didn't cache
> >the partial download somewhere.
>
> Hmm, if you are serving big files off of mod_perl, memory leaks are the
> least of your problems :) That doesn't apply to Apache::MP3 of course, for
> which it's normal, but in no case should your mod_perl server be serving
> your big files.
>
> >On Mon, 20 May 2002, Matt Sergeant wrote:
> >
> > > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote:
> > > > I too thought of setting a cron job to restart the server once per day in
> > > > order to keep the memory "fresh".
> > > >
> > > > In a production environment, are there any downsides to doing this, i.e.,
> > > > server inaccessibility, etc..?
> > >
> > > It's very rare to have a site that can't cope with just a few seconds
> > > downtime. Most users won't even notice, save for some slight delay in
> > getting
> > > their request through. Users tend to be pretty used to trying again in this
> > > world of "reliable" computing.
>
>
Re: Memory Leaks
Posted by Per Einar Ellefsen <pe...@skynet.be>.
At 23:54 20.05.2002, Allen Day wrote:
>I've noticed that if I restart apache while I'm in the middle of a
>download (MP3 stream), after the buffer in my MP3 player runs out, it
>skips to the next track -- presumably because the connection was closed.
>
>This might cause a problem for you if your users are downloading big
>files. They might have to restart from the beginning if they didn't cache
>the partial download somewhere.
Hmm, if you are serving big files off of mod_perl, memory leaks are the
least of your problems :) That doesn't apply to Apache::MP3 of course, for
which it's normal, but in no case should your mod_perl server be serving
your big files.
>On Mon, 20 May 2002, Matt Sergeant wrote:
>
> > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote:
> > > I too thought of setting a cron job to restart the server once per day in
> > > order to keep the memory "fresh".
> > >
> > > In a production environment, are there any downsides to doing this, i.e.,
> > > server inaccessibility, etc..?
> >
> > It's very rare to have a site that can't cope with just a few seconds
> > downtime. Most users won't even notice, save for some slight delay in
> getting
> > their request through. Users tend to be pretty used to trying again in this
> > world of "reliable" computing.
--
Per Einar Ellefsen
per.einar@skynet.be
Re: Memory Leaks
Posted by Allen Day <al...@ucla.edu>.
I've noticed that if I restart apache while I'm in the middle of a
download (MP3 stream), after the buffer in my MP3 player runs out, it
skips to the next track -- presumably because the connection was closed.
This might cause a problem for you if your users are downloading big
files. They might have to restart from the beginning if they didn't cache
the partial download somewhere.
-Allen
On Mon, 20 May 2002, Matt Sergeant wrote:
> On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote:
> > I too thought of setting a cron job to restart the server once per day in
> > order to keep the memory "fresh".
> >
> > In a production environment, are there any downsides to doing this, i.e.,
> > server inaccessibility, etc..?
>
> It's very rare to have a site that can't cope with just a few seconds
> downtime. Most users won't even notice, save for some slight delay in getting
> their request through. Users tend to be pretty used to trying again in this
> world of "reliable" computing.
>
> Matt.
>
Re: Memory Leaks
Posted by Matt Sergeant <ma...@sergeant.org>.
On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote:
> I too thought of setting a cron job to restart the server once per day in
> order to keep the memory "fresh".
>
> In a production environment, are there any downsides to doing this, i.e.,
> server inaccessibility, etc..?
It's very rare to have a site that can't cope with just a few seconds
downtime. Most users won't even notice, save for some slight delay in getting
their request through. Users tend to be pretty used to trying again in this
world of "reliable" computing.
Matt.
Re: Memory Leaks
Posted by Perrin Harkins <pe...@elem.com>.
Gregory Matthews wrote:
> I too thought of setting a cron job to restart the server once per day
> in order to keep the memory "fresh".
>
> In a production environment, are there any downsides to doing this,
> i.e., server inaccessibility, etc..?
There have been some discussion on the list about this in the past. The
ideal situation is to have a cluster that you can do a rolling restart
on without making the service totally unavailable.
- Perrin
Re: Memory Leaks
Posted by Gregory Matthews <gr...@iwebtips.com>.
I too thought of setting a cron job to restart the server once per day in
order to keep the memory "fresh".
In a production environment, are there any downsides to doing this, i.e.,
server inaccessibility, etc..?
Thanks.
Gregory
At 08:25 AM 5/20/2002 -0400, you wrote:
>It is more an issue of it being worth tracking down a small memory
>leak vs a large memory leak. Our software still has some very small
>leaks, on the order of 10kv every hour... it would probably take us a
>month to track down and solve all these problems. I find it easier to
>restart the web servers daily.
>
>We did have some enourmous leaks as well, based on circular reference,
>and those ate up 1 GB of memory in about 30 minutes... It took us
>about three weeks to find it.
>
>Gregory Matthews writes:
> > So am I being overly paranoid concerning the "leak" potential of mod_perl
> > programming?
> >
> > If I start with "strict" code to begin with and try my best to stay away
> > from the problems you mentioned, then any potential memory leak/drain
> > issues will be avoided?
> >
> > Keep in mind, although my application is not designed to launch the space
> > shuttle, I do want it to be solid/stable/peformance-packed from the
> ground up.
> >
> > I will be also be using MySql with the Apache::DBI module.
> >
> > Thanks in advance.
> >
> > Gregory
> >
> >
> > At 11:34 PM 5/19/2002 -0400, you wrote:
> > > > I have a couple of questions regarding leaking memory in mod_perl:
> > > >
> > > > 1. What are the main culprits, in order of severity, of memory leaks,
> > >i.e.:
> > > >
> > > > a. global variables (NOT lexically scoped via my)
> > > > b. ...
> > > > c. ...
> > > >
> > > > 2. When writing code from scratch (a new application), what is the
> > >best
> > > > way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn
> > >On,
> > > > etc.. ?
> > >
> > >There are actually not very many ways you can leak memory in Perl (and
> > >thus mod_perl). Most people confuse memory growth with memory leakage.
> > >If you want to know how to avoid memory growth, look at the performance
> > >tuning stuff in the Guide, like passing references, avoiding slurping of
> > >large files, controlling the buffering of DBI result sets, etc.
> > >
> > >Leaks are caused by circular references, the string form of eval (at
> > >least it used to leak a little), nested closures (sometimes created
> > >accidentally with the Error module), and one or two obscure syntax
> > >problems. I think one of them involved code like "my $x = 7 if $y;".
> > >Matt Sergeant got bitten by this in the early stages of AxKit
> > >development, and the details are in the mailing list archive.
> > >
> > >Global variables by themselves are not a source of leaks or growth. If
> > >you slurp a large file into a global, your process will grow, but the
> > >same is true for a lexical.
> > >
> > >- Perrin
> >
> >
>
>--
>C Wayne Huling <c....@pobox.com>
Re: Memory Leaks
Posted by Perrin Harkins <pe...@elem.com>.
> So am I being overly paranoid concerning the "leak" potential of
mod_perl
> programming?
No, memory management is very important with mod_perl.
> If I start with "strict" code to begin with and try my best to stay
away
> from the problems you mentioned, then any potential memory leak/drain
> issues will be avoided?
Using strict is an absolute must for many reasons. It won't prevent you
from using up tons of memory though. You have to become tuned in to the
things that use up memory so that you can avoid them. Start with the
Guide.
- Perrin
Re: Memory Leaks
Posted by "Doran L. Barton" <fo...@iodynamics.com>.
Not long ago, Gregory Matthews proclaimed...
> So am I being overly paranoid concerning the "leak" potential of mod_perl
> programming?
No... But once you do it "right" it comes natural.
The thing that killed me when I first started doing mod_perl development
was code that pushed items onto lists in Apache::Registry scripts. I wasn't
initializing those list variables and before you knew it, lists were
getting populated with values in duplicate, triplicate, and so on.
Nothing has helped me become a better Perl programming than mod_perl. :)
-=Fozz
--
Doran L. Barton <fo...@iodynamics.com> - Chief Super Hero - Iodynamics LLC
< http://www.iodynamics.com/ > - Linux solutions and dynamic websites
"Dog for sale: eats anything and is fond of children."
-- Classified ad in newspaper
Re: Memory Leaks
Posted by Gregory Matthews <gr...@iwebtips.com>.
So am I being overly paranoid concerning the "leak" potential of mod_perl
programming?
If I start with "strict" code to begin with and try my best to stay away
from the problems you mentioned, then any potential memory leak/drain
issues will be avoided?
Keep in mind, although my application is not designed to launch the space
shuttle, I do want it to be solid/stable/peformance-packed from the ground up.
I will be also be using MySql with the Apache::DBI module.
Thanks in advance.
Gregory
At 11:34 PM 5/19/2002 -0400, you wrote:
> > I have a couple of questions regarding leaking memory in mod_perl:
> >
> > 1. What are the main culprits, in order of severity, of memory leaks,
>i.e.:
> >
> > a. global variables (NOT lexically scoped via my)
> > b. ...
> > c. ...
> >
> > 2. When writing code from scratch (a new application), what is the
>best
> > way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn
>On,
> > etc.. ?
>
>There are actually not very many ways you can leak memory in Perl (and
>thus mod_perl). Most people confuse memory growth with memory leakage.
>If you want to know how to avoid memory growth, look at the performance
>tuning stuff in the Guide, like passing references, avoiding slurping of
>large files, controlling the buffering of DBI result sets, etc.
>
>Leaks are caused by circular references, the string form of eval (at
>least it used to leak a little), nested closures (sometimes created
>accidentally with the Error module), and one or two obscure syntax
>problems. I think one of them involved code like "my $x = 7 if $y;".
>Matt Sergeant got bitten by this in the early stages of AxKit
>development, and the details are in the mailing list archive.
>
>Global variables by themselves are not a source of leaks or growth. If
>you slurp a large file into a global, your process will grow, but the
>same is true for a lexical.
>
>- Perrin
Re: Memory Leaks
Posted by Matt Sergeant <ma...@sergeant.org>.
On Mon, 20 May 2002, Mark Fowler wrote:
> On Mon, 20 May 2002, Matt Sergeant wrote:
>
> > if ($@ && $@->isa('Exception::DB')) {
> > debug "Exception: $@";
> > $um->dbh->rollback;
> > }
> >
> > (note: if you expect all exceptions to be references like this, you had
> > better have a $SIG{__DIE__} handler installed to bless non-blessed
> > exceptions before re-throwing them
>
> Can't you just use UNIVERSAL's ISA method directly?
>
> if (UNIVERSAL::isa($@,'Exception::DB')) {
>
> This of course might fail if you got the string "Exception::DB" or
> likewise back as an error message.
>
> Alternativly, check if it's blessed
>
> use Scalar::Util qw(blessed);
>
> if (blessed($@) && $@->isa('Exception::DB')) {
Yeah, I know all the tricks. Ultimately it's a matter of how ugly you want
your code to get, and how many external modules you want to rely on (I
believe Scalar::Util is going to be part of 5.8 though).
--
<!-- Matt -->
<:->Get a smart net</:->
Re: Memory Leaks
Posted by Mark Fowler <ma...@twoshortplanks.com>.
On Mon, 20 May 2002, Matt Sergeant wrote:
> if ($@ && $@->isa('Exception::DB')) {
> debug "Exception: $@";
> $um->dbh->rollback;
> }
>
> (note: if you expect all exceptions to be references like this, you had
> better have a $SIG{__DIE__} handler installed to bless non-blessed
> exceptions before re-throwing them
Can't you just use UNIVERSAL's ISA method directly?
if (UNIVERSAL::isa($@,'Exception::DB')) {
This of course might fail if you got the string "Exception::DB" or
likewise back as an error message.
Alternativly, check if it's blessed
use Scalar::Util qw(blessed);
if (blessed($@) && $@->isa('Exception::DB')) {
Later.
Mark.
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ mark@twoshortplanks.com
';use Term'Cap;$t=Tgetent Term'Cap{};print$t->Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t->Tgoto(cm,$_,$y)." $w";select$k,$k,$k,.03}$y+=2}
Re: Memory Leaks
Posted by Matt Sergeant <ma...@sergeant.org>.
On Mon, 20 May 2002, F. Xavier Noria wrote:
> On Mon, 20 May 2002 10:15:02 +0100 (BST)
> Matt Sergeant <ma...@sergeant.org> wrote:
>
> : > my $um = UserManager->new;
> : > # ...
> : > try {
> : > $um->write_user($user);
> : > $um->dbh->commit;
> : > } catch Exception::DB with {
> : > my $e = shift;
> : > debug "Exception: $e";
> : > $um->dbh->rollback;
> : > };
> :
> : No. $um is caught in a closure, which could potentially leak.
>
> Wow, thank you, I have that pattern repeated in the code many times.
>
> That is the way I would write that try/catch in Java, where you need to
> have $um in the scope of the try and the catch blocks, what is the right
> way to write that in Perl/Error.pm?
I gave up on Error.pm's try/catch syntax a long time ago - I think it's
hidden closure system combined with perl bugs is just too broken for
production use. Instead I use good old eval:
my $um = UserManager->new;
...
eval {
$um->write_user($user);
$um->dbh->commit;
};
if ($@ && $@->isa('Exception::DB')) {
debug "Exception: $@";
$um->dbh->rollback;
}
(note: if you expect all exceptions to be references like this, you had
better have a $SIG{__DIE__} handler installed to bless non-blessed
exceptions before re-throwing them - ask me if you need an example of
that)
--
<!-- Matt -->
<:->Get a smart net</:->
Re: Memory Leaks
Posted by "F. Xavier Noria" <fx...@isoco.com>.
On Mon, 20 May 2002 10:15:02 +0100 (BST)
Matt Sergeant <ma...@sergeant.org> wrote:
: > my $um = UserManager->new;
: > # ...
: > try {
: > $um->write_user($user);
: > $um->dbh->commit;
: > } catch Exception::DB with {
: > my $e = shift;
: > debug "Exception: $e";
: > $um->dbh->rollback;
: > };
:
: No. $um is caught in a closure, which could potentially leak.
Wow, thank you, I have that pattern repeated in the code many times.
That is the way I would write that try/catch in Java, where you need to
have $um in the scope of the try and the catch blocks, what is the right
way to write that in Perl/Error.pm?
-- fxn
Re: Memory Leaks
Posted by Matt Sergeant <ma...@sergeant.org>.
On Mon, 20 May 2002, F. Xavier Noria wrote:
> On Sun, 19 May 2002 23:34:24 -0400
> "Perrin Harkins" <pe...@elem.com> wrote:
>
> : Leaks are caused by circular references, the string form of eval (at
> : least it used to leak a little), nested closures (sometimes created
> : accidentally with the Error module)
>
> I am using the Error module in my current project, what kind of
> constructs should one avoid? Is this safe?
>
> my $um = UserManager->new;
> # ...
> try {
> $um->write_user($user);
> $um->dbh->commit;
> } catch Exception::DB with {
> my $e = shift;
> debug "Exception: $e";
> $um->dbh->rollback;
> };
No. $um is caught in a closure, which could potentially leak.
--
<!-- Matt -->
<:->Get a smart net</:->
Re: Memory Leaks
Posted by "F. Xavier Noria" <fx...@isoco.com>.
On Sun, 19 May 2002 23:34:24 -0400
"Perrin Harkins" <pe...@elem.com> wrote:
: Leaks are caused by circular references, the string form of eval (at
: least it used to leak a little), nested closures (sometimes created
: accidentally with the Error module)
I am using the Error module in my current project, what kind of
constructs should one avoid? Is this safe?
my $um = UserManager->new;
# ...
try {
$um->write_user($user);
$um->dbh->commit;
} catch Exception::DB with {
my $e = shift;
debug "Exception: $e";
$um->dbh->rollback;
};
-- fxn
Re: sending CGI ouput through a handler
Posted by Perrin Harkins <pe...@elem.com>.
> My::Handler takes the requested file and adds some markup to it with
the
> Template Toolkit if the MIMEtype of the file is text/html. I want to
be
> able to use the same handler to also add markup to the output of
executed
> CGI. What is the best way to do this?
You can't feed the output of mod_cgi to mod_perl, at least not in Apache
1.x. The best thing to do (at least for performance) would be to
rewrite the CGI so that you call it as a module from your handler.
Another approach would be to use Apache::Filter to grab the output from
PerlRun or Regsistry. You could also sublass PerlRun or Registry and
add your handler code to it.
- Perrin
Re: sending CGI ouput through a handler
Posted by Stas Bekman <st...@stason.org>.
Allen Day wrote:
> Okay, I realize this has probably been covered before, but I couldn't find
> anything in the archive...
>
> I have a configuration like this:
>
> <Location />
> SetHandler perl-script
> PerlHandler My::Handler
> </Location>
>
> My::Handler takes the requested file and adds some markup to it with the
> Template Toolkit if the MIMEtype of the file is text/html. I want to be
> able to use the same handler to also add markup to the output of executed
> CGI. What is the best way to do this?
>
> Should I set up a separate location for the CGI (I'm doing this,
> actually) ? Are there any extra directives needed?
by CGI do you mean a script running under mod_cgi? If it's registry you
should be able to use stacked handlers:
http://perl.apache.org/release/docs/1.0/guide/config.html#Stacked_Handlers
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
sending CGI ouput through a handler
Posted by Allen Day <al...@ucla.edu>.
Okay, I realize this has probably been covered before, but I couldn't find
anything in the archive...
I have a configuration like this:
<Location />
SetHandler perl-script
PerlHandler My::Handler
</Location>
My::Handler takes the requested file and adds some markup to it with the
Template Toolkit if the MIMEtype of the file is text/html. I want to be
able to use the same handler to also add markup to the output of executed
CGI. What is the best way to do this?
Should I set up a separate location for the CGI (I'm doing this,
actually) ? Are there any extra directives needed?
TIA
Allen
Re: Memory Leaks
Posted by Perrin Harkins <pe...@elem.com>.
> I have a couple of questions regarding leaking memory in mod_perl:
>
> 1. What are the main culprits, in order of severity, of memory leaks,
i.e.:
>
> a. global variables (NOT lexically scoped via my)
> b. ...
> c. ...
>
> 2. When writing code from scratch (a new application), what is the
best
> way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn
On,
> etc.. ?
There are actually not very many ways you can leak memory in Perl (and
thus mod_perl). Most people confuse memory growth with memory leakage.
If you want to know how to avoid memory growth, look at the performance
tuning stuff in the Guide, like passing references, avoiding slurping of
large files, controlling the buffering of DBI result sets, etc.
Leaks are caused by circular references, the string form of eval (at
least it used to leak a little), nested closures (sometimes created
accidentally with the Error module), and one or two obscure syntax
problems. I think one of them involved code like "my $x = 7 if $y;".
Matt Sergeant got bitten by this in the early stages of AxKit
development, and the details are in the mailing list archive.
Global variables by themselves are not a source of leaks or growth. If
you slurp a large file into a global, your process will grow, but the
same is true for a lexical.
- Perrin