You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Jonathan Vanasco <mo...@2xlp.com> on 2006/08/16 20:54:24 UTC
acceptable memory leaks?
still profiling stuff like crazy, and using ab to bench single pages
while watching top / ps
i've been isolating stuff i know to leak so i can continue looking
for issues
i'm finding a ton of stuff in random libraries, and have been
prioritizing on what to fix
obviously , 4k/request is at the top....
but i've been finding a lot of random things that leak .024k/request
ie
120k / 5000 requests
or some that are
120k / 2500 requests
thats easily fixed with a max request limit of 1000 or so per child
i'm just wondering what people here consider to be acceptable memory
leakage. i'm not going to both fixing the .024k stuff myself, but
anything thats 1k or more per request i think i need to try and fix.
Re: acceptable memory leaks?
Posted by David Scott <ds...@earthlink.net>.
Memory leak != memory hog. The leak could indicate instability and is a
much higher priority than "a measly couple of KB". Even little leaks
need to be plugged.
d
Joel Bernstein wrote:
> On Wed, Aug 16, 2006 at 03:24:32PM -0500, Matthew wrote:
>
>>> Memory is cheap / CPU is cheap
>>>
>> My boss makes this same argument and I hate it. Wasting is still wasting.
>>
>
> Wasting developer time over a measly couple of KB, you mean?
>
> /joel
>
>
Re: acceptable memory leaks?
Posted by Perrin Harkins <pe...@elem.com>.
On Thu, 2006-08-17 at 15:42 -0400, Jonathan Vanasco wrote:
> the dynamic method evals couldn't be fixed. but if that is a bug
> that is fixed in the future (and not a behavior) , thats awesome.
Here's a relevant post:
http://www.nntp.perl.org/group/perl.perl5.porters/115762
I think there have historically been issues with eval STRING even when
the string is not a syntax error.
If you want to play around a bit you can try recompiling your perl with
Perl's malloc or the system's malloc (whichever you are not using now)
to see if it changes things.
- Perrin
Re: acceptable memory leaks?
Posted by Jonathan Vanasco <jo...@2xlp.com>.
On Aug 17, 2006, at 3:52 PM, Michael Peters wrote:
> Most of the problems seem to be with syntactically incorrect string
> evals, not
> code evals, since code evals are compiled when the rest of it is
> compiled anyways.
Interesting. i'm doing a backlog of new features on my project right
now, but when i get back to benching and profiling them, i'll make
sure to profile my code evals, just to see.
i just need to say that its really amazing what a bit of leak fixing
and profiling can do.
this weekend, my processes were at ~30mb with rapid growth, which i
felt was wholly unacceptable.
on tuesday, they were down to 12mb with a 20-40k growth per request
today they're at 10mb with a 4k growth per request.
Re: acceptable memory leaks?
Posted by Michael Peters <mp...@plusthree.com>.
Jonathan Vanasco wrote:
>> Which version of Perl are you running?
>
> 5.8.6 and 5.8.8 . both are showing it.
Just for giggles, you might try the latest 5.9.
> the dynamic method evals couldn't be fixed. but if that is a bug that
> is fixed in the future (and not a behavior) , thats awesome.
Most of the problems seem to be with syntactically incorrect string evals, not
code evals, since code evals are compiled when the rest of it is compiled anyways.
--
Michael Peters
Developer
Plus Three, LP
Re: acceptable memory leaks?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Aug 17, 2006, at 3:29 PM, Michael Peters wrote:
> I remember some Perl bug reports about memory leaks with eval'ing
> some strings.
> I believe these have been fixed in blead-perl and the more recent
> 5.8 are better
> than previous ones. I believe most of these are also being ported
> for the next
> 5.8 release. But I can't find any links to prove that right now.
>
> Which version of Perl are you running?
5.8.6 and 5.8.8 . both are showing it.
i just tore out every string eval I could, and coded around that-- or
tried to code around it in other ways.
it was mostly for class default info, so I was able to get away with
constants for 70% of my code. maybe 20% of my problem code was
solvable with some extended variable variable use. it probably
should have been like that in the beginning anyways.
the dynamic method evals couldn't be fixed. but if that is a bug
that is fixed in the future (and not a behavior) , thats awesome.
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: acceptable memory leaks?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Jonathan Vanasco wrote:
> I just wanted to give the list an update on my recent exploits
>
> Devel::Cycle
> Find memory cycles in objects
> pro:
> easy to integrate into mp2 handlers.
> con:
> it only detects one type of leak , which most people avoid
> creating in the first place
>
> Devel::Leak
> Utility for looking for perl objects that are not reclaimed
> pro:
> easy to integrate into mp2 handlers
> con:
> Note that you need a perl built with -DDEBUGGING for sv_dump()
> to print anything, but counts are valid in any perl.
Isn't there Apache::Leak ? (stas ?)
> Dev'ing on OSX is a PITA. many things won't compile right, the bsd
> layer by apple is severely crippled ( getrusage barely works, with 1/3
> the functionality of what it should have. misc. terminal commands like
> 'top' are similar )
I was not aware it worked at all ?
Its quite disappointing since the man page at least
is nearly identical across NetBsd, OpenBsd, FreeBSD, and Darwin.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
Re: acceptable memory leaks?
Posted by Matt Sergeant <ma...@sergeant.org>.
It does. But you may need:
sub DBI::FIRSTKEY {}
somewhere in your code. Not sure why. I'll ask Artur.
On 21-Aug-06, at 5:48 PM, Jonathan Vanasco wrote:
> Devel::GC::Helper
> con:
> doesn't work with DBI installed
Re: acceptable memory leaks?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
I just wanted to give the list an update on my recent exploits
Devel::Cycle
Find memory cycles in objects
pro:
easy to integrate into mp2 handlers.
con:
it only detects one type of leak , which most people avoid creating
in the first place
Devel::Leak
Utility for looking for perl objects that are not reclaimed
pro:
easy to integrate into mp2 handlers
con:
Note that you need a perl built with -DDEBUGGING for sv_dump() to
print anything, but counts are valid in any perl.
Devel::LeakTrace
Indicate where leaked variables are coming from
con:
I haven't been able to get this to work with mod_perl, as the
persistent environment is a bit counterintutive
Devel::LeakObject
Detect leaks of objects
con:
this seems to do nothing in a persistent environment
Devel::GC::Helper
con:
doesn't work with DBI installed
Dev'ing on OSX is a PITA. many things won't compile right, the bsd
layer by apple is severely crippled ( getrusage barely works, with
1/3 the functionality of what it should have. misc. terminal
commands like 'top' are similar )
Personally, I found this behavior (approximate) using Devel::Leak:
Starting out my server, i have this many 'objects' when i hit a
fairly static page (run through template engine, but nothing dynamic
changes )
197913
I hit reload
+20
I hit reload
+10
I hit reload
+ 7
I hit reload thereafter
+1
To me, it seems the +40 etc from the large growths are normal. the
+1 however looks to be a leak to me. hopefully i can track that
sucker down and end this nightmare.
Re: acceptable memory leaks?
Posted by Michael Peters <mp...@plusthree.com>.
Jonathan Vanasco wrote:
> FWIW, I've found the following things to be the worst:
>
> eval EXPR
> my $x= eval("");
>
> under ab, it grows about 4k per eval per request. that memory
> never seems to be reclaimed under mod_perl.
I remember some Perl bug reports about memory leaks with eval'ing some strings.
I believe these have been fixed in blead-perl and the more recent 5.8 are better
than previous ones. I believe most of these are also being ported for the next
5.8 release. But I can't find any links to prove that right now.
Which version of Perl are you running?
--
Michael Peters
Developer
Plus Three, LP
Re: acceptable memory leaks?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Aug 17, 2006, at 2:54 AM, Leo Lapworth wrote:
>
> Sounds like a totally sensible approach.
>
> I was being slightly flippant with my RAM/CPU is cheep comment
> (though for me personally it does have some mileage) and I'm not
> saying that
> code should be a hack, and of course it must be tested properly.
Ram / CPU is cheap, but to an extent. Once you max your machine
out, it becomes another machine-- which means about a minimum $2400/
year committment ( colo , electricity , bandwidth, hardware , or
dedicated rental ) + dealing with another machine to setup (another 5
minutes every time you have to upgrade ).
Fixing / Tracking stuff like this, for me, is just a lot of long term
savings of time money and aggravation. sure i'll need to cluster out
more eventually and assume those costs, but i won't have to do it as
soon.
FWIW, I've found the following things to be the worst:
eval EXPR
my $x= eval("");
under ab, it grows about 4k per eval per request. that memory
never seems to be reclaimed under mod_perl.
( i haven't profiled eval BLOCK ( eval{} ) yet. i 'm going to
guess there's no issue with it though, as i use it to trap errors
everywhere, and none of my db routines or page generation sections
are growing )
creating a populated anonymous hash ?
the growth i found in Apache::Session ( under ab ) was in TIEHASH :
my $self = {
args => $args,
data => { _session_id => $session_id },
serialized => undef,
lock => 0,
status => 0,
lock_manager => undef, # These two are object refs ...
object_store => undef,
generate => undef, # but these three are subroutine refs
serialize => undef,
unserialize => undef,
};
that too leaked 4k per request under ab, and never got reclaimed
i put the ? in there , because i'm not sure what is at fault.
as i'm not sure exactly how TIEHASH works with tie. the growth
happens in that anonymous hash creation.
there's no growth if
my $self= {};
i checked to make sure the objects were being destroyed correctly,
and they were. i couldn't find any references to the data or any of
the other vars that would have resulted in those objects not being
expired, but I very well could have missed something.
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: acceptable memory leaks?
Posted by Leo Lapworth <le...@cuckoo.org>.
On 16 Aug 2006, at 22:40, Jonathan Vanasco wrote:
> if there's an issue, big or small, i want to know where it is. if
> i can fix it-- great. if i can't fix it, i can document what the
> issue is, so maybe i can fix it when able to or when something
> goes wrong, i know where to look.
Sounds like a totally sensible approach.
I was being slightly flippant with my RAM/CPU is cheep comment
(though for me personally it does have some mileage) and I'm not
saying that
code should be a hack, and of course it must be tested properly.
I applaud your skill and commitment to tracking this down, even if
only to document it.
Good luck :)
Leo
Re: acceptable memory leaks?
Posted by Jonathan Vanasco <jo...@2xlp.com>.
On Aug 16, 2006, at 5:08 PM, Michael Peters wrote:
>
> A memory leak is when memory is used and then lost and can't be
> recovered. If
> the process can still access or use that memory, it's not a leak,
> it's just
> memory growth. Leaks should be completely eliminated. Growth should
> be managed.
I don't know if my problem is growth or leak or what. I do know that
1000 requests to a single URL , which renders essentially static
content , grows several megabytes, it needs to be addressed. Could
it be growth? Sure. Could it be a leak on the perl integration?
Sure- but doubtfully.
What i do know is that somehow some variables, somewhere (i'm
guessing anon hashes ) are not getting cleaned up when they should
be. I can either spend a few hours fixing that now, or leave it as-
is and support it. i used to work at a place that did countless hack
jobs with no QA and tons of bugs, because they made the cash on
support & maintenance (clients didn't care, because they had
something online in 2 weeks instead of 4 months). I HATED , beyond
belief, dealing with that stuff.
if there's an issue, big or small, i want to know where it is. if i
can fix it-- great. if i can't fix it, i can document what the issue
is, so maybe i can fix it when able to or when something goes wrong,
i know where to look.
Re: acceptable memory leaks?
Posted by Michael Peters <mp...@plusthree.com>.
David Scott wrote:
> "...uses more memory over time..." Hmm, sounds like an actual leak to
> me, unless there's some justification for it.
>
> I agree that the problem is probably in someone's XS somewhere. But a
> leak is a leak and is bad.
A memory leak is when memory is used and then lost and can't be recovered. If
the process can still access or use that memory, it's not a leak, it's just
memory growth. Leaks should be completely eliminated. Growth should be managed.
--
Michael Peters
Developer
Plus Three, LP
Re: acceptable memory leaks?
Posted by David Scott <ds...@earthlink.net>.
"...uses more memory over time..." Hmm, sounds like an actual leak to
me, unless there's some justification for it.
I agree that the problem is probably in someone's XS somewhere. But a
leak is a leak and is bad.
d
Perrin Harkins wrote:
> On Wed, 2006-08-16 at 13:35 -0700, David Scott wrote:
>
>> Memory leak != memory hog. The leak could indicate instability and is a
>> much higher priority than "a measly couple of KB". Even little leaks
>> need to be plugged.
>>
>
> When people talk about "memory leaks" on this list, they almost never
> mean actual leaks. What they mean is "uses more memory over time." A
> leak would mean that some memory is allocated and then lost in the
> cracks, i.e. some kind of pointer bug or something at the C level. Perl
> and apache have been cleaned of these pretty well by now.
>
> One place where people do find actual leaks sometimes is in modules that
> use XS code. Fixing that kind of leak can be hard and certainly
> requires some serious C chops.
>
> - Perrin
>
>
>
Re: acceptable memory leaks?
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2006-08-16 at 14:01 -0700, David Scott wrote:
> "...uses more memory over time..." Hmm, sounds like an actual leak to
> me, unless there's some justification for it.
Usually it's just normal perl behavior. You use variables, they need
memory, they don't give the memory back, and the whole thing gets bigger
over time.
- Perrin
Re: acceptable memory leaks?
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2006-08-16 at 13:35 -0700, David Scott wrote:
> Memory leak != memory hog. The leak could indicate instability and is a
> much higher priority than "a measly couple of KB". Even little leaks
> need to be plugged.
When people talk about "memory leaks" on this list, they almost never
mean actual leaks. What they mean is "uses more memory over time." A
leak would mean that some memory is allocated and then lost in the
cracks, i.e. some kind of pointer bug or something at the C level. Perl
and apache have been cleaned of these pretty well by now.
One place where people do find actual leaks sometimes is in modules that
use XS code. Fixing that kind of leak can be hard and certainly
requires some serious C chops.
- Perrin
Re: acceptable memory leaks?
Posted by Joel Bernstein <jo...@fysh.org>.
On Wed, Aug 16, 2006 at 03:24:32PM -0500, Matthew wrote:
> > Memory is cheap / CPU is cheap
>
> My boss makes this same argument and I hate it. Wasting is still wasting.
Wasting developer time over a measly couple of KB, you mean?
/joel
Re: acceptable memory leaks?
Posted by Matthew <ma...@matthewboehm.com>.
> Memory is cheap / CPU is cheap
My boss makes this same argument and I hate it. Wasting is still wasting.
-Matthew
Re: acceptable memory leaks?
Posted by Jonathan Vanasco <jo...@2xlp.com>.
On Aug 16, 2006, at 4:21 PM, Leo Lapworth wrote:
> Memory is cheap / CPU is cheap (for when you reach
> Apache::SizeLimit and need to spawn a new process) - your (and
> other developers) time is not.
practical response:
Because I'm using some modules with known large memory leaks ( open
ssl wrappers )-- which i have to deal with.
I'm also sysadmin on the machine, so I can either spend 16hrs dealing
with mod_perl leaks now, or just ignore them and spend 2hrs a day for
eternity dealing with the ramifications of the leaks and keeping
things up and running.
if someone else had to manage apache and the server, i'd say f-it and
make them deal. but its me on both ends.
devils advocate response:
I'm not saying i want things perfect. I'm fine with some loss. But
a bunch of misc functions i've got going on are leaking 1-4k /request
on average. multiply that by 10-20 functions going on, and thats 40/
request. yeah, i can just use sizelimit, but sizelimit is working on
normal process growth ( increasing result sets, post data ) and the
leaks i can't control. 40k /request @ an average size of 12mb /child
means i've doubled the size every 300 requests-- off those bugs
alone. I'd rather keep max-requests @1000 before i start doubling
the size of the process. i'd rather have spare memory on my machine
and leaner code than something that breaks
Re: acceptable memory leaks?
Posted by Leo Lapworth <le...@cuckoo.org>.
Playing devils advocate...
On 16 Aug 2006, at 19:54, Jonathan Vanasco wrote:
> but anything thats 1k or more per request i think i need to try
> and fix.
Why :) ?
Memory is cheap / CPU is cheap (for when you reach Apache::SizeLimit
and need to spawn a new process) - your (and other developers) time
is not.
If it's a fun project your enjoying doing - that's a different matter :)
Leo