You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Jonathan Vanasco <mo...@2xlp.com> on 2006/08/15 21:24:09 UTC
Dumping / Profiling Memory ?
Does anyone know if there's something in httpd / mod_perl that will
let me dump and analyze the memory in the child (and possibly the
parent)?
My children are pegged at 6-13MB of private memory, on top of
80mb ?!? of parent memory
I'm trying to figure out what in the hell isn't sharing, or causing
the growth. Everything in my modules is ' use strict', OOP and tons
of caching before forking. i've been able to test that the cached
data (which is only like 400k) is indeed staying shared using
Apache::Peek
I haven't found anything in CPAN, and none of the stuff in Perlguts
looks like it could handle this.
Re: Dumping / Profiling Memory ?
Posted by Matt Sergeant <ma...@sergeant.org>.
On 15-Aug-06, at 4:25 PM, Arshavir Grigorian wrote:
> I guess I was more interested in type of structures that continuously
> increase the memory footprint with each request (memory leaks).
I've had moderate success with Devel::GC::Helper [*].
It's not mod_perl integrated, but easy enough to add to your app
dumping to the error log. It tells you what variables are leaking
(though not their names) so you can Data::Dumper them to get an idea
of what the variables might actually be.
Matt.
[*] Outside of mod_perl - I have no idea how well it will work within
a mod_perl environment.
Re: Dumping / Profiling Memory ?
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2006-08-15 at 13:25 -0700, Arshavir Grigorian wrote:
> I guess I was more interested in type of structures that continuously
> increase the memory footprint with each request (memory leaks).
> Barring a database table getting larger, etc, is there any reason why
> the memory footprint should grow with each request (for the same page,
> obviously) ?
There used to be some bugs with code refs that would do this, but I
think they're all fixed in recent versions of Perl. Aside from that, it
would mostly be obvious things: continuously adding to some array or
hash that never goes out of scope maybe. There could be a variable that
you accidentally made into a closure var. Most of the time when I found
one of these it was the code ref bug, as seen when using Error.pm or
other modules that take code refs.
> Interesting ... I thought once a variable went out of scope, its
> memory would be available for other uses within the same Perl process.
> So would you recommend undef-ing all variables that store large data chunks?
If you know you the variable will be rarely used it could be good. If
it's being used frequently, it will just need the memory again next
time.
- Perrin
Re: Dumping / Profiling Memory ?
Posted by Arshavir Grigorian <gr...@gmail.com>.
On 8/15/06, Perrin Harkins <pe...@elem.com> wrote:
> On Tue, 2006-08-15 at 12:49 -0700, Arshavir Grigorian wrote:
> > What would be a good tool for finding where this type of a problem
> > problem is?
>
> Memory growth? You basically comment out code until it stops happening.
>
> > Also, considering that Perl does automagic garbage
> > collection, what sort of coding "style" would produce such a problem
> > (I guess circular references would be one, others?).
>
> There is a lot of information on what causes things to be
> shared/unshared and what coding techniques reduce your memory footprint
> in the Practical mod_perl book. In particular, here:
> http://www.modperlbook.org/html/ch10_01.html
> http://www.modperlbook.org/html/ch14_02.html
>
> People often either slurp a large file or read results from a large
> query (many databases load the entire result set into your process
> unless you tell them not to) and then their process stays big.
I guess I was more interested in type of structures that continuously
increase the memory footprint with each request (memory leaks).
Barring a database table getting larger, etc, is there any reason why
the memory footprint should grow with each request (for the same page,
obviously) ?
> Keep in mind, Perl's garbage collection does not return the memory that
> a variable used when it goes out of scope. If you load up "my $file"
> with 5MB of data, it will be held onto forever and not used for other
> variables, even in the same process. Most of the time this is what you
> want, since it improves performance by avoiding reallocating memory next
> time you use that variable, but you can return the memory by undef'ing
> $file.
Interesting ... I thought once a variable went out of scope, its
memory would be available for other uses within the same Perl process.
So would you recommend undef-ing all variables that store large data chunks?
Re: Dumping / Profiling Memory ?
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2006-08-15 at 12:49 -0700, Arshavir Grigorian wrote:
> What would be a good tool for finding where this type of a problem
> problem is?
Memory growth? You basically comment out code until it stops happening.
> Also, considering that Perl does automagic garbage
> collection, what sort of coding "style" would produce such a problem
> (I guess circular references would be one, others?).
There is a lot of information on what causes things to be
shared/unshared and what coding techniques reduce your memory footprint
in the Practical mod_perl book. In particular, here:
http://www.modperlbook.org/html/ch10_01.html
http://www.modperlbook.org/html/ch14_02.html
People often either slurp a large file or read results from a large
query (many databases load the entire result set into your process
unless you tell them not to) and then their process stays big.
Keep in mind, Perl's garbage collection does not return the memory that
a variable used when it goes out of scope. If you load up "my $file"
with 5MB of data, it will be held onto forever and not used for other
variables, even in the same process. Most of the time this is what you
want, since it improves performance by avoiding reallocating memory next
time you use that variable, but you can return the memory by undef'ing
$file.
- Perrin
Re: Dumping / Profiling Memory ?
Posted by Arshavir Grigorian <gr...@gmail.com>.
On 8/15/06, Perrin Harkins <pe...@elem.com> wrote:
> On Tue, 2006-08-15 at 15:24 -0400, Jonathan Vanasco wrote:
> > Does anyone know if there's something in httpd / mod_perl that will
> > let me dump and analyze the memory in the child (and possibly the
> > parent)?
>
> Linux::Smaps, if you're on Linux.
>
> > My children are pegged at 6-13MB of private memory, on top of
> > 80mb ?!? of parent memory
>
> That doesn't sound very big to me.
>
> > I'm trying to figure out what in the hell isn't sharing, or causing
> > the growth.
>
> Lots of things won't be shared. Just reading a variable can unshare the
> RAM page it's stored in. It should not be growing on every hit though.
> If you run one process and hit it with ab doing the same request over
> and over and it keeps growing, you have a code problem.
>
> - Perrin
>
What would be a good tool for findign where this type of a problem
problem is? Also, considering that Perl does automagic garbage
collection, what sort of coding "style" would produce such a problem
(I guess circular references would be one, others?).
Arshavir
Re: Dumping / Profiling Memory ?
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2006-08-16 at 03:51 -0400, Jonathan Vanasco wrote:
> Can someone confirm if there is a memory leak in Apache Session ?
Your best bet is to just keep going using the same technique until you
find it. You have all the source, so there's no reason to stop when you
hit someone else's code.
> unless i die by passing in bad data, i seem to get a 4k/request loss -
That makes me suspicious because it's really unusual to see a loss on
EVERY request. Perl allocates memory in chunks rather than grabbing
exactly what it needs, which means it doesn't grow every time a variable
grows. Usually you would have to hit a process repeatedly with ab or
something to see that it's continuously growing.
- Perrin
Re: Dumping / Profiling Memory ?
Posted by Jonathan Vanasco <jo...@2xlp.com>.
On Aug 16, 2006, at 7:48 AM, Frank Maas wrote:
> Which one? The first, the latter or both?
any tie.
my $options= $self->ApacheSessionOptions->{ $store };
tie %session , 'Apache::Session::Postgres', $sessionID, $options ;
my %options= $self->ApacheSessionOptions->{ $store };
tie %session , 'Apache::Session::Postgres', $sessionID, \%$options ;
my %options;
while( my($key, $val) = each(%{$self->ApacheSessionOptions->
{ $store }}) ) {
$options{$key}= $val;
}
tie %session , 'Apache::Session::Postgres', $sessionID, \%options ;
tie %session , 'Apache::Session::Postgres', $sessionID, {
'UserName'=> $options->{'UserName'} ,
'Password'=> $options->{'Password'} ,
'DataSource'=> $options->{'Datasource'} ,
'Commit'=> $options->{'Commit'} ,
};
what doesnt create a loss
tie %session , 'Apache::Session::Postgres', $sessionID, %{$options};
however that creates a 'strict refs' error within the whatever
package is causing the error ( i tried encapsulating the code in a no
strict refs block, but it must be reinstated elsewhere.
On Aug 16, 2006, at 9:57 AM, Perrin Harkins wrote:
> That makes me suspicious because it's really unusual to see a loss on
> EVERY request. Perl allocates memory in chunks rather than grabbing
> exactly what it needs, which means it doesn't grow every time a
> variable
> grows. Usually you would have to hit a process repeatedly with ab or
> something to see that it's continuously growing.
its losing 4k/request running on ab
its losing slightly less than 4k every request when i hit it manually
( 4k sometimes, 4k ever 2 items some other times),
RE: Dumping / Profiling Memory ?
Posted by Frank Maas <fr...@cheiron-it.nl>.
> the loss happens directly after the tie.
Which one? The first, the latter or both?
Regards,
Frank
Re: Dumping / Profiling Memory ?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Aug 15, 2006, at 4:28 PM, Perrin Harkins wrote:
> If that happens even when you just send the same request every time to
> the same process, there's something extra going on, like a variable
> not
> going out of scope or being added to each time.
It took me 4 hours to track this down, I finally did.
Can someone confirm if there is a memory leak in Apache Session ?
my %session;
my $store= $self->ApacheSessionStore ;
my $options= $self->ApacheSessionOptions->{ $store };
DEBUG && print STDERR "\nTying? Apache::Session::Postgres -
$sessionID";
eval{
if ( $options->{'Handle'} ) {
tie %session , 'Apache::Session::Postgres', $sessionID, {
'Handle'=> $options->{'Handle'} ,
'Commit'=> $options->{'Commit'} ,
};
}
else {
tie %session , 'Apache::Session::Postgres', $sessionID, $options ;
}
} ;
if ( $@ ) {
$error= 1;
$sessionID= undef;
}
In my example $options comes from a constant in a child class of the
method's class -- but i've tried converting it to a hash (walking it)
and creating a new hash by hand-- neither has leak in memory from
reading it.
No matter what $options is (tested on pg / mysql ) -- i've tried $, \
% , { k=> v }
unless i die by passing in bad data, i seem to get a 4k/request loss -
the loss happens directly after the tie.
Re: Dumping / Profiling Memory ?
Posted by Jonathan Vanasco <jo...@2xlp.com>.
On Aug 15, 2006, at 4:54 PM, Leo Lapworth wrote:
> Just as a reference I run a big site which has processes which
> start at about 30 MBs
> but over tens of thousands of request do grow, I decided it was
> better to spend the time
> doing more coding than days of tracking down small memory leaks
> (good to test for
> large ones obviously).
>
> So in our startup.pl we use SizeLimit:
>
> use Apache::SizeLimit;
> $Apache::SizeLimit::MAX_PROCESS_SIZE = 55000;
Yeah, I've got that running on my production box. I guess I just
expected MP to be smaller with all the optimizations.
i do need to track down my 12k memory leak though. that's killing me.
Re: Dumping / Profiling Memory ?
Posted by Leo Lapworth <le...@cuckoo.org>.
On 15 Aug 2006, at 21:28, Perrin Harkins wrote:
> On Tue, 2006-08-15 at 16:20 -0400, Jonathan Vanasco wrote:
>>>> My children are pegged at 6-13MB of private memory, on top of
>>>> 80mb ?!? of parent memory
>>> That doesn't sound very big to me.
>> Really?
>
> Yeah, that's really not surprising. And you can still run a lot of
> those on a 1GB machine.
>
>> I was under impression that with proper memory management, you could
>> have MP servers down to around 3-6 MBs (at least the modperlbook and
>> docs suggest that )
>
> I don't think I've ever seen a real app that small.
Just as a reference I run a big site which has processes which start
at about 30 MBs
but over tens of thousands of request do grow, I decided it was
better to spend the time
doing more coding than days of tracking down small memory leaks (good
to test for
large ones obviously).
So in our startup.pl we use SizeLimit:
use Apache::SizeLimit;
$Apache::SizeLimit::MAX_PROCESS_SIZE = 55000;
Hope that helps
Leo
Re: Dumping / Profiling Memory ?
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2006-08-15 at 16:20 -0400, Jonathan Vanasco wrote:
> >> My children are pegged at 6-13MB of private memory, on top of
> >> 80mb ?!? of parent memory
> > That doesn't sound very big to me.
> Really?
Yeah, that's really not surprising. And you can still run a lot of
those on a 1GB machine.
> I was under impression that with proper memory management, you could
> have MP servers down to around 3-6 MBs (at least the modperlbook and
> docs suggest that )
I don't think I've ever seen a real app that small.
> i thought unshare on read should only happen if you're accessing a
> string as a number of vice versa.
Sure, I'm just saying that all kinds of little things make pages
unshared and it's hard to predict.
> I have a slight grow , about 10k per request.
If that happens even when you just send the same request every time to
the same process, there's something extra going on, like a variable not
going out of scope or being added to each time.
- Perrin
Re: Dumping / Profiling Memory ?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Aug 15, 2006, at 3:43 PM, Perrin Harkins wrote:
> Linux::Smaps, if you're on Linux.
f*&(. FreeBSD.
i'll dig up a debian box.
>> My children are pegged at 6-13MB of private memory, on top of
>> 80mb ?!? of parent memory
> That doesn't sound very big to me.
Really?
I was under impression that with proper memory management, you could
have MP servers down to around 3-6 MBs (at least the modperlbook and
docs suggest that )
I tried doing some dumbed down servers/handlers where the only thing
involved was the Apache2/APR libraries-- on first hit the children's
memory spiked to 6mb each.
> Lots of things won't be shared. Just reading a variable can
> unshare the
> RAM page it's stored in. It should not be growing on every hit
> though.
> If you run one process and hit it with ab doing the same request over
> and over and it keeps growing, you have a code problem.
i thought unshare on read should only happen if you're accessing a
string as a number of vice versa. I'll see whats going on once I've
got the Debian box up. The 12MB just seems too big.
I have a slight grow , about 10k per request. I need to track that
down. I think that should be normal, and I've got max-requests set
to 1000 so it never gets out of hand.
I'm glad you don't think thats a big size though. It was a lot
larger than I was expecting.
Re: Dumping / Profiling Memory ?
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2006-08-15 at 15:24 -0400, Jonathan Vanasco wrote:
> Does anyone know if there's something in httpd / mod_perl that will
> let me dump and analyze the memory in the child (and possibly the
> parent)?
Linux::Smaps, if you're on Linux.
> My children are pegged at 6-13MB of private memory, on top of
> 80mb ?!? of parent memory
That doesn't sound very big to me.
> I'm trying to figure out what in the hell isn't sharing, or causing
> the growth.
Lots of things won't be shared. Just reading a variable can unshare the
RAM page it's stored in. It should not be growing on every hit though.
If you run one process and hit it with ab doing the same request over
and over and it keeps growing, you have a code problem.
- Perrin