You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Jonathan Vanasco <mo...@2xlp.com> on 2006/09/06 23:36:50 UTC
Re: Memory leak in mod_perl?
On Aug 18, 2006, at 6:24 AM, Andreas Rieke wrote:
> Hi,
>
> after booting a redhat enterprise linux 3 machine with apache 2.0.58,
> perl 5.8.8 and mod_perl 2.0.2,
> it runs well using about 300 M of 1 G physical RAM.
> However, the remaining RAM decreases day by day, and after 2 or 3
> weeks, the machine crashes because swapping takes too much time.
> However, all processes together take about 250 MBytes according to ps,
> thus I assume that the kernel takes the rest. free tells me in fact
> that
> much swap space is used an nearly no physical RAM is left.
>
> The strange thing is that all the memory is gone even after
> stopping all
> apache processes.
> The only thing which helps is to reboot the machine.
I'm experiencing nearly the same thing now on
FreeBSD 6.0 REL
Apache/2.0.58
mod_perl 2.0.2
Postgres 8.1.3
From what I can tell, this is happening:
bootup: ( pg ) 861 Free
apache start: ( apache , pg , 3x pg clients ) 785 Free
apache stop: ( pg, 3x pg clients ) 840 Free
apache start: ( apache , pg , 3x pg clients ) 774 Free
apache stop: ( pg, 3x pg clients ) 829 Free
apache start: ( apache , pg , 3x pg clients ) 773 Free
apache stop: ( pg, 3x pg clients ) 829 Free
Note: the large drop in free memory is correlated to an equal
increase in Pg's shared memory- so i'm not worried about that. Yet.
pg stop: 833 free
oh, wait. whats's that? i killed all those pg connections and no
shared memory was shared? what the...
ugh... lets check with ipcs?
nope, no shared mem for postgres found. that stuff just disappeared
fine, let's just go on with the test
free memory @ start: 773 ( apache , pg , + clients each )
free memory after 10,000 requests : 697M ( apache , pg , + clients
each )
difference : 76M
no worries! This is attributed to:
mp copy-on-write
postgres slurping memory into shared
but wait...
stop apache (pg): 762M
stop postgres () : 773 free
we now have as much free memory NOT running apache & pg as we did
when we WERE running them. thats just a coincidence that the numbers
are the same.
Note that this is a pretty bad test.
I ***THINK*** that this is all to do with pg's shared memory. I
don't have the facilities, however, to test this using MP+ mysql
( which would rule out Apache / modperl and DBI as being the
culprit ), or another postgres method.
So, in regards to the original poster:
a- I think you're going into swap, because your Postgres config
doesn't have a limit on memory Perhaps your apache too. Check
that. Load a ton of crap into Apache before the fork, and set max-
requests to 500 or so. That should severely limit your memory use.
b- I think only reboots work to reclaim memory, because there seems
to be an oddity with Postgres releasing shared memory.
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: Memory leak in mod_perl?
Posted by Jonathan <mo...@2xlp.com>.
> IIRC, didn't we track this to Postgresql ? If so, we should
> relocate to the postgresql lists.
Not officially/definitively, though i've already posted it to the
pgsql lists. ( and ruled out bsd as being the cause thanks to some
friends with NYC BSD UG )
Re: Memory leak in mod_perl?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Fred Tyler wrote:
> Linux 2.6.12.6
> Apache 1.3.33
> Postgresql 7.4.9
> mod_perl 1.29
> 350-400 viritual hosted domains, all running a mod_perl/postgres backed
> CMS.
IIRC, didn't we track this to Postgresql ? If so, we should relocate to the postgresql lists.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
When I call your name, Girl, it starts to flame
Burning in my heart, Tearing it all apart..
No matter how I try My love I cannot hide....
Re: Memory leak in mod_perl?
Posted by Fred Tyler <fr...@gmail.com>.
> > after booting a redhat enterprise linux 3 machine with apache 2.0.58,
> > perl 5.8.8 and mod_perl 2.0.2,
> > it runs well using about 300 M of 1 G physical RAM.
> > However, the remaining RAM decreases day by day, and after 2 or 3
> > weeks, the machine crashes because swapping takes too much time.
> > However, all processes together take about 250 MBytes according to ps,
> > thus I assume that the kernel takes the rest. free tells me in fact
> > that much swap space is used an nearly no physical RAM is left.
> >
> > The strange thing is that all the memory is gone even after
> > stopping all apache processes.
> > The only thing which helps is to reboot the machine.
>
>
> I'm experiencing nearly the same thing now on
> FreeBSD 6.0 REL
> Apache/2.0.58
> mod_perl 2.0.2
> Postgres 8.1.3
I wish I'd caught this thread from the beginning. I've been seeing
this exact same problem for several months now. Here is the
environment where I first noticed it:
Linux 2.6.12.6
Apache 1.3.33
Postgresql 7.4.9
mod_perl 1.29
350-400 viritual hosted domains, all running a mod_perl/postgres backed CMS.
To make it clear: The memory is definitely lost. After 2-3 weeks on a
machine with 2GB of RAM, I can shut down *every single daemon* on the
machine and the RAM is not freed. And it is not a misreading of "top"
or "free". That memory is gone, and trying to keep the machine running
results in it going heavily into swap just for basic processes, and
eventually crashing.
It looks a lot like a kernel memory leak, and for the past few months
that is what I've assumed it was. I don't know how else to explain the
fact that no processes are running but the kernel still sees all the
RAM taken. However, the fact that it was reported on 2.4 *and* BSD
here makes me reconsider.
Andreas: Are you also running PostgreSQL as Jonathan and I are?
Andreas & Jonathan: Are either of you running reiserfs? (Perhaps it is
the filesystem causing this.)
Ok, now possibly some light at the end of the tunnel: I have set up a
second machine with the following specs:
Linux 2.6.16.27
Postgresql 7.4.13
Apache 1.3.37
mod_perl 1.29
~100 virtual hosted domains, all running mod_perl/postgres CMS
I have not seen any memory issues with this machine. When all of the
processes are stopped, the memory usage goes right back down to where
it should be. However, I don't recall seeing the memory problems on
the other machine until it hit around 300-350 vhosted domains. But
then again, I wasn't paying very much attention until it actually
became a problem. Right now I'd say it's a little too early to tell,
but in a couple months I should have ~300 domains on this box as well,
and then I'll know if this particular configuration is immune to the
memory problems.
Also, I've eliminated Apache as the cause, as the following
configuration also has the memory problem:
Linux 2.6.12.6
Apache 1.3.37
Postgresql 7.4.9
mod_perl 1.29
Perhaps this weekend I'll upgrade the postgres on this box to 7.4.13
and see if it makes a difference. If not, the only thing left is the
kernel...
Re: Memory leak in mod_perl?
Posted by Frank Wiles <fr...@wiles.org>.
On Wed, 6 Sep 2006 19:09:15 -0400
Jonathan Vanasco <mo...@2xlp.com> wrote:
> On Sep 6, 2006, at 6:41 PM, Frank Wiles wrote:
>
> > Is that 500MB that "vanished" in used, buffers, or cached? Just
> > because it isn't listed in "free" doesn't mean it isn't "free"
> > from a "Available memory I can use" standpoint. For example, your
> > system
> > will reclaim memory from cached and buffers when it runs out of
> > "free".
>
> off my original discovery, a few restarts of apache/pg and i started
> running into swap- like the original poster.
>
> i haven't been able to run memory down that low on purpose yet-- if
> you've got any clues into examining the cache/buffers , I'll gladly
> give it a go.
Well if you're running into swap then yeah, you've got a problem.
Sorry haven't been following this thread as closely as I probably
should have before commenting. :)
I'd suggest running just pg on the system for awhile and do some
stop/starts and see if you can force it to swap with just pg.
That'll remove apache/mp from the picture entirely as being a
possible source of the problem.
I'd also check to make sure you updated DBD::Pg recently as there
were a few memory leaks fixed not long ago and also to make sure it
has been compiled against the 8.1.3 libpq ( in case you did a pg
upgrade since DBD::Pg was built ).
If running with just pg doesn't force you to go into swap, if
possible try running your mp code without pg running. Again doing
the stop/starts to see if you can get into swap that way.
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: Memory leak in mod_perl?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Sep 6, 2006, at 6:41 PM, Frank Wiles wrote:
> Is that 500MB that "vanished" in used, buffers, or cached? Just
> because it isn't listed in "free" doesn't mean it isn't "free" from
> a "Available memory I can use" standpoint. For example, your
> system
> will reclaim memory from cached and buffers when it runs out of
> "free".
off my original discovery, a few restarts of apache/pg and i started
running into swap- like the original poster.
i haven't been able to run memory down that low on purpose yet-- if
you've got any clues into examining the cache/buffers , I'll gladly
give it a go.
Re: Memory leak in mod_perl?
Posted by Frank Wiles <fr...@wiles.org>.
On Wed, 6 Sep 2006 18:26:46 -0400
Jonathan Vanasco <mo...@2xlp.com> wrote:
>
> On Sep 6, 2006, at 6:01 PM, Frank Wiles wrote:
>
> > It didn't disappear, and it isn't "shared" like the type of
> > sharing we talk about with mod_perl/Apache. It's not CoW sharing.
> I know.
>
> > The shared memory you're talking about here is held by the
> > postmaster
> > daemon and is used to store frequently requested data to be used
> > by the forked children via SysV IPC.
>
> Yes. I know that. pg_stop is the stopping of the postmaster daemon
> -- not clients ( apache stop will kill the pids of the clients ).
>
> > And yeah, PostgreSQL cleans up it's shared memory when it exits
> > like it should.
>
> No, its not. thats the point of my post.
Sorry I guess I misunderstood what you were saying.
> On my system, shared memory that was allocated to postgres is not
> being returned. postgres's exit releases the lock on memory-- ipcs
> shows nothing allocated. but the kernel never gains any of that
> shared mem. i only see about the size of the resident memory of the
> postmaster instance and the clients returned.
>
> I'm inclined to think this 'leak' is in postgres and not in
> Apache/MP/ DBI, because it happens to pg via the command line
>
> the long running process where i found this behavior started off at
> about 850mb of free memory. earlier today my machine was down to
> 241mb free- a stop/wait/start of apache brought me to 317 and a
> restart of pg brought me to 321. thats about 500mb vanished.
>
> then i remembered this post, started poking around , and saw memory
> disappear and never made available in proportion to the pg vsize
> ( the clients themselves are nothing).
Is that 500MB that "vanished" in used, buffers, or cached? Just
because it isn't listed in "free" doesn't mean it isn't "free" from
a "Available memory I can use" standpoint. For example, your system
will reclaim memory from cached and buffers when it runs out of
"free".
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: Memory leak in mod_perl?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Sep 6, 2006, at 6:01 PM, Frank Wiles wrote:
> It didn't disappear, and it isn't "shared" like the type of sharing
> we talk about with mod_perl/Apache. It's not CoW sharing.
I know.
> The shared memory you're talking about here is held by the
> postmaster
> daemon and is used to store frequently requested data to be used by
> the forked children via SysV IPC.
Yes. I know that. pg_stop is the stopping of the postmaster daemon
-- not clients ( apache stop will kill the pids of the clients ).
> And yeah, PostgreSQL cleans up it's shared memory when it exits
> like
> it should.
No, its not. thats the point of my post.
On my system, shared memory that was allocated to postgres is not
being returned. postgres's exit releases the lock on memory-- ipcs
shows nothing allocated. but the kernel never gains any of that
shared mem. i only see about the size of the resident memory of the
postmaster instance and the clients returned.
I'm inclined to think this 'leak' is in postgres and not in Apache/MP/
DBI, because it happens to pg via the command line
the long running process where i found this behavior started off at
about 850mb of free memory. earlier today my machine was down to
241mb free- a stop/wait/start of apache brought me to 317 and a
restart of pg brought me to 321. thats about 500mb vanished.
then i remembered this post, started poking around , and saw memory
disappear and never made available in proportion to the pg vsize
( the clients themselves are nothing).
Re: Memory leak in mod_perl?
Posted by Frank Wiles <fr...@wiles.org>.
On Wed, 6 Sep 2006 17:36:50 -0400
Jonathan Vanasco <mo...@2xlp.com> wrote:
> From what I can tell, this is happening:
> bootup: ( pg ) 861 Free
> apache start: ( apache , pg , 3x pg clients ) 785 Free
> apache stop: ( pg, 3x pg clients ) 840 Free
> apache start: ( apache , pg , 3x pg clients ) 774 Free
> apache stop: ( pg, 3x pg clients ) 829 Free
> apache start: ( apache , pg , 3x pg clients ) 773 Free
> apache stop: ( pg, 3x pg clients ) 829 Free
>
> Note: the large drop in free memory is correlated to an
> equal increase in Pg's shared memory- so i'm not worried about that.
> Yet.
>
> pg stop: 833 free
>
> oh, wait. whats's that? i killed all those pg connections
> and no shared memory was shared? what the...
>
> ugh... lets check with ipcs?
>
> nope, no shared mem for postgres found. that stuff just
> disappeared
It didn't disappear, and it isn't "shared" like the type of sharing
we talk about with mod_perl/Apache. It's not CoW sharing.
The shared memory you're talking about here is held by the postmaster
daemon and is used to store frequently requested data to be used by
the forked children via SysV IPC.
And yeah, PostgreSQL cleans up it's shared memory when it exits like
it should.
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------