You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Andreas Rieke <an...@isl.de> on 2006/08/18 12:24:56 UTC
Memory leak in mod_perl?
Hi,
after booting a redhat enterprise linux 3 machine with apache 2.0.58,
perl 5.8.8 and mod_perl 2.0.2,
it runs well using about 300 M of 1 G physical RAM.
However, the remaining RAM decreases day by day, and after 2 or 3
weeks, the machine crashes because swapping takes too much time.
However, all processes together take about 250 MBytes according to ps,
thus I assume that the kernel takes the rest. free tells me in fact that
much swap space is used an nearly no physical RAM is left.
The strange thing is that all the memory is gone even after stopping all
apache processes.
The only thing which helps is to reboot the machine.
This behaviour has been seen not only on Red Hat Enterprise Linux 3 with
a 2.4
kernel, but also on SuSE Linux 10 with a 2.6.13-15-default kernel.
For that reason, I do not assume that the kernel is the problem. There are
no unusual things running on the machine, the main application is the
apache web server with mod_perl and a PostgreSQL database.
I received some hints that mod_perl might cause this problem, and
searched this mailing list without results.
Maybe anyone has seen a similar problem and found a solution,
thanks in advance,
Andreas
Re: Memory leak in mod_perl?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Perrin Harkins wrote:
> On Fri, 2006-08-18 at 12:24 +0200, Andreas Rieke wrote:
>> it runs well using about 300 M of 1 G physical RAM.
>> However, the remaining RAM decreases day by day, and after 2 or 3
>> weeks, the machine crashes because swapping takes too much time.
>> However, all processes together take about 250 MBytes according to ps,
>> thus I assume that the kernel takes the rest. free tells me in fact that
>> much swap space is used an nearly no physical RAM is left.
>
> I think you'd better get some monitoring on that system and find out
> what's really happening. Log the size of the apache processes on a
> regular basis (or graph them with Cacti or MRTG) and see if they are the
> problem or not.
/dev/shm is allocated out of kernel heap space typically ~1GB, though you can change
this by changing the address space the kernel is loaded into. When you hit the max, its a hard panic/reboot
instantaneously.
You might try dropping the number of inodes for this to keep the size smaller.
At least thats my best guess.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
Re: Memory leak in mod_perl?
Posted by Perrin Harkins <pe...@elem.com>.
On Fri, 2006-08-18 at 19:35 +0200, Andreas Rieke wrote:
> as the memory is even gone after apache with mod_perl has been stopped,
> I do not think that this really helps. I do know that the kernel should
> recover all the memory when processes stop even if these did not free
> the memory, thus it seems that the kernel is responsible for that
> behaviour. However, I have seen these strange things on several kernels
> (2.4, 2.6) from different distributions (Redhat, SuSE) so that I can not
> believe that this is in fact a kernel problem.
I don't understand what you're saying. You don't think it's mod_perl
because stopping apache doesn't help, but you don't think it's the
kernel either? It must be something.
Get some monitoring on your system, reboot it and start fresh, and watch
what happens. See what takes the memory.
- Perrin
Re: Memory leak in mod_perl?
Posted by Andreas Rieke <an...@isl.de>.
Perrin Harkins wrote:
>On Fri, 2006-08-18 at 12:24 +0200, Andreas Rieke wrote:
>
>
>>it runs well using about 300 M of 1 G physical RAM.
>>However, the remaining RAM decreases day by day, and after 2 or 3
>>weeks, the machine crashes because swapping takes too much time.
>>However, all processes together take about 250 MBytes according to ps,
>>thus I assume that the kernel takes the rest. free tells me in fact that
>>much swap space is used an nearly no physical RAM is left.
>>
>>
>
>I think you'd better get some monitoring on that system and find out
>what's really happening. Log the size of the apache processes on a
>regular basis (or graph them with Cacti or MRTG) and see if they are the
>problem or not.
>
>- Perrin
>
>
>
Perrin,
as the memory is even gone after apache with mod_perl has been stopped,
I do not think that this really helps. I do know that the kernel should
recover all the memory when processes stop even if these did not free
the memory, thus it seems that the kernel is responsible for that
behaviour. However, I have seen these strange things on several kernels
(2.4, 2.6) from different distributions (Redhat, SuSE) so that I can not
believe that this is in fact a kernel problem.
Any more ideas?
Thanks in advance,
Andreas
Re: Memory leak in mod_perl?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
Are you using the RH / SuSE versions of mp/apache and postgres ? Or
did you compile your own?
I my experience, on those systems its best to compile your own. The
packages are often way out of date and have issues that have been
solved.
I've found that running Postgres and MP on the same machine is
annoying at best. Modperl wants tons of memory. Postgres needs tons
of memory for the main process, and quite a bit for each connection.
Since each apache child gets its own DB connection, you end up having
to tweak the semaphores and kernel params to get both working right.
( i think by default Postgres on most systems can only handle 40
connections-- if nothing else is running. )
That said, I'd be doubly sure of a few things:
a- chances are you had to tweak some kernel params to get both apps
running right. could that be a cause.
b- did you build from source or packages? source good. packages
bad ( they almost always have an exploit or leak that has been found
and fixed )
c- did you check to see if your code has anything that creates known
memory leaks in perl itself ? we all love perl, its just had some
issues in the past.
also,
d- are you running sizelimit or maxrequests to manage the growth?
250m can easily grow to 500m without an actual leak, just because
of the way memory is managed by mp.
maybe the growth is normal, and then you're leaking from all the
swapping thats going on.
personally, i've found ps and top to give me spotty numbers in
regards to mod_perl. my production box ( FreeBSD ) seems to not show
any sort of shared memory, even though the memory is shared. i can
tell, because there's not much swapping going on, and 30 processes
that claim to use 25.3% of my system memory each. numbers on my osx
box are skewed in other manners.
Re: Memory leak in mod_perl?
Posted by Perrin Harkins <pe...@elem.com>.
On Fri, 2006-08-18 at 12:24 +0200, Andreas Rieke wrote:
> it runs well using about 300 M of 1 G physical RAM.
> However, the remaining RAM decreases day by day, and after 2 or 3
> weeks, the machine crashes because swapping takes too much time.
> However, all processes together take about 250 MBytes according to ps,
> thus I assume that the kernel takes the rest. free tells me in fact that
> much swap space is used an nearly no physical RAM is left.
I think you'd better get some monitoring on that system and find out
what's really happening. Log the size of the apache processes on a
regular basis (or graph them with Cacti or MRTG) and see if they are the
problem or not.
- Perrin
Re: Memory leak in mod_perl?
Posted by Jonathan <mo...@2xlp.com>.
> IIRC, didn't we track this to Postgresql ? If so, we should
> relocate to the postgresql lists.
Not officially/definitively, though i've already posted it to the
pgsql lists. ( and ruled out bsd as being the cause thanks to some
friends with NYC BSD UG )
Re: Memory leak in mod_perl?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Fred Tyler wrote:
> Linux 2.6.12.6
> Apache 1.3.33
> Postgresql 7.4.9
> mod_perl 1.29
> 350-400 viritual hosted domains, all running a mod_perl/postgres backed
> CMS.
IIRC, didn't we track this to Postgresql ? If so, we should relocate to the postgresql lists.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
When I call your name, Girl, it starts to flame
Burning in my heart, Tearing it all apart..
No matter how I try My love I cannot hide....
Re: Memory leak in mod_perl?
Posted by Fred Tyler <fr...@gmail.com>.
> > after booting a redhat enterprise linux 3 machine with apache 2.0.58,
> > perl 5.8.8 and mod_perl 2.0.2,
> > it runs well using about 300 M of 1 G physical RAM.
> > However, the remaining RAM decreases day by day, and after 2 or 3
> > weeks, the machine crashes because swapping takes too much time.
> > However, all processes together take about 250 MBytes according to ps,
> > thus I assume that the kernel takes the rest. free tells me in fact
> > that much swap space is used an nearly no physical RAM is left.
> >
> > The strange thing is that all the memory is gone even after
> > stopping all apache processes.
> > The only thing which helps is to reboot the machine.
>
>
> I'm experiencing nearly the same thing now on
> FreeBSD 6.0 REL
> Apache/2.0.58
> mod_perl 2.0.2
> Postgres 8.1.3
I wish I'd caught this thread from the beginning. I've been seeing
this exact same problem for several months now. Here is the
environment where I first noticed it:
Linux 2.6.12.6
Apache 1.3.33
Postgresql 7.4.9
mod_perl 1.29
350-400 viritual hosted domains, all running a mod_perl/postgres backed CMS.
To make it clear: The memory is definitely lost. After 2-3 weeks on a
machine with 2GB of RAM, I can shut down *every single daemon* on the
machine and the RAM is not freed. And it is not a misreading of "top"
or "free". That memory is gone, and trying to keep the machine running
results in it going heavily into swap just for basic processes, and
eventually crashing.
It looks a lot like a kernel memory leak, and for the past few months
that is what I've assumed it was. I don't know how else to explain the
fact that no processes are running but the kernel still sees all the
RAM taken. However, the fact that it was reported on 2.4 *and* BSD
here makes me reconsider.
Andreas: Are you also running PostgreSQL as Jonathan and I are?
Andreas & Jonathan: Are either of you running reiserfs? (Perhaps it is
the filesystem causing this.)
Ok, now possibly some light at the end of the tunnel: I have set up a
second machine with the following specs:
Linux 2.6.16.27
Postgresql 7.4.13
Apache 1.3.37
mod_perl 1.29
~100 virtual hosted domains, all running mod_perl/postgres CMS
I have not seen any memory issues with this machine. When all of the
processes are stopped, the memory usage goes right back down to where
it should be. However, I don't recall seeing the memory problems on
the other machine until it hit around 300-350 vhosted domains. But
then again, I wasn't paying very much attention until it actually
became a problem. Right now I'd say it's a little too early to tell,
but in a couple months I should have ~300 domains on this box as well,
and then I'll know if this particular configuration is immune to the
memory problems.
Also, I've eliminated Apache as the cause, as the following
configuration also has the memory problem:
Linux 2.6.12.6
Apache 1.3.37
Postgresql 7.4.9
mod_perl 1.29
Perhaps this weekend I'll upgrade the postgres on this box to 7.4.13
and see if it makes a difference. If not, the only thing left is the
kernel...
Re: Memory leak in mod_perl?
Posted by Frank Wiles <fr...@wiles.org>.
On Wed, 6 Sep 2006 19:09:15 -0400
Jonathan Vanasco <mo...@2xlp.com> wrote:
> On Sep 6, 2006, at 6:41 PM, Frank Wiles wrote:
>
> > Is that 500MB that "vanished" in used, buffers, or cached? Just
> > because it isn't listed in "free" doesn't mean it isn't "free"
> > from a "Available memory I can use" standpoint. For example, your
> > system
> > will reclaim memory from cached and buffers when it runs out of
> > "free".
>
> off my original discovery, a few restarts of apache/pg and i started
> running into swap- like the original poster.
>
> i haven't been able to run memory down that low on purpose yet-- if
> you've got any clues into examining the cache/buffers , I'll gladly
> give it a go.
Well if you're running into swap then yeah, you've got a problem.
Sorry haven't been following this thread as closely as I probably
should have before commenting. :)
I'd suggest running just pg on the system for awhile and do some
stop/starts and see if you can force it to swap with just pg.
That'll remove apache/mp from the picture entirely as being a
possible source of the problem.
I'd also check to make sure you updated DBD::Pg recently as there
were a few memory leaks fixed not long ago and also to make sure it
has been compiled against the 8.1.3 libpq ( in case you did a pg
upgrade since DBD::Pg was built ).
If running with just pg doesn't force you to go into swap, if
possible try running your mp code without pg running. Again doing
the stop/starts to see if you can get into swap that way.
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: Memory leak in mod_perl?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Sep 6, 2006, at 6:41 PM, Frank Wiles wrote:
> Is that 500MB that "vanished" in used, buffers, or cached? Just
> because it isn't listed in "free" doesn't mean it isn't "free" from
> a "Available memory I can use" standpoint. For example, your
> system
> will reclaim memory from cached and buffers when it runs out of
> "free".
off my original discovery, a few restarts of apache/pg and i started
running into swap- like the original poster.
i haven't been able to run memory down that low on purpose yet-- if
you've got any clues into examining the cache/buffers , I'll gladly
give it a go.
Re: Memory leak in mod_perl?
Posted by Frank Wiles <fr...@wiles.org>.
On Wed, 6 Sep 2006 18:26:46 -0400
Jonathan Vanasco <mo...@2xlp.com> wrote:
>
> On Sep 6, 2006, at 6:01 PM, Frank Wiles wrote:
>
> > It didn't disappear, and it isn't "shared" like the type of
> > sharing we talk about with mod_perl/Apache. It's not CoW sharing.
> I know.
>
> > The shared memory you're talking about here is held by the
> > postmaster
> > daemon and is used to store frequently requested data to be used
> > by the forked children via SysV IPC.
>
> Yes. I know that. pg_stop is the stopping of the postmaster daemon
> -- not clients ( apache stop will kill the pids of the clients ).
>
> > And yeah, PostgreSQL cleans up it's shared memory when it exits
> > like it should.
>
> No, its not. thats the point of my post.
Sorry I guess I misunderstood what you were saying.
> On my system, shared memory that was allocated to postgres is not
> being returned. postgres's exit releases the lock on memory-- ipcs
> shows nothing allocated. but the kernel never gains any of that
> shared mem. i only see about the size of the resident memory of the
> postmaster instance and the clients returned.
>
> I'm inclined to think this 'leak' is in postgres and not in
> Apache/MP/ DBI, because it happens to pg via the command line
>
> the long running process where i found this behavior started off at
> about 850mb of free memory. earlier today my machine was down to
> 241mb free- a stop/wait/start of apache brought me to 317 and a
> restart of pg brought me to 321. thats about 500mb vanished.
>
> then i remembered this post, started poking around , and saw memory
> disappear and never made available in proportion to the pg vsize
> ( the clients themselves are nothing).
Is that 500MB that "vanished" in used, buffers, or cached? Just
because it isn't listed in "free" doesn't mean it isn't "free" from
a "Available memory I can use" standpoint. For example, your system
will reclaim memory from cached and buffers when it runs out of
"free".
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: Memory leak in mod_perl?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Sep 6, 2006, at 6:01 PM, Frank Wiles wrote:
> It didn't disappear, and it isn't "shared" like the type of sharing
> we talk about with mod_perl/Apache. It's not CoW sharing.
I know.
> The shared memory you're talking about here is held by the
> postmaster
> daemon and is used to store frequently requested data to be used by
> the forked children via SysV IPC.
Yes. I know that. pg_stop is the stopping of the postmaster daemon
-- not clients ( apache stop will kill the pids of the clients ).
> And yeah, PostgreSQL cleans up it's shared memory when it exits
> like
> it should.
No, its not. thats the point of my post.
On my system, shared memory that was allocated to postgres is not
being returned. postgres's exit releases the lock on memory-- ipcs
shows nothing allocated. but the kernel never gains any of that
shared mem. i only see about the size of the resident memory of the
postmaster instance and the clients returned.
I'm inclined to think this 'leak' is in postgres and not in Apache/MP/
DBI, because it happens to pg via the command line
the long running process where i found this behavior started off at
about 850mb of free memory. earlier today my machine was down to
241mb free- a stop/wait/start of apache brought me to 317 and a
restart of pg brought me to 321. thats about 500mb vanished.
then i remembered this post, started poking around , and saw memory
disappear and never made available in proportion to the pg vsize
( the clients themselves are nothing).
Re: Memory leak in mod_perl?
Posted by Frank Wiles <fr...@wiles.org>.
On Wed, 6 Sep 2006 17:36:50 -0400
Jonathan Vanasco <mo...@2xlp.com> wrote:
> From what I can tell, this is happening:
> bootup: ( pg ) 861 Free
> apache start: ( apache , pg , 3x pg clients ) 785 Free
> apache stop: ( pg, 3x pg clients ) 840 Free
> apache start: ( apache , pg , 3x pg clients ) 774 Free
> apache stop: ( pg, 3x pg clients ) 829 Free
> apache start: ( apache , pg , 3x pg clients ) 773 Free
> apache stop: ( pg, 3x pg clients ) 829 Free
>
> Note: the large drop in free memory is correlated to an
> equal increase in Pg's shared memory- so i'm not worried about that.
> Yet.
>
> pg stop: 833 free
>
> oh, wait. whats's that? i killed all those pg connections
> and no shared memory was shared? what the...
>
> ugh... lets check with ipcs?
>
> nope, no shared mem for postgres found. that stuff just
> disappeared
It didn't disappear, and it isn't "shared" like the type of sharing
we talk about with mod_perl/Apache. It's not CoW sharing.
The shared memory you're talking about here is held by the postmaster
daemon and is used to store frequently requested data to be used by
the forked children via SysV IPC.
And yeah, PostgreSQL cleans up it's shared memory when it exits like
it should.
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: Memory leak in mod_perl?
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Aug 18, 2006, at 6:24 AM, Andreas Rieke wrote:
> Hi,
>
> after booting a redhat enterprise linux 3 machine with apache 2.0.58,
> perl 5.8.8 and mod_perl 2.0.2,
> it runs well using about 300 M of 1 G physical RAM.
> However, the remaining RAM decreases day by day, and after 2 or 3
> weeks, the machine crashes because swapping takes too much time.
> However, all processes together take about 250 MBytes according to ps,
> thus I assume that the kernel takes the rest. free tells me in fact
> that
> much swap space is used an nearly no physical RAM is left.
>
> The strange thing is that all the memory is gone even after
> stopping all
> apache processes.
> The only thing which helps is to reboot the machine.
I'm experiencing nearly the same thing now on
FreeBSD 6.0 REL
Apache/2.0.58
mod_perl 2.0.2
Postgres 8.1.3
From what I can tell, this is happening:
bootup: ( pg ) 861 Free
apache start: ( apache , pg , 3x pg clients ) 785 Free
apache stop: ( pg, 3x pg clients ) 840 Free
apache start: ( apache , pg , 3x pg clients ) 774 Free
apache stop: ( pg, 3x pg clients ) 829 Free
apache start: ( apache , pg , 3x pg clients ) 773 Free
apache stop: ( pg, 3x pg clients ) 829 Free
Note: the large drop in free memory is correlated to an equal
increase in Pg's shared memory- so i'm not worried about that. Yet.
pg stop: 833 free
oh, wait. whats's that? i killed all those pg connections and no
shared memory was shared? what the...
ugh... lets check with ipcs?
nope, no shared mem for postgres found. that stuff just disappeared
fine, let's just go on with the test
free memory @ start: 773 ( apache , pg , + clients each )
free memory after 10,000 requests : 697M ( apache , pg , + clients
each )
difference : 76M
no worries! This is attributed to:
mp copy-on-write
postgres slurping memory into shared
but wait...
stop apache (pg): 762M
stop postgres () : 773 free
we now have as much free memory NOT running apache & pg as we did
when we WERE running them. thats just a coincidence that the numbers
are the same.
Note that this is a pretty bad test.
I ***THINK*** that this is all to do with pg's shared memory. I
don't have the facilities, however, to test this using MP+ mysql
( which would rule out Apache / modperl and DBI as being the
culprit ), or another postgres method.
So, in regards to the original poster:
a- I think you're going into swap, because your Postgres config
doesn't have a limit on memory Perhaps your apache too. Check
that. Load a ton of crap into Apache before the fork, and set max-
requests to 500 or so. That should severely limit your memory use.
b- I think only reboots work to reclaim memory, because there seems
to be an oddity with Postgres releasing shared memory.
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: Memory leak in mod_perl?
Posted by Andreas Rieke <an...@isl.de>.
Clinton Gormley wrote:
>On Fri, 2006-08-18 at 12:24 +0200, Andreas Rieke wrote:
>
>
>>Hi,
>>
>>
>>
>
>
>
>>The strange thing is that all the memory is gone even after stopping all
>>apache processes.
>>The only thing which helps is to reboot the machine.
>>
>>
>>
>
>Hi Andreas
>
>Not my field of expertise, but this sounds like a kernel problem, or a
>problem of some other process running on the machine, maybe Postgres?
>
>I had exactly the same scenario once many years ago, using RedHat 6, and
>then it was a kernel problem. but as you say, you've tried 2 distinctly
>different kernels without any issues...
>
>so i would look at other processes for problems.
>
>good luck
>
>Clint
>
>
>
Clint,
thanks for your suggestion, but there are no other processes, only those
belonging to the operating system. Using ps, I can calculate that they
are in the maximum using about 200 MBytes (and they really do so after
rebooting), but the machine has 1 Gigybyte of RAM and free tells me that
nearly no memory is left (there is also nearly no memory cached and
buffered). I also tried ipcs to find out if there is any shared memory,
but it isn't. For that reason, the only one who is able to allocate
memory seems to be the kernel, but as you already stated, it seems quite
strange that the problem has been seen on several (old and new) kernels.
Any more ideas?
Thanks in advance,
Andreas
Re: Memory leak in mod_perl?
Posted by Clinton Gormley <cl...@traveljury.com>.
On Fri, 2006-08-18 at 12:24 +0200, Andreas Rieke wrote:
> Hi,
>
> The strange thing is that all the memory is gone even after stopping all
> apache processes.
> The only thing which helps is to reboot the machine.
>
Hi Andreas
Not my field of expertise, but this sounds like a kernel problem, or a
problem of some other process running on the machine, maybe Postgres?
I had exactly the same scenario once many years ago, using RedHat 6, and
then it was a kernel problem. but as you say, you've tried 2 distinctly
different kernels without any issues...
so i would look at other processes for problems.
good luck
Clint