You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Marco Rolappe <m_...@web.de> on 2003/03/20 19:14:05 UTC
[HELP] determining possible leak
hi group,
there's a problem I currently have with a webapp, that makes me go crazy.
the webapp has been live for about two months now, and I think the problem
didn't exist in the beginning.
the problem itself seems to be some leak.
config:
- linux box (dual processor)
- tomcat 4.1.18
- cocoon 2.0.4
symptoms:
- webapp runs fine for hours (12-24 hours) with a max. cpu usage of about
4%
- memory usage is continuously rising
- at some point cpu usage continuously around 100%, distributed to the
processors
strange points:
- though having specified java option -Xmx128 the memory usage rises above
that value (when viewed in top)
- discrepancy between memory usage shown by ps/top and memory usage logged
by the store-janitor
- in the store-janitors's logs the 'JVM total memory' is never above the
maximum specified via -Xmx
- according to the store-janitor's logs the webapp (cocoon) always has
enough memory (> 15mb)
can anyone explain
- the discrepancy between the memory usage reported by the store-janitor
and ps/top?
- where the leak may be? (from the janitor's values I assume it's not
within the JVM but within the java process?)
- how to determine the kind of leak/where the leak may be
finally, a snip from the janitor's log. please note that top reports a
memory usage of about 223MB. -Xmx is -Xmx196. cpu usage is currently normal.
but this maybe because the JVM's heap size hasn't reached its maximum yet
(according to the janitor).
...
DEBUG (2003-03-20) 19:01.15:335 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM total Memory: 157827072
DEBUG (2003-03-20) 19:01.15:336 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM free Memory: 16551888
DEBUG (2003-03-20) 19:01.15:336 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: Memory is low = false
DEBUG (2003-03-20) 19:01.25:349 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM total Memory: 157827072
DEBUG (2003-03-20) 19:01.25:349 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM free Memory: 15666808
DEBUG (2003-03-20) 19:01.25:350 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: Memory is low = false
DEBUG (2003-03-20) 19:01.35:369 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM total Memory: 157827072
DEBUG (2003-03-20) 19:01.35:370 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM free Memory: 18108320
DEBUG (2003-03-20) 19:01.35:370 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: Memory is low = false
DEBUG (2003-03-20) 19:01.45:389 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM total Memory: 157827072
DEBUG (2003-03-20) 19:01.45:390 [core.store.janitor] (Unknown-URI)
Unknown-t
hread/StoreJanitorImpl: JVM free Memory: 17840184
...
Re: [HELP] determining possible leak
Posted by Berin Loritsch <bl...@apache.org>.
Marco Rolappe wrote:
> hi group,
>
> there's a problem I currently have with a webapp, that makes me go crazy.
>
> the webapp has been live for about two months now, and I think the problem
> didn't exist in the beginning.
>
>
> the problem itself seems to be some leak.
Which JVM are you using? Some people have confirmed that the
StringBuffer class in JDK 1.4.1 has a memory leak.
AW: [HELP] determining possible leak
Posted by Marco Rolappe <m_...@web.de>.
thanx for your tips christopher, pier and berin,
1. fortunately it wasn't my webapp that had the problem. I tried isolating
the webapps and then it showed up. problem solved - for me.
2. it is (I think) a StringBuffer issue. I infer this from the thread dump I
get when -QUIT killing the buggy webapp's tomcat; threads are shown blocked
waiting for StringBuffers' capacities to be expanded.
3. the buggy webapp is a (supposedly) simple static wml/jsp webapp. I read
about a memory leak regarding JSP compilation, but I don't think that one
applies here.
> -----Ursprungliche Nachricht-----
> Von: cocoon-dev-return-38872-m_rolappe=web.de@xml.apache.org
> [mailto:cocoon-dev-return-38872-m_rolappe=web.de@xml.apache.org]Im
> Auftrag von Christopher Watson
> Gesendet: Donnerstag, 20. Marz 2003 21:49
> An: cocoon-dev@xml.apache.org
> Betreff: RE: [HELP] determining possible leak
>
>
> Marco
>
> At risk of a red herring ...
>
> I had what sounds a very similar problem with tomcat 4.1.12 and cocoon
> 2.0.3 and sun jdk1.3 on w2k
> The behaviour of
> OK for 24hrs then 100%cpu - with no apparent application errors
> is identical to what I saw
>
> I looked on tomcat mailing lists and found someone else had similar
> problem which stopped when they went to jdk1.4.1
> I did same and problem went away !
>
> Can't speak for jdk1.4.0 since by the time I moved off jdk1.3, jdk1.4.1
> was out...
>
> But 100%CPU and no application error messages does 'seem' the kind of
> thing that might happen with bad garbage collection due to a rogue JVM.
> So if it doesn't create additional hassle perhaps jdk1.4.1 is worth a
> try ?
>
> Christopher Watson
>
> > -----Original Message-----
> > From: m_rolappe@web.de [mailto:m_rolappe@web.de]
> > Sent: 20 March 2003 19:26
> > To: cocoon-dev@xml.apache.org
> > Subject: AW: [HELP] determining possible leak
> >
> >
> > JDK version is:
> >
> > java version "1.4.0_02"
> > Java(TM) 2 Runtime Environment, Standard Edition (build
> > 1.4.0_02-b02) Java HotSpot(TM) Client VM (build 1.4.0_02-b02,
> > mixed mode)
> >
> > but as already mentioned I think the problem didn't exist in
> > the beginning on the live machine (but maybe it didn't have
> > the time to come up, since in the beginning I set -Xmx256m).
> >
> > but even if it's that StringBuffer issue, wouldn't that show
> > up in the janitor's memory reports?
> >
> > tnx
> >
> > > -----Ursprungliche Nachricht-----
> > > Von: cocoon-dev-return-38861-m_rolappe=web.de@xml.apache.org
> > > [mailto:cocoon-dev-return-38861-m_rolappe=web.de@xml.apache.org]Im
> > > Auftrag von Gianugo Rabellino
> > > Gesendet: Donnerstag, 20. Marz 2003 19:26
> > > An: cocoon-dev@xml.apache.org
> > > Betreff: Re: [HELP] determining possible leak
> > >
> > >
> > > Marco Rolappe wrote:
> > > > hi group,
> > > >
> > > > there's a problem I currently have with a webapp, that makes me
> > > go crazy.
> > > >
> > > > the webapp has been live for about two months now, and I think
> > > the problem
> > > > didn't exist in the beginning.
> > >
> > > Using JDK 1.4.1? If so, look for the "profiling Cocoon" recent
> > > threads: there is a bug in StringBuffer that leads to a memory leak.
> > >
> > > Ciao,
> > >
> > > --
> > > Gianugo Rabellino
> > > Pro-netics s.r.l.
> > > http://www.pro-netics.com
> > >
> >
> >
>
RE: [HELP] determining possible leak
Posted by Christopher Watson <c....@zen.co.uk>.
Marco
At risk of a red herring ...
I had what sounds a very similar problem with tomcat 4.1.12 and cocoon
2.0.3 and sun jdk1.3 on w2k
The behaviour of
OK for 24hrs then 100%cpu - with no apparent application errors
is identical to what I saw
I looked on tomcat mailing lists and found someone else had similar
problem which stopped when they went to jdk1.4.1
I did same and problem went away !
Can't speak for jdk1.4.0 since by the time I moved off jdk1.3, jdk1.4.1
was out...
But 100%CPU and no application error messages does 'seem' the kind of
thing that might happen with bad garbage collection due to a rogue JVM.
So if it doesn't create additional hassle perhaps jdk1.4.1 is worth a
try ?
Christopher Watson
> -----Original Message-----
> From: m_rolappe@web.de [mailto:m_rolappe@web.de]
> Sent: 20 March 2003 19:26
> To: cocoon-dev@xml.apache.org
> Subject: AW: [HELP] determining possible leak
>
>
> JDK version is:
>
> java version "1.4.0_02"
> Java(TM) 2 Runtime Environment, Standard Edition (build
> 1.4.0_02-b02) Java HotSpot(TM) Client VM (build 1.4.0_02-b02,
> mixed mode)
>
> but as already mentioned I think the problem didn't exist in
> the beginning on the live machine (but maybe it didn't have
> the time to come up, since in the beginning I set -Xmx256m).
>
> but even if it's that StringBuffer issue, wouldn't that show
> up in the janitor's memory reports?
>
> tnx
>
> > -----Ursprungliche Nachricht-----
> > Von: cocoon-dev-return-38861-m_rolappe=web.de@xml.apache.org
> > [mailto:cocoon-dev-return-38861-m_rolappe=web.de@xml.apache.org]Im
> > Auftrag von Gianugo Rabellino
> > Gesendet: Donnerstag, 20. Marz 2003 19:26
> > An: cocoon-dev@xml.apache.org
> > Betreff: Re: [HELP] determining possible leak
> >
> >
> > Marco Rolappe wrote:
> > > hi group,
> > >
> > > there's a problem I currently have with a webapp, that makes me
> > go crazy.
> > >
> > > the webapp has been live for about two months now, and I think
> > the problem
> > > didn't exist in the beginning.
> >
> > Using JDK 1.4.1? If so, look for the "profiling Cocoon" recent
> > threads: there is a bug in StringBuffer that leads to a memory leak.
> >
> > Ciao,
> >
> > --
> > Gianugo Rabellino
> > Pro-netics s.r.l.
> > http://www.pro-netics.com
> >
>
>
AW: [HELP] determining possible leak
Posted by Marco Rolappe <m_...@web.de>.
JDK version is:
java version "1.4.0_02"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0_02-b02)
Java HotSpot(TM) Client VM (build 1.4.0_02-b02, mixed mode)
but as already mentioned I think the problem didn't exist in the beginning
on the live machine (but maybe it didn't have the time to come up, since in
the beginning I set -Xmx256m).
but even if it's that StringBuffer issue, wouldn't that show up in the
janitor's memory reports?
tnx
> -----Ursprungliche Nachricht-----
> Von: cocoon-dev-return-38861-m_rolappe=web.de@xml.apache.org
> [mailto:cocoon-dev-return-38861-m_rolappe=web.de@xml.apache.org]Im
> Auftrag von Gianugo Rabellino
> Gesendet: Donnerstag, 20. Marz 2003 19:26
> An: cocoon-dev@xml.apache.org
> Betreff: Re: [HELP] determining possible leak
>
>
> Marco Rolappe wrote:
> > hi group,
> >
> > there's a problem I currently have with a webapp, that makes me
> go crazy.
> >
> > the webapp has been live for about two months now, and I think
> the problem
> > didn't exist in the beginning.
>
> Using JDK 1.4.1? If so, look for the "profiling Cocoon" recent threads:
> there is a bug in StringBuffer that leads to a memory leak.
>
> Ciao,
>
> --
> Gianugo Rabellino
> Pro-netics s.r.l.
> http://www.pro-netics.com
>
Re: [HELP] determining possible leak
Posted by Gianugo Rabellino <gi...@apache.org>.
Marco Rolappe wrote:
> hi group,
>
> there's a problem I currently have with a webapp, that makes me go crazy.
>
> the webapp has been live for about two months now, and I think the problem
> didn't exist in the beginning.
Using JDK 1.4.1? If so, look for the "profiling Cocoon" recent threads:
there is a bug in StringBuffer that leads to a memory leak.
Ciao,
--
Gianugo Rabellino
Pro-netics s.r.l.
http://www.pro-netics.com
AW: [HELP] determining possible leak
Posted by Marco Rolappe <m_...@web.de>.
> -----Ursprungliche Nachricht-----
> Von: cocoon-dev-return-38868-m_rolappe=web.de@xml.apache.org
> [mailto:cocoon-dev-return-38868-m_rolappe=web.de@xml.apache.org]Im
> Auftrag von Pier Fumagalli
> Gesendet: Donnerstag, 20. Marz 2003 20:37
> An: cocoon-dev@xml.apache.org
> Betreff: Re: [HELP] determining possible leak
>
>
> > symptoms:
> > - webapp runs fine for hours (12-24 hours) with a max. cpu
> usage of about
> > 4%
> > - memory usage is continuously rising
> > - at some point cpu usage continuously around 100%, distributed to the
> > processors
>
> Nothing in the logs? No output to standard output/error? Do you see an
> OutOfMemoryException?
no OutOfMemoryException.
>
> > can anyone explain
> > - the discrepancy between the memory usage reported by the store-janitor
> > and ps/top?
>
> Well, what you set by -Xmx is how much memory the JVM will
> allocate for JAVA
> objects... Top reports how much memory the JVM actually occupies (the JVM
> itself PLUS the Java Heap)...
yes, I know, the maximum heap size. but I thought the JVM's consumption
would be fixed size; if I set -Xmx128m and get the mem usage rising higher
than 200MB I assume it's not the memory the JVM needs itself. or is it?
> > - where the leak may be? (from the janitor's values I assume it's not
> > within the JVM but within the java process?)
>
> Are you sure it's a memory leak? How can you be sure of that? The JVM will
> grow slowly to basically occupy all its heap, but will never release back
> its "unused" memory to the underlying OS...
I was not sure, it was my best guess ;-) as mentioned after a fresh start
everything is fine. and the only constantly changing factor directly visible
to me was the memory usage. and the high cpu usage only occuring at high mem
usage. strange though that the cpu usage wouldn't decrease afterwards until
the whole webapp was stopped.
another thing I remember now; before shutting tomcat completely down I tried
it with restarting the context. but to no avail; instead of going back to
normal/decreasing another fixed amount of memory (about 5MB) would be gone
with the wind instantly. the same with the second and third context restart.
> > finally, a snip from the janitor's log. please note that top reports a
> > memory usage of about 223MB. -Xmx is -Xmx196. cpu usage is
> currently normal.
> > but this maybe because the JVM's heap size hasn't reached its
> maximum yet
> > (according to the janitor).
>
> Yes, there's nothing wrong with this log... If you set the VM to run with
> -Xms196 -Xmx196, you'll have a VM which after initialization should not
> claim any more memory from the OS... You should first use this to
> see if you
> actually have a memory leak or not. And then only use the output from the
> store janitor.
will try if the problem shows up again. for some strange reason everything
seems currently normal. murphy's law I guess ;-)
anyway, tnx.
Re: [HELP] determining possible leak
Posted by Pier Fumagalli <pi...@betaversion.org>.
"Marco Rolappe" <m_...@web.de> wrote:
> symptoms:
> - webapp runs fine for hours (12-24 hours) with a max. cpu usage of about
> 4%
> - memory usage is continuously rising
> - at some point cpu usage continuously around 100%, distributed to the
> processors
Nothing in the logs? No output to standard output/error? Do you see an
OutOfMemoryException?
> can anyone explain
> - the discrepancy between the memory usage reported by the store-janitor
> and ps/top?
Well, what you set by -Xmx is how much memory the JVM will allocate for JAVA
objects... Top reports how much memory the JVM actually occupies (the JVM
itself PLUS the Java Heap)...
> - where the leak may be? (from the janitor's values I assume it's not
> within the JVM but within the java process?)
Are you sure it's a memory leak? How can you be sure of that? The JVM will
grow slowly to basically occupy all its heap, but will never release back
its "unused" memory to the underlying OS...
> finally, a snip from the janitor's log. please note that top reports a
> memory usage of about 223MB. -Xmx is -Xmx196. cpu usage is currently normal.
> but this maybe because the JVM's heap size hasn't reached its maximum yet
> (according to the janitor).
Yes, there's nothing wrong with this log... If you set the VM to run with
-Xms196 -Xmx196, you'll have a VM which after initialization should not
claim any more memory from the OS... You should first use this to see if you
actually have a memory leak or not. And then only use the output from the
store janitor.
Pier