You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Donald Armstrong <do...@gmail.com> on 2010/08/11 18:15:20 UTC
JreMemoryLeakPreventionListener and hourly Full GC
Hello,
We have recently deployed tomcat-6.0.28 in our organization and are
noticing every hour, a Full GC is occurring. The same application,
same JVM, same JVM args, just a new tomcat release.
I have addressed the issue by adjusting the server.xml and disabling
gcDaemonProtection.
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
gcDaemonProtection="false"/>
Ideas on what is causing the Full GC and why every hour?
our environment:
tomcat-6.0.28
java.vm.specification.vendor:Sun Microsystems Inc.
java.runtime.version:1.6.0_18-b07
sun.arch.data.model:64
os.version:5.10
os.arch:sparcv9
GC.log
212621.426: [Full GC (System) 212621.427:
[CMS:81164K->68018K(8005248K), 4.0688219 secs]
240526K->68018K(8350272K),[CMS Perm : 91877K->91830K(131072K)],
4.0694842 secs] [Times:user=4.08 sys=0.01, real=4.07 secs]
216225.566: [Full GC (System) 216225.567:
[CMS:82629K->64655K(8005248K), 3.9778052 secs]
210097K->64655K(8350272K),[CMS Perm : 91874K->91837K(131072K)],
3.9784728 secs] [Times:
user=3.98 sys=0.00, real=3.98 secs]
219829.596: [Full GC (System) 219829.597:
[CMS:87605K->73336K(8005248K), 4.2279768 secs]
313716K->73336K(8350272K),[CMS Perm : 91930K->91841K(131072K)],
4.2286745 secs] [Times:
user=4.23 sys=0.00, real=4.23 secs]
Thanks,
Donnie
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JreMemoryLeakPreventionListener and hourly Full GC
Posted by do...@gmail.com.
Chris you must have misread. Do not blame yourself, my op was not as clear as it should have been. Apologies
Sent from my Verizon Wireless BlackBerry
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Fri, 13 Aug 2010 16:02:36
To: Tomcat Users List<us...@tomcat.apache.org>
Reply-To: "Tomcat Users List" <us...@tomcat.apache.org>
Subject: Re: JreMemoryLeakPreventionListener and hourly Full GC
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Donald,
On 8/12/2010 5:04 PM, Donald Armstrong wrote:
> 3) <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
> gcDaemonProtection="false"/>
[snip]
> We've decided to go with option 3.
I'm confused... I thought you tried that and you were still seeing
hourly GCs. Did I misread?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkxlpNwACgkQ9CaO5/Lv0PC++gCcDlOxLpOWR1G6agyGjtdxG7UD
IaoAnRQkJQpND07ToEqZFGHTO6eqFzG/
=qIle
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JreMemoryLeakPreventionListener and hourly Full GC
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Donald,
On 8/12/2010 5:04 PM, Donald Armstrong wrote:
> 3) <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
> gcDaemonProtection="false"/>
[snip]
> We've decided to go with option 3.
I'm confused... I thought you tried that and you were still seeing
hourly GCs. Did I misread?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkxlpNwACgkQ9CaO5/Lv0PC++gCcDlOxLpOWR1G6agyGjtdxG7UD
IaoAnRQkJQpND07ToEqZFGHTO6eqFzG/
=qIle
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JreMemoryLeakPreventionListener and hourly Full GC
Posted by Donald Armstrong <do...@gmail.com>.
Thank you Konstantin and Chris for your attention.
As stated in the initial post:
'We have recently deployed tomcat-6.0.28 in our organization and are
noticing every hour, a Full GC is occurring. The same application,
same JVM, same JVM args, just a new tomcat release.'
Using the default JreMemoryLeakPreventionListener configuration that
has 'gcDaemonProtection=true' will result in 1hr FullGCs using Sun
1.6 b18, b20 and b21on Solaris and Windows. We've tested and
successfully 'contained' the FullGC behavior using one of the below
configurations:
1) suppress the FullGC using JVM arg -XX:+DisableExplicitGC
2) keep the FullGC but to defer to the CMS collector using JVM arg
-XX:+ExplicitGCInvokesConcurrent
3) <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
gcDaemonProtection="false"/>
4) Disable the listener altogether
We've decided to go with option 3.
Thanks,
Donnie
On Thu, Aug 12, 2010 at 9:15 AM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Konstantin,
>
> On 8/12/2010 7:48 AM, Konstantin Kolinko wrote:
>> It looks that with your version of JRE and with your setting it is
>> better to run with gcDaemonProtection="false". I certainly do not
>> know how other JRE versions behave here. (Feedback is welcomed).
>
> That's what he's got, now: gcDaemonProtection="false" (see the OP), but
> the GC is still happening.
>
> Donald, the reason the JreMemoryLeakPreventionListener makes the call to
> GC.requestLatency is to un-do the effect of some other component (RMI,
> it seems).
>
> It's possible that some component other than the
> JreMemoryLeakPreventionListener is causing the periodic full GC.
>
> Try disabling the listener altogether and see if the GCs go away.
>
> If they don't, consider recompiling the JreMemoryLeakPreventionListener
> with some debugging code in there (the current code does not emit any
> logging unless there is a problem calling GC.requestLatency... it does
> not announce it's intention of doing so, even at a TRACE level).
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkxkAfAACgkQ9CaO5/Lv0PBgjACgkeQyW4H949HJ/k+28hjKR3JH
> 9tsAoLIxzWfypWAhR9PzC4IQpGFF0Kwy
> =5GDP
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JreMemoryLeakPreventionListener and hourly Full GC
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Konstantin,
On 8/12/2010 7:48 AM, Konstantin Kolinko wrote:
> It looks that with your version of JRE and with your setting it is
> better to run with gcDaemonProtection="false". I certainly do not
> know how other JRE versions behave here. (Feedback is welcomed).
That's what he's got, now: gcDaemonProtection="false" (see the OP), but
the GC is still happening.
Donald, the reason the JreMemoryLeakPreventionListener makes the call to
GC.requestLatency is to un-do the effect of some other component (RMI,
it seems).
It's possible that some component other than the
JreMemoryLeakPreventionListener is causing the periodic full GC.
Try disabling the listener altogether and see if the GCs go away.
If they don't, consider recompiling the JreMemoryLeakPreventionListener
with some debugging code in there (the current code does not emit any
logging unless there is a problem calling GC.requestLatency... it does
not announce it's intention of doing so, even at a TRACE level).
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkxkAfAACgkQ9CaO5/Lv0PBgjACgkeQyW4H949HJ/k+28hjKR3JH
9tsAoLIxzWfypWAhR9PzC4IQpGFF0Kwy
=5GDP
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JreMemoryLeakPreventionListener and hourly Full GC
Posted by Konstantin Kolinko <kn...@gmail.com>.
2010/8/12 Donald Armstrong <do...@gmail.com>:
> The documentation does not indicate if the
> JreMemoryLeakPreventionListener should be enabled for production
> environments, but based on this information, probably not?
>
I think you are too fast jumping to conclusions.
BTW, even if you do not see JavaDoc here
http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/catalina/core/JreMemoryLeakPreventionListener.html
the sources are always available,
http://svn.apache.org/viewvc/tomcat/tc6.0.x/trunk/java/org/apache/catalina/core/JreMemoryLeakPreventionListener.java?view=markup
JreMemoryLeakPreventionListener is a combination of several
independent tricks to prevent certain known memory leak problems.
Each of those tricks can be enabled or disabled independently.
It looks that with your version of JRE and with your setting it is
better to run with gcDaemonProtection="false". I certainly do not
know how other JRE versions behave here. (Feedback is welcomed).
> Thank you for your opinion.
>
> Donnie
>
By the way, there is a page in the wiki regarding the memory leaks
protection. Additions are welcomed there as well.
http://wiki.apache.org/tomcat/MemoryLeakProtection
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JreMemoryLeakPreventionListener and hourly Full GC
Posted by Donald Armstrong <do...@gmail.com>.
Thanks for your attention chris,
We've identified why every hour: sun.misc.GC.requestLatency(3600000l)
I was able to confirm with the tomcat documentation that with
gcDaemonProtection enabled, the "GC Daemon" thread is running.
We've identified why the Full GC; I've looked at the decompiled code
for the sun.misc.GC class and there is a System.gc() call buried in
there. It is hard to read, but likely here the Full GC is happening.
{
synchronized(GC.lock)
{
l = GC.latencyTarget;
if(l != 9223372036854775807L)
break label0;
GC.daemon = null;
}
return;
}
long l1 = GC.maxObjectInspectionAge();
if(l1 >= l)
{
System.gc();
l1 = 0L;
}
The documentation does not indicate if the
JreMemoryLeakPreventionListener should be enabled for production
environments, but based on this information, probably not?
Thank you for your opinion.
Donnie
On Wed, Aug 11, 2010 at 12:34 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Donald,
>
> On 8/11/2010 12:15 PM, Donald Armstrong wrote:
>> We have recently deployed tomcat-6.0.28 in our organization and are
>> noticing every hour, a Full GC is occurring. The same application,
>> same JVM, same JVM args, just a new tomcat release.
>>
>> I have addressed the issue by adjusting the server.xml and disabling
>> gcDaemonProtection.
>
> Why?
>
>> Ideas on what is causing the Full GC and why every hour?
>
> There is no javadoc for this class which is unfortunate, but the
> documentation in
> http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html is likely
> where you found this configuration option.
>
> Here's what it says about gcDaemonProtection:
>
> "
> Enables protection so that calls to sun.misc.GC.requestLatency(long)
> triggered by a web application do not result in a memory leak. Use of
> RMI is likely to trigger a call to this method. A side effect of
> enabling this protection is the creation of a thread named "GC Daemon".
> The protection is uses reflection to access internal Sun classes and may
> generate errors on startup on non-Sun JVMs. The default is true.
> "
>
> Looking at the code for JreMemoryLeakPreventionListener, no threads are
> fired from that code. The code in there calls the static method
> sun.misc.GC.requestLatency(3600000l) - that's one hour, if the argument
> is in ms. I'm having trouble following the decompiled code, but I can't
> find anywhere that a Thread is actually started.
>
> Have you tried to take a thread dump to see what threads are running?
> Presumably, you ought to see something interesting in there. Are you
> able to run a profiler against the process to see where the thread is
> created?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkxi3w4ACgkQ9CaO5/Lv0PAYBQCfe7+POwLPcW6p9QQyWlLbSspZ
> Z+EAn1H1FrU2QQ40fUiywug7Ohz2fpP7
> =GC7n
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: JreMemoryLeakPreventionListener and hourly Full GC
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Donald,
On 8/11/2010 12:15 PM, Donald Armstrong wrote:
> We have recently deployed tomcat-6.0.28 in our organization and are
> noticing every hour, a Full GC is occurring. The same application,
> same JVM, same JVM args, just a new tomcat release.
>
> I have addressed the issue by adjusting the server.xml and disabling
> gcDaemonProtection.
Why?
> Ideas on what is causing the Full GC and why every hour?
There is no javadoc for this class which is unfortunate, but the
documentation in
http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html is likely
where you found this configuration option.
Here's what it says about gcDaemonProtection:
"
Enables protection so that calls to sun.misc.GC.requestLatency(long)
triggered by a web application do not result in a memory leak. Use of
RMI is likely to trigger a call to this method. A side effect of
enabling this protection is the creation of a thread named "GC Daemon".
The protection is uses reflection to access internal Sun classes and may
generate errors on startup on non-Sun JVMs. The default is true.
"
Looking at the code for JreMemoryLeakPreventionListener, no threads are
fired from that code. The code in there calls the static method
sun.misc.GC.requestLatency(3600000l) - that's one hour, if the argument
is in ms. I'm having trouble following the decompiled code, but I can't
find anywhere that a Thread is actually started.
Have you tried to take a thread dump to see what threads are running?
Presumably, you ought to see something interesting in there. Are you
able to run a profiler against the process to see where the thread is
created?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkxi3w4ACgkQ9CaO5/Lv0PAYBQCfe7+POwLPcW6p9QQyWlLbSspZ
Z+EAn1H1FrU2QQ40fUiywug7Ohz2fpP7
=GC7n
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org