You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by "Janko Heilgeist (JIRA)" <ji...@apache.org> on 2009/02/19 07:55:02 UTC
[jira] Created: (GERONIMO-4543) EAR classloader not garbage
collected
EAR classloader not garbage collected
-------------------------------------
Key: GERONIMO-4543
URL: https://issues.apache.org/jira/browse/GERONIMO-4543
Project: Geronimo
Issue Type: Bug
Security Level: public (Regular issues)
Components: transaction manager
Affects Versions: 2.2
Reporter: Janko Heilgeist
The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Janko Heilgeist (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Janko Heilgeist updated GERONIMO-4543:
--------------------------------------
Attachment: privileged_currenttime.patch
This patch starts the CurrentTime thread in a privileged block to prevent it from inheriting the AccessControlContext of the current thread.
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Kevan Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675399#action_12675399 ]
Kevan Miller commented on GERONIMO-4543:
----------------------------------------
Do you also need to specify -J-XX:+CMSPermGenSweepingEnabled? I don't run with -XX:+UseConcMarkSweepGC. So, I'm not very familiar with these sets of options.
Thanks for the additional info. That helps. I ran some tests with:
JAVA_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError"
And didn't see any major problems through multiple deploy/undeploy cycles. EAR/WAR ClassLoaders (and their Classes) were definitely being GC'ed. So, I don't see a systemic problem. Quite possibly we're losing the first ClassLoader. Also, possible that we're temporarily holding onto ClassLoader's through ThreadLocals. Will have to dig a bit further with a finer tooth comb. May be a few days...
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Memory Leaks, transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Priority: Blocker
> Fix For: 2.2
>
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Janko Heilgeist (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674936#action_12674936 ]
Janko Heilgeist commented on GERONIMO-4543:
-------------------------------------------
There is still another reference from CurrentTime to my EAR's classloader through the inheritableThreadLocals field:
{code}
Java Local Reference (from org.apache.geronimo.transaction.manager.TransactionTimer$CurrentTime@0x2aaab1e8cd68) exclude :
--> org.apache.geronimo.transaction.manager.TransactionTimer$CurrentTime@0x2aaab1e8cd68 (164 bytes) (field inheritableThreadLocals:) exclude
--> java.lang.ThreadLocal$ThreadLocalMap@0x2aaab2485938 (32 bytes) (field table:) exclude
--> [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@0x2aaab2454dc0 (144 bytes) (Element 1 of [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@0x2aaab2454dc0:) exclude
--> java.lang.ThreadLocal$ThreadLocalMap$Entry@0x2aaab2fe39e8 (56 bytes) (field value:) exclude
--> org.apache.xbean.naming.context.WritableContext$NestedWritableContext@0x2aaab055f088 (89 bytes) (field bindingsRef:) exclude
--> java.util.concurrent.atomic.AtomicReference@0x2aaab2031940 (24 bytes) (field value:) exclude
--> java.util.HashMap@0x2aaab22a0040 (64 bytes) (field table:) exclude
--> [Ljava.util.HashMap$Entry;@0x2aaab2031c00 (144 bytes) (Element 3 of [Ljava.util.HashMap$Entry;@0x2aaab2031c00:) exclude
--> java.util.HashMap$Entry@0x2aaab20322d0 (44 bytes) (field value:) exclude
--> org.apache.xbean.naming.context.WritableContext$NestedWritableContext@0x2aaab055f1a8 (89 bytes) (field contextFederation:) exclude
--> org.apache.xbean.naming.context.ContextFederation@0x2aaab0a133e8 (32 bytes) (field federatedContextRef:) exclude
--> java.util.concurrent.atomic.AtomicReference@0x2aaab2031a30 (24 bytes) (field value:) exclude
--> java.util.Collections$UnmodifiableSet@0x2aaab2031a48 (24 bytes) (field c:) exclude
--> java.util.LinkedHashSet@0x2aaab2031a60 (24 bytes) (field map:) exclude
--> java.util.LinkedHashMap@0x2aaab1f2a660 (73 bytes) (field header:) exclude
--> java.util.LinkedHashMap$Entry@0x2aaab22a0140 (60 bytes) (field after:) exclude
--> java.util.LinkedHashMap$Entry@0x2aaab22a0180 (60 bytes) (field key:) exclude
--> org.apache.xbean.naming.context.ImmutableContext@0x2aaab1f2a918 (65 bytes) (field absoluteIndex:) exclude
--> java.util.Collections$UnmodifiableMap@0x2aaab2032600 (48 bytes) (field m:) exclude
--> java.util.HashMap@0x2aaab0a13568 (64 bytes) (field table:) exclude
--> [Ljava.util.HashMap$Entry;@0x2aaab2031d30 (144 bytes) (Element 15 of [Ljava.util.HashMap$Entry;@0x2aaab2031d30:) exclude
--> java.util.HashMap$Entry@0x2aaab2032690 (44 bytes) (field value:) exclude
--> org.apache.geronimo.naming.reference.HandleDelegateReference@0x2aaab1f2a6b0 (80 bytes) (field classLoader:) exclude
--> org.apache.geronimo.kernel.config.MultiParentClassLoader@0x2aaab2fbc000 (156 bytes) exclude
{code}
I assume, that it is the inheritable component context from org.apache.geronimo.naming.java.RootContext. But I can't find a way to fix this...
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Janko Heilgeist (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675265#action_12675265 ]
Janko Heilgeist commented on GERONIMO-4543:
-------------------------------------------
Hi Kevan,
I've set my JAVA_OPTS to "-XX:MaxPermSize=265m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled" for some time now and never saw any OOM errors. But last Friday I must have accidentally reset the environment variable somehow without noticing it. On Monday I started seeing the OOM PermGen space errors immediately during first deployment and began investigating the cause. I didn't recognize the missing JAVA_OPTS and instead debugged the application with jmap/jhat. Thus, I noticed that the classes of my EAR never get GCed after undeploying and that got me hooked up on the issue. Shortly after, I switched back to the old JAVA_OPTS and everything worked again. But its just my own private Geronimo installation for development purposes that gets restarted a few times a day. So the OOM may be pushed back long enough to never occur.
So much for the background. I currently just deploy/undeploy my app and don't actually drive it. I already noticed a few additional issues when I enable an included Quartz scheduled job and when I call my JUnitE2 test cases. For now, I've disabled them and focus just on getting the app GCed after a single deploy/undeploy. I also noticed that the PermGen space increases minimally (that is a few KB maybe) with each deploy/undeploy cycle. It's not a serious issue yet but I'll investigate it further if it continues.
I've only seen the EAR's MultiParentClassLoader, its child WAR's MultiParentClassLoader, and an intermediate ChildConfigurationClassLoader not being GCed. SUN's Frank Kieviet describes a servlet in his blog, that repeatedly loads a class with its own classloader to force a PermGen space GC. Thus, the relevant classloaders should have been GCed if they had been eligible.
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Memory Leaks, transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Priority: Blocker
> Fix For: 2.2
>
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Janko Heilgeist (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Janko Heilgeist updated GERONIMO-4543:
--------------------------------------
Attachment: ear-with-tx.tar.gz
This EAR packages an EJB-jar and a WAR. The EJB exports an empty transactional method which the WAR's servlet accesses during deployment. After deploying/undeploying the EAR there are still live connections to the rootset:
{code}
Java Local Reference (from org.apache.geronimo.transaction.manager.TransactionTimer$CurrentTime@0x2aaab30aaed8) exclude :
--> org.apache.geronimo.transaction.manager.TransactionTimer$CurrentTime@0x2aaab30aaed8 (164 bytes) (field inheritedAccessControlContext:) exclude
--> java.security.AccessControlContext@0x2aaab30aafb0 (41 bytes) (field context:) exclude
--> [Ljava.security.ProtectionDomain;@0x2aaab30aba00 (136 bytes) (Element 14 of [Ljava.security.ProtectionDomain;@0x2aaab30aba00:) exclude
--> java.security.ProtectionDomain@0x2aaab30aba90 (50 bytes) (field classloader:) exclude
--> org.apache.geronimo.kernel.config.MultiParentClassLoader@0x2aaab2a376b8 (156 bytes) (field parents:) exclude
--> [Ljava.lang.ClassLoader;@0x2aaab3086ff0 (88 bytes) (Element 1 of [Ljava.lang.ClassLoader;@0x2aaab3086ff0:) exclude
--> org.apache.geronimo.kernel.config.ChildrenConfigurationClassLoader@0x2aaab03bb338 (115 bytes) (field parent:) exclude
--> org.apache.geronimo.kernel.config.MultiParentClassLoader@0x2aaab2a37608 (156 bytes) exclude
{code}
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Attachments: ear-with-tx.tar.gz
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Donald Woods (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Donald Woods updated GERONIMO-4543:
-----------------------------------
Component/s: Memory Leaks
Priority: Blocker (was: Major)
Patch Info: [Patch Available]
Fix Version/s: 2.2
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Memory Leaks, transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Priority: Blocker
> Fix For: 2.2
>
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "David Jencks (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Jencks closed GERONIMO-4543.
----------------------------------
Resolution: Fixed
Patch applied in txmanager trunk rev 780434 and 2.1 branch rev 780437
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Memory Leaks, transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Priority: Blocker
> Fix For: 2.2
>
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Kevan Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675147#action_12675147 ]
Kevan Miller commented on GERONIMO-4543:
----------------------------------------
Hi Janko,
How many ClassLoader's are you seeing that aren't being GC'ed? Are you seeing OOME PermGen errors? Or just noticing that some ClassLoader's aren't being GC'ed? Are multiple deploy/undeploy cycles going to drive the problem? Or are you driving the app, also?
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Memory Leaks, transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Priority: Blocker
> Fix For: 2.2
>
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (GERONIMO-4543) EAR classloader not garbage
collected
Posted by "Janko Heilgeist (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GERONIMO-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681311#action_12681311 ]
Janko Heilgeist commented on GERONIMO-4543:
-------------------------------------------
Java6 says "Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future" if it is started with "-XX:+CMSPermGenSweepingEnabled" so I stopped using this option recently.
I'll admit that this report is not a major issue. I just discovered the leaks when I debugged an application. They haven't caused any real problems yet.
> EAR classloader not garbage collected
> -------------------------------------
>
> Key: GERONIMO-4543
> URL: https://issues.apache.org/jira/browse/GERONIMO-4543
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Memory Leaks, transaction manager
> Affects Versions: 2.2
> Reporter: Janko Heilgeist
> Priority: Blocker
> Fix For: 2.2
>
> Attachments: ear-with-tx.tar.gz, privileged_currenttime.patch
>
>
> The TransactionTimer$CurrentTime thread inherits the AccessControlContext of the first EAR/WAR/EJB-jar that carries out a transaction. Thus, the particular EAR/WAR/EJB-jar's classloader is prevented from being GCed.
> See http://www.nabble.com/PermGen-space-issues-td22079768s134.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.