You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Costin Leau (JIRA)" <ji...@apache.org> on 2012/07/30 16:17:33 UTC
[jira] [Created] (HADOOP-8632) Configuration leaking class-loaders
Costin Leau created HADOOP-8632:
-----------------------------------
Summary: Configuration leaking class-loaders
Key: HADOOP-8632
URL: https://issues.apache.org/jira/browse/HADOOP-8632
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Costin Leau
The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated HADOOP-8632:
----------------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
3.0.0
Status: Resolved (was: Patch Available)
Thanks Constin,
I merged this into trunk, and branch-2. If you wanted it in another release please add a comment to indicate it.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435791#comment-13435791 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
By the way, tried patch -p 0 < HADOOP-8632-trunk.patch and it applied cleanly against the trunk.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433375#comment-13433375 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
By the way, in the same vein, ReflectionUtils#CONSTRUCTOR_CACHE also leaks classes (see HADOOP-8605 - I'm happy to fix that as well if you want).
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438181#comment-13438181 ]
Robert Joseph Evans commented on HADOOP-8632:
---------------------------------------------
The code looks good and the existing tests all seem to pass. Please remove the tabs from your patch, our coding standard requires the use of spaces instead, and then I am a +1.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas reassigned HADOOP-8632:
---------------------------------------
Assignee: Costin Leau
Costin, I added you as a contributor to Hadoop and assigned this jira to you. Thanks for contributing to Hadoop.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Assignee: Costin Leau
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431938#comment-13431938 ]
Robert Joseph Evans commented on HADOOP-8632:
---------------------------------------------
Costin,
I understand your issue more fully now, and I am fine if you want to add in WeakReferences to the ClassLoaders. If you have a patch for this leak, I would be happy to review it.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440387#comment-13440387 ]
Hudson commented on HADOOP-8632:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2626 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2626/])
HADOOP-8632. Configuration leaking class-loaders (Costin Leau via bobby) (Revision 1376543)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376543
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Costin Leau updated HADOOP-8632:
--------------------------------
Attachment: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch
git patch
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435217#comment-13435217 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
Guys I'm not sure why the patch doesn't apply - I've followed verbatim the instructions from the wiki. Any ideas on what's missing?
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HADOOP-8632.
---------------------------------
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Assignee: Costin Leau
> Fix For: 3.0.0, 2.0.2-alpha
>
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435808#comment-13435808 ]
Hadoop QA commented on HADOOP-8632:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541186/HADOOP-8632-trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1312//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1312//console
This message is automatically generated.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425186#comment-13425186 ]
Todd Lipcon commented on HADOOP-8632:
-------------------------------------
Hi Costin. We added the {{CACHE_CLASSES}} member in HADOOP-6133 to solve a performance regression which we saw in practice. HADOOP-6502 also talks about another case where the performance of this code was critical. So I don't think removing the cache is a good idea.
That said, I do think changing the cache to have weak values as well as weak keys makes sense. Guava's MapMaker has a nice utility to do this, or we could simply change the class values to be Map<String, WeakReference<Class<?>>>.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440172#comment-13440172 ]
Hadoop QA commented on HADOOP-8632:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542100/HADOOP-8632-trunk-no-tabs.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1351//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1351//console
This message is automatically generated.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450569#comment-13450569 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
Thanks!
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Assignee: Costin Leau
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433262#comment-13433262 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
I've attached my patch. I picked it up from my fork on GitHub (of Hadoop Commons) - based it on hadoop-2.0.1 branch.
See the code here: https://github.com/costin/hadoop-common/commit/57d9df37e600dd588a737d67b271657561ebfea2
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated HADOOP-8632:
----------------------------------------
Status: Patch Available (was: Open)
Kicking Jenkins so it will test the patch.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430490#comment-13430490 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
@Robert
My issue is not with the cache itself but with the leakage. If a client submits several big jobs, she has to either launch a new JVM for each submission or somehow patch the leak from outside. Or face OOM.
Addressing this in the framework directly obviously is much better.
@Todd
Wrapping the value with a WeakReference probably it's the easiest solution since it doesn't introduce a new library dependency. It can later be upgraded to MapMaker if the pattern occurs often.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435253#comment-13435253 ]
Robert Joseph Evans commented on HADOOP-8632:
---------------------------------------------
When I try to apply the patch to trunk I get
{noformat}
$ patch -p 0 < HADOOP-8632.patch
patching file hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
Hunk #3 succeeded at 1532 (offset 55 lines).
patching file hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
Hunk #2 FAILED at 1044.
1 out of 2 hunks FAILED -- saving rejects to file hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java.rej
{noformat}
You may want to try upmerging the patch to trunk.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434170#comment-13434170 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
Since the previous patch (the standard git/GitHub patch [1]) didn't apply correctly, I've attached the git diff result as specified on the Hadoop wiki [2]
[1] https://github.com/costin/hadoop-common/commit/57d9df37e600dd588a737d67b271657561ebfea2.patch
[2] http://wiki.apache.org/hadoop/GitAndHadoop
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435830#comment-13435830 ]
Costin Leau commented on HADOOP-8632:
-------------------------------------
The ZK failures seem to be caused by the CI host, independently from this patch.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Costin Leau updated HADOOP-8632:
--------------------------------
Attachment: HADOOP-8632-trunk-no-tabs.patch
the patch w/o any tabs.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440389#comment-13440389 ]
Hudson commented on HADOOP-8632:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2690 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2690/])
HADOOP-8632. Configuration leaking class-loaders (Costin Leau via bobby) (Revision 1376543)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376543
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Costin Leau updated HADOOP-8632:
--------------------------------
Attachment: HADOOP-8632-trunk.patch
Attaching the patch against the trunk (the previous patch as mentioned, was against branch-2.1.0-alpha).
Ran
git diff --no-prefix 7a3427de68 518c39814e
The github link is: http://j.mp/Nor44u
Did a cherry pick of the commit against 2.1.0-alpha.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433633#comment-13433633 ]
Hadoop QA commented on HADOOP-8632:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12540677/0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1286//console
This message is automatically generated.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440391#comment-13440391 ]
Hudson commented on HADOOP-8632:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2654 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2654/])
HADOOP-8632. Configuration leaking class-loaders (Costin Leau via bobby) (Revision 1376543)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376543
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Fix For: 3.0.0, 2.2.0-alpha
>
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking
class-loaders
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434635#comment-13434635 ]
Hadoop QA commented on HADOOP-8632:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12540887/HADOOP-8632.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1295//console
This message is automatically generated.
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
Posted by "Costin Leau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Costin Leau updated HADOOP-8632:
--------------------------------
Attachment: HADOOP-8632.patch
attaching result of:
git diff --no-prefix fa23683720 57d9df3
> Configuration leaking class-loaders
> -----------------------------------
>
> Key: HADOOP-8632
> URL: https://issues.apache.org/jira/browse/HADOOP-8632
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 2.0.0-alpha
> Reporter: Costin Leau
> Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch
>
>
> The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed.
> One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything.
> In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself.
> Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed.
> And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira