You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sem Mulder (JIRA)" <ji...@apache.org> on 2015/10/23 09:34:27 UTC
[jira] [Updated] (SPARK-11276) SizeEstimator prevents class
unloading
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sem Mulder updated SPARK-11276:
-------------------------------
Description:
The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys.
Which results in strong references to these class objects. If these classes are dynamically created
this prevents the corresponding ClassLoader from being GCed. Leading to PermGen exhaustion.
An easy fix would be to use a WeakRef for the keys. A proposed fix can be found here:
[https://github.com/Site2Mobile/spark/commit/21c572cbda5607d0c7c6643bfaf43e53c8aa6f8c]
We are currently running this in production and it seems to resolve the issue.
was:
The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys which results in strong references to these class objects.
If these classes are dynamically created this prevents the corresponding ClassLoader from being GCed. Leading to PermGen exhaustion.
An easy fix would be to use a WeakRef for the keys. A proposed fix can be found here:
[https://github.com/Site2Mobile/spark/commit/21c572cbda5607d0c7c6643bfaf43e53c8aa6f8c]
We are currently running this in production and it seems to resolve the issue.
> SizeEstimator prevents class unloading
> --------------------------------------
>
> Key: SPARK-11276
> URL: https://issues.apache.org/jira/browse/SPARK-11276
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
> Affects Versions: 1.5.1
> Reporter: Sem Mulder
>
> The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys.
> Which results in strong references to these class objects. If these classes are dynamically created
> this prevents the corresponding ClassLoader from being GCed. Leading to PermGen exhaustion.
> An easy fix would be to use a WeakRef for the keys. A proposed fix can be found here:
> [https://github.com/Site2Mobile/spark/commit/21c572cbda5607d0c7c6643bfaf43e53c8aa6f8c]
> We are currently running this in production and it seems to resolve the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org