You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2015/10/27 08:55:17 UTC
spark git commit: [SPARK-11276][CORE] SizeEstimator prevents class
unloading
Repository: spark
Updated Branches:
refs/heads/master d77d198fc -> feb8d6a44
[SPARK-11276][CORE] SizeEstimator prevents class unloading
The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys.
Which results in strong references to the Class objects. If these classes are dynamically created
this prevents the corresponding ClassLoader from being GCed. Leading to PermGen exhaustion.
We use a Map with WeakKeys to prevent this issue.
Author: Sem Mulder <se...@site2mobile.com>
Closes #9244 from SemMulder/fix-sizeestimator-classunloading.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/feb8d6a4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/feb8d6a4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/feb8d6a4
Branch: refs/heads/master
Commit: feb8d6a44fbfc31a880aaaac0cfcaadc91786073
Parents: d77d198
Author: Sem Mulder <se...@site2mobile.com>
Authored: Tue Oct 27 07:55:10 2015 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Tue Oct 27 07:55:10 2015 +0000
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/util/SizeEstimator.scala | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/feb8d6a4/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala b/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
index 14b1f2a..23ee4ef 100644
--- a/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
+++ b/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
@@ -17,6 +17,8 @@
package org.apache.spark.util
+import com.google.common.collect.MapMaker
+
import java.lang.management.ManagementFactory
import java.lang.reflect.{Field, Modifier}
import java.util.{IdentityHashMap, Random}
@@ -29,7 +31,6 @@ import org.apache.spark.Logging
import org.apache.spark.annotation.DeveloperApi
import org.apache.spark.util.collection.OpenHashSet
-
/**
* :: DeveloperApi ::
* Estimates the sizes of Java objects (number of bytes of memory they occupy), for use in
@@ -73,7 +74,8 @@ object SizeEstimator extends Logging {
private val ALIGN_SIZE = 8
// A cache of ClassInfo objects for each class
- private val classInfos = new ConcurrentHashMap[Class[_], ClassInfo]
+ // We use weakKeys to allow GC of dynamically created classes
+ private val classInfos = new MapMaker().weakKeys().makeMap[Class[_], ClassInfo]()
// Object and pointer sizes are arch dependent
private var is64bit = false
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org