You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jo...@apache.org on 2014/10/07 21:20:32 UTC
git commit: [SPARK-3731] [PySpark] fix memory leak in PythonRDD
Repository: spark
Updated Branches:
refs/heads/master 655032965 -> bc87cc410
[SPARK-3731] [PySpark] fix memory leak in PythonRDD
The parent.getOrCompute() of PythonRDD is executed in a separated thread, it should release the memory reserved for shuffle and unrolling finally.
Author: Davies Liu <da...@gmail.com>
Closes #2668 from davies/leak and squashes the following commits:
ae98be2 [Davies Liu] fix memory leak in PythonRDD
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bc87cc41
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bc87cc41
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bc87cc41
Branch: refs/heads/master
Commit: bc87cc410fae59660c13b6ae1c14204df77237b8
Parents: 6550329
Author: Davies Liu <da...@gmail.com>
Authored: Tue Oct 7 12:20:12 2014 -0700
Committer: Josh Rosen <jo...@apache.org>
Committed: Tue Oct 7 12:20:12 2014 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/bc87cc41/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala b/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
index ad6eb9e..c74f865 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
@@ -247,6 +247,11 @@ private[spark] class PythonRDD(
// will kill the whole executor (see org.apache.spark.executor.Executor).
_exception = e
worker.shutdownOutput()
+ } finally {
+ // Release memory used by this thread for shuffles
+ env.shuffleMemoryManager.releaseMemoryForThisThread()
+ // Release memory used by this thread for unrolling blocks
+ env.blockManager.memoryStore.releaseUnrollMemoryForThisThread()
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org