You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2015/04/28 13:51:05 UTC
spark git commit: [SPARK-7100] [MLLIB] Fix persisted RDD leak in
GradientBoostTrees
Repository: spark
Updated Branches:
refs/heads/master 7f3b3b7eb -> 75905c57c
[SPARK-7100] [MLLIB] Fix persisted RDD leak in GradientBoostTrees
This fixes a leak of a persisted RDD where GradientBoostTrees can call persist but never unpersists.
Jira: https://issues.apache.org/jira/browse/SPARK-7100
Discussion: http://apache-spark-developers-list.1001551.n3.nabble.com/GradientBoostTrees-leaks-a-persisted-RDD-td11750.html
Author: Jim Carroll <ji...@dontcallme.com>
Closes #5669 from jimfcarroll/gb-unpersist-fix and squashes the following commits:
45f4b03 [Jim Carroll] [SPARK-7100][MLLib] Fix persisted RDD leak in GradientBoostTrees
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/75905c57
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/75905c57
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/75905c57
Branch: refs/heads/master
Commit: 75905c57cd57bc5b650ac5f486580ef8a229b260
Parents: 7f3b3b7
Author: Jim Carroll <ji...@dontcallme.com>
Authored: Tue Apr 28 07:51:02 2015 -0400
Committer: Sean Owen <so...@cloudera.com>
Committed: Tue Apr 28 07:51:02 2015 -0400
----------------------------------------------------------------------
.../org/apache/spark/mllib/tree/GradientBoostedTrees.scala | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/75905c57/mllib/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala b/mllib/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala
index 0e31c7e..deac390 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/tree/GradientBoostedTrees.scala
@@ -177,9 +177,10 @@ object GradientBoostedTrees extends Logging {
treeStrategy.assertValid()
// Cache input
- if (input.getStorageLevel == StorageLevel.NONE) {
+ val persistedInput = if (input.getStorageLevel == StorageLevel.NONE) {
input.persist(StorageLevel.MEMORY_AND_DISK)
- }
+ true
+ } else false
timer.stop("init")
@@ -265,6 +266,9 @@ object GradientBoostedTrees extends Logging {
logInfo("Internal timing for DecisionTree:")
logInfo(s"$timer")
+
+ if (persistedInput) input.unpersist()
+
if (validate) {
new GradientBoostedTreesModel(
boostingStrategy.treeStrategy.algo,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org