You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jo...@apache.org on 2014/12/16 01:06:27 UTC
spark git commit: SPARK-785 [CORE] ClosureCleaner not invoked on most
PairRDDFunctions
Repository: spark
Updated Branches:
refs/heads/master 8176b7a02 -> 2a28bc610
SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions
This looked like perhaps a simple and important one. `combineByKey` looks like it should clean its arguments' closures, and that in turn covers apparently all remaining functions in `PairRDDFunctions` which delegate to it.
Author: Sean Owen <so...@cloudera.com>
Closes #3690 from srowen/SPARK-785 and squashes the following commits:
8df68fe [Sean Owen] Clean context of most remaining functions in PairRDDFunctions, which ultimately call combineByKey
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2a28bc61
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2a28bc61
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2a28bc61
Branch: refs/heads/master
Commit: 2a28bc61009a170af3853c78f7f36970898a6d56
Parents: 8176b7a
Author: Sean Owen <so...@cloudera.com>
Authored: Mon Dec 15 16:06:15 2014 -0800
Committer: Josh Rosen <jo...@databricks.com>
Committed: Mon Dec 15 16:06:15 2014 -0800
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/2a28bc61/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala b/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
index c43e1f2..b0434c9 100644
--- a/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
@@ -84,7 +84,10 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
throw new SparkException("Default partitioner cannot partition array keys.")
}
}
- val aggregator = new Aggregator[K, V, C](createCombiner, mergeValue, mergeCombiners)
+ val aggregator = new Aggregator[K, V, C](
+ self.context.clean(createCombiner),
+ self.context.clean(mergeValue),
+ self.context.clean(mergeCombiners))
if (self.partitioner == Some(partitioner)) {
self.mapPartitions(iter => {
val context = TaskContext.get()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org