You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2015/07/23 08:48:09 UTC
spark git commit: [SPARK-8695] [CORE] [MLLIB] TreeAggregation
shouldn't be triggered when it doesn't save wall-clock time.
Repository: spark
Updated Branches:
refs/heads/master 6d0d8b406 -> b983d493b
[SPARK-8695] [CORE] [MLLIB] TreeAggregation shouldn't be triggered when it doesn't save wall-clock time.
Author: Perinkulam I. Ganesh <gi...@us.ibm.com>
Closes #7397 from piganesh/SPARK-8695 and squashes the following commits:
041620c [Perinkulam I. Ganesh] [SPARK-8695][CORE][MLlib] TreeAggregation shouldn't be triggered when it doesn't save wall-clock time.
9ad067c [Perinkulam I. Ganesh] [SPARK-8695] [core] [WIP] TreeAggregation shouldn't be triggered for 5 partitions
a6fed07 [Perinkulam I. Ganesh] [SPARK-8695] [core] [WIP] TreeAggregation shouldn't be triggered for 5 partitions
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b983d493
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b983d493
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b983d493
Branch: refs/heads/master
Commit: b983d493b490ca8bafe7eb988b62a250987ae353
Parents: 6d0d8b4
Author: Perinkulam I. Ganesh <gi...@us.ibm.com>
Authored: Thu Jul 23 07:46:20 2015 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Thu Jul 23 07:47:42 2015 +0100
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/rdd/RDD.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/b983d493/core/src/main/scala/org/apache/spark/rdd/RDD.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
index 394c668..6d61d22 100644
--- a/core/src/main/scala/org/apache/spark/rdd/RDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/RDD.scala
@@ -1082,7 +1082,9 @@ abstract class RDD[T: ClassTag](
val scale = math.max(math.ceil(math.pow(numPartitions, 1.0 / depth)).toInt, 2)
// If creating an extra level doesn't help reduce
// the wall-clock time, we stop tree aggregation.
- while (numPartitions > scale + numPartitions / scale) {
+
+ // Don't trigger TreeAggregation when it doesn't save wall-clock time
+ while (numPartitions > scale + math.ceil(numPartitions.toDouble / scale)) {
numPartitions /= scale
val curNumPartitions = numPartitions
partiallyAggregated = partiallyAggregated.mapPartitionsWithIndex {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org