You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by da...@apache.org on 2015/12/02 07:41:56 UTC
spark git commit: [SPARK-12090] [PYSPARK] consider shuffle in
coalesce()
Repository: spark
Updated Branches:
refs/heads/master 0f37d1d7e -> 4375eb3f4
[SPARK-12090] [PYSPARK] consider shuffle in coalesce()
Author: Davies Liu <da...@databricks.com>
Closes #10090 from davies/fix_coalesce.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4375eb3f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4375eb3f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4375eb3f
Branch: refs/heads/master
Commit: 4375eb3f48fc7ae90caf6c21a0d3ab0b66bf4efa
Parents: 0f37d1d
Author: Davies Liu <da...@databricks.com>
Authored: Tue Dec 1 22:41:48 2015 -0800
Committer: Davies Liu <da...@gmail.com>
Committed: Tue Dec 1 22:41:48 2015 -0800
----------------------------------------------------------------------
python/pyspark/rdd.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/4375eb3f/python/pyspark/rdd.py
----------------------------------------------------------------------
diff --git a/python/pyspark/rdd.py b/python/pyspark/rdd.py
index 4b4d596..00bb9a6 100644
--- a/python/pyspark/rdd.py
+++ b/python/pyspark/rdd.py
@@ -2015,7 +2015,7 @@ class RDD(object):
>>> sc.parallelize([1, 2, 3, 4, 5], 3).coalesce(1).glom().collect()
[[1, 2, 3, 4, 5]]
"""
- jrdd = self._jrdd.coalesce(numPartitions)
+ jrdd = self._jrdd.coalesce(numPartitions, shuffle)
return RDD(jrdd, self.ctx, self._jrdd_deserializer)
def zip(self, other):
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org