You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2019/01/23 02:24:11 UTC
[spark] branch master updated: [SPARK-22465][CORE][FOLLOWUP] Use
existing partitioner when defaultNumPartitions is equal to
maxPartitioner.numPartitions
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3da71f2 [SPARK-22465][CORE][FOLLOWUP] Use existing partitioner when defaultNumPartitions is equal to maxPartitioner.numPartitions
3da71f2 is described below
commit 3da71f2da192276af041024b73e85e0acaac66a4
Author: Ngone51 <ng...@163.com>
AuthorDate: Wed Jan 23 10:23:40 2019 +0800
[SPARK-22465][CORE][FOLLOWUP] Use existing partitioner when defaultNumPartitions is equal to maxPartitioner.numPartitions
## What changes were proposed in this pull request?
Followup of #20091. We could also use existing partitioner when defaultNumPartitions is equal to the maxPartitioner's numPartitions.
## How was this patch tested?
Existed.
Closes #23581 from Ngone51/dev-use-existing-partitioner-when-defaultNumPartitions-equalTo-MaxPartitioner#-numPartitions.
Authored-by: Ngone51 <ng...@163.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
core/src/main/scala/org/apache/spark/Partitioner.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/Partitioner.scala b/core/src/main/scala/org/apache/spark/Partitioner.scala
index 5152375..a0cba8a 100644
--- a/core/src/main/scala/org/apache/spark/Partitioner.scala
+++ b/core/src/main/scala/org/apache/spark/Partitioner.scala
@@ -51,8 +51,8 @@ object Partitioner {
*
* When available, we choose the partitioner from rdds with maximum number of partitions. If this
* partitioner is eligible (number of partitions within an order of maximum number of partitions
- * in rdds), or has partition number higher than default partitions number - we use this
- * partitioner.
+ * in rdds), or has partition number higher than or equal to default partitions number - we use
+ * this partitioner.
*
* Otherwise, we'll use a new HashPartitioner with the default partitions number.
*
@@ -79,9 +79,9 @@ object Partitioner {
}
// If the existing max partitioner is an eligible one, or its partitions number is larger
- // than the default number of partitions, use the existing partitioner.
+ // than or equal to the default number of partitions, use the existing partitioner.
if (hasMaxPartitioner.nonEmpty && (isEligiblePartitioner(hasMaxPartitioner.get, rdds) ||
- defaultNumPartitions < hasMaxPartitioner.get.getNumPartitions)) {
+ defaultNumPartitions <= hasMaxPartitioner.get.getNumPartitions)) {
hasMaxPartitioner.get.partitioner.get
} else {
new HashPartitioner(defaultNumPartitions)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org