[iceberg] branch master updated: Spark: Do not pass numPartitions in 3.1 to let AQE size stages (#3203)

[bl...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

blue pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iceberg.git


The following commit(s) were added to refs/heads/master by this push:
     new adc6d15  Spark: Do not pass numPartitions in 3.1 to let AQE size stages (#3203)
adc6d15 is described below

commit adc6d153b26e7f982d75b427f62002e32f1913fc
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Wed Oct 6 23:47:33 2021 +0800

    Spark: Do not pass numPartitions in 3.1 to let AQE size stages (#3203)
---
 .../src/main/scala/org/apache/spark/sql/catalyst/utils/PlanUtils.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/spark3/src/main/scala/org/apache/spark/sql/catalyst/utils/PlanUtils.scala b/spark3/src/main/scala/org/apache/spark/sql/catalyst/utils/PlanUtils.scala
index e6cdf89..2902223 100644
--- a/spark3/src/main/scala/org/apache/spark/sql/catalyst/utils/PlanUtils.scala
+++ b/spark3/src/main/scala/org/apache/spark/sql/catalyst/utils/PlanUtils.scala
@@ -61,7 +61,8 @@ object PlanUtils {
     if (Spark3VersionUtil.isSpark30) {
       repartitionByExpressionCtor.newInstance(partitionExpressions, child, Integer.valueOf(numPartitions))
     } else {
-      repartitionByExpressionCtor.newInstance(partitionExpressions, child, Some(numPartitions))
+      // Do not pass numPartitions because it is set automatically for AQE
+      repartitionByExpressionCtor.newInstance(partitionExpressions, child, None)
     }
   }
 }