You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/23 01:41:25 UTC
[GitHub] [spark] ulysses-you commented on a change in pull request #32932: [SPARK-35786][SQL] Add a new operator to distingush if AQE can optimize safely
ulysses-you commented on a change in pull request #32932:
URL: https://github.com/apache/spark/pull/32932#discussion_r656693575
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##########
@@ -1351,6 +1351,26 @@ object RepartitionByExpression {
}
}
+/**
+ * This operator does not guarantee the output partitioning, because the partition number will be
+ * optimized by AQE.
+ */
+case class AdaptiveRepartition(
+ partitionExpressions: Seq[Expression],
+ child: LogicalPlan) extends RepartitionOperation {
Review comment:
yea it's more safer to don't extend `RepartitionOperation`.
##########
File path: docs/sql-performance-tuning.md
##########
@@ -228,6 +228,8 @@ The "REPARTITION_BY_RANGE" hint must have column names and a partition number is
SELECT /*+ REPARTITION */ * FROM t
SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t
SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t
+ SELECT /*+ ADAPTIVE_REPARTITION */ * FROM t
+ SELECT /*+ ADAPTIVE_REPARTITION(c) */ * FROM t
Review comment:
+1 for `REPARTITION_BY_AQE `
##########
File path: docs/sql-performance-tuning.md
##########
@@ -228,6 +228,8 @@ The "REPARTITION_BY_RANGE" hint must have column names and a partition number is
SELECT /*+ REPARTITION */ * FROM t
SELECT /*+ REPARTITION_BY_RANGE(c) */ * FROM t
SELECT /*+ REPARTITION_BY_RANGE(3, c) */ * FROM t
+ SELECT /*+ ADAPTIVE_REPARTITION */ * FROM t
+ SELECT /*+ ADAPTIVE_REPARTITION(c) */ * FROM t
For more details please refer to the documentation of [Partitioning Hints](sql-ref-syntax-qry-select-hints.html#partitioning-hints).
Review comment:
yea, it's better to add docs for the new hint. Since this PR only adds a new hint that has no big difference with previous hint, we can add more docs and use case after adding new rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org