[GitHub] [spark] wangyum commented on a diff in pull request #36874: [SPARK-39475][SQL] Pull out complex join keys for shuffled join

[GitBox <gi...@apache.org>]

wangyum commented on code in PR #36874:
URL: https://github.com/apache/spark/pull/36874#discussion_r898559792


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -210,6 +210,8 @@ abstract class Optimizer(catalogManager: CatalogManager)
     // idempotence enforcement on this batch. We thus make it FixedPoint(1) instead of Once.
     Batch("Join Reorder", FixedPoint(1),
       CostBasedJoinReorder) :+
+    Batch("Pull Out Complex Join Keys", Once,
+      PullOutComplexJoinKeys) :+

Review Comment:
   There is an advantage to putting it here:
   - Reduce complex join key runs from 3 to 1 for SMJ.
   
   However, a disadvantage cannot be avoided:
   - It may increase the data size of shuffle. For example: the join key is: `concat(col1, col2, col3, col4 ...)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org