[GitHub] [iceberg] singhpk234 commented on a change in pull request #3992: Spark 3.2: Fix cardinality check for alternative join implementations

[GitBox <gi...@apache.org>]

singhpk234 commented on a change in pull request #3992:
URL: https://github.com/apache/iceberg/pull/3992#discussion_r793328800



##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -46,6 +48,16 @@ case class MergeRowsExec(
     output: Seq[Attribute],
     child: SparkPlan) extends UnaryExecNode {
 
+  override def requiredChildOrdering: Seq[Seq[SortOrder]] = {
+    // request a local sort by the row ID attrs if shuffle hash joins are enabled
+    // this is needed to co-locate matches for the same target row after the shuffle
+    if (performCardinalityCheck && !conf.preferSortMergeJoin) {

Review comment:
       Hello Anton, 
   
   I have a small doubt, as per my understanding, at present not preferingSMJ doesn't implies SHJ being selected it's conjuncted with canBuildLocalHashMapBySize threshold as well as muchSmaller check's ([CP](https://github.com/apache/spark/blob/v3.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala#L277)).
   
   Also there are conf's such as "spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold" in AQE which even works slightly orthogonal by adding internal "PREFER_SHUFFLE_HASH" hint
   
   Any pointer's how are we thinking of handling it or is the handling even required (appologies I am very new to iceberg)  ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org