[GitHub] [spark] ulysses-you edited a comment on pull request #34602: [SPARK-37328][SQL] Fix bug that OptimizeSkewedJoin may not work after it was moved from queryStageOptimizerRules to queryStagePreparationRules.

[GitBox <gi...@apache.org>]

ulysses-you edited a comment on pull request #34602:
URL: https://github.com/apache/spark/pull/34602#issuecomment-972458746


   > And it will not only work in cases where just 2 tables join, many complex combination need to be considered, such as multiple table joins in same stage.
   
   Why it can not work in such case ? if multiple table joins in same stage, the plan should be :
   ```
   SHJ2
     SHJ1
       ShuffleStage
       ShuffleStage
   ShuffleStage
   ```
   So we can still optimize the SHJ1 by transformUp this plan if we allow introduce extra shuffle.
   
   It seems to me that the check `if (shuffleStages.length == 2)` is unnecessary. aslo cc @JkSelf 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org