[GitHub] [spark] cloud-fan commented on pull request #28676: [SPARK-31869][SQL] BroadcastHashJoinExec can utilize the build side for its output partitioning

[GitBox <gi...@apache.org>]

cloud-fan commented on pull request #28676:
URL: https://github.com/apache/spark/pull/28676#issuecomment-652894414


   > However, if the join type of BroadcastHashJoinExec is inner join, the build side's info (the join keys) can be added to BroadcastHashJoinExec's outputPartitioning.
   
   In general, I agree that we can leverage the partitioning from the build side, but I don't think simply union the partitioning of both sides is correct. Say the streaming side is partitioned by `a, b, c` and the join condition is `b = x AND c = y`. Then we can say the join output is partitioned by `a, b, c` or `a, x, c` or `a, b, y` or `a, x, y`. It's kind of replacing one or more columns from the streaming side partitioning with the corresponding join keys, and get all the combinations.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org