[GitHub] [spark] ulysses-you commented on pull request #36483: [SPARK-39126][SQL] After eliminating join to one side, that side should take advantage of LocalShuffleRead optimization

[GitBox <gi...@apache.org>]

ulysses-you commented on PR #36483:
URL: https://github.com/apache/spark/pull/36483#issuecomment-1120638033

   We do not need a new shuffle origin and the current `OptimizeShuffleWithLocalRead` almost supports this case except one thing:
   
   The `OptimizeShuffleWithLocalRead` only match the root plan who is a shuffle or shuffle reader, but the case you point is a Project which is from the original output of the Join (join is converted to project). So the physical plan after optimized looks like:
   ```
   Project
     AQEShuffleReader
       ...
   ```
   
   It seems we can enhance `OptimizeShuffleWithLocalRead` that support optimzie the shuffle which is under the Project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org