You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "clubycoder (via GitHub)" <gi...@apache.org> on 2023/02/06 19:45:27 UTC
[GitHub] [spark] clubycoder opened a new pull request, #39911: [SPARK-36478][SQL] Optimize out unused left-outer Join under Project
clubycoder opened a new pull request, #39911:
URL: https://github.com/apache/spark/pull/39911
This is an alternate approach to PR's previously submitted for - https://issues.apache.org/jira/browse/SPARK-36478
### What changes were proposed in this pull request?
This PR adds an additional Join optimizer that will eliminate a left-outer join that is under a Project where the columns needed are on all the right side of the join. In this case the join is removed and the right child of the Join is put directly under the Project.
### Why are the changes needed?
This optimization is meant to optimize queries that don't depend on a left-outer join that has been applied to an upstream view. The use-case is a group has standardized their analysis around a view that includes one or more left-outer joins to bring in additional columns that enhance the core data. This view has been saved off and is now a blackbox for multiple downstream queries that are required to use this view for standardization. Many of the queries on this view will not depend on columns coming from the view's left-joins. If this joined columns are not being used, we should avoid doing the join and reading the joined dataset.
For example we would like: ```
== Optimized Logical Plan ==
Project [timestamp#16L, customerId#17]
+- Join LeftOuter, (productId#18 = productId_products_productId#64)
:- Project [timestamp#16L, customerId#17, productId#18]
: +- Join LeftOuter, (customerId#17 = customerId_customers_customerId#58)
: :- LocalRelation [timestamp#16L, customerId#17, productId#18]
: +- LocalRelation [customerId_customers_customerId#58]
+- LocalRelation [productId_products_productId#64]
```
to become: ```
== Optimized Logical Plan ==
LocalRelation [timestamp#16L, customerId#17]
```
because the columns that come from the joins are not being used.
**NOTE**: The removal of the redundant Project(s) is handled by other existing optimizers.
### Does this PR introduce _any_ user-facing change?
The only use-facing change would be the explain of a query that matches the plan optimization. The optimized plan will have the left-outer join removed if the join is not used by the wrapping project.
### How was this patch tested?
Additional tests added to JoinOptimizationSuite covering plans that match and don't match the optimization as well as tested manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] clubycoder commented on pull request #39911: [SPARK-36478][SQL][WIP] Optimize out unused left-outer Join under Project
Posted by "clubycoder (via GitHub)" <gi...@apache.org>.
clubycoder commented on PR #39911:
URL: https://github.com/apache/spark/pull/39911#issuecomment-1420023995
Looks like this doesn't work when the left side has multiple rows that match and should cause duplication of the rows on the right even though columns from the left are not being used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] clubycoder closed pull request #39911: [SPARK-36478][SQL][WIP] Optimize out unused left-outer Join under Project
Posted by "clubycoder (via GitHub)" <gi...@apache.org>.
clubycoder closed pull request #39911: [SPARK-36478][SQL][WIP] Optimize out unused left-outer Join under Project
URL: https://github.com/apache/spark/pull/39911
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] clubycoder commented on pull request #39911: [SPARK-36478][SQL][WIP] Optimize out unused left-outer Join under Project
Posted by "clubycoder (via GitHub)" <gi...@apache.org>.
clubycoder commented on PR #39911:
URL: https://github.com/apache/spark/pull/39911#issuecomment-1419993504
Looks like there may be an issue with this combined with a lateral join...investigating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org