You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/06 23:24:38 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request, #37108: [SPARK-39551][SQL][3.2] Add AQE invalid plan check
dongjoon-hyun opened a new pull request, #37108:
URL: https://github.com/apache/spark/pull/37108
### What changes were proposed in this pull request?
This is a back port of https://github.com/apache/spark/pull/36953
This PR adds a check for invalid plans in AQE replanning process. The check will throw exceptions when it detects an invalid plan, causing AQE to void the current replanning result and keep using the latest valid plan.
### Why are the changes needed?
AQE logical optimization rules can lead to invalid physical plans and cause runtime exceptions as certain physical plan nodes are not compatible with others. E.g., `BroadcastExchangeExec` can only work as a direct child of broadcast join nodes, but it could appear under other incompatible physical plan nodes because of empty relation propagation.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #37108: [SPARK-39551][SQL][3.2] Add AQE invalid plan check
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #37108:
URL: https://github.com/apache/spark/pull/37108#issuecomment-1176945065
thank you @dongjoon-hyun help backport , I'm a bit late
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37108: [SPARK-39551][SQL][3.2] Add AQE invalid plan check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37108:
URL: https://github.com/apache/spark/pull/37108#issuecomment-1176945686
No problem. I wanted to have this at Apache Spark 3.2.2 RC1 next week. That's the reason why I hurried. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37108: [SPARK-39551][SQL][3.2] Add AQE invalid plan check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37108:
URL: https://github.com/apache/spark/pull/37108#issuecomment-1177055604
All tests passed. Thank you, @ulysses-you and @cloud-fan .
Merged to branch-3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #37108: [SPARK-39551][SQL][3.2] Add AQE invalid plan check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #37108: [SPARK-39551][SQL][3.2] Add AQE invalid plan check
URL: https://github.com/apache/spark/pull/37108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37108: [SPARK-39551][SQL][3.2] Add AQE invalid plan check
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37108:
URL: https://github.com/apache/spark/pull/37108#issuecomment-1176860401
cc @maryannxue , @cloud-fan , @ulysses-you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org