You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/08 09:04:53 UTC
[GitHub] [spark] ulysses-you opened a new pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
ulysses-you opened a new pull request #32816:
URL: https://github.com/apache/spark/pull/32816
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error message, please read the guideline first:
https://spark.apache.org/error-message-guidelines.html
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
- move `OptimizeSkewedJoin` from stage optimizer rules to preparation stage rules
- add rule `EnsureRequirements` behind the `OptimizeSkewedJoin`
- add `SkewJoinAwareCost` to support estimate skewed join cost
- add `finalPreparationStageRules` to avoid use shuffle reader for repartition
- add new config to decide if force optimize skewed join
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
In general, skewed join has more impact on performance than once more shuffle. It makes sense to force optimize skewed join even if introduce extra shuffle.
A common case:
```
HashAggregate
SortMergJoin
Sort
Exchange
Sort
Exchange
```
and after this PR, the plan looks like:
```
HashAggregate
Exchange
SortMergJoin (isSkew=true)
Sort
Exchange
Sort
Exchange
```
Note that, the new introduced shuffle also can be optimized by AQE.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, a new config.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
* Add new test
* pass exists test `SPARK-30524: Do not optimize skew join if introduce additional shuffle`
* pass exists test `SPARK-33551: Do not use custom shuffle reader for repartition`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-872729734
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877108186
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45360/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906233248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856646979
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-891788575
**[Test build #141993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141993/testReport)** for PR 32816 at commit [`30b7de0`](https://github.com/apache/spark/commit/30b7de03dc20ab8f2d9aaa07ed37a18d6081f672).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902559282
**[Test build #142663 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142663/testReport)** for PR 32816 at commit [`f5ad40e`](https://github.com/apache/spark/commit/f5ad40e7ef8fafc4e084ef15d7956d0b3eaf214e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917565931
**[Test build #143175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143175/testReport)** for PR 32816 at commit [`f5e4b91`](https://github.com/apache/spark/commit/f5e4b911ea608cdf863215feb6e13996a57cb294).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-913203465
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143001/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r704901886
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -38,26 +38,40 @@ import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoin
* but can be false in AQE when AQE optimization may change the plan
* output partitioning and need to retain the user-specified
* repartition shuffles in the plan.
+ * @param requiredDistribution The root required distribution we should ensure. This value is used
+ * in AQE in case we change final stage output partitioning.
*/
-case class EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Rule[SparkPlan] {
-
- private def ensureDistributionAndOrdering(operator: SparkPlan): SparkPlan = {
- val requiredChildDistributions: Seq[Distribution] = operator.requiredChildDistribution
- val requiredChildOrderings: Seq[Seq[SortOrder]] = operator.requiredChildOrdering
- var children: Seq[SparkPlan] = operator.children
- assert(requiredChildDistributions.length == children.length)
- assert(requiredChildOrderings.length == children.length)
+case class EnsureRequirements(
+ optimizeOutRepartition: Boolean = true,
+ requiredDistribution: Option[Distribution] = None)
+ extends Rule[SparkPlan] {
+ private def ensureDistributionAndOrdering(
+ originChildren: Seq[SparkPlan],
+ requiredChildDistributions: Seq[Distribution],
+ requiredChildOrderings: Seq[Seq[SortOrder]],
+ isRootDistribution: Boolean): Seq[SparkPlan] = {
+ assert(requiredChildDistributions.length == originChildren.length)
+ assert(requiredChildOrderings.length == originChildren.length)
// Ensure that the operator's children satisfy their output distribution requirements.
- children = children.zip(requiredChildDistributions).map {
+ var children = originChildren.zip(requiredChildDistributions).map {
case (child, distribution) if child.outputPartitioning.satisfies(distribution) =>
child
case (child, BroadcastDistribution(mode)) =>
BroadcastExchangeExec(mode, child)
case (child, distribution) =>
val numPartitions = distribution.requiredNumPartitions
.getOrElse(conf.numShufflePartitions)
- ShuffleExchangeExec(distribution.createPartitioning(numPartitions), child)
+ val shuffleOrigin = if (isRootDistribution) {
Review comment:
Gotcha, so can we pass this shuffleOrigin from the parameter list instead of the boolean `isRootDistribution`? won't reduce the code change, but seems nicer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917380025
**[Test build #143164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143164/testReport)** for PR 32816 at commit [`f5e4b91`](https://github.com/apache/spark/commit/f5e4b911ea608cdf863215feb6e13996a57cb294).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914107971
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143042/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905273236
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142753/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902426885
**[Test build #142664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142664/testReport)** for PR 32816 at commit [`8058fe9`](https://github.com/apache/spark/commit/8058fe9354ed28b32f0ea0bdc8c9243f79c58fba).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902438633
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856865935
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139480/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-913203224
**[Test build #143001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143001/testReport)** for PR 32816 at commit [`23ebea0`](https://github.com/apache/spark/commit/23ebea0a408469f5c2a70e04982ffa6261dae96c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class OptimizeSkewedJoin(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696399885
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -169,15 +179,34 @@ case class AdaptiveSparkPlanExec(
optimized
}
+ def prepareQueryStages(
+ plan: SparkPlan,
+ optimizeSkewedJoin: Boolean): SparkPlan = {
+ if (optimizeSkewedJoin) {
+ AQEUtils.ensureRequiredDistribution(
Review comment:
I think it doesn't hurt to always run it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906230828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906263140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915720214
**[Test build #143101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143101/testReport)** for PR 32816 at commit [`ca63321`](https://github.com/apache/spark/commit/ca6332167f20ca85c4ae23100a475530f9118a8e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r688276500
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -169,10 +162,33 @@ case class AdaptiveSparkPlanExec(
optimized
}
+ private def checkDistribution(
+ newPlan: SparkPlan,
+ originPlan: SparkPlan,
+ isFinalStage: Boolean,
+ ruleName: String): SparkPlan = {
Review comment:
this method is used by `optimizeQueryStage` and `optimizeSkewedJoin` for checking final stage distribution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r692075920
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/SkewJoinAwareCost.scala
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.exchange.ShuffleExchangeLike
+import org.apache.spark.sql.execution.joins.ShuffledJoin
+
+/**
+ * A skew join aware implementation of [[Cost]], which consider shuffle number and skew join number
+ */
+case class SkewJoinAwareCost(
+ numShuffles: Int,
+ numSkewJoins: Int) extends Cost {
+ override def compare(that: Cost): Int = that match {
+ case other: SkewJoinAwareCost =>
+ if (numSkewJoins > other.numSkewJoins || numShuffles < other.numShuffles) {
Review comment:
yes, if a plan has now skew join and less shuffle nodes than b plan, we will pick a plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906233216
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905307929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142751/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r695389108
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEUtils.scala
##########
@@ -57,4 +58,20 @@ object AQEUtils {
}
case _ => Some(UnspecifiedDistribution)
}
+
+ // Add an extra shuffle if input plan does not satisfy the required distribution.
+ // This method is invoked after optimizing skewed join in case we change final stage
+ // output partitioning.
+ def ensureRequiredDistribution(
+ plan: SparkPlan, distribution: Option[Distribution]): SparkPlan = distribution match {
+ case Some(d) if !plan.outputPartitioning.satisfies(d) =>
+ val numPartitions = d.requiredNumPartitions.getOrElse(conf.numShufflePartitions)
+ val shuffleOrigin = if (d.requiredNumPartitions.isDefined) {
+ REPARTITION_BY_NUM
+ } else {
+ REPARTITION_BY_COL
+ }
+ ShuffleExchangeExec(d.createPartitioning(numPartitions), plan, shuffleOrigin)
Review comment:
I think the origin should always be `ENSIRE_REQUIREMENT`? The shuffle is added to satisfy the required distribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856780537
**[Test build #139466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139466/testReport)** for PR 32816 at commit [`368d1e0`](https://github.com/apache/spark/commit/368d1e0d3d20d763085a8a910ed6d6befe8fb9db).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class SkewJoinAwareCost(numShuffles: Int, numSkewJoins: Int) extends Cost `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696404589
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##########
@@ -1908,6 +1914,70 @@ class AdaptiveQueryExecSuite
}
}
+ test("SPARK-33832: Support optimize skew join even if introduce extra shuffle") {
+ withSQLConf(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_OPTIMIZE_SKEWS_IN_REBALANCE_PARTITIONS_ENABLED.key -> "false",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.SKEW_JOIN_SKEWED_PARTITION_THRESHOLD.key -> "100",
+ SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.key -> "100",
+ SQLConf.COALESCE_PARTITIONS_MIN_PARTITION_NUM.key -> "1",
+ SQLConf.SHUFFLE_PARTITIONS.key -> "10",
+ SQLConf.ADAPTIVE_FORCE_OPTIMIZE_SKEWED_JOIN.key -> "true") {
+ withTempView("skewData1", "skewData2") {
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 3 as key1", "id as value1")
+ .createOrReplaceTempView("skewData1")
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 1 as key2", "id as value2")
+ .createOrReplaceTempView("skewData2")
+
+ // check if optimized skewed join does not satisfy the required distribution
+ Seq(true, false).foreach { hasRequiredDistribution =>
+ Seq(true, false).foreach { hasPartitionNumber =>
+ val repartition = if (hasRequiredDistribution) {
+ s"/*+ repartition(${ if (hasPartitionNumber) "10," else ""}key1) */"
+ } else {
+ ""
+ }
+
+ // check required distribution and extra shuffle
+ val (_, adaptive1) =
+ runAdaptiveAndVerifyResult(s"SELECT $repartition key1 FROM skewData1 " +
+ s"JOIN skewData2 ON key1 = key2 GROUP BY key1")
+ val shuffles1 = findTopLevelShuffle(adaptive1)
+ assert(shuffles1.size == 3)
+ // the head shuffle is from second EnsureRequirements in queryStagePreparationRules
Review comment:
`// shuffles1.head is the top-level shuffle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857577812
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139541/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857417815
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44056/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915750888
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915755827
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47604/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-901966258
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47146/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857388691
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44056/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-872731988
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877075687
**[Test build #140849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140849/testReport)** for PR 32816 at commit [`7734d3e`](https://github.com/apache/spark/commit/7734d3eec1078a56f600ce7b6e718824c09a0e3b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-891782878
**[Test build #141993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141993/testReport)** for PR 32816 at commit [`30b7de0`](https://github.com/apache/spark/commit/30b7de03dc20ab8f2d9aaa07ed37a18d6081f672).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696356920
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -258,7 +258,8 @@ object OptimizeSkewedJoin extends AQEShuffleReadRule {
object ShuffleStage {
def unapply(plan: SparkPlan): Option[ShuffleQueryStageExec] = plan match {
- case s: ShuffleQueryStageExec if s.mapStats.isDefined && isSupported(s.shuffle) =>
+ case s: ShuffleQueryStageExec if s.isMaterialized && s.mapStats.isDefined &&
+ isSupported(s.shuffle) =>
Review comment:
nit: seems this rule doesn't need to extend `AQEShuffleReadRule` any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905203266
**[Test build #142753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142753/testReport)** for PR 32816 at commit [`3135585`](https://github.com/apache/spark/commit/3135585184307c9584dc47ee9375dc3de9470729).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r701922272
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -640,14 +668,12 @@ case class AdaptiveSparkPlanExec(
* Re-optimize and run physical planning on the current logical plan based on the latest stats.
*/
private def reOptimize(
- logicalPlan: LogicalPlan): (SparkPlan, LogicalPlan) = context.qe.withCteMap {
+ logicalPlan: LogicalPlan): (Seq[SparkPlan], LogicalPlan) = context.qe.withCteMap {
logicalPlan.invalidateStatsCache()
val optimized = optimizer.execute(logicalPlan)
val sparkPlan = context.session.sessionState.planner.plan(ReturnAnswer(optimized)).next()
- val newPlan = applyPhysicalRules(
- sparkPlan,
- preprocessingRules ++ queryStagePreparationRules,
- Some((planChangeLogger, "AQE Replanning")))
+ val optimizedPhysicalPlan = prepareQueryStages(sparkPlan, false)
Review comment:
The code changes won't be reduced by much, but at least it won't "contaminate" the framework. Why does AQE framework has to be aware of this skew join rule specifically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856720169
**[Test build #139480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139480/testReport)** for PR 32816 at commit [`ccc45ab`](https://github.com/apache/spark/commit/ccc45ab019beddc7eae0193f82883986a4a513fa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917391057
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47668/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912611240
**[Test build #142980 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142980/testReport)** for PR 32816 at commit [`4712986`](https://github.com/apache/spark/commit/4712986d217e515197a4f6bee1ae9790fda62087).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912423787
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914145069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47546/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r703278537
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -250,15 +253,28 @@ object OptimizeSkewedJoin extends AQEShuffleReadRule {
// SHJ
// Shuffle
// Shuffle
- optimizeSkewJoin(plan)
+ val optimized =
+ EnsureRequirements(requiredDistribution.isDefined, requiredDistribution)
+ .apply(optimizeSkewJoin(plan))
+ val originCost = costEvaluator.evaluateCost(plan)
+ val optimizedCost = costEvaluator.evaluateCost(optimized)
+ // two cases we will pick new plan:
+ // 1. optimize the skew join without extra shuffle
+ // 2. optimize the skew join with extra shuffle but the costEvaluator think it's better
+ if (optimizedCost < originCost || (originCost == optimizedCost && optimized != plan)) {
Review comment:
yea, since we are only optimize physical plan here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902566875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902443736
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47165/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902426885
**[Test build #142664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142664/testReport)** for PR 32816 at commit [`8058fe9`](https://github.com/apache/spark/commit/8058fe9354ed28b32f0ea0bdc8c9243f79c58fba).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898180606
**[Test build #142412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142412/testReport)** for PR 32816 at commit [`d305894`](https://github.com/apache/spark/commit/d305894ef7efd7dd8f7792febe328c5ebe447184).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857494775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858319082
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905199821
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47252/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905372169
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47258/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r692179974
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -656,13 +687,54 @@ case class AdaptiveSparkPlanExec(
// node to prevent the loss of the `BroadcastExchangeExec` node in DPP subquery.
// Here, we also need to avoid to insert the `BroadcastExchangeExec` node when the newPlan
// is already the `BroadcastExchangeExec` plan after apply the `LogicalQueryStageStrategy` rule.
- val finalPlan = currentPhysicalPlan match {
+ def updateBroadcastExchange(plan: SparkPlan): SparkPlan = currentPhysicalPlan match {
case b: BroadcastExchangeLike
- if (!newPlan.isInstanceOf[BroadcastExchangeLike]) => b.withNewChildren(Seq(newPlan))
- case _ => newPlan
+ if (!plan.isInstanceOf[BroadcastExchangeLike]) => b.withNewChildren(Seq(plan))
+ case _ => plan
}
- (finalPlan, optimized)
+ val optimizedWithSkewedJoin = applyPhysicalRules(
+ optimizedPhysicalPlan,
+ optimizeSkewedJoinWithExtraShuffleRules,
Review comment:
I find this a bit hard to reason about. In general, we have different ways to optimize the query. We produce multiple physical plans and pick the one with the least cost.
Here we apply extra rules to the plan and get a new plan. This doesn't match the general idea. That's why I proposed https://github.com/apache/spark/pull/32816/files#r691844920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-899537814
**[Test build #142493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142493/testReport)** for PR 32816 at commit [`6725f97`](https://github.com/apache/spark/commit/6725f975ca9f15eb8f233381fcfa9fc88cdef760).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r692181227
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -656,13 +687,54 @@ case class AdaptiveSparkPlanExec(
// node to prevent the loss of the `BroadcastExchangeExec` node in DPP subquery.
// Here, we also need to avoid to insert the `BroadcastExchangeExec` node when the newPlan
// is already the `BroadcastExchangeExec` plan after apply the `LogicalQueryStageStrategy` rule.
- val finalPlan = currentPhysicalPlan match {
+ def updateBroadcastExchange(plan: SparkPlan): SparkPlan = currentPhysicalPlan match {
case b: BroadcastExchangeLike
- if (!newPlan.isInstanceOf[BroadcastExchangeLike]) => b.withNewChildren(Seq(newPlan))
- case _ => newPlan
+ if (!plan.isInstanceOf[BroadcastExchangeLike]) => b.withNewChildren(Seq(plan))
+ case _ => plan
}
- (finalPlan, optimized)
+ val optimizedWithSkewedJoin = applyPhysicalRules(
+ optimizedPhysicalPlan,
+ optimizeSkewedJoinWithExtraShuffleRules,
+ Some((planChangeLogger, "AQE Optimize Skewed Join With Extra Shuffle"))
+ )
+ val validatedWithSkewedJoin =
+ checkDistribution(
+ optimizedWithSkewedJoin,
+ optimizedPhysicalPlan,
+ isFinalStage(optimizedWithSkewedJoin),
Review comment:
previously we only validate in the query optimization phase, not preparation phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858424083
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44148/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906593780
**[Test build #142804 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142804/testReport)** for PR 32816 at commit [`bc39694`](https://github.com/apache/spark/commit/bc39694258c88e338d41df7db51aa6b123f0a32a).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696411043
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -100,24 +100,34 @@ case class AdaptiveSparkPlanExec(
// A list of physical plan rules to be applied before creation of query stages. The physical
// plan should reach a final status of query stages (i.e., no more addition or removal of
// Exchange nodes) after running these rules.
- @transient private val queryStagePreparationRules: Seq[Rule[SparkPlan]] = Seq(
- RemoveRedundantProjects,
- // For cases like `df.repartition(a, b).select(c)`, there is no distribution requirement for
- // the final plan, but we do need to respect the user-specified repartition. Here we ask
- // `EnsureRequirements` to not optimize out the user-specified repartition-by-col to work
- // around this case.
- EnsureRequirements(optimizeOutRepartition = requiredDistribution.isDefined),
- RemoveRedundantSorts,
- DisableUnnecessaryBucketedScan
- ) ++ context.session.sessionState.queryStagePrepRules
+ private def queryStagePreparationRules(
+ optimizeSkewedJoin: Boolean = false): Seq[Rule[SparkPlan]] = {
Review comment:
Im OK to remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858252115
**[Test build #139608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139608/testReport)** for PR 32816 at commit [`a80bb5c`](https://github.com/apache/spark/commit/a80bb5c06d76f35a80afcad6e242158ef809875e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877217755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140846/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912431161
**[Test build #142980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142980/testReport)** for PR 32816 at commit [`4712986`](https://github.com/apache/spark/commit/4712986d217e515197a4f6bee1ae9790fda62087).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r702426889
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -250,15 +253,28 @@ object OptimizeSkewedJoin extends AQEShuffleReadRule {
// SHJ
// Shuffle
// Shuffle
- optimizeSkewJoin(plan)
+ val optimized =
+ EnsureRequirements(requiredDistribution.isDefined, requiredDistribution)
+ .apply(optimizeSkewJoin(plan))
+ val originCost = costEvaluator.evaluateCost(plan)
+ val optimizedCost = costEvaluator.evaluateCost(optimized)
+ // two cases we will pick new plan:
+ // 1. optimize the skew join without extra shuffle
+ // 2. optimize the skew join with extra shuffle but the costEvaluator think it's better
+ if (optimizedCost < originCost || (originCost == optimizedCost && optimized != plan)) {
Review comment:
Moved the cost evaluation into `OptimizeSkewJoin`, so now we have two cases that will pick the new plan (see the comment).
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -169,12 +174,6 @@ case class AdaptiveSparkPlanExec(
optimized
}
- @transient private val costEvaluator =
- conf.getConf(SQLConf.ADAPTIVE_CUSTOM_COST_EVALUATOR_CLASS) match {
- case Some(className) => CostEvaluator.instantiate(className, session.sparkContext.getConf)
- case _ => SimpleCostEvaluator
- }
Review comment:
just move this code up
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -38,26 +38,40 @@ import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoin
* but can be false in AQE when AQE optimization may change the plan
* output partitioning and need to retain the user-specified
* repartition shuffles in the plan.
+ * @param requiredDistribution The root required distribution we should ensure. This value is used
+ * in AQE in case we change final stage output partitioning.
*/
-case class EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Rule[SparkPlan] {
-
- private def ensureDistributionAndOrdering(operator: SparkPlan): SparkPlan = {
- val requiredChildDistributions: Seq[Distribution] = operator.requiredChildDistribution
- val requiredChildOrderings: Seq[Seq[SortOrder]] = operator.requiredChildOrdering
- var children: Seq[SparkPlan] = operator.children
- assert(requiredChildDistributions.length == children.length)
- assert(requiredChildOrderings.length == children.length)
+case class EnsureRequirements(
+ optimizeOutRepartition: Boolean = true,
+ requiredDistribution: Option[Distribution] = None)
+ extends Rule[SparkPlan] {
+ private def ensureDistributionAndOrdering(
+ originChildren: Seq[SparkPlan],
+ requiredChildDistributions: Seq[Distribution],
+ requiredChildOrderings: Seq[Seq[SortOrder]],
+ isRootDistribution: Boolean): Seq[SparkPlan] = {
+ assert(requiredChildDistributions.length == originChildren.length)
+ assert(requiredChildOrderings.length == originChildren.length)
// Ensure that the operator's children satisfy their output distribution requirements.
- children = children.zip(requiredChildDistributions).map {
+ var children = originChildren.zip(requiredChildDistributions).map {
case (child, distribution) if child.outputPartitioning.satisfies(distribution) =>
child
case (child, BroadcastDistribution(mode)) =>
BroadcastExchangeExec(mode, child)
case (child, distribution) =>
val numPartitions = distribution.requiredNumPartitions
.getOrElse(conf.numShufflePartitions)
- ShuffleExchangeExec(distribution.createPartitioning(numPartitions), child)
+ val shuffleOrigin = if (isRootDistribution) {
Review comment:
@maryannxue this condition is what I said before. if the required distribution is root then we should not give it the `ENSURE_REQUIREMENTS`. The reason is `ENSURE_REQUIREMENTS` will be optimized by other rules (e.g. `CoalesceShufflePartitions`) that will break the required distribution again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914103700
**[Test build #143042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143042/testReport)** for PR 32816 at commit [`76c363d`](https://github.com/apache/spark/commit/76c363daacfcf35a08378859c710f3bd916bc153).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917380025
**[Test build #143164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143164/testReport)** for PR 32816 at commit [`f5e4b91`](https://github.com/apache/spark/commit/f5e4b911ea608cdf863215feb6e13996a57cb294).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856648191
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139471/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858287594
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44135/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r700729711
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEUtils.scala
##########
@@ -57,4 +58,22 @@ object AQEUtils {
}
case _ => Some(UnspecifiedDistribution)
}
+
+ // Add an extra shuffle if input plan does not satisfy the required distribution.
+ // This method is invoked after optimizing skewed join in case we change final stage
+ // output partitioning.
+ def ensureRequiredDistribution(
Review comment:
Can we take advantage of `EnsureRequirements` here by passing it a root distro requirement instead of using this adhoc logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #32816:
URL: https://github.com/apache/spark/pull/32816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r704911852
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -38,26 +38,40 @@ import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoin
* but can be false in AQE when AQE optimization may change the plan
* output partitioning and need to retain the user-specified
* repartition shuffles in the plan.
+ * @param requiredDistribution The root required distribution we should ensure. This value is used
+ * in AQE in case we change final stage output partitioning.
*/
-case class EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Rule[SparkPlan] {
-
- private def ensureDistributionAndOrdering(operator: SparkPlan): SparkPlan = {
- val requiredChildDistributions: Seq[Distribution] = operator.requiredChildDistribution
- val requiredChildOrderings: Seq[Seq[SortOrder]] = operator.requiredChildOrdering
- var children: Seq[SparkPlan] = operator.children
- assert(requiredChildDistributions.length == children.length)
- assert(requiredChildOrderings.length == children.length)
+case class EnsureRequirements(
+ optimizeOutRepartition: Boolean = true,
+ requiredDistribution: Option[Distribution] = None)
+ extends Rule[SparkPlan] {
+ private def ensureDistributionAndOrdering(
+ originChildren: Seq[SparkPlan],
+ requiredChildDistributions: Seq[Distribution],
+ requiredChildOrderings: Seq[Seq[SortOrder]],
+ isRootDistribution: Boolean): Seq[SparkPlan] = {
+ assert(requiredChildDistributions.length == originChildren.length)
+ assert(requiredChildOrderings.length == originChildren.length)
// Ensure that the operator's children satisfy their output distribution requirements.
- children = children.zip(requiredChildDistributions).map {
+ var children = originChildren.zip(requiredChildDistributions).map {
case (child, distribution) if child.outputPartitioning.satisfies(distribution) =>
child
case (child, BroadcastDistribution(mode)) =>
BroadcastExchangeExec(mode, child)
case (child, distribution) =>
val numPartitions = distribution.requiredNumPartitions
.getOrElse(conf.numShufflePartitions)
- ShuffleExchangeExec(distribution.createPartitioning(numPartitions), child)
+ val shuffleOrigin = if (isRootDistribution) {
Review comment:
agree, pulled out the shuffle origin to make thie more cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877237063
**[Test build #140849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140849/testReport)** for PR 32816 at commit [`7734d3e`](https://github.com/apache/spark/commit/7734d3eec1078a56f600ce7b6e718824c09a0e3b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898180606
**[Test build #142412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142412/testReport)** for PR 32816 at commit [`d305894`](https://github.com/apache/spark/commit/d305894ef7efd7dd8f7792febe328c5ebe447184).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898192900
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856609792
**[Test build #139466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139466/testReport)** for PR 32816 at commit [`368d1e0`](https://github.com/apache/spark/commit/368d1e0d3d20d763085a8a910ed6d6befe8fb9db).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877074968
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45357/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902424243
**[Test build #142662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142662/testReport)** for PR 32816 at commit [`b54e9c2`](https://github.com/apache/spark/commit/b54e9c232a43ce05e320d5459b16b770261ee38b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905305090
**[Test build #142751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142751/testReport)** for PR 32816 at commit [`b215e2d`](https://github.com/apache/spark/commit/b215e2d0fdf959bd1c735297a1a33c1a078f2361).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-901889681
**[Test build #142645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142645/testReport)** for PR 32816 at commit [`60b7b9d`](https://github.com/apache/spark/commit/60b7b9da8f268f15706c30462976ffc389c95c45).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898283032
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857365002
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139532/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696415767
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -169,15 +179,34 @@ case class AdaptiveSparkPlanExec(
optimized
}
+ def prepareQueryStages(
+ plan: SparkPlan,
+ optimizeSkewedJoin: Boolean): SparkPlan = {
+ if (optimizeSkewedJoin) {
+ AQEUtils.ensureRequiredDistribution(
Review comment:
yeah we can always run it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902442032
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47165/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912454881
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47481/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r701214497
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -254,25 +259,40 @@ case class EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Ru
}
}
- def apply(plan: SparkPlan): SparkPlan = plan.transformUp {
- case operator @ ShuffleExchangeExec(upper: HashPartitioning, child, shuffleOrigin)
- if optimizeOutRepartition &&
- (shuffleOrigin == REPARTITION_BY_COL || shuffleOrigin == REPARTITION_BY_NUM) =>
- def hasSemanticEqualPartitioning(partitioning: Partitioning): Boolean = {
- partitioning match {
- case lower: HashPartitioning if upper.semanticEquals(lower) => true
- case lower: PartitioningCollection =>
- lower.partitionings.exists(hasSemanticEqualPartitioning)
- case _ => false
+ def apply(plan: SparkPlan): SparkPlan = {
+ val newPlan = plan.transformUp {
+ case operator @ ShuffleExchangeExec(upper: HashPartitioning, child, shuffleOrigin)
+ if optimizeOutRepartition &&
+ (shuffleOrigin == REPARTITION_BY_COL || shuffleOrigin == REPARTITION_BY_NUM) =>
+ def hasSemanticEqualPartitioning(partitioning: Partitioning): Boolean = {
+ partitioning match {
+ case lower: HashPartitioning if upper.semanticEquals(lower) => true
+ case lower: PartitioningCollection =>
+ lower.partitionings.exists(hasSemanticEqualPartitioning)
+ case _ => false
+ }
+ }
+ if (hasSemanticEqualPartitioning(child.outputPartitioning)) {
+ child
+ } else {
+ operator
}
- }
- if (hasSemanticEqualPartitioning(child.outputPartitioning)) {
- child
- } else {
- operator
- }
- case operator: SparkPlan =>
- ensureDistributionAndOrdering(reorderJoinPredicates(operator))
+ case operator: SparkPlan =>
+ ensureDistributionAndOrdering(reorderJoinPredicates(operator))
+ }
+
+ requiredDistribution match {
Review comment:
yeah, I considered about this approach, but not sure it's worth to do this integration. The requiredDistribution's shuffle origin is always different with the `ensureDistributionAndOrdering`, then if we want to merge them we need one more condition. Or if you have some other thought ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r701542919
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -640,14 +668,12 @@ case class AdaptiveSparkPlanExec(
* Re-optimize and run physical planning on the current logical plan based on the latest stats.
*/
private def reOptimize(
- logicalPlan: LogicalPlan): (SparkPlan, LogicalPlan) = context.qe.withCteMap {
+ logicalPlan: LogicalPlan): (Seq[SparkPlan], LogicalPlan) = context.qe.withCteMap {
logicalPlan.invalidateStatsCache()
val optimized = optimizer.execute(logicalPlan)
val sparkPlan = context.session.sessionState.planner.plan(ReturnAnswer(optimized)).next()
- val newPlan = applyPhysicalRules(
- sparkPlan,
- preprocessingRules ++ queryStagePreparationRules,
- Some((planChangeLogger, "AQE Replanning")))
+ val optimizedPhysicalPlan = prepareQueryStages(sparkPlan, false)
Review comment:
@maryannxue Do you mean the logic if we should apply `OptimizeSkewedJoin` + `EnsureRequirements` or not moves into `OptimizeSkewedJoin` itself ? If so, we aslo need add an extra cost evaluator check in `OptimizeSkewedJoin`. Seems it' not clearer than current approach that we only evaluate the cost in one place.
Am I following your thought ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877075687
**[Test build #140849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140849/testReport)** for PR 32816 at commit [`7734d3e`](https://github.com/apache/spark/commit/7734d3eec1078a56f600ce7b6e718824c09a0e3b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877257683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r692656713
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -100,24 +100,34 @@ case class AdaptiveSparkPlanExec(
// A list of physical plan rules to be applied before creation of query stages. The physical
// plan should reach a final status of query stages (i.e., no more addition or removal of
// Exchange nodes) after running these rules.
- @transient private val queryStagePreparationRules: Seq[Rule[SparkPlan]] = Seq(
- RemoveRedundantProjects,
- // For cases like `df.repartition(a, b).select(c)`, there is no distribution requirement for
- // the final plan, but we do need to respect the user-specified repartition. Here we ask
- // `EnsureRequirements` to not optimize out the user-specified repartition-by-col to work
- // around this case.
- EnsureRequirements(optimizeOutRepartition = requiredDistribution.isDefined),
- RemoveRedundantSorts,
- DisableUnnecessaryBucketedScan
- ) ++ context.session.sessionState.queryStagePrepRules
+ private def queryStagePreparationRules(
+ optimizeSkewedJoin: Boolean = false): Seq[Rule[SparkPlan]] = {
+ val optimizeSkewedJoinRules = if (optimizeSkewedJoin) {
+ Seq(OptimizeSkewedJoin,
+ // Add the EnsureRequirements rule here since OptimizeSkewedJoin will change
+ // output partitioning, make sure we have right distribution.
+ EnsureRequirements(optimizeOutRepartition = requiredDistribution.isDefined))
+ } else {
+ Nil
+ }
+
+ Seq(
+ RemoveRedundantProjects,
+ // For cases like `df.repartition(a, b).select(c)`, there is no distribution requirement for
+ // the final plan, but we do need to respect the user-specified repartition. Here we ask
+ // `EnsureRequirements` to not optimize out the user-specified repartition-by-col to work
+ // around this case.
+ EnsureRequirements(optimizeOutRepartition = requiredDistribution.isDefined),
+ RemoveRedundantSorts,
+ DisableUnnecessaryBucketedScan
+ ) ++ optimizeSkewedJoinRules ++ context.session.sessionState.queryStagePrepRules
Review comment:
We need two `EnsureRequirements` for `OptimizeSkewedJoin`.
The first `EnsureRequirements` is to add enough exchange for Join that ensure the physical plan can be executed. And the second `EnsureRequirements` is to make sure we have right distribution after `OptimizeSkewedJoin ` applied since `OptimizeSkewedJoin` can change the output partitioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906065584
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47278/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696352301
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -640,29 +660,44 @@ case class AdaptiveSparkPlanExec(
* Re-optimize and run physical planning on the current logical plan based on the latest stats.
*/
private def reOptimize(
- logicalPlan: LogicalPlan): (SparkPlan, LogicalPlan) = context.qe.withCteMap {
+ logicalPlan: LogicalPlan): (Seq[SparkPlan], LogicalPlan) = context.qe.withCteMap {
logicalPlan.invalidateStatsCache()
val optimized = optimizer.execute(logicalPlan)
val sparkPlan = context.session.sessionState.planner.plan(ReturnAnswer(optimized)).next()
- val newPlan = applyPhysicalRules(
+
+ val optimizedPhysicalPlan = applyPhysicalRules(
sparkPlan,
- preprocessingRules ++ queryStagePreparationRules,
+ preprocessingRules ++ queryStagePreparationRules(),
Some((planChangeLogger, "AQE Replanning")))
+ val optimizedWithSkewedJoin =
+ AQEUtils.ensureRequiredDistribution(
+ applyPhysicalRules(
Review comment:
shall we follow `optimizeQueryStage` and move this to a new method?
```
def prepareQueryStages(optimizeSkewJoin: Boolean) = {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856647927
**[Test build #139471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139471/testReport)** for PR 32816 at commit [`b02bc0f`](https://github.com/apache/spark/commit/b02bc0f149f8706e1c221d836538af87c49a16ea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906222059
**[Test build #142804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142804/testReport)** for PR 32816 at commit [`bc39694`](https://github.com/apache/spark/commit/bc39694258c88e338d41df7db51aa6b123f0a32a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906263140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906433033
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142801/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905266003
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902019650
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47147/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917537155
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47678/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902019650
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47147/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856774160
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856648180
**[Test build #139471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139471/testReport)** for PR 32816 at commit [`b02bc0f`](https://github.com/apache/spark/commit/b02bc0f149f8706e1c221d836538af87c49a16ea).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856812077
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139466/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905160429
**[Test build #142751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142751/testReport)** for PR 32816 at commit [`b215e2d`](https://github.com/apache/spark/commit/b215e2d0fdf959bd1c735297a1a33c1a078f2361).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915757149
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905389675
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47258/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914145069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47546/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857417815
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44056/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696410142
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -100,24 +100,34 @@ case class AdaptiveSparkPlanExec(
// A list of physical plan rules to be applied before creation of query stages. The physical
// plan should reach a final status of query stages (i.e., no more addition or removal of
// Exchange nodes) after running these rules.
- @transient private val queryStagePreparationRules: Seq[Rule[SparkPlan]] = Seq(
- RemoveRedundantProjects,
- // For cases like `df.repartition(a, b).select(c)`, there is no distribution requirement for
- // the final plan, but we do need to respect the user-specified repartition. Here we ask
- // `EnsureRequirements` to not optimize out the user-specified repartition-by-col to work
- // around this case.
- EnsureRequirements(optimizeOutRepartition = requiredDistribution.isDefined),
- RemoveRedundantSorts,
- DisableUnnecessaryBucketedScan
- ) ++ context.session.sessionState.queryStagePrepRules
+ private def queryStagePreparationRules(
+ optimizeSkewedJoin: Boolean = false): Seq[Rule[SparkPlan]] = {
Review comment:
this is mainly for `initalPlan` which is not needed optimize skewed join
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877257683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r706241249
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -38,26 +38,31 @@ import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoin
* but can be false in AQE when AQE optimization may change the plan
* output partitioning and need to retain the user-specified
* repartition shuffles in the plan.
+ * @param requiredDistribution The root required distribution we should ensure. This value is used
+ * in AQE in case we change final stage output partitioning.
*/
-case class EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Rule[SparkPlan] {
-
- private def ensureDistributionAndOrdering(operator: SparkPlan): SparkPlan = {
- val requiredChildDistributions: Seq[Distribution] = operator.requiredChildDistribution
- val requiredChildOrderings: Seq[Seq[SortOrder]] = operator.requiredChildOrdering
- var children: Seq[SparkPlan] = operator.children
- assert(requiredChildDistributions.length == children.length)
- assert(requiredChildOrderings.length == children.length)
+case class EnsureRequirements(
+ optimizeOutRepartition: Boolean = true,
+ requiredDistribution: Option[Distribution] = None)
+ extends Rule[SparkPlan] {
+ private def ensureDistributionAndOrdering(
+ originChildren: Seq[SparkPlan],
Review comment:
nit: `originalChildren`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917453740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r706241626
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -38,26 +38,31 @@ import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoin
* but can be false in AQE when AQE optimization may change the plan
* output partitioning and need to retain the user-specified
* repartition shuffles in the plan.
+ * @param requiredDistribution The root required distribution we should ensure. This value is used
+ * in AQE in case we change final stage output partitioning.
*/
-case class EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Rule[SparkPlan] {
-
- private def ensureDistributionAndOrdering(operator: SparkPlan): SparkPlan = {
- val requiredChildDistributions: Seq[Distribution] = operator.requiredChildDistribution
- val requiredChildOrderings: Seq[Seq[SortOrder]] = operator.requiredChildOrdering
- var children: Seq[SparkPlan] = operator.children
- assert(requiredChildDistributions.length == children.length)
- assert(requiredChildOrderings.length == children.length)
+case class EnsureRequirements(
+ optimizeOutRepartition: Boolean = true,
+ requiredDistribution: Option[Distribution] = None)
+ extends Rule[SparkPlan] {
+ private def ensureDistributionAndOrdering(
+ originChildren: Seq[SparkPlan],
Review comment:
Still, i'd prefer using `children` here and `newChildren` below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915757149
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917568896
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143175/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r703234376
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -250,15 +253,28 @@ object OptimizeSkewedJoin extends AQEShuffleReadRule {
// SHJ
// Shuffle
// Shuffle
- optimizeSkewJoin(plan)
+ val optimized =
+ EnsureRequirements(requiredDistribution.isDefined, requiredDistribution)
+ .apply(optimizeSkewJoin(plan))
+ val originCost = costEvaluator.evaluateCost(plan)
+ val optimizedCost = costEvaluator.evaluateCost(optimized)
+ // two cases we will pick new plan:
+ // 1. optimize the skew join without extra shuffle
+ // 2. optimize the skew join with extra shuffle but the costEvaluator think it's better
+ if (optimizedCost < originCost || (originCost == optimizedCost && optimized != plan)) {
Review comment:
how about just `optimizedCost <= originCost`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917526377
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902566875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914222462
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r706244614
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/simpleCosting.scala
##########
@@ -35,15 +36,48 @@ case class SimpleCost(value: Long) extends Cost {
}
/**
- * A simple implementation of [[CostEvaluator]], which counts the number of
- * [[ShuffleExchangeLike]] nodes in the plan.
+ * A skew join aware implementation of [[Cost]], which consider shuffle number and skew join number
Review comment:
Can we add more description on how the cost is calculated in the presence of skew joins? What happens if there's two or more extra shuffles by adding a skew join optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915722290
**[Test build #143101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143101/testReport)** for PR 32816 at commit [`ca63321`](https://github.com/apache/spark/commit/ca6332167f20ca85c4ae23100a475530f9118a8e).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917391659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47668/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858536436
**[Test build #139621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139621/testReport)** for PR 32816 at commit [`094ee0c`](https://github.com/apache/spark/commit/094ee0cf7d3895c97b85c1c6935a6d93e3175acd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856643015
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902563139
**[Test build #142664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142664/testReport)** for PR 32816 at commit [`8058fe9`](https://github.com/apache/spark/commit/8058fe9354ed28b32f0ea0bdc8c9243f79c58fba).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877107317
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45357/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906197637
**[Test build #142801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142801/testReport)** for PR 32816 at commit [`580a0a4`](https://github.com/apache/spark/commit/580a0a4548e6c25c8191dbf9de7256045bc86e0c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906594849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142804/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r695390047
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -640,29 +658,44 @@ case class AdaptiveSparkPlanExec(
* Re-optimize and run physical planning on the current logical plan based on the latest stats.
*/
private def reOptimize(
- logicalPlan: LogicalPlan): (SparkPlan, LogicalPlan) = context.qe.withCteMap {
+ logicalPlan: LogicalPlan): (Seq[SparkPlan], LogicalPlan) = context.qe.withCteMap {
logicalPlan.invalidateStatsCache()
val optimized = optimizer.execute(logicalPlan)
val sparkPlan = context.session.sessionState.planner.plan(ReturnAnswer(optimized)).next()
- val newPlan = applyPhysicalRules(
+
+ val optimizedPhysicalPlan = applyPhysicalRules(
sparkPlan,
- preprocessingRules ++ queryStagePreparationRules,
+ preprocessingRules ++ queryStagePreparationRules(),
Some((planChangeLogger, "AQE Replanning")))
+ val optimizedWithSkewedJoin =
+ AQEUtils.ensureRequiredDistribution(
+ applyPhysicalRules(
+ sparkPlan,
+ preprocessingRules ++ queryStagePreparationRules(true),
+ Some((planChangeLogger, "AQE Replanning With Optimize Skewed Join"))),
Review comment:
let's include it in the `AQE Replanning` phase as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906193681
**[Test build #142800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142800/testReport)** for PR 32816 at commit [`73f787b`](https://github.com/apache/spark/commit/73f787bd7faeffdb8f8db45bd884e228dc669126).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915755840
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47604/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-899239127
**[Test build #142493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142493/testReport)** for PR 32816 at commit [`6725f97`](https://github.com/apache/spark/commit/6725f975ca9f15eb8f233381fcfa9fc88cdef760).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906013667
**[Test build #142778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142778/testReport)** for PR 32816 at commit [`bc45d70`](https://github.com/apache/spark/commit/bc45d7040d5cd2a8017594605604dadceeea8fb2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902466437
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902424243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906433033
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142801/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902439939
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856774202
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856845591
**[Test build #139480 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139480/testReport)** for PR 32816 at commit [`ccc45ab`](https://github.com/apache/spark/commit/ccc45ab019beddc7eae0193f82883986a4a513fa).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917453740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914222462
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906230910
**[Test build #142800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142800/testReport)** for PR 32816 at commit [`73f787b`](https://github.com/apache/spark/commit/73f787bd7faeffdb8f8db45bd884e228dc669126).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r701542919
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -640,14 +668,12 @@ case class AdaptiveSparkPlanExec(
* Re-optimize and run physical planning on the current logical plan based on the latest stats.
*/
private def reOptimize(
- logicalPlan: LogicalPlan): (SparkPlan, LogicalPlan) = context.qe.withCteMap {
+ logicalPlan: LogicalPlan): (Seq[SparkPlan], LogicalPlan) = context.qe.withCteMap {
logicalPlan.invalidateStatsCache()
val optimized = optimizer.execute(logicalPlan)
val sparkPlan = context.session.sessionState.planner.plan(ReturnAnswer(optimized)).next()
- val newPlan = applyPhysicalRules(
- sparkPlan,
- preprocessingRules ++ queryStagePreparationRules,
- Some((planChangeLogger, "AQE Replanning")))
+ val optimizedPhysicalPlan = prepareQueryStages(sparkPlan, false)
Review comment:
@maryannxue Do you mean the logic if we should apply `OptimizeSkewedJoin` + `EnsureRequirements` or not moves into `OptimizeSkewedJoin` itself ? If so, we aslo need add a nextra cost evaluator check in `OptimizeSkewedJoin`. Seems it' not cleaner than current approach that we only evaluate the cost in one place.
Am I following your thought ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-918090786
thank you all !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906594849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142804/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914114540
**[Test build #143043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143043/testReport)** for PR 32816 at commit [`8961084`](https://github.com/apache/spark/commit/89610842f656d87f531a23337d027f6ec1f1ba08).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857362820
**[Test build #139532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139532/testReport)** for PR 32816 at commit [`4236958`](https://github.com/apache/spark/commit/42369588df8b4ae9bafa6be9b4889e5a33d9c7f6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858350225
**[Test build #139621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139621/testReport)** for PR 32816 at commit [`094ee0c`](https://github.com/apache/spark/commit/094ee0cf7d3895c97b85c1c6935a6d93e3175acd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905201936
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47252/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914314991
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143043/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905312587
**[Test build #142758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142758/testReport)** for PR 32816 at commit [`bc45d70`](https://github.com/apache/spark/commit/bc45d7040d5cd2a8017594605604dadceeea8fb2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-891782878
**[Test build #141993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141993/testReport)** for PR 32816 at commit [`30b7de0`](https://github.com/apache/spark/commit/30b7de03dc20ab8f2d9aaa07ed37a18d6081f672).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877140277
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45360/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-901988125
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47147/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r695402449
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEUtils.scala
##########
@@ -57,4 +58,20 @@ object AQEUtils {
}
case _ => Some(UnspecifiedDistribution)
}
+
+ // Add an extra shuffle if input plan does not satisfy the required distribution.
+ // This method is invoked after optimizing skewed join in case we change final stage
+ // output partitioning.
+ def ensureRequiredDistribution(
+ plan: SparkPlan, distribution: Option[Distribution]): SparkPlan = distribution match {
+ case Some(d) if !plan.outputPartitioning.satisfies(d) =>
+ val numPartitions = d.requiredNumPartitions.getOrElse(conf.numShufflePartitions)
+ val shuffleOrigin = if (d.requiredNumPartitions.isDefined) {
+ REPARTITION_BY_NUM
+ } else {
+ REPARTITION_BY_COL
+ }
+ ShuffleExchangeExec(d.createPartitioning(numPartitions), plan, shuffleOrigin)
Review comment:
We should retain origin shuffle here, otherwise some stage optimize rules can break the `ENSIRE_REQUIREMENT` output partitioning in next stage. e.g. `CoalesceShufflePartitions` when the input plan's origin is `REPARTITION_BY_NUM`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856609792
**[Test build #139466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139466/testReport)** for PR 32816 at commit [`368d1e0`](https://github.com/apache/spark/commit/368d1e0d3d20d763085a8a910ed6d6befe8fb9db).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898278242
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46926/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-891791382
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898249519
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46926/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898392464
**[Test build #142419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142419/testReport)** for PR 32816 at commit [`ba09cd4`](https://github.com/apache/spark/commit/ba09cd4be45f3394b4ab9d30ec878a11b6b1c081).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898204665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696350387
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEUtils.scala
##########
@@ -57,4 +58,20 @@ object AQEUtils {
}
case _ => Some(UnspecifiedDistribution)
}
+
+ // Add an extra shuffle if input plan does not satisfy the required distribution.
+ // This method is invoked after optimizing skewed join in case we change final stage
+ // output partitioning.
+ def ensureRequiredDistribution(
+ plan: SparkPlan, distribution: Option[Distribution]): SparkPlan = distribution match {
Review comment:
nit: can we pass in the `conf` as a parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905201936
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47252/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902556496
**[Test build #142662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142662/testReport)** for PR 32816 at commit [`b54e9c2`](https://github.com/apache/spark/commit/b54e9c232a43ce05e320d5459b16b770261ee38b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906445777
cc @maryannxue to take another look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906966854
**[Test build #142828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142828/testReport)** for PR 32816 at commit [`bc39694`](https://github.com/apache/spark/commit/bc39694258c88e338d41df7db51aa6b123f0a32a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r691844920
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -111,13 +111,22 @@ case class AdaptiveSparkPlanExec(
DisableUnnecessaryBucketedScan
) ++ context.session.sessionState.queryStagePrepRules
+ // This list rules are applied between queryStagePreparationRules and estimate physical plan cost
+ // so that we can support introduce extra shuffle
Review comment:
I'm thinking about a more efficient way that runs `EnsureRequirements` only once
```
def queryStagePreparationRules(optimizeSkewJoin: Boolean) = {
val rules = Seq(
RemoveRedundantProjects,
OptimizeSkewedJoin,
...
)
if (optimizeSkewJoin) rules else rules.filterNot(_ != OptimizeSkewedJoin)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r692077822
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -656,13 +687,54 @@ case class AdaptiveSparkPlanExec(
// node to prevent the loss of the `BroadcastExchangeExec` node in DPP subquery.
// Here, we also need to avoid to insert the `BroadcastExchangeExec` node when the newPlan
// is already the `BroadcastExchangeExec` plan after apply the `LogicalQueryStageStrategy` rule.
- val finalPlan = currentPhysicalPlan match {
+ def updateBroadcastExchange(plan: SparkPlan): SparkPlan = currentPhysicalPlan match {
case b: BroadcastExchangeLike
- if (!newPlan.isInstanceOf[BroadcastExchangeLike]) => b.withNewChildren(Seq(newPlan))
- case _ => newPlan
+ if (!plan.isInstanceOf[BroadcastExchangeLike]) => b.withNewChildren(Seq(plan))
+ case _ => plan
}
- (finalPlan, optimized)
+ val optimizedWithSkewedJoin = applyPhysicalRules(
+ optimizedPhysicalPlan,
+ optimizeSkewedJoinWithExtraShuffleRules,
+ Some((planChangeLogger, "AQE Optimize Skewed Join With Extra Shuffle"))
+ )
+ val validatedWithSkewedJoin =
+ checkDistribution(
+ optimizedWithSkewedJoin,
+ optimizedPhysicalPlan,
+ isFinalStage(optimizedWithSkewedJoin),
+ OptimizeSkewedJoin.ruleName)
+
+ // here are three reasons if validatedWithSkewedJoin is equal to optimizedPhysicalPlan:
+ // 1. no skewed join optimized
+ // 2. optimize skewed join introduce extra shuffle and force optimize is disabled
+ // 3. optimize skewed join change final stage output partitioning
+ val newPhysicalPlans = if (validatedWithSkewedJoin.fastEquals(optimizedPhysicalPlan)) {
+ updateBroadcastExchange(optimizedPhysicalPlan) :: Nil
Review comment:
@cloud-fan in these three case, we will not introduce extra overhead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r695392832
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -297,12 +307,20 @@ case class AdaptiveSparkPlanExec(
// plans are updated, we can clear the query stage list because at this point the two plans
// are semantically and physically in sync again.
val logicalPlan = replaceWithQueryStagesInLogicalPlan(currentLogicalPlan, stagesToReplace)
- val (newPhysicalPlan, newLogicalPlan) = reOptimize(logicalPlan)
- val origCost = costEvaluator.evaluateCost(currentPhysicalPlan)
- val newCost = costEvaluator.evaluateCost(newPhysicalPlan)
- if (newCost < origCost ||
- (newCost == origCost && currentPhysicalPlan != newPhysicalPlan)) {
- logOnLevel(s"Plan changed from $currentPhysicalPlan to $newPhysicalPlan")
+ val (newPhysicalPlans, newLogicalPlan) = reOptimize(logicalPlan)
+ val newPhysicalPlan =
+ (Seq(currentPhysicalPlan) ++ newPhysicalPlans)
Review comment:
I think the code can be
```
val preferredNewPhysicalPlan = ... // the plan with lowest cost and comes first
if (newCost < oldCost || ...) ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915754542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696358035
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/simpleCosting.scala
##########
@@ -35,15 +36,48 @@ case class SimpleCost(value: Long) extends Cost {
}
/**
- * A simple implementation of [[CostEvaluator]], which counts the number of
- * [[ShuffleExchangeLike]] nodes in the plan.
+ * A skew join aware implementation of [[Cost]], which consider shuffle number and skew join number
*/
-object SimpleCostEvaluator extends CostEvaluator {
+case class SkewJoinAwareCost(
+ numShuffles: Int,
+ numSkewJoins: Int) extends Cost {
+ override def compare(that: Cost): Int = that match {
+ case other: SkewJoinAwareCost =>
+ // If more skew joins are optimized or less shuffle nodes, it means the cost is lower
+ if (numSkewJoins > other.numSkewJoins) {
+ -1
+ } else if (numSkewJoins < other.numSkewJoins) {
+ 1
+ } else if (numShuffles < other.numShuffles) {
+ -1
+ } else if (numShuffles > other.numShuffles) {
+ 1
+ } else {
+ 0
+ }
+ case _ =>
+ throw QueryExecutionErrors.cannotCompareCostWithTargetCostError(that.toString)
+ }
+}
+
+/**
+ * A skew join aware implementation of [[CostEvaluator]], which counts the number of
+ * [[ShuffleExchangeLike]] nodes and skew join nodes in the plan.
+ */
+case class SimpleCostEvaluator(forceOptimizeSkewedJoin: Boolean) extends CostEvaluator {
override def evaluateCost(plan: SparkPlan): Cost = {
- val cost = plan.collect {
+ val shuffleNumber = plan.collect {
case s: ShuffleExchangeLike => s
}.size
- SimpleCost(cost)
+
+ if (forceOptimizeSkewedJoin) {
+ val skewJoinNumber = plan.collect {
Review comment:
nit: `numShuffles` and `numSkewJoins`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906970029
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142828/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-913156614
**[Test build #143001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143001/testReport)** for PR 32816 at commit [`23ebea0`](https://github.com/apache/spark/commit/23ebea0a408469f5c2a70e04982ffa6261dae96c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r662769610
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -252,17 +275,26 @@ case class AdaptiveSparkPlanExec(
// plans are updated, we can clear the query stage list because at this point the two plans
// are semantically and physically in sync again.
val logicalPlan = replaceWithQueryStagesInLogicalPlan(currentLogicalPlan, stagesToReplace)
- val (newPhysicalPlan, newLogicalPlan) = reOptimize(logicalPlan)
+ val (reOptimizePhysicalPlan, newLogicalPlan) = reOptimize(logicalPlan)
+ val planWithExtraShuffle = rePlanWithExtraShuffle(reOptimizePhysicalPlan)
val origCost = costEvaluator.evaluateCost(currentPhysicalPlan)
- val newCost = costEvaluator.evaluateCost(newPhysicalPlan)
- if (newCost < origCost ||
- (newCost == origCost && currentPhysicalPlan != newPhysicalPlan)) {
+ val newCost = costEvaluator.evaluateCost(reOptimizePhysicalPlan)
+ val extraShuffleCost = costEvaluator.evaluateCost(planWithExtraShuffle)
+ def updateCurrentPlan(newPhysicalPlan: SparkPlan): Unit = {
logOnLevel(s"Plan changed from $currentPhysicalPlan to $newPhysicalPlan")
cleanUpTempTags(newPhysicalPlan)
currentPhysicalPlan = newPhysicalPlan
currentLogicalPlan = newLogicalPlan
stagesToReplace = Seq.empty[QueryStageExec]
}
+
+ if (extraShuffleCost < newCost ||
+ (extraShuffleCost == newCost && planWithExtraShuffle != reOptimizePhysicalPlan)) {
+ updateCurrentPlan(planWithExtraShuffle)
+ } else if (newCost < origCost ||
+ (newCost == origCost && currentPhysicalPlan != reOptimizePhysicalPlan)) {
+ updateCurrentPlan(reOptimizePhysicalPlan)
+ }
Review comment:
Yes, and we pick one from these 3 with the lowest cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906887946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902153686
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142646/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877188314
**[Test build #140846 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140846/testReport)** for PR 32816 at commit [`8bc22ad`](https://github.com/apache/spark/commit/8bc22ad61f2a76da6a3430f5eebe9646cf5fb841).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906013667
**[Test build #142778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142778/testReport)** for PR 32816 at commit [`bc45d70`](https://github.com/apache/spark/commit/bc45d7040d5cd2a8017594605604dadceeea8fb2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-899551474
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142493/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906197637
**[Test build #142801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142801/testReport)** for PR 32816 at commit [`580a0a4`](https://github.com/apache/spark/commit/580a0a4548e6c25c8191dbf9de7256045bc86e0c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r662493000
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -91,13 +91,22 @@ case class AdaptiveSparkPlanExec(
DisableUnnecessaryBucketedScan
) ++ context.session.sessionState.queryStagePrepRules
Review comment:
we can make `queryStagePreparationRules` to take a parameter `optimzeSkewJoin: Boolean`, and call it twice in `reOptimize`, to produce 2 plans. Then I don't think we need the hack for the final stage anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r691851018
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -297,17 +315,26 @@ case class AdaptiveSparkPlanExec(
// plans are updated, we can clear the query stage list because at this point the two plans
// are semantically and physically in sync again.
val logicalPlan = replaceWithQueryStagesInLogicalPlan(currentLogicalPlan, stagesToReplace)
- val (newPhysicalPlan, newLogicalPlan) = reOptimize(logicalPlan)
+ val (reOptimizationPhysicalPlan, newLogicalPlan) = reOptimize(logicalPlan)
Review comment:
If the conf is false, the returned `Seq[SparkPlan]` will be Nil, so that we don't introduce extra overhead if force skew join is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-901885922
**[Test build #142645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142645/testReport)** for PR 32816 at commit [`60b7b9d`](https://github.com/apache/spark/commit/60b7b9da8f268f15706c30462976ffc389c95c45).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898285367
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46925/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r692075920
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/SkewJoinAwareCost.scala
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.exchange.ShuffleExchangeLike
+import org.apache.spark.sql.execution.joins.ShuffledJoin
+
+/**
+ * A skew join aware implementation of [[Cost]], which consider shuffle number and skew join number
+ */
+case class SkewJoinAwareCost(
+ numShuffles: Int,
+ numSkewJoins: Int) extends Cost {
+ override def compare(that: Cost): Int = that match {
+ case other: SkewJoinAwareCost =>
+ if (numSkewJoins > other.numSkewJoins || numShuffles < other.numShuffles) {
Review comment:
yes, if a plan has now skew join and less shuffle nodes than b plan, we will pick a plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r695392177
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -297,12 +307,20 @@ case class AdaptiveSparkPlanExec(
// plans are updated, we can clear the query stage list because at this point the two plans
// are semantically and physically in sync again.
val logicalPlan = replaceWithQueryStagesInLogicalPlan(currentLogicalPlan, stagesToReplace)
- val (newPhysicalPlan, newLogicalPlan) = reOptimize(logicalPlan)
- val origCost = costEvaluator.evaluateCost(currentPhysicalPlan)
- val newCost = costEvaluator.evaluateCost(newPhysicalPlan)
- if (newCost < origCost ||
- (newCost == origCost && currentPhysicalPlan != newPhysicalPlan)) {
- logOnLevel(s"Plan changed from $currentPhysicalPlan to $newPhysicalPlan")
+ val (newPhysicalPlans, newLogicalPlan) = reOptimize(logicalPlan)
+ val newPhysicalPlan =
+ (Seq(currentPhysicalPlan) ++ newPhysicalPlans)
Review comment:
Let's make the rule clear:
1. If the cost is the same, we prefer to pick the new physical plan
2. If the new physical plans have the same cost, we prefer to first plan (that has no skew join optimization)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902450754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898393759
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142419/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856865935
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139480/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857475508
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856748368
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856692484
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43994/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857389090
**[Test build #139541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139541/testReport)** for PR 32816 at commit [`e5526c5`](https://github.com/apache/spark/commit/e5526c524b5c794d8aa6f130d7dc73eaa1411ace).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858424083
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44148/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858542183
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139621/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-872709101
**[Test build #140561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140561/testReport)** for PR 32816 at commit [`9c985da`](https://github.com/apache/spark/commit/9c985da9870e107311cef0383a2a40703e4f4f07).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-901889753
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906233248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856646963
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43989/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905389675
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47258/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914399301
**[Test build #143045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143045/testReport)** for PR 32816 at commit [`5ba73c4`](https://github.com/apache/spark/commit/5ba73c4e2958e807b11f5de4f9be56ea4b854964).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-913175394
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47503/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906262361
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898204654
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898285339
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46925/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-913164745
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47503/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915877242
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143103/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898225440
**[Test build #142419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142419/testReport)** for PR 32816 at commit [`ba09cd4`](https://github.com/apache/spark/commit/ba09cd4be45f3394b4ab9d30ec878a11b6b1c081).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906887946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r692092985
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/SkewJoinAwareCost.scala
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.exchange.ShuffleExchangeLike
+import org.apache.spark.sql.execution.joins.ShuffledJoin
+
+/**
+ * A skew join aware implementation of [[Cost]], which consider shuffle number and skew join number
+ */
+case class SkewJoinAwareCost(
+ numShuffles: Int,
+ numSkewJoins: Int) extends Cost {
+ override def compare(that: Cost): Int = that match {
+ case other: SkewJoinAwareCost =>
+ if (numSkewJoins > other.numSkewJoins || numShuffles < other.numShuffles) {
Review comment:
Good point I think we should always pick the bigger skew join number first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912612898
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142980/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906421305
**[Test build #142801 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142801/testReport)** for PR 32816 at commit [`580a0a4`](https://github.com/apache/spark/commit/580a0a4548e6c25c8191dbf9de7256045bc86e0c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r691843423
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -677,6 +677,13 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val ADAPTIVE_FORCE_OPTIMIZE_SKEWED_JOIN =
+ buildConf("spark.sql.adaptive.forceOptimizeSkewedJoin")
+ .doc("When true, force enable OptimizeSkewedJoin even if it introduces extra shuffle.")
+ .version("3.2.0")
Review comment:
3.3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902443736
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47165/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912485320
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47481/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905230095
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-872711527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912612898
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142980/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914314991
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143043/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917529565
**[Test build #143175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143175/testReport)** for PR 32816 at commit [`f5e4b91`](https://github.com/apache/spark/commit/f5e4b911ea608cdf863215feb6e13996a57cb294).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-912431161
**[Test build #142980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142980/testReport)** for PR 32816 at commit [`4712986`](https://github.com/apache/spark/commit/4712986d217e515197a4f6bee1ae9790fda62087).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914190540
**[Test build #143045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143045/testReport)** for PR 32816 at commit [`5ba73c4`](https://github.com/apache/spark/commit/5ba73c4e2958e807b11f5de4f9be56ea4b854964).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r703233772
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -250,15 +253,28 @@ object OptimizeSkewedJoin extends AQEShuffleReadRule {
// SHJ
// Shuffle
// Shuffle
- optimizeSkewJoin(plan)
+ val optimized =
+ EnsureRequirements(requiredDistribution.isDefined, requiredDistribution)
Review comment:
shall we simply pass in the `EnsureRequirements` instance to `OptimizeSkewedJoin`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914107079
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906864442
**[Test build #142828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142828/testReport)** for PR 32816 at commit [`bc39694`](https://github.com/apache/spark/commit/bc39694258c88e338d41df7db51aa6b123f0a32a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905179205
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47252/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877217755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140846/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696361665
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##########
@@ -1908,6 +1914,69 @@ class AdaptiveQueryExecSuite
}
}
+ test("SPARK-33832: Support optimize skew join even if introduce extra shuffle") {
+ withSQLConf(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
+ SQLConf.ADAPTIVE_OPTIMIZE_SKEWS_IN_REBALANCE_PARTITIONS_ENABLED.key -> "false",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.SKEW_JOIN_SKEWED_PARTITION_THRESHOLD.key -> "100",
+ SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.key -> "100",
+ SQLConf.COALESCE_PARTITIONS_MIN_PARTITION_NUM.key -> "1",
+ SQLConf.SHUFFLE_PARTITIONS.key -> "10",
+ SQLConf.ADAPTIVE_FORCE_OPTIMIZE_SKEWED_JOIN.key -> "true") {
+ withTempView("skewData1", "skewData2") {
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 3 as key1", "id as value1")
+ .createOrReplaceTempView("skewData1")
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 1 as key2", "id as value2")
+ .createOrReplaceTempView("skewData2")
+
+ // check if optimized skewed join does not satisfy the required distribution
+ Seq(true, false).foreach { hasRequiredDistribution =>
+ Seq(true, false).foreach { hasPartitionNumber =>
+ val repartition = if (hasRequiredDistribution) {
+ s"/*+ repartition(${ if (hasPartitionNumber) "10," else ""}key1) */"
+ } else {
+ ""
+ }
+
+ // check required distribution and extra shuffle
+ val (_, adaptive1) =
+ runAdaptiveAndVerifyResult(s"SELECT $repartition key1 FROM skewData1 " +
+ s"JOIN skewData2 ON key1 = key2 GROUP BY key1")
+ val shuffles1 = findTopLevelShuffle(adaptive1)
+ assert(shuffles1.size == 3)
+ assert(shuffles1.head.shuffleOrigin == ENSURE_REQUIREMENTS)
+ val smj1 = findTopLevelSortMergeJoin(adaptive1)
+ assert(smj1.size == 1 && smj1.exists(_.isSkewJoin))
Review comment:
`smj1.exists(_.isSkewJoin)` -> `smj1.head.isSkewJoin`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856720169
**[Test build #139480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139480/testReport)** for PR 32816 at commit [`ccc45ab`](https://github.com/apache/spark/commit/ccc45ab019beddc7eae0193f82883986a4a513fa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857389090
**[Test build #139541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139541/testReport)** for PR 32816 at commit [`e5526c5`](https://github.com/apache/spark/commit/e5526c524b5c794d8aa6f130d7dc73eaa1411ace).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-857494775
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914219323
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915734918
**[Test build #143103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143103/testReport)** for PR 32816 at commit [`ca63321`](https://github.com/apache/spark/commit/ca6332167f20ca85c4ae23100a475530f9118a8e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902452047
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905251903
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47253/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906142371
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906849284
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r703317241
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -108,16 +114,17 @@ case class AdaptiveSparkPlanExec(
// around this case.
EnsureRequirements(optimizeOutRepartition = requiredDistribution.isDefined),
Review comment:
is it safer to pass in the `requiredDistribution` to this `EnsureRequirements` instance as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-914213889
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-917536377
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47678/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-905160429
**[Test build #142751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142751/testReport)** for PR 32816 at commit [`b215e2d`](https://github.com/apache/spark/commit/b215e2d0fdf959bd1c735297a1a33c1a078f2361).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #32816:
URL: https://github.com/apache/spark/pull/32816#discussion_r696399833
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -258,7 +258,8 @@ object OptimizeSkewedJoin extends AQEShuffleReadRule {
object ShuffleStage {
def unapply(plan: SparkPlan): Option[ShuffleQueryStageExec] = plan match {
- case s: ShuffleQueryStageExec if s.mapStats.isDefined && isSupported(s.shuffle) =>
+ case s: ShuffleQueryStageExec if s.isMaterialized && s.mapStats.isDefined &&
+ isSupported(s.shuffle) =>
Review comment:
removed the `AQEShuffleReadRule` and simplify the condition inline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856774202
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44003/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906226256
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-877094388
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45357/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-906864442
**[Test build #142828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142828/testReport)** for PR 32816 at commit [`bc39694`](https://github.com/apache/spark/commit/bc39694258c88e338d41df7db51aa6b123f0a32a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-898225440
**[Test build #142419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142419/testReport)** for PR 32816 at commit [`ba09cd4`](https://github.com/apache/spark/commit/ba09cd4be45f3394b4ab9d30ec878a11b6b1c081).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-856717755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43994/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915735936
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47604/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-858299743
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44135/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-902450734
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32816: [SPARK-33832][SQL] Support optimize skewed join even if introduce extra shuffle
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32816:
URL: https://github.com/apache/spark/pull/32816#issuecomment-915757136
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org