You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/25 05:12:40 UTC

[GitHub] [spark] maryannxue opened a new pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

maryannxue opened a new pull request #30494:
URL: https://github.com/apache/spark/pull/30494


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   This PR fixes an AQE issue where local shuffle reader, partition coalescing, or skew join optimization can be mistakenly applied to a shuffle introduced by repartition or a regular shuffle that logically replaces a repartition shuffle.
   The proposed solution checks for the presence of any repartition shuffle and filters out not applicable optimization rules for the final stage in an AQE plan.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   Without the change, the output of a repartition query may not be correct.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   No.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   Added UT.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733955659


   **[Test build #131789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131789/testReport)** for PR 30494 at commit [`366f703`](https://github.com/apache/spark/commit/366f703a8026ba99ee349b016211eb08bbd0b56b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `case class CoalesceShufflePartitions(session: SparkSession) extends CustomShuffleReaderRule `
     * `trait CustomShuffleReaderRule extends Rule[SparkPlan] `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733470435


   **[Test build #131738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131738/testReport)** for PR 30494 at commit [`1a6628e`](https://github.com/apache/spark/commit/1a6628eb9cc35e51dfda601e11d56288309955c4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
manuzhang commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734731545


   If it's not the final stage, `OptimizeLocalShuffleReader` will not be applied to the probe side when the output partitioning can not `ensureRequirements`. I think we lack this `ensureRequirements` for the final stage, which is usually the requirement of target table.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733637339






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733486871






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
manuzhang commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734270568


   @maryannxue, @cloud-fan, 
   
   Sorry for not raising up earlier but I'd like to discuss a case which seems to not have been covered here.
   
   1. The final stage *before write* is a `SortMergeJoin` with partitioning that match the target table.
   2. AQE switches the `SortMergeJoin` to `BroadcastHashJoin` as one side is smaller than broadcast threshold.
   3. The probe side, with a different partitioning, is applied `OptimizeLocalShuffleReader`, which breaks the user intended partitioning.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733486871






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733835324


   **[Test build #131789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131789/testReport)** for PR 30494 at commit [`366f703`](https://github.com/apache/spark/commit/366f703a8026ba99ee349b016211eb08bbd0b56b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733680528


   It's unfortunate that we can't retain the shuffle origin info when we optimize out the repartition shuffle. The approach here looks like a good compromise.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
manuzhang commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-735518365


   > To be clear, we can't make any guarantee about the internal shuffles, as it's not reliable (depends on join strategy, aggregate strategy, etc.)
   
   When it's the final stage, it's crossing the boundary and interacting with target table and downstream jobs. It's no longer internal and I think we need have some guarantee here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733927636


   **[Test build #131810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131810/testReport)** for PR 30494 at commit [`97f58f5`](https://github.com/apache/spark/commit/97f58f5744ace0d869f3660eec4f05a48e710e97).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734692905


   then it's not a user-specified partitioning and we don't need to respect it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
manuzhang commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-735539228


   @maryannxue, @cloud-fan 
   I got your point. I just wish we could do more to create less surprise for users.
   Anyway, thanks for patiently answering my questions through holidays.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30494:
URL: https://github.com/apache/spark/pull/30494#discussion_r530339227



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -55,6 +55,11 @@ import org.apache.spark.sql.internal.SQLConf
  */
 object OptimizeSkewedJoin extends Rule[SparkPlan] {
 
+  /**
+   * The list of [[ShuffleOrigin]]s supported by this rule.
+   */
+  val supportedShuffleOrigins: Seq[ShuffleOrigin] = Seq(ENSURE_REQUIREMENTS)

Review comment:
       should the rule `OptimizeSkewedJoin` itself check it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maryannxue commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
maryannxue commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-735522219


   The output partitioning and sort order are totally implementation
   dependent. Say an SMJ would preserve partitioning from both sides while a
   BHJ would only do one side. And we could well have any random join
   implementation that does not preserve partititioning from either side. And
   how can spark possibly guarantee that?
   
   On the other hand, if the user asks for a certain partitioning, they should
   specify it by using "repartition", and it's spark's job to optimize it out
   if possible.
   
   On Sun, Nov 29, 2020, 9:05 PM Manu Zhang <no...@github.com> wrote:
   
   > To be clear, we can't make any guarantee about the internal shuffles, as
   > it's not reliable (depends on join strategy, aggregate strategy, etc.)
   >
   > When it's the final stage, it's crossing the boundary and interacting with
   > target table and downstream jobs. It's no longer internal and I think we
   > need have some guarantee here.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/30494#issuecomment-735518365>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AA72RAFSGGYH4KUTSI4BOZLSSMDYDANCNFSM4UB37LRA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733470435


   **[Test build #131738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131738/testReport)** for PR 30494 at commit [`1a6628e`](https://github.com/apache/spark/commit/1a6628eb9cc35e51dfda601e11d56288309955c4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan edited a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734741387


   The final stage can be just `df.rdd()` where there is no target table. To be clear, we can't make any guarantee about the internal shuffles, as it's not reliable (depends on join strategy, aggregate strategy, etc.). We only need to respect user-given `repartition`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733637319






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733897718






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734715842


   > why ? User set spark.sql.shuffle.partitions
   
   Then we can't do AQE at all as it changes the number of partitions...
   
   `spark.sql.shuffle.partitions` is a config for internal shuffles, and it does not have any guarantee for user-visible behaviors.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733927636


   **[Test build #131810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131810/testReport)** for PR 30494 at commit [`97f58f5`](https://github.com/apache/spark/commit/97f58f5744ace0d869f3660eec4f05a48e710e97).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733999514






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733897718






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734010117


   **[Test build #131810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131810/testReport)** for PR 30494 at commit [`97f58f5`](https://github.com/apache/spark/commit/97f58f5744ace0d869f3660eec4f05a48e710e97).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
manuzhang commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734702792


   > then it's not a user-specified partitioning and we don't need to respect it.
   
   why ? User set  `spark.sql.shuffle.partitions`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
manuzhang commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734686995


   @cloud-fan 
   No explicit `repartition` but implicit through join with `spark.sql.shuffle.partitions` configuration. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733480069


   **[Test build #131738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131738/testReport)** for PR 30494 at commit [`1a6628e`](https://github.com/apache/spark/commit/1a6628eb9cc35e51dfda601e11d56288309955c4).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gatorsmile closed pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
gatorsmile closed pull request #30494:
URL: https://github.com/apache/spark/pull/30494


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] manuzhang edited a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
manuzhang edited a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734270568


   @maryannxue, @cloud-fan, 
   
   Sorry for not raising up earlier but I'd like to discuss a case which seems to not have been covered here.
   
   1. The final stage *before write* is a `SortMergeJoin` with partitioning that matches the target table.
   2. AQE switches the `SortMergeJoin` to `BroadcastHashJoin` as one side is smaller than broadcast threshold.
   3. The probe side, with a different partitioning, is applied `OptimizeLocalShuffleReader`, which breaks the user intended partitioning.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734741387


   The final stage can be just `df.collect()` where there is no target table. To be clear, we can't make any guarantee about the internal shuffles, as it's not reliable (depends on join strategy, aggregate strategy, etc.). We only need to respect user-given `repartition`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maryannxue commented on a change in pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #30494:
URL: https://github.com/apache/spark/pull/30494#discussion_r530490828



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -55,6 +55,11 @@ import org.apache.spark.sql.internal.SQLConf
  */
 object OptimizeSkewedJoin extends Rule[SparkPlan] {
 
+  /**
+   * The list of [[ShuffleOrigin]]s supported by this rule.
+   */
+  val supportedShuffleOrigins: Seq[ShuffleOrigin] = Seq(ENSURE_REQUIREMENTS)

Review comment:
       Yes, we better do, to make it more robust, although it's not really necessary here as we exclude these rules if there's been a repartition shuffle.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733966636






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733835324


   **[Test build #131789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131789/testReport)** for PR 30494 at commit [`366f703`](https://github.com/apache/spark/commit/366f703a8026ba99ee349b016211eb08bbd0b56b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maryannxue commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
maryannxue commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734672087


   I think the case you mentioned is exactly one of the test cases. Can you
   find a case that is not working? And if repartition is not specified, we
   should not guarantee any partitioning.
   
   On Fri, Nov 27, 2020, 12:31 AM Wenchen Fan <no...@github.com> wrote:
   
   > is repartition involved in your case?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/30494#issuecomment-734669644>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AA72RADPGEPC7DPPEB73FN3SR5BV5ANCNFSM4UB37LRA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] maryannxue commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
maryannxue commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733469415


   cc @cloud-fan @manuzhang 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-733966636






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734669644


   is `repartition` involved in your case?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gatorsmile commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition

Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #30494:
URL: https://github.com/apache/spark/pull/30494#issuecomment-734053423


   Thanks! Merged to master


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org