You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/07 14:48:34 UTC

[GitHub] [spark] zhongyu09 opened a new pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

zhongyu09 opened a new pull request #31084:
URL: https://github.com/apache/spark/pull/31084


   This PR is the same as https://github.com/apache/spark/pull/30998 to merge to branch 3.0
   
   ### What changes were proposed in this pull request?
   In AdaptiveSparkPlanExec.getFinalPhysicalPlan, when newStages are generated, sort the new stages by class type to make sure BroadcastQueryState precede others.
   It can make sure the broadcast job are submitted before map jobs to avoid waiting for job schedule and cause broadcast timeout. 
   
   ### Why are the changes needed?
   When enable AQE, in getFinalPhysicalPlan, spark traversal the physical plan bottom up and create query stage for materialized part by createQueryStages and materialize those new created query stages to submit map stages or broadcasting. When ShuffleQueryStage are materializing before BroadcastQueryStage, the map job and broadcast job are submitted almost at the same time, but map job will hold all the computing resources. If the map job runs slow (when lots of data needs to process and the resource is limited), the broadcast job cannot be started(and finished) before spark.sql.broadcastTimeout, thus cause whole job failed (introduced in SPARK-31475).
   The workaround to increase spark.sql.broadcastTimeout doesn't make sense and graceful, because the data to broadcast is very small.
   
   
   ### Does this PR introduce _any_ user-facing change?
   NO
   
   
   ### How was this patch tested?
   1. Add UT
   2. Test the code using dev environment in https://issues.apache.org/jira/browse/SPARK-33933


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756571492


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38409/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756175999


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756991818


   Got it. I'm not sure about removing UTs. BTW, Jenkins passed at least.
   
   WDYT, @cloud-fan and @HyukjinKwon ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757028611


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38446/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757020946


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38446/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757028598


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38446/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757415994


   I reverted it for now for RC preparation . Let's make a PR with clarifying which case it doesn't cover, and why this is a partial fix.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756649006






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757028611


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38446/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756551294


   **[Test build #133820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133820/testReport)** for PR 31084 at commit [`03c9b09`](https://github.com/apache/spark/commit/03c9b091b79b5f98d54eef912b6d3fc377ab7f9c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756672359


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133820/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756571511


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38409/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] zhongyu09 commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
zhongyu09 commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756599168


   Hi @dongjoon-hyun 
   I try the test tens of times and the test failed twice. 
   As we discussed with @viirya and @cloud-fan in https://github.com/apache/spark/pull/30998, the solution is not perfect. The order of calling materialize can guarantee that the order of task to be scheduled in normal circumstances, but, the guarantee is not strict since the submit of broadcast job and shuffle map job are in different thread. So there's still risk for the shuffle map job schedule earlier before broadcast job. I wonder should we need to remove the UT until we thorough resolve the issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757059034


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133857/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31084:
URL: https://github.com/apache/spark/pull/31084


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757096494


   The same test case failed in Jenkins Maven environment again. (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133857/testReport)
   ```
   - SPARK-33933: AQE broadcast should not timeout with slow map tasks *** FAILED ***
     1693 was not greater than 2000 (AdaptiveQueryExecSuite.scala:926)
   14:59:38.190 WARN org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite: 
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756467472


   The failure is relevant to AQE. Could you take a look at it, @zhongyu09 ?
   ![Screen Shot 2021-01-07 at 4 19 54 PM](https://user-images.githubusercontent.com/9700541/103959320-396b8980-5104-11eb-94d2-48fa508d5cb1.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756551294


   **[Test build #133820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133820/testReport)** for PR 31084 at commit [`03c9b09`](https://github.com/apache/spark/commit/03c9b091b79b5f98d54eef912b6d3fc377ab7f9c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756672359


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133820/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757058967


   **[Test build #133857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133857/testReport)** for PR 31084 at commit [`03c9b09`](https://github.com/apache/spark/commit/03c9b091b79b5f98d54eef912b6d3fc377ab7f9c).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757096581


   I'm wondering if this test is stable in `master` and `branch-3.1`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756547625


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756649006


   **[Test build #133820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133820/testReport)** for PR 31084 at commit [`03c9b09`](https://github.com/apache/spark/commit/03c9b091b79b5f98d54eef912b6d3fc377ab7f9c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] zhongyu09 commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
zhongyu09 commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757995580


   > I reverted it for now for RC preparation . Let's make a PR with clarifying which case it doesn't cover, and why this is a partial fix.
   
   For partial fix, it is difficult to give an stable UT. I would rather give an stable fix. I think two directions:
   1.  make sure broadcast job is submitted before shuffle map job, the calling of materialize() for non-broadcast query stage should wait until all the broadcast jobs are submitted. 
   2. excluded the schedule time for broadcast job when we calculate time out. This is very hard to measure. For downgrade, perhaps we can measure the time for pure broadcast, that is, minus collect time. But this also has big changes, as well as changes for non-AQE.
   
   I prefer for #1, it behavior more like non-AQE and is this PR's original intention and will have less impact to non-AQE.
   
        


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757403648


   A partial fix is fine but let's make sure to mention/document what cases the partial fix does not cover.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756991818






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756563807


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38409/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757002672


   **[Test build #133857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133857/testReport)** for PR 31084 at commit [`03c9b09`](https://github.com/apache/spark/commit/03c9b091b79b5f98d54eef912b6d3fc377ab7f9c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] zhongyu09 commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
zhongyu09 commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756502599


   > The failure is relevant to AQE. Could you take a look at it, @zhongyu09 ?
   > ![Screen Shot 2021-01-07 at 4 19 54 PM](https://user-images.githubusercontent.com/9700541/103959320-396b8980-5104-11eb-94d2-48fa508d5cb1.png)
   
   I will have a look.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756992150


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] zhongyu09 commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
zhongyu09 commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757101007


   > I'm wondering if this test is stable in `master` and `branch-3.1`.
   
   Yes, also not stable in `master`. I create a PR to remove the UT.  https://github.com/apache/spark/pull/31099


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757403648


   A partial fix is fine but let's make sure to mention/document what cases the partial fix does not cover.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756571511


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38409/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756672359






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756175999


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756991818


   Got it. I'm not sure about removing UTs.
   
   WDYT, @cloud-fan and @HyukjinKwon ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756672359






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757059034


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133857/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757002672


   **[Test build #133857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133857/testReport)** for PR 31084 at commit [`03c9b09`](https://github.com/apache/spark/commit/03c9b091b79b5f98d54eef912b6d3fc377ab7f9c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756504646


   Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] zhongyu09 commented on pull request #31084: [SPARK-33933][SQL][3.0][test-maven] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
zhongyu09 commented on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-757101007


   > I'm wondering if this test is stable in `master` and `branch-3.1`.
   
   Yes, also not stable in `master`. I create a PR to remove the UT.  https://github.com/apache/spark/pull/31099


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756551294






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31084: [SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #31084:
URL: https://github.com/apache/spark/pull/31084#issuecomment-756991818


   Got it. I'm not sure about removing UTs. BTW, Jenkins passed at least.
   
   WDYT, @cloud-fan and @HyukjinKwon ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org