You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by jinxing64 <gi...@git.apache.org> on 2017/07/22 14:42:40 UTC

[GitHub] spark pull request #18713: [SPARK-21509][SQL] Add a config to enable adaptiv...

GitHub user jinxing64 opened a pull request:

    https://github.com/apache/spark/pull/18713

    [SPARK-21509][SQL] Add a config to enable adaptive query execution only for the last que…

    ## What changes were proposed in this pull request?
    Feature of adaptive query execution is a good way to avoid generating too many small files on HDFS, like mentioned in SPARK-16188.
    When feature of adaptive query execution is enabled, all shuffles will be coordinated. The drawbacks:
    1. It's hard to balance the num of reducers(this decides the processing speed) and file size on HDFS
    2. It generates some more shuffles(https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala#L101)
    3. It generates lots of jobs, which have extra cost for scheduling.
    
    We can add a config and enable adaptive query execution only for the last shuffle.
    
    ## How was this patch tested?
    
    Unit test.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jinxing64/spark SPARK-21509

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18713
    
----
commit efb8cd36de0e0b96a8efe855b6243c1f553ce195
Author: jinxing <ji...@126.com>
Date:   2017-07-22T04:22:36Z

    Add a config to enable adaptive query execution only for the last query execution.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18713: [SPARK-21509][SQL] Add a config to enable adaptive query...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18713
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79869/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18713: [SPARK-21509][SQL] Add a config to enable adaptive query...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18713
  
    **[Test build #79869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79869/testReport)** for PR 18713 at commit [`efb8cd3`](https://github.com/apache/spark/commit/efb8cd36de0e0b96a8efe855b6243c1f553ce195).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18713: [SPARK-21509][SQL] Add a config to enable adaptive query...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18713
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18713: [SPARK-21509][SQL] Add a config to enable adaptive query...

Posted by jinxing64 <gi...@git.apache.org>.

Github user jinxing64 commented on the issue:

    https://github.com/apache/spark/pull/18713
  
    cc @cloud-fan @jiangxb1987 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18713: [SPARK-21509][SQL] Add a config to enable adaptive query...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18713
  
    **[Test build #79869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79869/testReport)** for PR 18713 at commit [`efb8cd3`](https://github.com/apache/spark/commit/efb8cd36de0e0b96a8efe855b6243c1f553ce195).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18713: [SPARK-21509][SQL] Add a config to enable adaptive query...

Posted by cloud-fan <gi...@git.apache.org>.

Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/18713
  
    adaptive query is really an experimental and uncompleted feature, I'm hesitant to modify it unless we have a holistic plan about how to improve it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18713: [SPARK-21509][SQL] Add a config to enable adaptive query...

Posted by jinxing64 <gi...@git.apache.org>.

Github user jinxing64 commented on the issue:

    https://github.com/apache/spark/pull/18713
  
    Ok, I will close this for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org