You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by 10110346 <gi...@git.apache.org> on 2018/08/10 04:27:26 UTC

[GitHub] spark pull request #22065: [SPARK-23992][CORE] ShuffleDependency does not ne...

GitHub user 10110346 opened a pull request:

    https://github.com/apache/spark/pull/22065

    [SPARK-23992][CORE] ShuffleDependency does not need to be deserialized every time

    In the same stage, 'ShuffleDependency' is not necessary to be deserialized each time.
    
    I have tested 3 times in my production environment , it has a bit of performance improvement( about 0.7%):
    Before this PR：
    duration: 20189(s)
    App count: 4736
    the sum of time for all Apps: 530470(s)
    
    Afterthis PR：
    duration: 20035(s)
    the count of Apps : 4736
    the sum of time for all Apps: 525220(s)
    How was this patch tested?
    
    Existing unit test

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/10110346/spark notdeserializedep22

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22065.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22065
    
----
commit a99769dd1aac779e972ed2e23aa7598e6d7c7105
Author: liuxian <li...@...>
Date:   2018-08-10T04:20:54Z

    fix

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #96671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96671/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    This is end-to-end performance improvement, although our data is very small.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #94714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94714/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Are we talking about a 0.7% margin improvement? It doesn't seem like it's worth the complexity.



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #94541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94541/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #94541 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94541/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2154/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2029/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #94551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94551/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #94551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94551/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #96671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96671/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2037/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94551/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94541/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94714/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96671/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22065: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22065
  
    **[Test build #94714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94714/testReport)** for PR 22065 at commit [`a99769d`](https://github.com/apache/spark/commit/a99769dd1aac779e972ed2e23aa7598e6d7c7105).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org