You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by 10110346 <gi...@git.apache.org> on 2018/04/16 12:09:10 UTC

[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...

GitHub user 10110346 opened a pull request:

    https://github.com/apache/spark/pull/21079

    [SPARK-23992][CORE] ShuffleDependency does not need to be deserialized every time

    ## What changes were proposed in this pull request?
    
    In the same stage, 'ShuffleDependency' is not necessary to be deserialized each time.
    
    ## How was this patch tested?
    Existing unit test


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/10110346/spark notdeserializedep

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21079
    
----
commit 2239e786cf0b631457ce6c6acfa173e21e314e61
Author: liuxian <li...@...>
Date:   2018-04-16T11:48:09Z

    fix

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    **[Test build #89394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89394/testReport)** for PR 21079 at commit [`2239e78`](https://github.com/apache/spark/commit/2239e786cf0b631457ce6c6acfa173e21e314e61).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89394/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 closed the pull request at:

    https://github.com/apache/spark/pull/21079


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2343/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    **[Test build #89433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89433/testReport)** for PR 21079 at commit [`c647f0a`](https://github.com/apache/spark/commit/c647f0a461156d658e5771a0af2e07bb4a837530).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Have you seen any performance improvement with this change?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21079#discussion_r181942230
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
    @@ -113,3 +118,24 @@ private[spark] class ShuffleMapTask(
     
       override def toString: String = "ShuffleMapTask(%d, %d)".format(stageId, partitionId)
     }
    +
    +object ShuffleMapTask extends Logging {
    +  private val cache = CacheBuilder.newBuilder()
    --- End diff --
    
    we don't need to clear, this cache can  recovery itself


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    **[Test build #89394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89394/testReport)** for PR 21079 at commit [`2239e78`](https://github.com/apache/spark/commit/2239e786cf0b631457ce6c6acfa173e21e314e61).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    ping @10110346 for ^


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2374/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94505/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89433/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    **[Test build #89433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89433/testReport)** for PR 21079 at commit [`c647f0a`](https://github.com/apache/spark/commit/c647f0a461156d658e5771a0af2e07bb4a837530).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    **[Test build #94505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94505/testReport)** for PR 21079 at commit [`c647f0a`](https://github.com/apache/spark/commit/c647f0a461156d658e5771a0af2e07bb4a837530).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:

    https://github.com/apache/spark/pull/21079
  
     I'm sorry to be so late reply to you!
    I have tested in my production environment,  it has a bit of performance improvement. @jiangxb1987 @HyukjinKwon


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21079: [SPARK-23992][CORE] ShuffleDependency does not ne...

Posted by Ngone51 <gi...@git.apache.org>.
Github user Ngone51 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21079#discussion_r181938225
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
    @@ -113,3 +118,24 @@ private[spark] class ShuffleMapTask(
     
       override def toString: String = "ShuffleMapTask(%d, %d)".format(stageId, partitionId)
     }
    +
    +object ShuffleMapTask extends Logging {
    +  private val cache = CacheBuilder.newBuilder()
    --- End diff --
    
    Do we need to clear this `cache` at the end of a app ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    **[Test build #94505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94505/testReport)** for PR 21079 at commit [`c647f0a`](https://github.com/apache/spark/commit/c647f0a461156d658e5771a0af2e07bb4a837530).
     * This patch **fails Spark unit tests**.
     * This patch **does not merge cleanly**.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21079: [SPARK-23992][CORE] ShuffleDependency does not need to b...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/21079
  
    Mind if I ask to share the number and post it in the PR description?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org