You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by lipzhu <gi...@git.apache.org> on 2018/10/18 11:21:06 UTC

[GitHub] spark pull request #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.1...

GitHub user lipzhu opened a pull request:

    https://github.com/apache/spark/pull/22765

    [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final to 4.1.30.Final

    ## What changes were proposed in this pull request?
    Upgrade netty dependency from 4.1.17 to 4.1.30.
    
    Explanation:
    Currently when sending a ChunkedByteBuffer with more than 16 chunks over the network will trigger a "merge" of all the blocks into one big transient array that is then sent over the network. This is problematic as the total memory for all chunks can be high (2GB) and this would then trigger an allocation of 2GB to merge everything, which will create OOM errors.
    And we can avoid this issue by upgrade the netty. https://github.com/netty/netty/pull/8038
    
    
    ## How was this patch tested?
    
    Manual tests in some spark jobs.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lipzhu/spark SPARK-25757

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22765.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22765
    
----
commit 0753c0fe907070b50812b3fb40868dfcdb2bd969
Author: Zhu, Lipeng <li...@...>
Date:   2018-10-17T11:58:05Z

    upgrade netty-all from 4.1.17.Final to 4.1.30.Final

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97543/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Thank you, @lipzhu , @srowen , @HyukjinKwon , @felixcheung , and @kiszk .
    Let's merge this and proceed `removing Netty 3.x` in a different PR.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    **[Test build #97672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97672/testReport)** for PR 22765 at commit [`0753c0f`](https://github.com/apache/spark/commit/0753c0fe907070b50812b3fb40868dfcdb2bd969).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Retest this please.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    **[Test build #97666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97666/testReport)** for PR 22765 at commit [`0753c0f`](https://github.com/apache/spark/commit/0753c0fe907070b50812b3fb40868dfcdb2bd969).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    **[Test build #97543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97543/testReport)** for PR 22765 at commit [`0753c0f`](https://github.com/apache/spark/commit/0753c0fe907070b50812b3fb40868dfcdb2bd969).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by kiszk <gi...@git.apache.org>.

Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    That's probably fine.
    As an aside, I think we can remove all instances of Netty 3.x in the code base, if any, now that Flume is gone.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97672/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by lipzhu <gi...@git.apache.org>.

Github user lipzhu commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    cc @srowen @zsxwing @HyukjinKwon 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97666/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    **[Test build #97666 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97666/testReport)** for PR 22765 at commit [`0753c0f`](https://github.com/apache/spark/commit/0753c0fe907070b50812b3fb40868dfcdb2bd969).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    **[Test build #97543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97543/testReport)** for PR 22765 at commit [`0753c0f`](https://github.com/apache/spark/commit/0753c0fe907070b50812b3fb40868dfcdb2bd969).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by kiszk <gi...@git.apache.org>.

Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    LGTM, pending Jenkins


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    **[Test build #97672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97672/testReport)** for PR 22765 at commit [`0753c0f`](https://github.com/apache/spark/commit/0753c0fe907070b50812b3fb40868dfcdb2bd969).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    ok to test.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22765: [SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22765
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org