You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by mukulmurthy <gi...@git.apache.org> on 2018/08/21 23:21:10 UTC

[GitHub] spark pull request #22176: [SPARK-25181][CORE] Limit Thread Pool size in Blo...

GitHub user mukulmurthy opened a pull request:

    https://github.com/apache/spark/pull/22176

    [SPARK-25181][CORE] Limit Thread Pool size in BlockManager Master and Slave endpoints

    ## What changes were proposed in this pull request?
    
    Limit Thread Pool size in BlockManager Master and Slave endpoints.
    
    Currently, BlockManagerMasterEndpoint and BlockManagerSlaveEndpoint both have thread pools with nearly unbounded (Integer.MAX_VALUE) numbers of threads. In certain cases, this can lead to driver OOM errors. This change limits the thread pools to 100 threads; this should not break any existing behavior because any tasks beyond that number will get queued.
    
    ## How was this patch tested?
    
    Manual testing
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mukulmurthy/oss-spark 25181-threads

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22176.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22176
    
----
commit c2223c3862619d2191ea787f3a2ee3c0d8d67ff2
Author: Mukul Murthy <mu...@...>
Date:   2018-08-21T23:16:11Z

    Limit thread pools in BlockManager Master and Slave endpoints

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22176: [SPARK-25181][CORE] Limit Thread Pool size in BlockManag...

Posted by zsxwing <gi...@git.apache.org>.

Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/22176
  
    LGTM. Merging to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22176: [SPARK-25181][CORE] Limit Thread Pool size in BlockManag...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22176
  
    **[Test build #95066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95066/testReport)** for PR 22176 at commit [`c2223c3`](https://github.com/apache/spark/commit/c2223c3862619d2191ea787f3a2ee3c0d8d67ff2).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22176: [SPARK-25181][CORE] Limit Thread Pool size in BlockManag...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22176
  
    **[Test build #95057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95057/testReport)** for PR 22176 at commit [`c2223c3`](https://github.com/apache/spark/commit/c2223c3862619d2191ea787f3a2ee3c0d8d67ff2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22176: [SPARK-25181][CORE] Limit Thread Pool size in BlockManag...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22176
  
    **[Test build #95066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95066/testReport)** for PR 22176 at commit [`c2223c3`](https://github.com/apache/spark/commit/c2223c3862619d2191ea787f3a2ee3c0d8d67ff2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22176: [SPARK-25181][CORE] Limit Thread Pool size in BlockManag...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22176
  
    **[Test build #95057 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95057/testReport)** for PR 22176 at commit [`c2223c3`](https://github.com/apache/spark/commit/c2223c3862619d2191ea787f3a2ee3c0d8d67ff2).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22176: [SPARK-25181][CORE] Limit Thread Pool size in Blo...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/22176


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org