You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2018/03/05 13:12:25 UTC

[GitHub] spark pull request #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-c...

GitHub user wangyum opened a pull request:

    https://github.com/apache/spark/pull/20735

    [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-enabled option to memLimitExceededLogMessage

    ## What changes were proposed in this pull request?
    My spark application sometimes will throw `Container killed by YARN for exceeding memory limits`.
    Even I increased `spark.yarn.executor.memoryOverhead` to 10G, this error still happen.  The latest config:
    <img width="685" alt="memory-config" src="https://user-images.githubusercontent.com/5399861/36975716-f5c548d2-20b5-11e8-95e5-b228d50917b9.png">
    
    And error message:
    ```
    ExecutorLostFailure (executor 121 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 30.7 GB of 30 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
    ```
    
    This is because of [Linux glibc >= 2.10 (RHEL 6) malloc may show excessive virtual memory usage](https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en). So disable `yarn.nodemanager.vmem-check-enabled` looks like a good option as [MapR mentioned ](https://mapr.com/blog/best-practices-yarn-resource-management).
    
    This PR add disable `yarn.nodemanager.vmem-check-enabled` option to memLimitExceededLogMessage.
    
    More details:
    https://issues.apache.org/jira/browse/YARN-4714
    https://stackoverflow.com/a/31450291
    https://stackoverflow.com/a/42091255
    
    After this PR:
    <img width="898" alt="yarn" src="https://user-images.githubusercontent.com/5399861/36975949-c8e7bbbe-20b6-11e8-9513-9f903b868d8d.png">
    
    ## How was this patch tested?
    
    N/A


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/wangyum/spark YARN-4714

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20735.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20735
    
----
commit 3fc05b4f8599ee65e8c4f808aee238d212c22b17
Author: Yuming Wang <yu...@...>
Date:   2018-03-05T12:38:21Z

    Update memLimitExceededLogMessage

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    LGTM, merging to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-c...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20735#discussion_r172732614
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
    @@ -736,7 +736,8 @@ private object YarnAllocator {
       def memLimitExceededLogMessage(diagnostics: String, pattern: Pattern): String = {
         val matcher = pattern.matcher(diagnostics)
         val diag = if (matcher.find()) " " + matcher.group() + "." else ""
    -    ("Container killed by YARN for exceeding memory limits." + diag
    -      + " Consider boosting spark.yarn.executor.memoryOverhead.")
    +    s"Container killed by YARN for exceeding memory limits. $diag " +
    +      "Consider boosting spark.yarn.executor.memoryOverhead or " +
    +      "disable yarn.nodemanager.vmem-check-enabled because of YARN-4714."
    --- End diff --
    
    Thank you for confirmation!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-c...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20735#discussion_r172661657
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
    @@ -736,7 +736,8 @@ private object YarnAllocator {
       def memLimitExceededLogMessage(diagnostics: String, pattern: Pattern): String = {
         val matcher = pattern.matcher(diagnostics)
         val diag = if (matcher.find()) " " + matcher.group() + "." else ""
    -    ("Container killed by YARN for exceeding memory limits." + diag
    -      + " Consider boosting spark.yarn.executor.memoryOverhead.")
    +    s"Container killed by YARN for exceeding memory limits. $diag " +
    +      "Consider boosting spark.yarn.executor.memoryOverhead or " +
    +      "disable yarn.nodemanager.vmem-check-enabled because of YARN-4714."
    --- End diff --
    
    I also met this situation in my docker cluster environment and this was a workaround for that.
    But, I'm not sure it's a recommendable way in Apache Spark warning.
    How do you think about this, @jerryshao ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87956/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Sorry my merge environment has some issues, please wait for a while.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    **[Test build #87956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87956/testReport)** for PR 20735 at commit [`3fc05b4`](https://github.com/apache/spark/commit/3fc05b4f8599ee65e8c4f808aee238d212c22b17).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    **[Test build #88038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88038/testReport)** for PR 20735 at commit [`a9d3fa5`](https://github.com/apache/spark/commit/a9d3fa5ead2ebec5f44615dc272056fe59f6130a).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-c...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20735#discussion_r172729670
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
    @@ -736,7 +736,8 @@ private object YarnAllocator {
       def memLimitExceededLogMessage(diagnostics: String, pattern: Pattern): String = {
         val matcher = pattern.matcher(diagnostics)
         val diag = if (matcher.find()) " " + matcher.group() + "." else ""
    -    ("Container killed by YARN for exceeding memory limits." + diag
    -      + " Consider boosting spark.yarn.executor.memoryOverhead.")
    +    s"Container killed by YARN for exceeding memory limits. $diag " +
    +      "Consider boosting spark.yarn.executor.memoryOverhead or " +
    +      "disable yarn.nodemanager.vmem-check-enabled because of YARN-4714."
    --- End diff --
    
    The changes looks fine to me.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    **[Test build #87956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87956/testReport)** for PR 20735 at commit [`3fc05b4`](https://github.com/apache/spark/commit/3fc05b4f8599ee65e8c4f808aee238d212c22b17).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-c...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20735#discussion_r172732010
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
    @@ -736,7 +736,8 @@ private object YarnAllocator {
       def memLimitExceededLogMessage(diagnostics: String, pattern: Pattern): String = {
         val matcher = pattern.matcher(diagnostics)
         val diag = if (matcher.find()) " " + matcher.group() + "." else ""
    -    ("Container killed by YARN for exceeding memory limits." + diag
    -      + " Consider boosting spark.yarn.executor.memoryOverhead.")
    +    s"Container killed by YARN for exceeding memory limits. $diag " +
    +      "Consider boosting spark.yarn.executor.memoryOverhead or " +
    +      "disable yarn.nodemanager.vmem-check-enabled because of YARN-4714."
    --- End diff --
    
    nit: "disable" -> "disabling"?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1346/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    **[Test build #88038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88038/testReport)** for PR 20735 at commit [`a9d3fa5`](https://github.com/apache/spark/commit/a9d3fa5ead2ebec5f44615dc272056fe59f6130a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-c...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20735


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1272/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20735: [MINOR][YARN] Add disable yarn.nodemanager.vmem-check-en...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20735
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88038/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org