You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by js...@apache.org on 2018/03/07 09:02:06 UTC
spark git commit: [MINOR][YARN] Add disable
yarn.nodemanager.vmem-check-enabled option to memLimitExceededLogMessage
Repository: spark
Updated Branches:
refs/heads/master 4c587eb48 -> 04e71c316
[MINOR][YARN] Add disable yarn.nodemanager.vmem-check-enabled option to memLimitExceededLogMessage
My spark application sometimes will throw `Container killed by YARN for exceeding memory limits`.
Even I increased `spark.yarn.executor.memoryOverhead` to 10G, this error still happen. The latest config:
<img width="685" alt="memory-config" src="https://user-images.githubusercontent.com/5399861/36975716-f5c548d2-20b5-11e8-95e5-b228d50917b9.png">
And error message:
```
ExecutorLostFailure (executor 121 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 30.7 GB of 30 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
```
This is because of [Linux glibc >= 2.10 (RHEL 6) malloc may show excessive virtual memory usage](https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en). So disable `yarn.nodemanager.vmem-check-enabled` looks like a good option as [MapR mentioned ](https://mapr.com/blog/best-practices-yarn-resource-management).
This PR add disable `yarn.nodemanager.vmem-check-enabled` option to memLimitExceededLogMessage.
More details:
https://issues.apache.org/jira/browse/YARN-4714
https://stackoverflow.com/a/31450291
https://stackoverflow.com/a/42091255
After this PR:
<img width="898" alt="yarn" src="https://user-images.githubusercontent.com/5399861/36975949-c8e7bbbe-20b6-11e8-9513-9f903b868d8d.png">
N/A
Author: Yuming Wang <yu...@ebay.com>
Author: Yuming Wang <wg...@gmail.com>
Closes #20735 from wangyum/YARN-4714.
Change-Id: Ie10836e2c07b6384d228c3f9e89f802823bd9f16
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/04e71c31
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/04e71c31
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/04e71c31
Branch: refs/heads/master
Commit: 04e71c31603af3a13bc13300df799f003fe185f7
Parents: 4c587eb
Author: Yuming Wang <yu...@ebay.com>
Authored: Wed Mar 7 17:01:29 2018 +0800
Committer: jerryshao <ss...@hortonworks.com>
Committed: Wed Mar 7 17:01:29 2018 +0800
----------------------------------------------------------------------
.../main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/04e71c31/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
----------------------------------------------------------------------
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index 506adb3..a537243 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -736,7 +736,8 @@ private object YarnAllocator {
def memLimitExceededLogMessage(diagnostics: String, pattern: Pattern): String = {
val matcher = pattern.matcher(diagnostics)
val diag = if (matcher.find()) " " + matcher.group() + "." else ""
- ("Container killed by YARN for exceeding memory limits." + diag
- + " Consider boosting spark.yarn.executor.memoryOverhead.")
+ s"Container killed by YARN for exceeding memory limits. $diag " +
+ "Consider boosting spark.yarn.executor.memoryOverhead or " +
+ "disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714."
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org