You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org> on 2017/12/22 00:36:00 UTC
[jira] [Commented] (YARN-7284) NodeManager crashes with OOM when
Debug log enabled for ContainerLocalizer
[ https://issues.apache.org/jira/browse/YARN-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300766#comment-16300766 ]
Vinod Kumar Vavilapalli commented on YARN-7284:
-----------------------------------------------
Turning off DEBUG for ContainerLocalizer should avoid this problem.
But presumably it was turned on to debug some issues with localization.
It's high time we send the ContainerLocalizer output also to a file that eventually gets aggregated. The key challenge is that ContainerLocalizer isn't tied to one container - it can download resources for multiple containers. So, tying to log-aggregation will need some non-trivial tricks.
> NodeManager crashes with OOM when Debug log enabled for ContainerLocalizer
> ---------------------------------------------------------------------------
>
> Key: YARN-7284
> URL: https://issues.apache.org/jira/browse/YARN-7284
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.7.3
> Reporter: Prabhu Joseph
> Attachments: Screen Shot 2017-10-03 at 1.29.35 PM.png, Screen Shot 2017-10-03 at 1.29.48 PM.png
>
>
> NodeManager crashes with OOM when DEBUG log enabled for ContainerLocalizer.
> {code}
> 2017-10-03 07:25:20,066 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[Thread-2114,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
> at java.lang.StringBuffer.append(StringBuffer.java:272)
> at org.apache.hadoop.util.Shell$1.run(Shell.java:900)
> {code}
> errThread part of Hadoop Common Shell reads all the DEBUG log lines and appends to StringBuffer errMsg. As per the heap dump, the errMsg stores more than 1GB of contents. (attached image)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org