You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Karam Singh (JIRA)" <ji...@apache.org> on 2016/06/27 16:23:52 UTC
[jira] [Commented] (YARN-5296) NMs going OutOfMemory because
ContainerMetrics leak in ContainerMonitorImpl
[ https://issues.apache.org/jira/browse/YARN-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351351#comment-15351351 ]
Karam Singh commented on YARN-5296:
-----------------------------------
From offline discussion with [~djp]
Root cause is in YARN-4811 where we launch tasks (scheduleAtFixedRate) in MutableQuantiles but never get chance to terminate these tasks
> NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl
> ---------------------------------------------------------------------------
>
> Key: YARN-5296
> URL: https://issues.apache.org/jira/browse/YARN-5296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.8.0, 2.9.0
> Reporter: Karam Singh
>
> Ran tests in following manner,
> 1. Run GridMix of 768 sequestionally around 17 times to execute about 12.9K apps.
> 2. After 4-5hrs take Check NM Heap using Memory Analyser. It report around 96% Heap is being used my ContainerMetrics
> 3. Run 7 more GridMix run for have around 18.2apps ran in total. Again check NM heap using Memory Analyser again 96% heap is being used by ContainerMetrics.
> 4. Start one more grimdmix run, while run going on , NMs started going down with OOM, around running 18.7K+, On analysing NM heap using Memory analyser, OOM was caused by ContainerMetrics
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org