[jira] [Comment Edited] (YARN-6966) NodeManager metrics may return wrong negative values when NM restart

["Robert Kanter (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548546#comment-16548546 ] 

Robert Kanter edited comment on YARN-6966 at 7/18/18 11:01 PM:
---------------------------------------------------------------

It looks like the job is broken even though it's up now.  All of the tests from today ran for < 1 min before failing :(

I think it's fine if making a testcase for that will be too tricky.

+1 LGM pending Jenkins
[~haibochen] any other comments?


was (Author: rkanter):
It looks like the job is broken even though it's up now.  All of the tests from today ran for < 1 min before failing :(

+1 LGM pending Jenkins
[~haibochen] any other comments?

> NodeManager metrics may return wrong negative values when NM restart
> --------------------------------------------------------------------
>
>                 Key: YARN-6966
>                 URL: https://issues.apache.org/jira/browse/YARN-6966
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Yang Wang
>            Assignee: Szilard Nemeth
>            Priority: Major
>         Attachments: YARN-6966.001.patch, YARN-6966.002.patch, YARN-6966.003.patch, YARN-6966.004.patch, YARN-6966.005.patch
>
>
> Just as YARN-6212. However, I think it is not a duplicate of YARN-3933.
> The primary cause of negative values is that metrics do not recover properly when NM restart.
> AllocatedContainers,ContainersLaunched,AllocatedGB,AvailableGB,AllocatedVCores,AvailableVCores in metrics also need to recover when NM restart.
> This should be done in ContainerManagerImpl#recoverContainer.
> The scenario could be reproduction by the following steps:
> # Make sure YarnConfiguration.NM_RECOVERY_ENABLED=true,YarnConfiguration.NM_RECOVERY_SUPERVISED=true in NM
> # Submit an application and keep running
> # Restart NM
> # Stop the application
> # Now you get the negative values
> {code}
> /jmx?qry=Hadoop:service=NodeManager,name=NodeManagerMetrics
> {code}
> {code}
> {
> name: "Hadoop:service=NodeManager,name=NodeManagerMetrics",
> modelerType: "NodeManagerMetrics",
> tag.Context: "yarn",
> tag.Hostname: "hadoop1111.com",
> ContainersLaunched: 0,
> ContainersCompleted: 0,
> ContainersFailed: 2,
> ContainersKilled: 0,
> ContainersIniting: 0,
> ContainersRunning: 0,
> AllocatedGB: 0,
> AllocatedContainers: -2,
> AvailableGB: 160,
> AllocatedVCores: -11,
> AvailableVCores: 3611,
> ContainerLaunchDurationNumOps: 2,
> ContainerLaunchDurationAvgTime: 6,
> BadLocalDirs: 0,
> BadLogDirs: 0,
> GoodLocalDirsDiskUtilizationPerc: 2,
> GoodLogDirsDiskUtilizationPerc: 2
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org