[jira] [Commented] (YARN-3496) Add a configuration to disable/enable storing localization state in NMLeveldbStateStore

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498088#comment-14498088 ] 

Jason Lowe commented on YARN-3496:
----------------------------------

How are we going to support recovery of containers without tracking which resources we have localized?  We cannot re-localize resources for already running containers -- they're already running on the existing localized resources.  If we fail to track those resources then we leak them on the local filesystem, potentially colliding with them later.  So from my perspective there already is a way to enable/disable this behavior, and that's enabling/disabling NM recovery.  I don't see how we can support recovery without storing localization state.

Do we have hard evidence storing to leveldb is a huge NM performance impact, or is this theoretical?  In our testing we have not seen enabling NM recovery as having a significant impact on NM performance.

> Add a configuration to disable/enable storing localization state in NMLeveldbStateStore
> ---------------------------------------------------------------------------------------
>
>                 Key: YARN-3496
>                 URL: https://issues.apache.org/jira/browse/YARN-3496
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.7.0
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>
> Add a configuration to disable/enable storing localization state in NMLeveldbStateStore.
> Store Localization state in the levelDB may have some overhead, which may affect NM performance.
> It would better to have a configuration to disable/enable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)