[jira] [Updated] (YARN-1336) Work-preserving nodemanager restart

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated YARN-1336:
-----------------------------

    Attachment: YARN-1336-rollup.patch

Attaching a rollup patch for the prototype that [~raviprak] and I developed.  This recovers resource localization state, applications and containers, tokens, log aggregation, deletion service, and the MR shuffle auxiliary service.  A quick high-level overview:

- Restart functionality is enabled by configuring yarn.nodemanager.recovery.enabled to true and yarn.nodemanager.recovery.dir to a directory on the local filesystem where the state will be stored.
- Containers are launched with an additional shell layer which places the exit code of the container in an .exitcode file.  This allows the restarted NM instance to recover containers that are already running or have exited since the last NM instance.
- NMStateStoreService is the abstraction layer for the state store.  NMNullStateStoreService is used when recovery is disabled and NMLevelDBStateStoreService is used when it is enabled.
- Rather than explicitly record localized resource reference counts, resources are recovered with no references and recovered containers re-request their resources as during a normal container lifecycle to restore the reference counts.

Some things that are still missing:
- ability to distinguish shutdown for restart vs. decommission
- proper handling of state store errors
- adding unit tests
- adding formal documentation.

Feedback is greatly appreciated.  I'll be working on addressing the missing items and splitting the patch into smaller pieces across the appropriate subtasks to simplify reviews.

> Work-preserving nodemanager restart
> -----------------------------------
>
>                 Key: YARN-1336
>                 URL: https://issues.apache.org/jira/browse/YARN-1336
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>    Affects Versions: 2.3.0
>            Reporter: Jason Lowe
>         Attachments: YARN-1336-rollup.patch
>
>
> This serves as an umbrella ticket for tasks related to work-preserving nodemanager restart.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)