[jira] [Updated] (YARN-4771) Some containers can be skipped during log aggregation after NM restart

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod Kumar Vavilapalli updated YARN-4771:
------------------------------------------
    Target Version/s: 2.7.4  (was: 2.7.3)

bq. However it may have issues with very long-running apps that churn a lot of containers, since the container state won't be released until the application completes.
This is going to be problematic, impacting NM memory usage.

I think the right solution is to decouple log-aggregation state completely from the rest of the container-state, and persist that separately in state-store etc irrespective of container / application state.

IAC, I am dropping this off 2.7.3 given that this is a long standing issue and as I'd to proceed with that release now.

> Some containers can be skipped during log aggregation after NM restart
> ----------------------------------------------------------------------
>
>                 Key: YARN-4771
>                 URL: https://issues.apache.org/jira/browse/YARN-4771
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.7.2
>            Reporter: Jason Lowe
>            Priority: Critical
>         Attachments: YARN-4771.001.patch, YARN-4771.002.patch
>
>
> A container can be skipped during log aggregation after a work-preserving nodemanager restart if the following events occur:
> # Container completes more than yarn.nodemanager.duration-to-track-stopped-containers milliseconds before the restart
> # At least one other container completes after the above container and before the restart



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)