[jira] [Commented] (YARN-2261) YARN should have a way to run post-application cleanup

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054571#comment-14054571 ] 

Vinod Kumar Vavilapalli commented on YARN-2261:
-----------------------------------------------

The proposal here is to have a YARN application-level cleanup container that runs only as the last thing for an application in the cluster.
 - In a way, we already have this today, as we let AM's hang around for a while (by default - 10mins) *after* unregister - this feature makes it explicit.
 - For those who have lived in this space around for a while, this is akin to MR job-cleanup.
 - This feature lets apps submit a separate container-launch-context for cleanup, one that is only run after the app is done for real.
 - Clearly, it will
    -- be optional.
    -- Have timeouts on how much time it can take to finish (default, overridable, and upper limit. Default = today's time for AMs to exit after unregister?)
    -- Have resource requests limits like usual
    -- May have its own retries (Cleanup failure != Application failure as today?)

Some challenges
 - Cleanup container may not get resources because cluster may have gotten busy after the final AM exit. Solution is to reserve (part of) resources used by the last AM for use by the cleanup container

> YARN should have a way to run post-application cleanup
> ------------------------------------------------------
>
>                 Key: YARN-2261
>                 URL: https://issues.apache.org/jira/browse/YARN-2261
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: resourcemanager
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>
> See MAPREDUCE-5956 for context. Specific options are at https://issues.apache.org/jira/browse/MAPREDUCE-5956?focusedCommentId=14054562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14054562.



--
This message was sent by Atlassian JIRA
(v6.2#6252)