[jira] [Commented] (MAPREDUCE-4099) ApplicationMaster may fail to remove staging directory

["Jason Lowe (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245751#comment-13245751 ] 

Jason Lowe commented on MAPREDUCE-4099:
---------------------------------------

Initially I thought a quick fix would be to change MRAppMaster so the call to cleanupStagingDir() occurs before stopping all the services (and therefore before the RM tries to kill the AM).  However this introduces another problem: if something goes wrong with the AM (killed, crashed, hung) between the time it has removed the staging directory and before it has notified the RM then the RM will think the AM did not complete successfully and it will either report the job as failed (after the AM already told the client it was successful) or worse, the RM will launch another AM attempt and fail because the staging directory has been removed.

Seems like we need another RMApp state to track the fact that we've succeeded but are still cleaning up.  For example, when the AM unregisters, we go from RUNNING into a new FINISHING state.  The RM then gives the AM so many seconds to exit on its own.  If the AM container doesn't exit on its own within the time limit then the RM kills the container, but in either case we move to the FINISHED state (i.e.: once we're FINISHING, we're going to get to FINISHED one way or another).

I'm not thrilled with the idea of adding yet another state to app/attempts, but other alternatives seem to open doors to the AM failing at just the wrong time and we end up failing the job after the AM has already told the client the job was successful.
                
> ApplicationMaster may fail to remove staging directory
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-4099
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4099
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.2
>            Reporter: Jason Lowe
>            Priority: Critical
>
> When the ApplicationMaster shuts down it's supposed to remove the staging directory, assuming properties weren't set to override this behavior. During shutdown the AM tells the ResourceManager that it has finished before it cleans up the staging directory.  However upon hearing the AM has finished, the RM turns right around and kills the AM container.  If the AM is too slow, the AM will be killed before the staging directory is removed.
> We're seeing the AM lose this race fairly consistently on our clusters, and the lack of staging directory cleanup quickly leads to filesystem quota issues for some users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira