[jira] [Comment Edited] (MAPREDUCE-5547) Job history should not be flushed to JHS until AM gets unregistered

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787380#comment-13787380 ] 

Vinod Kumar Vavilapalli edited comment on MAPREDUCE-5547 at 10/5/13 10:28 PM:
------------------------------------------------------------------------------

We finally sat down and reasoned about all things in MR App that are broken because of various race conditions during RM restart. While fixing that is a larger effort, looking at this specific problem, I think we should stick to the invariant that if RM sees the app as failed, we should make sure clients also see the same.

In such cases where the AM has successfully finished the 'job', but failed to unregister or failed to write history file(we ran into this also), the client will still the job as running till the last attempt. And if the last attempt also fails with the same reason, it sees the job as failed. In corner cases, we will lose work but that's better than clients struggling with successful jobs with no history files or a failure information on RM.

That said, to really fix this issue, we should change the order of things in the AM - unregister should be the first thing that should happen. Previously we moved JobHistory flush/close to be before unregister as we didn't have the AM grace period as we have now. Given that we now have the AM grace period, we can do the following
 - Flush history events & close the current history file
 - unregister and loop till RM acknowledges (YARN-540)
 - If unregister fails, don't do anything any more - irrespective of whether this is the last retry or not. This is done at MAPREDUCE-5562.
 - Otherwise, if this is not the last retry, then
    -- let the client loop (safeTermination flag) (MAPREDUCE-5505)
    -- Don't copy the history file (TODO, this JIRA)
    -- Don't send the job-end notification (MAPREDUCE-5538)
    -- Don't delete the staging directory (Done)
    -- Exit
 - Otherwise, this is the last retry (usual code path, all DONE)
    -- copy the history file to intermediate done directory
    -- send the job notification URL and
    -- let the client know job succeeded/failed/killed
    -- remove the staging directory.
    -- Exit

There are a couple more things that need fixing in separate JIRAs
 - AM reports final tracking URL as part of finishApplicationRequest. RM should set this URL for clients to see ONLY after the state operations are done
 - There is a duration during which AM unregistered successfully with RM, but hasn't yet copied the history file to the intermediate done directory and so JHS cannot serve this job to the client. We should change JHS to throw a ComeBackToMeLaterException and client retrying for the availability of the hist file till a timeout.

Few more things that need more thought
 - _SUCCESS files and their creation time given the above design
 - The commit start and end files and their cleanup


was (Author: vinodkv):
We finally sat down and reasoned about all things in MR App that are broken because of various race conditions during RM restart. While fixing that is a larger effort, looking at this specific problem, I think we should stick to the invariant that if RM sees the app as failed, we should make sure clients also see the same.

In such cases where the AM has successfully finished the 'job', but failed to unregister or failed to write history file(we ran into this also), the client will still the job as running till the last attempt. And if the last attempt also fails with the same reason, it sees the job as failed. In corner cases, we will lose work but that's better than clients struggling with successful jobs with no history files or a failure information on RM.

That said, to really fix this issue, we should change the order of things in the AM - unregister should be the first thing that should happen. Previously we moved JobHistory flush/close to be before unregister as we didn't have the AM grace period as we have now. Given that we now have the AM grace period, we can do the following
 - Flush history events & close the current history file
 - unregister
 - If unregister fails, don't do anything any more - irrespective of whether this is the last retry or not. This is done at MAPREDUCE-5562.
 - Otherwise, if this is not the last retry, then
    -- let the client loop (safeTermination flag)
    -- Don't copy the history file
    -- Don't send the job-end notification
    -- Don't delete the staging directory
    -- Exit
 - Otherwise, this is the last retry
    -- copy the history file to intermediate done directory
    -- send the job notification URL and
    -- let the client know job succeeded/failed/killed
    -- remove the staging directory.

> Job history should not be flushed to JHS until AM gets unregistered
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5547
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)