[jira] [Commented] (MAPREDUCE-3028) Support job end notification in .next /0.23

["Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129768#comment-13129768 ] 

Ravi Prakash commented on MAPREDUCE-3028:
-----------------------------------------

Hi Arun! Yes I've been working on this.

>From your and Amol's discussions:
Spec is as such (details to be borrowed from current JT). Please use same config variable names to avoid compatibility.
If the name has JT (or jobtracker)then lets discuss. The spec is as such

- AM should trigger a http call upon job completion
- The http call should be fire and forget by default.
- If user specifies number of tries then the following should be true
  1. Allow hadoop config to be set to ignore this in case becomes a problem (too many retries)
  2. Set max retries for all AM; i.e. users cannot specify a number above this. If Hadoop config does not specify it
the value should be 1 (i.e fire and forget)
  3. Lets say that the Hadoop config on max retries is set to 3. Then AM should try 3 times with timeout (again a
Hadoop variable). If timeout is not specified, the value should be 30 seconds.

The aim is to let AM get off quickly. Since unlike JT there is one AM per job there would not bottleneck and no hanging
of queue. So 30 seconds is ok.

                
> Support job end notification in .next /0.23
> -------------------------------------------
>
>                 Key: MAPREDUCE-3028
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Mohammad Kamrul Islam
>            Assignee: Ravi Prakash
>            Priority: Blocker
>             Fix For: 0.23.0
>
>
> Oozie primarily depends on  the job end notification to determine when the job finishes. In the current version,  job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion.
> Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well.
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira