[jira] Commented: (HADOOP-4261) Jobs failing in the init stage will never cleanup

["Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634111#action_12634111 ] 

Amareshwari Sriramadasu commented on HADOOP-4261:
-------------------------------------------------

One solution i could see is JobClient should not do setupJob, even setupJob should also be done as a seperate task (InitTask), similar to cleanup task. Then, after the completion of 'job initialization', setup will be done by the InitTask. If initializations fails, there will be no setup or cleanup task launched.

> Jobs failing in the init stage will never cleanup
> -------------------------------------------------
>
>                 Key: HADOOP-4261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4261
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amar Kamat
>            Assignee: Amareshwari Sriramadasu
>            Priority: Blocker
>
> Pre HADOOP-3150, if the job fails in the init stage, {{job.kill()}} was called. This used to make sure that the job was cleaned up w.r.t 
> - staus set to KILLED/FAILED
> - job files from the system dir are deleted
> - closing of job history files
> - making jobtracker aware of this through {{jobTracker.finalizeJob()}}
> - cleaning up the data structures via {{JobInProgress.garbageCollect()}}
> Now if the job fails in the init stage, {{job.fail()}} is called which doesnt do the cleanup. HADOOP-3150 introduces cleanup tasks which are launched once the job completes i.e killed/failed/succeeded.  Jobtracker will never consider this job for scheduling as the job will be in the {{PREP}} state forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.