[jira] Commented: (HADOOP-3140) JobTracker should not try to promote a (map) task if it does not write to DFS at all

["Amar Kamat (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584183#action_12584183 ] 

Amar Kamat commented on HADOOP-3140:
------------------------------------

How about this
1) {{Task.done()}} method checks if the task has data to be promoted and passes this info to the TaskTracker via the {{TaskTracker.done()}} api.
2) If there is no data to promote, the TaskTracker sets the task status as {{SUCCEEDED}} or {{FAILED}} depending on whether the task succeeds or fails.
3) JobInProgress adds only {{COMMIT_PENDING}} tasks to the commit-pending queue. The commit-pending queue deals with {{KILLED/FAILED}} tasks only if the commit-pending thread fails to save the task output or if the TaskTracker is lost.
4) Temporary data from {{FAILED/KILLED}} tasks will be deleted once the job completes (see HADOOP-2391).
5) {{JobInProgress.updateTaskStatus()}} can now be called with {{SUCCEEDED}} state from TaskTracker (via heartbeat) or from the commit-pending queue.
5) If a JobInProgress.updateTaskStatus() is called with {{SUCCEEDED}} state for a completed TIP it will be marked as {{KILLED}}.


> JobTracker should not try to promote a (map) task if it does not write to DFS at all
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3140
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Runping Qi
>
> In most cases, map tasks do not write to dfs.
> Thus, when they complete, they should not be put into commit_pending queue at all.
> This will improve the task promotion significantly.
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.