[jira] Commented: (HADOOP-3150) Move task file promotion into the task

["Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609801#action_12609801 ] 

Amareshwari Sriramadasu commented on HADOOP-3150:
-------------------------------------------------

Today we give promotion to all jobs that have an mapred.output.dir defined. So, one more thing that needs to be fixed here is 'Applications creating side files with OutputFormat which is not an instance of FileOutputFormat'. For example, Hadoop archives has output format as NullOutputFormat, but the archive is created using task's side files. Thus 'moving the task commit to OutputFormat' will ignore the sidefiles. And also, 'with setupJob and setupTask moved to OutputFormat', the facility to create side files is removed.

To support creating side files even with OutputFormat which is not a FileOutputFormat, we can have SideFileOutputFormat which extends FileOutputFormat. If the job OutputFormat is not a FileOutputFormat and mapred.output.dir is defined, the framework will instantiate SideFileOutputFormat. 
The following apis in FileOutputFormat will be moved to SideFileOutputFormat
{noformat}
static void setWorkOutputPath(JobConf conf, Path outputDir)
public static Path getWorkOutputPath(JobConf conf) 
{noformat}

SideFileOutputFormat.getRecordWriter() will use TextOutputFormat's RecordWriter.

Finally task commit will constitute commit of Job's OutputFormat and SideFileOutputFormat ( if Job's OutputFormat is not a FileOutputFormat). 
Thoughts?


> Move task file promotion into the task
> --------------------------------------
>
>                 Key: HADOOP-3150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3150
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.19.0
>
>         Attachments: 3150.patch
>
>
> We need to move the task file promotion from the JobTracker to the Task and move it down into the output format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.