[jira] [Commented] (MAPREDUCE-2715) submitAndMonitorJob() doesn't play nice with MultipleOutputFile

["Harsh J (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068573#comment-13068573 ] 

Harsh J commented on MAPREDUCE-2715:
------------------------------------

A few questions here:

* What is {{MultipleOutputFile}}? Pardon my ignorance, but I could not find a class or type like that in my workspace, so would be good to know if its not {{MultipleOutputs}} or {{MultiFileOutputFormat}} we're talking about here.
* The output directory existence check is done by the {{OutputFormat}} class in use, and not the {{submitAndMonitorJob()}} call. The specific method to override in this case would usually be {{OutputFormat#checkOutputSpecs}}, which gets invoked pre-submit. Is this what you are talking about too?
* Does {{MultipleOutputFile}} (or if this is one of the two classes I mentioned earlier) create arbitrary directories in the HDFS? How does it ensure task commits in this case?

> submitAndMonitorJob() doesn't play nice with MultipleOutputFile
> ---------------------------------------------------------------
>
>                 Key: MAPREDUCE-2715
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2715
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Geoffrey Young
>
> part of submitAndMonitorJob() balks if the output directory currently exists but is non-empty:
>   "Error launching job , Output path already exists : "
> this logic actually conflicts with the ideas behind MultipleOutputFile, where the output file path is calculated later on.
> it would be really nice to remove the restriction for non-empty output directories in submitAndMonitorJob() so that MultipleOutputFile becomes more useful - as it stands now, I can't, for example, specify a base output path then use MutlipleOutputFile to partition by date on a daily basis.
> thanks.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira