You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Sharad Agarwal (JIRA)" <ji...@apache.org> on 2011/05/13 08:38:47 UTC
[jira] [Created] (MAPREDUCE-2493) New Api FileOutputFormat does not
honour user specified OutputCommitter
New Api FileOutputFormat does not honour user specified OutputCommitter
-----------------------------------------------------------------------
Key: MAPREDUCE-2493
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Sharad Agarwal
o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2493) New Api FileOutputFormat does
not honour user specified OutputCommitter
Posted by "Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149646#comment-13149646 ]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-2493:
-----------------------------------------------------------
When we use new API, the OutputCommitter associated with the FileOutputFormat is FileOutputCommitter.
Where as for the old API, OutputCommitter is reading from the conf object.
{code}
if (useNewApi) {
if (LOG.isDebugEnabled()) {
LOG.debug("using new api for output committer");
}
outputFormat =
ReflectionUtils.newInstance(taskContext.getOutputFormatClass(), job);
committer = outputFormat.getOutputCommitter(taskContext);
} else {
committer = conf.getOutputCommitter();
}
{code}
> New Api FileOutputFormat does not honour user specified OutputCommitter
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-2493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Sharad Agarwal
>
> o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2493) New Api FileOutputFormat does not
honour user specified OutputCommitter
Posted by "Bhallamudi Venkata Siva Kamesh (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-2493:
------------------------------------------------------
Component/s: mrv2
Affects Version/s: 0.24.0
0.23.0
> New Api FileOutputFormat does not honour user specified OutputCommitter
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-2493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Sharad Agarwal
> Attachments: MAPREDUCE-2493.patch
>
>
> o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2493) New Api FileOutputFormat does not
honour user specified OutputCommitter
Posted by "Bhallamudi Venkata Siva Kamesh (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-2493:
------------------------------------------------------
Attachment: MAPREDUCE-2493.patch
here I am attaching a solution against trunk. Pls review the solution
> New Api FileOutputFormat does not honour user specified OutputCommitter
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-2493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Sharad Agarwal
> Attachments: MAPREDUCE-2493.patch
>
>
> o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2493) New Api FileOutputFormat does
not honour user specified OutputCommitter
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280251#comment-13280251 ]
Robert Joseph Evans commented on MAPREDUCE-2493:
------------------------------------------------
Bh.V.S.Kamesh,
I am not an expert on the code, but I thought that this was by design. In the older APIs the configs for output format and output committer were separate, but the committer is tied quite closely to the output format. If I am outputting to a DB using a DB Output Format I now have to set two configs instead of just one to make this work. What is more I may need to play some odd games to make it so that a DB output committer even works so that I can commit/roll back the results, something that the current DB output format does not implement. This comes at the expense of making it more difficult to override the OutputCommitter, but my experience with the FileOutputCommitter, it is not really designed so that it can be subclassed in a clean extensible way.
That being said I am fine with adding in the ability to override the output committer through a configuration on the newer API, I am just not sure that this is the proper way to do it. I have not had time to really think through it. At a minimum please upmerge the patch. It no longer applies.
> New Api FileOutputFormat does not honour user specified OutputCommitter
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-2493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Sharad Agarwal
> Assignee: Bhallamudi Venkata Siva Kamesh
> Attachments: MAPREDUCE-2493.patch
>
>
> o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2493) New Api FileOutputFormat does not
honour user specified OutputCommitter
Posted by "Bhallamudi Venkata Siva Kamesh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-2493:
------------------------------------------------------
Attachment: MAPREDUCE-2493-1.patch
Hi Robert,
Thanks for looking into this issue. I have found the necessity of a pluggable output committer associated with FOF, when I was fixing [MAPREDUCE-3130|https://issues.apache.org/jira//browse/MAPREDUCE-3130]. I fixed [MAPREDUCE-3130|https://issues.apache.org/jira//browse/MAPREDUCE-3130] by associating [MAPREDUCE-3471|https://issues.apache.org/jira//browse/MAPREDUCE-3471] with *FOF*. I just attached the way I fixed this.
Upmerging the perviously submitted patch. Please provide your feedback
If this is not a proper way to fix, I am happy to have fix for this.:)
> New Api FileOutputFormat does not honour user specified OutputCommitter
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-2493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Sharad Agarwal
> Assignee: Bhallamudi Venkata Siva Kamesh
> Attachments: MAPREDUCE-2493-1.patch, MAPREDUCE-2493.patch
>
>
> o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2493) New Api FileOutputFormat does
not honour user specified OutputCommitter
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves reassigned MAPREDUCE-2493:
----------------------------------------
Assignee: Bhallamudi Venkata Siva Kamesh
> New Api FileOutputFormat does not honour user specified OutputCommitter
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-2493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Sharad Agarwal
> Assignee: Bhallamudi Venkata Siva Kamesh
> Attachments: MAPREDUCE-2493.patch
>
>
> o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2493) New Api FileOutputFormat does
not honour user specified OutputCommitter
Posted by "Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223227#comment-13223227 ]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-2493:
-----------------------------------------------------------
Can someone please review the attached patch.
> New Api FileOutputFormat does not honour user specified OutputCommitter
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-2493
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2493
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.24.0
> Reporter: Sharad Agarwal
> Attachments: MAPREDUCE-2493.patch
>
>
> o.a.h.mapreduce.lib.output.FileOutputFormat always uses the default FileOutputCommitter. It ignores the user specified OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira