You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Rohit Dandona (JIRA)" <ji...@apache.org> on 2012/11/06 07:36:15 UTC
[jira] [Created] (MAPREDUCE-4773) MultipleOutput with different
output path for each
Rohit Dandona created MAPREDUCE-4773:
----------------------------------------
Summary: MultipleOutput with different output path for each
Key: MAPREDUCE-4773
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4773
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Rohit Dandona
Is it possible to have multiple outputs in a map reduce code where each output is directed to a different path ?
e.g.
FileOutputFormat.setOutputPath(job, new Path(outputPath));
MultipleOutputs.addNamedOutput(job, "Output 1", TextOutputFormat.class, Text.class, Text.class);
MultipleOutputs.addNamedOutput(job, "Output 2", TextOutputFormat.class, Text.class, Text.class);
Can "Output 1" & "Output 2" be alloted seperate paths ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4773) MultipleOutput with different
output path for each
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J resolved MAPREDUCE-4773.
--------------------------------
Resolution: Not A Problem
Good to know. In future, please use the user-lists when you have an issue in your development you wish to discuss/get an answer for. It is a pretty active list today. The JIRA exists for identified bugs and/or feature requests and not user help.
Resolving JIRA as "Not a Problem" (For now).
> MultipleOutput with different output path for each
> ---------------------------------------------------
>
> Key: MAPREDUCE-4773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4773
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Rohit Dandona
>
> Is it possible to have multiple outputs in a map reduce code where each output is directed to a different path ?
> e.g.
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> MultipleOutputs.addNamedOutput(job, "Output 1", TextOutputFormat.class, Text.class, Text.class);
> MultipleOutputs.addNamedOutput(job, "Output 2", TextOutputFormat.class, Text.class, Text.class);
> Can "Output 1" & "Output 2" be alloted seperate paths ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4773) MultipleOutput with different
output path for each
Posted by "Rohit Dandona (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491383#comment-13491383 ]
Rohit Dandona commented on MAPREDUCE-4773:
------------------------------------------
Hey Harsh...thnx for ur reply. Could you please give an example to elaborate the use of baseOutputPath.
> MultipleOutput with different output path for each
> ---------------------------------------------------
>
> Key: MAPREDUCE-4773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4773
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Rohit Dandona
>
> Is it possible to have multiple outputs in a map reduce code where each output is directed to a different path ?
> e.g.
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> MultipleOutputs.addNamedOutput(job, "Output 1", TextOutputFormat.class, Text.class, Text.class);
> MultipleOutputs.addNamedOutput(job, "Output 2", TextOutputFormat.class, Text.class, Text.class);
> Can "Output 1" & "Output 2" be alloted seperate paths ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4773) MultipleOutput with different
output path for each
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491397#comment-13491397 ]
Harsh J commented on MAPREDUCE-4773:
------------------------------------
The specific method: http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html#write(KEYOUT,%20VALUEOUT,%20java.lang.String) in new API MO, allows you to specify calls such as {{mo.write(keyObj, valObj, "dynamic/custom/subdir/path/string");}}. This ends up writing in job output dir subdirectory {{dynamic/custom/subdir}} with partitioned files prefixed with {{string-}}.
> MultipleOutput with different output path for each
> ---------------------------------------------------
>
> Key: MAPREDUCE-4773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4773
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Rohit Dandona
>
> Is it possible to have multiple outputs in a map reduce code where each output is directed to a different path ?
> e.g.
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> MultipleOutputs.addNamedOutput(job, "Output 1", TextOutputFormat.class, Text.class, Text.class);
> MultipleOutputs.addNamedOutput(job, "Output 2", TextOutputFormat.class, Text.class, Text.class);
> Can "Output 1" & "Output 2" be alloted seperate paths ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4773) MultipleOutput with different
output path for each
Posted by "Rohit Dandona (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491444#comment-13491444 ]
Rohit Dandona commented on MAPREDUCE-4773:
------------------------------------------
Thanks a lot Harsh...i got the solution.
> MultipleOutput with different output path for each
> ---------------------------------------------------
>
> Key: MAPREDUCE-4773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4773
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Rohit Dandona
>
> Is it possible to have multiple outputs in a map reduce code where each output is directed to a different path ?
> e.g.
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> MultipleOutputs.addNamedOutput(job, "Output 1", TextOutputFormat.class, Text.class, Text.class);
> MultipleOutputs.addNamedOutput(job, "Output 2", TextOutputFormat.class, Text.class, Text.class);
> Can "Output 1" & "Output 2" be alloted seperate paths ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4773) MultipleOutput with different
output path for each
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491360#comment-13491360 ]
Harsh J commented on MAPREDUCE-4773:
------------------------------------
Depends - separate sub-dir paths are supported by the direct baseOutputPath argument method, but not wholly different output directories, as it then gets slightly more complex to do the file output committing for the task outputs.
> MultipleOutput with different output path for each
> ---------------------------------------------------
>
> Key: MAPREDUCE-4773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4773
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Rohit Dandona
>
> Is it possible to have multiple outputs in a map reduce code where each output is directed to a different path ?
> e.g.
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> MultipleOutputs.addNamedOutput(job, "Output 1", TextOutputFormat.class, Text.class, Text.class);
> MultipleOutputs.addNamedOutput(job, "Output 2", TextOutputFormat.class, Text.class, Text.class);
> Can "Output 1" & "Output 2" be alloted seperate paths ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira