You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Tomasz <to...@gmail.com> on 2013/04/24 20:20:27 UTC
MultipleInputs and multiple mappers
Hi,
I've searched but didn't find any answers, so decided to post to this list.
I'm trying to convert MultipleInputs into oozie <map-reduce> action.
My perfectly working code in main() is:
MultipleInputs.addInputPath(conf, new Path(inFirm),
SequenceFileInputFormat.class, FirmMapper.class);
MultipleInputs.addInputPath(conf, new Path(inOffice),
SequenceFileInputFormat.class, OfficeMapper.class);
FileOutputFormat.setOutputPath(conf, new Path(mr1Out));
conf.setOutputFormat(SequenceFileOutputFormat.class);
conf.setMapOutputKeyClass(TextPair.class);
conf.setMapOutputValueClass(Text.class);
conf.setPartitionerClass(TextPair.FirstPartitioner.class);
conf.setOutputValueGroupingComparator(TextPair.FirstComparator.class);
conf.setReducerClass(SingleOfficeFirmReducer.class);
conf.setOutputKeyClass(TextPair.class);
conf.setOutputValueClass(Text.class);
How do I translate MultipleInputs and multiple mappers into oozie workflow?
All the examples show only the single mapper.
br,
Tom
Re: MultipleInputs and multiple mappers
Posted by Rohini Palaniswamy <ro...@gmail.com>.
Hi Tom,
You can either
1) write a Java action which has the above code. (or)
2) You can mimick what MultipleInputs.addInputPath does, and set those
settings yourself in the action conf of the MR action.
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.0/src/mapred/org/apache/hadoop/mapred/lib/MultipleInputs.java
Regards,
Rohini
On Wed, Apr 24, 2013 at 11:20 AM, Tomasz <to...@gmail.com> wrote:
> Hi,
>
> I've searched but didn't find any answers, so decided to post to this list.
> I'm trying to convert MultipleInputs into oozie <map-reduce> action.
> My perfectly working code in main() is:
> MultipleInputs.addInputPath(**conf, new Path(inFirm),
> SequenceFileInputFormat.class, FirmMapper.class);
> MultipleInputs.addInputPath(**conf, new Path(inOffice),
> SequenceFileInputFormat.class, OfficeMapper.class);
> FileOutputFormat.**setOutputPath(conf, new Path(mr1Out));
> conf.setOutputFormat(**SequenceFileOutputFormat.**class);
> conf.setMapOutputKeyClass(**TextPair.class);
> conf.setMapOutputValueClass(**Text.class);
> conf.setPartitionerClass(**TextPair.FirstPartitioner.**class);
> conf.**setOutputValueGroupingComparat**or(TextPair.FirstComparator.**
> class);
> conf.setReducerClass(**SingleOfficeFirmReducer.class)**;
> conf.setOutputKeyClass(**TextPair.class);
> conf.setOutputValueClass(Text.**class);
> How do I translate MultipleInputs and multiple mappers into oozie workflow?
> All the examples show only the single mapper.
>
> br,
> Tom
>