You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Tomasz <to...@gmail.com> on 2013/04/24 20:20:27 UTC

MultipleInputs and multiple mappers

Hi,

I've searched but didn't find any answers, so decided to post to this list.
I'm trying to convert MultipleInputs into oozie <map-reduce> action.
My perfectly working code in main() is:
         MultipleInputs.addInputPath(conf, new Path(inFirm), 
SequenceFileInputFormat.class, FirmMapper.class);
         MultipleInputs.addInputPath(conf, new Path(inOffice), 
SequenceFileInputFormat.class, OfficeMapper.class);
         FileOutputFormat.setOutputPath(conf, new Path(mr1Out));
         conf.setOutputFormat(SequenceFileOutputFormat.class);
         conf.setMapOutputKeyClass(TextPair.class);
         conf.setMapOutputValueClass(Text.class);
         conf.setPartitionerClass(TextPair.FirstPartitioner.class);
conf.setOutputValueGroupingComparator(TextPair.FirstComparator.class);
         conf.setReducerClass(SingleOfficeFirmReducer.class);
         conf.setOutputKeyClass(TextPair.class);
         conf.setOutputValueClass(Text.class);
How do I translate MultipleInputs and multiple mappers into oozie workflow?
All the examples show only the single mapper.

br,
Tom

Re: MultipleInputs and multiple mappers

Posted by Rohini Palaniswamy <ro...@gmail.com>.
Hi Tom,
  You can either
  1) write a Java action which has the above code. (or)
  2) You can mimick what MultipleInputs.addInputPath does, and set those
settings yourself in the action conf of the MR action.

http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.0/src/mapred/org/apache/hadoop/mapred/lib/MultipleInputs.java

Regards,
Rohini


On Wed, Apr 24, 2013 at 11:20 AM, Tomasz <to...@gmail.com> wrote:

> Hi,
>
> I've searched but didn't find any answers, so decided to post to this list.
> I'm trying to convert MultipleInputs into oozie <map-reduce> action.
> My perfectly working code in main() is:
>         MultipleInputs.addInputPath(**conf, new Path(inFirm),
> SequenceFileInputFormat.class, FirmMapper.class);
>         MultipleInputs.addInputPath(**conf, new Path(inOffice),
> SequenceFileInputFormat.class, OfficeMapper.class);
>         FileOutputFormat.**setOutputPath(conf, new Path(mr1Out));
>         conf.setOutputFormat(**SequenceFileOutputFormat.**class);
>         conf.setMapOutputKeyClass(**TextPair.class);
>         conf.setMapOutputValueClass(**Text.class);
>         conf.setPartitionerClass(**TextPair.FirstPartitioner.**class);
> conf.**setOutputValueGroupingComparat**or(TextPair.FirstComparator.**
> class);
>         conf.setReducerClass(**SingleOfficeFirmReducer.class)**;
>         conf.setOutputKeyClass(**TextPair.class);
>         conf.setOutputValueClass(Text.**class);
> How do I translate MultipleInputs and multiple mappers into oozie workflow?
> All the examples show only the single mapper.
>
> br,
> Tom
>