Write to different files based on key from mapper

[Srihari Anantha Padmanabhan <sr...@yahoo-inc.com>]

Hi,

I want to write the records to different hdfs files (instead of the default part-m..0000) into different output dirs based on the key generated by the mapper. I would like to implement this using the hadoop mapreduce API.

Has anyone faced this problem? Is there any sample program that I can refer to?

Thanks,
Srihari

Re: Write to different files based on key from mapper

[Shrijeet Paliwal <sh...@rocketfuel.com>]

The class MultipleOutputs in package org.apache.hadoop.mapreduce.lib.output
describes its usage as following :

The MultipleOutputs class simplifies writing output data to multiple outputs


It also has a sample program in its comments. Here is the link belonging to
0.21
http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html

-Shrijeet

On Mon, Nov 22, 2010 at 10:58 AM, Srihari Anantha Padmanabhan <
sriharia@yahoo-inc.com> wrote:
> Hi,
>
> I want to write the records to different hdfs files (instead of the
default part-m..0000) into different output dirs based on the key generated
by the mapper. I would like to implement this using the hadoop mapreduce
API.
>
> Has anyone faced this problem? Is there any sample program that I can
refer to?
>
> Thanks,
> Srihari