You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-user@hadoop.apache.org by Ke Zhai <zh...@cs.umd.edu> on 2011/10/12 21:00:36 UTC

Split Mapper Output

Hi, all,

I am trying to write an application which needs the mapper to split its output to file and reducer.
For example, if a mapper produces two key-value pairs (a, 1) and (b, 2), how can I write record (a, 1) directly to a file on HDFS, while still pass record (b, 2) to a reducer?

Best,
Ke

Re: Split Mapper Output

Posted by Ke Zhai <zh...@cs.umd.edu>.

Thanks.
Ke

On Oct 12, 2011, at 3:06 PM, Justin Woody wrote:

> Ke,
> 
> Check out the MultipleOutputs class. I think that's what you need. The
> Javadoc has an example as well.
> 
> Justin
> 
> On Wed, Oct 12, 2011 at 3:00 PM, Ke Zhai <zh...@cs.umd.edu> wrote:
>> Hi, all,
>> 
>> I am trying to write an application which needs the mapper to split its output to file and reducer.
>> For example, if a mapper produces two key-value pairs (a, 1) and (b, 2), how can I write record (a, 1) directly to a file on HDFS, while still pass record (b, 2) to a reducer?
>> 
>> Best,
>> Ke

Re: Split Mapper Output

Posted by Justin Woody <ju...@gmail.com>.

Ke,

Check out the MultipleOutputs class. I think that's what you need. The
Javadoc has an example as well.

Justin

On Wed, Oct 12, 2011 at 3:00 PM, Ke Zhai <zh...@cs.umd.edu> wrote:
> Hi, all,
>
> I am trying to write an application which needs the mapper to split its output to file and reducer.
> For example, if a mapper produces two key-value pairs (a, 1) and (b, 2), how can I write record (a, 1) directly to a file on HDFS, while still pass record (b, 2) to a reducer?
>
> Best,
> Ke