You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Denim Live <de...@yahoo.com> on 2010/07/08 11:13:34 UTC

naming the output fle of reduce to the partition number

Hi Everyone,
I am having some problem with naming the output file of each reduce task with 
the partition number. First of all, how can  I get the partition number within 
each reduce? Second, How am I going to name the output file with that partition 
number? 

I have looked to the MultipleTextOutputFormat. It can generate a new file with 
the name of my choice for each key. But I want to name the output file for each 
partition with the name of my choice i.e. with the partition number. 

Please help me in this regard.

Thanks

Re: naming the output fle of reduce to the partition number

Posted by Denim Live <de...@yahoo.com>.

yes, I can get the partition number using 
jobconf.getInt("mapred.task.partition") but how can I custom name my output file 
of each reducer with just this partition number?




________________________________
From: Ted Yu <yu...@gmail.com>
To: mapreduce-user@hadoop.apache.org
Sent: Thu, July 8, 2010 6:22:54 PM
Subject: Re: naming the output fle of reduce to the partition number

Please take a look at getUniqueName() method of 
src/mapred/org/apache/hadoop/mapred/FileOutputFormat.java
It retrieves "mapred.task.partition"


On Thu, Jul 8, 2010 at 2:13 AM, Denim Live <de...@yahoo.com> wrote:

Hi Everyone,
>I am having some problem with naming the output file of each reduce task with 
>the partition number. First of all, how can  I get the partition number within 
>each reduce? Second, How am I going to name the output file with that partition 
>number? 
>
>I have looked to the MultipleTextOutputFormat. It can generate a new file with 
>the name of my choice for each key. But I want to name the output file for each 
>partition with the name of my choice i.e. with the partition number. 
>
>Please help me in this regard.
> 
>Thanks
>

Re: naming the output fle of reduce to the partition number

Posted by Ted Yu <yu...@gmail.com>.
Please take a look at getUniqueName() method of
src/mapred/org/apache/hadoop/mapred/FileOutputFormat.java
It retrieves "mapred.task.partition"

On Thu, Jul 8, 2010 at 2:13 AM, Denim Live <de...@yahoo.com> wrote:

> Hi Everyone,
> I am having some problem with naming the output file of each reduce task
> with the partition number. First of all, how can  I get the partition number
> within each reduce? Second, How am I going to name the output file with that
> partition number?
> I have looked to the MultipleTextOutputFormat. It can generate a new file
> with the name of my choice for each key. But I want to name the output file
> for each partition with the name of my choice i.e. with the partition
> number.
> Please help me in this regard.
>
> Thanks
>
>