Merge Reducers Outputs

[Mohamed Riadh Trad <Mo...@inria.fr>]

Dear All,

Is it possible to set up a task with multiple reducers and merge reducers outputs into one single file?

Bests,

Trad Mohamed Riadh, M.Sc, Ing.
PhD. student
INRIA-TELECOM PARISTECH - ENPC School of International Management

Office: 11-15
Phone: (33)-1 39 63 59 33
Fax: (33)-1 39 63 56 74
Email: riadh.trad@inria.fr
Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/

Re: Merge Reducers Outputs

[David Rosenstrauch <da...@darose.net>]

On 07/26/2011 06:52 PM, Mohamed Riadh Trad wrote:
> Dear All,
>
> Is it possible to set up a task with multiple reducers and merge reducers outputs into one single file?
>
> Bests,
>
> Trad Mohamed Riadh, M.Sc, Ing.

Not within the map-reduce job, but you can merge it after the job is 
done.  At my previous company we used FileUtil.copyMerge() to do this, 
and it worked quite well.

See:

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileUtil.html#copyMerge%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20boolean,%20org.apache.hadoop.conf.Configuration,%20java.lang.String%29

DR

Re: Merge Reducers Outputs

[Arun C Murthy <ac...@hortonworks.com>]

No, you either have small enough data that you can have all go to a single reducer or you can setup a (sampling) partitioner so that the partitions are sorted and you can get globally sorted output from multiple reduces - take a look at the TeraSort example for this.

Arun

On Jul 26, 2011, at 3:52 PM, Mohamed Riadh Trad wrote:

> Dear All,
> 
> Is it possible to set up a task with multiple reducers and merge reducers outputs into one single file?
> 
> Bests,
> 
> Trad Mohamed Riadh, M.Sc, Ing.
> PhD. student
> INRIA-TELECOM PARISTECH - ENPC School of International Management
> 
> Office: 11-15
> Phone: (33)-1 39 63 59 33
> Fax: (33)-1 39 63 56 74
> Email: riadh.trad@inria.fr
> Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/