reduce more than one way

[Elia Mazzawi <el...@casalemedia.com>]

Hello,

I'm writing hadoop programs in Java,
I have 2 hadooop map/reduce programs that have the same map, but a 
different reduce methods.

can i run them in a way so that the map only happens once?

maybe store the intermediate result or something?

Re: reduce more than one way

[Owen O'Malley <om...@apache.org>]

On Nov 7, 2008, at 12:35 PM, Elia Mazzawi wrote:

> I have 2 hadooop map/reduce programs that have the same map, but a  
> different reduce methods.
>
> can i run them in a way so that the map only happens once?

If the input to the reduces is the same, you can put the two reduces  
together and use one of the multiple output libraries. That will let  
your reducer produce two different output directories.

-- Owen

Re: reduce more than one way

[Amar Kamat <am...@yahoo-inc.com>]

Elia Mazzawi wrote:
> Hello,
>
> I'm writing hadoop programs in Java,
> I have 2 hadooop map/reduce programs that have the same map, but a 
> different reduce methods. 
Look how MultipleOutputFormat is used. This provides the facility to 
write to multiple files.
Amar
>
> can i run them in a way so that the map only happens once?
>
> maybe store the intermediate result or something?

Re: reduce more than one way

[lohit <lo...@yahoo.com>]

There is mapper called IdentityMapper (look of IdentityMapper.java), which basically reads input and outputs without doing anything.
May be you can run your mapper with no reducers and store intermediate output and then run your 2 hadoop programs with Identity mapper and different set of reducers.
Thanks,
Lohit



----- Original Message ----
From: Elia Mazzawi <el...@casalemedia.com>
To: core-user@hadoop.apache.org
Sent: Friday, November 7, 2008 12:35:44 PM
Subject: reduce more than one way

Hello,

I'm writing hadoop programs in Java,
I have 2 hadooop map/reduce programs that have the same map, but a different reduce methods.

can i run them in a way so that the map only happens once?

maybe store the intermediate result or something?

Re: reduce more than one way

[Miles Osborne <mi...@inf.ed.ac.uk>]

why not just merge the two reducer

2008/11/7 Elia Mazzawi <el...@casalemedia.com>:
> Hello,
>
> I'm writing hadoop programs in Java,
> I have 2 hadooop map/reduce programs that have the same map, but a different
> reduce methods.
>
> can i run them in a way so that the map only happens once?
>
> maybe store the intermediate result or something?
>



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.