You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Vibhooti Verma <ve...@gmail.com> on 2008/04/23 06:23:01 UTC
Getting map ouput as final output by setting number of reduce to zero
Has any one tried setting number of reduce to zero and getting map's output
as the final output?
I tried doing the same but my map output does not come to specified output
path for mapred.
let me know if someone has already done that. I am not able to find out,
where my map outputs are written.
http://hadoop.apache.org/core/docs/r0.16.3/mapred_tutorial.html say the
following
Reducer NONE
It is legal to set the number of reduce-tasks to *zero* if no reduction is
desired.
In this case the outputs of the map-tasks go directly to the FileSystem,
into the output path set by
setOutputPath(Path)<http://hadoop.apache.org/core/docs/r0.16.3/api/org/apache/hadoop/mapred/JobConf.html#setOutputPath%28org.apache.hadoop.fs.Path%29>.
The framework does not sort the map-outputs before writing them out to the
FileSystem.
--
cheers,
Vibhooti
Re: Getting map ouput as final output by setting number of reduce
to zero
Posted by Amar Kamat <am...@yahoo-inc.com>.
Vibhooti Verma wrote:
> Has any one tried setting number of reduce to zero and getting map's output
> as the final output?
>
Look at the RandomWriter example
(src/examples/org/apache/hadoop/examples/RandomWriter.java).
Amar
> I tried doing the same but my map output does not come to specified output
> path for mapred.
> let me know if someone has already done that. I am not able to find out,
> where my map outputs are written.
> http://hadoop.apache.org/core/docs/r0.16.3/mapred_tutorial.html say the
> following
>
> Reducer NONE
>
> It is legal to set the number of reduce-tasks to *zero* if no reduction is
> desired.
>
> In this case the outputs of the map-tasks go directly to the FileSystem,
> into the output path set by
> setOutputPath(Path)<http://hadoop.apache.org/core/docs/r0.16.3/api/org/apache/hadoop/mapred/JobConf.html#setOutputPath%28org.apache.hadoop.fs.Path%29>.
> The framework does not sort the map-outputs before writing them out to the
> FileSystem.
>
>
>
Re: Getting map ouput as final output by setting number of reduce to zero
Posted by Chris Dyer <re...@umd.edu>.
Setting the number of reducers to zero has the advantage that no
sorting of the intermediate values will occur, which can save
considerable amounts of time if the output from the mapper is large.
Chris
On Wed, Apr 30, 2008 at 6:47 PM, jkupferman <jk...@umail.ucsb.edu> wrote:
>
> You actually dont need to do that, you can just use an IdentityReducer. The
> IdentityReducer simply acts as a pass through, so whatever is written out
> from your mapper, is passed through it and then written out to file.
>
>
> vibhooti wrote:
> >
> > Has any one tried setting number of reduce to zero and getting map's
> > output
> > as the final output?
> > I tried doing the same but my map output does not come to specified output
> > path for mapred.
> > let me know if someone has already done that. I am not able to find out,
> > where my map outputs are written.
> > http://hadoop.apache.org/core/docs/r0.16.3/mapred_tutorial.html say the
> > following
> >
> > Reducer NONE
> >
> > It is legal to set the number of reduce-tasks to *zero* if no reduction is
> > desired.
> >
> > In this case the outputs of the map-tasks go directly to the FileSystem,
> > into the output path set by
> > setOutputPath(Path)<http://hadoop.apache.org/core/docs/r0.16.3/api/org/apache/hadoop/mapred/JobConf.html#setOutputPath%28org.apache.hadoop.fs.Path%29>.
> > The framework does not sort the map-outputs before writing them out to the
> > FileSystem.
> >
> >
> > --
> > cheers,
> > Vibhooti
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Getting-map-ouput-as-final-output-by-setting-number-of-reduce-to-zero-tp16847897p16992742.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Re: Getting map ouput as final output by setting number of reduce
to zero
Posted by jkupferman <jk...@umail.ucsb.edu>.
You actually dont need to do that, you can just use an IdentityReducer. The
IdentityReducer simply acts as a pass through, so whatever is written out
from your mapper, is passed through it and then written out to file.
vibhooti wrote:
>
> Has any one tried setting number of reduce to zero and getting map's
> output
> as the final output?
> I tried doing the same but my map output does not come to specified output
> path for mapred.
> let me know if someone has already done that. I am not able to find out,
> where my map outputs are written.
> http://hadoop.apache.org/core/docs/r0.16.3/mapred_tutorial.html say the
> following
>
> Reducer NONE
>
> It is legal to set the number of reduce-tasks to *zero* if no reduction is
> desired.
>
> In this case the outputs of the map-tasks go directly to the FileSystem,
> into the output path set by
> setOutputPath(Path)<http://hadoop.apache.org/core/docs/r0.16.3/api/org/apache/hadoop/mapred/JobConf.html#setOutputPath%28org.apache.hadoop.fs.Path%29>.
> The framework does not sort the map-outputs before writing them out to the
> FileSystem.
>
>
> --
> cheers,
> Vibhooti
>
>
--
View this message in context: http://www.nabble.com/Getting-map-ouput-as-final-output-by-setting-number-of-reduce-to-zero-tp16847897p16992742.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.