You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rakhi Khatwani <ra...@gmail.com> on 2009/07/17 15:06:49 UTC

Using chainmapper and chain reducer in hadoop

Hi
        I am trying out a simple example in which you have two maps and one
reduce, Map1 and Map2 and Reduce1

now i wanna execute my job in the following fashion

Map1 -> Map2 -> Reduce1 (o/p of map1 goes into map2 and o/p of map2 goes
into reduce1)


i have declared my conf as follows:


JobConf conf1 = new JobConf(ChainingJobsMain.class);
        conf1.setJobName("Job1");

        JobConf mapper1Conf = new JobConf();
        JobConf mapper2Conf = new JobConf();
        JobConf reducerConf = new JobConf();

ChainMapper chainMapper = new ChainMapper();
        ChainReducer chainReducer = new ChainReducer();
        chainMapper.addMapper(conf1, Map1.class, LongWritable.class,
Text.class, Text.class, Text.class, false, mapper1Conf);
        chainMapper.addMapper(conf1, Map2.class, Text.class, Text.class,
Text.class, Text.class, false, mapper2Conf);
        chainReducer.setReducer(conf1, Reduce1.class, LongWritable.class,
Text.class, Text.class, Text.class, false, reducerConf);

    //Input and output format
        conf1.setInputFormat(TextInputFormat.class);
        conf1.setOutputFormat(NullOutputFormat.class);
        FileInputFormat.setInputPaths(conf1, "/user/root/input");


        try {
            JobClient.runJob(conf1);
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }


but i get an output of a single map reduce. (i.e. map1 and reduce1.... i
dont see any map2 code executing).

Has anyone tried this out? and am i going wrong anywhere?? or does anyone
have a simple example which runs chainmapper???

Regards,
Raakhi

Re: Using chainmapper and chain reducer in hadoop

Posted by jason hadoop <ja...@gmail.com>.
The examples in ch8 and ch9 use chain mapping,
I will pick this conversation in the www.prohadoop.com forum

On Fri, Jul 17, 2009 at 9:34 AM, Rakhi Khatwani <ra...@gmail.com>wrote:

> Hi Jason,
>              I have the book with me and reading that book i got a fair
> idea about chaining maps/reduce. but i wanted to try out a simple example
> to
> get my concepts clear. i coudn't find any example which depicts the use of
> chainMapper/chainReducer. so i downloaded prohadoop.jar. but that also
> just contains the API's.
>
> actually i did output stuff from each map. i could view output only from
> Map1 and nothing from Map2 in the logs.
>
> Regards,
> Raakhi
>
> On Fri, Jul 17, 2009 at 6:59 PM, jason hadoop <jason.hadoop@gmail.com
> >wrote:
>
> > There are some examples for chain mapping in the example bundle for
> chapter
> > 8 of Pro Hadoop.
> > One think that may not be clear is that the chain of mappers execute
> within
> > the single task jvm assigned for each map task,
> > and the mappers chained to the reduce execute in the jvm assigned to the
> > reduce.
> > So: if your job has 1 input split and 1 reduce, you would only see 1 map
> > task and one reduce task in the JobTracker Web interface.
> >
> > To clarify:
> >
> > The 2 maps you define above would execute simultaneously. When you call
> > output.collect from Map1, there is a little bit of glue code, then
> > Map2.mapper will be called with the key, value pair from the
> > output.collect,
> > when map2.mapper returns, the line of code after the Map1 output.collect
> > will execute.
> >
> > If you pass true, instead of false, in your example above, a copy of the
> > key/value pair will be the passed.
> >
> >
> > On Fri, Jul 17, 2009 at 6:06 AM, Rakhi Khatwani <
> rakhi.khatwani@gmail.com
> > >wrote:
> >
> > > Hi
> > >        I am trying out a simple example in which you have two maps and
> > one
> > > reduce, Map1 and Map2 and Reduce1
> > >
> > > now i wanna execute my job in the following fashion
> > >
> > > Map1 -> Map2 -> Reduce1 (o/p of map1 goes into map2 and o/p of map2
> goes
> > > into reduce1)
> > >
> > >
> > > i have declared my conf as follows:
> > >
> > >
> > > JobConf conf1 = new JobConf(ChainingJobsMain.class);
> > >        conf1.setJobName("Job1");
> > >
> > >        JobConf mapper1Conf = new JobConf();
> > >        JobConf mapper2Conf = new JobConf();
> > >        JobConf reducerConf = new JobConf();
> > >
> > > ChainMapper chainMapper = new ChainMapper();
> > >        ChainReducer chainReducer = new ChainReducer();
> > >        chainMapper.addMapper(conf1, Map1.class, LongWritable.class,
> > > Text.class, Text.class, Text.class, false, mapper1Conf);
> > >        chainMapper.addMapper(conf1, Map2.class, Text.class, Text.class,
> > > Text.class, Text.class, false, mapper2Conf);
> > >        chainReducer.setReducer(conf1, Reduce1.class,
> LongWritable.class,
> > > Text.class, Text.class, Text.class, false, reducerConf);
> > >
> > >    //Input and output format
> > >        conf1.setInputFormat(TextInputFormat.class);
> > >        conf1.setOutputFormat(NullOutputFormat.class);
> > >        FileInputFormat.setInputPaths(conf1, "/user/root/input");
> > >
> > >
> > >        try {
> > >            JobClient.runJob(conf1);
> > >        } catch (IOException e) {
> > >            // TODO Auto-generated catch block
> > >            e.printStackTrace();
> > >        }
> > >
> > >
> > > but i get an output of a single map reduce. (i.e. map1 and reduce1....
> i
> > > dont see any map2 code executing).
> > >
> > > Has anyone tried this out? and am i going wrong anywhere?? or does
> anyone
> > > have a simple example which runs chainmapper???
> > >
> > > Regards,
> > > Raakhi
> > >
> >
> >
> >
> > --
> > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > www.prohadoopbook.com a community for Hadoop Professionals
> >
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Re: Using chainmapper and chain reducer in hadoop

Posted by Rakhi Khatwani <ra...@gmail.com>.
Hi Jason,
              I have the book with me and reading that book i got a fair
idea about chaining maps/reduce. but i wanted to try out a simple example to
get my concepts clear. i coudn't find any example which depicts the use of
chainMapper/chainReducer. so i downloaded prohadoop.jar. but that also
just contains the API's.

actually i did output stuff from each map. i could view output only from
Map1 and nothing from Map2 in the logs.

Regards,
Raakhi

On Fri, Jul 17, 2009 at 6:59 PM, jason hadoop <ja...@gmail.com>wrote:

> There are some examples for chain mapping in the example bundle for chapter
> 8 of Pro Hadoop.
> One think that may not be clear is that the chain of mappers execute within
> the single task jvm assigned for each map task,
> and the mappers chained to the reduce execute in the jvm assigned to the
> reduce.
> So: if your job has 1 input split and 1 reduce, you would only see 1 map
> task and one reduce task in the JobTracker Web interface.
>
> To clarify:
>
> The 2 maps you define above would execute simultaneously. When you call
> output.collect from Map1, there is a little bit of glue code, then
> Map2.mapper will be called with the key, value pair from the
> output.collect,
> when map2.mapper returns, the line of code after the Map1 output.collect
> will execute.
>
> If you pass true, instead of false, in your example above, a copy of the
> key/value pair will be the passed.
>
>
> On Fri, Jul 17, 2009 at 6:06 AM, Rakhi Khatwani <rakhi.khatwani@gmail.com
> >wrote:
>
> > Hi
> >        I am trying out a simple example in which you have two maps and
> one
> > reduce, Map1 and Map2 and Reduce1
> >
> > now i wanna execute my job in the following fashion
> >
> > Map1 -> Map2 -> Reduce1 (o/p of map1 goes into map2 and o/p of map2 goes
> > into reduce1)
> >
> >
> > i have declared my conf as follows:
> >
> >
> > JobConf conf1 = new JobConf(ChainingJobsMain.class);
> >        conf1.setJobName("Job1");
> >
> >        JobConf mapper1Conf = new JobConf();
> >        JobConf mapper2Conf = new JobConf();
> >        JobConf reducerConf = new JobConf();
> >
> > ChainMapper chainMapper = new ChainMapper();
> >        ChainReducer chainReducer = new ChainReducer();
> >        chainMapper.addMapper(conf1, Map1.class, LongWritable.class,
> > Text.class, Text.class, Text.class, false, mapper1Conf);
> >        chainMapper.addMapper(conf1, Map2.class, Text.class, Text.class,
> > Text.class, Text.class, false, mapper2Conf);
> >        chainReducer.setReducer(conf1, Reduce1.class, LongWritable.class,
> > Text.class, Text.class, Text.class, false, reducerConf);
> >
> >    //Input and output format
> >        conf1.setInputFormat(TextInputFormat.class);
> >        conf1.setOutputFormat(NullOutputFormat.class);
> >        FileInputFormat.setInputPaths(conf1, "/user/root/input");
> >
> >
> >        try {
> >            JobClient.runJob(conf1);
> >        } catch (IOException e) {
> >            // TODO Auto-generated catch block
> >            e.printStackTrace();
> >        }
> >
> >
> > but i get an output of a single map reduce. (i.e. map1 and reduce1.... i
> > dont see any map2 code executing).
> >
> > Has anyone tried this out? and am i going wrong anywhere?? or does anyone
> > have a simple example which runs chainmapper???
> >
> > Regards,
> > Raakhi
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>

Re: Using chainmapper and chain reducer in hadoop

Posted by jason hadoop <ja...@gmail.com>.
There are some examples for chain mapping in the example bundle for chapter
8 of Pro Hadoop.
One think that may not be clear is that the chain of mappers execute within
the single task jvm assigned for each map task,
and the mappers chained to the reduce execute in the jvm assigned to the
reduce.
So: if your job has 1 input split and 1 reduce, you would only see 1 map
task and one reduce task in the JobTracker Web interface.

To clarify:

The 2 maps you define above would execute simultaneously. When you call
output.collect from Map1, there is a little bit of glue code, then
Map2.mapper will be called with the key, value pair from the output.collect,
when map2.mapper returns, the line of code after the Map1 output.collect
will execute.

If you pass true, instead of false, in your example above, a copy of the
key/value pair will be the passed.


On Fri, Jul 17, 2009 at 6:06 AM, Rakhi Khatwani <ra...@gmail.com>wrote:

> Hi
>        I am trying out a simple example in which you have two maps and one
> reduce, Map1 and Map2 and Reduce1
>
> now i wanna execute my job in the following fashion
>
> Map1 -> Map2 -> Reduce1 (o/p of map1 goes into map2 and o/p of map2 goes
> into reduce1)
>
>
> i have declared my conf as follows:
>
>
> JobConf conf1 = new JobConf(ChainingJobsMain.class);
>        conf1.setJobName("Job1");
>
>        JobConf mapper1Conf = new JobConf();
>        JobConf mapper2Conf = new JobConf();
>        JobConf reducerConf = new JobConf();
>
> ChainMapper chainMapper = new ChainMapper();
>        ChainReducer chainReducer = new ChainReducer();
>        chainMapper.addMapper(conf1, Map1.class, LongWritable.class,
> Text.class, Text.class, Text.class, false, mapper1Conf);
>        chainMapper.addMapper(conf1, Map2.class, Text.class, Text.class,
> Text.class, Text.class, false, mapper2Conf);
>        chainReducer.setReducer(conf1, Reduce1.class, LongWritable.class,
> Text.class, Text.class, Text.class, false, reducerConf);
>
>    //Input and output format
>        conf1.setInputFormat(TextInputFormat.class);
>        conf1.setOutputFormat(NullOutputFormat.class);
>        FileInputFormat.setInputPaths(conf1, "/user/root/input");
>
>
>        try {
>            JobClient.runJob(conf1);
>        } catch (IOException e) {
>            // TODO Auto-generated catch block
>            e.printStackTrace();
>        }
>
>
> but i get an output of a single map reduce. (i.e. map1 and reduce1.... i
> dont see any map2 code executing).
>
> Has anyone tried this out? and am i going wrong anywhere?? or does anyone
> have a simple example which runs chainmapper???
>
> Regards,
> Raakhi
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals