You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by hong <mi...@163.com> on 2008/06/03 17:31:47 UTC
how to execute two consecutive map-reduce pairs?
Hi all
A job must be done in two pairs of map reduce, That is, Map1==>
reduce1 ==> map2==>reduce2. "==>" means the output file of left is
the input of the right.
To do that job, can I just create only one JobConf instance, and
invoke JobClient.runJob(conf) once?
Is there any similar example?
Thank you, in advance!
Re: how to execute two consecutive map-reduce pairs?
Posted by Christophe Taton <ta...@apache.org>.
Maybe you can check org.apache.hadoop.mapred.jobcontrol.*
I did not try it myself but it looks like this is what you need.
Cheers,
Christophe
On Tue, Jun 3, 2008 at 5:55 PM, Ted Dunning <te...@gmail.com> wrote:
> No.
>
> At least you need to call runJob twice. Typically, it is safer to create
> two job configurations so you don't forget to change something from the
> first jobs.
>
> It isn't a big deal. Just do it!
>
> On Tue, Jun 3, 2008 at 8:31 AM, hong <mi...@163.com> wrote:
>
> > Hi all
> >
> > A job must be done in two pairs of map reduce, That is, Map1==> reduce1
> ==>
> > map2==>reduce2. "==>" means the output file of left is the input of the
> > right.
> > To do that job, can I just create only one JobConf instance, and invoke
> > JobClient.runJob(conf) once?
> >
> > Is there any similar example?
> >
> > Thank you, in advance!
>
>
>
>
> --
> ted
>
Re: how to execute two consecutive map-reduce pairs?
Posted by Ted Dunning <te...@gmail.com>.
No.
At least you need to call runJob twice. Typically, it is safer to create
two job configurations so you don't forget to change something from the
first jobs.
It isn't a big deal. Just do it!
On Tue, Jun 3, 2008 at 8:31 AM, hong <mi...@163.com> wrote:
> Hi all
>
> A job must be done in two pairs of map reduce, That is, Map1==> reduce1 ==>
> map2==>reduce2. "==>" means the output file of left is the input of the
> right.
> To do that job, can I just create only one JobConf instance, and invoke
> JobClient.runJob(conf) once?
>
> Is there any similar example?
>
> Thank you, in advance!
--
ted