You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Tina Samuel <ti...@gmail.com> on 2014/07/02 08:00:59 UTC
Duplicating map reduce tasks
Hi,
I would like to provide the map reduce jobs in the following format :-
<Map Reduce task> <number_of_replicas>
I want to execute the specified task the specified number of
times(number_of_replicas) on different nodes and then I want to compare the
results produced by these task copies. Is it possible to do this in Hadoop
Map reduce? If not possible, is there any means by which I can modify the
code of map reduce so that I can do it?
Thanks & Regards,
Tina
Re: Duplicating map reduce tasks
Posted by Tina Samuel <ti...@gmail.com>.
Hi,
Thanks for responding. Is there any method to schedule a map reduce task ,
say WordCount on all the live worker nodes?
Regards,
Tina
Regards,
Tina
On Wed, Jul 2, 2014 at 7:47 PM, sudhakara st <su...@gmail.com> wrote:
> I don't think we can achieve this by changing one class or two class, Look
> into from JobSubmitter.java, TaskAttemptID,java ... to speculative
> execution may it get helps you. Running the code on different replicas of
> the same block will produce will always produce same results,
>
>
> On Wed, Jul 2, 2014 at 7:14 PM, Tina Samuel <ti...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Thanks for responding.Is it possible to achieve my requirement by
> modifying
> > some code of map reduce,by which I will be able to replicate every map
> > reduce task that is submitted?
> >
> > Thanks & Regards,
> > Tina
> >
> >
> > Regards,
> > Tina
> >
> >
> > On Wed, Jul 2, 2014 at 6:47 PM, Wellington Chevreuil <
> > wellington.chevreuil@gmail.com> wrote:
> >
> > > Hi Tina,
> > >
> > > That's not controllable with hadoop mr. Hadoop sometimes will do this
> on
> > > its own (if you have speculative execution enabled), just for
> performance
> > > reasons case a given task is taking to much to complete under a given
> > node,
> > > and running the same code under replica of the same file will always
> > > produce same results, then it launches the same task under different
> > nodes,
> > > and once the first is finished, the result is ready and the others are
> > > killed.
> > >
> > > Cheers.
> > >
> > > On 2 Jul 2014, at 07:00, Tina Samuel <ti...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > I would like to provide the map reduce jobs in the following format
> :-
> > > >
> > > > <Map Reduce task> <number_of_replicas>
> > > >
> > > > I want to execute the specified task the specified number of
> > > > times(number_of_replicas) on different nodes and then I want to
> compare
> > > the
> > > > results produced by these task copies. Is it possible to do this in
> > > Hadoop
> > > > Map reduce? If not possible, is there any means by which I can modify
> > the
> > > > code of map reduce so that I can do it?
> > > >
> > > > Thanks & Regards,
> > > > Tina
> > >
> > >
> >
>
>
>
> --
>
> Regards,
> ...sudhakara
>
Re: Duplicating map reduce tasks
Posted by sudhakara st <su...@gmail.com>.
I don't think we can achieve this by changing one class or two class, Look
into from JobSubmitter.java, TaskAttemptID,java ... to speculative
execution may it get helps you. Running the code on different replicas of
the same block will produce will always produce same results,
On Wed, Jul 2, 2014 at 7:14 PM, Tina Samuel <ti...@gmail.com> wrote:
> Hi,
>
> Thanks for responding.Is it possible to achieve my requirement by modifying
> some code of map reduce,by which I will be able to replicate every map
> reduce task that is submitted?
>
> Thanks & Regards,
> Tina
>
>
> Regards,
> Tina
>
>
> On Wed, Jul 2, 2014 at 6:47 PM, Wellington Chevreuil <
> wellington.chevreuil@gmail.com> wrote:
>
> > Hi Tina,
> >
> > That's not controllable with hadoop mr. Hadoop sometimes will do this on
> > its own (if you have speculative execution enabled), just for performance
> > reasons case a given task is taking to much to complete under a given
> node,
> > and running the same code under replica of the same file will always
> > produce same results, then it launches the same task under different
> nodes,
> > and once the first is finished, the result is ready and the others are
> > killed.
> >
> > Cheers.
> >
> > On 2 Jul 2014, at 07:00, Tina Samuel <ti...@gmail.com> wrote:
> >
> > > Hi,
> > > I would like to provide the map reduce jobs in the following format :-
> > >
> > > <Map Reduce task> <number_of_replicas>
> > >
> > > I want to execute the specified task the specified number of
> > > times(number_of_replicas) on different nodes and then I want to compare
> > the
> > > results produced by these task copies. Is it possible to do this in
> > Hadoop
> > > Map reduce? If not possible, is there any means by which I can modify
> the
> > > code of map reduce so that I can do it?
> > >
> > > Thanks & Regards,
> > > Tina
> >
> >
>
--
Regards,
...sudhakara
Re: Duplicating map reduce tasks
Posted by Tina Samuel <ti...@gmail.com>.
Hi,
Thanks for responding.Is it possible to achieve my requirement by modifying
some code of map reduce,by which I will be able to replicate every map
reduce task that is submitted?
Thanks & Regards,
Tina
Regards,
Tina
On Wed, Jul 2, 2014 at 6:47 PM, Wellington Chevreuil <
wellington.chevreuil@gmail.com> wrote:
> Hi Tina,
>
> That's not controllable with hadoop mr. Hadoop sometimes will do this on
> its own (if you have speculative execution enabled), just for performance
> reasons case a given task is taking to much to complete under a given node,
> and running the same code under replica of the same file will always
> produce same results, then it launches the same task under different nodes,
> and once the first is finished, the result is ready and the others are
> killed.
>
> Cheers.
>
> On 2 Jul 2014, at 07:00, Tina Samuel <ti...@gmail.com> wrote:
>
> > Hi,
> > I would like to provide the map reduce jobs in the following format :-
> >
> > <Map Reduce task> <number_of_replicas>
> >
> > I want to execute the specified task the specified number of
> > times(number_of_replicas) on different nodes and then I want to compare
> the
> > results produced by these task copies. Is it possible to do this in
> Hadoop
> > Map reduce? If not possible, is there any means by which I can modify the
> > code of map reduce so that I can do it?
> >
> > Thanks & Regards,
> > Tina
>
>
Re: Duplicating map reduce tasks
Posted by Wellington Chevreuil <we...@gmail.com>.
Hi Tina,
That's not controllable with hadoop mr. Hadoop sometimes will do this on its own (if you have speculative execution enabled), just for performance reasons case a given task is taking to much to complete under a given node, and running the same code under replica of the same file will always produce same results, then it launches the same task under different nodes, and once the first is finished, the result is ready and the others are killed.
Cheers.
On 2 Jul 2014, at 07:00, Tina Samuel <ti...@gmail.com> wrote:
> Hi,
> I would like to provide the map reduce jobs in the following format :-
>
> <Map Reduce task> <number_of_replicas>
>
> I want to execute the specified task the specified number of
> times(number_of_replicas) on different nodes and then I want to compare the
> results produced by these task copies. Is it possible to do this in Hadoop
> Map reduce? If not possible, is there any means by which I can modify the
> code of map reduce so that I can do it?
>
> Thanks & Regards,
> Tina