You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Suresh S <su...@gmail.com> on 2013/01/02 13:23:30 UTC
running Combiner for all the task on the node.
Hello,
I think, running combiner function at node level (to combine all the
map task output of the node) may reduce the intermediate data movement.
I don't know this technique is already available or not. Is it worth
for working in this direction?
Any suggestions? Thanks in advance.
*Regards*
*S.Suresh,*
*Research Scholar,*
*Department of Computer Applications,*
*National Institute of Technology,*
*Tiruchirappalli - 620015.*
*+91-9941506562*
Re: running Combiner for all the task on the node.
Posted by Harsh J <ha...@cloudera.com>.
Hi Suresh,
This has been in good progress for a while, please see
https://issues.apache.org/jira/browse/MAPREDUCE-4502 and sub-tasks.
On Wed, Jan 2, 2013 at 5:53 PM, Suresh S <su...@gmail.com> wrote:
> Hello,
>
> I think, running combiner function at node level (to combine all the
> map task output of the node) may reduce the intermediate data movement.
>
> I don't know this technique is already available or not. Is it worth
> for working in this direction?
> Any suggestions? Thanks in advance.
> *Regards*
> *S.Suresh,*
> *Research Scholar,*
> *Department of Computer Applications,*
> *National Institute of Technology,*
> *Tiruchirappalli - 620015.*
> *+91-9941506562*
--
Harsh J
Re: running Combiner for all the task on the node.
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Suresh,
I mean, in current approach combine phase will occur per mapper
instance only, but NOT per node.
I think additional local shuffle and sort phase should happen so
that you can combine per node.
But NOT really sure whether this can be achievable or not. You
can give a try and see whether you find any complications.
Thanks,
Mahesh Balija,
Calsoft Labs.
On Wed, Jan 2, 2013 at 6:34 PM, Suresh S <su...@gmail.com> wrote:
> Definitely it will cost some overload. This lead to less intermediate
> movement and less time for reduce phase. This benefit may improve the
> overall performance.
> *Regards*
> *S.Suresh,*
> *Research Scholar,*
> *Department of Computer Applications,*
> *National Institute of Technology,*
> *Tiruchirappalli - 620015.*
> *+91-9941506562*
>
>
> On Wed, Jan 2, 2013 at 6:20 PM, Mahesh Balija <balijamahesh.mca@gmail.com
> >wrote:
>
> > Continued,
> >
> > Also one more shuffle and sort phase should occur so that you can
> > merge/combine them properly.
> > So you should decide whether additional shuffle and sort phase will be
> > overhead in contrast with combine per node.
> >
> > Best,
> > Mahesh Balija,
> > Calsoft Labs.
> >
> > On Wed, Jan 2, 2013 at 6:14 PM, Mahesh Balija <
> balijamahesh.mca@gmail.com
> > >wrote:
> >
> > > Hi Suresh,
> > >
> > > The combiner function will aggregate the data from a
> > single
> > > map instance. But NOT for all the maps running in a given node.
> > > AFAIK As the maps will be running in the individual
> child
> > > JVMs, still the intermediate data need to be serialized (moved) so that
> > > your combiner can aggregate the data at Node level.
> > >
> > > Best,
> > > Mahesh Balija,
> > > Calsoft Labs.
> > >
> > >
> > >
> > > On Wed, Jan 2, 2013 at 5:53 PM, Suresh S <su...@gmail.com> wrote:
> > >
> > >> Hello,
> > >>
> > >> I think, running combiner function at node level (to combine all
> > the
> > >> map task output of the node) may reduce the intermediate data
> movement.
> > >>
> > >> I don't know this technique is already available or not. Is it
> > worth
> > >> for working in this direction?
> > >> Any suggestions? Thanks in advance.
> > >> *Regards*
> > >> *S.Suresh,*
> > >> *Research Scholar,*
> > >> *Department of Computer Applications,*
> > >> *National Institute of Technology,*
> > >> *Tiruchirappalli - 620015.*
> > >> *+91-9941506562*
> > >>
> > >
> > >
> >
>
Re: running Combiner for all the task on the node.
Posted by Suresh S <su...@gmail.com>.
Definitely it will cost some overload. This lead to less intermediate
movement and less time for reduce phase. This benefit may improve the
overall performance.
*Regards*
*S.Suresh,*
*Research Scholar,*
*Department of Computer Applications,*
*National Institute of Technology,*
*Tiruchirappalli - 620015.*
*+91-9941506562*
On Wed, Jan 2, 2013 at 6:20 PM, Mahesh Balija <ba...@gmail.com>wrote:
> Continued,
>
> Also one more shuffle and sort phase should occur so that you can
> merge/combine them properly.
> So you should decide whether additional shuffle and sort phase will be
> overhead in contrast with combine per node.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
> On Wed, Jan 2, 2013 at 6:14 PM, Mahesh Balija <balijamahesh.mca@gmail.com
> >wrote:
>
> > Hi Suresh,
> >
> > The combiner function will aggregate the data from a
> single
> > map instance. But NOT for all the maps running in a given node.
> > AFAIK As the maps will be running in the individual child
> > JVMs, still the intermediate data need to be serialized (moved) so that
> > your combiner can aggregate the data at Node level.
> >
> > Best,
> > Mahesh Balija,
> > Calsoft Labs.
> >
> >
> >
> > On Wed, Jan 2, 2013 at 5:53 PM, Suresh S <su...@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> I think, running combiner function at node level (to combine all
> the
> >> map task output of the node) may reduce the intermediate data movement.
> >>
> >> I don't know this technique is already available or not. Is it
> worth
> >> for working in this direction?
> >> Any suggestions? Thanks in advance.
> >> *Regards*
> >> *S.Suresh,*
> >> *Research Scholar,*
> >> *Department of Computer Applications,*
> >> *National Institute of Technology,*
> >> *Tiruchirappalli - 620015.*
> >> *+91-9941506562*
> >>
> >
> >
>
Re: running Combiner for all the task on the node.
Posted by Mahesh Balija <ba...@gmail.com>.
Continued,
Also one more shuffle and sort phase should occur so that you can
merge/combine them properly.
So you should decide whether additional shuffle and sort phase will be
overhead in contrast with combine per node.
Best,
Mahesh Balija,
Calsoft Labs.
On Wed, Jan 2, 2013 at 6:14 PM, Mahesh Balija <ba...@gmail.com>wrote:
> Hi Suresh,
>
> The combiner function will aggregate the data from a single
> map instance. But NOT for all the maps running in a given node.
> AFAIK As the maps will be running in the individual child
> JVMs, still the intermediate data need to be serialized (moved) so that
> your combiner can aggregate the data at Node level.
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
>
>
> On Wed, Jan 2, 2013 at 5:53 PM, Suresh S <su...@gmail.com> wrote:
>
>> Hello,
>>
>> I think, running combiner function at node level (to combine all the
>> map task output of the node) may reduce the intermediate data movement.
>>
>> I don't know this technique is already available or not. Is it worth
>> for working in this direction?
>> Any suggestions? Thanks in advance.
>> *Regards*
>> *S.Suresh,*
>> *Research Scholar,*
>> *Department of Computer Applications,*
>> *National Institute of Technology,*
>> *Tiruchirappalli - 620015.*
>> *+91-9941506562*
>>
>
>
Re: running Combiner for all the task on the node.
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Suresh,
The combiner function will aggregate the data from a single
map instance. But NOT for all the maps running in a given node.
AFAIK As the maps will be running in the individual child
JVMs, still the intermediate data need to be serialized (moved) so that
your combiner can aggregate the data at Node level.
Best,
Mahesh Balija,
Calsoft Labs.
On Wed, Jan 2, 2013 at 5:53 PM, Suresh S <su...@gmail.com> wrote:
> Hello,
>
> I think, running combiner function at node level (to combine all the
> map task output of the node) may reduce the intermediate data movement.
>
> I don't know this technique is already available or not. Is it worth
> for working in this direction?
> Any suggestions? Thanks in advance.
> *Regards*
> *S.Suresh,*
> *Research Scholar,*
> *Department of Computer Applications,*
> *National Institute of Technology,*
> *Tiruchirappalli - 620015.*
> *+91-9941506562*
>