You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Chhaya Vishwakarma <Ch...@lntinfotech.com> on 2014/07/04 07:29:09 UTC
Working of combiner in hadoop
Hi,
If have two map tasks running on one node , i have written combiner class also.
Will combiner be called once for each map task or just once for both the map tasks
Can i write a logic inside map which will work as combiner ? if yes will there be any side effect?
Regards,
Chhaya Vishwakarma
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"
Re: Working of combiner in hadoop
Posted by JAGANADH G <ja...@gmail.com>.
On Fri, Jul 4, 2014 at 10:59 AM, Chhaya Vishwakarma <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
>
>
Hi Chaya,
Refer the following URLS
http://java.dzone.com/articles/designing-mapreduce-algorithms
http://isaacslavitt.com/2014/01/01/in-mapper-combiner-pattern-for-mapreduce/
http://alpinenow.com/blog/in-mapper-combiner/
Best regards
--
**********************************
JAGANADH G
http://jaganadhg.in
*ILUGCBE*
http://ilugcbe.org.in
Re: Working of combiner in hadoop
Posted by Chris Mawata <ch...@gmail.com>.
The key/value pairs are processes by the mapper independently of each
other. The combiner logic deals with all the outputs from multiple
key/value pairs do that logic can not be in the map method.
On Jul 4, 2014 1:29 AM, "Chhaya Vishwakarma" <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
> Regards,
>
> Chhaya Vishwakarma
>
>
>
> ------------------------------
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
Re: Working of combiner in hadoop
Posted by Chris Mawata <ch...@gmail.com>.
The key/value pairs are processes by the mapper independently of each
other. The combiner logic deals with all the outputs from multiple
key/value pairs do that logic can not be in the map method.
On Jul 4, 2014 1:29 AM, "Chhaya Vishwakarma" <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
> Regards,
>
> Chhaya Vishwakarma
>
>
>
> ------------------------------
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
Re: Working of combiner in hadoop
Posted by JAGANADH G <ja...@gmail.com>.
On Fri, Jul 4, 2014 at 10:59 AM, Chhaya Vishwakarma <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
>
>
Hi Chaya,
Refer the following URLS
http://java.dzone.com/articles/designing-mapreduce-algorithms
http://isaacslavitt.com/2014/01/01/in-mapper-combiner-pattern-for-mapreduce/
http://alpinenow.com/blog/in-mapper-combiner/
Best regards
--
**********************************
JAGANADH G
http://jaganadhg.in
*ILUGCBE*
http://ilugcbe.org.in
Re: Working of combiner in hadoop
Posted by JAGANADH G <ja...@gmail.com>.
On Fri, Jul 4, 2014 at 10:59 AM, Chhaya Vishwakarma <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
>
>
Hi Chaya,
Refer the following URLS
http://java.dzone.com/articles/designing-mapreduce-algorithms
http://isaacslavitt.com/2014/01/01/in-mapper-combiner-pattern-for-mapreduce/
http://alpinenow.com/blog/in-mapper-combiner/
Best regards
--
**********************************
JAGANADH G
http://jaganadhg.in
*ILUGCBE*
http://ilugcbe.org.in
Re: Working of combiner in hadoop
Posted by Chris Mawata <ch...@gmail.com>.
The key/value pairs are processes by the mapper independently of each
other. The combiner logic deals with all the outputs from multiple
key/value pairs do that logic can not be in the map method.
On Jul 4, 2014 1:29 AM, "Chhaya Vishwakarma" <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
> Regards,
>
> Chhaya Vishwakarma
>
>
>
> ------------------------------
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
Re: Working of combiner in hadoop
Posted by Chris Mawata <ch...@gmail.com>.
The key/value pairs are processes by the mapper independently of each
other. The combiner logic deals with all the outputs from multiple
key/value pairs do that logic can not be in the map method.
On Jul 4, 2014 1:29 AM, "Chhaya Vishwakarma" <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
> Regards,
>
> Chhaya Vishwakarma
>
>
>
> ------------------------------
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
Re: Working of combiner in hadoop
Posted by JAGANADH G <ja...@gmail.com>.
On Fri, Jul 4, 2014 at 10:59 AM, Chhaya Vishwakarma <
Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi,
>
>
>
> If have two map tasks running on one node , i have written combiner class
> also.
>
> Will combiner be called once for each map task or just once for both the
> map tasks
>
>
>
> Can i write a logic inside map which will work as combiner ? if yes will
> there be any side effect?
>
>
>
>
>
Hi Chaya,
Refer the following URLS
http://java.dzone.com/articles/designing-mapreduce-algorithms
http://isaacslavitt.com/2014/01/01/in-mapper-combiner-pattern-for-mapreduce/
http://alpinenow.com/blog/in-mapper-combiner/
Best regards
--
**********************************
JAGANADH G
http://jaganadhg.in
*ILUGCBE*
http://ilugcbe.org.in