You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Lucian Iordache <lu...@gmail.com> on 2011/07/01 10:55:21 UTC
Relation between Mapper and Combiner
Hello guys,
Can anybody tell me which is the relation between map task and combine
tasks?
I would like to know if there is a 1:1 relation between them, or is a *:1
(many to one).
For example:
If I have *2 map tasks* ran on the same machine, will I have *1 combine task
* on that machine to combine the maps outputs, *or 2 combine tasks*?
Best Regards,
--
Lucian
Re: Relation between Mapper and Combiner
Posted by Lucian Iordache <lu...@gmail.com>.
Ok, that is what I wanted to know.
Thanks you!
Best Regards,
Lucian
On Fri, Jul 1, 2011 at 2:47 PM, Devaraj K <de...@huawei.com> wrote:
> Hi Lucian,****
>
> ** **
>
> For every map task, combiner will be executed multiple times
> before writing the map output. Combine step is not a separate task and it is
> part of map task execution. Reducer will copy the output of the map task
> which is reduced by the combiner.****
>
> ** **
>
> >For example:
> >If I have *2 map tasks* ran on the same machine, will I have *1 combine
> task* on that machine to combine the maps outputs, *or 2 combine tasks*?**
> **
>
> ** **
>
> In this case, combiner will be executed for each map task independent to
> each other. This combiner step will execute multiple times till it gets same
> output for 1 or more runs of combiner.
>
> ****
>
> ** **
>
> You can go through Combiner section here for more info :
> http://wiki.apache.org/hadoop/HadoopMapReduce****
>
> ** **
>
> Devaraj K ****
>
>
> -------------------------------------------------------------------------------------------------------------------------------------
> This e-mail and its attachments contain confidential information from
> HUAWEI, which
> is intended only for the person or entity whose address is listed above.
> Any use of the
> information contained herein in any way (including, but not limited to,
> total or partial
> disclosure, reproduction, or dissemination) by persons other than the
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, please
> notify the sender by
> phone or email immediately and delete it!ss****
>
> ****
> ------------------------------
>
> *From:* Lucian Iordache [mailto:lucian.george.iordache@gmail.com]
> *Sent:* Friday, July 01, 2011 2:25 PM
> *To:* mapreduce-user@hadoop.apache.org
> *Subject:* Relation between Mapper and Combiner****
>
> ** **
>
> Hello guys,
>
> Can anybody tell me which is the relation between map task and combine
> tasks?
> I would like to know if there is a 1:1 relation between them, or is a *:1
> (many to one).
>
> For example:
> If I have *2 map tasks* ran on the same machine, will I have *1 combine
> task* on that machine to combine the maps outputs, *or 2 combine tasks*?
>
> Best Regards,
> --
> Lucian****
>
RE: Relation between Mapper and Combiner
Posted by Devaraj K <de...@huawei.com>.
Hi Lucian,
For every map task, combiner will be executed multiple times
before writing the map output. Combine step is not a separate task and it is
part of map task execution. Reducer will copy the output of the map task
which is reduced by the combiner.
>For example:
>If I have 2 map tasks ran on the same machine, will I have 1 combine task
on that machine to combine the maps outputs, or 2 combine tasks?
In this case, combiner will be executed for each map task independent to
each other. This combiner step will execute multiple times till it gets same
output for 1 or more runs of combiner.
You can go through Combiner section here for more info :
http://wiki.apache.org/hadoop/HadoopMapReduce
Devaraj K
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to,
total or partial
disclosure, reproduction, or dissemination) by persons other than the
intended
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by
phone or email immediately and delete it!ss
_____
From: Lucian Iordache [mailto:lucian.george.iordache@gmail.com]
Sent: Friday, July 01, 2011 2:25 PM
To: mapreduce-user@hadoop.apache.org
Subject: Relation between Mapper and Combiner
Hello guys,
Can anybody tell me which is the relation between map task and combine
tasks?
I would like to know if there is a 1:1 relation between them, or is a *:1
(many to one).
For example:
If I have 2 map tasks ran on the same machine, will I have 1 combine task on
that machine to combine the maps outputs, or 2 combine tasks?
Best Regards,
--
Lucian