You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Todd <bi...@163.com> on 2014/12/20 17:53:19 UTC
Combiner doesn't take effect if there is no reducer phrase
Hi, Hadoopers,
It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Re:Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Todd <bi...@163.com>.
Thanks Kai, I understood, thanks.
At 2014-12-21 01:13:15, "Kai Voigt" <k...@123.org> wrote:
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
Hi, Hadoopers,
It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai VoigtAm Germaniahafen 1k@123.org
24143 Kiel+49 160 96683050
Germany@KaiVoigt
Re:Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Todd <bi...@163.com>.
Thanks Kai, I understood, thanks.
At 2014-12-21 01:13:15, "Kai Voigt" <k...@123.org> wrote:
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
Hi, Hadoopers,
It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai VoigtAm Germaniahafen 1k@123.org
24143 Kiel+49 160 96683050
Germany@KaiVoigt
Re:Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Todd <bi...@163.com>.
Thanks Kai, I understood, thanks.
At 2014-12-21 01:13:15, "Kai Voigt" <k...@123.org> wrote:
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
Hi, Hadoopers,
It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai VoigtAm Germaniahafen 1k@123.org
24143 Kiel+49 160 96683050
Germany@KaiVoigt
Re:Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Todd <bi...@163.com>.
Thanks Kai, I understood, thanks.
At 2014-12-21 01:13:15, "Kai Voigt" <k...@123.org> wrote:
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
Hi, Hadoopers,
It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai VoigtAm Germaniahafen 1k@123.org
24143 Kiel+49 160 96683050
Germany@KaiVoigt
Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Kai Voigt <k...@123.org>.
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
> Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
>
> Hi, Hadoopers,
> It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
> I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Kai Voigt <k...@123.org>.
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
> Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
>
> Hi, Hadoopers,
> It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
> I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Kai Voigt <k...@123.org>.
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
> Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
>
> Hi, Hadoopers,
> It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
> I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: Combiner doesn't take effect if there is no reducer phrase
Posted by Kai Voigt <k...@123.org>.
Your observation is correct, the combine code will not execute if you disable the reduce phase, i.e. set numbers of reducers to 0.
Actually, the combine code might not be executed with a reduce phase, or it might be executed more than once on the map output. It’s the framework’s decision, so don’t put anything into the combine code you expect to run.
Kai
> Am 20.12.2014 um 17:53 schrieb Todd <bi...@163.com>:
>
> Hi, Hadoopers,
> It looks to me that when I provide the combiner but doesn't provide the reducer,the combiner doesn't take effect or combiner doesn't get run at all.
> I don't understand here because the combiner is running on the Mapper node, so could some one please explain the behavior of Combiner?Thanks.
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt