You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Nick Cen <ce...@gmail.com> on 2009/02/08 08:52:05 UTC
only one reducer running in a hadoop cluster
Hi,
I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
lucene together, so i copy some of the source code from nutch's Indexer
class, but when i run my job, i found that there is only 1 reducer running
on 1 pc, so the performance is not as far as expect.
--
http://daily.appspot.com/food/
Re: only one reducer running in a hadoop cluster
Posted by Nick Cen <ce...@gmail.com>.
Thanks everyone. I find the solution for this one, in my main method, i call
the setNumReductTask() on JobConf with the value i want.
2009/2/9 Owen O'Malley <om...@apache.org>
>
> On Feb 7, 2009, at 11:52 PM, Nick Cen wrote:
>
> Hi,
>>
>> I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
>> lucene together, so i copy some of the source code from nutch's Indexer
>> class, but when i run my job, i found that there is only 1 reducer running
>> on 1 pc, so the performance is not as far as expect.
>>
>
> Set mapred.reduce.tasks in your configuration to the number of reduces, you
> want your jobs to have by default. Typically this should be 0.99 *
> mapred.tasktracker.reduce.tasks.maximum * number of computers.
>
--
http://daily.appspot.com/food/
Re: only one reducer running in a hadoop cluster
Posted by Owen O'Malley <om...@apache.org>.
On Feb 7, 2009, at 11:52 PM, Nick Cen wrote:
> Hi,
>
> I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
> lucene together, so i copy some of the source code from nutch's
> Indexer
> class, but when i run my job, i found that there is only 1 reducer
> running
> on 1 pc, so the performance is not as far as expect.
Set mapred.reduce.tasks in your configuration to the number of
reduces, you want your jobs to have by default. Typically this should
be 0.99 * mapred.tasktracker.reduce.tasks.maximum * number of computers.
Re: only one reducer running in a hadoop cluster
Posted by Amareshwari Sriramadasu <am...@yahoo-inc.com>.
Nick Cen wrote:
> Hi,
>
> I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
> lucene together, so i copy some of the source code from nutch's Indexer
> class, but when i run my job, i found that there is only 1 reducer running
> on 1 pc, so the performance is not as far as expect.
>
>
what is the configuration of mapred.tasktracker.reduce.tasks.maximum ?
-Amareshwari