You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Murat Ali Bayir <mu...@agmlab.com> on 2006/08/10 14:29:45 UTC
number of mapper
Hi everbody, Although I change the number of mappers in hadoop-site.xml
and use job.setNumMapTasks method the system gives another number as a
number of mapper, the problem only occurs for number of mapper, number
of reducers works correctly. What I have to do for setting the number
of mappers in the system?
Re: number of mapper
Posted by Dennis Kubes <nu...@dragonflymc.com>.
Take a look at this,
http://wiki.apache.org/lucene-hadoop/HowManyMapsAndReduces
It will answer why you have a few more map tasks that are set in the
configuration.
Dennis
Murat Ali Bayir wrote:
> my configs are given below:
> in hadoop-site number of mapper = 130
> in my code I use job.setNumMapTasks = 130
> in hadoop-default numberof mapper = 2
> in this configuration I have taken 135 mapper in my job. However there
> is no problem in number of reducer.
>
> Andrzej Bialecki wrote:
>
>> Murat Ali Bayir wrote:
>>
>>> Hi everbody, Although I change the number of mappers in
>>> hadoop-site.xml and use job.setNumMapTasks method the system gives
>>> another number as a number of mapper, the problem only occurs for
>>> number of mapper, number of reducers works correctly. What I have
>>> to do for setting the number of mappers in the system?
>>
>>
>> Any value that you put in hadoop-site.xml will always override any
>> other config settings, even those set programatically in
>> job.setNumMapTasks. You should remove these settings from
>> hadoop-site, and put them into mapred-default.xml.
>>
>
Re: number of mapper
Posted by Murat Ali Bayir <mu...@agmlab.com>.
my configs are given below:
in hadoop-site number of mapper = 130
in my code I use job.setNumMapTasks = 130
in hadoop-default numberof mapper = 2
in this configuration I have taken 135 mapper in my job. However there
is no problem in number of reducer.
Andrzej Bialecki wrote:
> Murat Ali Bayir wrote:
>
>> Hi everbody, Although I change the number of mappers in
>> hadoop-site.xml and use job.setNumMapTasks method the system gives
>> another number as a number of mapper, the problem only occurs for
>> number of mapper, number of reducers works correctly. What I have to
>> do for setting the number of mappers in the system?
>
>
> Any value that you put in hadoop-site.xml will always override any
> other config settings, even those set programatically in
> job.setNumMapTasks. You should remove these settings from hadoop-site,
> and put them into mapred-default.xml.
>
Re: number of mapper
Posted by Andrzej Bialecki <ab...@getopt.org>.
Murat Ali Bayir wrote:
> Hi everbody, Although I change the number of mappers in
> hadoop-site.xml and use job.setNumMapTasks method the system gives
> another number as a number of mapper, the problem only occurs for
> number of mapper, number of reducers works correctly. What I have to
> do for setting the number of mappers in the system?
Any value that you put in hadoop-site.xml will always override any other
config settings, even those set programatically in job.setNumMapTasks.
You should remove these settings from hadoop-site, and put them into
mapred-default.xml.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: number of mapper
Posted by Murat Ali Bayir <mu...@agmlab.com>.
it can not be problem, it only restrict the number of tasks running
simultaneously, there can be pending tasks also, i check that this not
problem. I am not sure but I notice that the number of mapper tasks is
equal to k*number of different parts in input path. To illusrate I have
15 parts in my input path, I set the number of mappers 130 in
hadoop-site.xml however when I run the job I have 135 mapper which is 9
times of number of input part.
Dennis Kubes wrote:
> There is also a mapred.tasktracker.tasks.maximum variable which may be
> causing the task number to be different.
>
> Dennis
>
> Murat Ali Bayir wrote:
>
>> Hi everbody, Although I change the number of mappers in
>> hadoop-site.xml and use job.setNumMapTasks method the system gives
>> another number as a number of mapper, the problem only occurs for
>> number of mapper, number of reducers works correctly. What I have to
>> do for setting the number of mappers in the system?
>
>
>
> .
>
Re: number of mapper
Posted by Dennis Kubes <nu...@dragonflymc.com>.
There is also a mapred.tasktracker.tasks.maximum variable which may be
causing the task number to be different.
Dennis
Murat Ali Bayir wrote:
> Hi everbody, Although I change the number of mappers in
> hadoop-site.xml and use job.setNumMapTasks method the system gives
> another number as a number of mapper, the problem only occurs for
> number of mapper, number of reducers works correctly. What I have to
> do for setting the number of mappers in the system?