You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by javateck javateck <ja...@gmail.com> on 2009/04/22 02:48:54 UTC
mapred.tasktracker.map.tasks.maximum not working
Hi,
I'm struggling with mapred.tasktracker.map.tasks.maximum, I set to 10 in
hadoop-site.xml, and I can see the job's configuration is also saying 10
when running the job, but the actual job is running at 2 map tasks which is
the default behavior in hadoop-default.xml. See below.
Anyone has this experience and how to set it right?
*Kind*
*% Complete*
*Num Tasks*
*Pending*
*Running*
*Complete*
*Killed*
*Failed/Killed*
*Task Attempts*
*map*
7.40%
189
173
2
14
0
0 / 0
*reduce*
0.00%
4
4
0
0
0
0 / 0
Re: mapred.tasktracker.map.tasks.maximum not working
Posted by javateck javateck <ja...@gmail.com>.
thanks, Zheng & Aaron. Actually I got it working as expected now, I guess it
could be because of my machine's resource constraint. I know it's related to
total number of map tasks for one machine, but previously it was just one
job that was running, I feel weird. Anyway it's running as expected now.
On Fri, Apr 24, 2009 at 3:05 AM, Zheng Shao <zs...@gmail.com> wrote:
> Hi javatech,
>
> mapred.tasktracker.map.tasks.maximum: This is a TaskTracker configuration
> - you might need to restart the TaskTracker after you made the modification
> to conf/hadoop-site.xml
> Also, as Aaron points out, this is the number of maximum tasks per
> TaskTracker - not the same number as the actual concurrent map tasks of a
> job - but they seem to be related in your specific case.
>
>
> Zheng
>
>
> On Fri, Apr 24, 2009 at 12:20 AM, Aaron Kimball <aa...@cloudera.com>wrote:
>
>> mapred.tasktracker.map.tasks.maximum only sets the maximum number of map
>> tasks which may be run on a given machine. It is unrelated to the number of
>> tasks actually used in a given job.
>>
>> That's mapred.map.tasks
>>
>> - Aaron
>>
>>
>> On Wed, Apr 22, 2009 at 9:48 AM, javateck javateck <ja...@gmail.com>wrote:
>>
>>> Hi,
>>> I'm struggling with mapred.tasktracker.map.tasks.maximum, I set to 10
>>> in hadoop-site.xml, and I can see the job's configuration is also saying 10
>>> when running the job, but the actual job is running at 2 map tasks which is
>>> the default behavior in hadoop-default.xml. See below.
>>>
>>> Anyone has this experience and how to set it right?
>>>
>>> *Kind*
>>>
>>> *% Complete*
>>>
>>> *Num Tasks*
>>>
>>> *Pending*
>>>
>>> *Running*
>>>
>>> *Complete*
>>>
>>> *Killed*
>>>
>>> *Failed/Killed*
>>>
>>> *Task Attempts*
>>>
>>> *map*
>>>
>>> 7.40%
>>>
>>>
>>>
>>> 189
>>>
>>> 173
>>>
>>> 2
>>>
>>> 14
>>>
>>> 0
>>>
>>> 0 / 0
>>>
>>> *reduce*
>>>
>>> 0.00%
>>>
>>>
>>> 4
>>>
>>> 4
>>>
>>> 0
>>>
>>> 0
>>>
>>> 0
>>>
>>> 0 / 0
>>>
>>
>>
>
>
> --
> Yours,
> Zheng
>
Re: mapred.tasktracker.map.tasks.maximum not working
Posted by Zheng Shao <zs...@gmail.com>.
Hi javatech,
mapred.tasktracker.map.tasks.maximum: This is a TaskTracker configuration -
you might need to restart the TaskTracker after you made the modification to
conf/hadoop-site.xml
Also, as Aaron points out, this is the number of maximum tasks per
TaskTracker - not the same number as the actual concurrent map tasks of a
job - but they seem to be related in your specific case.
Zheng
On Fri, Apr 24, 2009 at 12:20 AM, Aaron Kimball <aa...@cloudera.com> wrote:
> mapred.tasktracker.map.tasks.maximum only sets the maximum number of map
> tasks which may be run on a given machine. It is unrelated to the number of
> tasks actually used in a given job.
>
> That's mapred.map.tasks
>
> - Aaron
>
>
> On Wed, Apr 22, 2009 at 9:48 AM, javateck javateck <ja...@gmail.com>wrote:
>
>> Hi,
>> I'm struggling with mapred.tasktracker.map.tasks.maximum, I set to 10
>> in hadoop-site.xml, and I can see the job's configuration is also saying 10
>> when running the job, but the actual job is running at 2 map tasks which is
>> the default behavior in hadoop-default.xml. See below.
>>
>> Anyone has this experience and how to set it right?
>>
>> *Kind*
>>
>> *% Complete*
>>
>> *Num Tasks*
>>
>> *Pending*
>>
>> *Running*
>>
>> *Complete*
>>
>> *Killed*
>>
>> *Failed/Killed*
>>
>> *Task Attempts*
>>
>> *map*
>>
>> 7.40%
>>
>>
>>
>> 189
>>
>> 173
>>
>> 2
>>
>> 14
>>
>> 0
>>
>> 0 / 0
>>
>> *reduce*
>>
>> 0.00%
>>
>>
>> 4
>>
>> 4
>>
>> 0
>>
>> 0
>>
>> 0
>>
>> 0 / 0
>>
>
>
--
Yours,
Zheng
Re: mapred.tasktracker.map.tasks.maximum not working
Posted by Aaron Kimball <aa...@cloudera.com>.
mapred.tasktracker.map.tasks.maximum only sets the maximum number of map
tasks which may be run on a given machine. It is unrelated to the number of
tasks actually used in a given job.
That's mapred.map.tasks
- Aaron
On Wed, Apr 22, 2009 at 9:48 AM, javateck javateck <ja...@gmail.com>wrote:
> Hi,
> I'm struggling with mapred.tasktracker.map.tasks.maximum, I set to 10 in
> hadoop-site.xml, and I can see the job's configuration is also saying 10
> when running the job, but the actual job is running at 2 map tasks which is
> the default behavior in hadoop-default.xml. See below.
>
> Anyone has this experience and how to set it right?
>
> *Kind*
>
> *% Complete*
>
> *Num Tasks*
>
> *Pending*
>
> *Running*
>
> *Complete*
>
> *Killed*
>
> *Failed/Killed*
>
> *Task Attempts*
>
> *map*
>
> 7.40%
>
>
>
> 189
>
> 173
>
> 2
>
> 14
>
> 0
>
> 0 / 0
>
> *reduce*
>
> 0.00%
>
>
> 4
>
> 4
>
> 0
>
> 0
>
> 0
>
> 0 / 0
>