You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Keren Ouaknine <ke...@gmail.com> on 2011/11/25 04:15:55 UTC
#number of mappers
Hello,
I configured mapred-site.xml with one mapper, still the web ui shows:
Map shows capacity 40 = 2 x #machines. I was expecting it to be 20.
Any idea?
Thanks,
Keren
Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map
SlotsOccupied Reduce Slots Reserved Map SlotsReserved Reduce SlotsMap Task
CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted NodesGraylisted
NodesExcluded
Nodes 3420720 <http://10.239.24.12:50030/machines.jsp?type=active>3420004020
3.000 <http://10.239.24.12:50030/machines.jsp?type=blacklisted>
0<http://10.239.24.12:50030/machines.jsp?type=graylisted>
0 <http://10.239.24.12:50030/machines.jsp?type=excluded>
--
Keren Ouaknine
Cell: +972 54 2565404
Web: www.kereno.com
Re: #number of mappers
Posted by Keren Ouaknine <ke...@gmail.com>.
Indeed, that was the problem, thanks!
Keren
On Mon, Nov 28, 2011 at 10:46 PM, Harsh J <ha...@cloudera.com> wrote:
> Keren,
>
> Seems to work alright for me once I set these and restart each
> TaskTracker. Perhaps the problem is merely that you're missing an opening
> <property> XML tag for the second configuration, in that copy-paste below?
>
>
> On Tue, Nov 29, 2011 at 8:44 AM, Keren Ouaknine <ke...@gmail.com> wrote:
>
>> Thanks Harsh, the property in already set to one on all the nodes (see
>> below).
>> What do you reckon?
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <value>1</value>
>> <description>The maximum number of map tasks that will be run
>> simultaneously by a task tracker.</description>
>> </property>
>> <name>mapred.tasktracker.reduce.tasks.maximum</name>
>> <value>1</value>
>> <description>The maximum number of reduce tasks that will be run
>> simultaneously by a task tracker.</description>
>> </property>
>> </configuration>
>>
>>
>> On Thu, Nov 24, 2011 at 10:45 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Keren,
>>>
>>> What was your configured property at the TaskTrackers? The right
>>> property is mapred.tasktracker.{map,reduce}.tasks.maximum, and applies to
>>> each TaskTracker individually (so all your TT mapred-site.xml need to carry
>>> that property to limit their slots).
>>>
>>> On 25-Nov-2011, at 8:45 AM, Keren Ouaknine wrote:
>>>
>>> Hello,
>>>
>>> I configured mapred-site.xml with one mapper, still the web ui shows:
>>> Map shows capacity 40 = 2 x #machines. I was expecting it to be 20.
>>> Any idea?
>>>
>>> Thanks,
>>> Keren
>>>
>>>
>>> Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map
>>> SlotsOccupied Reduce Slots Reserved Map SlotsReserved Reduce SlotsMap
>>> Task CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted NodesGraylisted
>>> NodesExcluded Nodes 3420720<http://10.239.24.12:50030/machines.jsp?type=active>
>>> 34200040203.000<http://10.239.24.12:50030/machines.jsp?type=blacklisted>
>>> 0 <http://10.239.24.12:50030/machines.jsp?type=graylisted>0<http://10.239.24.12:50030/machines.jsp?type=excluded>
>>> --
>>> Keren Ouaknine
>>> Cell: +972 54 2565404
>>> Web: www.kereno.com
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Keren Ouaknine
>> Cell: +972 54 2565404
>> Web: www.kereno.com
>>
>>
>>
>
>
> --
> Harsh J
>
--
Keren Ouaknine
Cell: +972 54 2565404
Web: www.kereno.com
Re: #number of mappers
Posted by Harsh J <ha...@cloudera.com>.
Keren,
Seems to work alright for me once I set these and restart each TaskTracker.
Perhaps the problem is merely that you're missing an opening <property> XML
tag for the second configuration, in that copy-paste below?
On Tue, Nov 29, 2011 at 8:44 AM, Keren Ouaknine <ke...@gmail.com> wrote:
> Thanks Harsh, the property in already set to one on all the nodes (see
> below).
> What do you reckon?
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <value>1</value>
> <description>The maximum number of map tasks that will be run
> simultaneously by a task tracker.</description>
> </property>
> <name>mapred.tasktracker.reduce.tasks.maximum</name>
> <value>1</value>
> <description>The maximum number of reduce tasks that will be run
> simultaneously by a task tracker.</description>
> </property>
> </configuration>
>
>
> On Thu, Nov 24, 2011 at 10:45 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Keren,
>>
>> What was your configured property at the TaskTrackers? The right property
>> is mapred.tasktracker.{map,reduce}.tasks.maximum, and applies to each
>> TaskTracker individually (so all your TT mapred-site.xml need to carry that
>> property to limit their slots).
>>
>> On 25-Nov-2011, at 8:45 AM, Keren Ouaknine wrote:
>>
>> Hello,
>>
>> I configured mapred-site.xml with one mapper, still the web ui shows:
>> Map shows capacity 40 = 2 x #machines. I was expecting it to be 20.
>> Any idea?
>>
>> Thanks,
>> Keren
>>
>>
>> Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map
>> SlotsOccupied Reduce Slots Reserved Map SlotsReserved Reduce SlotsMap
>> Task CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted NodesGraylisted
>> NodesExcluded Nodes 3420720<http://10.239.24.12:50030/machines.jsp?type=active>
>> 34200040203.000 <http://10.239.24.12:50030/machines.jsp?type=blacklisted>
>> 0 <http://10.239.24.12:50030/machines.jsp?type=graylisted>0<http://10.239.24.12:50030/machines.jsp?type=excluded>
>> --
>> Keren Ouaknine
>> Cell: +972 54 2565404
>> Web: www.kereno.com
>>
>>
>>
>>
>
>
> --
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
>
>
>
--
Harsh J
Re: #number of mappers
Posted by Ronnie Dove <ro...@oceansync.com>.
Hello Karen,
Did you restart the TaskTracker after making the maximum change (it sounds like it was already set for you)? That mapred.tasktracker.map.tasks.maximum property sets the maximum number of map tasks which may be run on a given machine (TaskTracker). There is also a property for setting the number of tasks actually used in a given job (mapred.map.tasks). Which version of Hadoop are you running? Here is some good information: http://wiki.apache.org/hadoop/HowManyMapsAndReduces
________________________________
Ronnie Dove
OceanSync Hadoop Management
http://www.oceansync.com
________________________________
From: Keren Ouaknine <ke...@gmail.com>
To: mapreduce-user@hadoop.apache.org
Sent: Monday, November 28, 2011 10:14 PM
Subject: Re: #number of mappers
Thanks Harsh, the property in already set to one on all the nodes (see below).
What do you reckon?
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
<description>The maximum number of map tasks that will be run simultaneously by a task tracker.</description>
</property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
<description>The maximum number of reduce tasks that will be run simultaneously by a task tracker.</description>
</property>
</configuration>
On Thu, Nov 24, 2011 at 10:45 PM, Harsh J <ha...@cloudera.com> wrote:
Keren,
>
>
>What was your configured property at the TaskTrackers? The right property is mapred.tasktracker.{map,reduce}.tasks.maximum, and applies to each TaskTracker individually (so all your TT mapred-site.xml need to carry that property to limit their slots).
>
>
>On 25-Nov-2011, at 8:45 AM, Keren Ouaknine wrote:
>
>Hello,
>>
>>I configured mapred-site.xml with one mapper, still the web ui shows:
>>Map shows capacity 40 = 2 x #machines. I was expecting it to be 20.
>>Any idea?
>>
>>Thanks,
>>Keren
>>
>>
>>
>>Running Map Tasks
>>Running Reduce Tasks
>>Total Submissions
>>Nodes
>>Occupied Map Slots
>>Occupied Reduce Slots
>>Reserved Map Slots
>>Reserved Reduce Slots
>>Map Task Capacity
>>Reduce Task Capacity
>>Avg. Tasks/Node
>>Blacklisted Nodes
>>Graylisted Nodes
>>Excluded Nodes
>>34 20 7 20 34 20 0 0 40 20 3.00 0 0 0
>>--
>>
>>Keren OuaknineCell: +972 54 2565404
>>Web: www.kereno.com
>>
>>
>>
>
--
Keren OuaknineCell: +972 54 2565404
Web: www.kereno.com
25-Nov-2011, at 8:45 AM, Keren Ouaknine wrote:
Hello,
>
>I configured mapred-site.xml with one mapper, still the web ui shows:
>Map shows capacity 40 = 2 x #machines. I was expecting it to be 20.
>Any idea?
>
>Thanks,
>Keren
>
>
>
>Running Map Tasks
>Running Reduce Tasks
>Total Submissions
>Nodes
>Occupied Map Slots
>Occupied Reduce Slots
>Reserved Map Slots
>Reserved Reduce Slots
>Map Task Capacity
>Reduce Task Capacity
>Avg. Tasks/Node
>Blacklisted Nodes
>Graylisted Nodes
>Excluded Nodes
>34 20 7 20 34 20 0 0 40 20 3.00 0 0 0
>--
>
>Keren OuaknineCell: +972 54 2565404
>Web: www.kereno.com
>
>
>
--
Keren OuaknineCell: +972 54 2565404
Web: www.kereno.com
Re: #number of mappers
Posted by Keren Ouaknine <ke...@gmail.com>.
Thanks Harsh, the property in already set to one on all the nodes (see
below).
What do you reckon?
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
<description>The maximum number of map tasks that will be run
simultaneously by a task tracker.</description>
</property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
<description>The maximum number of reduce tasks that will be run
simultaneously by a task tracker.</description>
</property>
</configuration>
On Thu, Nov 24, 2011 at 10:45 PM, Harsh J <ha...@cloudera.com> wrote:
> Keren,
>
> What was your configured property at the TaskTrackers? The right property
> is mapred.tasktracker.{map,reduce}.tasks.maximum, and applies to each
> TaskTracker individually (so all your TT mapred-site.xml need to carry that
> property to limit their slots).
>
> On 25-Nov-2011, at 8:45 AM, Keren Ouaknine wrote:
>
> Hello,
>
> I configured mapred-site.xml with one mapper, still the web ui shows:
> Map shows capacity 40 = 2 x #machines. I was expecting it to be 20.
> Any idea?
>
> Thanks,
> Keren
>
>
> Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map
> SlotsOccupied Reduce Slots Reserved Map SlotsReserved Reduce SlotsMap
> Task CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted NodesGraylisted
> NodesExcluded Nodes 3420720<http://10.239.24.12:50030/machines.jsp?type=active>
> 34200040203.000 <http://10.239.24.12:50030/machines.jsp?type=blacklisted>
> 0 <http://10.239.24.12:50030/machines.jsp?type=graylisted>0<http://10.239.24.12:50030/machines.jsp?type=excluded>
> --
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
>
>
>
>
--
Keren Ouaknine
Cell: +972 54 2565404
Web: www.kereno.com
Re: #number of mappers
Posted by Harsh J <ha...@cloudera.com>.
Keren,
What was your configured property at the TaskTrackers? The right property is mapred.tasktracker.{map,reduce}.tasks.maximum, and applies to each TaskTracker individually (so all your TT mapred-site.xml need to carry that property to limit their slots).
On 25-Nov-2011, at 8:45 AM, Keren Ouaknine wrote:
> Hello,
>
> I configured mapred-site.xml with one mapper, still the web ui shows:
> Map shows capacity 40 = 2 x #machines. I was expecting it to be 20.
> Any idea?
>
> Thanks,
> Keren
>
>
> Running Map Tasks Running Reduce Tasks Total Submissions Nodes Occupied Map Slots Occupied Reduce Slots Reserved Map Slots Reserved Reduce Slots Map Task Capacity Reduce Task Capacity Avg. Tasks/Node Blacklisted Nodes Graylisted Nodes Excluded Nodes
> 34 20 7 20 34 20 0 0 40 20 3.00 0 0 0
>
> --
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
>
>