You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Mark Kerzner <ma...@shmsoft.com> on 2012/11/13 16:16:13 UTC
mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value
for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this
situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
I am using a Hadoop manager, and it stores the values in a database,
outside the configuration files.
On Tue, Nov 13, 2012 at 10:27 AM, Jay Vyas <ja...@gmail.com> wrote:
> Hmmm What do you mean wrong configuration file.? How could that ever
> happen?
>
> Jay Vyas
>
> On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com>
> wrote:
>
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
>> ><ma...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value
>> for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
>> it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this
>> situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>>
>>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
I am using a Hadoop manager, and it stores the values in a database,
outside the configuration files.
On Tue, Nov 13, 2012 at 10:27 AM, Jay Vyas <ja...@gmail.com> wrote:
> Hmmm What do you mean wrong configuration file.? How could that ever
> happen?
>
> Jay Vyas
>
> On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com>
> wrote:
>
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
>> ><ma...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value
>> for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
>> it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this
>> situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>>
>>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
I am using a Hadoop manager, and it stores the values in a database,
outside the configuration files.
On Tue, Nov 13, 2012 at 10:27 AM, Jay Vyas <ja...@gmail.com> wrote:
> Hmmm What do you mean wrong configuration file.? How could that ever
> happen?
>
> Jay Vyas
>
> On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com>
> wrote:
>
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
>> ><ma...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value
>> for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
>> it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this
>> situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>>
>>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
I am using a Hadoop manager, and it stores the values in a database,
outside the configuration files.
On Tue, Nov 13, 2012 at 10:27 AM, Jay Vyas <ja...@gmail.com> wrote:
> Hmmm What do you mean wrong configuration file.? How could that ever
> happen?
>
> Jay Vyas
>
> On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com>
> wrote:
>
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
>> mark.kerzner@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
>> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
>> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
>> ><ma...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value
>> for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
>> it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this
>> situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>>
>>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Jay Vyas <ja...@gmail.com>.
Hmmm What do you mean wrong configuration file.? How could that ever happen?
Jay Vyas
On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com> wrote:
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com> wrote:
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <us...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Jay Vyas <ja...@gmail.com>.
Hmmm What do you mean wrong configuration file.? How could that ever happen?
Jay Vyas
On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com> wrote:
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com> wrote:
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <us...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Jay Vyas <ja...@gmail.com>.
Hmmm What do you mean wrong configuration file.? How could that ever happen?
Jay Vyas
On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com> wrote:
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com> wrote:
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <us...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Jay Vyas <ja...@gmail.com>.
Hmmm What do you mean wrong configuration file.? How could that ever happen?
Jay Vyas
On Nov 13, 2012, at 10:25 AM, Mark Kerzner <ma...@shmsoft.com> wrote:
> Exactly! I found the right one, and it is 80.
>
> Thank you,
> Mark
>
> On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com> wrote:
>> Look on the job tracker web UI is parameter has taken effect (port 50030)
>>
>> It might be in the wrong config file or something
>>
>>
>> Serge
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Date: Tuesday, November 13, 2012 5:19 PM
>> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
>> Subject: Re: mapred.tasktracker.map.tasks.maximum
>>
>> 1.0.1
>>
>> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
>> What hadoop version are we talking about?
>>
>> From: Mark Kerzner <ma...@shmsoft.com>>>
>> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
>> Date: Tuesday, November 13, 2012 5:16 PM
>> To: Hadoop User <us...@hadoop.apache.org>>>
>> Subject: mapred.tasktracker.map.tasks.maximum
>>
>> Hi,
>>
>> I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
>>
>> <property>
>> <name>mapred.tasktracker.map.tasks.maximum</name>
>> <!-- see other kb entry about this one. -->
>> <value>1</value>
>> <final>true</final>
>> </property>
>> (which I think is wrong).
>>
>> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
>>
>> Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
>>
>> Thank you. Sincerely,
>> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
;)
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:25 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>>" <us...@hadoop.apache.org>>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
;)
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:25 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>>" <us...@hadoop.apache.org>>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
;)
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:25 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>>" <us...@hadoop.apache.org>>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
;)
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:25 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>>" <us...@hadoop.apache.org>>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> Look on the job tracker web UI is parameter has taken effect (port 50030)
>
> It might be in the wrong config file or something
>
>
> Serge
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:19 PM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: mapred.tasktracker.map.tasks.maximum
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
> ><ma...@hadoop.apache.org>>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> Look on the job tracker web UI is parameter has taken effect (port 50030)
>
> It might be in the wrong config file or something
>
>
> Serge
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:19 PM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: mapred.tasktracker.map.tasks.maximum
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
> ><ma...@hadoop.apache.org>>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> Look on the job tracker web UI is parameter has taken effect (port 50030)
>
> It might be in the wrong config file or something
>
>
> Serge
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:19 PM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: mapred.tasktracker.map.tasks.maximum
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
> ><ma...@hadoop.apache.org>>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
Exactly! I found the right one, and it is 80.
Thank you,
Mark
On Tue, Nov 13, 2012 at 10:23 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> Look on the job tracker web UI is parameter has taken effect (port 50030)
>
> It might be in the wrong config file or something
>
>
> Serge
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:19 PM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: mapred.tasktracker.map.tasks.maximum
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com<ma...@nice.com>> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org><mailto:
> user@hadoop.apache.org<ma...@hadoop.apache.org>>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org
> ><ma...@hadoop.apache.org>>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
An important distinction, max slots per node is one thing, and total
mappers is something else - it is the total that ran. But now I know where
to look.
thank you,
mark
On Tue, Nov 13, 2012 at 10:26 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Mark,
>
>
>
> The way I understand it…
>
>
>
> # of mappers is calculated by deviding your input by file split size (64Mb
> default).
>
>
>
> So, say your input is 64Gb in size so you will end up with 1000 mappers.
>
>
>
> Slots are a different story. It is number of map tasks processed by a
> single node in parrallel. It is normally calculated by the number of cores
> that node is running on (one per core).
>
>
>
> Rgds,
>
>
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Tuesday, November 13, 2012 10:20 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: mapred.tasktracker.map.tasks.maximum
>
>
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
An important distinction, max slots per node is one thing, and total
mappers is something else - it is the total that ran. But now I know where
to look.
thank you,
mark
On Tue, Nov 13, 2012 at 10:26 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Mark,
>
>
>
> The way I understand it…
>
>
>
> # of mappers is calculated by deviding your input by file split size (64Mb
> default).
>
>
>
> So, say your input is 64Gb in size so you will end up with 1000 mappers.
>
>
>
> Slots are a different story. It is number of map tasks processed by a
> single node in parrallel. It is normally calculated by the number of cores
> that node is running on (one per core).
>
>
>
> Rgds,
>
>
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Tuesday, November 13, 2012 10:20 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: mapred.tasktracker.map.tasks.maximum
>
>
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
An important distinction, max slots per node is one thing, and total
mappers is something else - it is the total that ran. But now I know where
to look.
thank you,
mark
On Tue, Nov 13, 2012 at 10:26 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Mark,
>
>
>
> The way I understand it…
>
>
>
> # of mappers is calculated by deviding your input by file split size (64Mb
> default).
>
>
>
> So, say your input is 64Gb in size so you will end up with 1000 mappers.
>
>
>
> Slots are a different story. It is number of map tasks processed by a
> single node in parrallel. It is normally calculated by the number of cores
> that node is running on (one per core).
>
>
>
> Rgds,
>
>
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Tuesday, November 13, 2012 10:20 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: mapred.tasktracker.map.tasks.maximum
>
>
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
An important distinction, max slots per node is one thing, and total
mappers is something else - it is the total that ran. But now I know where
to look.
thank you,
mark
On Tue, Nov 13, 2012 at 10:26 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Mark,
>
>
>
> The way I understand it…
>
>
>
> # of mappers is calculated by deviding your input by file split size (64Mb
> default).
>
>
>
> So, say your input is 64Gb in size so you will end up with 1000 mappers.
>
>
>
> Slots are a different story. It is number of map tasks processed by a
> single node in parrallel. It is normally calculated by the number of cores
> that node is running on (one per core).
>
>
>
> Rgds,
>
>
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Tuesday, November 13, 2012 10:20 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: mapred.tasktracker.map.tasks.maximum
>
>
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
RE: mapred.tasktracker.map.tasks.maximum
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Mark,
The way I understand it...
# of mappers is calculated by deviding your input by file split size (64Mb default).
So, say your input is 64Gb in size so you will end up with 1000 mappers.
Slots are a different story. It is number of map tasks processed by a single node in parrallel. It is normally calculated by the number of cores that node is running on (one per core).
Rgds,
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Tuesday, November 13, 2012 10:20 AM
To: user@hadoop.apache.org
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
RE: mapred.tasktracker.map.tasks.maximum
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Mark,
The way I understand it...
# of mappers is calculated by deviding your input by file split size (64Mb default).
So, say your input is 64Gb in size so you will end up with 1000 mappers.
Slots are a different story. It is number of map tasks processed by a single node in parrallel. It is normally calculated by the number of cores that node is running on (one per core).
Rgds,
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Tuesday, November 13, 2012 10:20 AM
To: user@hadoop.apache.org
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
Look on the job tracker web UI is parameter has taken effect (port 50030)
It might be in the wrong config file or something
Serge
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
RE: mapred.tasktracker.map.tasks.maximum
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Mark,
The way I understand it...
# of mappers is calculated by deviding your input by file split size (64Mb default).
So, say your input is 64Gb in size so you will end up with 1000 mappers.
Slots are a different story. It is number of map tasks processed by a single node in parrallel. It is normally calculated by the number of cores that node is running on (one per core).
Rgds,
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Tuesday, November 13, 2012 10:20 AM
To: user@hadoop.apache.org
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
RE: mapred.tasktracker.map.tasks.maximum
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Mark,
The way I understand it...
# of mappers is calculated by deviding your input by file split size (64Mb default).
So, say your input is 64Gb in size so you will end up with 1000 mappers.
Slots are a different story. It is number of map tasks processed by a single node in parrallel. It is normally calculated by the number of cores that node is running on (one per core).
Rgds,
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Tuesday, November 13, 2012 10:20 AM
To: user@hadoop.apache.org
Subject: Re: mapred.tasktracker.map.tasks.maximum
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>>" <us...@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Mark Kerzner <ma...@shmsoft.com>.
1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <us...@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <!-- see other kb entry about this one. -->
> <value>1</value>
> <final>true</final>
> </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark
Re: mapred.tasktracker.map.tasks.maximum
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
What hadoop version are we talking about?
From: Mark Kerzner <ma...@shmsoft.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <us...@hadoop.apache.org>>
Subject: mapred.tasktracker.map.tasks.maximum
Hi,
I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum number of mappers per slot is 1:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<!-- see other kb entry about this one. -->
<value>1</value>
<final>true</final>
</property>
(which I think is wrong).
Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more than 1 mapper per node.
Questions: what maximum number of mappers would be appropriate in this situation? Is that the right way to set them?
Thank you. Sincerely,
Mark