You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Cogan, Peter (Peter)" <Pe...@alcatel-lucent.com> on 2012/11/01 14:43:13 UTC
Set the number of maps
Hi
I understand that the maximum number of concurrent map tasks is set by mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller number of maps (am testing disk IO). I thought that I could set that within the main program using
conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
to run with 4 maps – but that seems to have no impact. I know I could just change the mapred-site.xml and restart map reduce but that's kind of a pain. Can it be set from within the code?
Thanks
Peter
Re: Set the number of maps
Posted by "Cogan, Peter (Peter)" <Pe...@alcatel-lucent.com>.
Thanks for your answers!
From: Marcos Ortiz <ml...@uci.cu>>
Date: Thu, 1 Nov 2012 19:03:23 +0100
To: peter cogan <pe...@alcatel-lucent.com>>
Cc: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Set the number of maps
The option since 0.21 was renamed to mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
Hi
I understand that the maximum number of concurrent map tasks is set by
mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
number of maps (am testing disk IO). I thought that I could set that withinthe main program using
conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
to run with 4 maps – but that seems to have no impact. I know I could just
change the mapred-site.xml and restart map reduce but that's kind of a pain.
Can it be set from within the code?
Thanks
Peter
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz<http://about.me/marcosortiz>
@marcosluis2186<http://twitter.com/marcosluis2186>
[http://universidad.uci.cu/email.gif]<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
Re: Set the number of maps
Posted by "Cogan, Peter (Peter)" <Pe...@alcatel-lucent.com>.
Thanks for your answers!
From: Marcos Ortiz <ml...@uci.cu>>
Date: Thu, 1 Nov 2012 19:03:23 +0100
To: peter cogan <pe...@alcatel-lucent.com>>
Cc: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Set the number of maps
The option since 0.21 was renamed to mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
Hi
I understand that the maximum number of concurrent map tasks is set by
mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
number of maps (am testing disk IO). I thought that I could set that withinthe main program using
conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
to run with 4 maps – but that seems to have no impact. I know I could just
change the mapred-site.xml and restart map reduce but that's kind of a pain.
Can it be set from within the code?
Thanks
Peter
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz<http://about.me/marcosortiz>
@marcosluis2186<http://twitter.com/marcosluis2186>
[http://universidad.uci.cu/email.gif]<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
Re: Set the number of maps
Posted by "Cogan, Peter (Peter)" <Pe...@alcatel-lucent.com>.
Thanks for your answers!
From: Marcos Ortiz <ml...@uci.cu>>
Date: Thu, 1 Nov 2012 19:03:23 +0100
To: peter cogan <pe...@alcatel-lucent.com>>
Cc: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Set the number of maps
The option since 0.21 was renamed to mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
Hi
I understand that the maximum number of concurrent map tasks is set by
mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
number of maps (am testing disk IO). I thought that I could set that withinthe main program using
conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
to run with 4 maps – but that seems to have no impact. I know I could just
change the mapred-site.xml and restart map reduce but that's kind of a pain.
Can it be set from within the code?
Thanks
Peter
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz<http://about.me/marcosortiz>
@marcosluis2186<http://twitter.com/marcosluis2186>
[http://universidad.uci.cu/email.gif]<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
Re: Set the number of maps
Posted by "Cogan, Peter (Peter)" <Pe...@alcatel-lucent.com>.
Thanks for your answers!
From: Marcos Ortiz <ml...@uci.cu>>
Date: Thu, 1 Nov 2012 19:03:23 +0100
To: peter cogan <pe...@alcatel-lucent.com>>
Cc: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Set the number of maps
The option since 0.21 was renamed to mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
Hi
I understand that the maximum number of concurrent map tasks is set by
mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
number of maps (am testing disk IO). I thought that I could set that withinthe main program using
conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
to run with 4 maps – but that seems to have no impact. I know I could just
change the mapred-site.xml and restart map reduce but that's kind of a pain.
Can it be set from within the code?
Thanks
Peter
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz<http://about.me/marcosortiz>
@marcosluis2186<http://twitter.com/marcosluis2186>
[http://universidad.uci.cu/email.gif]<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
<http://www.uci.cu/>
Re: Set the number of maps
Posted by Marcos Ortiz <ml...@uci.cu>.
The option since 0.21 was renamed to
mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the
mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
> It can't be set from the code this way - the slot property is applied
> at the TaskTracker service level (as the name goes).
>
> Since you're just testing at the moment, try to set these values,
> restart TTs, and run your jobs again. You do not need to restart JT at
> any point for tweaking these values.
>
> On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
> <Pe...@alcatel-lucent.com> wrote:
>> Hi
>>
>> I understand that the maximum number of concurrent map tasks is set by
>> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
>> number of maps (am testing disk IO). I thought that I could set that within
>> the main program using
>>
>> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>>
>>
>> to run with 4 maps – but that seems to have no impact. I know I could just
>> change the mapred-site.xml and restart map reduce but that's kind of a pain.
>> Can it be set from within the code?
>>
>>
>> Thanks
>>
>> Peter
>
>
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Set the number of maps
Posted by Marcos Ortiz <ml...@uci.cu>.
The option since 0.21 was renamed to
mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the
mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
> It can't be set from the code this way - the slot property is applied
> at the TaskTracker service level (as the name goes).
>
> Since you're just testing at the moment, try to set these values,
> restart TTs, and run your jobs again. You do not need to restart JT at
> any point for tweaking these values.
>
> On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
> <Pe...@alcatel-lucent.com> wrote:
>> Hi
>>
>> I understand that the maximum number of concurrent map tasks is set by
>> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
>> number of maps (am testing disk IO). I thought that I could set that within
>> the main program using
>>
>> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>>
>>
>> to run with 4 maps – but that seems to have no impact. I know I could just
>> change the mapred-site.xml and restart map reduce but that's kind of a pain.
>> Can it be set from within the code?
>>
>>
>> Thanks
>>
>> Peter
>
>
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Set the number of maps
Posted by Marcos Ortiz <ml...@uci.cu>.
The option since 0.21 was renamed to
mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the
mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
> It can't be set from the code this way - the slot property is applied
> at the TaskTracker service level (as the name goes).
>
> Since you're just testing at the moment, try to set these values,
> restart TTs, and run your jobs again. You do not need to restart JT at
> any point for tweaking these values.
>
> On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
> <Pe...@alcatel-lucent.com> wrote:
>> Hi
>>
>> I understand that the maximum number of concurrent map tasks is set by
>> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
>> number of maps (am testing disk IO). I thought that I could set that within
>> the main program using
>>
>> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>>
>>
>> to run with 4 maps – but that seems to have no impact. I know I could just
>> change the mapred-site.xml and restart map reduce but that's kind of a pain.
>> Can it be set from within the code?
>>
>>
>> Thanks
>>
>> Peter
>
>
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Set the number of maps
Posted by Marcos Ortiz <ml...@uci.cu>.
The option since 0.21 was renamed to
mapreduce.tasktracker.map.tasks.maximum, and like
Harsh said to you, is is a TaskTracker service level option.
Another thing is that this option is very united to the
mapreduce.child.java.opts, so , make sure
to monitor constantly the effect of these changes in your cluster.
On 11/01/2012 11:55 AM, Harsh J wrote:
> It can't be set from the code this way - the slot property is applied
> at the TaskTracker service level (as the name goes).
>
> Since you're just testing at the moment, try to set these values,
> restart TTs, and run your jobs again. You do not need to restart JT at
> any point for tweaking these values.
>
> On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
> <Pe...@alcatel-lucent.com> wrote:
>> Hi
>>
>> I understand that the maximum number of concurrent map tasks is set by
>> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
>> number of maps (am testing disk IO). I thought that I could set that within
>> the main program using
>>
>> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>>
>>
>> to run with 4 maps – but that seems to have no impact. I know I could just
>> change the mapred-site.xml and restart map reduce but that's kind of a pain.
>> Can it be set from within the code?
>>
>>
>> Thanks
>>
>> Peter
>
>
--
Marcos Luis Ortíz Valmaseda
about.me/marcosortiz <http://about.me/marcosortiz>
@marcosluis2186 <http://twitter.com/marcosluis2186>
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Set the number of maps
Posted by Harsh J <ha...@cloudera.com>.
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set by
> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
> number of maps (am testing disk IO). I thought that I could set that within
> the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a pain.
> Can it be set from within the code?
>
>
> Thanks
>
> Peter
--
Harsh J
Re: Set the number of maps
Posted by Bejoy KS <be...@gmail.com>.
Hi Peter
'mapred.tasktracker.map.tasks.maximum' is not for setting an upper cap on the map tasks spawned by a job. This property is used to set the map slots on each TaskTracker. It is TaskTracker level property and cannot be overriden on a per job basis.
To control the number of map tasks for your job you need to increase the value for min and max split sizes.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Ted Dunning <td...@maprtech.com>
Date: Thu, 1 Nov 2012 09:50:10
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Re: Set the number of maps
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>
Re: Set the number of maps
Posted by Bejoy KS <be...@gmail.com>.
Hi Peter
'mapred.tasktracker.map.tasks.maximum' is not for setting an upper cap on the map tasks spawned by a job. This property is used to set the map slots on each TaskTracker. It is TaskTracker level property and cannot be overriden on a per job basis.
To control the number of map tasks for your job you need to increase the value for min and max split sizes.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Ted Dunning <td...@maprtech.com>
Date: Thu, 1 Nov 2012 09:50:10
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Re: Set the number of maps
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>
Re: Set the number of maps
Posted by Bejoy KS <be...@gmail.com>.
Hi Peter
'mapred.tasktracker.map.tasks.maximum' is not for setting an upper cap on the map tasks spawned by a job. This property is used to set the map slots on each TaskTracker. It is TaskTracker level property and cannot be overriden on a per job basis.
To control the number of map tasks for your job you need to increase the value for min and max split sizes.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Ted Dunning <td...@maprtech.com>
Date: Thu, 1 Nov 2012 09:50:10
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Re: Set the number of maps
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>
Re: Set the number of maps
Posted by Bejoy KS <be...@gmail.com>.
Hi Peter
'mapred.tasktracker.map.tasks.maximum' is not for setting an upper cap on the map tasks spawned by a job. This property is used to set the map slots on each TaskTracker. It is TaskTracker level property and cannot be overriden on a per job basis.
To control the number of map tasks for your job you need to increase the value for min and max split sizes.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Ted Dunning <td...@maprtech.com>
Date: Thu, 1 Nov 2012 09:50:10
To: <us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Re: Set the number of maps
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>
Re: Set the number of maps
Posted by Ted Dunning <td...@maprtech.com>.
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>
Re: Set the number of maps
Posted by Harsh J <ha...@cloudera.com>.
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set by
> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
> number of maps (am testing disk IO). I thought that I could set that within
> the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a pain.
> Can it be set from within the code?
>
>
> Thanks
>
> Peter
--
Harsh J
Re: Set the number of maps
Posted by Ted Dunning <td...@maprtech.com>.
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>
Re: Set the number of maps
Posted by Harsh J <ha...@cloudera.com>.
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set by
> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
> number of maps (am testing disk IO). I thought that I could set that within
> the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a pain.
> Can it be set from within the code?
>
>
> Thanks
>
> Peter
--
Harsh J
Re: Set the number of maps
Posted by Harsh J <ha...@cloudera.com>.
It can't be set from the code this way - the slot property is applied
at the TaskTracker service level (as the name goes).
Since you're just testing at the moment, try to set these values,
restart TTs, and run your jobs again. You do not need to restart JT at
any point for tweaking these values.
On Thu, Nov 1, 2012 at 7:13 PM, Cogan, Peter (Peter)
<Pe...@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set by
> mapred.tasktracker.map.tasks.maximum - however I wish to run with a smaller
> number of maps (am testing disk IO). I thought that I could set that within
> the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a pain.
> Can it be set from within the code?
>
>
> Thanks
>
> Peter
--
Harsh J
Re: Set the number of maps
Posted by Ted Dunning <td...@maprtech.com>.
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>
Re: Set the number of maps
Posted by Ted Dunning <td...@maprtech.com>.
Is the spelling of the option correct?
On Thu, Nov 1, 2012 at 6:43 AM, Cogan, Peter (Peter) <
Peter.Cogan@alcatel-lucent.com> wrote:
> Hi
>
> I understand that the maximum number of concurrent map tasks is set
> by mapred.tasktracker.map.tasks.maximum - however I wish to run with a
> smaller number of maps (am testing disk IO). I thought that I could set
> that within the main program using
>
> conf.set("mapred.tasktracker.map.tasks.maximuma", "4");
>
>
> to run with 4 maps – but that seems to have no impact. I know I could just
> change the mapred-site.xml and restart map reduce but that's kind of a
> pain. Can it be set from within the code?
>
>
> Thanks
>
> Peter
>