You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Manoj Babu <ma...@gmail.com> on 2013/01/28 06:06:18 UTC
reg max map task config
Hi All,
I am trying to override the value of mapred.tasktracker.map.tasks.maximum
in mapred-site.xml through my job configuration object will it reflect in
the corresponding task trackers?
or do i have to change it in mapred-site.xml following a restart for the
changes to apply?
Thanks in advance.
Cheers!
Manoj.
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
There's probably different variables (esp. since you run HBase,
calculated resources) involved for this, but you can generally find
some reading material on this in books and
http://wiki.apache.org/hadoop/HowManyMapsAndReduces which should help
you come to a good value. Pro-tip I usually get from Todd:
Consistently monitor and tune, for it is harmless to restart a slave
to set in new values.
On Mon, Jan 28, 2013 at 6:04 PM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi Harsh,
>
> Is there a recommanded way to configure this value? I think I read
> something like 0.7 x cores but I'm not 100% sure...
>
> JM
>
> 2013/1/28, Manoj Babu <ma...@gmail.com>:
>> Thank you Harsh.
>>
>> Cheers!
>> Manoj.
>>
>>
>> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> As noted on
>>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>>> ,
>>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>>> applies to a TaskTracker daemon and therefore needs to be applied to
>>> its local mapred-site.xml and has to be restarted to take new values
>>> into effect.
>>>
>>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>>> > Hi All,
>>> >
>>> > I am trying to override the value of
>>> mapred.tasktracker.map.tasks.maximum in
>>> > mapred-site.xml through my job configuration object will it reflect in
>>> the
>>> > corresponding task trackers?
>>> > or do i have to change it in mapred-site.xml following a restart for
>>> > the
>>> > changes to apply?
>>> >
>>> > Thanks in advance.
>>> >
>>> > Cheers!
>>> > Manoj.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
--
Harsh J
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
There's probably different variables (esp. since you run HBase,
calculated resources) involved for this, but you can generally find
some reading material on this in books and
http://wiki.apache.org/hadoop/HowManyMapsAndReduces which should help
you come to a good value. Pro-tip I usually get from Todd:
Consistently monitor and tune, for it is harmless to restart a slave
to set in new values.
On Mon, Jan 28, 2013 at 6:04 PM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi Harsh,
>
> Is there a recommanded way to configure this value? I think I read
> something like 0.7 x cores but I'm not 100% sure...
>
> JM
>
> 2013/1/28, Manoj Babu <ma...@gmail.com>:
>> Thank you Harsh.
>>
>> Cheers!
>> Manoj.
>>
>>
>> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> As noted on
>>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>>> ,
>>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>>> applies to a TaskTracker daemon and therefore needs to be applied to
>>> its local mapred-site.xml and has to be restarted to take new values
>>> into effect.
>>>
>>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>>> > Hi All,
>>> >
>>> > I am trying to override the value of
>>> mapred.tasktracker.map.tasks.maximum in
>>> > mapred-site.xml through my job configuration object will it reflect in
>>> the
>>> > corresponding task trackers?
>>> > or do i have to change it in mapred-site.xml following a restart for
>>> > the
>>> > changes to apply?
>>> >
>>> > Thanks in advance.
>>> >
>>> > Cheers!
>>> > Manoj.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
--
Harsh J
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
There's probably different variables (esp. since you run HBase,
calculated resources) involved for this, but you can generally find
some reading material on this in books and
http://wiki.apache.org/hadoop/HowManyMapsAndReduces which should help
you come to a good value. Pro-tip I usually get from Todd:
Consistently monitor and tune, for it is harmless to restart a slave
to set in new values.
On Mon, Jan 28, 2013 at 6:04 PM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi Harsh,
>
> Is there a recommanded way to configure this value? I think I read
> something like 0.7 x cores but I'm not 100% sure...
>
> JM
>
> 2013/1/28, Manoj Babu <ma...@gmail.com>:
>> Thank you Harsh.
>>
>> Cheers!
>> Manoj.
>>
>>
>> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> As noted on
>>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>>> ,
>>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>>> applies to a TaskTracker daemon and therefore needs to be applied to
>>> its local mapred-site.xml and has to be restarted to take new values
>>> into effect.
>>>
>>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>>> > Hi All,
>>> >
>>> > I am trying to override the value of
>>> mapred.tasktracker.map.tasks.maximum in
>>> > mapred-site.xml through my job configuration object will it reflect in
>>> the
>>> > corresponding task trackers?
>>> > or do i have to change it in mapred-site.xml following a restart for
>>> > the
>>> > changes to apply?
>>> >
>>> > Thanks in advance.
>>> >
>>> > Cheers!
>>> > Manoj.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
--
Harsh J
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
There's probably different variables (esp. since you run HBase,
calculated resources) involved for this, but you can generally find
some reading material on this in books and
http://wiki.apache.org/hadoop/HowManyMapsAndReduces which should help
you come to a good value. Pro-tip I usually get from Todd:
Consistently monitor and tune, for it is harmless to restart a slave
to set in new values.
On Mon, Jan 28, 2013 at 6:04 PM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi Harsh,
>
> Is there a recommanded way to configure this value? I think I read
> something like 0.7 x cores but I'm not 100% sure...
>
> JM
>
> 2013/1/28, Manoj Babu <ma...@gmail.com>:
>> Thank you Harsh.
>>
>> Cheers!
>> Manoj.
>>
>>
>> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Hi,
>>>
>>> As noted on
>>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>>> ,
>>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>>> applies to a TaskTracker daemon and therefore needs to be applied to
>>> its local mapred-site.xml and has to be restarted to take new values
>>> into effect.
>>>
>>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>>> > Hi All,
>>> >
>>> > I am trying to override the value of
>>> mapred.tasktracker.map.tasks.maximum in
>>> > mapred-site.xml through my job configuration object will it reflect in
>>> the
>>> > corresponding task trackers?
>>> > or do i have to change it in mapred-site.xml following a restart for
>>> > the
>>> > changes to apply?
>>> >
>>> > Thanks in advance.
>>> >
>>> > Cheers!
>>> > Manoj.
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
--
Harsh J
Re: reg max map task config
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Harsh,
Is there a recommanded way to configure this value? I think I read
something like 0.7 x cores but I'm not 100% sure...
JM
2013/1/28, Manoj Babu <ma...@gmail.com>:
> Thank you Harsh.
>
> Cheers!
> Manoj.
>
>
> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> As noted on
>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>> ,
>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>> applies to a TaskTracker daemon and therefore needs to be applied to
>> its local mapred-site.xml and has to be restarted to take new values
>> into effect.
>>
>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I am trying to override the value of
>> mapred.tasktracker.map.tasks.maximum in
>> > mapred-site.xml through my job configuration object will it reflect in
>> the
>> > corresponding task trackers?
>> > or do i have to change it in mapred-site.xml following a restart for
>> > the
>> > changes to apply?
>> >
>> > Thanks in advance.
>> >
>> > Cheers!
>> > Manoj.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: reg max map task config
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Harsh,
Is there a recommanded way to configure this value? I think I read
something like 0.7 x cores but I'm not 100% sure...
JM
2013/1/28, Manoj Babu <ma...@gmail.com>:
> Thank you Harsh.
>
> Cheers!
> Manoj.
>
>
> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> As noted on
>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>> ,
>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>> applies to a TaskTracker daemon and therefore needs to be applied to
>> its local mapred-site.xml and has to be restarted to take new values
>> into effect.
>>
>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I am trying to override the value of
>> mapred.tasktracker.map.tasks.maximum in
>> > mapred-site.xml through my job configuration object will it reflect in
>> the
>> > corresponding task trackers?
>> > or do i have to change it in mapred-site.xml following a restart for
>> > the
>> > changes to apply?
>> >
>> > Thanks in advance.
>> >
>> > Cheers!
>> > Manoj.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: reg max map task config
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Harsh,
Is there a recommanded way to configure this value? I think I read
something like 0.7 x cores but I'm not 100% sure...
JM
2013/1/28, Manoj Babu <ma...@gmail.com>:
> Thank you Harsh.
>
> Cheers!
> Manoj.
>
>
> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> As noted on
>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>> ,
>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>> applies to a TaskTracker daemon and therefore needs to be applied to
>> its local mapred-site.xml and has to be restarted to take new values
>> into effect.
>>
>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I am trying to override the value of
>> mapred.tasktracker.map.tasks.maximum in
>> > mapred-site.xml through my job configuration object will it reflect in
>> the
>> > corresponding task trackers?
>> > or do i have to change it in mapred-site.xml following a restart for
>> > the
>> > changes to apply?
>> >
>> > Thanks in advance.
>> >
>> > Cheers!
>> > Manoj.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: reg max map task config
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Harsh,
Is there a recommanded way to configure this value? I think I read
something like 0.7 x cores but I'm not 100% sure...
JM
2013/1/28, Manoj Babu <ma...@gmail.com>:
> Thank you Harsh.
>
> Cheers!
> Manoj.
>
>
> On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> As noted on
>> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
>> ,
>> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
>> applies to a TaskTracker daemon and therefore needs to be applied to
>> its local mapred-site.xml and has to be restarted to take new values
>> into effect.
>>
>> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I am trying to override the value of
>> mapred.tasktracker.map.tasks.maximum in
>> > mapred-site.xml through my job configuration object will it reflect in
>> the
>> > corresponding task trackers?
>> > or do i have to change it in mapred-site.xml following a restart for
>> > the
>> > changes to apply?
>> >
>> > Thanks in advance.
>> >
>> > Cheers!
>> > Manoj.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: reg max map task config
Posted by Manoj Babu <ma...@gmail.com>.
Thank you Harsh.
Cheers!
Manoj.
On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> As noted on
> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
> ,
> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
> applies to a TaskTracker daemon and therefore needs to be applied to
> its local mapred-site.xml and has to be restarted to take new values
> into effect.
>
> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> > Hi All,
> >
> > I am trying to override the value of
> mapred.tasktracker.map.tasks.maximum in
> > mapred-site.xml through my job configuration object will it reflect in
> the
> > corresponding task trackers?
> > or do i have to change it in mapred-site.xml following a restart for the
> > changes to apply?
> >
> > Thanks in advance.
> >
> > Cheers!
> > Manoj.
>
>
>
> --
> Harsh J
>
Re: reg max map task config
Posted by Manoj Babu <ma...@gmail.com>.
Thank you Harsh.
Cheers!
Manoj.
On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> As noted on
> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
> ,
> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
> applies to a TaskTracker daemon and therefore needs to be applied to
> its local mapred-site.xml and has to be restarted to take new values
> into effect.
>
> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> > Hi All,
> >
> > I am trying to override the value of
> mapred.tasktracker.map.tasks.maximum in
> > mapred-site.xml through my job configuration object will it reflect in
> the
> > corresponding task trackers?
> > or do i have to change it in mapred-site.xml following a restart for the
> > changes to apply?
> >
> > Thanks in advance.
> >
> > Cheers!
> > Manoj.
>
>
>
> --
> Harsh J
>
Re: reg max map task config
Posted by Manoj Babu <ma...@gmail.com>.
Thank you Harsh.
Cheers!
Manoj.
On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> As noted on
> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
> ,
> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
> applies to a TaskTracker daemon and therefore needs to be applied to
> its local mapred-site.xml and has to be restarted to take new values
> into effect.
>
> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> > Hi All,
> >
> > I am trying to override the value of
> mapred.tasktracker.map.tasks.maximum in
> > mapred-site.xml through my job configuration object will it reflect in
> the
> > corresponding task trackers?
> > or do i have to change it in mapred-site.xml following a restart for the
> > changes to apply?
> >
> > Thanks in advance.
> >
> > Cheers!
> > Manoj.
>
>
>
> --
> Harsh J
>
Re: reg max map task config
Posted by Manoj Babu <ma...@gmail.com>.
Thank you Harsh.
Cheers!
Manoj.
On Mon, Jan 28, 2013 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> As noted on
> http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons
> ,
> and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
> applies to a TaskTracker daemon and therefore needs to be applied to
> its local mapred-site.xml and has to be restarted to take new values
> into effect.
>
> On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> > Hi All,
> >
> > I am trying to override the value of
> mapred.tasktracker.map.tasks.maximum in
> > mapred-site.xml through my job configuration object will it reflect in
> the
> > corresponding task trackers?
> > or do i have to change it in mapred-site.xml following a restart for the
> > changes to apply?
> >
> > Thanks in advance.
> >
> > Cheers!
> > Manoj.
>
>
>
> --
> Harsh J
>
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
Hi,
As noted on http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons,
and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
applies to a TaskTracker daemon and therefore needs to be applied to
its local mapred-site.xml and has to be restarted to take new values
into effect.
On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> Hi All,
>
> I am trying to override the value of mapred.tasktracker.map.tasks.maximum in
> mapred-site.xml through my job configuration object will it reflect in the
> corresponding task trackers?
> or do i have to change it in mapred-site.xml following a restart for the
> changes to apply?
>
> Thanks in advance.
>
> Cheers!
> Manoj.
--
Harsh J
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
Hi,
As noted on http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons,
and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
applies to a TaskTracker daemon and therefore needs to be applied to
its local mapred-site.xml and has to be restarted to take new values
into effect.
On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> Hi All,
>
> I am trying to override the value of mapred.tasktracker.map.tasks.maximum in
> mapred-site.xml through my job configuration object will it reflect in the
> corresponding task trackers?
> or do i have to change it in mapred-site.xml following a restart for the
> changes to apply?
>
> Thanks in advance.
>
> Cheers!
> Manoj.
--
Harsh J
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
Hi,
As noted on http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons,
and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
applies to a TaskTracker daemon and therefore needs to be applied to
its local mapred-site.xml and has to be restarted to take new values
into effect.
On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> Hi All,
>
> I am trying to override the value of mapred.tasktracker.map.tasks.maximum in
> mapred-site.xml through my job configuration object will it reflect in the
> corresponding task trackers?
> or do i have to change it in mapred-site.xml following a restart for the
> changes to apply?
>
> Thanks in advance.
>
> Cheers!
> Manoj.
--
Harsh J
Re: reg max map task config
Posted by Harsh J <ha...@cloudera.com>.
Hi,
As noted on http://hadoop.apache.org/docs/stable/cluster_setup.html#Configuring+the+Hadoop+Daemons,
and by its own name, the config 'mapred.tasktracker.map.tasks.maximum'
applies to a TaskTracker daemon and therefore needs to be applied to
its local mapred-site.xml and has to be restarted to take new values
into effect.
On Mon, Jan 28, 2013 at 10:36 AM, Manoj Babu <ma...@gmail.com> wrote:
> Hi All,
>
> I am trying to override the value of mapred.tasktracker.map.tasks.maximum in
> mapred-site.xml through my job configuration object will it reflect in the
> corresponding task trackers?
> or do i have to change it in mapred-site.xml following a restart for the
> changes to apply?
>
> Thanks in advance.
>
> Cheers!
> Manoj.
--
Harsh J