You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by pr...@nokia.com on 2011/07/01 22:03:05 UTC
mapred.tasktracker.map.tasks.maximum is not taking into effect
Hi all,
I am using hadoop 0.20.2. I am setting the property mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my job conf but I am still seeing max of only 2 map and reduce tasks on each node. I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this a bug in 0.20.2 or am I doing something wrong?
Configuration conf = new Configuration();
conf.set("mapred.tasktracker.map.tasks.maximum", "4");
conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
Thanks
Praveen
Re: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by Joey Echeverria <jo...@cloudera.com>.
This property applies to a tasktraker rather that an individual job.
Therefore it needs to be set in the mapred-site.xml and the daemon
restarted.
-Joey
On Jul 1, 2011 7:01 PM, <pr...@nokia.com> wrote:
> Are you sure? AFAIK all mapred.xxx properties can be set via job config. I
also read on yahoo tutorial that this property can be either set in
hadoop-site.XML or job config. May be someone can confirm this who have
really used this property.
>
> Praveen
>
> On Jul 1, 2011, at 4:46 PM, "ext Anthony Urso" <an...@cs.ucla.edu>
wrote:
>
>> On Fri, Jul 1, 2011 at 1:03 PM, <pr...@nokia.com> wrote:
>>> Hi all,
>>>
>>> I am using hadoop 0.20.2. I am setting the property
>>> mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my
job
>>> conf but I am still seeing max of only 2 map and reduce tasks on each
node.
>>> I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this
a
>>> bug in 0.20.2 or am I doing something wrong?
>>>
>>>
>>
>> If I remember correctly, you have to set this in your hadoop-site.xml
>> and restart your job tracker and task trackers.
>>
>>>
>>> Configuration conf = new Configuration();
>>>
>>> conf.set("mapred.tasktracker.map.tasks.maximum", "4");
>>>
>>> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
>>>
>>>
>>>
>>> Thanks
>>>
>>> Praveen
Re: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by Mostafa Gaber <mo...@gmail.com>.
If your datanode has 2 HDFS-chunks (blocks) of the input file, the scheduler
will first prefer to run 2 map tasks on the tasktracker where this datanode
resides.
On Fri, Jul 1, 2011 at 10:33 PM, Juwei Shi <sh...@gmail.com> wrote:
> I think that Anthony is right. Task capacity has to been set at
> mapred-default.html, and restart the cluster.
>
> Anthony Urso
>
>
>
> 2011/7/2 <pr...@nokia.com>
>
> Are you sure? AFAIK all mapred.xxx properties can be set via job config. I
>> also read on yahoo tutorial that this property can be either set in
>> hadoop-site.XML or job config. May be someone can confirm this who have
>> really used this property.
>>
>> Praveen
>>
>> On Jul 1, 2011, at 4:46 PM, "ext Anthony Urso" <an...@cs.ucla.edu>
>> wrote:
>>
>> > On Fri, Jul 1, 2011 at 1:03 PM, <pr...@nokia.com> wrote:
>> >> Hi all,
>> >>
>> >> I am using hadoop 0.20.2. I am setting the property
>> >> mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my
>> job
>> >> conf but I am still seeing max of only 2 map and reduce tasks on each
>> node.
>> >> I know my machine can run 4 maps and 4 reduce tasks in parallel. Is
>> this a
>> >> bug in 0.20.2 or am I doing something wrong?
>> >>
>> >>
>> >
>> > If I remember correctly, you have to set this in your hadoop-site.xml
>> > and restart your job tracker and task trackers.
>> >
>> >>
>> >> Configuration conf = new Configuration();
>> >>
>> >> conf.set("mapred.tasktracker.map.tasks.maximum", "4");
>> >>
>> >> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
>> >>
>> >>
>> >>
>> >> Thanks
>> >>
>> >> Praveen
>>
>
>
>
> --
> - Juwei
>
--
Best Regards,
Mostafa Ead
Re: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by Juwei Shi <sh...@gmail.com>.
I think that Anthony is right. Task capacity has to been set at
mapred-default.html, and restart the cluster.
Anthony Urso
2011/7/2 <pr...@nokia.com>
> Are you sure? AFAIK all mapred.xxx properties can be set via job config. I
> also read on yahoo tutorial that this property can be either set in
> hadoop-site.XML or job config. May be someone can confirm this who have
> really used this property.
>
> Praveen
>
> On Jul 1, 2011, at 4:46 PM, "ext Anthony Urso" <an...@cs.ucla.edu>
> wrote:
>
> > On Fri, Jul 1, 2011 at 1:03 PM, <pr...@nokia.com> wrote:
> >> Hi all,
> >>
> >> I am using hadoop 0.20.2. I am setting the property
> >> mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my
> job
> >> conf but I am still seeing max of only 2 map and reduce tasks on each
> node.
> >> I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this
> a
> >> bug in 0.20.2 or am I doing something wrong?
> >>
> >>
> >
> > If I remember correctly, you have to set this in your hadoop-site.xml
> > and restart your job tracker and task trackers.
> >
> >>
> >> Configuration conf = new Configuration();
> >>
> >> conf.set("mapred.tasktracker.map.tasks.maximum", "4");
> >>
> >> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
> >>
> >>
> >>
> >> Thanks
> >>
> >> Praveen
>
--
- Juwei
Re: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by pr...@nokia.com.
Are you sure? AFAIK all mapred.xxx properties can be set via job config. I also read on yahoo tutorial that this property can be either set in hadoop-site.XML or job config. May be someone can confirm this who have really used this property.
Praveen
On Jul 1, 2011, at 4:46 PM, "ext Anthony Urso" <an...@cs.ucla.edu> wrote:
> On Fri, Jul 1, 2011 at 1:03 PM, <pr...@nokia.com> wrote:
>> Hi all,
>>
>> I am using hadoop 0.20.2. I am setting the property
>> mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my job
>> conf but I am still seeing max of only 2 map and reduce tasks on each node.
>> I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this a
>> bug in 0.20.2 or am I doing something wrong?
>>
>>
>
> If I remember correctly, you have to set this in your hadoop-site.xml
> and restart your job tracker and task trackers.
>
>>
>> Configuration conf = new Configuration();
>>
>> conf.set("mapred.tasktracker.map.tasks.maximum", "4");
>>
>> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
>>
>>
>>
>> Thanks
>>
>> Praveen
Re: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by Anthony Urso <an...@cs.ucla.edu>.
On Fri, Jul 1, 2011 at 1:03 PM, <pr...@nokia.com> wrote:
> Hi all,
>
> I am using hadoop 0.20.2. I am setting the property
> mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my job
> conf but I am still seeing max of only 2 map and reduce tasks on each node.
> I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this a
> bug in 0.20.2 or am I doing something wrong?
>
>
If I remember correctly, you have to set this in your hadoop-site.xml
and restart your job tracker and task trackers.
>
> Configuration conf = new Configuration();
>
> conf.set("mapred.tasktracker.map.tasks.maximum", "4");
>
> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
>
>
>
> Thanks
>
> Praveen
Re: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by Harsh Chouraria <ha...@cloudera.com>.
Saket,
Do note that while FileInputFormats and most of its derivatives would respect the "mapred.map.tasks" as a property to get a desired number of splits [1], other formats may not consider it at all (i.e. you need to check sources to make sure they can).
[1] - The getSplits(…) interface of InputFormats accept a number of splits to create, but that's about the level of enforcement there, and returned splits' sizes are not checked against the required amount by the framework.
On 06-Jul-2011, at 12:32 AM, Saket Joshi wrote:
> Praveen,
>
> You also need to set the property “mapred.map.tasks” to the value derived from the following formula ( mapred.tasktracker.map.tasks.maximum * Number of tasktracker Nodes in your cluster) . I am using Hadoop 0.20.2 and I am able to increase the number of maps using these two mentoined properties. Similar settings apply for the reducer .
>
> -Saket
>
> From: praveen.peddi@nokia.com [mailto:praveen.peddi@nokia.com]
> Sent: Friday, July 01, 2011 1:03 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: mapred.tasktracker.map.tasks.maximum is not taking into effect
>
> Hi all,
> I am using hadoop 0.20.2. I am setting the property mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my job conf but I am still seeing max of only 2 map and reduce tasks on each node. I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this a bug in 0.20.2 or am I doing something wrong?
>
> Configuration conf = new Configuration();
> conf.set("mapred.tasktracker.map.tasks.maximum", "4");
> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
>
> Thanks
> Praveen
>
> This communication, along with any attachments, may include confidential TouchCommerce Inc. information. Any distribution or copying of this material is strictly prohibited. If you are not the intended recipient of this transmission, please contact the sender immediately.
Re: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by Devaraj Das <dd...@hortonworks.com>.
The properties mapred.tasktracker.{map,reduce}.tasks.maximum are node level
parameters, and are read by tasktrackers when they start up. Passing them in
the job confing doesn't do any change ...
On Tue, Jul 5, 2011 at 12:02 PM, Saket Joshi <sj...@touchcommerce.com>wrote:
> Praveen, ****
>
> ** **
>
> You also need to set the property “mapred.map.tasks” to the value derived
> from the following formula ( mapred.tasktracker.map.tasks.maximum * Number
> of tasktracker Nodes in your cluster) . I am using Hadoop 0.20.2 and I am
> able to increase the number of maps using these two mentoined properties.
> Similar settings apply for the reducer .****
>
> ** **
>
> -Saket****
>
> ** **
>
> *From:* praveen.peddi@nokia.com [mailto:praveen.peddi@nokia.com]
> *Sent:* Friday, July 01, 2011 1:03 PM
> *To:* mapreduce-user@hadoop.apache.org
> *Subject:* mapred.tasktracker.map.tasks.maximum is not taking into effect*
> ***
>
> ** **
>
> Hi all,****
>
> I am using hadoop 0.20.2. I am setting the property
> mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my job
> conf but I am still seeing max of only 2 map and reduce tasks on each node.
> I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this a
> bug in 0.20.2 or am I doing something wrong?****
>
> ** **
>
> Configuration conf = new Configuration();****
>
> conf.set("mapred.tasktracker.map.tasks.maximum", "4");****
>
> conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");****
>
> ** **
>
> Thanks****
>
> Praveen****
>
> This communication, along with any attachments, may include confidential
> TouchCommerce Inc. information. Any distribution or copying of this material
> is strictly prohibited. If you are not the intended recipient of this
> transmission, please contact the sender immediately.
>
RE: mapred.tasktracker.map.tasks.maximum is not taking into effect
Posted by Saket Joshi <sj...@touchcommerce.com>.
Praveen,
You also need to set the property "mapred.map.tasks" to the value derived from the following formula ( mapred.tasktracker.map.tasks.maximum * Number of tasktracker Nodes in your cluster) . I am using Hadoop 0.20.2 and I am able to increase the number of maps using these two mentoined properties. Similar settings apply for the reducer .
-Saket
From: praveen.peddi@nokia.com [mailto:praveen.peddi@nokia.com]
Sent: Friday, July 01, 2011 1:03 PM
To: mapreduce-user@hadoop.apache.org
Subject: mapred.tasktracker.map.tasks.maximum is not taking into effect
Hi all,
I am using hadoop 0.20.2. I am setting the property mapred.tasktracker.map.tasks.maximum = 4 (same for reduce also) on my job conf but I am still seeing max of only 2 map and reduce tasks on each node. I know my machine can run 4 maps and 4 reduce tasks in parallel. Is this a bug in 0.20.2 or am I doing something wrong?
Configuration conf = new Configuration();
conf.set("mapred.tasktracker.map.tasks.maximum", "4");
conf.set("mapred.tasktracker.reduce.tasks.maximum", "4");
Thanks
Praveen
This communication, along with any attachments, may include confidential TouchCommerce Inc. information. Any distribution or copying of this material is strictly prohibited. If you are not the intended recipient of this transmission, please contact the sender immediately.