You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Edmar Ferreira <ed...@gmail.com> on 2012/01/30 18:44:12 UTC
Set max number of tasks per node
Hi guys,
I want to set my maximum number of tasks per node to 1, so I put this line
on my whirr config file :
hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
However when I run my job with verbose mode I see the value set as 2.
My full config file:
1 whirr.cluster-name=hadoopcluster
2 whirr.instance-templates= 1 hadoop-jobtracker+hadoop-namenode,1
hadoop-datanode+hadoop-tasktracker
3 whirr.provider=aws-ec2
4 #whirr.image-id=us-west-1/ami-f197c9b4
5 #whirr.location-id=us-west-1
6 #whirr.aws-ec2-spot-price=0.085
7 whirr.identity=${env:AWS_ACCESS_KEY_ID}
8 whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
9 whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
10 whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
11 hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
12 whirr.hadoop.version=0.20.2
13 whirr.hadoop.tarball.url=
http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr.hadoop.version}.tar.gz
Thanks.
--
Edmar Ferreira
Co-Founder at Everwrite
Re: Set max number of tasks per node
Posted by Edmar Ferreira <ed...@gmail.com>.
The local file don't have the entry related to max tasks. I will add it and
see what happens.
On Mon, Jan 30, 2012 at 5:01 PM, Andrei Savu <sa...@gmail.com> wrote:
>
> On Mon, Jan 30, 2012 at 8:57 PM, Edmar Ferreira <
> edmaroliveiraferreira@gmail.com> wrote:
>
>> I my local machine whirr creates a hadoop-site.xml how can I change to
>> the more update configuration files ?
>
>
> So the local config file has a different value? If so this is a bug and we
> should fix it.
>
--
Edmar Ferreira
Co-Founder at Everwrite
Re: Set max number of tasks per node
Posted by Andrei Savu <sa...@gmail.com>.
On Mon, Jan 30, 2012 at 8:57 PM, Edmar Ferreira <
edmaroliveiraferreira@gmail.com> wrote:
> I my local machine whirr creates a hadoop-site.xml how can I change to the
> more update configuration files ?
So the local config file has a different value? If so this is a bug and we
should fix it.
Re: Set max number of tasks per node
Posted by Edmar Ferreira <ed...@gmail.com>.
Found it. In the cluster machines it's set right.
I my local machine whirr creates a hadoop-site.xml how can I change to the
more update configuration files ?
I am using hadoop 0.20.2
On Mon, Jan 30, 2012 at 4:48 PM, Edmar Ferreira <
edmaroliveiraferreira@gmail.com> wrote:
> In what directory whirr put the config files int the cluster machines ?
>
>
> On Mon, Jan 30, 2012 at 4:27 PM, Andrei Savu <sa...@gmail.com>wrote:
>
>> I have submitted the new test as part of
>> https://issues.apache.org/jira/browse/WHIRR-493
>>
>> -- Andrei Savu
>>
>>
>> On Mon, Jan 30, 2012 at 8:19 PM, Andrei Savu <sa...@gmail.com>wrote:
>>
>>> I have just checked HadoopConfigurationBuilder
>>> / HadoopConfigurationBuilderTest - everything looks
>>> fine and it's possible to override the value as expected.
>>>
>>> Can you check the Hadoop config files on the remote machines?
>>> Is mapred.tasktracker.map.tasks.maximum set to "2"?
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>>
>>> On Mon, Jan 30, 2012 at 7:44 PM, Edmar Ferreira <
>>> edmaroliveiraferreira@gmail.com> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I want to set my maximum number of tasks per node to 1, so I put this
>>>> line on my whirr config file :
>>>>
>>>> hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
>>>>
>>>> However when I run my job with verbose mode I see the value set as 2.
>>>>
>>>> My full config file:
>>>>
>>>> 1 whirr.cluster-name=hadoopcluster
>>>> 2 whirr.instance-templates= 1 hadoop-jobtracker+hadoop-namenode,1
>>>> hadoop-datanode+hadoop-tasktracker
>>>> 3 whirr.provider=aws-ec2
>>>> 4 #whirr.image-id=us-west-1/ami-f197c9b4
>>>> 5 #whirr.location-id=us-west-1
>>>> 6 #whirr.aws-ec2-spot-price=0.085
>>>> 7 whirr.identity=${env:AWS_ACCESS_KEY_ID}
>>>> 8 whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
>>>> 9 whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>>>> 10 whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>>> 11 hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
>>>> 12 whirr.hadoop.version=0.20.2
>>>> 13 whirr.hadoop.tarball.url=
>>>> http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr.hadoop.version}.tar.gz
>>>>
>>>>
>>>>
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Edmar Ferreira
>>>> Co-Founder at Everwrite
>>>>
>>>>
>>>
>>
>
>
> --
> Edmar Ferreira
> Co-Founder at Everwrite
>
>
--
Edmar Ferreira
Co-Founder at Everwrite
Re: Set max number of tasks per node
Posted by Edmar Ferreira <ed...@gmail.com>.
In what directory whirr put the config files int the cluster machines ?
On Mon, Jan 30, 2012 at 4:27 PM, Andrei Savu <sa...@gmail.com> wrote:
> I have submitted the new test as part of
> https://issues.apache.org/jira/browse/WHIRR-493
>
> -- Andrei Savu
>
>
> On Mon, Jan 30, 2012 at 8:19 PM, Andrei Savu <sa...@gmail.com>wrote:
>
>> I have just checked HadoopConfigurationBuilder
>> / HadoopConfigurationBuilderTest - everything looks
>> fine and it's possible to override the value as expected.
>>
>> Can you check the Hadoop config files on the remote machines?
>> Is mapred.tasktracker.map.tasks.maximum set to "2"?
>>
>> -- Andrei Savu / andreisavu.ro
>>
>>
>> On Mon, Jan 30, 2012 at 7:44 PM, Edmar Ferreira <
>> edmaroliveiraferreira@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I want to set my maximum number of tasks per node to 1, so I put this
>>> line on my whirr config file :
>>>
>>> hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
>>>
>>> However when I run my job with verbose mode I see the value set as 2.
>>>
>>> My full config file:
>>>
>>> 1 whirr.cluster-name=hadoopcluster
>>> 2 whirr.instance-templates= 1 hadoop-jobtracker+hadoop-namenode,1
>>> hadoop-datanode+hadoop-tasktracker
>>> 3 whirr.provider=aws-ec2
>>> 4 #whirr.image-id=us-west-1/ami-f197c9b4
>>> 5 #whirr.location-id=us-west-1
>>> 6 #whirr.aws-ec2-spot-price=0.085
>>> 7 whirr.identity=${env:AWS_ACCESS_KEY_ID}
>>> 8 whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
>>> 9 whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>>> 10 whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>> 11 hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
>>> 12 whirr.hadoop.version=0.20.2
>>> 13 whirr.hadoop.tarball.url=
>>> http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr.hadoop.version}.tar.gz
>>>
>>>
>>>
>>>
>>> Thanks.
>>>
>>> --
>>> Edmar Ferreira
>>> Co-Founder at Everwrite
>>>
>>>
>>
>
--
Edmar Ferreira
Co-Founder at Everwrite
Re: Set max number of tasks per node
Posted by Andrei Savu <sa...@gmail.com>.
I have submitted the new test as part of
https://issues.apache.org/jira/browse/WHIRR-493
-- Andrei Savu
On Mon, Jan 30, 2012 at 8:19 PM, Andrei Savu <sa...@gmail.com> wrote:
> I have just checked HadoopConfigurationBuilder
> / HadoopConfigurationBuilderTest - everything looks
> fine and it's possible to override the value as expected.
>
> Can you check the Hadoop config files on the remote machines?
> Is mapred.tasktracker.map.tasks.maximum set to "2"?
>
> -- Andrei Savu / andreisavu.ro
>
>
> On Mon, Jan 30, 2012 at 7:44 PM, Edmar Ferreira <
> edmaroliveiraferreira@gmail.com> wrote:
>
>> Hi guys,
>>
>> I want to set my maximum number of tasks per node to 1, so I put this
>> line on my whirr config file :
>>
>> hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
>>
>> However when I run my job with verbose mode I see the value set as 2.
>>
>> My full config file:
>>
>> 1 whirr.cluster-name=hadoopcluster
>> 2 whirr.instance-templates= 1 hadoop-jobtracker+hadoop-namenode,1
>> hadoop-datanode+hadoop-tasktracker
>> 3 whirr.provider=aws-ec2
>> 4 #whirr.image-id=us-west-1/ami-f197c9b4
>> 5 #whirr.location-id=us-west-1
>> 6 #whirr.aws-ec2-spot-price=0.085
>> 7 whirr.identity=${env:AWS_ACCESS_KEY_ID}
>> 8 whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
>> 9 whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>> 10 whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>> 11 hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
>> 12 whirr.hadoop.version=0.20.2
>> 13 whirr.hadoop.tarball.url=
>> http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr.hadoop.version}.tar.gz
>>
>>
>>
>>
>> Thanks.
>>
>> --
>> Edmar Ferreira
>> Co-Founder at Everwrite
>>
>>
>
Re: Set max number of tasks per node
Posted by Andrei Savu <sa...@gmail.com>.
I have just checked HadoopConfigurationBuilder
/ HadoopConfigurationBuilderTest - everything looks
fine and it's possible to override the value as expected.
Can you check the Hadoop config files on the remote machines?
Is mapred.tasktracker.map.tasks.maximum set to "2"?
-- Andrei Savu / andreisavu.ro
On Mon, Jan 30, 2012 at 7:44 PM, Edmar Ferreira <
edmaroliveiraferreira@gmail.com> wrote:
> Hi guys,
>
> I want to set my maximum number of tasks per node to 1, so I put this line
> on my whirr config file :
>
> hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
>
> However when I run my job with verbose mode I see the value set as 2.
>
> My full config file:
>
> 1 whirr.cluster-name=hadoopcluster
> 2 whirr.instance-templates= 1 hadoop-jobtracker+hadoop-namenode,1
> hadoop-datanode+hadoop-tasktracker
> 3 whirr.provider=aws-ec2
> 4 #whirr.image-id=us-west-1/ami-f197c9b4
> 5 #whirr.location-id=us-west-1
> 6 #whirr.aws-ec2-spot-price=0.085
> 7 whirr.identity=${env:AWS_ACCESS_KEY_ID}
> 8 whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
> 9 whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
> 10 whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
> 11 hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=1
> 12 whirr.hadoop.version=0.20.2
> 13 whirr.hadoop.tarball.url=
> http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr.hadoop.version}.tar.gz
>
>
>
>
> Thanks.
>
> --
> Edmar Ferreira
> Co-Founder at Everwrite
>
>