You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Frank Scholten <fr...@frankscholten.nl> on 2011/06/14 10:12:56 UTC
[WHIRR-55] Whirr generates a hadoop-site.xml without overridden
hadoop properties
Hi all,
When I start a hadoop cluster with overridden properties using
0.5-incubating I only get
a hadoop-site.xml and no mapred-site.xml with the additional
properties. I am running Ubuntu 11.4
I am using the following test setup:
whirr.service-name=hadoop
whirr.cluster-name=test-cluster
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode, 2
hadoop-datanode+hadoop-tasktracker
whirr.provider=aws-ec2
whirr.location-id=eu-west-1
whirr.hardware-id=m1.small
whirr.image-id=eu-west-1/ami-1b9fa86f
whirr.identity=${AWS_ACCESS_KEY_ID}
whirr.credential=${AWS_SECRET_KEY}
whirr.private-key-file=/home/frank/.ssh/id_rsa_whirr
whirr.public-key-file=/home/frank/.ssh/id_rsa_whirr.pub
whirr.cluster-user=frank
hadoop-mapreduce.mapred.child.java.opts=-Xmx1000m
hadoop-mapreduce.mapred.child.ulimit=1500000
hadoop-mapreduce.mapred.map.tasks=2
hadoop-mapreduce.mapred.reduce.tasks=2
hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=2
hadoop-mapreduce.mapred.tasktracker.reduce.tasks.maximum=2
and I run
$ whirr launch-cluster path/to/property/file
Cheers,
Frank
Re: [WHIRR-55] Whirr generates a hadoop-site.xml without overridden
hadoop properties
Posted by Frank Scholten <fr...@frankscholten.nl>.
On Tue, Jun 14, 2011 at 5:59 PM, Andrei Savu <sa...@gmail.com> wrote:
> I've started a cluster using the same recipe and everything looks fine to me.
>
>> When I start a hadoop cluster with overridden properties using
>> 0.5-incubating I only get a hadoop-site.xml and no mapred-site.xml
>> with the additional properties.
>
> On the local machine or on the amazon cloud?
>
> If you are looking at
> <local-machine>~/.whirr/test-cluster/hadoop-site.xml then this is the
> expected behavior. I'm not sure but I think this is not important as
> long as the online cluster is configured as expected.
Yes, I noticed it later on. Just was confused that my local machine
didn't have the overridden properties in hadoop-site.xml but
doesn't seem to be a problem.
Frank
>
> Cheers,
>
> -- Andrei Savu / andreisavu.ro
>
> On Tue, Jun 14, 2011 at 11:12 AM, Frank Scholten <fr...@frankscholten.nl> wrote:
>> Hi all,
>>
>> When I start a hadoop cluster with overridden properties using
>> 0.5-incubating I only get
>> a hadoop-site.xml and no mapred-site.xml with the additional
>> properties. I am running Ubuntu 11.4
>>
>> I am using the following test setup:
>>
>> whirr.service-name=hadoop
>> whirr.cluster-name=test-cluster
>> whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode, 2
>> hadoop-datanode+hadoop-tasktracker
>> whirr.provider=aws-ec2
>> whirr.location-id=eu-west-1
>> whirr.hardware-id=m1.small
>> whirr.image-id=eu-west-1/ami-1b9fa86f
>> whirr.identity=${AWS_ACCESS_KEY_ID}
>> whirr.credential=${AWS_SECRET_KEY}
>> whirr.private-key-file=/home/frank/.ssh/id_rsa_whirr
>> whirr.public-key-file=/home/frank/.ssh/id_rsa_whirr.pub
>> whirr.cluster-user=frank
>>
>> hadoop-mapreduce.mapred.child.java.opts=-Xmx1000m
>> hadoop-mapreduce.mapred.child.ulimit=1500000
>> hadoop-mapreduce.mapred.map.tasks=2
>> hadoop-mapreduce.mapred.reduce.tasks=2
>> hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=2
>> hadoop-mapreduce.mapred.tasktracker.reduce.tasks.maximum=2
>>
>> and I run
>>
>> $ whirr launch-cluster path/to/property/file
>>
>> Cheers,
>>
>> Frank
>>
>
Re: [WHIRR-55] Whirr generates a hadoop-site.xml without overridden
hadoop properties
Posted by Andrei Savu <sa...@gmail.com>.
I've started a cluster using the same recipe and everything looks fine to me.
> When I start a hadoop cluster with overridden properties using
> 0.5-incubating I only get a hadoop-site.xml and no mapred-site.xml
> with the additional properties.
On the local machine or on the amazon cloud?
If you are looking at
<local-machine>~/.whirr/test-cluster/hadoop-site.xml then this is the
expected behavior. I'm not sure but I think this is not important as
long as the online cluster is configured as expected.
Cheers,
-- Andrei Savu / andreisavu.ro
On Tue, Jun 14, 2011 at 11:12 AM, Frank Scholten <fr...@frankscholten.nl> wrote:
> Hi all,
>
> When I start a hadoop cluster with overridden properties using
> 0.5-incubating I only get
> a hadoop-site.xml and no mapred-site.xml with the additional
> properties. I am running Ubuntu 11.4
>
> I am using the following test setup:
>
> whirr.service-name=hadoop
> whirr.cluster-name=test-cluster
> whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode, 2
> hadoop-datanode+hadoop-tasktracker
> whirr.provider=aws-ec2
> whirr.location-id=eu-west-1
> whirr.hardware-id=m1.small
> whirr.image-id=eu-west-1/ami-1b9fa86f
> whirr.identity=${AWS_ACCESS_KEY_ID}
> whirr.credential=${AWS_SECRET_KEY}
> whirr.private-key-file=/home/frank/.ssh/id_rsa_whirr
> whirr.public-key-file=/home/frank/.ssh/id_rsa_whirr.pub
> whirr.cluster-user=frank
>
> hadoop-mapreduce.mapred.child.java.opts=-Xmx1000m
> hadoop-mapreduce.mapred.child.ulimit=1500000
> hadoop-mapreduce.mapred.map.tasks=2
> hadoop-mapreduce.mapred.reduce.tasks=2
> hadoop-mapreduce.mapred.tasktracker.map.tasks.maximum=2
> hadoop-mapreduce.mapred.tasktracker.reduce.tasks.maximum=2
>
> and I run
>
> $ whirr launch-cluster path/to/property/file
>
> Cheers,
>
> Frank
>