You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Benjamin Clark <be...@daltonclark.com> on 2011/03/12 04:56:58 UTC
hadoop config property override problem
Yes, thanks Tom, that works.
Many features and design changes in this branch are much appreciated, especially the config property override in the whirr config file.
I found one problem, at least in the 0.4 branch. If you override a property with a comma-separated list, for example:
hadoop-common.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
then what actually shows up in core-site.xml is surrounded by brackets and has spaces between the elements of the list, so
[org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec]
Hadoop is not chomping or trimming that kind of thing, so you need to remove the spaces and brackets to get that to work. It's easy enough to patch the file, deploy to the slaves and restart, but I'm wondering if that's accounted for anywhere. I scanned CHANGES.txt in trunk and I don't see it.
--Ben
On Mar 10, 2011, at 5:41 PM, Tom White wrote:
> On Thu, Mar 10, 2011 at 1:19 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>> Thank you both, Tom and Andrei. Now that I know where the faq is, I hope to bother you less with things that are documented!
>>
>> I think I should be all set with customization, but I need to build. BUILD.txt says 'mvn clean install' or mvn package -Ppackage. I can do that, and mvn reports success but then I try to use whirr-cli-0.4.0-incubating.jar as the jar, and the manifest has no main class (OK, I can supply 'org.apache.whirr.cli.Main' if I need to), and in any case the jar is not a fat jar, as I see all the publicly distributed versions are. It looks in the poms as if you have the maven assembly plugins trying to make a fat jar, but it doesn't seem to be doing it for me.
>
> Whirr no longer produces a shaded (fat) JAR as of 0.4.0 and trunk, so
> perhaps it is working. Try bin/whirr and it should list the roles for
> you.
>
> Cheers
> Tom
>
>>
>> What am I doing wrong?
>>
>> I'm doing this on a Mac like so:
>>
>> $ ruby --version
>> ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10]
>> $ mvn --version
>> Apache Maven 3.0.2 (r1056850; 2011-01-08 19:58:10-0500)
>> Java version: 1.6.0_24, vendor: Apple Inc.
>> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>> Default locale: en_US, platform encoding: MacRoman
>> OS name: "mac os x", version: "10.6.6", arch: "x86_64", family: "mac"
>> $ java -version
>> java version "1.6.0_24"
>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
>>
>>
>>
>>
>> Now I think my only problem is that
>> On Mar 10, 2011, at 2:35 PM, Andrei Savu wrote:
>>
>>> Starting with the upcoming 0.4.0 release Whirr is no longer using S3
>>> for storing the install and configure scripts. You can grab the
>>> scripts from:
>>>
>>> ${WHIRR_HOME}/services/${SERVICE_NAME}/src/main/resources/functions/{install,configure}_SERVICE.sh
>>>
>>> It's also easier to customize the scripts. You just need to place your
>>> version in ${WHIRR_HOME}/functions (I believe it should have the same
>>> name).
>>>
>>> From the 0.4.0 FAQ: "If you want to change the scripts then you can
>>> place a modified copy of the
>>> scripts in a _functions_ directory in Whirr's installation directory. The
>>> original versions of the scripts can be found in _functions_ directories in the
>>> source trees."
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>> On Thu, Mar 10, 2011 at 9:27 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>> So if we grab install_cdh_hadoop.sh from the source tree, and put a customized version in our own bucket, and set whirr.run-url-base to the root of that bucket, it should work, even in 4.0 and after?
>>>>
>>>> Based on the FAQ I tried a few of these to attempt to verify I was on the right track:
>>>>
>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop
>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop
>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop
>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop.sh
>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop.sh
>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop.sh
>>>>
>>>> but all give 404s.
>>>>
>>>>
>>>>
>>>> On Mar 10, 2011, at 12:01 AM, Tom White wrote:
>>>>
>>>>> Sorry I missed this thread. On 0.4.0 and later
>>>>>
>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>
>>>>> changes to
>>>>>
>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>
>>>>> Cheers,
>>>>> Tom
>>>>>
>>>>> On Wed, Mar 9, 2011 at 8:23 AM, Sebastian Schoenherr
>>>>> <se...@uibk.ac.at> wrote:
>>>>>> Hi Saptarshi,
>>>>>> I tried to execute my working whirr 0.3.0 configuration (identical to your
>>>>>> property file, using cloudera scripts) on branch-0.4 and the same issues
>>>>>> arised for me. Unfortunately I'm not sure yet why it's not working with
>>>>>> branch-0.4. Is using branch-0.3 an option for you?
>>>>>> Any other guesses?
>>>>>> cheers,
>>>>>> sebastian
>>>>>>
>>>>>>
>>>>>> On 08.03.2011 05:41, Saptarshi Guha wrote:
>>>>>>>
>>>>>>> once again, i've changed the secret identity ..
>>>>>>>
>>>>>>> On Mon, Mar 7, 2011 at 8:41 PM, Saptarshi Guha
>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>> No such luck on my end This is my script file, you can test that the
>>>>>>>> scripts download. But when I log in
>>>>>>>> hadoop version is. (I pulled the latest git). Also my scripts (you can
>>>>>>>> confirm if you download) echo a small line to files in /tmp.
>>>>>>>> They are not being created.
>>>>>>>>
>>>>>>>> Hadoop 0.20.2
>>>>>>>> Subversion
>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>> -r 911707
>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>
>>>>>>>> whirr.cluster-name=revotesting2
>>>>>>>> whirr.service-name=hadoop
>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>> whirr.provider=aws-ec2
>>>>>>>> whirr.identity= AKIAJH5JBSI5KJ7YZQ6A
>>>>>>>> whirr.credential= b/kqLJAHOdRA4L30n7Zt8Edz383B1ARtPI3wiyD6
>>>>>>>> whirr.location-id=us-east-1
>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>> whirr.run-url-base=http://ml.stat.purdue.edu/whirr-scripts/
>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>
>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>
>>>>>>>> ##http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 7, 2011 at 9:59 AM, Benjamin Clark<be...@daltonclark.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> In my experience you need
>>>>>>>>>
>>>>>>>>> whirr.run-url-base=http://name-of-my-bucket-with-customized-scripts/
>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>
>>>>>>>>> and then you *also* need to have a copy of sun/java/install in that same
>>>>>>>>> bucket.
>>>>>>>>>
>>>>>>>>> And both of those scripts need to be public-readable.
>>>>>>>>>
>>>>>>>>> So in the end you should be able to do
>>>>>>>>> curl
>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/cloudera/cdh/install
>>>>>>>>>
>>>>>>>>> and
>>>>>>>>> curl
>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/sun/java/install
>>>>>>>>>
>>>>>>>>> Even if you haven't customizied sun/java/install, it needs to be there.
>>>>>>>>>
>>>>>>>>> If you do all that, the scripts will run and you will have the versions
>>>>>>>>> you asked for.
>>>>>>>>>
>>>>>>>>> `hadoop version` on the name node then says, in my case:
>>>>>>>>>
>>>>>>>>> Hadoop 0.20.2-CDH3B4
>>>>>>>>>
>>>>>>>>> On Mar 7, 2011, at 12:41 PM, Saptarshi Guha wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Fixed the security slip up. Did the hadoop version thing and got this
>>>>>>>>>>
>>>>>>>>>> [root@domU-12-31-39-0B-CC-41 ~]# hadoop version
>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>> Subversion
>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>> -r 911707
>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>
>>>>>>>>>> So i guess its not CDH.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Saptarshi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 7, 2011 at 9:22 AM, Saptarshi Guha
>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> dear me! thanks, will do right away.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 7, 2011 at 1:46 AM, Sebastian Schoenherr
>>>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>>>> Try to execute "hadoop version" on your namenode, if the output is
>>>>>>>>>>>> Hadoop
>>>>>>>>>>>> 0.20.2-CDH3B4, the current cloudera distribution has been installed.
>>>>>>>>>>>> Btw, I would recommend to set your current Access Key ID and Secret
>>>>>>>>>>>> Key
>>>>>>>>>>>> inactive, since you posted it in your prop file.
>>>>>>>>>>>> cheers
>>>>>>>>>>>> sebastian
>>>>>>>>>>>>
>>>>>>>>>>>> On 06.03.2011 06:54, Saptarshi Guha wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I did git clone of the latest whirr and copied cloudera scripts into
>>>>>>>>>>>>> the script directory (copied over
>>>>>>>>>>>>> from whirr-0.3-incubating).
>>>>>>>>>>>>>
>>>>>>>>>>>>> My properties file is at the end of this email.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, I don't think the scripts are being run because the
>>>>>>>>>>>>> jobtracker
>>>>>>>>>>>>> is the default Apache hadoop jobtracker and not the cloudera
>>>>>>>>>>>>> jobtracker.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Have i missed something?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>
>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>
>>>>>>>>>>>>> ## Properties
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> whirr.cluster-name=revotesting
>>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>>> whirr.identity= AKIAI3FUFFXAPYLE7CJA
>>>>>>>>>>>>> whirr.credential= 2Yq3Ar2HSxK/hbwZHs6aN6yrh0yfGNSPTpVw3t2n
>>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>>>
>>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>>
>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: aws 64-bit c1.xlarge problems
Posted by Andrei Savu <sa...@gmail.com>.
Thanks for sharing. I'm thinking about defining a set of of supported
AMIs / OSes for Whirr and test them all when building a new release.
On Fri, Mar 18, 2011 at 9:57 PM, Benjamin Clark <be...@daltonclark.com> wrote:
> I read the manual and figured a bit more of this out. Amazon may change the
> defaults in their console without an announcement, but they document what
> they're doing here: http://aws.amazon.com/amazon-linux-ami/
> The /media/ephemeral0 is for one of their Amazon linux instances that has
> S3-backed non-durable storage. It seems as if the ebs-backed ones have no
> non-durable storage by default, and the S3-backed ones do, but in that
> eccentric location (eccentric relative to what everybody else does on
> Amazon). So if we like the S3-backed instances, we can hack the
> install_cdh_hadoop.sh script by adding
> rm -rf /mnt
> ln -s /media/ephemeral0 /mnt
> or we can write a whole thing that spins up an ebs volume per node and
> attaches it, for the ebs-backed ones.
> Is there any experience among the users as to which will be more stable and
> perform better? I've got the S3-backed one working, so I'll use that and
> just bake it off against the Alestic/ubuntu system that now also works for
> me, unless there's a compelling case for the ebs-backed thing.
> --Ben
>
>
> Andrei,
>
> The release candidate code does work. Perhaps something is different,
> relative to the patched frankenstein I was using, perhaps I had some local
> corruption or config problem.
>
> It sets up everything as whatever my local user is, by default, and the
> override as whirr.cluster-user works as well.
>
> In any case, at the rate AWS seems to be changing the configuration of
> 'amazon linux' perhaps it's less useful than I thought. Last week the
> default amis in the console had a bunch of spare disk space on the
> /media/ephemeral0 partition, which I could symlink /mnt to in the
> install_cdh_hadoop.sh script, and then hdfs would have a decent amount of
> space. Now there is no such thing, so I suppose I would have to launch an
> ebs volume per node and mount that. This is now tipping over into the "too
> much trouble" zone for me. And in the mean time I got all my native stuff
> (hadoop-lzo and R/Rhipe) working on ubuntu, so I think I'm going to use the
> Alestic image from the recipe for a while. If there's an obvious candidate
> up there for "reasonably-modern redhat derivative ami from a source on the
> good lists that behaves well," I'd like to know what it is. By 'reasonably
> modern' I mean having default python >= 2.5.
>
> I liked the old custom of having /mnt be a separate partition of a decent
> size. I hope this is just a glitch with AWS. I suspect it may be because
> jclouds/whirr is showing (e.g.) in the output:
> volumes=[[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false,
> isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc,
> durable=false, isBootDevice=false]
> So theoretically the disk space is still there on those non-boot,
> non-durable devices, but I cannot mount them.
>
>
> I also tried the cluster ami, because I am intrigued by the possibilities
> for good performance. Sounds great for hadoop, doesn't it? But it won't
> even start the nodes, giving this:
>
> Configuring template
> Unexpected error while starting 1 nodes, minimum 1 nodes for
> [hadoop-namenode, hadoop-jobtracker] of cluster bhcLA
> java.util.concurrent.ExecutionException:
> org.jclouds.http.HttpResponseException: command: POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with response: HTTP/1.1
> 400 Bad Request; content: [Non-Windows AMIs with a virtualization type of
> 'hvm' currently may only be used with Cluster Compute instance types.]
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.waitForOutcomes(BootstrapClusterAction.java:307)
> at
> org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:260)
> at
> org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:221)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: org.jclouds.http.HttpResponseException: command: POST
> https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with response: HTTP/1.1
> 400 Bad Request; content: [Non-Windows AMIs with a virtualization type of
> 'hvm' currently may only be used with Cluster Compute instance types.]
> at
> org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
>
> There must be something a bit more involved to specify cluster instances in
> the amazon api, perhaps not (yet) supported by jclouds? I'm afraid I don't
> need this enough right now to justify digging further .
>
>
> Anyway, thanks for all your help and advice on this.
>
> --Ben
>
>
> On Mar 17, 2011, at 7:01 PM, Andrei Savu wrote:
>
> Strange! I will try your properties file tomorrow.
>
> If you want to try again you can find the artifacts for 0.4.0 RC1 here:
>
> http://people.apache.org/~asavu/whirr-0.4.0-incubating-candidate-1
>
> On Thu, Mar 17, 2011 at 8:41 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>
> Andrei,
>
> Thanks for looking at this. Unfortunately it does not seem to work.
>
> Using the Amazon linux 64-bit ami with no whirr.cluster-user, or if I set it
> to 'ben' or whatever else, I get this.
>
> 1) SshException on node us-east-1/i-62de280d:
>
> org.jclouds.ssh.SshException: ec2-user@72.44.35.254:22: Error connecting to
> session.
>
> at
> org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>
> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>
> at
> org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>
> at
> org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>
> So it doesn't seem to be honoring that property, and it's definitely not
> allowing me to log in to any nodes, 'ben', 'ec2-user' or 'root'.
>
> The ubuntu ami from the recipes continues to work fine.
>
> Here's the full config file I'm using. I grabbed the recipe from trunk and
> put my stuff back in, to make sure I'm not missing a new setting:
>
> whirr.cluster-name=bhcTL
>
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
> hadoop-datanode+hadoop-tasktracker
>
> whirr.hadoop-install-function=install_cdh_hadoop
>
> whirr.hadoop-configure-function=configure_cdh_hadoop
>
> whirr.provider=aws-ec2
>
> whirr.identity=${env:AWS_ACCESS_KEY_ID}
>
> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
>
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-hkey
>
> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-hkey.pub
>
> whirr.cluster-user=ben
>
> # Amazon linux 32-bit--works
>
> #whirr.hardware-id=c1.medium
>
> #whirr.image-id=us-east-1/ami-d59d6bbc
>
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/ -- works
>
> #whirr.hardware-id=c1.xlarge
>
> #whirr.image-id=us-east-1/ami-da0cf8b3
>
> # Amazon linux 64-bit as of 3/11:--doesn't work
>
> whirr.hardware-id=c1.xlarge
>
> whirr.image-id=us-east-1/ami-8e1fece7
>
> #Cluster compute --doesn't work
>
> #whirr.hardward-id=cc1.4xlarge
>
> #whirr.image-id=us-east-1/ami-321eed5b
>
> whirr.location-id=us-east-1d
>
> hadoop-hdfs.dfs.permissions=false
>
> hadoop-hdfs.dfs.replication=2
>
>
> --Ben
>
>
>
>
> On Mar 17, 2011, at 1:08 PM, Andrei Savu wrote:
>
> Ben, could you give it one more try using the current trunk?
>
> You can specify the user by setting the option whirr.cluster-user
>
> (defaults to current system user).
>
> On Wed, Mar 16, 2011 at 11:23 PM, Benjamin Clark <be...@daltonclark.com>
> wrote:
>
> Andrei,
>
> Thanks.
>
> After patching with 158, it launches fine as me on that Ubuntu image from
> the recipe (i.e. on my client machine I am 'ben', so now the aws user that
> has sudo, and as whom I can log in is also 'ben'), so that looks good.
>
> But it's now doing this with amazon linux (ami-da0cf8b3, which was the
> default 64-bit ami a few days ago, and may still be) during launch:
>
> 1) SshException on node us-east-1/i-b2678ddd:
>
> org.jclouds.ssh.SshException: ben@50.16.96.211:22: Error connecting to
> session.
>
> at
> org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>
> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>
> at
> org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>
> at
> org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>
> at
> org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:45)
>
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> So it seems as if the key part of jclouds authentication setup is still
> failing for the amazon linux/ec2-user scenario, i.e. trying to set up as the
> local user, but failing.
>
> Is there a property for the user it launches as? Or does it just do
> whichever user you are locally, instead of ec2-user/ubuntu/root, depending
> on the default, as before?
>
> I can switch to ubuntu, but I have a fair amount of native code setup in my
> custom scripts and would prefer to stick with a redhattish version if
> possible.
>
> Looking ahead, I want to benchmark plain old 64-bit instances against
> cluster instances, to see if the allegedly improved networking gives us a
> boost, and the available ones I see are Suse and Amazon linux. When I
> switch to the amazon linux one, like so:
>
> whirr.hardward-id=cc1.4xlarge
>
> whirr.image-id=us-east-1/ami-321eed5b
>
> I get different a different problem:
>
> Exception in thread "main" java.util.NoSuchElementException: hardwares don't
> support any images: [biggest=false, fastest=false, imageName=null,
> imageDescription=Amazon Linux AMI x86_64 HVM EBS EXT4,
> imageId=us-east-1/ami-321eed5b, imageVersion=ext4, location=[id=us-east-1,
> scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA],
> metadata={}], minCores=0.0, minRam=0, osFamily=unrecognized, osName=null,
> osDescription=amazon/amzn-hvm-ami-2011.02.1-beta.x86_64-ext4, osVersion=,
> osArch=hvm, os64Bit=true, hardwareId=m1.small]
>
> [[id=cc1.4xlarge, providerId=cc1.4xlarge, name=null, processors=[[cores=4.0,
> speed=4.0], [cores=4.0, speed=4.0]], ram=23552, volumes=[[id=null,
> type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
> [id=null, type=LOCAL, size=840.0, device=/dev/sdb, durable=false,
> isBootDevice=false], [id=null, type=LOCAL, size=840.0, device=/dev/sdc,
> durable=false, isBootDevice=false]], supportsI
>
> but I imagine that if using cluster instances is going to be possible,
> support for amazon linux will be needed.
>
> --Ben
>
>
> On Mar 16, 2011, at 4:07 PM, Andrei Savu wrote:
>
> I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
>
> going to commit WHIRR-158 [1] tomorrow and it should fix the problem
>
> you are seeing. We should be able to restart the vote for the 0.4.0
>
> release after fixing this issue.
>
> [0] https://issues.apache.org/jira/browse/WHIRR-264
>
> [1] https://issues.apache.org/jira/browse/WHIRR-158
>
> -- Andrei Savu / andreisavu.ro
>
> On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>
> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon
> linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default
> for new amazon linux instances, a few days ago) with good success. I took
> the default hadoop-ec2.properties recipe and modified it slightly to suit my
> needs. I'm now trying with basically the same properties file, but when I
> use
>
> whirr.hardware-id=c1.xlarge
>
> and then either this (from the recipe)
>
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>
> whirr.image-id=us-east-1/ami-da0cf8b3
>
> or this:
>
> # Amazon linux 64-bit, default as of 3/11:
>
> whirr.image-id=us-east-1/ami-8e1fece7
>
> I get a a failure to install the right public key, so that I can't log into
> the name node (or any other nodes, for that matter).
>
>
> My whole config file is this:
>
> whirr.cluster-name=bhcL4
>
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4
> hadoop-datanode+hadoop-tasktracker
>
> whirr.hadoop-install-function=install_cdh_hadoop
>
> whirr.hadoop-configure-function=configure_cdh_hadoop
>
> whirr.provider=aws-ec2
>
> whirr.identity=...
>
> whirr.credential=...
>
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
>
> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
>
> whirr.hardware-id=c1.xlarge
>
> #whirr.hardware-id=c1.medium
>
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>
> whirr.image-id=us-east-1/ami-da0cf8b3
>
> # Amazon linux as of 3/11:
>
> #whirr.image-id=us-east-1/ami-8e1fece7
>
> # If you choose a different location, make sure whirr.image-id is updated
> too
>
> whirr.location-id=us-east-1d
>
> hadoop-hdfs.dfs.permissions=false
>
> hadoop-hdfs.dfs.replication=2
>
>
>
> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d
> and whirr.location-id=us-east-1
>
>
>
>
>
>
>
>
Re: aws 64-bit c1.xlarge problems
Posted by Benjamin Clark <be...@daltonclark.com>.
I read the manual and figured a bit more of this out. Amazon may change the defaults in their console without an announcement, but they document what they're doing here: http://aws.amazon.com/amazon-linux-ami/
The /media/ephemeral0 is for one of their Amazon linux instances that has S3-backed non-durable storage. It seems as if the ebs-backed ones have no non-durable storage by default, and the S3-backed ones do, but in that eccentric location (eccentric relative to what everybody else does on Amazon). So if we like the S3-backed instances, we can hack the install_cdh_hadoop.sh script by adding
rm -rf /mnt
ln -s /media/ephemeral0 /mnt
or we can write a whole thing that spins up an ebs volume per node and attaches it, for the ebs-backed ones.
Is there any experience among the users as to which will be more stable and perform better? I've got the S3-backed one working, so I'll use that and just bake it off against the Alestic/ubuntu system that now also works for me, unless there's a compelling case for the ebs-backed thing.
--Ben
>
>> Andrei,
>>
>> The release candidate code does work. Perhaps something is different, relative to the patched frankenstein I was using, perhaps I had some local corruption or config problem.
>>
>> It sets up everything as whatever my local user is, by default, and the override as whirr.cluster-user works as well.
>>
>> In any case, at the rate AWS seems to be changing the configuration of 'amazon linux' perhaps it's less useful than I thought. Last week the default amis in the console had a bunch of spare disk space on the /media/ephemeral0 partition, which I could symlink /mnt to in the install_cdh_hadoop.sh script, and then hdfs would have a decent amount of space. Now there is no such thing, so I suppose I would have to launch an ebs volume per node and mount that. This is now tipping over into the "too much trouble" zone for me. And in the mean time I got all my native stuff (hadoop-lzo and R/Rhipe) working on ubuntu, so I think I'm going to use the Alestic image from the recipe for a while. If there's an obvious candidate up there for "reasonably-modern redhat derivative ami from a source on the good lists that behaves well," I'd like to know what it is. By 'reasonably modern' I mean having default python >= 2.5.
>>
>> I liked the old custom of having /mnt be a separate partition of a decent size. I hope this is just a glitch with AWS. I suspect it may be because jclouds/whirr is showing (e.g.) in the output:
>> volumes=[[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]
>> So theoretically the disk space is still there on those non-boot, non-durable devices, but I cannot mount them.
>>
>>
>> I also tried the cluster ami, because I am intrigued by the possibilities for good performance. Sounds great for hadoop, doesn't it? But it won't even start the nodes, giving this:
>>
>> Configuring template
>> Unexpected error while starting 1 nodes, minimum 1 nodes for [hadoop-namenode, hadoop-jobtracker] of cluster bhcLA
>> java.util.concurrent.ExecutionException: org.jclouds.http.HttpResponseException: command: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [Non-Windows AMIs with a virtualization type of 'hvm' currently may only be used with Cluster Compute instance types.]
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.waitForOutcomes(BootstrapClusterAction.java:307)
>> at org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:260)
>> at org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:221)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:680)
>> Caused by: org.jclouds.http.HttpResponseException: command: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [Non-Windows AMIs with a virtualization type of 'hvm' currently may only be used with Cluster Compute instance types.]
>> at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
>>
>> There must be something a bit more involved to specify cluster instances in the amazon api, perhaps not (yet) supported by jclouds? I'm afraid I don't need this enough right now to justify digging further .
>>
>>
>> Anyway, thanks for all your help and advice on this.
>>
>> --Ben
>>
>>
>> On Mar 17, 2011, at 7:01 PM, Andrei Savu wrote:
>>
>>> Strange! I will try your properties file tomorrow.
>>>
>>> If you want to try again you can find the artifacts for 0.4.0 RC1 here:
>>> http://people.apache.org/~asavu/whirr-0.4.0-incubating-candidate-1
>>>
>>> On Thu, Mar 17, 2011 at 8:41 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>> Andrei,
>>>>
>>>> Thanks for looking at this. Unfortunately it does not seem to work.
>>>>
>>>> Using the Amazon linux 64-bit ami with no whirr.cluster-user, or if I set it to 'ben' or whatever else, I get this.
>>>>
>>>> 1) SshException on node us-east-1/i-62de280d:
>>>> org.jclouds.ssh.SshException: ec2-user@72.44.35.254:22: Error connecting to session.
>>>> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>>>> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>>>> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>>>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>>>>
>>>> So it doesn't seem to be honoring that property, and it's definitely not allowing me to log in to any nodes, 'ben', 'ec2-user' or 'root'.
>>>>
>>>> The ubuntu ami from the recipes continues to work fine.
>>>>
>>>> Here's the full config file I'm using. I grabbed the recipe from trunk and put my stuff back in, to make sure I'm not missing a new setting:
>>>>
>>>> whirr.cluster-name=bhcTL
>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2 hadoop-datanode+hadoop-tasktracker
>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>> whirr.provider=aws-ec2
>>>> whirr.identity=${env:AWS_ACCESS_KEY_ID}
>>>> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-hkey
>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-hkey.pub
>>>> whirr.cluster-user=ben
>>>> # Amazon linux 32-bit--works
>>>> #whirr.hardware-id=c1.medium
>>>> #whirr.image-id=us-east-1/ami-d59d6bbc
>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/ -- works
>>>> #whirr.hardware-id=c1.xlarge
>>>> #whirr.image-id=us-east-1/ami-da0cf8b3
>>>> # Amazon linux 64-bit as of 3/11:--doesn't work
>>>> whirr.hardware-id=c1.xlarge
>>>> whirr.image-id=us-east-1/ami-8e1fece7
>>>> #Cluster compute --doesn't work
>>>> #whirr.hardward-id=cc1.4xlarge
>>>> #whirr.image-id=us-east-1/ami-321eed5b
>>>> whirr.location-id=us-east-1d
>>>> hadoop-hdfs.dfs.permissions=false
>>>> hadoop-hdfs.dfs.replication=2
>>>>
>>>>
>>>> --Ben
>>>>
>>>>
>>>>
>>>>
>>>> On Mar 17, 2011, at 1:08 PM, Andrei Savu wrote:
>>>>
>>>>> Ben, could you give it one more try using the current trunk?
>>>>>
>>>>> You can specify the user by setting the option whirr.cluster-user
>>>>> (defaults to current system user).
>>>>>
>>>>> On Wed, Mar 16, 2011 at 11:23 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>> Andrei,
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> After patching with 158, it launches fine as me on that Ubuntu image from the recipe (i.e. on my client machine I am 'ben', so now the aws user that has sudo, and as whom I can log in is also 'ben'), so that looks good.
>>>>>>
>>>>>> But it's now doing this with amazon linux (ami-da0cf8b3, which was the default 64-bit ami a few days ago, and may still be) during launch:
>>>>>>
>>>>>> 1) SshException on node us-east-1/i-b2678ddd:
>>>>>> org.jclouds.ssh.SshException: ben@50.16.96.211:22: Error connecting to session.
>>>>>> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>>>>>> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>>>>>> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>>>>>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>>>>>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:45)
>>>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>
>>>>>> So it seems as if the key part of jclouds authentication setup is still failing for the amazon linux/ec2-user scenario, i.e. trying to set up as the local user, but failing.
>>>>>>
>>>>>> Is there a property for the user it launches as? Or does it just do whichever user you are locally, instead of ec2-user/ubuntu/root, depending on the default, as before?
>>>>>>
>>>>>> I can switch to ubuntu, but I have a fair amount of native code setup in my custom scripts and would prefer to stick with a redhattish version if possible.
>>>>>>
>>>>>> Looking ahead, I want to benchmark plain old 64-bit instances against cluster instances, to see if the allegedly improved networking gives us a boost, and the available ones I see are Suse and Amazon linux. When I switch to the amazon linux one, like so:
>>>>>>
>>>>>> whirr.hardward-id=cc1.4xlarge
>>>>>> whirr.image-id=us-east-1/ami-321eed5b
>>>>>>
>>>>>> I get different a different problem:
>>>>>>
>>>>>> Exception in thread "main" java.util.NoSuchElementException: hardwares don't support any images: [biggest=false, fastest=false, imageName=null, imageDescription=Amazon Linux AMI x86_64 HVM EBS EXT4, imageId=us-east-1/ami-321eed5b, imageVersion=ext4, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=unrecognized, osName=null, osDescription=amazon/amzn-hvm-ami-2011.02.1-beta.x86_64-ext4, osVersion=, osArch=hvm, os64Bit=true, hardwareId=m1.small]
>>>>>> [[id=cc1.4xlarge, providerId=cc1.4xlarge, name=null, processors=[[cores=4.0, speed=4.0], [cores=4.0, speed=4.0]], ram=23552, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=840.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=840.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsI
>>>>>>
>>>>>> but I imagine that if using cluster instances is going to be possible, support for amazon linux will be needed.
>>>>>>
>>>>>> --Ben
>>>>>>
>>>>>>
>>>>>> On Mar 16, 2011, at 4:07 PM, Andrei Savu wrote:
>>>>>>
>>>>>>> I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
>>>>>>> going to commit WHIRR-158 [1] tomorrow and it should fix the problem
>>>>>>> you are seeing. We should be able to restart the vote for the 0.4.0
>>>>>>> release after fixing this issue.
>>>>>>>
>>>>>>> [0] https://issues.apache.org/jira/browse/WHIRR-264
>>>>>>> [1] https://issues.apache.org/jira/browse/WHIRR-158
>>>>>>>
>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>
>>>>>>> On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>>>> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
>>>>>>>>
>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>
>>>>>>>> and then either this (from the recipe)
>>>>>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>>>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>>>>>>
>>>>>>>> or this:
>>>>>>>> # Amazon linux 64-bit, default as of 3/11:
>>>>>>>> whirr.image-id=us-east-1/ami-8e1fece7
>>>>>>>>
>>>>>>>> I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
>>>>>>>>
>>>>>>>>
>>>>>>>> My whole config file is this:
>>>>>>>>
>>>>>>>> whirr.cluster-name=bhcL4
>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
>>>>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>>>> whirr.provider=aws-ec2
>>>>>>>> whirr.identity=...
>>>>>>>> whirr.credential=...
>>>>>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
>>>>>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>> #whirr.hardware-id=c1.medium
>>>>>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>>>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>>>>>> # Amazon linux as of 3/11:
>>>>>>>> #whirr.image-id=us-east-1/ami-8e1fece7
>>>>>>>> # If you choose a different location, make sure whirr.image-id is updated too
>>>>>>>> whirr.location-id=us-east-1d
>>>>>>>> hadoop-hdfs.dfs.permissions=false
>>>>>>>> hadoop-hdfs.dfs.replication=2
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
>>>>>>
>>>>>>
>>>>
>>>>
>>
>
Re: aws 64-bit c1.xlarge problems
Posted by Benjamin Clark <be...@daltonclark.com>.
Andrei,
The release candidate code does work. Perhaps something is different, relative to the patched frankenstein I was using, perhaps I had some local corruption or config problem.
It sets up everything as whatever my local user is, by default, and the override as whirr.cluster-user works as well.
In any case, at the rate AWS seems to be changing the configuration of 'amazon linux' perhaps it's less useful than I thought. Last week the default amis in the console had a bunch of spare disk space on the /media/ephemeral0 partition, which I could symlink /mnt to in the install_cdh_hadoop.sh script, and then hdfs would have a decent amount of space. Now there is no such thing, so I suppose I would have to launch an ebs volume per node and mount that. This is now tipping over into the "too much trouble" zone for me. And in the mean time I got all my native stuff (hadoop-lzo and R/Rhipe) working on ubuntu, so I think I'm going to use the Alestic image from the recipe for a while. If there's an obvious candidate up there for "reasonably-modern redhat derivative ami from a source on the good lists that behaves well," I'd like to know what it is. By 'reasonably modern' I mean having default python >= 2.5.
I liked the old custom of having /mnt be a separate partition of a decent size. I hope this is just a glitch with AWS. I suspect it may be because jclouds/whirr is showing (e.g.) in the output:
volumes=[[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]
So theoretically the disk space is still there on those non-boot, non-durable devices, but I cannot mount them.
I also tried the cluster ami, because I am intrigued by the possibilities for good performance. Sounds great for hadoop, doesn't it? But it won't even start the nodes, giving this:
Configuring template
Unexpected error while starting 1 nodes, minimum 1 nodes for [hadoop-namenode, hadoop-jobtracker] of cluster bhcLA
java.util.concurrent.ExecutionException: org.jclouds.http.HttpResponseException: command: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [Non-Windows AMIs with a virtualization type of 'hvm' currently may only be used with Cluster Compute instance types.]
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.waitForOutcomes(BootstrapClusterAction.java:307)
at org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:260)
at org.apache.whirr.cluster.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:221)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.jclouds.http.HttpResponseException: command: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [Non-Windows AMIs with a virtualization type of 'hvm' currently may only be used with Cluster Compute instance types.]
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
There must be something a bit more involved to specify cluster instances in the amazon api, perhaps not (yet) supported by jclouds? I'm afraid I don't need this enough right now to justify digging further .
Anyway, thanks for all your help and advice on this.
--Ben
On Mar 17, 2011, at 7:01 PM, Andrei Savu wrote:
> Strange! I will try your properties file tomorrow.
>
> If you want to try again you can find the artifacts for 0.4.0 RC1 here:
> http://people.apache.org/~asavu/whirr-0.4.0-incubating-candidate-1
>
> On Thu, Mar 17, 2011 at 8:41 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>> Andrei,
>>
>> Thanks for looking at this. Unfortunately it does not seem to work.
>>
>> Using the Amazon linux 64-bit ami with no whirr.cluster-user, or if I set it to 'ben' or whatever else, I get this.
>>
>> 1) SshException on node us-east-1/i-62de280d:
>> org.jclouds.ssh.SshException: ec2-user@72.44.35.254:22: Error connecting to session.
>> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>>
>> So it doesn't seem to be honoring that property, and it's definitely not allowing me to log in to any nodes, 'ben', 'ec2-user' or 'root'.
>>
>> The ubuntu ami from the recipes continues to work fine.
>>
>> Here's the full config file I'm using. I grabbed the recipe from trunk and put my stuff back in, to make sure I'm not missing a new setting:
>>
>> whirr.cluster-name=bhcTL
>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2 hadoop-datanode+hadoop-tasktracker
>> whirr.hadoop-install-function=install_cdh_hadoop
>> whirr.hadoop-configure-function=configure_cdh_hadoop
>> whirr.provider=aws-ec2
>> whirr.identity=${env:AWS_ACCESS_KEY_ID}
>> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-hkey
>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-hkey.pub
>> whirr.cluster-user=ben
>> # Amazon linux 32-bit--works
>> #whirr.hardware-id=c1.medium
>> #whirr.image-id=us-east-1/ami-d59d6bbc
>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/ -- works
>> #whirr.hardware-id=c1.xlarge
>> #whirr.image-id=us-east-1/ami-da0cf8b3
>> # Amazon linux 64-bit as of 3/11:--doesn't work
>> whirr.hardware-id=c1.xlarge
>> whirr.image-id=us-east-1/ami-8e1fece7
>> #Cluster compute --doesn't work
>> #whirr.hardward-id=cc1.4xlarge
>> #whirr.image-id=us-east-1/ami-321eed5b
>> whirr.location-id=us-east-1d
>> hadoop-hdfs.dfs.permissions=false
>> hadoop-hdfs.dfs.replication=2
>>
>>
>> --Ben
>>
>>
>>
>>
>> On Mar 17, 2011, at 1:08 PM, Andrei Savu wrote:
>>
>>> Ben, could you give it one more try using the current trunk?
>>>
>>> You can specify the user by setting the option whirr.cluster-user
>>> (defaults to current system user).
>>>
>>> On Wed, Mar 16, 2011 at 11:23 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>> Andrei,
>>>>
>>>> Thanks.
>>>>
>>>> After patching with 158, it launches fine as me on that Ubuntu image from the recipe (i.e. on my client machine I am 'ben', so now the aws user that has sudo, and as whom I can log in is also 'ben'), so that looks good.
>>>>
>>>> But it's now doing this with amazon linux (ami-da0cf8b3, which was the default 64-bit ami a few days ago, and may still be) during launch:
>>>>
>>>> 1) SshException on node us-east-1/i-b2678ddd:
>>>> org.jclouds.ssh.SshException: ben@50.16.96.211:22: Error connecting to session.
>>>> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>>>> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>>>> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>>>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>>>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:45)
>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>
>>>> So it seems as if the key part of jclouds authentication setup is still failing for the amazon linux/ec2-user scenario, i.e. trying to set up as the local user, but failing.
>>>>
>>>> Is there a property for the user it launches as? Or does it just do whichever user you are locally, instead of ec2-user/ubuntu/root, depending on the default, as before?
>>>>
>>>> I can switch to ubuntu, but I have a fair amount of native code setup in my custom scripts and would prefer to stick with a redhattish version if possible.
>>>>
>>>> Looking ahead, I want to benchmark plain old 64-bit instances against cluster instances, to see if the allegedly improved networking gives us a boost, and the available ones I see are Suse and Amazon linux. When I switch to the amazon linux one, like so:
>>>>
>>>> whirr.hardward-id=cc1.4xlarge
>>>> whirr.image-id=us-east-1/ami-321eed5b
>>>>
>>>> I get different a different problem:
>>>>
>>>> Exception in thread "main" java.util.NoSuchElementException: hardwares don't support any images: [biggest=false, fastest=false, imageName=null, imageDescription=Amazon Linux AMI x86_64 HVM EBS EXT4, imageId=us-east-1/ami-321eed5b, imageVersion=ext4, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=unrecognized, osName=null, osDescription=amazon/amzn-hvm-ami-2011.02.1-beta.x86_64-ext4, osVersion=, osArch=hvm, os64Bit=true, hardwareId=m1.small]
>>>> [[id=cc1.4xlarge, providerId=cc1.4xlarge, name=null, processors=[[cores=4.0, speed=4.0], [cores=4.0, speed=4.0]], ram=23552, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=840.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=840.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsI
>>>>
>>>> but I imagine that if using cluster instances is going to be possible, support for amazon linux will be needed.
>>>>
>>>> --Ben
>>>>
>>>>
>>>> On Mar 16, 2011, at 4:07 PM, Andrei Savu wrote:
>>>>
>>>>> I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
>>>>> going to commit WHIRR-158 [1] tomorrow and it should fix the problem
>>>>> you are seeing. We should be able to restart the vote for the 0.4.0
>>>>> release after fixing this issue.
>>>>>
>>>>> [0] https://issues.apache.org/jira/browse/WHIRR-264
>>>>> [1] https://issues.apache.org/jira/browse/WHIRR-158
>>>>>
>>>>> -- Andrei Savu / andreisavu.ro
>>>>>
>>>>> On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
>>>>>>
>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>
>>>>>> and then either this (from the recipe)
>>>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>>>>
>>>>>> or this:
>>>>>> # Amazon linux 64-bit, default as of 3/11:
>>>>>> whirr.image-id=us-east-1/ami-8e1fece7
>>>>>>
>>>>>> I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
>>>>>>
>>>>>>
>>>>>> My whole config file is this:
>>>>>>
>>>>>> whirr.cluster-name=bhcL4
>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
>>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>> whirr.provider=aws-ec2
>>>>>> whirr.identity=...
>>>>>> whirr.credential=...
>>>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
>>>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
>>>>>> whirr.hardware-id=c1.xlarge
>>>>>> #whirr.hardware-id=c1.medium
>>>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>>>> # Amazon linux as of 3/11:
>>>>>> #whirr.image-id=us-east-1/ami-8e1fece7
>>>>>> # If you choose a different location, make sure whirr.image-id is updated too
>>>>>> whirr.location-id=us-east-1d
>>>>>> hadoop-hdfs.dfs.permissions=false
>>>>>> hadoop-hdfs.dfs.replication=2
>>>>>>
>>>>>>
>>>>>>
>>>>>> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
>>>>
>>>>
>>
>>
Re: aws 64-bit c1.xlarge problems
Posted by Andrei Savu <sa...@gmail.com>.
Strange! I will try your properties file tomorrow.
If you want to try again you can find the artifacts for 0.4.0 RC1 here:
http://people.apache.org/~asavu/whirr-0.4.0-incubating-candidate-1
On Thu, Mar 17, 2011 at 8:41 PM, Benjamin Clark <be...@daltonclark.com> wrote:
> Andrei,
>
> Thanks for looking at this. Unfortunately it does not seem to work.
>
> Using the Amazon linux 64-bit ami with no whirr.cluster-user, or if I set it to 'ben' or whatever else, I get this.
>
> 1) SshException on node us-east-1/i-62de280d:
> org.jclouds.ssh.SshException: ec2-user@72.44.35.254:22: Error connecting to session.
> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>
> So it doesn't seem to be honoring that property, and it's definitely not allowing me to log in to any nodes, 'ben', 'ec2-user' or 'root'.
>
> The ubuntu ami from the recipes continues to work fine.
>
> Here's the full config file I'm using. I grabbed the recipe from trunk and put my stuff back in, to make sure I'm not missing a new setting:
>
> whirr.cluster-name=bhcTL
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2 hadoop-datanode+hadoop-tasktracker
> whirr.hadoop-install-function=install_cdh_hadoop
> whirr.hadoop-configure-function=configure_cdh_hadoop
> whirr.provider=aws-ec2
> whirr.identity=${env:AWS_ACCESS_KEY_ID}
> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-hkey
> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-hkey.pub
> whirr.cluster-user=ben
> # Amazon linux 32-bit--works
> #whirr.hardware-id=c1.medium
> #whirr.image-id=us-east-1/ami-d59d6bbc
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/ -- works
> #whirr.hardware-id=c1.xlarge
> #whirr.image-id=us-east-1/ami-da0cf8b3
> # Amazon linux 64-bit as of 3/11:--doesn't work
> whirr.hardware-id=c1.xlarge
> whirr.image-id=us-east-1/ami-8e1fece7
> #Cluster compute --doesn't work
> #whirr.hardward-id=cc1.4xlarge
> #whirr.image-id=us-east-1/ami-321eed5b
> whirr.location-id=us-east-1d
> hadoop-hdfs.dfs.permissions=false
> hadoop-hdfs.dfs.replication=2
>
>
> --Ben
>
>
>
>
> On Mar 17, 2011, at 1:08 PM, Andrei Savu wrote:
>
>> Ben, could you give it one more try using the current trunk?
>>
>> You can specify the user by setting the option whirr.cluster-user
>> (defaults to current system user).
>>
>> On Wed, Mar 16, 2011 at 11:23 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>> Andrei,
>>>
>>> Thanks.
>>>
>>> After patching with 158, it launches fine as me on that Ubuntu image from the recipe (i.e. on my client machine I am 'ben', so now the aws user that has sudo, and as whom I can log in is also 'ben'), so that looks good.
>>>
>>> But it's now doing this with amazon linux (ami-da0cf8b3, which was the default 64-bit ami a few days ago, and may still be) during launch:
>>>
>>> 1) SshException on node us-east-1/i-b2678ddd:
>>> org.jclouds.ssh.SshException: ben@50.16.96.211:22: Error connecting to session.
>>> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>>> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>>> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:45)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>
>>> So it seems as if the key part of jclouds authentication setup is still failing for the amazon linux/ec2-user scenario, i.e. trying to set up as the local user, but failing.
>>>
>>> Is there a property for the user it launches as? Or does it just do whichever user you are locally, instead of ec2-user/ubuntu/root, depending on the default, as before?
>>>
>>> I can switch to ubuntu, but I have a fair amount of native code setup in my custom scripts and would prefer to stick with a redhattish version if possible.
>>>
>>> Looking ahead, I want to benchmark plain old 64-bit instances against cluster instances, to see if the allegedly improved networking gives us a boost, and the available ones I see are Suse and Amazon linux. When I switch to the amazon linux one, like so:
>>>
>>> whirr.hardward-id=cc1.4xlarge
>>> whirr.image-id=us-east-1/ami-321eed5b
>>>
>>> I get different a different problem:
>>>
>>> Exception in thread "main" java.util.NoSuchElementException: hardwares don't support any images: [biggest=false, fastest=false, imageName=null, imageDescription=Amazon Linux AMI x86_64 HVM EBS EXT4, imageId=us-east-1/ami-321eed5b, imageVersion=ext4, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=unrecognized, osName=null, osDescription=amazon/amzn-hvm-ami-2011.02.1-beta.x86_64-ext4, osVersion=, osArch=hvm, os64Bit=true, hardwareId=m1.small]
>>> [[id=cc1.4xlarge, providerId=cc1.4xlarge, name=null, processors=[[cores=4.0, speed=4.0], [cores=4.0, speed=4.0]], ram=23552, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=840.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=840.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsI
>>>
>>> but I imagine that if using cluster instances is going to be possible, support for amazon linux will be needed.
>>>
>>> --Ben
>>>
>>>
>>> On Mar 16, 2011, at 4:07 PM, Andrei Savu wrote:
>>>
>>>> I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
>>>> going to commit WHIRR-158 [1] tomorrow and it should fix the problem
>>>> you are seeing. We should be able to restart the vote for the 0.4.0
>>>> release after fixing this issue.
>>>>
>>>> [0] https://issues.apache.org/jira/browse/WHIRR-264
>>>> [1] https://issues.apache.org/jira/browse/WHIRR-158
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>> On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
>>>>>
>>>>> whirr.hardware-id=c1.xlarge
>>>>>
>>>>> and then either this (from the recipe)
>>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>>>
>>>>> or this:
>>>>> # Amazon linux 64-bit, default as of 3/11:
>>>>> whirr.image-id=us-east-1/ami-8e1fece7
>>>>>
>>>>> I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
>>>>>
>>>>>
>>>>> My whole config file is this:
>>>>>
>>>>> whirr.cluster-name=bhcL4
>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>> whirr.provider=aws-ec2
>>>>> whirr.identity=...
>>>>> whirr.credential=...
>>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
>>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
>>>>> whirr.hardware-id=c1.xlarge
>>>>> #whirr.hardware-id=c1.medium
>>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>>> # Amazon linux as of 3/11:
>>>>> #whirr.image-id=us-east-1/ami-8e1fece7
>>>>> # If you choose a different location, make sure whirr.image-id is updated too
>>>>> whirr.location-id=us-east-1d
>>>>> hadoop-hdfs.dfs.permissions=false
>>>>> hadoop-hdfs.dfs.replication=2
>>>>>
>>>>>
>>>>>
>>>>> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
>>>
>>>
>
>
Re: aws 64-bit c1.xlarge problems
Posted by Benjamin Clark <be...@daltonclark.com>.
Andrei,
Thanks for looking at this. Unfortunately it does not seem to work.
Using the Amazon linux 64-bit ami with no whirr.cluster-user, or if I set it to 'ben' or whatever else, I get this.
1) SshException on node us-east-1/i-62de280d:
org.jclouds.ssh.SshException: ec2-user@72.44.35.254:22: Error connecting to session.
at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
So it doesn't seem to be honoring that property, and it's definitely not allowing me to log in to any nodes, 'ben', 'ec2-user' or 'root'.
The ubuntu ami from the recipes continues to work fine.
Here's the full config file I'm using. I grabbed the recipe from trunk and put my stuff back in, to make sure I'm not missing a new setting:
whirr.cluster-name=bhcTL
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2 hadoop-datanode+hadoop-tasktracker
whirr.hadoop-install-function=install_cdh_hadoop
whirr.hadoop-configure-function=configure_cdh_hadoop
whirr.provider=aws-ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-hkey
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-hkey.pub
whirr.cluster-user=ben
# Amazon linux 32-bit--works
#whirr.hardware-id=c1.medium
#whirr.image-id=us-east-1/ami-d59d6bbc
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/ -- works
#whirr.hardware-id=c1.xlarge
#whirr.image-id=us-east-1/ami-da0cf8b3
# Amazon linux 64-bit as of 3/11:--doesn't work
whirr.hardware-id=c1.xlarge
whirr.image-id=us-east-1/ami-8e1fece7
#Cluster compute --doesn't work
#whirr.hardward-id=cc1.4xlarge
#whirr.image-id=us-east-1/ami-321eed5b
whirr.location-id=us-east-1d
hadoop-hdfs.dfs.permissions=false
hadoop-hdfs.dfs.replication=2
--Ben
On Mar 17, 2011, at 1:08 PM, Andrei Savu wrote:
> Ben, could you give it one more try using the current trunk?
>
> You can specify the user by setting the option whirr.cluster-user
> (defaults to current system user).
>
> On Wed, Mar 16, 2011 at 11:23 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>> Andrei,
>>
>> Thanks.
>>
>> After patching with 158, it launches fine as me on that Ubuntu image from the recipe (i.e. on my client machine I am 'ben', so now the aws user that has sudo, and as whom I can log in is also 'ben'), so that looks good.
>>
>> But it's now doing this with amazon linux (ami-da0cf8b3, which was the default 64-bit ami a few days ago, and may still be) during launch:
>>
>> 1) SshException on node us-east-1/i-b2678ddd:
>> org.jclouds.ssh.SshException: ben@50.16.96.211:22: Error connecting to session.
>> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
>> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
>> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
>> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:45)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>
>> So it seems as if the key part of jclouds authentication setup is still failing for the amazon linux/ec2-user scenario, i.e. trying to set up as the local user, but failing.
>>
>> Is there a property for the user it launches as? Or does it just do whichever user you are locally, instead of ec2-user/ubuntu/root, depending on the default, as before?
>>
>> I can switch to ubuntu, but I have a fair amount of native code setup in my custom scripts and would prefer to stick with a redhattish version if possible.
>>
>> Looking ahead, I want to benchmark plain old 64-bit instances against cluster instances, to see if the allegedly improved networking gives us a boost, and the available ones I see are Suse and Amazon linux. When I switch to the amazon linux one, like so:
>>
>> whirr.hardward-id=cc1.4xlarge
>> whirr.image-id=us-east-1/ami-321eed5b
>>
>> I get different a different problem:
>>
>> Exception in thread "main" java.util.NoSuchElementException: hardwares don't support any images: [biggest=false, fastest=false, imageName=null, imageDescription=Amazon Linux AMI x86_64 HVM EBS EXT4, imageId=us-east-1/ami-321eed5b, imageVersion=ext4, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=unrecognized, osName=null, osDescription=amazon/amzn-hvm-ami-2011.02.1-beta.x86_64-ext4, osVersion=, osArch=hvm, os64Bit=true, hardwareId=m1.small]
>> [[id=cc1.4xlarge, providerId=cc1.4xlarge, name=null, processors=[[cores=4.0, speed=4.0], [cores=4.0, speed=4.0]], ram=23552, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=840.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=840.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsI
>>
>> but I imagine that if using cluster instances is going to be possible, support for amazon linux will be needed.
>>
>> --Ben
>>
>>
>> On Mar 16, 2011, at 4:07 PM, Andrei Savu wrote:
>>
>>> I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
>>> going to commit WHIRR-158 [1] tomorrow and it should fix the problem
>>> you are seeing. We should be able to restart the vote for the 0.4.0
>>> release after fixing this issue.
>>>
>>> [0] https://issues.apache.org/jira/browse/WHIRR-264
>>> [1] https://issues.apache.org/jira/browse/WHIRR-158
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>> On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
>>>>
>>>> whirr.hardware-id=c1.xlarge
>>>>
>>>> and then either this (from the recipe)
>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>>
>>>> or this:
>>>> # Amazon linux 64-bit, default as of 3/11:
>>>> whirr.image-id=us-east-1/ami-8e1fece7
>>>>
>>>> I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
>>>>
>>>>
>>>> My whole config file is this:
>>>>
>>>> whirr.cluster-name=bhcL4
>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>> whirr.provider=aws-ec2
>>>> whirr.identity=...
>>>> whirr.credential=...
>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
>>>> whirr.hardware-id=c1.xlarge
>>>> #whirr.hardware-id=c1.medium
>>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>> # Amazon linux as of 3/11:
>>>> #whirr.image-id=us-east-1/ami-8e1fece7
>>>> # If you choose a different location, make sure whirr.image-id is updated too
>>>> whirr.location-id=us-east-1d
>>>> hadoop-hdfs.dfs.permissions=false
>>>> hadoop-hdfs.dfs.replication=2
>>>>
>>>>
>>>>
>>>> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
>>
>>
Re: aws 64-bit c1.xlarge problems
Posted by Andrei Savu <sa...@gmail.com>.
Ben, could you give it one more try using the current trunk?
You can specify the user by setting the option whirr.cluster-user
(defaults to current system user).
On Wed, Mar 16, 2011 at 11:23 PM, Benjamin Clark <be...@daltonclark.com> wrote:
> Andrei,
>
> Thanks.
>
> After patching with 158, it launches fine as me on that Ubuntu image from the recipe (i.e. on my client machine I am 'ben', so now the aws user that has sudo, and as whom I can log in is also 'ben'), so that looks good.
>
> But it's now doing this with amazon linux (ami-da0cf8b3, which was the default 64-bit ami a few days ago, and may still be) during launch:
>
> 1) SshException on node us-east-1/i-b2678ddd:
> org.jclouds.ssh.SshException: ben@50.16.96.211:22: Error connecting to session.
> at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
> at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
> at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
> at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:45)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> So it seems as if the key part of jclouds authentication setup is still failing for the amazon linux/ec2-user scenario, i.e. trying to set up as the local user, but failing.
>
> Is there a property for the user it launches as? Or does it just do whichever user you are locally, instead of ec2-user/ubuntu/root, depending on the default, as before?
>
> I can switch to ubuntu, but I have a fair amount of native code setup in my custom scripts and would prefer to stick with a redhattish version if possible.
>
> Looking ahead, I want to benchmark plain old 64-bit instances against cluster instances, to see if the allegedly improved networking gives us a boost, and the available ones I see are Suse and Amazon linux. When I switch to the amazon linux one, like so:
>
> whirr.hardward-id=cc1.4xlarge
> whirr.image-id=us-east-1/ami-321eed5b
>
> I get different a different problem:
>
> Exception in thread "main" java.util.NoSuchElementException: hardwares don't support any images: [biggest=false, fastest=false, imageName=null, imageDescription=Amazon Linux AMI x86_64 HVM EBS EXT4, imageId=us-east-1/ami-321eed5b, imageVersion=ext4, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=unrecognized, osName=null, osDescription=amazon/amzn-hvm-ami-2011.02.1-beta.x86_64-ext4, osVersion=, osArch=hvm, os64Bit=true, hardwareId=m1.small]
> [[id=cc1.4xlarge, providerId=cc1.4xlarge, name=null, processors=[[cores=4.0, speed=4.0], [cores=4.0, speed=4.0]], ram=23552, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=840.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=840.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsI
>
> but I imagine that if using cluster instances is going to be possible, support for amazon linux will be needed.
>
> --Ben
>
>
> On Mar 16, 2011, at 4:07 PM, Andrei Savu wrote:
>
>> I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
>> going to commit WHIRR-158 [1] tomorrow and it should fix the problem
>> you are seeing. We should be able to restart the vote for the 0.4.0
>> release after fixing this issue.
>>
>> [0] https://issues.apache.org/jira/browse/WHIRR-264
>> [1] https://issues.apache.org/jira/browse/WHIRR-158
>>
>> -- Andrei Savu / andreisavu.ro
>>
>> On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
>>>
>>> whirr.hardware-id=c1.xlarge
>>>
>>> and then either this (from the recipe)
>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>
>>> or this:
>>> # Amazon linux 64-bit, default as of 3/11:
>>> whirr.image-id=us-east-1/ami-8e1fece7
>>>
>>> I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
>>>
>>>
>>> My whole config file is this:
>>>
>>> whirr.cluster-name=bhcL4
>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
>>> whirr.hadoop-install-function=install_cdh_hadoop
>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>> whirr.provider=aws-ec2
>>> whirr.identity=...
>>> whirr.credential=...
>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
>>> whirr.hardware-id=c1.xlarge
>>> #whirr.hardware-id=c1.medium
>>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>> # Amazon linux as of 3/11:
>>> #whirr.image-id=us-east-1/ami-8e1fece7
>>> # If you choose a different location, make sure whirr.image-id is updated too
>>> whirr.location-id=us-east-1d
>>> hadoop-hdfs.dfs.permissions=false
>>> hadoop-hdfs.dfs.replication=2
>>>
>>>
>>>
>>> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
>
>
Re: aws 64-bit c1.xlarge problems
Posted by Benjamin Clark <be...@daltonclark.com>.
Andrei,
Thanks.
After patching with 158, it launches fine as me on that Ubuntu image from the recipe (i.e. on my client machine I am 'ben', so now the aws user that has sudo, and as whom I can log in is also 'ben'), so that looks good.
But it's now doing this with amazon linux (ami-da0cf8b3, which was the default 64-bit ami a few days ago, and may still be) during launch:
1) SshException on node us-east-1/i-b2678ddd:
org.jclouds.ssh.SshException: ben@50.16.96.211:22: Error connecting to session.
at org.jclouds.ssh.jsch.JschSshClient.propagate(JschSshClient.java:252)
at org.jclouds.ssh.jsch.JschSshClient.connect(JschSshClient.java:206)
at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:90)
at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:70)
at org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:45)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
So it seems as if the key part of jclouds authentication setup is still failing for the amazon linux/ec2-user scenario, i.e. trying to set up as the local user, but failing.
Is there a property for the user it launches as? Or does it just do whichever user you are locally, instead of ec2-user/ubuntu/root, depending on the default, as before?
I can switch to ubuntu, but I have a fair amount of native code setup in my custom scripts and would prefer to stick with a redhattish version if possible.
Looking ahead, I want to benchmark plain old 64-bit instances against cluster instances, to see if the allegedly improved networking gives us a boost, and the available ones I see are Suse and Amazon linux. When I switch to the amazon linux one, like so:
whirr.hardward-id=cc1.4xlarge
whirr.image-id=us-east-1/ami-321eed5b
I get different a different problem:
Exception in thread "main" java.util.NoSuchElementException: hardwares don't support any images: [biggest=false, fastest=false, imageName=null, imageDescription=Amazon Linux AMI x86_64 HVM EBS EXT4, imageId=us-east-1/ami-321eed5b, imageVersion=ext4, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=unrecognized, osName=null, osDescription=amazon/amzn-hvm-ami-2011.02.1-beta.x86_64-ext4, osVersion=, osArch=hvm, os64Bit=true, hardwareId=m1.small]
[[id=cc1.4xlarge, providerId=cc1.4xlarge, name=null, processors=[[cores=4.0, speed=4.0], [cores=4.0, speed=4.0]], ram=23552, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=840.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=840.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsI
but I imagine that if using cluster instances is going to be possible, support for amazon linux will be needed.
--Ben
On Mar 16, 2011, at 4:07 PM, Andrei Savu wrote:
> I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
> going to commit WHIRR-158 [1] tomorrow and it should fix the problem
> you are seeing. We should be able to restart the vote for the 0.4.0
> release after fixing this issue.
>
> [0] https://issues.apache.org/jira/browse/WHIRR-264
> [1] https://issues.apache.org/jira/browse/WHIRR-158
>
> -- Andrei Savu / andreisavu.ro
>
> On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
>>
>> whirr.hardware-id=c1.xlarge
>>
>> and then either this (from the recipe)
>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>> whirr.image-id=us-east-1/ami-da0cf8b3
>>
>> or this:
>> # Amazon linux 64-bit, default as of 3/11:
>> whirr.image-id=us-east-1/ami-8e1fece7
>>
>> I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
>>
>>
>> My whole config file is this:
>>
>> whirr.cluster-name=bhcL4
>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
>> whirr.hadoop-install-function=install_cdh_hadoop
>> whirr.hadoop-configure-function=configure_cdh_hadoop
>> whirr.provider=aws-ec2
>> whirr.identity=...
>> whirr.credential=...
>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
>> whirr.hardware-id=c1.xlarge
>> #whirr.hardware-id=c1.medium
>> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
>> whirr.image-id=us-east-1/ami-da0cf8b3
>> # Amazon linux as of 3/11:
>> #whirr.image-id=us-east-1/ami-8e1fece7
>> # If you choose a different location, make sure whirr.image-id is updated too
>> whirr.location-id=us-east-1d
>> hadoop-hdfs.dfs.permissions=false
>> hadoop-hdfs.dfs.replication=2
>>
>>
>>
>> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
Re: aws 64-bit c1.xlarge problems
Posted by Andrei Savu <sa...@gmail.com>.
I've seen something similar while testing Whirr: WHIRR-264 [0]. We are
going to commit WHIRR-158 [1] tomorrow and it should fix the problem
you are seeing. We should be able to restart the vote for the 0.4.0
release after fixing this issue.
[0] https://issues.apache.org/jira/browse/WHIRR-264
[1] https://issues.apache.org/jira/browse/WHIRR-158
-- Andrei Savu / andreisavu.ro
On Wed, Mar 16, 2011 at 6:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
> I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
>
> whirr.hardware-id=c1.xlarge
>
> and then either this (from the recipe)
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
>
> or this:
> # Amazon linux 64-bit, default as of 3/11:
> whirr.image-id=us-east-1/ami-8e1fece7
>
> I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
>
>
> My whole config file is this:
>
> whirr.cluster-name=bhcL4
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
> whirr.hadoop-install-function=install_cdh_hadoop
> whirr.hadoop-configure-function=configure_cdh_hadoop
> whirr.provider=aws-ec2
> whirr.identity=...
> whirr.credential=...
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
> whirr.hardware-id=c1.xlarge
> #whirr.hardware-id=c1.medium
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # Amazon linux as of 3/11:
> #whirr.image-id=us-east-1/ami-8e1fece7
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1d
> hadoop-hdfs.dfs.permissions=false
> hadoop-hdfs.dfs.replication=2
>
>
>
> Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
aws 64-bit c1.xlarge problems
Posted by Benjamin Clark <be...@daltonclark.com>.
I have been using whirr 0.4 branch to launch clusters of c1.medium amazon linux machines (whirr.image-id=us-east-1/ami-d59d6bbc, which was the default for new amazon linux instances, a few days ago) with good success. I took the default hadoop-ec2.properties recipe and modified it slightly to suit my needs. I'm now trying with basically the same properties file, but when I use
whirr.hardware-id=c1.xlarge
and then either this (from the recipe)
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/
whirr.image-id=us-east-1/ami-da0cf8b3
or this:
# Amazon linux 64-bit, default as of 3/11:
whirr.image-id=us-east-1/ami-8e1fece7
I get a a failure to install the right public key, so that I can't log into the name node (or any other nodes, for that matter).
My whole config file is this:
whirr.cluster-name=bhcL4
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,4 hadoop-datanode+hadoop-tasktracker
whirr.hadoop-install-function=install_cdh_hadoop
whirr.hadoop-configure-function=configure_cdh_hadoop
whirr.provider=aws-ec2
whirr.identity=...
whirr.credential=...
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa-formyhadoop.pub
whirr.hardware-id=c1.xlarge
#whirr.hardware-id=c1.medium
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/
whirr.image-id=us-east-1/ami-da0cf8b3
# Amazon linux as of 3/11:
#whirr.image-id=us-east-1/ami-8e1fece7
# If you choose a different location, make sure whirr.image-id is updated too
whirr.location-id=us-east-1d
hadoop-hdfs.dfs.permissions=false
hadoop-hdfs.dfs.replication=2
Am I doing something wrong here? I tried with whirr.location-id=us-east-1d and whirr.location-id=us-east-1
OutOfMemoryError, mapred-site option
Posted by Sebastian Schoenherr <se...@uibk.ac.at>.
Hi guys,
I tried to start a 64 bit image with WHIRR (branch 0.4.0 or trunk) on
EC2 and get a OutOfMemoryError in the child processes. It reminds me of
this: https://issues.apache.org/jira/browse/WHIRR-146. I get the same
distcp behavior;
After setting the mapred.child.ulimit via the property file to null
(hadoop-mapreduce.mapred.child.ulimit=null) the distcp runs as it
should. Maybe the option is set on a too small value or it is not
necessary as the patch of 146 suggest.
Can anyone else reproduce this error or is it a problem of my custom image?
Cheers
Sebastian
Re: hadoop config property override problem
Posted by Benjamin Clark <be...@daltonclark.com>.
Andrei
I actually found the time to test this sooner than I thought--it works! Thanks a million.
I tested this on branch 0.4 and I patched my trunk but haven't actually tried it on trunk
--Ben
On Mar 12, 2011, at 10:52 AM, Andrei Savu wrote:
> I've created a patch [1] for this. Ben, I believe it should work for
> you. Tom, let me know what you think about this approach. Unit and
> integration tests are passing for me.
>
> [1] https://issues.apache.org/jira/browse/WHIRR-259
>
> On Sat, Mar 12, 2011 at 2:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>> Just tried it. No, that doesn't change the result.
>>
>>
>> On Mar 12, 2011, at 1:21 AM, Tom White wrote:
>>
>>> Does escaping the comma with a backslash (i.e. \,) work? (See
>>> http://commons.apache.org/configuration/howto_properties.html)
>>>
>>> Tom
>>>
>>> On Fri, Mar 11, 2011 at 7:56 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>> Yes, thanks Tom, that works.
>>>>
>>>> Many features and design changes in this branch are much appreciated, especially the config property override in the whirr config file.
>>>>
>>>> I found one problem, at least in the 0.4 branch. If you override a property with a comma-separated list, for example:
>>>>
>>>> hadoop-common.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
>>>>
>>>> then what actually shows up in core-site.xml is surrounded by brackets and has spaces between the elements of the list, so
>>>>
>>>> [org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec]
>>>>
>>>> Hadoop is not chomping or trimming that kind of thing, so you need to remove the spaces and brackets to get that to work. It's easy enough to patch the file, deploy to the slaves and restart, but I'm wondering if that's accounted for anywhere. I scanned CHANGES.txt in trunk and I don't see it.
>>>>
>>>> --Ben
>>>>
>>>>
>>>> On Mar 10, 2011, at 5:41 PM, Tom White wrote:
>>>>
>>>>> On Thu, Mar 10, 2011 at 1:19 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>> Thank you both, Tom and Andrei. Now that I know where the faq is, I hope to bother you less with things that are documented!
>>>>>>
>>>>>> I think I should be all set with customization, but I need to build. BUILD.txt says 'mvn clean install' or mvn package -Ppackage. I can do that, and mvn reports success but then I try to use whirr-cli-0.4.0-incubating.jar as the jar, and the manifest has no main class (OK, I can supply 'org.apache.whirr.cli.Main' if I need to), and in any case the jar is not a fat jar, as I see all the publicly distributed versions are. It looks in the poms as if you have the maven assembly plugins trying to make a fat jar, but it doesn't seem to be doing it for me.
>>>>>
>>>>> Whirr no longer produces a shaded (fat) JAR as of 0.4.0 and trunk, so
>>>>> perhaps it is working. Try bin/whirr and it should list the roles for
>>>>> you.
>>>>>
>>>>> Cheers
>>>>> Tom
>>>>>
>>>>>>
>>>>>> What am I doing wrong?
>>>>>>
>>>>>> I'm doing this on a Mac like so:
>>>>>>
>>>>>> $ ruby --version
>>>>>> ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10]
>>>>>> $ mvn --version
>>>>>> Apache Maven 3.0.2 (r1056850; 2011-01-08 19:58:10-0500)
>>>>>> Java version: 1.6.0_24, vendor: Apple Inc.
>>>>>> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>>>>>> Default locale: en_US, platform encoding: MacRoman
>>>>>> OS name: "mac os x", version: "10.6.6", arch: "x86_64", family: "mac"
>>>>>> $ java -version
>>>>>> java version "1.6.0_24"
>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now I think my only problem is that
>>>>>> On Mar 10, 2011, at 2:35 PM, Andrei Savu wrote:
>>>>>>
>>>>>>> Starting with the upcoming 0.4.0 release Whirr is no longer using S3
>>>>>>> for storing the install and configure scripts. You can grab the
>>>>>>> scripts from:
>>>>>>>
>>>>>>> ${WHIRR_HOME}/services/${SERVICE_NAME}/src/main/resources/functions/{install,configure}_SERVICE.sh
>>>>>>>
>>>>>>> It's also easier to customize the scripts. You just need to place your
>>>>>>> version in ${WHIRR_HOME}/functions (I believe it should have the same
>>>>>>> name).
>>>>>>>
>>>>>>> From the 0.4.0 FAQ: "If you want to change the scripts then you can
>>>>>>> place a modified copy of the
>>>>>>> scripts in a _functions_ directory in Whirr's installation directory. The
>>>>>>> original versions of the scripts can be found in _functions_ directories in the
>>>>>>> source trees."
>>>>>>>
>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>
>>>>>>> On Thu, Mar 10, 2011 at 9:27 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>>>> So if we grab install_cdh_hadoop.sh from the source tree, and put a customized version in our own bucket, and set whirr.run-url-base to the root of that bucket, it should work, even in 4.0 and after?
>>>>>>>>
>>>>>>>> Based on the FAQ I tried a few of these to attempt to verify I was on the right track:
>>>>>>>>
>>>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop
>>>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop.sh
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop.sh
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop.sh
>>>>>>>>
>>>>>>>> but all give 404s.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mar 10, 2011, at 12:01 AM, Tom White wrote:
>>>>>>>>
>>>>>>>>> Sorry I missed this thread. On 0.4.0 and later
>>>>>>>>>
>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>
>>>>>>>>> changes to
>>>>>>>>>
>>>>>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Tom
>>>>>>>>>
>>>>>>>>> On Wed, Mar 9, 2011 at 8:23 AM, Sebastian Schoenherr
>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>> I tried to execute my working whirr 0.3.0 configuration (identical to your
>>>>>>>>>> property file, using cloudera scripts) on branch-0.4 and the same issues
>>>>>>>>>> arised for me. Unfortunately I'm not sure yet why it's not working with
>>>>>>>>>> branch-0.4. Is using branch-0.3 an option for you?
>>>>>>>>>> Any other guesses?
>>>>>>>>>> cheers,
>>>>>>>>>> sebastian
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08.03.2011 05:41, Saptarshi Guha wrote:
>>>>>>>>>>>
>>>>>>>>>>> once again, i've changed the secret identity ..
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 7, 2011 at 8:41 PM, Saptarshi Guha
>>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello
>>>>>>>>>>>>
>>>>>>>>>>>> No such luck on my end This is my script file, you can test that the
>>>>>>>>>>>> scripts download. But when I log in
>>>>>>>>>>>> hadoop version is. (I pulled the latest git). Also my scripts (you can
>>>>>>>>>>>> confirm if you download) echo a small line to files in /tmp.
>>>>>>>>>>>> They are not being created.
>>>>>>>>>>>>
>>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>>> Subversion
>>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>>> -r 911707
>>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>>
>>>>>>>>>>>> whirr.cluster-name=revotesting2
>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>> whirr.identity= AKIAJH5JBSI5KJ7YZQ6A
>>>>>>>>>>>> whirr.credential= b/kqLJAHOdRA4L30n7Zt8Edz383B1ARtPI3wiyD6
>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>> whirr.run-url-base=http://ml.stat.purdue.edu/whirr-scripts/
>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>
>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>
>>>>>>>>>>>> ##http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:59 AM, Benjamin Clark<be...@daltonclark.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> In my experience you need
>>>>>>>>>>>>>
>>>>>>>>>>>>> whirr.run-url-base=http://name-of-my-bucket-with-customized-scripts/
>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>>
>>>>>>>>>>>>> and then you *also* need to have a copy of sun/java/install in that same
>>>>>>>>>>>>> bucket.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And both of those scripts need to be public-readable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So in the end you should be able to do
>>>>>>>>>>>>> curl
>>>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/cloudera/cdh/install
>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>> curl
>>>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/sun/java/install
>>>>>>>>>>>>>
>>>>>>>>>>>>> Even if you haven't customizied sun/java/install, it needs to be there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you do all that, the scripts will run and you will have the versions
>>>>>>>>>>>>> you asked for.
>>>>>>>>>>>>>
>>>>>>>>>>>>> `hadoop version` on the name node then says, in my case:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hadoop 0.20.2-CDH3B4
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mar 7, 2011, at 12:41 PM, Saptarshi Guha wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> Fixed the security slip up. Did the hadoop version thing and got this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@domU-12-31-39-0B-CC-41 ~]# hadoop version
>>>>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>>>>> Subversion
>>>>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>>>>> -r 911707
>>>>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So i guess its not CDH.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:22 AM, Saptarshi Guha
>>>>>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dear me! thanks, will do right away.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Mar 7, 2011 at 1:46 AM, Sebastian Schoenherr
>>>>>>>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>>>>>>>> Try to execute "hadoop version" on your namenode, if the output is
>>>>>>>>>>>>>>>> Hadoop
>>>>>>>>>>>>>>>> 0.20.2-CDH3B4, the current cloudera distribution has been installed.
>>>>>>>>>>>>>>>> Btw, I would recommend to set your current Access Key ID and Secret
>>>>>>>>>>>>>>>> Key
>>>>>>>>>>>>>>>> inactive, since you posted it in your prop file.
>>>>>>>>>>>>>>>> cheers
>>>>>>>>>>>>>>>> sebastian
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 06.03.2011 06:54, Saptarshi Guha wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I did git clone of the latest whirr and copied cloudera scripts into
>>>>>>>>>>>>>>>>> the script directory (copied over
>>>>>>>>>>>>>>>>> from whirr-0.3-incubating).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My properties file is at the end of this email.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, I don't think the scripts are being run because the
>>>>>>>>>>>>>>>>> jobtracker
>>>>>>>>>>>>>>>>> is the default Apache hadoop jobtracker and not the cloudera
>>>>>>>>>>>>>>>>> jobtracker.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Have i missed something?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ## Properties
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> whirr.cluster-name=revotesting
>>>>>>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>>>>>>> whirr.identity= AKIAI3FUFFXAPYLE7CJA
>>>>>>>>>>>>>>>>> whirr.credential= 2Yq3Ar2HSxK/hbwZHs6aN6yrh0yfGNSPTpVw3t2n
>>>>>>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: hadoop config property override problem
Posted by Benjamin Clark <be...@daltonclark.com>.
Wow, that was quick, thanks.
I won't be able to check this out till a bit later in the weekend, but I'll report back as soon as I can.
--Ben
On Mar 12, 2011, at 10:52 AM, Andrei Savu wrote:
> I've created a patch [1] for this. Ben, I believe it should work for
> you. Tom, let me know what you think about this approach. Unit and
> integration tests are passing for me.
>
> [1] https://issues.apache.org/jira/browse/WHIRR-259
>
> On Sat, Mar 12, 2011 at 2:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>> Just tried it. No, that doesn't change the result.
>>
>>
>> On Mar 12, 2011, at 1:21 AM, Tom White wrote:
>>
>>> Does escaping the comma with a backslash (i.e. \,) work? (See
>>> http://commons.apache.org/configuration/howto_properties.html)
>>>
>>> Tom
>>>
>>> On Fri, Mar 11, 2011 at 7:56 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>> Yes, thanks Tom, that works.
>>>>
>>>> Many features and design changes in this branch are much appreciated, especially the config property override in the whirr config file.
>>>>
>>>> I found one problem, at least in the 0.4 branch. If you override a property with a comma-separated list, for example:
>>>>
>>>> hadoop-common.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
>>>>
>>>> then what actually shows up in core-site.xml is surrounded by brackets and has spaces between the elements of the list, so
>>>>
>>>> [org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec]
>>>>
>>>> Hadoop is not chomping or trimming that kind of thing, so you need to remove the spaces and brackets to get that to work. It's easy enough to patch the file, deploy to the slaves and restart, but I'm wondering if that's accounted for anywhere. I scanned CHANGES.txt in trunk and I don't see it.
>>>>
>>>> --Ben
>>>>
>>>>
>>>> On Mar 10, 2011, at 5:41 PM, Tom White wrote:
>>>>
>>>>> On Thu, Mar 10, 2011 at 1:19 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>> Thank you both, Tom and Andrei. Now that I know where the faq is, I hope to bother you less with things that are documented!
>>>>>>
>>>>>> I think I should be all set with customization, but I need to build. BUILD.txt says 'mvn clean install' or mvn package -Ppackage. I can do that, and mvn reports success but then I try to use whirr-cli-0.4.0-incubating.jar as the jar, and the manifest has no main class (OK, I can supply 'org.apache.whirr.cli.Main' if I need to), and in any case the jar is not a fat jar, as I see all the publicly distributed versions are. It looks in the poms as if you have the maven assembly plugins trying to make a fat jar, but it doesn't seem to be doing it for me.
>>>>>
>>>>> Whirr no longer produces a shaded (fat) JAR as of 0.4.0 and trunk, so
>>>>> perhaps it is working. Try bin/whirr and it should list the roles for
>>>>> you.
>>>>>
>>>>> Cheers
>>>>> Tom
>>>>>
>>>>>>
>>>>>> What am I doing wrong?
>>>>>>
>>>>>> I'm doing this on a Mac like so:
>>>>>>
>>>>>> $ ruby --version
>>>>>> ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10]
>>>>>> $ mvn --version
>>>>>> Apache Maven 3.0.2 (r1056850; 2011-01-08 19:58:10-0500)
>>>>>> Java version: 1.6.0_24, vendor: Apple Inc.
>>>>>> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>>>>>> Default locale: en_US, platform encoding: MacRoman
>>>>>> OS name: "mac os x", version: "10.6.6", arch: "x86_64", family: "mac"
>>>>>> $ java -version
>>>>>> java version "1.6.0_24"
>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now I think my only problem is that
>>>>>> On Mar 10, 2011, at 2:35 PM, Andrei Savu wrote:
>>>>>>
>>>>>>> Starting with the upcoming 0.4.0 release Whirr is no longer using S3
>>>>>>> for storing the install and configure scripts. You can grab the
>>>>>>> scripts from:
>>>>>>>
>>>>>>> ${WHIRR_HOME}/services/${SERVICE_NAME}/src/main/resources/functions/{install,configure}_SERVICE.sh
>>>>>>>
>>>>>>> It's also easier to customize the scripts. You just need to place your
>>>>>>> version in ${WHIRR_HOME}/functions (I believe it should have the same
>>>>>>> name).
>>>>>>>
>>>>>>> From the 0.4.0 FAQ: "If you want to change the scripts then you can
>>>>>>> place a modified copy of the
>>>>>>> scripts in a _functions_ directory in Whirr's installation directory. The
>>>>>>> original versions of the scripts can be found in _functions_ directories in the
>>>>>>> source trees."
>>>>>>>
>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>
>>>>>>> On Thu, Mar 10, 2011 at 9:27 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>>>> So if we grab install_cdh_hadoop.sh from the source tree, and put a customized version in our own bucket, and set whirr.run-url-base to the root of that bucket, it should work, even in 4.0 and after?
>>>>>>>>
>>>>>>>> Based on the FAQ I tried a few of these to attempt to verify I was on the right track:
>>>>>>>>
>>>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop
>>>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop.sh
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop.sh
>>>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop.sh
>>>>>>>>
>>>>>>>> but all give 404s.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mar 10, 2011, at 12:01 AM, Tom White wrote:
>>>>>>>>
>>>>>>>>> Sorry I missed this thread. On 0.4.0 and later
>>>>>>>>>
>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>
>>>>>>>>> changes to
>>>>>>>>>
>>>>>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Tom
>>>>>>>>>
>>>>>>>>> On Wed, Mar 9, 2011 at 8:23 AM, Sebastian Schoenherr
>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>> I tried to execute my working whirr 0.3.0 configuration (identical to your
>>>>>>>>>> property file, using cloudera scripts) on branch-0.4 and the same issues
>>>>>>>>>> arised for me. Unfortunately I'm not sure yet why it's not working with
>>>>>>>>>> branch-0.4. Is using branch-0.3 an option for you?
>>>>>>>>>> Any other guesses?
>>>>>>>>>> cheers,
>>>>>>>>>> sebastian
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08.03.2011 05:41, Saptarshi Guha wrote:
>>>>>>>>>>>
>>>>>>>>>>> once again, i've changed the secret identity ..
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 7, 2011 at 8:41 PM, Saptarshi Guha
>>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello
>>>>>>>>>>>>
>>>>>>>>>>>> No such luck on my end This is my script file, you can test that the
>>>>>>>>>>>> scripts download. But when I log in
>>>>>>>>>>>> hadoop version is. (I pulled the latest git). Also my scripts (you can
>>>>>>>>>>>> confirm if you download) echo a small line to files in /tmp.
>>>>>>>>>>>> They are not being created.
>>>>>>>>>>>>
>>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>>> Subversion
>>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>>> -r 911707
>>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>>
>>>>>>>>>>>> whirr.cluster-name=revotesting2
>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>> whirr.identity= AKIAJH5JBSI5KJ7YZQ6A
>>>>>>>>>>>> whirr.credential= b/kqLJAHOdRA4L30n7Zt8Edz383B1ARtPI3wiyD6
>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>> whirr.run-url-base=http://ml.stat.purdue.edu/whirr-scripts/
>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>
>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>
>>>>>>>>>>>> ##http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:59 AM, Benjamin Clark<be...@daltonclark.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> In my experience you need
>>>>>>>>>>>>>
>>>>>>>>>>>>> whirr.run-url-base=http://name-of-my-bucket-with-customized-scripts/
>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>>
>>>>>>>>>>>>> and then you *also* need to have a copy of sun/java/install in that same
>>>>>>>>>>>>> bucket.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And both of those scripts need to be public-readable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So in the end you should be able to do
>>>>>>>>>>>>> curl
>>>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/cloudera/cdh/install
>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>> curl
>>>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/sun/java/install
>>>>>>>>>>>>>
>>>>>>>>>>>>> Even if you haven't customizied sun/java/install, it needs to be there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you do all that, the scripts will run and you will have the versions
>>>>>>>>>>>>> you asked for.
>>>>>>>>>>>>>
>>>>>>>>>>>>> `hadoop version` on the name node then says, in my case:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hadoop 0.20.2-CDH3B4
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mar 7, 2011, at 12:41 PM, Saptarshi Guha wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> Fixed the security slip up. Did the hadoop version thing and got this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@domU-12-31-39-0B-CC-41 ~]# hadoop version
>>>>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>>>>> Subversion
>>>>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>>>>> -r 911707
>>>>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So i guess its not CDH.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:22 AM, Saptarshi Guha
>>>>>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dear me! thanks, will do right away.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Mar 7, 2011 at 1:46 AM, Sebastian Schoenherr
>>>>>>>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>>>>>>>> Try to execute "hadoop version" on your namenode, if the output is
>>>>>>>>>>>>>>>> Hadoop
>>>>>>>>>>>>>>>> 0.20.2-CDH3B4, the current cloudera distribution has been installed.
>>>>>>>>>>>>>>>> Btw, I would recommend to set your current Access Key ID and Secret
>>>>>>>>>>>>>>>> Key
>>>>>>>>>>>>>>>> inactive, since you posted it in your prop file.
>>>>>>>>>>>>>>>> cheers
>>>>>>>>>>>>>>>> sebastian
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 06.03.2011 06:54, Saptarshi Guha wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I did git clone of the latest whirr and copied cloudera scripts into
>>>>>>>>>>>>>>>>> the script directory (copied over
>>>>>>>>>>>>>>>>> from whirr-0.3-incubating).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My properties file is at the end of this email.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, I don't think the scripts are being run because the
>>>>>>>>>>>>>>>>> jobtracker
>>>>>>>>>>>>>>>>> is the default Apache hadoop jobtracker and not the cloudera
>>>>>>>>>>>>>>>>> jobtracker.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Have i missed something?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ## Properties
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> whirr.cluster-name=revotesting
>>>>>>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>>>>>>> whirr.identity= AKIAI3FUFFXAPYLE7CJA
>>>>>>>>>>>>>>>>> whirr.credential= 2Yq3Ar2HSxK/hbwZHs6aN6yrh0yfGNSPTpVw3t2n
>>>>>>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: hadoop config property override problem
Posted by Andrei Savu <sa...@gmail.com>.
I've created a patch [1] for this. Ben, I believe it should work for
you. Tom, let me know what you think about this approach. Unit and
integration tests are passing for me.
[1] https://issues.apache.org/jira/browse/WHIRR-259
On Sat, Mar 12, 2011 at 2:54 PM, Benjamin Clark <be...@daltonclark.com> wrote:
> Just tried it. No, that doesn't change the result.
>
>
> On Mar 12, 2011, at 1:21 AM, Tom White wrote:
>
>> Does escaping the comma with a backslash (i.e. \,) work? (See
>> http://commons.apache.org/configuration/howto_properties.html)
>>
>> Tom
>>
>> On Fri, Mar 11, 2011 at 7:56 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>> Yes, thanks Tom, that works.
>>>
>>> Many features and design changes in this branch are much appreciated, especially the config property override in the whirr config file.
>>>
>>> I found one problem, at least in the 0.4 branch. If you override a property with a comma-separated list, for example:
>>>
>>> hadoop-common.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
>>>
>>> then what actually shows up in core-site.xml is surrounded by brackets and has spaces between the elements of the list, so
>>>
>>> [org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec]
>>>
>>> Hadoop is not chomping or trimming that kind of thing, so you need to remove the spaces and brackets to get that to work. It's easy enough to patch the file, deploy to the slaves and restart, but I'm wondering if that's accounted for anywhere. I scanned CHANGES.txt in trunk and I don't see it.
>>>
>>> --Ben
>>>
>>>
>>> On Mar 10, 2011, at 5:41 PM, Tom White wrote:
>>>
>>>> On Thu, Mar 10, 2011 at 1:19 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>> Thank you both, Tom and Andrei. Now that I know where the faq is, I hope to bother you less with things that are documented!
>>>>>
>>>>> I think I should be all set with customization, but I need to build. BUILD.txt says 'mvn clean install' or mvn package -Ppackage. I can do that, and mvn reports success but then I try to use whirr-cli-0.4.0-incubating.jar as the jar, and the manifest has no main class (OK, I can supply 'org.apache.whirr.cli.Main' if I need to), and in any case the jar is not a fat jar, as I see all the publicly distributed versions are. It looks in the poms as if you have the maven assembly plugins trying to make a fat jar, but it doesn't seem to be doing it for me.
>>>>
>>>> Whirr no longer produces a shaded (fat) JAR as of 0.4.0 and trunk, so
>>>> perhaps it is working. Try bin/whirr and it should list the roles for
>>>> you.
>>>>
>>>> Cheers
>>>> Tom
>>>>
>>>>>
>>>>> What am I doing wrong?
>>>>>
>>>>> I'm doing this on a Mac like so:
>>>>>
>>>>> $ ruby --version
>>>>> ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10]
>>>>> $ mvn --version
>>>>> Apache Maven 3.0.2 (r1056850; 2011-01-08 19:58:10-0500)
>>>>> Java version: 1.6.0_24, vendor: Apple Inc.
>>>>> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>>>>> Default locale: en_US, platform encoding: MacRoman
>>>>> OS name: "mac os x", version: "10.6.6", arch: "x86_64", family: "mac"
>>>>> $ java -version
>>>>> java version "1.6.0_24"
>>>>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Now I think my only problem is that
>>>>> On Mar 10, 2011, at 2:35 PM, Andrei Savu wrote:
>>>>>
>>>>>> Starting with the upcoming 0.4.0 release Whirr is no longer using S3
>>>>>> for storing the install and configure scripts. You can grab the
>>>>>> scripts from:
>>>>>>
>>>>>> ${WHIRR_HOME}/services/${SERVICE_NAME}/src/main/resources/functions/{install,configure}_SERVICE.sh
>>>>>>
>>>>>> It's also easier to customize the scripts. You just need to place your
>>>>>> version in ${WHIRR_HOME}/functions (I believe it should have the same
>>>>>> name).
>>>>>>
>>>>>> From the 0.4.0 FAQ: "If you want to change the scripts then you can
>>>>>> place a modified copy of the
>>>>>> scripts in a _functions_ directory in Whirr's installation directory. The
>>>>>> original versions of the scripts can be found in _functions_ directories in the
>>>>>> source trees."
>>>>>>
>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>
>>>>>> On Thu, Mar 10, 2011 at 9:27 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>>> So if we grab install_cdh_hadoop.sh from the source tree, and put a customized version in our own bucket, and set whirr.run-url-base to the root of that bucket, it should work, even in 4.0 and after?
>>>>>>>
>>>>>>> Based on the FAQ I tried a few of these to attempt to verify I was on the right track:
>>>>>>>
>>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop
>>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop
>>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop
>>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop.sh
>>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop.sh
>>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop.sh
>>>>>>>
>>>>>>> but all give 404s.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mar 10, 2011, at 12:01 AM, Tom White wrote:
>>>>>>>
>>>>>>>> Sorry I missed this thread. On 0.4.0 and later
>>>>>>>>
>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>
>>>>>>>> changes to
>>>>>>>>
>>>>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Tom
>>>>>>>>
>>>>>>>> On Wed, Mar 9, 2011 at 8:23 AM, Sebastian Schoenherr
>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>> Hi Saptarshi,
>>>>>>>>> I tried to execute my working whirr 0.3.0 configuration (identical to your
>>>>>>>>> property file, using cloudera scripts) on branch-0.4 and the same issues
>>>>>>>>> arised for me. Unfortunately I'm not sure yet why it's not working with
>>>>>>>>> branch-0.4. Is using branch-0.3 an option for you?
>>>>>>>>> Any other guesses?
>>>>>>>>> cheers,
>>>>>>>>> sebastian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08.03.2011 05:41, Saptarshi Guha wrote:
>>>>>>>>>>
>>>>>>>>>> once again, i've changed the secret identity ..
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 7, 2011 at 8:41 PM, Saptarshi Guha
>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello
>>>>>>>>>>>
>>>>>>>>>>> No such luck on my end This is my script file, you can test that the
>>>>>>>>>>> scripts download. But when I log in
>>>>>>>>>>> hadoop version is. (I pulled the latest git). Also my scripts (you can
>>>>>>>>>>> confirm if you download) echo a small line to files in /tmp.
>>>>>>>>>>> They are not being created.
>>>>>>>>>>>
>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>> Subversion
>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>> -r 911707
>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>
>>>>>>>>>>> whirr.cluster-name=revotesting2
>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>> whirr.identity= AKIAJH5JBSI5KJ7YZQ6A
>>>>>>>>>>> whirr.credential= b/kqLJAHOdRA4L30n7Zt8Edz383B1ARtPI3wiyD6
>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>> whirr.run-url-base=http://ml.stat.purdue.edu/whirr-scripts/
>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>
>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>
>>>>>>>>>>> ##http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:59 AM, Benjamin Clark<be...@daltonclark.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> In my experience you need
>>>>>>>>>>>>
>>>>>>>>>>>> whirr.run-url-base=http://name-of-my-bucket-with-customized-scripts/
>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>
>>>>>>>>>>>> and then you *also* need to have a copy of sun/java/install in that same
>>>>>>>>>>>> bucket.
>>>>>>>>>>>>
>>>>>>>>>>>> And both of those scripts need to be public-readable.
>>>>>>>>>>>>
>>>>>>>>>>>> So in the end you should be able to do
>>>>>>>>>>>> curl
>>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/cloudera/cdh/install
>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>>>>>>> curl
>>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/sun/java/install
>>>>>>>>>>>>
>>>>>>>>>>>> Even if you haven't customizied sun/java/install, it needs to be there.
>>>>>>>>>>>>
>>>>>>>>>>>> If you do all that, the scripts will run and you will have the versions
>>>>>>>>>>>> you asked for.
>>>>>>>>>>>>
>>>>>>>>>>>> `hadoop version` on the name node then says, in my case:
>>>>>>>>>>>>
>>>>>>>>>>>> Hadoop 0.20.2-CDH3B4
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 7, 2011, at 12:41 PM, Saptarshi Guha wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> Fixed the security slip up. Did the hadoop version thing and got this
>>>>>>>>>>>>>
>>>>>>>>>>>>> [root@domU-12-31-39-0B-CC-41 ~]# hadoop version
>>>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>>>> Subversion
>>>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>>>> -r 911707
>>>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>>>
>>>>>>>>>>>>> So i guess its not CDH.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:22 AM, Saptarshi Guha
>>>>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> dear me! thanks, will do right away.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Mar 7, 2011 at 1:46 AM, Sebastian Schoenherr
>>>>>>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>>>>>>> Try to execute "hadoop version" on your namenode, if the output is
>>>>>>>>>>>>>>> Hadoop
>>>>>>>>>>>>>>> 0.20.2-CDH3B4, the current cloudera distribution has been installed.
>>>>>>>>>>>>>>> Btw, I would recommend to set your current Access Key ID and Secret
>>>>>>>>>>>>>>> Key
>>>>>>>>>>>>>>> inactive, since you posted it in your prop file.
>>>>>>>>>>>>>>> cheers
>>>>>>>>>>>>>>> sebastian
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 06.03.2011 06:54, Saptarshi Guha wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I did git clone of the latest whirr and copied cloudera scripts into
>>>>>>>>>>>>>>>> the script directory (copied over
>>>>>>>>>>>>>>>> from whirr-0.3-incubating).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My properties file is at the end of this email.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, I don't think the scripts are being run because the
>>>>>>>>>>>>>>>> jobtracker
>>>>>>>>>>>>>>>> is the default Apache hadoop jobtracker and not the cloudera
>>>>>>>>>>>>>>>> jobtracker.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Have i missed something?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ## Properties
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> whirr.cluster-name=revotesting
>>>>>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>>>>>> whirr.identity= AKIAI3FUFFXAPYLE7CJA
>>>>>>>>>>>>>>>> whirr.credential= 2Yq3Ar2HSxK/hbwZHs6aN6yrh0yfGNSPTpVw3t2n
>>>>>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
Re: hadoop config property override problem
Posted by Benjamin Clark <be...@daltonclark.com>.
Just tried it. No, that doesn't change the result.
On Mar 12, 2011, at 1:21 AM, Tom White wrote:
> Does escaping the comma with a backslash (i.e. \,) work? (See
> http://commons.apache.org/configuration/howto_properties.html)
>
> Tom
>
> On Fri, Mar 11, 2011 at 7:56 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>> Yes, thanks Tom, that works.
>>
>> Many features and design changes in this branch are much appreciated, especially the config property override in the whirr config file.
>>
>> I found one problem, at least in the 0.4 branch. If you override a property with a comma-separated list, for example:
>>
>> hadoop-common.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
>>
>> then what actually shows up in core-site.xml is surrounded by brackets and has spaces between the elements of the list, so
>>
>> [org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec]
>>
>> Hadoop is not chomping or trimming that kind of thing, so you need to remove the spaces and brackets to get that to work. It's easy enough to patch the file, deploy to the slaves and restart, but I'm wondering if that's accounted for anywhere. I scanned CHANGES.txt in trunk and I don't see it.
>>
>> --Ben
>>
>>
>> On Mar 10, 2011, at 5:41 PM, Tom White wrote:
>>
>>> On Thu, Mar 10, 2011 at 1:19 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>> Thank you both, Tom and Andrei. Now that I know where the faq is, I hope to bother you less with things that are documented!
>>>>
>>>> I think I should be all set with customization, but I need to build. BUILD.txt says 'mvn clean install' or mvn package -Ppackage. I can do that, and mvn reports success but then I try to use whirr-cli-0.4.0-incubating.jar as the jar, and the manifest has no main class (OK, I can supply 'org.apache.whirr.cli.Main' if I need to), and in any case the jar is not a fat jar, as I see all the publicly distributed versions are. It looks in the poms as if you have the maven assembly plugins trying to make a fat jar, but it doesn't seem to be doing it for me.
>>>
>>> Whirr no longer produces a shaded (fat) JAR as of 0.4.0 and trunk, so
>>> perhaps it is working. Try bin/whirr and it should list the roles for
>>> you.
>>>
>>> Cheers
>>> Tom
>>>
>>>>
>>>> What am I doing wrong?
>>>>
>>>> I'm doing this on a Mac like so:
>>>>
>>>> $ ruby --version
>>>> ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10]
>>>> $ mvn --version
>>>> Apache Maven 3.0.2 (r1056850; 2011-01-08 19:58:10-0500)
>>>> Java version: 1.6.0_24, vendor: Apple Inc.
>>>> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>>>> Default locale: en_US, platform encoding: MacRoman
>>>> OS name: "mac os x", version: "10.6.6", arch: "x86_64", family: "mac"
>>>> $ java -version
>>>> java version "1.6.0_24"
>>>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
>>>>
>>>>
>>>>
>>>>
>>>> Now I think my only problem is that
>>>> On Mar 10, 2011, at 2:35 PM, Andrei Savu wrote:
>>>>
>>>>> Starting with the upcoming 0.4.0 release Whirr is no longer using S3
>>>>> for storing the install and configure scripts. You can grab the
>>>>> scripts from:
>>>>>
>>>>> ${WHIRR_HOME}/services/${SERVICE_NAME}/src/main/resources/functions/{install,configure}_SERVICE.sh
>>>>>
>>>>> It's also easier to customize the scripts. You just need to place your
>>>>> version in ${WHIRR_HOME}/functions (I believe it should have the same
>>>>> name).
>>>>>
>>>>> From the 0.4.0 FAQ: "If you want to change the scripts then you can
>>>>> place a modified copy of the
>>>>> scripts in a _functions_ directory in Whirr's installation directory. The
>>>>> original versions of the scripts can be found in _functions_ directories in the
>>>>> source trees."
>>>>>
>>>>> -- Andrei Savu / andreisavu.ro
>>>>>
>>>>> On Thu, Mar 10, 2011 at 9:27 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>>> So if we grab install_cdh_hadoop.sh from the source tree, and put a customized version in our own bucket, and set whirr.run-url-base to the root of that bucket, it should work, even in 4.0 and after?
>>>>>>
>>>>>> Based on the FAQ I tried a few of these to attempt to verify I was on the right track:
>>>>>>
>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop
>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop
>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop
>>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop.sh
>>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop.sh
>>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop.sh
>>>>>>
>>>>>> but all give 404s.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mar 10, 2011, at 12:01 AM, Tom White wrote:
>>>>>>
>>>>>>> Sorry I missed this thread. On 0.4.0 and later
>>>>>>>
>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>
>>>>>>> changes to
>>>>>>>
>>>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Tom
>>>>>>>
>>>>>>> On Wed, Mar 9, 2011 at 8:23 AM, Sebastian Schoenherr
>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>> Hi Saptarshi,
>>>>>>>> I tried to execute my working whirr 0.3.0 configuration (identical to your
>>>>>>>> property file, using cloudera scripts) on branch-0.4 and the same issues
>>>>>>>> arised for me. Unfortunately I'm not sure yet why it's not working with
>>>>>>>> branch-0.4. Is using branch-0.3 an option for you?
>>>>>>>> Any other guesses?
>>>>>>>> cheers,
>>>>>>>> sebastian
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08.03.2011 05:41, Saptarshi Guha wrote:
>>>>>>>>>
>>>>>>>>> once again, i've changed the secret identity ..
>>>>>>>>>
>>>>>>>>> On Mon, Mar 7, 2011 at 8:41 PM, Saptarshi Guha
>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello
>>>>>>>>>>
>>>>>>>>>> No such luck on my end This is my script file, you can test that the
>>>>>>>>>> scripts download. But when I log in
>>>>>>>>>> hadoop version is. (I pulled the latest git). Also my scripts (you can
>>>>>>>>>> confirm if you download) echo a small line to files in /tmp.
>>>>>>>>>> They are not being created.
>>>>>>>>>>
>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>> Subversion
>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>> -r 911707
>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>
>>>>>>>>>> whirr.cluster-name=revotesting2
>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>> whirr.identity= AKIAJH5JBSI5KJ7YZQ6A
>>>>>>>>>> whirr.credential= b/kqLJAHOdRA4L30n7Zt8Edz383B1ARtPI3wiyD6
>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>> whirr.run-url-base=http://ml.stat.purdue.edu/whirr-scripts/
>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>
>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>
>>>>>>>>>> ##http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 7, 2011 at 9:59 AM, Benjamin Clark<be...@daltonclark.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> In my experience you need
>>>>>>>>>>>
>>>>>>>>>>> whirr.run-url-base=http://name-of-my-bucket-with-customized-scripts/
>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>
>>>>>>>>>>> and then you *also* need to have a copy of sun/java/install in that same
>>>>>>>>>>> bucket.
>>>>>>>>>>>
>>>>>>>>>>> And both of those scripts need to be public-readable.
>>>>>>>>>>>
>>>>>>>>>>> So in the end you should be able to do
>>>>>>>>>>> curl
>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/cloudera/cdh/install
>>>>>>>>>>>
>>>>>>>>>>> and
>>>>>>>>>>> curl
>>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/sun/java/install
>>>>>>>>>>>
>>>>>>>>>>> Even if you haven't customizied sun/java/install, it needs to be there.
>>>>>>>>>>>
>>>>>>>>>>> If you do all that, the scripts will run and you will have the versions
>>>>>>>>>>> you asked for.
>>>>>>>>>>>
>>>>>>>>>>> `hadoop version` on the name node then says, in my case:
>>>>>>>>>>>
>>>>>>>>>>> Hadoop 0.20.2-CDH3B4
>>>>>>>>>>>
>>>>>>>>>>> On Mar 7, 2011, at 12:41 PM, Saptarshi Guha wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> Fixed the security slip up. Did the hadoop version thing and got this
>>>>>>>>>>>>
>>>>>>>>>>>> [root@domU-12-31-39-0B-CC-41 ~]# hadoop version
>>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>>> Subversion
>>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>>> -r 911707
>>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>>
>>>>>>>>>>>> So i guess its not CDH.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:22 AM, Saptarshi Guha
>>>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> dear me! thanks, will do right away.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Mar 7, 2011 at 1:46 AM, Sebastian Schoenherr
>>>>>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>>>>>> Try to execute "hadoop version" on your namenode, if the output is
>>>>>>>>>>>>>> Hadoop
>>>>>>>>>>>>>> 0.20.2-CDH3B4, the current cloudera distribution has been installed.
>>>>>>>>>>>>>> Btw, I would recommend to set your current Access Key ID and Secret
>>>>>>>>>>>>>> Key
>>>>>>>>>>>>>> inactive, since you posted it in your prop file.
>>>>>>>>>>>>>> cheers
>>>>>>>>>>>>>> sebastian
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 06.03.2011 06:54, Saptarshi Guha wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I did git clone of the latest whirr and copied cloudera scripts into
>>>>>>>>>>>>>>> the script directory (copied over
>>>>>>>>>>>>>>> from whirr-0.3-incubating).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My properties file is at the end of this email.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, I don't think the scripts are being run because the
>>>>>>>>>>>>>>> jobtracker
>>>>>>>>>>>>>>> is the default Apache hadoop jobtracker and not the cloudera
>>>>>>>>>>>>>>> jobtracker.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Have i missed something?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ## Properties
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> whirr.cluster-name=revotesting
>>>>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>>>>> whirr.identity= AKIAI3FUFFXAPYLE7CJA
>>>>>>>>>>>>>>> whirr.credential= 2Yq3Ar2HSxK/hbwZHs6aN6yrh0yfGNSPTpVw3t2n
>>>>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: hadoop config property override problem
Posted by Tom White <to...@gmail.com>.
Does escaping the comma with a backslash (i.e. \,) work? (See
http://commons.apache.org/configuration/howto_properties.html)
Tom
On Fri, Mar 11, 2011 at 7:56 PM, Benjamin Clark <be...@daltonclark.com> wrote:
> Yes, thanks Tom, that works.
>
> Many features and design changes in this branch are much appreciated, especially the config property override in the whirr config file.
>
> I found one problem, at least in the 0.4 branch. If you override a property with a comma-separated list, for example:
>
> hadoop-common.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
>
> then what actually shows up in core-site.xml is surrounded by brackets and has spaces between the elements of the list, so
>
> [org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec]
>
> Hadoop is not chomping or trimming that kind of thing, so you need to remove the spaces and brackets to get that to work. It's easy enough to patch the file, deploy to the slaves and restart, but I'm wondering if that's accounted for anywhere. I scanned CHANGES.txt in trunk and I don't see it.
>
> --Ben
>
>
> On Mar 10, 2011, at 5:41 PM, Tom White wrote:
>
>> On Thu, Mar 10, 2011 at 1:19 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>> Thank you both, Tom and Andrei. Now that I know where the faq is, I hope to bother you less with things that are documented!
>>>
>>> I think I should be all set with customization, but I need to build. BUILD.txt says 'mvn clean install' or mvn package -Ppackage. I can do that, and mvn reports success but then I try to use whirr-cli-0.4.0-incubating.jar as the jar, and the manifest has no main class (OK, I can supply 'org.apache.whirr.cli.Main' if I need to), and in any case the jar is not a fat jar, as I see all the publicly distributed versions are. It looks in the poms as if you have the maven assembly plugins trying to make a fat jar, but it doesn't seem to be doing it for me.
>>
>> Whirr no longer produces a shaded (fat) JAR as of 0.4.0 and trunk, so
>> perhaps it is working. Try bin/whirr and it should list the roles for
>> you.
>>
>> Cheers
>> Tom
>>
>>>
>>> What am I doing wrong?
>>>
>>> I'm doing this on a Mac like so:
>>>
>>> $ ruby --version
>>> ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10]
>>> $ mvn --version
>>> Apache Maven 3.0.2 (r1056850; 2011-01-08 19:58:10-0500)
>>> Java version: 1.6.0_24, vendor: Apple Inc.
>>> Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>>> Default locale: en_US, platform encoding: MacRoman
>>> OS name: "mac os x", version: "10.6.6", arch: "x86_64", family: "mac"
>>> $ java -version
>>> java version "1.6.0_24"
>>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
>>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
>>>
>>>
>>>
>>>
>>> Now I think my only problem is that
>>> On Mar 10, 2011, at 2:35 PM, Andrei Savu wrote:
>>>
>>>> Starting with the upcoming 0.4.0 release Whirr is no longer using S3
>>>> for storing the install and configure scripts. You can grab the
>>>> scripts from:
>>>>
>>>> ${WHIRR_HOME}/services/${SERVICE_NAME}/src/main/resources/functions/{install,configure}_SERVICE.sh
>>>>
>>>> It's also easier to customize the scripts. You just need to place your
>>>> version in ${WHIRR_HOME}/functions (I believe it should have the same
>>>> name).
>>>>
>>>> From the 0.4.0 FAQ: "If you want to change the scripts then you can
>>>> place a modified copy of the
>>>> scripts in a _functions_ directory in Whirr's installation directory. The
>>>> original versions of the scripts can be found in _functions_ directories in the
>>>> source trees."
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>> On Thu, Mar 10, 2011 at 9:27 PM, Benjamin Clark <be...@daltonclark.com> wrote:
>>>>> So if we grab install_cdh_hadoop.sh from the source tree, and put a customized version in our own bucket, and set whirr.run-url-base to the root of that bucket, it should work, even in 4.0 and after?
>>>>>
>>>>> Based on the FAQ I tried a few of these to attempt to verify I was on the right track:
>>>>>
>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop
>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop
>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop
>>>>> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop.sh
>>>>> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop.sh
>>>>> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop.sh
>>>>>
>>>>> but all give 404s.
>>>>>
>>>>>
>>>>>
>>>>> On Mar 10, 2011, at 12:01 AM, Tom White wrote:
>>>>>
>>>>>> Sorry I missed this thread. On 0.4.0 and later
>>>>>>
>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>
>>>>>> changes to
>>>>>>
>>>>>> whirr.hadoop-install-function=install_cdh_hadoop
>>>>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>>>>>
>>>>>> Cheers,
>>>>>> Tom
>>>>>>
>>>>>> On Wed, Mar 9, 2011 at 8:23 AM, Sebastian Schoenherr
>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>> Hi Saptarshi,
>>>>>>> I tried to execute my working whirr 0.3.0 configuration (identical to your
>>>>>>> property file, using cloudera scripts) on branch-0.4 and the same issues
>>>>>>> arised for me. Unfortunately I'm not sure yet why it's not working with
>>>>>>> branch-0.4. Is using branch-0.3 an option for you?
>>>>>>> Any other guesses?
>>>>>>> cheers,
>>>>>>> sebastian
>>>>>>>
>>>>>>>
>>>>>>> On 08.03.2011 05:41, Saptarshi Guha wrote:
>>>>>>>>
>>>>>>>> once again, i've changed the secret identity ..
>>>>>>>>
>>>>>>>> On Mon, Mar 7, 2011 at 8:41 PM, Saptarshi Guha
>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>>
>>>>>>>>> No such luck on my end This is my script file, you can test that the
>>>>>>>>> scripts download. But when I log in
>>>>>>>>> hadoop version is. (I pulled the latest git). Also my scripts (you can
>>>>>>>>> confirm if you download) echo a small line to files in /tmp.
>>>>>>>>> They are not being created.
>>>>>>>>>
>>>>>>>>> Hadoop 0.20.2
>>>>>>>>> Subversion
>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>> -r 911707
>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>
>>>>>>>>> whirr.cluster-name=revotesting2
>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>> whirr.identity= AKIAJH5JBSI5KJ7YZQ6A
>>>>>>>>> whirr.credential= b/kqLJAHOdRA4L30n7Zt8Edz383B1ARtPI3wiyD6
>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>> whirr.run-url-base=http://ml.stat.purdue.edu/whirr-scripts/
>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>
>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>
>>>>>>>>> ##http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Mar 7, 2011 at 9:59 AM, Benjamin Clark<be...@daltonclark.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> In my experience you need
>>>>>>>>>>
>>>>>>>>>> whirr.run-url-base=http://name-of-my-bucket-with-customized-scripts/
>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>
>>>>>>>>>> and then you *also* need to have a copy of sun/java/install in that same
>>>>>>>>>> bucket.
>>>>>>>>>>
>>>>>>>>>> And both of those scripts need to be public-readable.
>>>>>>>>>>
>>>>>>>>>> So in the end you should be able to do
>>>>>>>>>> curl
>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/cloudera/cdh/install
>>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>> curl
>>>>>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/sun/java/install
>>>>>>>>>>
>>>>>>>>>> Even if you haven't customizied sun/java/install, it needs to be there.
>>>>>>>>>>
>>>>>>>>>> If you do all that, the scripts will run and you will have the versions
>>>>>>>>>> you asked for.
>>>>>>>>>>
>>>>>>>>>> `hadoop version` on the name node then says, in my case:
>>>>>>>>>>
>>>>>>>>>> Hadoop 0.20.2-CDH3B4
>>>>>>>>>>
>>>>>>>>>> On Mar 7, 2011, at 12:41 PM, Saptarshi Guha wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> Fixed the security slip up. Did the hadoop version thing and got this
>>>>>>>>>>>
>>>>>>>>>>> [root@domU-12-31-39-0B-CC-41 ~]# hadoop version
>>>>>>>>>>> Hadoop 0.20.2
>>>>>>>>>>> Subversion
>>>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>>>>>>>>>>> -r 911707
>>>>>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010
>>>>>>>>>>>
>>>>>>>>>>> So i guess its not CDH.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Saptarshi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 7, 2011 at 9:22 AM, Saptarshi Guha
>>>>>>>>>>> <sa...@revolutionanalytics.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> dear me! thanks, will do right away.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Mar 7, 2011 at 1:46 AM, Sebastian Schoenherr
>>>>>>>>>>>> <se...@uibk.ac.at> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Saptarshi,
>>>>>>>>>>>>> Try to execute "hadoop version" on your namenode, if the output is
>>>>>>>>>>>>> Hadoop
>>>>>>>>>>>>> 0.20.2-CDH3B4, the current cloudera distribution has been installed.
>>>>>>>>>>>>> Btw, I would recommend to set your current Access Key ID and Secret
>>>>>>>>>>>>> Key
>>>>>>>>>>>>> inactive, since you posted it in your prop file.
>>>>>>>>>>>>> cheers
>>>>>>>>>>>>> sebastian
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 06.03.2011 06:54, Saptarshi Guha wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I did git clone of the latest whirr and copied cloudera scripts into
>>>>>>>>>>>>>> the script directory (copied over
>>>>>>>>>>>>>> from whirr-0.3-incubating).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My properties file is at the end of this email.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, I don't think the scripts are being run because the
>>>>>>>>>>>>>> jobtracker
>>>>>>>>>>>>>> is the default Apache hadoop jobtracker and not the cloudera
>>>>>>>>>>>>>> jobtracker.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have i missed something?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Saptarshi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ## Properties
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> whirr.cluster-name=revotesting
>>>>>>>>>>>>>> whirr.service-name=hadoop
>>>>>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2
>>>>>>>>>>>>>> hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>>>>> whirr.provider=aws-ec2
>>>>>>>>>>>>>> whirr.identity= AKIAI3FUFFXAPYLE7CJA
>>>>>>>>>>>>>> whirr.credential= 2Yq3Ar2HSxK/hbwZHs6aN6yrh0yfGNSPTpVw3t2n
>>>>>>>>>>>>>> whirr.location-id=us-east-1
>>>>>>>>>>>>>> whirr.hardware-id=c1.xlarge
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ## Rightscales CentOS AMI
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes
>>>>>>>>>>>>>> jclouds.ec2.ami-owners=411009282317
>>>>>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install
>>>>>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>