You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Bob Briski <bb...@raybeam.com> on 2012/04/09 19:24:07 UTC
EC2: Security group not being destroyed
I'd say about 60-70% of the time, the "destroy-cluster" command doesn't
actually remove the security keys for the cluster on EC2. When I then try
to restart the cluster it fails with an error saying that the cluster
already the correct permissions defined.
Is there a way to force a delete? Or, even better, just have Whirr ignore
the security group definition if it already exists for this cluster
definition?
Thanks,
Bob
Re: EC2: Security group not being destroyed
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
Bob-
Guessing from the private key file you might be running as ubuntu which
could be doing something weird somewhere. A bug for sure, so logs would
be handy, but might have an easy workaround if you can run as a
different user locally (could be whirr, or could go with something else
to disambiguate in the log files).
--A
On 10/04/2012 22:05, Andrei Savu wrote:
> Can you post the relevant stack trace? Or log records?
>
> -- Andrei
>
> On Tue, Apr 10, 2012 at 11:55 PM, Bob Briski <bbriski@raybeam.com
> <ma...@raybeam.com>> wrote:
>
> Alright, this problem is happening again. Here's my properties file:
>
> whirr.cluster-name=sqoop
> whirr.cluster-user=whirr
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,6
> hadoop-datanode+hadoop-tasktracker
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> whirr.provider=aws-ec2
> whirr.identity=${env:AWS_ACCESS_KEY_ID}
> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
> whirr.hardware-id=m1.large
> whirr.image-id=us-east-1/ami-3f6ca156
> whirr.location-id=us-east-1
> whirr.private-key-file=/home/ubuntu/.ssh/id_rsa
> whirr.public-key-file=${whirr.private-key-file}.pub
> whirr.firewall-rules=3306
>
>
Re: EC2: Security group not being destroyed
Posted by Andrei Savu <sa...@gmail.com>.
Can you post the relevant stack trace? Or log records?
-- Andrei
On Tue, Apr 10, 2012 at 11:55 PM, Bob Briski <bb...@raybeam.com> wrote:
> Alright, this problem is happening again. Here's my properties file:
>
> whirr.cluster-name=sqoop
> whirr.cluster-user=whirr
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,6
> hadoop-datanode+hadoop-tasktracker
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> whirr.provider=aws-ec2
> whirr.identity=${env:AWS_ACCESS_KEY_ID}
> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
> whirr.hardware-id=m1.large
> whirr.image-id=us-east-1/ami-3f6ca156
> whirr.location-id=us-east-1
> whirr.private-key-file=/home/ubuntu/.ssh/id_rsa
> whirr.public-key-file=${whirr.private-key-file}.pub
> whirr.firewall-rules=3306
>
>
Re: EC2: Security group not being destroyed
Posted by Bob Briski <bb...@raybeam.com>.
Alright, this problem is happening again. Here's my properties file:
whirr.cluster-name=sqoop
whirr.cluster-user=whirr
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,6
hadoop-datanode+hadoop-tasktracker
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop
whirr.provider=aws-ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.hardware-id=m1.large
whirr.image-id=us-east-1/ami-3f6ca156
whirr.location-id=us-east-1
whirr.private-key-file=/home/ubuntu/.ssh/id_rsa
whirr.public-key-file=${whirr.private-key-file}.pub
whirr.firewall-rules=3306
Re: whirr.env.repo ignored for custom AMIs
Posted by Sebastian Schoenherr <se...@uibk.ac.at>.
Andrei,
I found the answer in /tmp/logs. The AMI I was using (it's a Ubuntu
10.04), already included the latest cdh3 release in its sources (|deb
http://archive.cloudera.com/debian maverick-cdh3 contrib||)| and I think
that's the reason I see cdh3u3 installed.
Thanks for your quick help!
Sebastian
On 10.04.2012 14:12, Andrei Savu wrote:
> It should work with a custom ami that's also running Ubuntu 10.04.
> Anything interesting in /tmp/logs?
>
> On Tue, Apr 10, 2012 at 3:05 PM, Sebastian Schoenherr
> <sebastian.schoenherr@uibk.ac.at
> <ma...@uibk.ac.at>> wrote:
>
> Hi Andrei,
> I'm using ubuntu AMIs (tried it with ubuntu 10.4 and ubuntu 11.10)
> cheers,
> sebastian
>
>
>
> On 10.04.2012 14:02, Andrei Savu wrote:
>> What OS is running your custom AMI?
>>
>> On Tue, Apr 10, 2012 at 2:55 PM, Sebastian Schoenherr
>> <sebastian.schoenherr@uibk.ac.at
>> <ma...@uibk.ac.at>> wrote:
>>
>> Hi guys,
>> I've some problems using the whirr.env.repo with whirr 0.7.1.
>> When specifying a custom AMI the newest version of cdh3
>> (currently cdh3u3) is getting installed even if the variable
>> whirr.env.repo=cdh3u2 is set.
>> I tried the exact same configuration with a standard AMI
>> (ami-da0cf8b3) and everything works as expected.
>> Any suggestions what I might be missing?
>> Cheers,
>> Sebastian
>>
>>
>
>
--
Sebastian Schoenherr
PhD student in Bioinformatics
Institute of Computer Science
Division of Genetic Epidemiology
sebastian.schoenherr@uibk.ac.at
http://dbis-informatik.uibk.ac.at
http://www.i-med.ac.at/genepi/
Re: whirr.env.repo ignored for custom AMIs
Posted by Andrei Savu <sa...@gmail.com>.
It should work with a custom ami that's also running Ubuntu 10.04. Anything
interesting in /tmp/logs?
On Tue, Apr 10, 2012 at 3:05 PM, Sebastian Schoenherr <
sebastian.schoenherr@uibk.ac.at> wrote:
> Hi Andrei,
> I'm using ubuntu AMIs (tried it with ubuntu 10.4 and ubuntu 11.10)
> cheers,
> sebastian
>
>
>
> On 10.04.2012 14:02, Andrei Savu wrote:
>
> What OS is running your custom AMI?
>
> On Tue, Apr 10, 2012 at 2:55 PM, Sebastian Schoenherr <
> sebastian.schoenherr@uibk.ac.at> wrote:
>
>> Hi guys,
>> I've some problems using the whirr.env.repo with whirr 0.7.1.
>> When specifying a custom AMI the newest version of cdh3 (currently
>> cdh3u3) is getting installed even if the variable whirr.env.repo=cdh3u2 is
>> set.
>> I tried the exact same configuration with a standard AMI (ami-da0cf8b3)
>> and everything works as expected.
>> Any suggestions what I might be missing?
>> Cheers,
>> Sebastian
>>
>
>
>
Re: whirr.env.repo ignored for custom AMIs
Posted by Sebastian Schoenherr <se...@uibk.ac.at>.
Hi Andrei,
I'm using ubuntu AMIs (tried it with ubuntu 10.4 and ubuntu 11.10)
cheers,
sebastian
On 10.04.2012 14:02, Andrei Savu wrote:
> What OS is running your custom AMI?
>
> On Tue, Apr 10, 2012 at 2:55 PM, Sebastian Schoenherr
> <sebastian.schoenherr@uibk.ac.at
> <ma...@uibk.ac.at>> wrote:
>
> Hi guys,
> I've some problems using the whirr.env.repo with whirr 0.7.1.
> When specifying a custom AMI the newest version of cdh3 (currently
> cdh3u3) is getting installed even if the variable
> whirr.env.repo=cdh3u2 is set.
> I tried the exact same configuration with a standard AMI
> (ami-da0cf8b3) and everything works as expected.
> Any suggestions what I might be missing?
> Cheers,
> Sebastian
>
>
Re: whirr.env.repo ignored for custom AMIs
Posted by Andrei Savu <sa...@gmail.com>.
What OS is running your custom AMI?
On Tue, Apr 10, 2012 at 2:55 PM, Sebastian Schoenherr <
sebastian.schoenherr@uibk.ac.at> wrote:
> Hi guys,
> I've some problems using the whirr.env.repo with whirr 0.7.1.
> When specifying a custom AMI the newest version of cdh3 (currently cdh3u3)
> is getting installed even if the variable whirr.env.repo=cdh3u2 is set.
> I tried the exact same configuration with a standard AMI (ami-da0cf8b3)
> and everything works as expected.
> Any suggestions what I might be missing?
> Cheers,
> Sebastian
>
whirr.env.repo ignored for custom AMIs
Posted by Sebastian Schoenherr <se...@uibk.ac.at>.
Hi guys,
I've some problems using the whirr.env.repo with whirr 0.7.1.
When specifying a custom AMI the newest version of cdh3 (currently
cdh3u3) is getting installed even if the variable whirr.env.repo=cdh3u2
is set.
I tried the exact same configuration with a standard AMI (ami-da0cf8b3)
and everything works as expected.
Any suggestions what I might be missing?
Cheers,
Sebastian
Re: EC2: Security group not being destroyed
Posted by Andrei Savu <sa...@gmail.com>.
This is something we've fixed in 0.7.2. Stay tunned for a new release.
On Tue, Apr 10, 2012 at 1:52 AM, Bob Briski <bb...@raybeam.com> wrote:
> That worked. Thanks.
>
> A word to anyone trying this that found the answer on
> https://issues.apache.org/jira/browse/WHIRR-378:
>
> Do NOT set the login-user to ubuntu. I was doing that while also setting
> the cluster-user and it continued to fail. Remove the login-user property
> and set the cluster-user to whirr.
>
> Thanks again Andrei
>
>
Re: EC2: Security group not being destroyed
Posted by Bob Briski <bb...@raybeam.com>.
That worked. Thanks.
A word to anyone trying this that found the answer on
https://issues.apache.org/jira/browse/WHIRR-378:
Do NOT set the login-user to ubuntu. I was doing that while also setting
the cluster-user and it continued to fail. Remove the login-user property
and set the cluster-user to whirr.
Thanks again Andrei
Re: EC2: Security group not being destroyed
Posted by Andrei Savu <sa...@gmail.com>.
Are you running Whirr inside Amazon? If so you need to add
whirr.cluster-user=whirr to your recipe.
Can you share your recipe?
On Tue, Apr 10, 2012 at 1:23 AM, Bob Briski <bb...@raybeam.com> wrote:
> That did fix the problem but introduced another problem. Actually, it's
> the reason I was using an older version in the first place.
>
> Just launching a cluster works the first time and then I get errors on
> authorization. It looks like it just can't contact the nodes. It results
> in the entire thing failing. It worked fine with 0.5.0. I haven't changed
> any configuration options and now I'm getting this:
>
> org.jclouds.rest.AuthorizationException:
> (ubuntu:rsa[fingerprint(8e:a4:cd:7d:87:5e:7e:de:f5:89:31:47:84:30:61:e0),sha1(f9:59:1f:ac:01:62:ba:f5:9c:a1:6c:b9:2e:21:4b:26:19:04:db:2b)]@
> 23.20.126.137:22)
> (ubuntu:rsa[fingerprint(8e:a4:cd:7d:87:5e:7e:de:f5:89:31:47:84:30:61:e0),sha1(f9:59:1f:ac:01:62:ba:f5:9c:a1:6c:b9:2e:21:4b:26:19:04:db:2b)]@
> 23.20.126.137:22) error acquiring SSHClient(timeout=60000): Exhausted
> available authentication methods
> at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:413)
> at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:244)
> at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)
> at
> org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150)
> at
> org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted
> available authentication methods
> at
> net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
> at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
> at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
> at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
> at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)
> at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)
> at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)
> ... 9 more
> Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth
> failed
> at
> net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
> at
> net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)
> at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
> at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
> at net.schmizz.sshj.transport.Reader.run(Reader.java:72)
>
>
>
Re: EC2: Security group not being destroyed
Posted by Bob Briski <bb...@raybeam.com>.
That did fix the problem but introduced another problem. Actually, it's
the reason I was using an older version in the first place.
Just launching a cluster works the first time and then I get errors on
authorization. It looks like it just can't contact the nodes. It results
in the entire thing failing. It worked fine with 0.5.0. I haven't changed
any configuration options and now I'm getting this:
org.jclouds.rest.AuthorizationException:
(ubuntu:rsa[fingerprint(8e:a4:cd:7d:87:5e:7e:de:f5:89:31:47:84:30:61:e0),sha1(f9:59:1f:ac:01:62:ba:f5:9c:a1:6c:b9:2e:21:4b:26:19:04:db:2b)]@
23.20.126.137:22)
(ubuntu:rsa[fingerprint(8e:a4:cd:7d:87:5e:7e:de:f5:89:31:47:84:30:61:e0),sha1(f9:59:1f:ac:01:62:ba:f5:9c:a1:6c:b9:2e:21:4b:26:19:04:db:2b)]@
23.20.126.137:22) error acquiring SSHClient(timeout=60000): Exhausted
available authentication methods
at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:413)
at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:244)
at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:255)
at
org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:89)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:150)
at
org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:57)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: net.schmizz.sshj.userauth.UserAuthException: Exhausted available
authentication methods
at
net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:199)
at org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:171)
at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:220)
... 9 more
Caused by: net.schmizz.sshj.userauth.UserAuthException: publickey auth
failed
at
net.schmizz.sshj.userauth.UserAuthImpl.handle(UserAuthImpl.java:157)
at
net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:474)
at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
at net.schmizz.sshj.transport.Reader.run(Reader.java:72)
Re: EC2: Security group not being destroyed
Posted by Andrei Savu <sa...@gmail.com>.
Upgrade to 0.7.1 - it should fix the problem for you.
On Mon, Apr 9, 2012 at 8:24 PM, Bob Briski <bb...@raybeam.com> wrote:
> I'd say about 60-70% of the time, the "destroy-cluster" command doesn't
> actually remove the security keys for the cluster on EC2. When I then try
> to restart the cluster it fails with an error saying that the cluster
> already the correct permissions defined.
>
> Is there a way to force a delete? Or, even better, just have Whirr ignore
> the security group definition if it already exists for this cluster
> definition?
>
> Thanks,
> Bob
>