You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Eric Danker <ed...@riotgames.com> on 2011/01/02 07:20:07 UTC
Issues spinning up a test cluster with Whirr 0.2.0
Whirr Folks,
I'm having trouble spinning up a small test Hadoop cluster for a POC. I'm following the instructions here:
http://incubator.apache.org/whirr/quick-start-guide.html (also tried the instructions here, which are similar: https://wiki.cloudera.com/display/DOC/Whirr+Installation)
My ec2.hadoop.properties file is:
whirr.service-name=hadoop
whirr.cluster-name=RiotTestHadoopCluster
whirr.instance-templates=1 jt+nn,2 dn+tt
whirr.provider=ec2
whirr.identity=REDACTED
whirr.credential=REDACTED
whirr.private-key-file=/path/to/private/key.pem
#whirr.public-key-file=
I can successfully start up the 3 node cluster with the following command:
whirr launch-cluster --config ec2.hadoop.properties
The next step is where I get tripped up. When I try to start the ssh proxy with the command:
. ~/.whirr/RiotTestHadoopCluster/hadoop-proxy.sh
I get an ssh error about the public key. Upon investigation, I see that the generated proxy script is trying to use the private key that I specified in my initial config, but the instances spun-up on EC2 are using a different key pair that is apparently created on-the-fly by JClouds (jclouds#RiotTestHadoopCluster#us-east-1#XX in this case).
To further complicated matters, when I try to connect to the master node via the EC2 control panel, it says to connect with the following ssh string:
ssh -i jclouds#RiotTestHadoopCluster#us-east-1#XX.pem root@ec2-50-16-136-233.compute-1.amazonaws.com<ma...@ec2-50-16-136-233.compute-1.amazonaws.com>
This doesn't seem to jive with the proxy script that is using a different private key and is trying to login with "ec2user" instead of "root".
Is the auto-created key pair only for user "root" and the private/public key pairs specified in the config used for the ec2user?
If that's the case, then I guess I need to figure out how to get the "public" version of the key from EC2 when I generate a key pair...it seems like it only gives you the option to download the private key..and just provides a signature for the keypair in the EC2 console.
So I guess my final question is:
- Is the generation of the JClouds key pairs transparent to this process? And if so,
- Is the reason why we need to specify both private AND public keys in the config file so that the public key can be installed under the "ec2user" on the new instance and the private key specified in the auto-gen'd proxy script?
Thanks in advance for your help!
- Eric
Re: Issues spinning up a test cluster with Whirr 0.2.0
Posted by Tom White <to...@cloudera.com>.
Hi Eric,
Did you generate a new keypair? It's usually best to do this - you
can't use the EC2 PEM file since there is no associated public key
available with it.
Cheers,
Tom
On Sat, Jan 1, 2011 at 10:20 PM, Eric Danker <ed...@riotgames.com> wrote:
> Whirr Folks,
>
>
>
> I’m having trouble spinning up a small test Hadoop cluster for a POC. I’m
> following the instructions here:
>
>
>
> http://incubator.apache.org/whirr/quick-start-guide.html (also tried the
> instructions here, which are similar:
> https://wiki.cloudera.com/display/DOC/Whirr+Installation)
>
>
>
> My ec2.hadoop.properties file is:
>
>
>
> whirr.service-name=hadoop
>
> whirr.cluster-name=RiotTestHadoopCluster
>
> whirr.instance-templates=1 jt+nn,2 dn+tt
>
> whirr.provider=ec2
>
> whirr.identity=REDACTED
>
> whirr.credential=REDACTED
>
> whirr.private-key-file=/path/to/private/key.pem
>
> #whirr.public-key-file=
>
>
>
> I can successfully start up the 3 node cluster with the following command:
>
>
>
> whirr launch-cluster --config ec2.hadoop.properties
>
>
>
> The next step is where I get tripped up. When I try to start the ssh proxy
> with the command:
>
>
>
> . ~/.whirr/RiotTestHadoopCluster/hadoop-proxy.sh
>
>
>
> I get an ssh error about the public key. Upon investigation, I see that the
> generated proxy script is trying to use the private key that I specified in
> my initial config, but the instances spun-up on EC2 are using a different
> key pair that is apparently created on-the-fly by JClouds
> (jclouds#RiotTestHadoopCluster#us-east-1#XX in this case).
>
>
>
> To further complicated matters, when I try to connect to the master node via
> the EC2 control panel, it says to connect with the following ssh string:
>
>
>
> ssh -i jclouds#RiotTestHadoopCluster#us-east-1#XX.pem
> root@ec2-50-16-136-233.compute-1.amazonaws.com
>
>
>
> This doesn’t seem to jive with the proxy script that is using a different
> private key and is trying to login with “ec2user” instead of “root”.
>
>
>
> Is the auto-created key pair only for user “root” and the private/public key
> pairs specified in the config used for the ec2user?
>
>
>
> If that’s the case, then I guess I need to figure out how to get the
> “public” version of the key from EC2 when I generate a key pair…it seems
> like it only gives you the option to download the private key..and just
> provides a signature for the keypair in the EC2 console.
>
>
>
> So I guess my final question is:
>
> - Is the generation of the JClouds key pairs transparent to this
> process? And if so,
>
> - Is the reason why we need to specify both private AND public keys
> in the config file so that the public key can be installed under the
> “ec2user” on the new instance and the private key specified in the
> auto-gen’d proxy script?
>
>
>
> Thanks in advance for your help!
>
>
>
> - Eric
>
>
>
>