You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Joris Poort <gp...@gmail.com> on 2011/08/25 04:07:30 UTC
EC2 Custom AMI's connection issue
Hi,
I'm new to whirr and I'm running custom AMI configuration (application
installed on working canonical image). Executing with whirr 0.6.0
everything executes fine until I get the following error:
"Dying because - java.net.SocketTimeoutException: Read timed out"
The instances are running fine, I can ssh into them, but the whirr
code stalls and I get the above error (2x number of instances), no
proxy shell is created. If I run the exact same code with vanilla
canonical images I don't have any issues.
Anyone have any ideas on things to test, debug or work around this?
Would really appreciate it!
Cheers,
Joris
Re: EC2 Custom AMI's connection issue
Posted by Paul Baclace <pa...@gmail.com>.
Try ssh -vv to see what is really happening. And of course make sure
id_rsa is only readable by the user executing ssh.
The whirr.log appears in the curr. dir. where the whirr command was run,
not on the spawned nodes.
Paul
On 20110824 21:01 , Joris Poort wrote:
> I'm just using the defaults. But you may be onto the problem here,
> when I try to ssh into the node using:
> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>
> I get a permission denied.
>
> Any idea how to fix this? Should I create my own set of SSH key pairs?
>
> Thanks again,
>
> Joris
>
> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu<sa...@gmail.com> wrote:
>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>
>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa& id_rsa.pub.
>>
>> -- Andrei Savu / andreisavu.ro
>>
>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort<gp...@gmail.com> wrote:
>>> I think you're probably right its an auth issue - although I was
>>> expecting a more direct/clear error message if the keypair wasn't
>>> working.
>>>
>>> I created the AMI by taking an EBS snapshot then converting to
>>> instance-store. I've tried both the ebs back ami and instance-store
>>> with the same results. My understanding is that the keypair used to
>>> create the AMI is generally one of the accepted keys in addition to
>>> the key pair used to launch the instance created by jclouds. I'm not
>>> sure how to confirm this for sure - is the jclouds keypair stored
>>> anywhere that can be used to test this?
>>>
>>> Thanks again for your help,
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu<sa...@gmail.com> wrote:
>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>> own key pair using the local SSH keys as specified in the properties
>>>> file.
>>>>
>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>> that custom ami with a different key pair?
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>>
>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort<gp...@gmail.com> wrote:
>>>>> Andrei - thanks for the response!
>>>>>
>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>> both spot instances and regular instances and am getting the same
>>>>> behavior.
>>>>>
>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>> and "Dying because" are not always there):
>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>> transport; encountered EOF
>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>> transport; encountered EOF
>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>> Broken transport; encountered EOF
>>>>> << (root@174.129.128.120:22) error acquiring
>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>
>>>>> Last few entries on whirr.log:
>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2)>>
>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>> ramdiskId=null, availabilityZone=null,
>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>> securityGroupIds=[],
>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4)<<
>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2)<<
>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4)<<
>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2)<<
>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8)>>
>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7)>>
>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>> seconds
>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7)<<
>>>>> socket [address=174.129.128.120, port=22] opened
>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8)<<
>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>
>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>
>>>>> Thanks again for any help on this!
>>>>>
>>>>> Joris
>>>>>
>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu<sa...@gmail.com> wrote:
>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>
>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>> in /tmp for jclouds script execution logs.
>>>>>>
>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>> same behavior with regular ones?
>>>>>>
>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort<gp...@gmail.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>> everything executes fine until I get the following error:
>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>
>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>> canonical images I don't have any issues.
>>>>>>>
>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>> Would really appreciate it!
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Joris
>>>>>>>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Great, thanks so much!
Joris
On Thu, Aug 25, 2011 at 12:45 AM, Andrei Savu <sa...@gmail.com> wrote:
> Check this presentation:
> http://www.oscon.com/oscon2011/public/schedule/detail/19214
>
> The ZooKeeper service should be easy to understand.
>
> Cheers,
>
> -- Andrei Savu / andreisavu.ro
>
> On Thu, Aug 25, 2011 at 12:41 AM, Joris Poort <gp...@gmail.com> wrote:
>> Does anyone know a good simple whirr service code to use as a base to
>> learn to write your own? Kind of new to this so would really
>> appreciate it.
>>
>> Cheers,
>>
>> Joris
>>
>> On Thu, Aug 25, 2011 at 12:16 AM, Joris Poort <gp...@gmail.com> wrote:
>>> Thanks for the recommendation on whirr services code, I think that
>>> will work - looking into how to do it now.
>>>
>>> Thanks again Andrei - really appreciate your help on all this.
>>>
>>> Cheers,
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 11:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>> It seems like for your custom ami the user scripts are not executed as expected.
>>>>
>>>> How about adding your own software later as Whirr services or using
>>>> bash scripts and start from a vanilla Ubuntu ami?
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>> On Wed, Aug 24, 2011 at 11:41 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> Thanks for offering your help Guillaume. Unfortunately I tried that
>>>>> and still have the same issues.
>>>>>
>>>>> On Wed, Aug 24, 2011 at 11:34 PM, tog <gu...@gmail.com> wrote:
>>>>>> I don't know if this can help but here are the 2 lines that I added to
>>>>>> my cluster property file in order to be able to log on the cluster:
>>>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>>>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>>>>>
>>>>>> then I was able to connect my local userid and not ubuntu.
>>>>>>
>>>>>> HTH
>>>>>> Guillaume
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 25, 2011 at 11:34 AM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>> I also tried to regenerate the ssh keys, but still no luck...
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>> Interestingly, as mentioned before I can ssh in using my keypair used
>>>>>>>> to create the custom AMI. So I'm thinking that it still has to do
>>>>>>>> with whirr not being able to get the right keypair access with the
>>>>>>>> local id_rsa.
>>>>>>>>
>>>>>>>> Thanks again for any help,
>>>>>>>>
>>>>>>>> Joris
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>>>>>>>>> adjusted to local-user "gpoort" I'm still not getting through. See
>>>>>>>>> output below from -vv (sorry about the long email).
>>>>>>>>>
>>>>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>>>>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>>>>> debug1: Applying options for *
>>>>>>>>> debug2: ssh_connect: needpriv 0
>>>>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>>> [184.72.86.108] port 22.
>>>>>>>>> debug1: Connection established.
>>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>>>>> debug2: dh_gen_key: priv key bits set: 138/256
>>>>>>>>> debug2: bits set: 502/1024
>>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>>>>> matches the RSA host key.
>>>>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>>>>> debug2: bits set: 497/1024
>>>>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>>>>> debug2: kex_derive_keys
>>>>>>>>> debug2: set_newkeys: mode 1
>>>>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>>>>> debug2: set_newkeys: mode 0
>>>>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>>>>> debug1: Roaming not allowed by server
>>>>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>>>>> debug2: service_accept: ssh-userauth
>>>>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>>>>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>>> debug1: Next authentication method: publickey
>>>>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>>> debug2: we did not send a packet, disable method
>>>>>>>>> debug1: No more authentication methods to try.
>>>>>>>>> Permission denied (publickey).
>>>>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>>>>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>>>>> debug1: Applying options for *
>>>>>>>>> debug2: ssh_connect: needpriv 0
>>>>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>>> [184.72.86.108] port 22.
>>>>>>>>> debug1: Connection established.
>>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>>>>> debug2: dh_gen_key: priv key bits set: 145/256
>>>>>>>>> debug2: bits set: 505/1024
>>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>>>>> matches the RSA host key.
>>>>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>>>>> debug2: bits set: 495/1024
>>>>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>>>>> debug2: kex_derive_keys
>>>>>>>>> debug2: set_newkeys: mode 1
>>>>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>>>>> debug2: set_newkeys: mode 0
>>>>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>>>>> debug1: Roaming not allowed by server
>>>>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>>>>> debug2: service_accept: ssh-userauth
>>>>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>>>>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>>> debug1: Next authentication method: publickey
>>>>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>>> debug2: we did not send a packet, disable method
>>>>>>>>> debug1: No more authentication methods to try.
>>>>>>>>> Permission denied (publickey).
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>> You should be able to login user the same user that is running Whirr.
>>>>>>>>>>
>>>>>>>>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>>>>>>>>
>>>>>>>>>> Permission denied most of the time means invalid key, user pair.
>>>>>>>>>>
>>>>>>>>>> It should be ok to use the keys generate by default.
>>>>>>>>>>
>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>> I'm just using the defaults. But you may be onto the problem here,
>>>>>>>>>>> when I try to ssh into the node using:
>>>>>>>>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>>>>>>>>
>>>>>>>>>>> I get a permission denied.
>>>>>>>>>>>
>>>>>>>>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>>>>>>>>
>>>>>>>>>>> Thanks again,
>>>>>>>>>>>
>>>>>>>>>>> Joris
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>>>>>>>>
>>>>>>>>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>>>>>>>>
>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>> I think you're probably right its an auth issue - although I was
>>>>>>>>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>>>>>>>>> working.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>>>>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>>>>>>>>> with the same results. My understanding is that the keypair used to
>>>>>>>>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>>>>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>>>>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>>>>>>>>> anywhere that can be used to test this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks again for your help,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>>>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>>>>>>>>> that custom ami with a different key pair?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>>>> Andrei - thanks for the response!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>>>>>>>>> and "Dying because" are not always there):
>>>>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>>>>>>>>> Broken transport; encountered EOF
>>>>>>>>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Last few entries on whirr.log:
>>>>>>>>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>>>>>>>>> securityGroupIds=[],
>>>>>>>>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>>>>>>>>> seconds
>>>>>>>>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks again for any help on this!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>>>>>>>>> same behavior with regular ones?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Andrei Savu <sa...@gmail.com>.
Check this presentation:
http://www.oscon.com/oscon2011/public/schedule/detail/19214
The ZooKeeper service should be easy to understand.
Cheers,
-- Andrei Savu / andreisavu.ro
On Thu, Aug 25, 2011 at 12:41 AM, Joris Poort <gp...@gmail.com> wrote:
> Does anyone know a good simple whirr service code to use as a base to
> learn to write your own? Kind of new to this so would really
> appreciate it.
>
> Cheers,
>
> Joris
>
> On Thu, Aug 25, 2011 at 12:16 AM, Joris Poort <gp...@gmail.com> wrote:
>> Thanks for the recommendation on whirr services code, I think that
>> will work - looking into how to do it now.
>>
>> Thanks again Andrei - really appreciate your help on all this.
>>
>> Cheers,
>>
>> Joris
>>
>> On Wed, Aug 24, 2011 at 11:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>> It seems like for your custom ami the user scripts are not executed as expected.
>>>
>>> How about adding your own software later as Whirr services or using
>>> bash scripts and start from a vanilla Ubuntu ami?
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>> On Wed, Aug 24, 2011 at 11:41 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> Thanks for offering your help Guillaume. Unfortunately I tried that
>>>> and still have the same issues.
>>>>
>>>> On Wed, Aug 24, 2011 at 11:34 PM, tog <gu...@gmail.com> wrote:
>>>>> I don't know if this can help but here are the 2 lines that I added to
>>>>> my cluster property file in order to be able to log on the cluster:
>>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>>>>
>>>>> then I was able to connect my local userid and not ubuntu.
>>>>>
>>>>> HTH
>>>>> Guillaume
>>>>>
>>>>>
>>>>> On Thu, Aug 25, 2011 at 11:34 AM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> I also tried to regenerate the ssh keys, but still no luck...
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>> Interestingly, as mentioned before I can ssh in using my keypair used
>>>>>>> to create the custom AMI. So I'm thinking that it still has to do
>>>>>>> with whirr not being able to get the right keypair access with the
>>>>>>> local id_rsa.
>>>>>>>
>>>>>>> Thanks again for any help,
>>>>>>>
>>>>>>> Joris
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>>>>>>>> adjusted to local-user "gpoort" I'm still not getting through. See
>>>>>>>> output below from -vv (sorry about the long email).
>>>>>>>>
>>>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>>>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>>>> debug1: Applying options for *
>>>>>>>> debug2: ssh_connect: needpriv 0
>>>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>> [184.72.86.108] port 22.
>>>>>>>> debug1: Connection established.
>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>>>> debug2: dh_gen_key: priv key bits set: 138/256
>>>>>>>> debug2: bits set: 502/1024
>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>>>> matches the RSA host key.
>>>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>>>> debug2: bits set: 497/1024
>>>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>>>> debug2: kex_derive_keys
>>>>>>>> debug2: set_newkeys: mode 1
>>>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>>>> debug2: set_newkeys: mode 0
>>>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>>>> debug1: Roaming not allowed by server
>>>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>>>> debug2: service_accept: ssh-userauth
>>>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>>>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>> debug1: Next authentication method: publickey
>>>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>> debug2: we did not send a packet, disable method
>>>>>>>> debug1: No more authentication methods to try.
>>>>>>>> Permission denied (publickey).
>>>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>>>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>>>> debug1: Applying options for *
>>>>>>>> debug2: ssh_connect: needpriv 0
>>>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>>> [184.72.86.108] port 22.
>>>>>>>> debug1: Connection established.
>>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit:
>>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>>>> debug2: dh_gen_key: priv key bits set: 145/256
>>>>>>>> debug2: bits set: 505/1024
>>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>>>> matches the RSA host key.
>>>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>>>> debug2: bits set: 495/1024
>>>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>>>> debug2: kex_derive_keys
>>>>>>>> debug2: set_newkeys: mode 1
>>>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>>>> debug2: set_newkeys: mode 0
>>>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>>>> debug1: Roaming not allowed by server
>>>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>>>> debug2: service_accept: ssh-userauth
>>>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>>>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>> debug1: Next authentication method: publickey
>>>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>>> debug2: we did not send a packet, disable method
>>>>>>>> debug1: No more authentication methods to try.
>>>>>>>> Permission denied (publickey).
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>> You should be able to login user the same user that is running Whirr.
>>>>>>>>>
>>>>>>>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>>>>>>>
>>>>>>>>> Permission denied most of the time means invalid key, user pair.
>>>>>>>>>
>>>>>>>>> It should be ok to use the keys generate by default.
>>>>>>>>>
>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>> I'm just using the defaults. But you may be onto the problem here,
>>>>>>>>>> when I try to ssh into the node using:
>>>>>>>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>>>>>>>
>>>>>>>>>> I get a permission denied.
>>>>>>>>>>
>>>>>>>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>>>>>>>
>>>>>>>>>> Thanks again,
>>>>>>>>>>
>>>>>>>>>> Joris
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>>>>>>>
>>>>>>>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>>>>>>>
>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>> I think you're probably right its an auth issue - although I was
>>>>>>>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>>>>>>>> working.
>>>>>>>>>>>>
>>>>>>>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>>>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>>>>>>>> with the same results. My understanding is that the keypair used to
>>>>>>>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>>>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>>>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>>>>>>>> anywhere that can be used to test this?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks again for your help,
>>>>>>>>>>>>
>>>>>>>>>>>> Joris
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>>>>>>>> file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>>>>>>>> that custom ami with a different key pair?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>>> Andrei - thanks for the response!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>>>>>>>> and "Dying because" are not always there):
>>>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>>>>>>>> Broken transport; encountered EOF
>>>>>>>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Last few entries on whirr.log:
>>>>>>>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>>>>>>>> securityGroupIds=[],
>>>>>>>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>>>>>>>> seconds
>>>>>>>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks again for any help on this!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>>>>>>>> same behavior with regular ones?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Does anyone know a good simple whirr service code to use as a base to
learn to write your own? Kind of new to this so would really
appreciate it.
Cheers,
Joris
On Thu, Aug 25, 2011 at 12:16 AM, Joris Poort <gp...@gmail.com> wrote:
> Thanks for the recommendation on whirr services code, I think that
> will work - looking into how to do it now.
>
> Thanks again Andrei - really appreciate your help on all this.
>
> Cheers,
>
> Joris
>
> On Wed, Aug 24, 2011 at 11:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>> It seems like for your custom ami the user scripts are not executed as expected.
>>
>> How about adding your own software later as Whirr services or using
>> bash scripts and start from a vanilla Ubuntu ami?
>>
>> -- Andrei Savu / andreisavu.ro
>>
>> On Wed, Aug 24, 2011 at 11:41 PM, Joris Poort <gp...@gmail.com> wrote:
>>> Thanks for offering your help Guillaume. Unfortunately I tried that
>>> and still have the same issues.
>>>
>>> On Wed, Aug 24, 2011 at 11:34 PM, tog <gu...@gmail.com> wrote:
>>>> I don't know if this can help but here are the 2 lines that I added to
>>>> my cluster property file in order to be able to log on the cluster:
>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>>>
>>>> then I was able to connect my local userid and not ubuntu.
>>>>
>>>> HTH
>>>> Guillaume
>>>>
>>>>
>>>> On Thu, Aug 25, 2011 at 11:34 AM, Joris Poort <gp...@gmail.com> wrote:
>>>>> I also tried to regenerate the ssh keys, but still no luck...
>>>>>
>>>>> On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> Interestingly, as mentioned before I can ssh in using my keypair used
>>>>>> to create the custom AMI. So I'm thinking that it still has to do
>>>>>> with whirr not being able to get the right keypair access with the
>>>>>> local id_rsa.
>>>>>>
>>>>>> Thanks again for any help,
>>>>>>
>>>>>> Joris
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>>>>>>> adjusted to local-user "gpoort" I'm still not getting through. See
>>>>>>> output below from -vv (sorry about the long email).
>>>>>>>
>>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>>> debug1: Applying options for *
>>>>>>> debug2: ssh_connect: needpriv 0
>>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>> [184.72.86.108] port 22.
>>>>>>> debug1: Connection established.
>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>>> debug2: dh_gen_key: priv key bits set: 138/256
>>>>>>> debug2: bits set: 502/1024
>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>>> matches the RSA host key.
>>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>>> debug2: bits set: 497/1024
>>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>>> debug2: kex_derive_keys
>>>>>>> debug2: set_newkeys: mode 1
>>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>>> debug2: set_newkeys: mode 0
>>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>>> debug1: Roaming not allowed by server
>>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>>> debug2: service_accept: ssh-userauth
>>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>> debug1: Next authentication method: publickey
>>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>> debug2: we did not send a packet, disable method
>>>>>>> debug1: No more authentication methods to try.
>>>>>>> Permission denied (publickey).
>>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>>> debug1: Applying options for *
>>>>>>> debug2: ssh_connect: needpriv 0
>>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>>> [184.72.86.108] port 22.
>>>>>>> debug1: Connection established.
>>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit:
>>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>>> debug2: mac_setup: found hmac-md5
>>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>>> debug2: dh_gen_key: priv key bits set: 145/256
>>>>>>> debug2: bits set: 505/1024
>>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>>> matches the RSA host key.
>>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>>> debug2: bits set: 495/1024
>>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>>> debug2: kex_derive_keys
>>>>>>> debug2: set_newkeys: mode 1
>>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>>> debug2: set_newkeys: mode 0
>>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>>> debug1: Roaming not allowed by server
>>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>>> debug2: service_accept: ssh-userauth
>>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>> debug1: Next authentication method: publickey
>>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>>> debug1: Authentications that can continue: publickey
>>>>>>> debug2: we did not send a packet, disable method
>>>>>>> debug1: No more authentication methods to try.
>>>>>>> Permission denied (publickey).
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>> You should be able to login user the same user that is running Whirr.
>>>>>>>>
>>>>>>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>>>>>>
>>>>>>>> Permission denied most of the time means invalid key, user pair.
>>>>>>>>
>>>>>>>> It should be ok to use the keys generate by default.
>>>>>>>>
>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>> I'm just using the defaults. But you may be onto the problem here,
>>>>>>>>> when I try to ssh into the node using:
>>>>>>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>>>>>>
>>>>>>>>> I get a permission denied.
>>>>>>>>>
>>>>>>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>>>>>>
>>>>>>>>> Thanks again,
>>>>>>>>>
>>>>>>>>> Joris
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>>>>>>
>>>>>>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>>>>>>
>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>> I think you're probably right its an auth issue - although I was
>>>>>>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>>>>>>> working.
>>>>>>>>>>>
>>>>>>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>>>>>>> with the same results. My understanding is that the keypair used to
>>>>>>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>>>>>>> anywhere that can be used to test this?
>>>>>>>>>>>
>>>>>>>>>>> Thanks again for your help,
>>>>>>>>>>>
>>>>>>>>>>> Joris
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>>>>>>> file.
>>>>>>>>>>>>
>>>>>>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>>>>>>> that custom ami with a different key pair?
>>>>>>>>>>>>
>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>> Andrei - thanks for the response!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>>>>>>> and "Dying because" are not always there):
>>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>>>>>>> Broken transport; encountered EOF
>>>>>>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>>
>>>>>>>>>>>>> Last few entries on whirr.log:
>>>>>>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>>>>>>> securityGroupIds=[],
>>>>>>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>>>>>>> seconds
>>>>>>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>>>>>>
>>>>>>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks again for any help on this!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>>>>>>> same behavior with regular ones?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Thanks for the recommendation on whirr services code, I think that
will work - looking into how to do it now.
Thanks again Andrei - really appreciate your help on all this.
Cheers,
Joris
On Wed, Aug 24, 2011 at 11:49 PM, Andrei Savu <sa...@gmail.com> wrote:
> It seems like for your custom ami the user scripts are not executed as expected.
>
> How about adding your own software later as Whirr services or using
> bash scripts and start from a vanilla Ubuntu ami?
>
> -- Andrei Savu / andreisavu.ro
>
> On Wed, Aug 24, 2011 at 11:41 PM, Joris Poort <gp...@gmail.com> wrote:
>> Thanks for offering your help Guillaume. Unfortunately I tried that
>> and still have the same issues.
>>
>> On Wed, Aug 24, 2011 at 11:34 PM, tog <gu...@gmail.com> wrote:
>>> I don't know if this can help but here are the 2 lines that I added to
>>> my cluster property file in order to be able to log on the cluster:
>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>>
>>> then I was able to connect my local userid and not ubuntu.
>>>
>>> HTH
>>> Guillaume
>>>
>>>
>>> On Thu, Aug 25, 2011 at 11:34 AM, Joris Poort <gp...@gmail.com> wrote:
>>>> I also tried to regenerate the ssh keys, but still no luck...
>>>>
>>>> On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> Interestingly, as mentioned before I can ssh in using my keypair used
>>>>> to create the custom AMI. So I'm thinking that it still has to do
>>>>> with whirr not being able to get the right keypair access with the
>>>>> local id_rsa.
>>>>>
>>>>> Thanks again for any help,
>>>>>
>>>>> Joris
>>>>>
>>>>> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>>>>>> adjusted to local-user "gpoort" I'm still not getting through. See
>>>>>> output below from -vv (sorry about the long email).
>>>>>>
>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>> debug1: Applying options for *
>>>>>> debug2: ssh_connect: needpriv 0
>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>> [184.72.86.108] port 22.
>>>>>> debug1: Connection established.
>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>> debug2: kex_parse_kexinit:
>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>> debug2: kex_parse_kexinit:
>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>> debug2: kex_parse_kexinit:
>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>> debug2: mac_setup: found hmac-md5
>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>> debug2: mac_setup: found hmac-md5
>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>> debug2: dh_gen_key: priv key bits set: 138/256
>>>>>> debug2: bits set: 502/1024
>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>> matches the RSA host key.
>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>> debug2: bits set: 497/1024
>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>> debug2: kex_derive_keys
>>>>>> debug2: set_newkeys: mode 1
>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>> debug2: set_newkeys: mode 0
>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>> debug1: Roaming not allowed by server
>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>> debug2: service_accept: ssh-userauth
>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>>>>>> debug1: Authentications that can continue: publickey
>>>>>> debug1: Next authentication method: publickey
>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>> debug1: Authentications that can continue: publickey
>>>>>> debug2: we did not send a packet, disable method
>>>>>> debug1: No more authentication methods to try.
>>>>>> Permission denied (publickey).
>>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>>> debug1: Applying options for *
>>>>>> debug2: ssh_connect: needpriv 0
>>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>>> [184.72.86.108] port 22.
>>>>>> debug1: Connection established.
>>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>>> debug2: kex_parse_kexinit:
>>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>> debug2: kex_parse_kexinit:
>>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>> debug2: kex_parse_kexinit:
>>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit:
>>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit:
>>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>>> debug2: mac_setup: found hmac-md5
>>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>>> debug2: mac_setup: found hmac-md5
>>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>>> debug2: dh_gen_key: priv key bits set: 145/256
>>>>>> debug2: bits set: 505/1024
>>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>>> matches the RSA host key.
>>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>>> debug2: bits set: 495/1024
>>>>>> debug1: ssh_rsa_verify: signature correct
>>>>>> debug2: kex_derive_keys
>>>>>> debug2: set_newkeys: mode 1
>>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>>> debug2: set_newkeys: mode 0
>>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>>> debug1: Roaming not allowed by server
>>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>>> debug2: service_accept: ssh-userauth
>>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>>>>>> debug1: Authentications that can continue: publickey
>>>>>> debug1: Next authentication method: publickey
>>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>>> debug2: we sent a publickey packet, wait for reply
>>>>>> debug1: Authentications that can continue: publickey
>>>>>> debug2: we did not send a packet, disable method
>>>>>> debug1: No more authentication methods to try.
>>>>>> Permission denied (publickey).
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>> You should be able to login user the same user that is running Whirr.
>>>>>>>
>>>>>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>>>>>
>>>>>>> Permission denied most of the time means invalid key, user pair.
>>>>>>>
>>>>>>> It should be ok to use the keys generate by default.
>>>>>>>
>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>> I'm just using the defaults. But you may be onto the problem here,
>>>>>>>> when I try to ssh into the node using:
>>>>>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>>>>>
>>>>>>>> I get a permission denied.
>>>>>>>>
>>>>>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>>>>>
>>>>>>>> Thanks again,
>>>>>>>>
>>>>>>>> Joris
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>>>>>
>>>>>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>>>>>
>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>> I think you're probably right its an auth issue - although I was
>>>>>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>>>>>> working.
>>>>>>>>>>
>>>>>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>>>>>> with the same results. My understanding is that the keypair used to
>>>>>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>>>>>> anywhere that can be used to test this?
>>>>>>>>>>
>>>>>>>>>> Thanks again for your help,
>>>>>>>>>>
>>>>>>>>>> Joris
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>>>>>> file.
>>>>>>>>>>>
>>>>>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>>>>>> that custom ami with a different key pair?
>>>>>>>>>>>
>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>> Andrei - thanks for the response!
>>>>>>>>>>>>
>>>>>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>>>>>> behavior.
>>>>>>>>>>>>
>>>>>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>>>>>> and "Dying because" are not always there):
>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>>>>>> Broken transport; encountered EOF
>>>>>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>>
>>>>>>>>>>>> Last few entries on whirr.log:
>>>>>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>>>>>> securityGroupIds=[],
>>>>>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>>>>>> seconds
>>>>>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>>>>>
>>>>>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks again for any help on this!
>>>>>>>>>>>>
>>>>>>>>>>>> Joris
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>>>>>> same behavior with regular ones?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Andrei Savu <sa...@gmail.com>.
It seems like for your custom ami the user scripts are not executed as expected.
How about adding your own software later as Whirr services or using
bash scripts and start from a vanilla Ubuntu ami?
-- Andrei Savu / andreisavu.ro
On Wed, Aug 24, 2011 at 11:41 PM, Joris Poort <gp...@gmail.com> wrote:
> Thanks for offering your help Guillaume. Unfortunately I tried that
> and still have the same issues.
>
> On Wed, Aug 24, 2011 at 11:34 PM, tog <gu...@gmail.com> wrote:
>> I don't know if this can help but here are the 2 lines that I added to
>> my cluster property file in order to be able to log on the cluster:
>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>
>> then I was able to connect my local userid and not ubuntu.
>>
>> HTH
>> Guillaume
>>
>>
>> On Thu, Aug 25, 2011 at 11:34 AM, Joris Poort <gp...@gmail.com> wrote:
>>> I also tried to regenerate the ssh keys, but still no luck...
>>>
>>> On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> Interestingly, as mentioned before I can ssh in using my keypair used
>>>> to create the custom AMI. So I'm thinking that it still has to do
>>>> with whirr not being able to get the right keypair access with the
>>>> local id_rsa.
>>>>
>>>> Thanks again for any help,
>>>>
>>>> Joris
>>>>
>>>> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>>>>> adjusted to local-user "gpoort" I'm still not getting through. See
>>>>> output below from -vv (sorry about the long email).
>>>>>
>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>> debug1: Applying options for *
>>>>> debug2: ssh_connect: needpriv 0
>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>> [184.72.86.108] port 22.
>>>>> debug1: Connection established.
>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>> debug2: kex_parse_kexinit:
>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>> debug2: kex_parse_kexinit:
>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>> debug2: kex_parse_kexinit:
>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>> debug2: mac_setup: found hmac-md5
>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>> debug2: mac_setup: found hmac-md5
>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>> debug2: dh_gen_key: priv key bits set: 138/256
>>>>> debug2: bits set: 502/1024
>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>> matches the RSA host key.
>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>> debug2: bits set: 497/1024
>>>>> debug1: ssh_rsa_verify: signature correct
>>>>> debug2: kex_derive_keys
>>>>> debug2: set_newkeys: mode 1
>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>> debug2: set_newkeys: mode 0
>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>> debug1: Roaming not allowed by server
>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>> debug2: service_accept: ssh-userauth
>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>>>>> debug1: Authentications that can continue: publickey
>>>>> debug1: Next authentication method: publickey
>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>> debug2: we sent a publickey packet, wait for reply
>>>>> debug1: Authentications that can continue: publickey
>>>>> debug2: we did not send a packet, disable method
>>>>> debug1: No more authentication methods to try.
>>>>> Permission denied (publickey).
>>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>> debug1: Applying options for *
>>>>> debug2: ssh_connect: needpriv 0
>>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>>> [184.72.86.108] port 22.
>>>>> debug1: Connection established.
>>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>>> debug1: identity file .ssh/id_rsa type 1
>>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>>> debug1: Remote protocol version 2.0, remote software version
>>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>>> debug2: fd 3 setting O_NONBLOCK
>>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>> debug1: SSH2_MSG_KEXINIT received
>>>>> debug2: kex_parse_kexinit:
>>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>> debug2: kex_parse_kexinit:
>>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>> debug2: kex_parse_kexinit:
>>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit:
>>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit:
>>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>>> debug2: kex_parse_kexinit: reserved 0
>>>>> debug2: mac_setup: found hmac-md5
>>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>>> debug2: mac_setup: found hmac-md5
>>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>> debug2: dh_gen_key: priv key bits set: 145/256
>>>>> debug2: bits set: 505/1024
>>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>>> matches the RSA host key.
>>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>>> debug2: bits set: 495/1024
>>>>> debug1: ssh_rsa_verify: signature correct
>>>>> debug2: kex_derive_keys
>>>>> debug2: set_newkeys: mode 1
>>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>> debug2: set_newkeys: mode 0
>>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>> debug1: Roaming not allowed by server
>>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>> debug2: service_accept: ssh-userauth
>>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>>>>> debug1: Authentications that can continue: publickey
>>>>> debug1: Next authentication method: publickey
>>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>>> debug2: we sent a publickey packet, wait for reply
>>>>> debug1: Authentications that can continue: publickey
>>>>> debug2: we did not send a packet, disable method
>>>>> debug1: No more authentication methods to try.
>>>>> Permission denied (publickey).
>>>>>
>>>>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>> You should be able to login user the same user that is running Whirr.
>>>>>>
>>>>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>>>>
>>>>>> Permission denied most of the time means invalid key, user pair.
>>>>>>
>>>>>> It should be ok to use the keys generate by default.
>>>>>>
>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>> I'm just using the defaults. But you may be onto the problem here,
>>>>>>> when I try to ssh into the node using:
>>>>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>>>>
>>>>>>> I get a permission denied.
>>>>>>>
>>>>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>>>>
>>>>>>> Thanks again,
>>>>>>>
>>>>>>> Joris
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>>>>
>>>>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>>>>
>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>> I think you're probably right its an auth issue - although I was
>>>>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>>>>> working.
>>>>>>>>>
>>>>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>>>>> with the same results. My understanding is that the keypair used to
>>>>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>>>>> anywhere that can be used to test this?
>>>>>>>>>
>>>>>>>>> Thanks again for your help,
>>>>>>>>>
>>>>>>>>> Joris
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>>>>> file.
>>>>>>>>>>
>>>>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>>>>> that custom ami with a different key pair?
>>>>>>>>>>
>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>> Andrei - thanks for the response!
>>>>>>>>>>>
>>>>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>>>>> behavior.
>>>>>>>>>>>
>>>>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>>>>> and "Dying because" are not always there):
>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>>> transport; encountered EOF
>>>>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>>>>> Broken transport; encountered EOF
>>>>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>>
>>>>>>>>>>> Last few entries on whirr.log:
>>>>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>>>>> securityGroupIds=[],
>>>>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>>>>> seconds
>>>>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>>>>
>>>>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>>>>
>>>>>>>>>>> Thanks again for any help on this!
>>>>>>>>>>>
>>>>>>>>>>> Joris
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>>>>
>>>>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>>>>
>>>>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>>>>> same behavior with regular ones?
>>>>>>>>>>>>
>>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>>>>
>>>>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Joris
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Thanks for offering your help Guillaume. Unfortunately I tried that
and still have the same issues.
On Wed, Aug 24, 2011 at 11:34 PM, tog <gu...@gmail.com> wrote:
> I don't know if this can help but here are the 2 lines that I added to
> my cluster property file in order to be able to log on the cluster:
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>
> then I was able to connect my local userid and not ubuntu.
>
> HTH
> Guillaume
>
>
> On Thu, Aug 25, 2011 at 11:34 AM, Joris Poort <gp...@gmail.com> wrote:
>> I also tried to regenerate the ssh keys, but still no luck...
>>
>> On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
>>> Interestingly, as mentioned before I can ssh in using my keypair used
>>> to create the custom AMI. So I'm thinking that it still has to do
>>> with whirr not being able to get the right keypair access with the
>>> local id_rsa.
>>>
>>> Thanks again for any help,
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>>>> adjusted to local-user "gpoort" I'm still not getting through. See
>>>> output below from -vv (sorry about the long email).
>>>>
>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>> debug1: Applying options for *
>>>> debug2: ssh_connect: needpriv 0
>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>> [184.72.86.108] port 22.
>>>> debug1: Connection established.
>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>> debug1: identity file .ssh/id_rsa type 1
>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>> debug1: Remote protocol version 2.0, remote software version
>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>> debug2: fd 3 setting O_NONBLOCK
>>>> debug1: SSH2_MSG_KEXINIT sent
>>>> debug1: SSH2_MSG_KEXINIT received
>>>> debug2: kex_parse_kexinit:
>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>> debug2: kex_parse_kexinit:
>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>> debug2: kex_parse_kexinit: reserved 0
>>>> debug2: kex_parse_kexinit:
>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>> debug2: kex_parse_kexinit: reserved 0
>>>> debug2: mac_setup: found hmac-md5
>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>> debug2: mac_setup: found hmac-md5
>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>> debug2: dh_gen_key: priv key bits set: 138/256
>>>> debug2: bits set: 502/1024
>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>> matches the RSA host key.
>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>> debug2: bits set: 497/1024
>>>> debug1: ssh_rsa_verify: signature correct
>>>> debug2: kex_derive_keys
>>>> debug2: set_newkeys: mode 1
>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>> debug2: set_newkeys: mode 0
>>>> debug1: SSH2_MSG_NEWKEYS received
>>>> debug1: Roaming not allowed by server
>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>> debug2: service_accept: ssh-userauth
>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>>>> debug1: Authentications that can continue: publickey
>>>> debug1: Next authentication method: publickey
>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>> debug2: we sent a publickey packet, wait for reply
>>>> debug1: Authentications that can continue: publickey
>>>> debug2: we did not send a packet, disable method
>>>> debug1: No more authentication methods to try.
>>>> Permission denied (publickey).
>>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>> debug1: Applying options for *
>>>> debug2: ssh_connect: needpriv 0
>>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>>> [184.72.86.108] port 22.
>>>> debug1: Connection established.
>>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>>> debug2: key_type_from_name: unknown key type '-----END'
>>>> debug1: identity file .ssh/id_rsa type 1
>>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>>> debug1: identity file .ssh/id_rsa-cert type -1
>>>> debug1: Remote protocol version 2.0, remote software version
>>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>>> debug2: fd 3 setting O_NONBLOCK
>>>> debug1: SSH2_MSG_KEXINIT sent
>>>> debug1: SSH2_MSG_KEXINIT received
>>>> debug2: kex_parse_kexinit:
>>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>> debug2: kex_parse_kexinit:
>>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>> debug2: kex_parse_kexinit: reserved 0
>>>> debug2: kex_parse_kexinit:
>>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit:
>>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit:
>>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>>> debug2: kex_parse_kexinit: reserved 0
>>>> debug2: mac_setup: found hmac-md5
>>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>>> debug2: mac_setup: found hmac-md5
>>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>> debug2: dh_gen_key: priv key bits set: 145/256
>>>> debug2: bits set: 505/1024
>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>>> matches the RSA host key.
>>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>>> debug2: bits set: 495/1024
>>>> debug1: ssh_rsa_verify: signature correct
>>>> debug2: kex_derive_keys
>>>> debug2: set_newkeys: mode 1
>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>> debug2: set_newkeys: mode 0
>>>> debug1: SSH2_MSG_NEWKEYS received
>>>> debug1: Roaming not allowed by server
>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>> debug2: service_accept: ssh-userauth
>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>>>> debug1: Authentications that can continue: publickey
>>>> debug1: Next authentication method: publickey
>>>> debug1: Offering RSA public key: .ssh/id_rsa
>>>> debug2: we sent a publickey packet, wait for reply
>>>> debug1: Authentications that can continue: publickey
>>>> debug2: we did not send a packet, disable method
>>>> debug1: No more authentication methods to try.
>>>> Permission denied (publickey).
>>>>
>>>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>> You should be able to login user the same user that is running Whirr.
>>>>>
>>>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>>>
>>>>> Permission denied most of the time means invalid key, user pair.
>>>>>
>>>>> It should be ok to use the keys generate by default.
>>>>>
>>>>> -- Andrei Savu / andreisavu.ro
>>>>>
>>>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> I'm just using the defaults. But you may be onto the problem here,
>>>>>> when I try to ssh into the node using:
>>>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>>>
>>>>>> I get a permission denied.
>>>>>>
>>>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> Joris
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>>>
>>>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>>>
>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>> I think you're probably right its an auth issue - although I was
>>>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>>>> working.
>>>>>>>>
>>>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>>>> with the same results. My understanding is that the keypair used to
>>>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>>>> anywhere that can be used to test this?
>>>>>>>>
>>>>>>>> Thanks again for your help,
>>>>>>>>
>>>>>>>> Joris
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>>>> file.
>>>>>>>>>
>>>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>>>> that custom ami with a different key pair?
>>>>>>>>>
>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>> Andrei - thanks for the response!
>>>>>>>>>>
>>>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>>>> behavior.
>>>>>>>>>>
>>>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>>>> and "Dying because" are not always there):
>>>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>> transport; encountered EOF
>>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>>> transport; encountered EOF
>>>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>>>> Broken transport; encountered EOF
>>>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>>
>>>>>>>>>> Last few entries on whirr.log:
>>>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>>>> securityGroupIds=[],
>>>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>>>> seconds
>>>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>>>
>>>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>>>
>>>>>>>>>> Thanks again for any help on this!
>>>>>>>>>>
>>>>>>>>>> Joris
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>>>
>>>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>>>
>>>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>>>> same behavior with regular ones?
>>>>>>>>>>>
>>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>>>
>>>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>>>
>>>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>
>>>>>>>>>>>> Joris
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
>
> --
> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
>
Re: EC2 Custom AMI's connection issue
Posted by tog <gu...@gmail.com>.
I don't know if this can help but here are the 2 lines that I added to
my cluster property file in order to be able to log on the cluster:
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
then I was able to connect my local userid and not ubuntu.
HTH
Guillaume
On Thu, Aug 25, 2011 at 11:34 AM, Joris Poort <gp...@gmail.com> wrote:
> I also tried to regenerate the ssh keys, but still no luck...
>
> On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
>> Interestingly, as mentioned before I can ssh in using my keypair used
>> to create the custom AMI. So I'm thinking that it still has to do
>> with whirr not being able to get the right keypair access with the
>> local id_rsa.
>>
>> Thanks again for any help,
>>
>> Joris
>>
>> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>>> adjusted to local-user "gpoort" I'm still not getting through. See
>>> output below from -vv (sorry about the long email).
>>>
>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>> debug1: Applying options for *
>>> debug2: ssh_connect: needpriv 0
>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>> [184.72.86.108] port 22.
>>> debug1: Connection established.
>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>> debug2: key_type_from_name: unknown key type '-----END'
>>> debug1: identity file .ssh/id_rsa type 1
>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>> debug1: identity file .ssh/id_rsa-cert type -1
>>> debug1: Remote protocol version 2.0, remote software version
>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>> debug1: Enabling compatibility mode for protocol 2.0
>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>> debug2: fd 3 setting O_NONBLOCK
>>> debug1: SSH2_MSG_KEXINIT sent
>>> debug1: SSH2_MSG_KEXINIT received
>>> debug2: kex_parse_kexinit:
>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>> debug2: kex_parse_kexinit:
>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>> debug2: kex_parse_kexinit: reserved 0
>>> debug2: kex_parse_kexinit:
>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>> debug2: kex_parse_kexinit: reserved 0
>>> debug2: mac_setup: found hmac-md5
>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>> debug2: mac_setup: found hmac-md5
>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>> debug2: dh_gen_key: priv key bits set: 138/256
>>> debug2: bits set: 502/1024
>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>> matches the RSA host key.
>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>> debug2: bits set: 497/1024
>>> debug1: ssh_rsa_verify: signature correct
>>> debug2: kex_derive_keys
>>> debug2: set_newkeys: mode 1
>>> debug1: SSH2_MSG_NEWKEYS sent
>>> debug1: expecting SSH2_MSG_NEWKEYS
>>> debug2: set_newkeys: mode 0
>>> debug1: SSH2_MSG_NEWKEYS received
>>> debug1: Roaming not allowed by server
>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>> debug2: service_accept: ssh-userauth
>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>>> debug1: Authentications that can continue: publickey
>>> debug1: Next authentication method: publickey
>>> debug1: Offering RSA public key: .ssh/id_rsa
>>> debug2: we sent a publickey packet, wait for reply
>>> debug1: Authentications that can continue: publickey
>>> debug2: we did not send a packet, disable method
>>> debug1: No more authentication methods to try.
>>> Permission denied (publickey).
>>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>> debug1: Applying options for *
>>> debug2: ssh_connect: needpriv 0
>>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>>> [184.72.86.108] port 22.
>>> debug1: Connection established.
>>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>>> debug2: key_type_from_name: unknown key type '-----END'
>>> debug1: identity file .ssh/id_rsa type 1
>>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>>> debug1: identity file .ssh/id_rsa-cert type -1
>>> debug1: Remote protocol version 2.0, remote software version
>>> OpenSSH_5.3p1 Debian-3ubuntu7
>>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>>> debug1: Enabling compatibility mode for protocol 2.0
>>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>>> debug2: fd 3 setting O_NONBLOCK
>>> debug1: SSH2_MSG_KEXINIT sent
>>> debug1: SSH2_MSG_KEXINIT received
>>> debug2: kex_parse_kexinit:
>>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>> debug2: kex_parse_kexinit:
>>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>> debug2: kex_parse_kexinit: reserved 0
>>> debug2: kex_parse_kexinit:
>>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit:
>>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit:
>>> debug2: kex_parse_kexinit: first_kex_follows 0
>>> debug2: kex_parse_kexinit: reserved 0
>>> debug2: mac_setup: found hmac-md5
>>> debug1: kex: server->client aes128-ctr hmac-md5 none
>>> debug2: mac_setup: found hmac-md5
>>> debug1: kex: client->server aes128-ctr hmac-md5 none
>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>> debug2: dh_gen_key: priv key bits set: 145/256
>>> debug2: bits set: 505/1024
>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>>> matches the RSA host key.
>>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>>> debug2: bits set: 495/1024
>>> debug1: ssh_rsa_verify: signature correct
>>> debug2: kex_derive_keys
>>> debug2: set_newkeys: mode 1
>>> debug1: SSH2_MSG_NEWKEYS sent
>>> debug1: expecting SSH2_MSG_NEWKEYS
>>> debug2: set_newkeys: mode 0
>>> debug1: SSH2_MSG_NEWKEYS received
>>> debug1: Roaming not allowed by server
>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>> debug2: service_accept: ssh-userauth
>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>>> debug1: Authentications that can continue: publickey
>>> debug1: Next authentication method: publickey
>>> debug1: Offering RSA public key: .ssh/id_rsa
>>> debug2: we sent a publickey packet, wait for reply
>>> debug1: Authentications that can continue: publickey
>>> debug2: we did not send a packet, disable method
>>> debug1: No more authentication methods to try.
>>> Permission denied (publickey).
>>>
>>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>> You should be able to login user the same user that is running Whirr.
>>>>
>>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>>
>>>> Permission denied most of the time means invalid key, user pair.
>>>>
>>>> It should be ok to use the keys generate by default.
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> I'm just using the defaults. But you may be onto the problem here,
>>>>> when I try to ssh into the node using:
>>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>>
>>>>> I get a permission denied.
>>>>>
>>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>>
>>>>> Thanks again,
>>>>>
>>>>> Joris
>>>>>
>>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>>
>>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>>
>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>> I think you're probably right its an auth issue - although I was
>>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>>> working.
>>>>>>>
>>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>>> with the same results. My understanding is that the keypair used to
>>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>>> anywhere that can be used to test this?
>>>>>>>
>>>>>>> Thanks again for your help,
>>>>>>>
>>>>>>> Joris
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>>> file.
>>>>>>>>
>>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>>> that custom ami with a different key pair?
>>>>>>>>
>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>> Andrei - thanks for the response!
>>>>>>>>>
>>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>>> behavior.
>>>>>>>>>
>>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>>> and "Dying because" are not always there):
>>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>> transport; encountered EOF
>>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>>> transport; encountered EOF
>>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>>> Broken transport; encountered EOF
>>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>>
>>>>>>>>> Last few entries on whirr.log:
>>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>>> securityGroupIds=[],
>>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>>> seconds
>>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>>
>>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>>
>>>>>>>>> Thanks again for any help on this!
>>>>>>>>>
>>>>>>>>> Joris
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>>
>>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>>
>>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>>> same behavior with regular ones?
>>>>>>>>>>
>>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>>
>>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>>
>>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Joris
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
I also tried to regenerate the ssh keys, but still no luck...
On Wed, Aug 24, 2011 at 10:13 PM, Joris Poort <gp...@gmail.com> wrote:
> Interestingly, as mentioned before I can ssh in using my keypair used
> to create the custom AMI. So I'm thinking that it still has to do
> with whirr not being able to get the right keypair access with the
> local id_rsa.
>
> Thanks again for any help,
>
> Joris
>
> On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
>> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
>> adjusted to local-user "gpoort" I'm still not getting through. See
>> output below from -vv (sorry about the long email).
>>
>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>> debug1: Reading configuration data /etc/ssh/ssh_config
>> debug1: Applying options for *
>> debug2: ssh_connect: needpriv 0
>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>> [184.72.86.108] port 22.
>> debug1: Connection established.
>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>> debug2: key_type_from_name: unknown key type '-----END'
>> debug1: identity file .ssh/id_rsa type 1
>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>> debug1: identity file .ssh/id_rsa-cert type -1
>> debug1: Remote protocol version 2.0, remote software version
>> OpenSSH_5.3p1 Debian-3ubuntu7
>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>> debug1: Enabling compatibility mode for protocol 2.0
>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>> debug2: fd 3 setting O_NONBLOCK
>> debug1: SSH2_MSG_KEXINIT sent
>> debug1: SSH2_MSG_KEXINIT received
>> debug2: kex_parse_kexinit:
>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>> debug2: kex_parse_kexinit:
>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit: first_kex_follows 0
>> debug2: kex_parse_kexinit: reserved 0
>> debug2: kex_parse_kexinit:
>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit: first_kex_follows 0
>> debug2: kex_parse_kexinit: reserved 0
>> debug2: mac_setup: found hmac-md5
>> debug1: kex: server->client aes128-ctr hmac-md5 none
>> debug2: mac_setup: found hmac-md5
>> debug1: kex: client->server aes128-ctr hmac-md5 none
>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>> debug2: dh_gen_key: priv key bits set: 138/256
>> debug2: bits set: 502/1024
>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>> matches the RSA host key.
>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>> debug2: bits set: 497/1024
>> debug1: ssh_rsa_verify: signature correct
>> debug2: kex_derive_keys
>> debug2: set_newkeys: mode 1
>> debug1: SSH2_MSG_NEWKEYS sent
>> debug1: expecting SSH2_MSG_NEWKEYS
>> debug2: set_newkeys: mode 0
>> debug1: SSH2_MSG_NEWKEYS received
>> debug1: Roaming not allowed by server
>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>> debug2: service_accept: ssh-userauth
>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
>> debug1: Authentications that can continue: publickey
>> debug1: Next authentication method: publickey
>> debug1: Offering RSA public key: .ssh/id_rsa
>> debug2: we sent a publickey packet, wait for reply
>> debug1: Authentications that can continue: publickey
>> debug2: we did not send a packet, disable method
>> debug1: No more authentication methods to try.
>> Permission denied (publickey).
>> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
>> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
>> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
>> debug1: Reading configuration data /etc/ssh/ssh_config
>> debug1: Applying options for *
>> debug2: ssh_connect: needpriv 0
>> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
>> [184.72.86.108] port 22.
>> debug1: Connection established.
>> debug2: key_type_from_name: unknown key type '-----BEGIN'
>> debug2: key_type_from_name: unknown key type '-----END'
>> debug1: identity file .ssh/id_rsa type 1
>> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
>> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
>> debug1: identity file .ssh/id_rsa-cert type -1
>> debug1: Remote protocol version 2.0, remote software version
>> OpenSSH_5.3p1 Debian-3ubuntu7
>> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
>> debug1: Enabling compatibility mode for protocol 2.0
>> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
>> debug2: fd 3 setting O_NONBLOCK
>> debug1: SSH2_MSG_KEXINIT sent
>> debug1: SSH2_MSG_KEXINIT received
>> debug2: kex_parse_kexinit:
>> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>> debug2: kex_parse_kexinit:
>> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit: first_kex_follows 0
>> debug2: kex_parse_kexinit: reserved 0
>> debug2: kex_parse_kexinit:
>> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit:
>> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>> debug2: kex_parse_kexinit: none,zlib@openssh.com
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit:
>> debug2: kex_parse_kexinit: first_kex_follows 0
>> debug2: kex_parse_kexinit: reserved 0
>> debug2: mac_setup: found hmac-md5
>> debug1: kex: server->client aes128-ctr hmac-md5 none
>> debug2: mac_setup: found hmac-md5
>> debug1: kex: client->server aes128-ctr hmac-md5 none
>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>> debug2: dh_gen_key: priv key bits set: 145/256
>> debug2: bits set: 505/1024
>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
>> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
>> matches the RSA host key.
>> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
>> debug2: bits set: 495/1024
>> debug1: ssh_rsa_verify: signature correct
>> debug2: kex_derive_keys
>> debug2: set_newkeys: mode 1
>> debug1: SSH2_MSG_NEWKEYS sent
>> debug1: expecting SSH2_MSG_NEWKEYS
>> debug2: set_newkeys: mode 0
>> debug1: SSH2_MSG_NEWKEYS received
>> debug1: Roaming not allowed by server
>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>> debug2: service_accept: ssh-userauth
>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
>> debug1: Authentications that can continue: publickey
>> debug1: Next authentication method: publickey
>> debug1: Offering RSA public key: .ssh/id_rsa
>> debug2: we sent a publickey packet, wait for reply
>> debug1: Authentications that can continue: publickey
>> debug2: we did not send a packet, disable method
>> debug1: No more authentication methods to try.
>> Permission denied (publickey).
>>
>> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>>> You should be able to login user the same user that is running Whirr.
>>>
>>> ssh -i ~/.ssh/id_rsa local-user@server
>>>
>>> Permission denied most of the time means invalid key, user pair.
>>>
>>> It should be ok to use the keys generate by default.
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> I'm just using the defaults. But you may be onto the problem here,
>>>> when I try to ssh into the node using:
>>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>>
>>>> I get a permission denied.
>>>>
>>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>>
>>>> Thanks again,
>>>>
>>>> Joris
>>>>
>>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>>
>>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>>
>>>>> -- Andrei Savu / andreisavu.ro
>>>>>
>>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> I think you're probably right its an auth issue - although I was
>>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>>> working.
>>>>>>
>>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>>> with the same results. My understanding is that the keypair used to
>>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>>> anywhere that can be used to test this?
>>>>>>
>>>>>> Thanks again for your help,
>>>>>>
>>>>>> Joris
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>>> file.
>>>>>>>
>>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>>> that custom ami with a different key pair?
>>>>>>>
>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>> Andrei - thanks for the response!
>>>>>>>>
>>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>>> behavior.
>>>>>>>>
>>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>>> and "Dying because" are not always there):
>>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>> transport; encountered EOF
>>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>>> transport; encountered EOF
>>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>>> Broken transport; encountered EOF
>>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>>
>>>>>>>> Last few entries on whirr.log:
>>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>>> securityGroupIds=[],
>>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>>> seconds
>>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>>
>>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>>
>>>>>>>> Thanks again for any help on this!
>>>>>>>>
>>>>>>>> Joris
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>>
>>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>>
>>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>>> same behavior with regular ones?
>>>>>>>>>
>>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>>
>>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>>
>>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>>> Would really appreciate it!
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Joris
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Interestingly, as mentioned before I can ssh in using my keypair used
to create the custom AMI. So I'm thinking that it still has to do
with whirr not being able to get the right keypair access with the
local id_rsa.
Thanks again for any help,
Joris
On Wed, Aug 24, 2011 at 10:11 PM, Joris Poort <gp...@gmail.com> wrote:
> Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
> adjusted to local-user "gpoort" I'm still not getting through. See
> output below from -vv (sorry about the long email).
>
> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
> ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: Applying options for *
> debug2: ssh_connect: needpriv 0
> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
> [184.72.86.108] port 22.
> debug1: Connection established.
> debug2: key_type_from_name: unknown key type '-----BEGIN'
> debug2: key_type_from_name: unknown key type '-----END'
> debug1: identity file .ssh/id_rsa type 1
> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
> debug1: identity file .ssh/id_rsa-cert type -1
> debug1: Remote protocol version 2.0, remote software version
> OpenSSH_5.3p1 Debian-3ubuntu7
> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
> debug2: fd 3 setting O_NONBLOCK
> debug1: SSH2_MSG_KEXINIT sent
> debug1: SSH2_MSG_KEXINIT received
> debug2: kex_parse_kexinit:
> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit:
> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: kex_parse_kexinit:
> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: mac_setup: found hmac-md5
> debug1: kex: server->client aes128-ctr hmac-md5 none
> debug2: mac_setup: found hmac-md5
> debug1: kex: client->server aes128-ctr hmac-md5 none
> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
> debug2: dh_gen_key: priv key bits set: 138/256
> debug2: bits set: 502/1024
> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
> matches the RSA host key.
> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
> debug2: bits set: 497/1024
> debug1: ssh_rsa_verify: signature correct
> debug2: kex_derive_keys
> debug2: set_newkeys: mode 1
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug2: set_newkeys: mode 0
> debug1: SSH2_MSG_NEWKEYS received
> debug1: Roaming not allowed by server
> debug1: SSH2_MSG_SERVICE_REQUEST sent
> debug2: service_accept: ssh-userauth
> debug1: SSH2_MSG_SERVICE_ACCEPT received
> debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
> debug1: Authentications that can continue: publickey
> debug1: Next authentication method: publickey
> debug1: Offering RSA public key: .ssh/id_rsa
> debug2: we sent a publickey packet, wait for reply
> debug1: Authentications that can continue: publickey
> debug2: we did not send a packet, disable method
> debug1: No more authentication methods to try.
> Permission denied (publickey).
> gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
> gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
> OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: Applying options for *
> debug2: ssh_connect: needpriv 0
> debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
> [184.72.86.108] port 22.
> debug1: Connection established.
> debug2: key_type_from_name: unknown key type '-----BEGIN'
> debug2: key_type_from_name: unknown key type '-----END'
> debug1: identity file .ssh/id_rsa type 1
> debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
> debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
> debug1: identity file .ssh/id_rsa-cert type -1
> debug1: Remote protocol version 2.0, remote software version
> OpenSSH_5.3p1 Debian-3ubuntu7
> debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
> debug1: Enabling compatibility mode for protocol 2.0
> debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
> debug2: fd 3 setting O_NONBLOCK
> debug1: SSH2_MSG_KEXINIT sent
> debug1: SSH2_MSG_KEXINIT received
> debug2: kex_parse_kexinit:
> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit:
> ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: kex_parse_kexinit:
> diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit:
> hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit: none,zlib@openssh.com
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit:
> debug2: kex_parse_kexinit: first_kex_follows 0
> debug2: kex_parse_kexinit: reserved 0
> debug2: mac_setup: found hmac-md5
> debug1: kex: server->client aes128-ctr hmac-md5 none
> debug2: mac_setup: found hmac-md5
> debug1: kex: client->server aes128-ctr hmac-md5 none
> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
> debug2: dh_gen_key: priv key bits set: 145/256
> debug2: bits set: 505/1024
> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
> debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
> debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
> matches the RSA host key.
> debug1: Found key in /home/gpoort/.ssh/known_hosts:7
> debug2: bits set: 495/1024
> debug1: ssh_rsa_verify: signature correct
> debug2: kex_derive_keys
> debug2: set_newkeys: mode 1
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug2: set_newkeys: mode 0
> debug1: SSH2_MSG_NEWKEYS received
> debug1: Roaming not allowed by server
> debug1: SSH2_MSG_SERVICE_REQUEST sent
> debug2: service_accept: ssh-userauth
> debug1: SSH2_MSG_SERVICE_ACCEPT received
> debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
> debug1: Authentications that can continue: publickey
> debug1: Next authentication method: publickey
> debug1: Offering RSA public key: .ssh/id_rsa
> debug2: we sent a publickey packet, wait for reply
> debug1: Authentications that can continue: publickey
> debug2: we did not send a packet, disable method
> debug1: No more authentication methods to try.
> Permission denied (publickey).
>
> On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
>> You should be able to login user the same user that is running Whirr.
>>
>> ssh -i ~/.ssh/id_rsa local-user@server
>>
>> Permission denied most of the time means invalid key, user pair.
>>
>> It should be ok to use the keys generate by default.
>>
>> -- Andrei Savu / andreisavu.ro
>>
>> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>>> I'm just using the defaults. But you may be onto the problem here,
>>> when I try to ssh into the node using:
>>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>>
>>> I get a permission denied.
>>>
>>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>>
>>> Thanks again,
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>>
>>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> I think you're probably right its an auth issue - although I was
>>>>> expecting a more direct/clear error message if the keypair wasn't
>>>>> working.
>>>>>
>>>>> I created the AMI by taking an EBS snapshot then converting to
>>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>>> with the same results. My understanding is that the keypair used to
>>>>> create the AMI is generally one of the accepted keys in addition to
>>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>>> anywhere that can be used to test this?
>>>>>
>>>>> Thanks again for your help,
>>>>>
>>>>> Joris
>>>>>
>>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>>> own key pair using the local SSH keys as specified in the properties
>>>>>> file.
>>>>>>
>>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>>> that custom ami with a different key pair?
>>>>>>
>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>> Andrei - thanks for the response!
>>>>>>>
>>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>>> both spot instances and regular instances and am getting the same
>>>>>>> behavior.
>>>>>>>
>>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>>> and "Dying because" are not always there):
>>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>> transport; encountered EOF
>>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>>> transport; encountered EOF
>>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>>> Broken transport; encountered EOF
>>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>>
>>>>>>> Last few entries on whirr.log:
>>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>>> securityGroupIds=[],
>>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>>> seconds
>>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>>
>>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>>
>>>>>>> Thanks again for any help on this!
>>>>>>>
>>>>>>> Joris
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>>
>>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>>
>>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>>> same behavior with regular ones?
>>>>>>>>
>>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>>> everything executes fine until I get the following error:
>>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>>
>>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>>> canonical images I don't have any issues.
>>>>>>>>>
>>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>>> Would really appreciate it!
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Joris
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Thanks guys. I guess I was using "ubuntu" as user incorrectly, now
adjusted to local-user "gpoort" I'm still not getting through. See
output below from -vv (sorry about the long email).
gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
ubuntu@ec2-184-72-86-108.compute-1.amazonaws.com
OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
[184.72.86.108] port 22.
debug1: Connection established.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file .ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file .ssh/id_rsa-cert type -1
debug1: Remote protocol version 2.0, remote software version
OpenSSH_5.3p1 Debian-3ubuntu7
debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit:
ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit:
ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 138/256
debug2: bits set: 502/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
matches the RSA host key.
debug1: Found key in /home/gpoort/.ssh/known_hosts:7
debug2: bits set: 497/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: .ssh/id_rsa (0x7f4ea3913cd0)
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: .ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
gpoort@jvb:~$ ssh -vv -i .ssh/id_rsa
gpoort@ec2-184-72-86-108.compute-1.amazonaws.com
OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ec2-184-72-86-108.compute-1.amazonaws.com
[184.72.86.108] port 22.
debug1: Connection established.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file .ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file .ssh/id_rsa-cert type -1
debug1: Remote protocol version 2.0, remote software version
OpenSSH_5.3p1 Debian-3ubuntu7
debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit:
ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit:
ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 145/256
debug2: bits set: 505/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 77:b6:b5:95:c6:0d:3d:82:94:91:d5:7d:70:8f:b8:1b
debug1: Host 'ec2-184-72-86-108.compute-1.amazonaws.com' is known and
matches the RSA host key.
debug1: Found key in /home/gpoort/.ssh/known_hosts:7
debug2: bits set: 495/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: .ssh/id_rsa (0x7fc32b462cd0)
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: .ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
On Wed, Aug 24, 2011 at 9:48 PM, Andrei Savu <sa...@gmail.com> wrote:
> You should be able to login user the same user that is running Whirr.
>
> ssh -i ~/.ssh/id_rsa local-user@server
>
> Permission denied most of the time means invalid key, user pair.
>
> It should be ok to use the keys generate by default.
>
> -- Andrei Savu / andreisavu.ro
>
> On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
>> I'm just using the defaults. But you may be onto the problem here,
>> when I try to ssh into the node using:
>> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>>
>> I get a permission denied.
>>
>> Any idea how to fix this? Should I create my own set of SSH key pairs?
>>
>> Thanks again,
>>
>> Joris
>>
>> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>>
>>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> I think you're probably right its an auth issue - although I was
>>>> expecting a more direct/clear error message if the keypair wasn't
>>>> working.
>>>>
>>>> I created the AMI by taking an EBS snapshot then converting to
>>>> instance-store. I've tried both the ebs back ami and instance-store
>>>> with the same results. My understanding is that the keypair used to
>>>> create the AMI is generally one of the accepted keys in addition to
>>>> the key pair used to launch the instance created by jclouds. I'm not
>>>> sure how to confirm this for sure - is the jclouds keypair stored
>>>> anywhere that can be used to test this?
>>>>
>>>> Thanks again for your help,
>>>>
>>>> Joris
>>>>
>>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>>> own key pair using the local SSH keys as specified in the properties
>>>>> file.
>>>>>
>>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>>> that custom ami with a different key pair?
>>>>>
>>>>> -- Andrei Savu / andreisavu.ro
>>>>>
>>>>>
>>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> Andrei - thanks for the response!
>>>>>>
>>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>>> both spot instances and regular instances and am getting the same
>>>>>> behavior.
>>>>>>
>>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>>> and "Dying because" are not always there):
>>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>> transport; encountered EOF
>>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>>> transport; encountered EOF
>>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>>> Broken transport; encountered EOF
>>>>>> << (root@174.129.128.120:22) error acquiring
>>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>>
>>>>>> Last few entries on whirr.log:
>>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>>> ramdiskId=null, availabilityZone=null,
>>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>>> securityGroupIds=[],
>>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>>> seconds
>>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>>> socket [address=174.129.128.120, port=22] opened
>>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>>
>>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>>
>>>>>> Thanks again for any help on this!
>>>>>>
>>>>>> Joris
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>>
>>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>>> in /tmp for jclouds script execution logs.
>>>>>>>
>>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>>> same behavior with regular ones?
>>>>>>>
>>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>>> everything executes fine until I get the following error:
>>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>>
>>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>>> canonical images I don't have any issues.
>>>>>>>>
>>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>>> Would really appreciate it!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Joris
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Andrei Savu <sa...@gmail.com>.
You should be able to login user the same user that is running Whirr.
ssh -i ~/.ssh/id_rsa local-user@server
Permission denied most of the time means invalid key, user pair.
It should be ok to use the keys generate by default.
-- Andrei Savu / andreisavu.ro
On Wed, Aug 24, 2011 at 9:01 PM, Joris Poort <gp...@gmail.com> wrote:
> I'm just using the defaults. But you may be onto the problem here,
> when I try to ssh into the node using:
> ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
>
> I get a permission denied.
>
> Any idea how to fix this? Should I create my own set of SSH key pairs?
>
> Thanks again,
>
> Joris
>
> On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
>> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>>
>> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>>
>> -- Andrei Savu / andreisavu.ro
>>
>> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>>> I think you're probably right its an auth issue - although I was
>>> expecting a more direct/clear error message if the keypair wasn't
>>> working.
>>>
>>> I created the AMI by taking an EBS snapshot then converting to
>>> instance-store. I've tried both the ebs back ami and instance-store
>>> with the same results. My understanding is that the keypair used to
>>> create the AMI is generally one of the accepted keys in addition to
>>> the key pair used to launch the instance created by jclouds. I'm not
>>> sure how to confirm this for sure - is the jclouds keypair stored
>>> anywhere that can be used to test this?
>>>
>>> Thanks again for your help,
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>>> own key pair using the local SSH keys as specified in the properties
>>>> file.
>>>>
>>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>>> that custom ami with a different key pair?
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>>
>>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> Andrei - thanks for the response!
>>>>>
>>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>>> both spot instances and regular instances and am getting the same
>>>>> behavior.
>>>>>
>>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>>> and "Dying because" are not always there):
>>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>> transport; encountered EOF
>>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>>> transport; encountered EOF
>>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>>> Broken transport; encountered EOF
>>>>> << (root@174.129.128.120:22) error acquiring
>>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>>
>>>>> Last few entries on whirr.log:
>>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>>> ramdiskId=null, availabilityZone=null,
>>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>>> securityGroupIds=[],
>>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>>> seconds
>>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>>> socket [address=174.129.128.120, port=22] opened
>>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>>> socket [address=50.17.135.8, port=22] opened
>>>>>
>>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>>
>>>>> Thanks again for any help on this!
>>>>>
>>>>> Joris
>>>>>
>>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>>
>>>>>> Also check whirr.log for more details and on the remote machines look
>>>>>> in /tmp for jclouds script execution logs.
>>>>>>
>>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>>> same behavior with regular ones?
>>>>>>
>>>>>> -- Andrei Savu / andreisavu.ro
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>>> everything executes fine until I get the following error:
>>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>>
>>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>>> canonical images I don't have any issues.
>>>>>>>
>>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>>> Would really appreciate it!
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Joris
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
I'm just using the defaults. But you may be onto the problem here,
when I try to ssh into the node using:
ssh -i ~/.ssh/id_rsa ubuntu@<public dns>
I get a permission denied.
Any idea how to fix this? Should I create my own set of SSH key pairs?
Thanks again,
Joris
On Wed, Aug 24, 2011 at 8:42 PM, Andrei Savu <sa...@gmail.com> wrote:
> Are you specifying a new set of SSH key pairs in the Whirr properties file?
>
> If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
>
> -- Andrei Savu / andreisavu.ro
>
> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>> I think you're probably right its an auth issue - although I was
>> expecting a more direct/clear error message if the keypair wasn't
>> working.
>>
>> I created the AMI by taking an EBS snapshot then converting to
>> instance-store. I've tried both the ebs back ami and instance-store
>> with the same results. My understanding is that the keypair used to
>> create the AMI is generally one of the accepted keys in addition to
>> the key pair used to launch the instance created by jclouds. I'm not
>> sure how to confirm this for sure - is the jclouds keypair stored
>> anywhere that can be used to test this?
>>
>> Thanks again for your help,
>>
>> Joris
>>
>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>> own key pair using the local SSH keys as specified in the properties
>>> file.
>>>
>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>> that custom ami with a different key pair?
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>>
>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> Andrei - thanks for the response!
>>>>
>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>> both spot instances and regular instances and am getting the same
>>>> behavior.
>>>>
>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>> and "Dying because" are not always there):
>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>> transport; encountered EOF
>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>> transport; encountered EOF
>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>> Broken transport; encountered EOF
>>>> << (root@174.129.128.120:22) error acquiring
>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>
>>>> Last few entries on whirr.log:
>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>> ramdiskId=null, availabilityZone=null,
>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>> securityGroupIds=[],
>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>> seconds
>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>> socket [address=174.129.128.120, port=22] opened
>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>> socket [address=50.17.135.8, port=22] opened
>>>>
>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>
>>>> Thanks again for any help on this!
>>>>
>>>> Joris
>>>>
>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>
>>>>> Also check whirr.log for more details and on the remote machines look
>>>>> in /tmp for jclouds script execution logs.
>>>>>
>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>> same behavior with regular ones?
>>>>>
>>>>> -- Andrei Savu / andreisavu.ro
>>>>>
>>>>>
>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>> everything executes fine until I get the following error:
>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>
>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>> canonical images I don't have any issues.
>>>>>>
>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>> Would really appreciate it!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Joris
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Andrei Savu <sa...@gmail.com>.
Are you specifying a new set of SSH key pairs in the Whirr properties file?
If not by default Whirr will use the keys found in ~/.ssh - id_rsa & id_rsa.pub.
-- Andrei Savu / andreisavu.ro
On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
> I think you're probably right its an auth issue - although I was
> expecting a more direct/clear error message if the keypair wasn't
> working.
>
> I created the AMI by taking an EBS snapshot then converting to
> instance-store. I've tried both the ebs back ami and instance-store
> with the same results. My understanding is that the keypair used to
> create the AMI is generally one of the accepted keys in addition to
> the key pair used to launch the instance created by jclouds. I'm not
> sure how to confirm this for sure - is the jclouds keypair stored
> anywhere that can be used to test this?
>
> Thanks again for your help,
>
> Joris
>
> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>> own key pair using the local SSH keys as specified in the properties
>> file.
>>
>> You've created the custom ami by taking an EBS snapshot? Can you use
>> that custom ami with a different key pair?
>>
>> -- Andrei Savu / andreisavu.ro
>>
>>
>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>> Andrei - thanks for the response!
>>>
>>> I logged into the custom AMI using ssh and a key pair on my local
>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>> both spot instances and regular instances and am getting the same
>>> behavior.
>>>
>>> Full output on terminal looks like (lines between "Starting 1 node"
>>> and "Dying because" are not always there):
>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>> transport; encountered EOF
>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>> transport; encountered EOF
>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>> Broken transport; encountered EOF
>>> << (root@174.129.128.120:22) error acquiring
>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>
>>> Last few entries on whirr.log:
>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>> ramdiskId=null, availabilityZone=null,
>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>> securityGroupIds=[],
>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>> started instances([region=us-east-1, name=sir-4f589c11])
>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>> started instances([region=us-east-1, name=sir-59cec612])
>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>> present instances([region=us-east-1, name=sir-4f589c11])
>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>> present instances([region=us-east-1, name=sir-59cec612])
>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>> seconds
>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>> socket [address=174.129.128.120, port=22] opened
>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>> socket [address=50.17.135.8, port=22] opened
>>>
>>> After ssh onto node, didn't find any logs in /tmp.
>>>
>>> Thanks again for any help on this!
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>
>>>> Also check whirr.log for more details and on the remote machines look
>>>> in /tmp for jclouds script execution logs.
>>>>
>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>> same behavior with regular ones?
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>>
>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>> everything executes fine until I get the following error:
>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>
>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>> code stalls and I get the above error (2x number of instances), no
>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>> canonical images I don't have any issues.
>>>>>
>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>> Would really appreciate it!
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Joris
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Additional info on the AMI I'm trying to run is:
ami-d1e525b8 (slightly customized version of ami-63be790a)
On Wed, Aug 24, 2011 at 8:35 PM, Joris Poort <gp...@gmail.com> wrote:
> Not sure if I'm testing the right thing here, but just made the AMI
> public to ensure it has no private settings giving issues. Executed
> same as before:
>
> Terminal:
> Bootstrapping cluster
> Configuring template
> Configuring template
> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
> Dying because - java.net.SocketTimeoutException: Read timed out
> Dying because - java.net.SocketTimeoutException: Read timed out
> Dying because - java.net.SocketTimeoutException: Read timed out
> Dying because - java.net.SocketTimeoutException: Read timed out
>
> Whirr.log:
> 2011-08-24 20:21:09,014 DEBUG [jclouds.compute] (pool-3-thread-3) <<
> started instances([region=us-east-1, name=sir-a65d6614])
> 2011-08-24 20:21:09,167 DEBUG [jclouds.compute] (pool-3-thread-4) <<
> started instances([region=us-east-1, name=sir-bf822814])
> 2011-08-24 20:21:09,422 DEBUG [jclouds.compute] (pool-3-thread-3) <<
> present instances([region=us-east-1, name=sir-a65d6614])
> 2011-08-24 20:21:09,672 DEBUG [jclouds.compute] (pool-3-thread-4) <<
> present instances([region=us-east-1, name=sir-bf822814])
> 2011-08-24 20:30:44,216 DEBUG [jclouds.compute] (user thread 2) >>
> blocking on socket [address=184.72.163.250, port=22] for 600000
> seconds
> 2011-08-24 20:30:58,316 DEBUG [jclouds.compute] (user thread 2) <<
> socket [address=184.72.163.250, port=22] opened
> 2011-08-24 20:31:05,739 DEBUG [jclouds.compute] (user thread 6) >>
> blocking on socket [address=50.16.31.67, port=22] for 600000 seconds
> 2011-08-24 20:31:12,968 DEBUG [jclouds.compute] (user thread 6) <<
> socket [address=50.16.31.67, port=22] opened
>
> Any further help on ideas of anything to try to help debug this issue
> would be greatly appreciated!
>
> Cheers,
>
> Joris
>
> On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
>> I think you're probably right its an auth issue - although I was
>> expecting a more direct/clear error message if the keypair wasn't
>> working.
>>
>> I created the AMI by taking an EBS snapshot then converting to
>> instance-store. I've tried both the ebs back ami and instance-store
>> with the same results. My understanding is that the keypair used to
>> create the AMI is generally one of the accepted keys in addition to
>> the key pair used to launch the instance created by jclouds. I'm not
>> sure how to confirm this for sure - is the jclouds keypair stored
>> anywhere that can be used to test this?
>>
>> Thanks again for your help,
>>
>> Joris
>>
>> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>>> own key pair using the local SSH keys as specified in the properties
>>> file.
>>>
>>> You've created the custom ami by taking an EBS snapshot? Can you use
>>> that custom ami with a different key pair?
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>>
>>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> Andrei - thanks for the response!
>>>>
>>>> I logged into the custom AMI using ssh and a key pair on my local
>>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>>> both spot instances and regular instances and am getting the same
>>>> behavior.
>>>>
>>>> Full output on terminal looks like (lines between "Starting 1 node"
>>>> and "Dying because" are not always there):
>>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>> transport; encountered EOF
>>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>>> transport; encountered EOF
>>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>>> Broken transport; encountered EOF
>>>> << (root@174.129.128.120:22) error acquiring
>>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>>
>>>> Last few entries on whirr.log:
>>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>>> ramdiskId=null, availabilityZone=null,
>>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>>> securityGroupIds=[],
>>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>> started instances([region=us-east-1, name=sir-4f589c11])
>>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>> started instances([region=us-east-1, name=sir-59cec612])
>>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>>> present instances([region=us-east-1, name=sir-4f589c11])
>>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>>> present instances([region=us-east-1, name=sir-59cec612])
>>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>>> seconds
>>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>>> socket [address=174.129.128.120, port=22] opened
>>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>>> socket [address=50.17.135.8, port=22] opened
>>>>
>>>> After ssh onto node, didn't find any logs in /tmp.
>>>>
>>>> Thanks again for any help on this!
>>>>
>>>> Joris
>>>>
>>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>>
>>>>> Also check whirr.log for more details and on the remote machines look
>>>>> in /tmp for jclouds script execution logs.
>>>>>
>>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>>> same behavior with regular ones?
>>>>>
>>>>> -- Andrei Savu / andreisavu.ro
>>>>>
>>>>>
>>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>>> everything executes fine until I get the following error:
>>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>>
>>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>>> code stalls and I get the above error (2x number of instances), no
>>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>>> canonical images I don't have any issues.
>>>>>>
>>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>>> Would really appreciate it!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Joris
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Not sure if I'm testing the right thing here, but just made the AMI
public to ensure it has no private settings giving issues. Executed
same as before:
Terminal:
Bootstrapping cluster
Configuring template
Configuring template
Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
Dying because - java.net.SocketTimeoutException: Read timed out
Dying because - java.net.SocketTimeoutException: Read timed out
Dying because - java.net.SocketTimeoutException: Read timed out
Dying because - java.net.SocketTimeoutException: Read timed out
Whirr.log:
2011-08-24 20:21:09,014 DEBUG [jclouds.compute] (pool-3-thread-3) <<
started instances([region=us-east-1, name=sir-a65d6614])
2011-08-24 20:21:09,167 DEBUG [jclouds.compute] (pool-3-thread-4) <<
started instances([region=us-east-1, name=sir-bf822814])
2011-08-24 20:21:09,422 DEBUG [jclouds.compute] (pool-3-thread-3) <<
present instances([region=us-east-1, name=sir-a65d6614])
2011-08-24 20:21:09,672 DEBUG [jclouds.compute] (pool-3-thread-4) <<
present instances([region=us-east-1, name=sir-bf822814])
2011-08-24 20:30:44,216 DEBUG [jclouds.compute] (user thread 2) >>
blocking on socket [address=184.72.163.250, port=22] for 600000
seconds
2011-08-24 20:30:58,316 DEBUG [jclouds.compute] (user thread 2) <<
socket [address=184.72.163.250, port=22] opened
2011-08-24 20:31:05,739 DEBUG [jclouds.compute] (user thread 6) >>
blocking on socket [address=50.16.31.67, port=22] for 600000 seconds
2011-08-24 20:31:12,968 DEBUG [jclouds.compute] (user thread 6) <<
socket [address=50.16.31.67, port=22] opened
Any further help on ideas of anything to try to help debug this issue
would be greatly appreciated!
Cheers,
Joris
On Wed, Aug 24, 2011 at 8:02 PM, Joris Poort <gp...@gmail.com> wrote:
> I think you're probably right its an auth issue - although I was
> expecting a more direct/clear error message if the keypair wasn't
> working.
>
> I created the AMI by taking an EBS snapshot then converting to
> instance-store. I've tried both the ebs back ami and instance-store
> with the same results. My understanding is that the keypair used to
> create the AMI is generally one of the accepted keys in addition to
> the key pair used to launch the instance created by jclouds. I'm not
> sure how to confirm this for sure - is the jclouds keypair stored
> anywhere that can be used to test this?
>
> Thanks again for your help,
>
> Joris
>
> On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
>> I'm not sure but it looks like an auth issue to me. Whirr creates it's
>> own key pair using the local SSH keys as specified in the properties
>> file.
>>
>> You've created the custom ami by taking an EBS snapshot? Can you use
>> that custom ami with a different key pair?
>>
>> -- Andrei Savu / andreisavu.ro
>>
>>
>> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>>> Andrei - thanks for the response!
>>>
>>> I logged into the custom AMI using ssh and a key pair on my local
>>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>>> both spot instances and regular instances and am getting the same
>>> behavior.
>>>
>>> Full output on terminal looks like (lines between "Starting 1 node"
>>> and "Dying because" are not always there):
>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>> transport; encountered EOF
>>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>>> transport; encountered EOF
>>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>>> Broken transport; encountered EOF
>>> << (root@174.129.128.120:22) error acquiring
>>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>> Dying because - java.net.SocketTimeoutException: Read timed out
>>>
>>> Last few entries on whirr.log:
>>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>>> requesting 1 spot instances region(us-east-1) price(0.250000)
>>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>>> ramdiskId=null, availabilityZone=null,
>>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>>> securityGroupIdToNames={}, blockDeviceMappings=[],
>>> securityGroupIds=[],
>>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>> started instances([region=us-east-1, name=sir-4f589c11])
>>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>> started instances([region=us-east-1, name=sir-59cec612])
>>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>>> present instances([region=us-east-1, name=sir-4f589c11])
>>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>>> present instances([region=us-east-1, name=sir-59cec612])
>>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>>> blocking on socket [address=174.129.128.120, port=22] for 600000
>>> seconds
>>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>>> socket [address=174.129.128.120, port=22] opened
>>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>>> socket [address=50.17.135.8, port=22] opened
>>>
>>> After ssh onto node, didn't find any logs in /tmp.
>>>
>>> Thanks again for any help on this!
>>>
>>> Joris
>>>
>>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>>
>>>> Also check whirr.log for more details and on the remote machines look
>>>> in /tmp for jclouds script execution logs.
>>>>
>>>> I know from IRC that you are using spot instances. Are you seeing the
>>>> same behavior with regular ones?
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>>
>>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>>> everything executes fine until I get the following error:
>>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>>
>>>>> The instances are running fine, I can ssh into them, but the whirr
>>>>> code stalls and I get the above error (2x number of instances), no
>>>>> proxy shell is created. If I run the exact same code with vanilla
>>>>> canonical images I don't have any issues.
>>>>>
>>>>> Anyone have any ideas on things to test, debug or work around this?
>>>>> Would really appreciate it!
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Joris
>>>>>
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
I think you're probably right its an auth issue - although I was
expecting a more direct/clear error message if the keypair wasn't
working.
I created the AMI by taking an EBS snapshot then converting to
instance-store. I've tried both the ebs back ami and instance-store
with the same results. My understanding is that the keypair used to
create the AMI is generally one of the accepted keys in addition to
the key pair used to launch the instance created by jclouds. I'm not
sure how to confirm this for sure - is the jclouds keypair stored
anywhere that can be used to test this?
Thanks again for your help,
Joris
On Wed, Aug 24, 2011 at 7:49 PM, Andrei Savu <sa...@gmail.com> wrote:
> I'm not sure but it looks like an auth issue to me. Whirr creates it's
> own key pair using the local SSH keys as specified in the properties
> file.
>
> You've created the custom ami by taking an EBS snapshot? Can you use
> that custom ami with a different key pair?
>
> -- Andrei Savu / andreisavu.ro
>
>
> On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
>> Andrei - thanks for the response!
>>
>> I logged into the custom AMI using ssh and a key pair on my local
>> machine (I'm executing whirr via ubuntu virtual machine). I've tried
>> both spot instances and regular instances and am getting the same
>> behavior.
>>
>> Full output on terminal looks like (lines between "Starting 1 node"
>> and "Dying because" are not always there):
>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>> transport; encountered EOF
>> Dying because - net.schmizz.sshj.transport.TransportException: Broken
>> transport; encountered EOF
>> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
>> Broken transport; encountered EOF
>> << (root@174.129.128.120:22) error acquiring
>> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
>> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
>> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
>> Dying because - java.net.SocketTimeoutException: Read timed out
>> Dying because - java.net.SocketTimeoutException: Read timed out
>> Dying because - java.net.SocketTimeoutException: Read timed out
>> Dying because - java.net.SocketTimeoutException: Read timed out
>>
>> Last few entries on whirr.log:
>> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>> requesting 1 spot instances region(us-east-1) price(0.250000)
>> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
>> ramdiskId=null, availabilityZone=null,
>> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
>> securityGroupIdToNames={}, blockDeviceMappings=[],
>> securityGroupIds=[],
>> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
>> monitoringEnabled=null, userData=null]) options([formParameters={}])
>> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>> started instances([region=us-east-1, name=sir-4f589c11])
>> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> started instances([region=us-east-1, name=sir-59cec612])
>> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>> present instances([region=us-east-1, name=sir-4f589c11])
>> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> present instances([region=us-east-1, name=sir-59cec612])
>> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
>> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
>> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
>> blocking on socket [address=174.129.128.120, port=22] for 600000
>> seconds
>> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
>> socket [address=174.129.128.120, port=22] opened
>> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
>> socket [address=50.17.135.8, port=22] opened
>>
>> After ssh onto node, didn't find any logs in /tmp.
>>
>> Thanks again for any help on this!
>>
>> Joris
>>
>> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>>
>>> Also check whirr.log for more details and on the remote machines look
>>> in /tmp for jclouds script execution logs.
>>>
>>> I know from IRC that you are using spot instances. Are you seeing the
>>> same behavior with regular ones?
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>>
>>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I'm new to whirr and I'm running custom AMI configuration (application
>>>> installed on working canonical image). Executing with whirr 0.6.0
>>>> everything executes fine until I get the following error:
>>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>>
>>>> The instances are running fine, I can ssh into them, but the whirr
>>>> code stalls and I get the above error (2x number of instances), no
>>>> proxy shell is created. If I run the exact same code with vanilla
>>>> canonical images I don't have any issues.
>>>>
>>>> Anyone have any ideas on things to test, debug or work around this?
>>>> Would really appreciate it!
>>>>
>>>> Cheers,
>>>>
>>>> Joris
>>>>
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Andrei Savu <sa...@gmail.com>.
I'm not sure but it looks like an auth issue to me. Whirr creates it's
own key pair using the local SSH keys as specified in the properties
file.
You've created the custom ami by taking an EBS snapshot? Can you use
that custom ami with a different key pair?
-- Andrei Savu / andreisavu.ro
On Wed, Aug 24, 2011 at 7:31 PM, Joris Poort <gp...@gmail.com> wrote:
> Andrei - thanks for the response!
>
> I logged into the custom AMI using ssh and a key pair on my local
> machine (I'm executing whirr via ubuntu virtual machine). I've tried
> both spot instances and regular instances and am getting the same
> behavior.
>
> Full output on terminal looks like (lines between "Starting 1 node"
> and "Dying because" are not always there):
> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
> Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
> Dying because - net.schmizz.sshj.transport.TransportException: Broken
> transport; encountered EOF
> Dying because - net.schmizz.sshj.transport.TransportException: Broken
> transport; encountered EOF
> <<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
> Broken transport; encountered EOF
> << (root@174.129.128.120:22) error acquiring
> SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
> net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
> at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
> Dying because - java.net.SocketTimeoutException: Read timed out
> Dying because - java.net.SocketTimeoutException: Read timed out
> Dying because - java.net.SocketTimeoutException: Read timed out
> Dying because - java.net.SocketTimeoutException: Read timed out
>
> Last few entries on whirr.log:
> 2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
> requesting 1 spot instances region(us-east-1) price(0.250000)
> spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
> ramdiskId=null, availabilityZone=null,
> keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
> securityGroupIdToNames={}, blockDeviceMappings=[],
> securityGroupIds=[],
> securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
> monitoringEnabled=null, userData=null]) options([formParameters={}])
> 2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
> started instances([region=us-east-1, name=sir-4f589c11])
> 2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
> started instances([region=us-east-1, name=sir-59cec612])
> 2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
> present instances([region=us-east-1, name=sir-4f589c11])
> 2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
> present instances([region=us-east-1, name=sir-59cec612])
> 2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
> blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
> 2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
> blocking on socket [address=174.129.128.120, port=22] for 600000
> seconds
> 2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
> socket [address=174.129.128.120, port=22] opened
> 2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
> socket [address=50.17.135.8, port=22] opened
>
> After ssh onto node, didn't find any logs in /tmp.
>
> Thanks again for any help on this!
>
> Joris
>
> On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
>> I suspect this is an authentication issue. How do you login to the custom AMI?
>>
>> Also check whirr.log for more details and on the remote machines look
>> in /tmp for jclouds script execution logs.
>>
>> I know from IRC that you are using spot instances. Are you seeing the
>> same behavior with regular ones?
>>
>> -- Andrei Savu / andreisavu.ro
>>
>>
>> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>>> Hi,
>>>
>>> I'm new to whirr and I'm running custom AMI configuration (application
>>> installed on working canonical image). Executing with whirr 0.6.0
>>> everything executes fine until I get the following error:
>>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>>
>>> The instances are running fine, I can ssh into them, but the whirr
>>> code stalls and I get the above error (2x number of instances), no
>>> proxy shell is created. If I run the exact same code with vanilla
>>> canonical images I don't have any issues.
>>>
>>> Anyone have any ideas on things to test, debug or work around this?
>>> Would really appreciate it!
>>>
>>> Cheers,
>>>
>>> Joris
>>>
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Joris Poort <gp...@gmail.com>.
Andrei - thanks for the response!
I logged into the custom AMI using ssh and a key pair on my local
machine (I'm executing whirr via ubuntu virtual machine). I've tried
both spot instances and regular instances and am getting the same
behavior.
Full output on terminal looks like (lines between "Starting 1 node"
and "Dying because" are not always there):
Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
Starting 1 node(s) with roles [hadoop-namenode, hadoop-jobtracker]
Dying because - net.schmizz.sshj.transport.TransportException: Broken
transport; encountered EOF
Dying because - net.schmizz.sshj.transport.TransportException: Broken
transport; encountered EOF
<<kex done>> woke to: net.schmizz.sshj.transport.TransportException:
Broken transport; encountered EOF
<< (root@174.129.128.120:22) error acquiring
SSHClient(root@174.129.128.120:22): Broken transport; encountered EOF
net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
at net.schmizz.sshj.transport.Reader.run(Reader.java:70)
Dying because - java.net.SocketTimeoutException: Read timed out
Dying because - java.net.SocketTimeoutException: Read timed out
Dying because - java.net.SocketTimeoutException: Read timed out
Dying because - java.net.SocketTimeoutException: Read timed out
Last few entries on whirr.log:
2011-08-24 19:20:05,428 DEBUG [jclouds.compute] (pool-3-thread-2) >>
requesting 1 spot instances region(us-east-1) price(0.250000)
spec([instanceType=m1.large, imageId=ami-d1e525b8, kernelId=null,
ramdiskId=null, availabilityZone=null,
keyName=jclouds#hadoop_custom_spot_1#us-east-1#45,
securityGroupIdToNames={}, blockDeviceMappings=[],
securityGroupIds=[],
securityGroupNames=[jclouds#hadoop_custom_spot_1#us-east-1],
monitoringEnabled=null, userData=null]) options([formParameters={}])
2011-08-24 19:20:05,642 DEBUG [jclouds.compute] (pool-3-thread-4) <<
started instances([region=us-east-1, name=sir-4f589c11])
2011-08-24 19:20:05,682 DEBUG [jclouds.compute] (pool-3-thread-2) <<
started instances([region=us-east-1, name=sir-59cec612])
2011-08-24 19:20:05,864 DEBUG [jclouds.compute] (pool-3-thread-4) <<
present instances([region=us-east-1, name=sir-4f589c11])
2011-08-24 19:20:05,917 DEBUG [jclouds.compute] (pool-3-thread-2) <<
present instances([region=us-east-1, name=sir-59cec612])
2011-08-24 19:27:18,150 DEBUG [jclouds.compute] (user thread 8) >>
blocking on socket [address=50.17.135.8, port=22] for 600000 seconds
2011-08-24 19:27:21,132 DEBUG [jclouds.compute] (user thread 7) >>
blocking on socket [address=174.129.128.120, port=22] for 600000
seconds
2011-08-24 19:27:24,222 DEBUG [jclouds.compute] (user thread 7) <<
socket [address=174.129.128.120, port=22] opened
2011-08-24 19:27:32,255 DEBUG [jclouds.compute] (user thread 8) <<
socket [address=50.17.135.8, port=22] opened
After ssh onto node, didn't find any logs in /tmp.
Thanks again for any help on this!
Joris
On Wed, Aug 24, 2011 at 7:12 PM, Andrei Savu <sa...@gmail.com> wrote:
> I suspect this is an authentication issue. How do you login to the custom AMI?
>
> Also check whirr.log for more details and on the remote machines look
> in /tmp for jclouds script execution logs.
>
> I know from IRC that you are using spot instances. Are you seeing the
> same behavior with regular ones?
>
> -- Andrei Savu / andreisavu.ro
>
>
> On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
>> Hi,
>>
>> I'm new to whirr and I'm running custom AMI configuration (application
>> installed on working canonical image). Executing with whirr 0.6.0
>> everything executes fine until I get the following error:
>> "Dying because - java.net.SocketTimeoutException: Read timed out"
>>
>> The instances are running fine, I can ssh into them, but the whirr
>> code stalls and I get the above error (2x number of instances), no
>> proxy shell is created. If I run the exact same code with vanilla
>> canonical images I don't have any issues.
>>
>> Anyone have any ideas on things to test, debug or work around this?
>> Would really appreciate it!
>>
>> Cheers,
>>
>> Joris
>>
>
Re: EC2 Custom AMI's connection issue
Posted by Andrei Savu <sa...@gmail.com>.
I suspect this is an authentication issue. How do you login to the custom AMI?
Also check whirr.log for more details and on the remote machines look
in /tmp for jclouds script execution logs.
I know from IRC that you are using spot instances. Are you seeing the
same behavior with regular ones?
-- Andrei Savu / andreisavu.ro
On Wed, Aug 24, 2011 at 7:07 PM, Joris Poort <gp...@gmail.com> wrote:
> Hi,
>
> I'm new to whirr and I'm running custom AMI configuration (application
> installed on working canonical image). Executing with whirr 0.6.0
> everything executes fine until I get the following error:
> "Dying because - java.net.SocketTimeoutException: Read timed out"
>
> The instances are running fine, I can ssh into them, but the whirr
> code stalls and I get the above error (2x number of instances), no
> proxy shell is created. If I run the exact same code with vanilla
> canonical images I don't have any issues.
>
> Anyone have any ideas on things to test, debug or work around this?
> Would really appreciate it!
>
> Cheers,
>
> Joris
>