You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Something Something <ma...@gmail.com> on 2009/11/22 01:57:34 UTC
Starting Hadoop cluster on EC2
Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's distribution
works well, but trying to do this myself so I can learn what's happening
behind the scene.)
I have a Master & a Slave. When I start HDFS on the master, I get a message
saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
the slave.
Basic problem (I think) is that I can't ssh from the master EC2 instance to
the Slave EC2 instance. What's the best way to fix it? I think I need the
"Key Pair" file on my master. I have a key pair on my local machine, but
how do I transfer it to the EC2 machine? (I know, I know, I agree.. I am
dumb :) Should I FTP it?
Please help. Thanks.
Re: Starting Hadoop cluster on EC2
Posted by Vaibhav Puranik <vp...@gmail.com>.
Thanks. I didn't know that you could do that.
Regards,
Vaibhav
On Sun, Nov 29, 2009 at 11:52 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Everything started working after I added the following line to
> hadoop-env.sh
>
> export HADOOP_SSH_OPTS="$HADOOP_SSH_OPTS -i /mnt/myVol/KeyPairName.pem "
>
> Thanks for your help.
>
>
> On Wed, Nov 25, 2009 at 10:25 AM, Vaibhav Puranik <vpuranik@gmail.com
> >wrote:
>
> > On master, in .ssh folder, there should be a file id_dsa.pub. This is
> > master's public key.
> >
> > Contents of this file should be copied to slave's authorized_keys file.
> >
> > Regards,
> > Vaibhav
> >
> > On Wed, Nov 25, 2009 at 9:45 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Can you please explain what you mean by "add master's public key to
> > slave's
> > > authorized_keys"? I have a feeling I am not doing this correctly.
> What
> > I
> > > did is this:
> > >
> > > 1) On Master, opened ~/.ssh/authorized_key and copied lines that look
> > like
> > > this...
> > >
> > > ssh-dss
> > >
> > >
> >
> AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpXuJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTLlssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4OijkAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNGnWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbHa89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfOn6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2MldKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ==
> > > root@domU-12-31-38-00-39-C8
> > >
> > >
> > > to the end of ~/.ssh/authorized_key on Slave.
> > >
> > > Something tells me there's a better way to do this. Is there?
> > >
> > >
> > > On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vp...@gmail.com>
> > > wrote:
> > >
> > > > Actually I was wrong about adding keypair. It should be resolved by
> > > adding
> > > > master's public key to slave's authorized_keys. Amazon puts your
> > keypair
> > > in
> > > > authroized_keys automatically. I missed that part.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Me again :(
> > > > >
> > > > > I decided to start fresh so that I can document the steps, but now
> I
> > > > can't
> > > > > even go further than I did before :(
> > > > >
> > > > > Please let me know what step I am missing:
> > > > >
> > > > > 1) Launched 2 instances on AWS console - one for Master, one for
> > > Slave.
> > > > >
> > > > > 2) In one command window, connected to instance 1:
> > > > > ssh -i MyKeyPair.pem <instance 1>
> > > > > (Got 'authenticity' warning.. typed 'yes')
> > > > >
> > > > > This is my "Master" window.
> > > > >
> > > > > 3) Opened another command window & connected to instance 2:
> > > > > ssh -i MyKeyPair.pem <instance 2>
> > > > > (Got 'authenticity' warning.. typed 'yes')
> > > > >
> > > > > This is my "Slave" window.
> > > > >
> > > > > 4) Opened another command window and 'scp' the key pair (from my
> > local
> > > > > machine) to instance 1 & instance 2.
> > > > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > > > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> > > > >
> > > > > 5) On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> > > > >
> > > > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> > > > >
> > > > > 6) To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > > > > ssh localhost (Got 'authenticiy' message. Typed 'yes'. Got
> > > Permission
> > > > > denied (public key)
> > > > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > > > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> > > > >
> > > > > 7) In order to ssh from Master to Slave (without entering
> KeyPair),
> > > > copied
> > > > > the key (that starts with 'sh-dss') from Slave's 'authorized_key'
> > file
> > > to
> > > > > Master's 'authoried_key' file.
> > > > >
> > > > > Now, every time I try (from Master):
> > > > >
> > > > > ssh <slave's ip address>
> > > > >
> > > > > I get - "Permission denied (publickey).
> > > > >
> > > > > But, when I type:
> > > > >
> > > > > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> > > > >
> > > > > It works.
> > > > >
> > > > > I thought step #5 above should have resolved this. What am I doing
> > > > wrong?
> > > > > I believe, in order for me to start the cluster, I should be able
> to
> > > > 'ssh'
> > > > > to slave without using key pair, correct?
> > > > >
> > > > >
> > > > > Thanks for your help.
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <
> vpuranik@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Install a program called Jps.
> > > > > >
> > > > > > It's a ps for java processes. execute jps, you should see the
> > > processes
> > > > > (if
> > > > > > they are running).
> > > > > >
> > > > > > Also make sure you check your logs.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Awesome. Thanks, Vaibhav. Made progress. No error messages
> > from
> > > > > > > ./all-start.sh
> > > > > > >
> > > > > > > I will test to see if the cluster is deployed properly.
> > > > > > >
> > > > > > > Question: On the slave, when I do:
> > > > > > >
> > > > > > > ps -eaf | grep 'hadoop'
> > > > > > >
> > > > > > > I don't see any processes. Shouldn't I see 2 processes - one
> for
> > > > > > Datanode
> > > > > > > and the other for TaskTracker?
> > > > > > >
> > > > > > > Please let me know. Thanks again.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <
> > > vpuranik@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes, I pretty much meant that.
> > > > > > > >
> > > > > > > > Add your master's public key to authorized_keys on slaves
> too.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Vaibhav
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Sorry.. but not sure what you mean by... "In order to
> achieve
> > > > that
> > > > > > you
> > > > > > > > need
> > > > > > > > > to add the keypair you are using to your
> > > > > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I tried this....
> > > > > > > > >
> > > > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > > > > >
> > > > > > > > > Is that what you meant by "add the keypair to
> > authorized_keys"?
> > > > > > > > >
> > > > > > > > > Still can't ssh from Slave to Master. It says.....
> > Permission
> > > > > > denied
> > > > > > > > > (publickey).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Sorry, I am a newbie when it comes to such environment
> > issues.
> > > > > > Thanks
> > > > > > > > for
> > > > > > > > > your patience. It's greatly appreciated.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > > > > vpuranik@gmail.com
> > > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > > > You need to be able to SSH without specifying the keypair
> > > from
> > > > a
> > > > > > > slave
> > > > > > > > to
> > > > > > > > > > master (I think reverse it true too, but I am not sure).
> > > > > > > > > >
> > > > > > > > > > In order to achieve that you need to add the keypair you
> > are
> > > > > using
> > > > > > > to
> > > > > > > > > your
> > > > > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > > > > >
> > > > > > > > > > Once you setup passphraseless ssh, it should start
> working.
> > > > > > > Furthrmore,
> > > > > > > > > > after you set this up, try to ssh manully (without key)
> and
> > > if
> > > > it
> > > > > > > asks
> > > > > > > > > you
> > > > > > > > > > the following message - say yes:
> > > > > > > > > >
> > > > > > > > > > The authenticity of host
> > > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > > (147.127.186.243)'
> > > > > > can't
> > > > > > > > > > be established.
> > > > > > > > > > RSA key fingerprint is
> > > > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > > > > Are you sure you want to continue connecting (yes/no)?
> yes
> > > > > > > > > > Warning: Permanently added
> > > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > > ,147.127.186.243'
> > > > > > > > > > (RSA) to the list of known hosts.
> > > > > > > > > >
> > > > > > > > > > In other words, you have to make sure that the machine
> you
> > > are
> > > > > > trying
> > > > > > > > to
> > > > > > > > > > ssh
> > > > > > > > > > is in your known_hosts file.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Vaibhav
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks, Vaibhav. That was helpful. I am able to scp
> > from
> > > my
> > > > > > local
> > > > > > > > > > machine
> > > > > > > > > > > after I opened port 22 under my security group.
> > > > > > > > > > >
> > > > > > > > > > > But my cluster is still not starting as expected. I
> > > mean...
> > > > on
> > > > > > my
> > > > > > > > > Master
> > > > > > > > > > > when I run:
> > > > > > > > > > >
> > > > > > > > > > > ./start-all.sh
> > > > > > > > > > >
> > > > > > > > > > > I get messages saying...
> > > > > > > > > > >
> > > > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > > > > >
> > > > > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > > > > >
> > > > > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Any suggestions? Thanks for your help.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > > > > vpuranik@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > In order to use scp from your machine to an ec2
> > instance
> > > > you
> > > > > > need
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > > > scp
> > > > > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > > > > >
> > > > > > > > > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > > > > > > > > >
> > > > > > > > > > > > Secondly in order to configure an ec2 cluster for
> > Hadoop,
> > > > you
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > > a
> > > > > > > > > > > > security group (let's call it hadoop) and give hadoop
> > > > access
> > > > > to
> > > > > > > > > hadoop.
> > > > > > > > > > > >
> > > > > > > > > > > > You can read more about security groups here -
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > > > > > > >
> > > > > > > > > > > > You also need to make sure that you can ssh from all
> > the
> > > > > slaves
> > > > > > > to
> > > > > > > > > > master
> > > > > > > > > > > > without specifying password.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Vaibhav Puranik
> > > > > > > > > > > > GumGum
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something
> <
> > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Seems like I am not explaining my problem
> correctly.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My Key Pair file is on my machine at work which is
> > > behind
> > > > a
> > > > > > > corp
> > > > > > > > > > > > firewall.
> > > > > > > > > > > > > As such I can't 'scp' from the ec2 instance to my
> > > local
> > > > > > > machine
> > > > > > > > at
> > > > > > > > > > > work
> > > > > > > > > > > > to
> > > > > > > > > > > > > 'get' the file. So I need a way to 'send' a file
> > from
> > > my
> > > > > > > machine
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > ec2
> > > > > > > > > > > > > instance. I tried using 'scp' (from my machine at
> > > work)
> > > > to
> > > > > > the
> > > > > > > > ec2
> > > > > > > > > > > > machine
> > > > > > > > > > > > > but it says "Permission denied". Does this make
> > sense?
> > > > > > > > > > > > >
> > > > > > > > > > > > > May be I need to use the command line tools for
> EC2.
> > I
> > > > am
> > > > > > > > looking
> > > > > > > > > > into
> > > > > > > > > > > > > those right now, but if there's a better/easier
> way,
> > > > please
> > > > > > let
> > > > > > > > me
> > > > > > > > > > > know.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks once again for your help.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > > > > zjffdu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > The ssh is installed on ec2 by default, otherwise
> > you
> > > > > have
> > > > > > no
> > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > > > > login
> > > > > > > > > > > > > > to ec2
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something
> > Something
> > > <
> > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As per Wikipedia (
> > > > > > http://en.wikipedia.org/wiki/Secure_copy
> > > > > > > )
> > > > > > > > > "It
> > > > > > > > > > > > does
> > > > > > > > > > > > > so
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > connecting to the host using SSH and there
> > executes
> > > > an
> > > > > > SCP
> > > > > > > > > server
> > > > > > > > > > > > > (scp)".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So if SSH isn't working SCP won't work, either.
> > In
> > > > any
> > > > > > > case
> > > > > > > > I
> > > > > > > > > > > tried
> > > > > > > > > > > > to
> > > > > > > > > > > > > > scp
> > > > > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Any other ideas? Thanks.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > > > > zjffdu@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> > > > Something
> > > > > <
> > > > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.
> > (Yes,
> > > > > > > > Cloudera's
> > > > > > > > > > > > > > distribution
> > > > > > > > > > > > > > > > > works well, but trying to do this myself so
> I
> > > can
> > > > > > learn
> > > > > > > > > > what's
> > > > > > > > > > > > > > > happening
> > > > > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have a Master & a Slave. When I start
> HDFS
> > > on
> > > > > the
> > > > > > > > > master,
> > > > > > > > > > I
> > > > > > > > > > > > get
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)"
> -
> > > > where
> > > > > > > > 10.xxx
> > > > > > > > > is
> > > > > > > > > > > IP
> > > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh
> > > from
> > > > > the
> > > > > > > > master
> > > > > > > > > > EC2
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the Slave EC2 instance. What's the best
> way
> > to
> > > > fix
> > > > > > it?
> > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > > I
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > "Key Pair" file on my master. I have a key
> > > pair
> > > > on
> > > > > > my
> > > > > > > > > local
> > > > > > > > > > > > > machine,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > how do I transfer it to the EC2 machine?
> (I
> > > > know,
> > > > > I
> > > > > > > > know,
> > > > > > > > > I
> > > > > > > > > > > > > agree..
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Please help. Thanks.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
Everything started working after I added the following line to hadoop-env.sh
export HADOOP_SSH_OPTS="$HADOOP_SSH_OPTS -i /mnt/myVol/KeyPairName.pem "
Thanks for your help.
On Wed, Nov 25, 2009 at 10:25 AM, Vaibhav Puranik <vp...@gmail.com>wrote:
> On master, in .ssh folder, there should be a file id_dsa.pub. This is
> master's public key.
>
> Contents of this file should be copied to slave's authorized_keys file.
>
> Regards,
> Vaibhav
>
> On Wed, Nov 25, 2009 at 9:45 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Can you please explain what you mean by "add master's public key to
> slave's
> > authorized_keys"? I have a feeling I am not doing this correctly. What
> I
> > did is this:
> >
> > 1) On Master, opened ~/.ssh/authorized_key and copied lines that look
> like
> > this...
> >
> > ssh-dss
> >
> >
> AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpXuJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTLlssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4OijkAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNGnWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbHa89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfOn6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2MldKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ==
> > root@domU-12-31-38-00-39-C8
> >
> >
> > to the end of ~/.ssh/authorized_key on Slave.
> >
> > Something tells me there's a better way to do this. Is there?
> >
> >
> > On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vp...@gmail.com>
> > wrote:
> >
> > > Actually I was wrong about adding keypair. It should be resolved by
> > adding
> > > master's public key to slave's authorized_keys. Amazon puts your
> keypair
> > in
> > > authroized_keys automatically. I missed that part.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > > On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Me again :(
> > > >
> > > > I decided to start fresh so that I can document the steps, but now I
> > > can't
> > > > even go further than I did before :(
> > > >
> > > > Please let me know what step I am missing:
> > > >
> > > > 1) Launched 2 instances on AWS console - one for Master, one for
> > Slave.
> > > >
> > > > 2) In one command window, connected to instance 1:
> > > > ssh -i MyKeyPair.pem <instance 1>
> > > > (Got 'authenticity' warning.. typed 'yes')
> > > >
> > > > This is my "Master" window.
> > > >
> > > > 3) Opened another command window & connected to instance 2:
> > > > ssh -i MyKeyPair.pem <instance 2>
> > > > (Got 'authenticity' warning.. typed 'yes')
> > > >
> > > > This is my "Slave" window.
> > > >
> > > > 4) Opened another command window and 'scp' the key pair (from my
> local
> > > > machine) to instance 1 & instance 2.
> > > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> > > >
> > > > 5) On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> > > >
> > > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> > > >
> > > > 6) To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > > > ssh localhost (Got 'authenticiy' message. Typed 'yes'. Got
> > Permission
> > > > denied (public key)
> > > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> > > >
> > > > 7) In order to ssh from Master to Slave (without entering KeyPair),
> > > copied
> > > > the key (that starts with 'sh-dss') from Slave's 'authorized_key'
> file
> > to
> > > > Master's 'authoried_key' file.
> > > >
> > > > Now, every time I try (from Master):
> > > >
> > > > ssh <slave's ip address>
> > > >
> > > > I get - "Permission denied (publickey).
> > > >
> > > > But, when I type:
> > > >
> > > > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> > > >
> > > > It works.
> > > >
> > > > I thought step #5 above should have resolved this. What am I doing
> > > wrong?
> > > > I believe, in order for me to start the cluster, I should be able to
> > > 'ssh'
> > > > to slave without using key pair, correct?
> > > >
> > > >
> > > > Thanks for your help.
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vpuranik@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Install a program called Jps.
> > > > >
> > > > > It's a ps for java processes. execute jps, you should see the
> > processes
> > > > (if
> > > > > they are running).
> > > > >
> > > > > Also make sure you check your logs.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Awesome. Thanks, Vaibhav. Made progress. No error messages
> from
> > > > > > ./all-start.sh
> > > > > >
> > > > > > I will test to see if the cluster is deployed properly.
> > > > > >
> > > > > > Question: On the slave, when I do:
> > > > > >
> > > > > > ps -eaf | grep 'hadoop'
> > > > > >
> > > > > > I don't see any processes. Shouldn't I see 2 processes - one for
> > > > > Datanode
> > > > > > and the other for TaskTracker?
> > > > > >
> > > > > > Please let me know. Thanks again.
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <
> > vpuranik@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Yes, I pretty much meant that.
> > > > > > >
> > > > > > > Add your master's public key to authorized_keys on slaves too.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vaibhav
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Sorry.. but not sure what you mean by... "In order to achieve
> > > that
> > > > > you
> > > > > > > need
> > > > > > > > to add the keypair you are using to your
> > > > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > > > >
> > > > > > > >
> > > > > > > > I tried this....
> > > > > > > >
> > > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > > > >
> > > > > > > > Is that what you meant by "add the keypair to
> authorized_keys"?
> > > > > > > >
> > > > > > > > Still can't ssh from Slave to Master. It says.....
> Permission
> > > > > denied
> > > > > > > > (publickey).
> > > > > > > >
> > > > > > > >
> > > > > > > > Sorry, I am a newbie when it comes to such environment
> issues.
> > > > > Thanks
> > > > > > > for
> > > > > > > > your patience. It's greatly appreciated.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > > > vpuranik@gmail.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > You need to be able to SSH without specifying the keypair
> > from
> > > a
> > > > > > slave
> > > > > > > to
> > > > > > > > > master (I think reverse it true too, but I am not sure).
> > > > > > > > >
> > > > > > > > > In order to achieve that you need to add the keypair you
> are
> > > > using
> > > > > > to
> > > > > > > > your
> > > > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > > > >
> > > > > > > > > Once you setup passphraseless ssh, it should start working.
> > > > > > Furthrmore,
> > > > > > > > > after you set this up, try to ssh manully (without key) and
> > if
> > > it
> > > > > > asks
> > > > > > > > you
> > > > > > > > > the following message - say yes:
> > > > > > > > >
> > > > > > > > > The authenticity of host
> > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > (147.127.186.243)'
> > > > > can't
> > > > > > > > > be established.
> > > > > > > > > RSA key fingerprint is
> > > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > > > > > Warning: Permanently added
> > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > ,147.127.186.243'
> > > > > > > > > (RSA) to the list of known hosts.
> > > > > > > > >
> > > > > > > > > In other words, you have to make sure that the machine you
> > are
> > > > > trying
> > > > > > > to
> > > > > > > > > ssh
> > > > > > > > > is in your known_hosts file.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Vaibhav
> > > > > > > > >
> > > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks, Vaibhav. That was helpful. I am able to scp
> from
> > my
> > > > > local
> > > > > > > > > machine
> > > > > > > > > > after I opened port 22 under my security group.
> > > > > > > > > >
> > > > > > > > > > But my cluster is still not starting as expected. I
> > mean...
> > > on
> > > > > my
> > > > > > > > Master
> > > > > > > > > > when I run:
> > > > > > > > > >
> > > > > > > > > > ./start-all.sh
> > > > > > > > > >
> > > > > > > > > > I get messages saying...
> > > > > > > > > >
> > > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > > > >
> > > > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > > > >
> > > > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Any suggestions? Thanks for your help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > > > vpuranik@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > In order to use scp from your machine to an ec2
> instance
> > > you
> > > > > need
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > scp
> > > > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > > > >
> > > > > > > > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > > > > > > > >
> > > > > > > > > > > Secondly in order to configure an ec2 cluster for
> Hadoop,
> > > you
> > > > > > need
> > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > a
> > > > > > > > > > > security group (let's call it hadoop) and give hadoop
> > > access
> > > > to
> > > > > > > > hadoop.
> > > > > > > > > > >
> > > > > > > > > > > You can read more about security groups here -
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > > > > > >
> > > > > > > > > > > You also need to make sure that you can ssh from all
> the
> > > > slaves
> > > > > > to
> > > > > > > > > master
> > > > > > > > > > > without specifying password.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Vaibhav Puranik
> > > > > > > > > > > GumGum
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > > > > > >
> > > > > > > > > > > > My Key Pair file is on my machine at work which is
> > behind
> > > a
> > > > > > corp
> > > > > > > > > > > firewall.
> > > > > > > > > > > > As such I can't 'scp' from the ec2 instance to my
> > local
> > > > > > machine
> > > > > > > at
> > > > > > > > > > work
> > > > > > > > > > > to
> > > > > > > > > > > > 'get' the file. So I need a way to 'send' a file
> from
> > my
> > > > > > machine
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > ec2
> > > > > > > > > > > > instance. I tried using 'scp' (from my machine at
> > work)
> > > to
> > > > > the
> > > > > > > ec2
> > > > > > > > > > > machine
> > > > > > > > > > > > but it says "Permission denied". Does this make
> sense?
> > > > > > > > > > > >
> > > > > > > > > > > > May be I need to use the command line tools for EC2.
> I
> > > am
> > > > > > > looking
> > > > > > > > > into
> > > > > > > > > > > > those right now, but if there's a better/easier way,
> > > please
> > > > > let
> > > > > > > me
> > > > > > > > > > know.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks once again for your help.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > > > zjffdu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The ssh is installed on ec2 by default, otherwise
> you
> > > > have
> > > > > no
> > > > > > > way
> > > > > > > > > to
> > > > > > > > > > > > login
> > > > > > > > > > > > > to ec2
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something
> Something
> > <
> > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As per Wikipedia (
> > > > > http://en.wikipedia.org/wiki/Secure_copy
> > > > > > )
> > > > > > > > "It
> > > > > > > > > > > does
> > > > > > > > > > > > so
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > connecting to the host using SSH and there
> executes
> > > an
> > > > > SCP
> > > > > > > > server
> > > > > > > > > > > > (scp)".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So if SSH isn't working SCP won't work, either.
> In
> > > any
> > > > > > case
> > > > > > > I
> > > > > > > > > > tried
> > > > > > > > > > > to
> > > > > > > > > > > > > scp
> > > > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Any other ideas? Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > > > zjffdu@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> > > Something
> > > > <
> > > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.
> (Yes,
> > > > > > > Cloudera's
> > > > > > > > > > > > > distribution
> > > > > > > > > > > > > > > > works well, but trying to do this myself so I
> > can
> > > > > learn
> > > > > > > > > what's
> > > > > > > > > > > > > > happening
> > > > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have a Master & a Slave. When I start HDFS
> > on
> > > > the
> > > > > > > > master,
> > > > > > > > > I
> > > > > > > > > > > get
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" -
> > > where
> > > > > > > 10.xxx
> > > > > > > > is
> > > > > > > > > > IP
> > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh
> > from
> > > > the
> > > > > > > master
> > > > > > > > > EC2
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the Slave EC2 instance. What's the best way
> to
> > > fix
> > > > > it?
> > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > I
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > "Key Pair" file on my master. I have a key
> > pair
> > > on
> > > > > my
> > > > > > > > local
> > > > > > > > > > > > machine,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > how do I transfer it to the EC2 machine? (I
> > > know,
> > > > I
> > > > > > > know,
> > > > > > > > I
> > > > > > > > > > > > agree..
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please help. Thanks.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Vaibhav Puranik <vp...@gmail.com>.
On master, in .ssh folder, there should be a file id_dsa.pub. This is
master's public key.
Contents of this file should be copied to slave's authorized_keys file.
Regards,
Vaibhav
On Wed, Nov 25, 2009 at 9:45 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Can you please explain what you mean by "add master's public key to slave's
> authorized_keys"? I have a feeling I am not doing this correctly. What I
> did is this:
>
> 1) On Master, opened ~/.ssh/authorized_key and copied lines that look like
> this...
>
> ssh-dss
>
> AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpXuJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTLlssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4OijkAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNGnWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbHa89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfOn6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2MldKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ==
> root@domU-12-31-38-00-39-C8
>
>
> to the end of ~/.ssh/authorized_key on Slave.
>
> Something tells me there's a better way to do this. Is there?
>
>
> On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vp...@gmail.com>
> wrote:
>
> > Actually I was wrong about adding keypair. It should be resolved by
> adding
> > master's public key to slave's authorized_keys. Amazon puts your keypair
> in
> > authroized_keys automatically. I missed that part.
> >
> > Regards,
> > Vaibhav
> >
> > On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Me again :(
> > >
> > > I decided to start fresh so that I can document the steps, but now I
> > can't
> > > even go further than I did before :(
> > >
> > > Please let me know what step I am missing:
> > >
> > > 1) Launched 2 instances on AWS console - one for Master, one for
> Slave.
> > >
> > > 2) In one command window, connected to instance 1:
> > > ssh -i MyKeyPair.pem <instance 1>
> > > (Got 'authenticity' warning.. typed 'yes')
> > >
> > > This is my "Master" window.
> > >
> > > 3) Opened another command window & connected to instance 2:
> > > ssh -i MyKeyPair.pem <instance 2>
> > > (Got 'authenticity' warning.. typed 'yes')
> > >
> > > This is my "Slave" window.
> > >
> > > 4) Opened another command window and 'scp' the key pair (from my local
> > > machine) to instance 1 & instance 2.
> > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> > >
> > > 5) On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> > >
> > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> > >
> > > 6) To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > > ssh localhost (Got 'authenticiy' message. Typed 'yes'. Got
> Permission
> > > denied (public key)
> > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> > >
> > > 7) In order to ssh from Master to Slave (without entering KeyPair),
> > copied
> > > the key (that starts with 'sh-dss') from Slave's 'authorized_key' file
> to
> > > Master's 'authoried_key' file.
> > >
> > > Now, every time I try (from Master):
> > >
> > > ssh <slave's ip address>
> > >
> > > I get - "Permission denied (publickey).
> > >
> > > But, when I type:
> > >
> > > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> > >
> > > It works.
> > >
> > > I thought step #5 above should have resolved this. What am I doing
> > wrong?
> > > I believe, in order for me to start the cluster, I should be able to
> > 'ssh'
> > > to slave without using key pair, correct?
> > >
> > >
> > > Thanks for your help.
> > >
> > >
> > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vp...@gmail.com>
> > > wrote:
> > >
> > > > Install a program called Jps.
> > > >
> > > > It's a ps for java processes. execute jps, you should see the
> processes
> > > (if
> > > > they are running).
> > > >
> > > > Also make sure you check your logs.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Awesome. Thanks, Vaibhav. Made progress. No error messages from
> > > > > ./all-start.sh
> > > > >
> > > > > I will test to see if the cluster is deployed properly.
> > > > >
> > > > > Question: On the slave, when I do:
> > > > >
> > > > > ps -eaf | grep 'hadoop'
> > > > >
> > > > > I don't see any processes. Shouldn't I see 2 processes - one for
> > > > Datanode
> > > > > and the other for TaskTracker?
> > > > >
> > > > > Please let me know. Thanks again.
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <
> vpuranik@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Yes, I pretty much meant that.
> > > > > >
> > > > > > Add your master's public key to authorized_keys on slaves too.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Sorry.. but not sure what you mean by... "In order to achieve
> > that
> > > > you
> > > > > > need
> > > > > > > to add the keypair you are using to your
> > > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > > >
> > > > > > >
> > > > > > > I tried this....
> > > > > > >
> > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > > >
> > > > > > > Is that what you meant by "add the keypair to authorized_keys"?
> > > > > > >
> > > > > > > Still can't ssh from Slave to Master. It says..... Permission
> > > > denied
> > > > > > > (publickey).
> > > > > > >
> > > > > > >
> > > > > > > Sorry, I am a newbie when it comes to such environment issues.
> > > > Thanks
> > > > > > for
> > > > > > > your patience. It's greatly appreciated.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > > vpuranik@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > You need to be able to SSH without specifying the keypair
> from
> > a
> > > > > slave
> > > > > > to
> > > > > > > > master (I think reverse it true too, but I am not sure).
> > > > > > > >
> > > > > > > > In order to achieve that you need to add the keypair you are
> > > using
> > > > > to
> > > > > > > your
> > > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > > >
> > > > > > > > Once you setup passphraseless ssh, it should start working.
> > > > > Furthrmore,
> > > > > > > > after you set this up, try to ssh manully (without key) and
> if
> > it
> > > > > asks
> > > > > > > you
> > > > > > > > the following message - say yes:
> > > > > > > >
> > > > > > > > The authenticity of host
> > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> (147.127.186.243)'
> > > > can't
> > > > > > > > be established.
> > > > > > > > RSA key fingerprint is
> > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > > > > Warning: Permanently added
> > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> ,147.127.186.243'
> > > > > > > > (RSA) to the list of known hosts.
> > > > > > > >
> > > > > > > > In other words, you have to make sure that the machine you
> are
> > > > trying
> > > > > > to
> > > > > > > > ssh
> > > > > > > > is in your known_hosts file.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Vaibhav
> > > > > > > >
> > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Thanks, Vaibhav. That was helpful. I am able to scp from
> my
> > > > local
> > > > > > > > machine
> > > > > > > > > after I opened port 22 under my security group.
> > > > > > > > >
> > > > > > > > > But my cluster is still not starting as expected. I
> mean...
> > on
> > > > my
> > > > > > > Master
> > > > > > > > > when I run:
> > > > > > > > >
> > > > > > > > > ./start-all.sh
> > > > > > > > >
> > > > > > > > > I get messages saying...
> > > > > > > > >
> > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > > >
> > > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > > >
> > > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Any suggestions? Thanks for your help.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > > vpuranik@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > In order to use scp from your machine to an ec2 instance
> > you
> > > > need
> > > > > > to
> > > > > > > > use
> > > > > > > > > > scp
> > > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > > >
> > > > > > > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > > > > > > >
> > > > > > > > > > Secondly in order to configure an ec2 cluster for Hadoop,
> > you
> > > > > need
> > > > > > to
> > > > > > > > > make
> > > > > > > > > > a
> > > > > > > > > > security group (let's call it hadoop) and give hadoop
> > access
> > > to
> > > > > > > hadoop.
> > > > > > > > > >
> > > > > > > > > > You can read more about security groups here -
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > > > > >
> > > > > > > > > > You also need to make sure that you can ssh from all the
> > > slaves
> > > > > to
> > > > > > > > master
> > > > > > > > > > without specifying password.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Vaibhav Puranik
> > > > > > > > > > GumGum
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > > > > >
> > > > > > > > > > > My Key Pair file is on my machine at work which is
> behind
> > a
> > > > > corp
> > > > > > > > > > firewall.
> > > > > > > > > > > As such I can't 'scp' from the ec2 instance to my
> local
> > > > > machine
> > > > > > at
> > > > > > > > > work
> > > > > > > > > > to
> > > > > > > > > > > 'get' the file. So I need a way to 'send' a file from
> my
> > > > > machine
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > ec2
> > > > > > > > > > > instance. I tried using 'scp' (from my machine at
> work)
> > to
> > > > the
> > > > > > ec2
> > > > > > > > > > machine
> > > > > > > > > > > but it says "Permission denied". Does this make sense?
> > > > > > > > > > >
> > > > > > > > > > > May be I need to use the command line tools for EC2. I
> > am
> > > > > > looking
> > > > > > > > into
> > > > > > > > > > > those right now, but if there's a better/easier way,
> > please
> > > > let
> > > > > > me
> > > > > > > > > know.
> > > > > > > > > > >
> > > > > > > > > > > Thanks once again for your help.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > > zjffdu@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The ssh is installed on ec2 by default, otherwise you
> > > have
> > > > no
> > > > > > way
> > > > > > > > to
> > > > > > > > > > > login
> > > > > > > > > > > > to ec2
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something
> <
> > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > As per Wikipedia (
> > > > http://en.wikipedia.org/wiki/Secure_copy
> > > > > )
> > > > > > > "It
> > > > > > > > > > does
> > > > > > > > > > > so
> > > > > > > > > > > > > by
> > > > > > > > > > > > > connecting to the host using SSH and there executes
> > an
> > > > SCP
> > > > > > > server
> > > > > > > > > > > (scp)".
> > > > > > > > > > > > >
> > > > > > > > > > > > > So if SSH isn't working SCP won't work, either. In
> > any
> > > > > case
> > > > > > I
> > > > > > > > > tried
> > > > > > > > > > to
> > > > > > > > > > > > scp
> > > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > > >
> > > > > > > > > > > > > Any other ideas? Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > > zjffdu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> > Something
> > > <
> > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes,
> > > > > > Cloudera's
> > > > > > > > > > > > distribution
> > > > > > > > > > > > > > > works well, but trying to do this myself so I
> can
> > > > learn
> > > > > > > > what's
> > > > > > > > > > > > > happening
> > > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have a Master & a Slave. When I start HDFS
> on
> > > the
> > > > > > > master,
> > > > > > > > I
> > > > > > > > > > get
> > > > > > > > > > > a
> > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" -
> > where
> > > > > > 10.xxx
> > > > > > > is
> > > > > > > > > IP
> > > > > > > > > > > > > address
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh
> from
> > > the
> > > > > > master
> > > > > > > > EC2
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the Slave EC2 instance. What's the best way to
> > fix
> > > > it?
> > > > > > I
> > > > > > > > > think
> > > > > > > > > > I
> > > > > > > > > > > > need
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > "Key Pair" file on my master. I have a key
> pair
> > on
> > > > my
> > > > > > > local
> > > > > > > > > > > machine,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > how do I transfer it to the EC2 machine? (I
> > know,
> > > I
> > > > > > know,
> > > > > > > I
> > > > > > > > > > > agree..
> > > > > > > > > > > > I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please help. Thanks.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
Can you please explain what you mean by "add master's public key to slave's
authorized_keys"? I have a feeling I am not doing this correctly. What I
did is this:
1) On Master, opened ~/.ssh/authorized_key and copied lines that look like
this...
ssh-dss
AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpXuJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTLlssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4OijkAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNGnWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbHa89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfOn6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2MldKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ==
root@domU-12-31-38-00-39-C8
to the end of ~/.ssh/authorized_key on Slave.
Something tells me there's a better way to do this. Is there?
On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vp...@gmail.com> wrote:
> Actually I was wrong about adding keypair. It should be resolved by adding
> master's public key to slave's authorized_keys. Amazon puts your keypair in
> authroized_keys automatically. I missed that part.
>
> Regards,
> Vaibhav
>
> On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Me again :(
> >
> > I decided to start fresh so that I can document the steps, but now I
> can't
> > even go further than I did before :(
> >
> > Please let me know what step I am missing:
> >
> > 1) Launched 2 instances on AWS console - one for Master, one for Slave.
> >
> > 2) In one command window, connected to instance 1:
> > ssh -i MyKeyPair.pem <instance 1>
> > (Got 'authenticity' warning.. typed 'yes')
> >
> > This is my "Master" window.
> >
> > 3) Opened another command window & connected to instance 2:
> > ssh -i MyKeyPair.pem <instance 2>
> > (Got 'authenticity' warning.. typed 'yes')
> >
> > This is my "Slave" window.
> >
> > 4) Opened another command window and 'scp' the key pair (from my local
> > machine) to instance 1 & instance 2.
> > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> >
> > 5) On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> >
> > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> >
> > 6) To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > ssh localhost (Got 'authenticiy' message. Typed 'yes'. Got Permission
> > denied (public key)
> > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> >
> > 7) In order to ssh from Master to Slave (without entering KeyPair),
> copied
> > the key (that starts with 'sh-dss') from Slave's 'authorized_key' file to
> > Master's 'authoried_key' file.
> >
> > Now, every time I try (from Master):
> >
> > ssh <slave's ip address>
> >
> > I get - "Permission denied (publickey).
> >
> > But, when I type:
> >
> > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> >
> > It works.
> >
> > I thought step #5 above should have resolved this. What am I doing
> wrong?
> > I believe, in order for me to start the cluster, I should be able to
> 'ssh'
> > to slave without using key pair, correct?
> >
> >
> > Thanks for your help.
> >
> >
> > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vp...@gmail.com>
> > wrote:
> >
> > > Install a program called Jps.
> > >
> > > It's a ps for java processes. execute jps, you should see the processes
> > (if
> > > they are running).
> > >
> > > Also make sure you check your logs.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Awesome. Thanks, Vaibhav. Made progress. No error messages from
> > > > ./all-start.sh
> > > >
> > > > I will test to see if the cluster is deployed properly.
> > > >
> > > > Question: On the slave, when I do:
> > > >
> > > > ps -eaf | grep 'hadoop'
> > > >
> > > > I don't see any processes. Shouldn't I see 2 processes - one for
> > > Datanode
> > > > and the other for TaskTracker?
> > > >
> > > > Please let me know. Thanks again.
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vpuranik@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Yes, I pretty much meant that.
> > > > >
> > > > > Add your master's public key to authorized_keys on slaves too.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Sorry.. but not sure what you mean by... "In order to achieve
> that
> > > you
> > > > > need
> > > > > > to add the keypair you are using to your
> > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > >
> > > > > >
> > > > > > I tried this....
> > > > > >
> > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > >
> > > > > > Is that what you meant by "add the keypair to authorized_keys"?
> > > > > >
> > > > > > Still can't ssh from Slave to Master. It says..... Permission
> > > denied
> > > > > > (publickey).
> > > > > >
> > > > > >
> > > > > > Sorry, I am a newbie when it comes to such environment issues.
> > > Thanks
> > > > > for
> > > > > > your patience. It's greatly appreciated.
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > vpuranik@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > You need to be able to SSH without specifying the keypair from
> a
> > > > slave
> > > > > to
> > > > > > > master (I think reverse it true too, but I am not sure).
> > > > > > >
> > > > > > > In order to achieve that you need to add the keypair you are
> > using
> > > > to
> > > > > > your
> > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > >
> > > > > > > Once you setup passphraseless ssh, it should start working.
> > > > Furthrmore,
> > > > > > > after you set this up, try to ssh manully (without key) and if
> it
> > > > asks
> > > > > > you
> > > > > > > the following message - say yes:
> > > > > > >
> > > > > > > The authenticity of host
> > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)'
> > > can't
> > > > > > > be established.
> > > > > > > RSA key fingerprint is
> > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > > > Warning: Permanently added
> > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > > > > (RSA) to the list of known hosts.
> > > > > > >
> > > > > > > In other words, you have to make sure that the machine you are
> > > trying
> > > > > to
> > > > > > > ssh
> > > > > > > is in your known_hosts file.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vaibhav
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Thanks, Vaibhav. That was helpful. I am able to scp from my
> > > local
> > > > > > > machine
> > > > > > > > after I opened port 22 under my security group.
> > > > > > > >
> > > > > > > > But my cluster is still not starting as expected. I mean...
> on
> > > my
> > > > > > Master
> > > > > > > > when I run:
> > > > > > > >
> > > > > > > > ./start-all.sh
> > > > > > > >
> > > > > > > > I get messages saying...
> > > > > > > >
> > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > >
> > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > >
> > > > > > > >
> > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > >
> > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > >
> > > > > > > >
> > > > > > > > Any suggestions? Thanks for your help.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > vpuranik@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > In order to use scp from your machine to an ec2 instance
> you
> > > need
> > > > > to
> > > > > > > use
> > > > > > > > > scp
> > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > >
> > > > > > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > > > > > >
> > > > > > > > > Secondly in order to configure an ec2 cluster for Hadoop,
> you
> > > > need
> > > > > to
> > > > > > > > make
> > > > > > > > > a
> > > > > > > > > security group (let's call it hadoop) and give hadoop
> access
> > to
> > > > > > hadoop.
> > > > > > > > >
> > > > > > > > > You can read more about security groups here -
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > > > >
> > > > > > > > > You also need to make sure that you can ssh from all the
> > slaves
> > > > to
> > > > > > > master
> > > > > > > > > without specifying password.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Vaibhav Puranik
> > > > > > > > > GumGum
> > > > > > > > >
> > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > > > >
> > > > > > > > > > My Key Pair file is on my machine at work which is behind
> a
> > > > corp
> > > > > > > > > firewall.
> > > > > > > > > > As such I can't 'scp' from the ec2 instance to my local
> > > > machine
> > > > > at
> > > > > > > > work
> > > > > > > > > to
> > > > > > > > > > 'get' the file. So I need a way to 'send' a file from my
> > > > machine
> > > > > > to
> > > > > > > > the
> > > > > > > > > > ec2
> > > > > > > > > > instance. I tried using 'scp' (from my machine at work)
> to
> > > the
> > > > > ec2
> > > > > > > > > machine
> > > > > > > > > > but it says "Permission denied". Does this make sense?
> > > > > > > > > >
> > > > > > > > > > May be I need to use the command line tools for EC2. I
> am
> > > > > looking
> > > > > > > into
> > > > > > > > > > those right now, but if there's a better/easier way,
> please
> > > let
> > > > > me
> > > > > > > > know.
> > > > > > > > > >
> > > > > > > > > > Thanks once again for your help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > zjffdu@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The ssh is installed on ec2 by default, otherwise you
> > have
> > > no
> > > > > way
> > > > > > > to
> > > > > > > > > > login
> > > > > > > > > > > to ec2
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jeff Zhang
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > As per Wikipedia (
> > > http://en.wikipedia.org/wiki/Secure_copy
> > > > )
> > > > > > "It
> > > > > > > > > does
> > > > > > > > > > so
> > > > > > > > > > > > by
> > > > > > > > > > > > connecting to the host using SSH and there executes
> an
> > > SCP
> > > > > > server
> > > > > > > > > > (scp)".
> > > > > > > > > > > >
> > > > > > > > > > > > So if SSH isn't working SCP won't work, either. In
> any
> > > > case
> > > > > I
> > > > > > > > tried
> > > > > > > > > to
> > > > > > > > > > > scp
> > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > >
> > > > > > > > > > > > Any other ideas? Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > zjffdu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> Something
> > <
> > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes,
> > > > > Cloudera's
> > > > > > > > > > > distribution
> > > > > > > > > > > > > > works well, but trying to do this myself so I can
> > > learn
> > > > > > > what's
> > > > > > > > > > > > happening
> > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have a Master & a Slave. When I start HDFS on
> > the
> > > > > > master,
> > > > > > > I
> > > > > > > > > get
> > > > > > > > > > a
> > > > > > > > > > > > > > message
> > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" -
> where
> > > > > 10.xxx
> > > > > > is
> > > > > > > > IP
> > > > > > > > > > > > address
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh from
> > the
> > > > > master
> > > > > > > EC2
> > > > > > > > > > > > instance
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > the Slave EC2 instance. What's the best way to
> fix
> > > it?
> > > > > I
> > > > > > > > think
> > > > > > > > > I
> > > > > > > > > > > need
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > "Key Pair" file on my master. I have a key pair
> on
> > > my
> > > > > > local
> > > > > > > > > > machine,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > how do I transfer it to the EC2 machine? (I
> know,
> > I
> > > > > know,
> > > > > > I
> > > > > > > > > > agree..
> > > > > > > > > > > I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please help. Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Vaibhav Puranik <vp...@gmail.com>.
Actually I was wrong about adding keypair. It should be resolved by adding
master's public key to slave's authorized_keys. Amazon puts your keypair in
authroized_keys automatically. I missed that part.
Regards,
Vaibhav
On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Me again :(
>
> I decided to start fresh so that I can document the steps, but now I can't
> even go further than I did before :(
>
> Please let me know what step I am missing:
>
> 1) Launched 2 instances on AWS console - one for Master, one for Slave.
>
> 2) In one command window, connected to instance 1:
> ssh -i MyKeyPair.pem <instance 1>
> (Got 'authenticity' warning.. typed 'yes')
>
> This is my "Master" window.
>
> 3) Opened another command window & connected to instance 2:
> ssh -i MyKeyPair.pem <instance 2>
> (Got 'authenticity' warning.. typed 'yes')
>
> This is my "Slave" window.
>
> 4) Opened another command window and 'scp' the key pair (from my local
> machine) to instance 1 & instance 2.
> scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
>
> 5) On BOTH (Master & Slave) added Key Pair to 'authorized_key'
>
> cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
>
> 6) To setup passphraseless ssh ran these on BOTH - Master & Slave:
> ssh localhost (Got 'authenticiy' message. Typed 'yes'. Got Permission
> denied (public key)
> ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
>
> 7) In order to ssh from Master to Slave (without entering KeyPair), copied
> the key (that starts with 'sh-dss') from Slave's 'authorized_key' file to
> Master's 'authoried_key' file.
>
> Now, every time I try (from Master):
>
> ssh <slave's ip address>
>
> I get - "Permission denied (publickey).
>
> But, when I type:
>
> ssh -i /tmp/MyKeyPair.pem <slave's ip address>
>
> It works.
>
> I thought step #5 above should have resolved this. What am I doing wrong?
> I believe, in order for me to start the cluster, I should be able to 'ssh'
> to slave without using key pair, correct?
>
>
> Thanks for your help.
>
>
> On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vp...@gmail.com>
> wrote:
>
> > Install a program called Jps.
> >
> > It's a ps for java processes. execute jps, you should see the processes
> (if
> > they are running).
> >
> > Also make sure you check your logs.
> >
> > Regards,
> > Vaibhav
> >
> > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Awesome. Thanks, Vaibhav. Made progress. No error messages from
> > > ./all-start.sh
> > >
> > > I will test to see if the cluster is deployed properly.
> > >
> > > Question: On the slave, when I do:
> > >
> > > ps -eaf | grep 'hadoop'
> > >
> > > I don't see any processes. Shouldn't I see 2 processes - one for
> > Datanode
> > > and the other for TaskTracker?
> > >
> > > Please let me know. Thanks again.
> > >
> > >
> > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vp...@gmail.com>
> > > wrote:
> > >
> > > > Yes, I pretty much meant that.
> > > >
> > > > Add your master's public key to authorized_keys on slaves too.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Sorry.. but not sure what you mean by... "In order to achieve that
> > you
> > > > need
> > > > > to add the keypair you are using to your
> > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > >
> > > > >
> > > > > I tried this....
> > > > >
> > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > >
> > > > > Is that what you meant by "add the keypair to authorized_keys"?
> > > > >
> > > > > Still can't ssh from Slave to Master. It says..... Permission
> > denied
> > > > > (publickey).
> > > > >
> > > > >
> > > > > Sorry, I am a newbie when it comes to such environment issues.
> > Thanks
> > > > for
> > > > > your patience. It's greatly appreciated.
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > vpuranik@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > You need to be able to SSH without specifying the keypair from a
> > > slave
> > > > to
> > > > > > master (I think reverse it true too, but I am not sure).
> > > > > >
> > > > > > In order to achieve that you need to add the keypair you are
> using
> > > to
> > > > > your
> > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > >
> > > > > > Once you setup passphraseless ssh, it should start working.
> > > Furthrmore,
> > > > > > after you set this up, try to ssh manully (without key) and if it
> > > asks
> > > > > you
> > > > > > the following message - say yes:
> > > > > >
> > > > > > The authenticity of host
> > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)'
> > can't
> > > > > > be established.
> > > > > > RSA key fingerprint is
> > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > > Warning: Permanently added
> > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > > > (RSA) to the list of known hosts.
> > > > > >
> > > > > > In other words, you have to make sure that the machine you are
> > trying
> > > > to
> > > > > > ssh
> > > > > > is in your known_hosts file.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks, Vaibhav. That was helpful. I am able to scp from my
> > local
> > > > > > machine
> > > > > > > after I opened port 22 under my security group.
> > > > > > >
> > > > > > > But my cluster is still not starting as expected. I mean... on
> > my
> > > > > Master
> > > > > > > when I run:
> > > > > > >
> > > > > > > ./start-all.sh
> > > > > > >
> > > > > > > I get messages saying...
> > > > > > >
> > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > >
> > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > >
> > > > > > >
> > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > >
> > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > >
> > > > > > >
> > > > > > > Any suggestions? Thanks for your help.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > vpuranik@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > In order to use scp from your machine to an ec2 instance you
> > need
> > > > to
> > > > > > use
> > > > > > > > scp
> > > > > > > > -i and give it the path of your keypair.
> > > > > > > >
> > > > > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > > > > >
> > > > > > > > Secondly in order to configure an ec2 cluster for Hadoop, you
> > > need
> > > > to
> > > > > > > make
> > > > > > > > a
> > > > > > > > security group (let's call it hadoop) and give hadoop access
> to
> > > > > hadoop.
> > > > > > > >
> > > > > > > > You can read more about security groups here -
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > > >
> > > > > > > > You also need to make sure that you can ssh from all the
> slaves
> > > to
> > > > > > master
> > > > > > > > without specifying password.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Vaibhav Puranik
> > > > > > > > GumGum
> > > > > > > >
> > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > > >
> > > > > > > > > My Key Pair file is on my machine at work which is behind a
> > > corp
> > > > > > > > firewall.
> > > > > > > > > As such I can't 'scp' from the ec2 instance to my local
> > > machine
> > > > at
> > > > > > > work
> > > > > > > > to
> > > > > > > > > 'get' the file. So I need a way to 'send' a file from my
> > > machine
> > > > > to
> > > > > > > the
> > > > > > > > > ec2
> > > > > > > > > instance. I tried using 'scp' (from my machine at work) to
> > the
> > > > ec2
> > > > > > > > machine
> > > > > > > > > but it says "Permission denied". Does this make sense?
> > > > > > > > >
> > > > > > > > > May be I need to use the command line tools for EC2. I am
> > > > looking
> > > > > > into
> > > > > > > > > those right now, but if there's a better/easier way, please
> > let
> > > > me
> > > > > > > know.
> > > > > > > > >
> > > > > > > > > Thanks once again for your help.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > zjffdu@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The ssh is installed on ec2 by default, otherwise you
> have
> > no
> > > > way
> > > > > > to
> > > > > > > > > login
> > > > > > > > > > to ec2
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jeff Zhang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > As per Wikipedia (
> > http://en.wikipedia.org/wiki/Secure_copy
> > > )
> > > > > "It
> > > > > > > > does
> > > > > > > > > so
> > > > > > > > > > > by
> > > > > > > > > > > connecting to the host using SSH and there executes an
> > SCP
> > > > > server
> > > > > > > > > (scp)".
> > > > > > > > > > >
> > > > > > > > > > > So if SSH isn't working SCP won't work, either. In any
> > > case
> > > > I
> > > > > > > tried
> > > > > > > > to
> > > > > > > > > > scp
> > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > >
> > > > > > > > > > > Any other ideas? Thanks.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > zjffdu@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > >
> > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something
> <
> > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes,
> > > > Cloudera's
> > > > > > > > > > distribution
> > > > > > > > > > > > > works well, but trying to do this myself so I can
> > learn
> > > > > > what's
> > > > > > > > > > > happening
> > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have a Master & a Slave. When I start HDFS on
> the
> > > > > master,
> > > > > > I
> > > > > > > > get
> > > > > > > > > a
> > > > > > > > > > > > > message
> > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> > > > 10.xxx
> > > > > is
> > > > > > > IP
> > > > > > > > > > > address
> > > > > > > > > > > > of
> > > > > > > > > > > > > the slave.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Basic problem (I think) is that I can't ssh from
> the
> > > > master
> > > > > > EC2
> > > > > > > > > > > instance
> > > > > > > > > > > > to
> > > > > > > > > > > > > the Slave EC2 instance. What's the best way to fix
> > it?
> > > > I
> > > > > > > think
> > > > > > > > I
> > > > > > > > > > need
> > > > > > > > > > > > the
> > > > > > > > > > > > > "Key Pair" file on my master. I have a key pair on
> > my
> > > > > local
> > > > > > > > > machine,
> > > > > > > > > > > but
> > > > > > > > > > > > > how do I transfer it to the EC2 machine? (I know,
> I
> > > > know,
> > > > > I
> > > > > > > > > agree..
> > > > > > > > > > I
> > > > > > > > > > > am
> > > > > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please help. Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
Me again :(
I decided to start fresh so that I can document the steps, but now I can't
even go further than I did before :(
Please let me know what step I am missing:
1) Launched 2 instances on AWS console - one for Master, one for Slave.
2) In one command window, connected to instance 1:
ssh -i MyKeyPair.pem <instance 1>
(Got 'authenticity' warning.. typed 'yes')
This is my "Master" window.
3) Opened another command window & connected to instance 2:
ssh -i MyKeyPair.pem <instance 2>
(Got 'authenticity' warning.. typed 'yes')
This is my "Slave" window.
4) Opened another command window and 'scp' the key pair (from my local
machine) to instance 1 & instance 2.
scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
5) On BOTH (Master & Slave) added Key Pair to 'authorized_key'
cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
6) To setup passphraseless ssh ran these on BOTH - Master & Slave:
ssh localhost (Got 'authenticiy' message. Typed 'yes'. Got Permission
denied (public key)
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
7) In order to ssh from Master to Slave (without entering KeyPair), copied
the key (that starts with 'sh-dss') from Slave's 'authorized_key' file to
Master's 'authoried_key' file.
Now, every time I try (from Master):
ssh <slave's ip address>
I get - "Permission denied (publickey).
But, when I type:
ssh -i /tmp/MyKeyPair.pem <slave's ip address>
It works.
I thought step #5 above should have resolved this. What am I doing wrong?
I believe, in order for me to start the cluster, I should be able to 'ssh'
to slave without using key pair, correct?
Thanks for your help.
On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vp...@gmail.com> wrote:
> Install a program called Jps.
>
> It's a ps for java processes. execute jps, you should see the processes (if
> they are running).
>
> Also make sure you check your logs.
>
> Regards,
> Vaibhav
>
> On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Awesome. Thanks, Vaibhav. Made progress. No error messages from
> > ./all-start.sh
> >
> > I will test to see if the cluster is deployed properly.
> >
> > Question: On the slave, when I do:
> >
> > ps -eaf | grep 'hadoop'
> >
> > I don't see any processes. Shouldn't I see 2 processes - one for
> Datanode
> > and the other for TaskTracker?
> >
> > Please let me know. Thanks again.
> >
> >
> > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vp...@gmail.com>
> > wrote:
> >
> > > Yes, I pretty much meant that.
> > >
> > > Add your master's public key to authorized_keys on slaves too.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > >
> > >
> > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Sorry.. but not sure what you mean by... "In order to achieve that
> you
> > > need
> > > > to add the keypair you are using to your
> > > > auhtorized_keys file in the home dir/.ssh folder."
> > > >
> > > >
> > > > I tried this....
> > > >
> > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > >
> > > > Is that what you meant by "add the keypair to authorized_keys"?
> > > >
> > > > Still can't ssh from Slave to Master. It says..... Permission
> denied
> > > > (publickey).
> > > >
> > > >
> > > > Sorry, I am a newbie when it comes to such environment issues.
> Thanks
> > > for
> > > > your patience. It's greatly appreciated.
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> vpuranik@gmail.com
> > > > >wrote:
> > > >
> > > > > You need to be able to SSH without specifying the keypair from a
> > slave
> > > to
> > > > > master (I think reverse it true too, but I am not sure).
> > > > >
> > > > > In order to achieve that you need to add the keypair you are using
> > to
> > > > your
> > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > >
> > > > > Once you setup passphraseless ssh, it should start working.
> > Furthrmore,
> > > > > after you set this up, try to ssh manully (without key) and if it
> > asks
> > > > you
> > > > > the following message - say yes:
> > > > >
> > > > > The authenticity of host
> > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)'
> can't
> > > > > be established.
> > > > > RSA key fingerprint is
> > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > Warning: Permanently added
> > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > > (RSA) to the list of known hosts.
> > > > >
> > > > > In other words, you have to make sure that the machine you are
> trying
> > > to
> > > > > ssh
> > > > > is in your known_hosts file.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Thanks, Vaibhav. That was helpful. I am able to scp from my
> local
> > > > > machine
> > > > > > after I opened port 22 under my security group.
> > > > > >
> > > > > > But my cluster is still not starting as expected. I mean... on
> my
> > > > Master
> > > > > > when I run:
> > > > > >
> > > > > > ./start-all.sh
> > > > > >
> > > > > > I get messages saying...
> > > > > >
> > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > >
> > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > >
> > > > > >
> > > > > > I can ssh from my Slave to master if I do this:
> > > > > >
> > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > >
> > > > > >
> > > > > > Any suggestions? Thanks for your help.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > vpuranik@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > In order to use scp from your machine to an ec2 instance you
> need
> > > to
> > > > > use
> > > > > > > scp
> > > > > > > -i and give it the path of your keypair.
> > > > > > >
> > > > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > > > >
> > > > > > > Secondly in order to configure an ec2 cluster for Hadoop, you
> > need
> > > to
> > > > > > make
> > > > > > > a
> > > > > > > security group (let's call it hadoop) and give hadoop access to
> > > > hadoop.
> > > > > > >
> > > > > > > You can read more about security groups here -
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > >
> > > > > > > You also need to make sure that you can ssh from all the slaves
> > to
> > > > > master
> > > > > > > without specifying password.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vaibhav Puranik
> > > > > > > GumGum
> > > > > > >
> > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > >
> > > > > > > > My Key Pair file is on my machine at work which is behind a
> > corp
> > > > > > > firewall.
> > > > > > > > As such I can't 'scp' from the ec2 instance to my local
> > machine
> > > at
> > > > > > work
> > > > > > > to
> > > > > > > > 'get' the file. So I need a way to 'send' a file from my
> > machine
> > > > to
> > > > > > the
> > > > > > > > ec2
> > > > > > > > instance. I tried using 'scp' (from my machine at work) to
> the
> > > ec2
> > > > > > > machine
> > > > > > > > but it says "Permission denied". Does this make sense?
> > > > > > > >
> > > > > > > > May be I need to use the command line tools for EC2. I am
> > > looking
> > > > > into
> > > > > > > > those right now, but if there's a better/easier way, please
> let
> > > me
> > > > > > know.
> > > > > > > >
> > > > > > > > Thanks once again for your help.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> zjffdu@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > The ssh is installed on ec2 by default, otherwise you have
> no
> > > way
> > > > > to
> > > > > > > > login
> > > > > > > > > to ec2
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jeff Zhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > As per Wikipedia (
> http://en.wikipedia.org/wiki/Secure_copy
> > )
> > > > "It
> > > > > > > does
> > > > > > > > so
> > > > > > > > > > by
> > > > > > > > > > connecting to the host using SSH and there executes an
> SCP
> > > > server
> > > > > > > > (scp)".
> > > > > > > > > >
> > > > > > > > > > So if SSH isn't working SCP won't work, either. In any
> > case
> > > I
> > > > > > tried
> > > > > > > to
> > > > > > > > > scp
> > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > >
> > > > > > > > > > Any other ideas? Thanks.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > zjffdu@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > >
> > > > > > > > > > > Jeff Zhang
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes,
> > > Cloudera's
> > > > > > > > > distribution
> > > > > > > > > > > > works well, but trying to do this myself so I can
> learn
> > > > > what's
> > > > > > > > > > happening
> > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > >
> > > > > > > > > > > > I have a Master & a Slave. When I start HDFS on the
> > > > master,
> > > > > I
> > > > > > > get
> > > > > > > > a
> > > > > > > > > > > > message
> > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> > > 10.xxx
> > > > is
> > > > > > IP
> > > > > > > > > > address
> > > > > > > > > > > of
> > > > > > > > > > > > the slave.
> > > > > > > > > > > >
> > > > > > > > > > > > Basic problem (I think) is that I can't ssh from the
> > > master
> > > > > EC2
> > > > > > > > > > instance
> > > > > > > > > > > to
> > > > > > > > > > > > the Slave EC2 instance. What's the best way to fix
> it?
> > > I
> > > > > > think
> > > > > > > I
> > > > > > > > > need
> > > > > > > > > > > the
> > > > > > > > > > > > "Key Pair" file on my master. I have a key pair on
> my
> > > > local
> > > > > > > > machine,
> > > > > > > > > > but
> > > > > > > > > > > > how do I transfer it to the EC2 machine? (I know, I
> > > know,
> > > > I
> > > > > > > > agree..
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > > > >
> > > > > > > > > > > > Please help. Thanks.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Vaibhav Puranik <vp...@gmail.com>.
Install a program called Jps.
It's a ps for java processes. execute jps, you should see the processes (if
they are running).
Also make sure you check your logs.
Regards,
Vaibhav
On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Awesome. Thanks, Vaibhav. Made progress. No error messages from
> ./all-start.sh
>
> I will test to see if the cluster is deployed properly.
>
> Question: On the slave, when I do:
>
> ps -eaf | grep 'hadoop'
>
> I don't see any processes. Shouldn't I see 2 processes - one for Datanode
> and the other for TaskTracker?
>
> Please let me know. Thanks again.
>
>
> On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vp...@gmail.com>
> wrote:
>
> > Yes, I pretty much meant that.
> >
> > Add your master's public key to authorized_keys on slaves too.
> >
> > Regards,
> > Vaibhav
> >
> >
> >
> > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Sorry.. but not sure what you mean by... "In order to achieve that you
> > need
> > > to add the keypair you are using to your
> > > auhtorized_keys file in the home dir/.ssh folder."
> > >
> > >
> > > I tried this....
> > >
> > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > >
> > > Is that what you meant by "add the keypair to authorized_keys"?
> > >
> > > Still can't ssh from Slave to Master. It says..... Permission denied
> > > (publickey).
> > >
> > >
> > > Sorry, I am a newbie when it comes to such environment issues. Thanks
> > for
> > > your patience. It's greatly appreciated.
> > >
> > >
> > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com
> > > >wrote:
> > >
> > > > You need to be able to SSH without specifying the keypair from a
> slave
> > to
> > > > master (I think reverse it true too, but I am not sure).
> > > >
> > > > In order to achieve that you need to add the keypair you are using
> to
> > > your
> > > > auhtorized_keys file in the home dir/.ssh folder.
> > > >
> > > > Once you setup passphraseless ssh, it should start working.
> Furthrmore,
> > > > after you set this up, try to ssh manully (without key) and if it
> asks
> > > you
> > > > the following message - say yes:
> > > >
> > > > The authenticity of host
> > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> > > > be established.
> > > > RSA key fingerprint is
> 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > Warning: Permanently added
> > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > (RSA) to the list of known hosts.
> > > >
> > > > In other words, you have to make sure that the machine you are trying
> > to
> > > > ssh
> > > > is in your known_hosts file.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Thanks, Vaibhav. That was helpful. I am able to scp from my local
> > > > machine
> > > > > after I opened port 22 under my security group.
> > > > >
> > > > > But my cluster is still not starting as expected. I mean... on my
> > > Master
> > > > > when I run:
> > > > >
> > > > > ./start-all.sh
> > > > >
> > > > > I get messages saying...
> > > > >
> > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > >
> > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > >
> > > > >
> > > > > I can ssh from my Slave to master if I do this:
> > > > >
> > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > >
> > > > >
> > > > > Any suggestions? Thanks for your help.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> vpuranik@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > In order to use scp from your machine to an ec2 instance you need
> > to
> > > > use
> > > > > > scp
> > > > > > -i and give it the path of your keypair.
> > > > > >
> > > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > > >
> > > > > > Secondly in order to configure an ec2 cluster for Hadoop, you
> need
> > to
> > > > > make
> > > > > > a
> > > > > > security group (let's call it hadoop) and give hadoop access to
> > > hadoop.
> > > > > >
> > > > > > You can read more about security groups here -
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > >
> > > > > > You also need to make sure that you can ssh from all the slaves
> to
> > > > master
> > > > > > without specifying password.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav Puranik
> > > > > > GumGum
> > > > > >
> > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > >
> > > > > > > My Key Pair file is on my machine at work which is behind a
> corp
> > > > > > firewall.
> > > > > > > As such I can't 'scp' from the ec2 instance to my local
> machine
> > at
> > > > > work
> > > > > > to
> > > > > > > 'get' the file. So I need a way to 'send' a file from my
> machine
> > > to
> > > > > the
> > > > > > > ec2
> > > > > > > instance. I tried using 'scp' (from my machine at work) to the
> > ec2
> > > > > > machine
> > > > > > > but it says "Permission denied". Does this make sense?
> > > > > > >
> > > > > > > May be I need to use the command line tools for EC2. I am
> > looking
> > > > into
> > > > > > > those right now, but if there's a better/easier way, please let
> > me
> > > > > know.
> > > > > > >
> > > > > > > Thanks once again for your help.
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > The ssh is installed on ec2 by default, otherwise you have no
> > way
> > > > to
> > > > > > > login
> > > > > > > > to ec2
> > > > > > > >
> > > > > > > >
> > > > > > > > Jeff Zhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy
> )
> > > "It
> > > > > > does
> > > > > > > so
> > > > > > > > > by
> > > > > > > > > connecting to the host using SSH and there executes an SCP
> > > server
> > > > > > > (scp)".
> > > > > > > > >
> > > > > > > > > So if SSH isn't working SCP won't work, either. In any
> case
> > I
> > > > > tried
> > > > > > to
> > > > > > > > scp
> > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > >
> > > > > > > > > Any other ideas? Thanks.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > zjffdu@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > >
> > > > > > > > > > Jeff Zhang
> > > > > > > > > >
> > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes,
> > Cloudera's
> > > > > > > > distribution
> > > > > > > > > > > works well, but trying to do this myself so I can learn
> > > > what's
> > > > > > > > > happening
> > > > > > > > > > > behind the scene.)
> > > > > > > > > > >
> > > > > > > > > > > I have a Master & a Slave. When I start HDFS on the
> > > master,
> > > > I
> > > > > > get
> > > > > > > a
> > > > > > > > > > > message
> > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> > 10.xxx
> > > is
> > > > > IP
> > > > > > > > > address
> > > > > > > > > > of
> > > > > > > > > > > the slave.
> > > > > > > > > > >
> > > > > > > > > > > Basic problem (I think) is that I can't ssh from the
> > master
> > > > EC2
> > > > > > > > > instance
> > > > > > > > > > to
> > > > > > > > > > > the Slave EC2 instance. What's the best way to fix it?
> > I
> > > > > think
> > > > > > I
> > > > > > > > need
> > > > > > > > > > the
> > > > > > > > > > > "Key Pair" file on my master. I have a key pair on my
> > > local
> > > > > > > machine,
> > > > > > > > > but
> > > > > > > > > > > how do I transfer it to the EC2 machine? (I know, I
> > know,
> > > I
> > > > > > > agree..
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > > >
> > > > > > > > > > > Please help. Thanks.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
Awesome. Thanks, Vaibhav. Made progress. No error messages from
./all-start.sh
I will test to see if the cluster is deployed properly.
Question: On the slave, when I do:
ps -eaf | grep 'hadoop'
I don't see any processes. Shouldn't I see 2 processes - one for Datanode
and the other for TaskTracker?
Please let me know. Thanks again.
On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vp...@gmail.com> wrote:
> Yes, I pretty much meant that.
>
> Add your master's public key to authorized_keys on slaves too.
>
> Regards,
> Vaibhav
>
>
>
> On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Sorry.. but not sure what you mean by... "In order to achieve that you
> need
> > to add the keypair you are using to your
> > auhtorized_keys file in the home dir/.ssh folder."
> >
> >
> > I tried this....
> >
> > cat KeyPair.pem >> /root/.ssh/authorized_keys
> >
> > Is that what you meant by "add the keypair to authorized_keys"?
> >
> > Still can't ssh from Slave to Master. It says..... Permission denied
> > (publickey).
> >
> >
> > Sorry, I am a newbie when it comes to such environment issues. Thanks
> for
> > your patience. It's greatly appreciated.
> >
> >
> > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com
> > >wrote:
> >
> > > You need to be able to SSH without specifying the keypair from a slave
> to
> > > master (I think reverse it true too, but I am not sure).
> > >
> > > In order to achieve that you need to add the keypair you are using to
> > your
> > > auhtorized_keys file in the home dir/.ssh folder.
> > >
> > > Once you setup passphraseless ssh, it should start working. Furthrmore,
> > > after you set this up, try to ssh manully (without key) and if it asks
> > you
> > > the following message - say yes:
> > >
> > > The authenticity of host
> > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> > > be established.
> > > RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > Are you sure you want to continue connecting (yes/no)? yes
> > > Warning: Permanently added
> > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > (RSA) to the list of known hosts.
> > >
> > > In other words, you have to make sure that the machine you are trying
> to
> > > ssh
> > > is in your known_hosts file.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Thanks, Vaibhav. That was helpful. I am able to scp from my local
> > > machine
> > > > after I opened port 22 under my security group.
> > > >
> > > > But my cluster is still not starting as expected. I mean... on my
> > Master
> > > > when I run:
> > > >
> > > > ./start-all.sh
> > > >
> > > > I get messages saying...
> > > >
> > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > >
> > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > >
> > > >
> > > > I can ssh from my Slave to master if I do this:
> > > >
> > > > ssh -i <keypairname>.pem <Master's ip address>
> > > >
> > > >
> > > > Any suggestions? Thanks for your help.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com
> >
> > > > wrote:
> > > >
> > > > > In order to use scp from your machine to an ec2 instance you need
> to
> > > use
> > > > > scp
> > > > > -i and give it the path of your keypair.
> > > > >
> > > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > > >
> > > > > Secondly in order to configure an ec2 cluster for Hadoop, you need
> to
> > > > make
> > > > > a
> > > > > security group (let's call it hadoop) and give hadoop access to
> > hadoop.
> > > > >
> > > > > You can read more about security groups here -
> > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > >
> > > > > You also need to make sure that you can ssh from all the slaves to
> > > master
> > > > > without specifying password.
> > > > >
> > > > > Regards,
> > > > > Vaibhav Puranik
> > > > > GumGum
> > > > >
> > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Seems like I am not explaining my problem correctly.
> > > > > >
> > > > > > My Key Pair file is on my machine at work which is behind a corp
> > > > > firewall.
> > > > > > As such I can't 'scp' from the ec2 instance to my local machine
> at
> > > > work
> > > > > to
> > > > > > 'get' the file. So I need a way to 'send' a file from my machine
> > to
> > > > the
> > > > > > ec2
> > > > > > instance. I tried using 'scp' (from my machine at work) to the
> ec2
> > > > > machine
> > > > > > but it says "Permission denied". Does this make sense?
> > > > > >
> > > > > > May be I need to use the command line tools for EC2. I am
> looking
> > > into
> > > > > > those right now, but if there's a better/easier way, please let
> me
> > > > know.
> > > > > >
> > > > > > Thanks once again for your help.
> > > > > >
> > > > > >
> > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > The ssh is installed on ec2 by default, otherwise you have no
> way
> > > to
> > > > > > login
> > > > > > > to ec2
> > > > > > >
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)
> > "It
> > > > > does
> > > > > > so
> > > > > > > > by
> > > > > > > > connecting to the host using SSH and there executes an SCP
> > server
> > > > > > (scp)".
> > > > > > > >
> > > > > > > > So if SSH isn't working SCP won't work, either. In any case
> I
> > > > tried
> > > > > to
> > > > > > > scp
> > > > > > > > but getting "Permission denied (public key)".
> > > > > > > >
> > > > > > > > Any other ideas? Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> zjffdu@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > >
> > > > > > > > > Jeff Zhang
> > > > > > > > >
> > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes,
> Cloudera's
> > > > > > > distribution
> > > > > > > > > > works well, but trying to do this myself so I can learn
> > > what's
> > > > > > > > happening
> > > > > > > > > > behind the scene.)
> > > > > > > > > >
> > > > > > > > > > I have a Master & a Slave. When I start HDFS on the
> > master,
> > > I
> > > > > get
> > > > > > a
> > > > > > > > > > message
> > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> 10.xxx
> > is
> > > > IP
> > > > > > > > address
> > > > > > > > > of
> > > > > > > > > > the slave.
> > > > > > > > > >
> > > > > > > > > > Basic problem (I think) is that I can't ssh from the
> master
> > > EC2
> > > > > > > > instance
> > > > > > > > > to
> > > > > > > > > > the Slave EC2 instance. What's the best way to fix it?
> I
> > > > think
> > > > > I
> > > > > > > need
> > > > > > > > > the
> > > > > > > > > > "Key Pair" file on my master. I have a key pair on my
> > local
> > > > > > machine,
> > > > > > > > but
> > > > > > > > > > how do I transfer it to the EC2 machine? (I know, I
> know,
> > I
> > > > > > agree..
> > > > > > > I
> > > > > > > > am
> > > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > > >
> > > > > > > > > > Please help. Thanks.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Vaibhav Puranik <vp...@gmail.com>.
Yes, I pretty much meant that.
Add your master's public key to authorized_keys on slaves too.
Regards,
Vaibhav
On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Sorry.. but not sure what you mean by... "In order to achieve that you need
> to add the keypair you are using to your
> auhtorized_keys file in the home dir/.ssh folder."
>
>
> I tried this....
>
> cat KeyPair.pem >> /root/.ssh/authorized_keys
>
> Is that what you meant by "add the keypair to authorized_keys"?
>
> Still can't ssh from Slave to Master. It says..... Permission denied
> (publickey).
>
>
> Sorry, I am a newbie when it comes to such environment issues. Thanks for
> your patience. It's greatly appreciated.
>
>
> On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com
> >wrote:
>
> > You need to be able to SSH without specifying the keypair from a slave to
> > master (I think reverse it true too, but I am not sure).
> >
> > In order to achieve that you need to add the keypair you are using to
> your
> > auhtorized_keys file in the home dir/.ssh folder.
> >
> > Once you setup passphraseless ssh, it should start working. Furthrmore,
> > after you set this up, try to ssh manully (without key) and if it asks
> you
> > the following message - say yes:
> >
> > The authenticity of host
> > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> > be established.
> > RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > Are you sure you want to continue connecting (yes/no)? yes
> > Warning: Permanently added
> > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > (RSA) to the list of known hosts.
> >
> > In other words, you have to make sure that the machine you are trying to
> > ssh
> > is in your known_hosts file.
> >
> > Regards,
> > Vaibhav
> >
> > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Thanks, Vaibhav. That was helpful. I am able to scp from my local
> > machine
> > > after I opened port 22 under my security group.
> > >
> > > But my cluster is still not starting as expected. I mean... on my
> Master
> > > when I run:
> > >
> > > ./start-all.sh
> > >
> > > I get messages saying...
> > >
> > > 10.252.xxx.xxx: Permission denied (publickey).
> > >
> > > where 10.252.xxx.xxx is the ip address of my Slave.
> > >
> > >
> > > I can ssh from my Slave to master if I do this:
> > >
> > > ssh -i <keypairname>.pem <Master's ip address>
> > >
> > >
> > > Any suggestions? Thanks for your help.
> > >
> > >
> > >
> > >
> > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vp...@gmail.com>
> > > wrote:
> > >
> > > > In order to use scp from your machine to an ec2 instance you need to
> > use
> > > > scp
> > > > -i and give it the path of your keypair.
> > > >
> > > > scp -i /path/my/ec2/keypair <souce> <destination>
> > > >
> > > > Secondly in order to configure an ec2 cluster for Hadoop, you need to
> > > make
> > > > a
> > > > security group (let's call it hadoop) and give hadoop access to
> hadoop.
> > > >
> > > > You can read more about security groups here -
> > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > >
> > > > You also need to make sure that you can ssh from all the slaves to
> > master
> > > > without specifying password.
> > > >
> > > > Regards,
> > > > Vaibhav Puranik
> > > > GumGum
> > > >
> > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Seems like I am not explaining my problem correctly.
> > > > >
> > > > > My Key Pair file is on my machine at work which is behind a corp
> > > > firewall.
> > > > > As such I can't 'scp' from the ec2 instance to my local machine at
> > > work
> > > > to
> > > > > 'get' the file. So I need a way to 'send' a file from my machine
> to
> > > the
> > > > > ec2
> > > > > instance. I tried using 'scp' (from my machine at work) to the ec2
> > > > machine
> > > > > but it says "Permission denied". Does this make sense?
> > > > >
> > > > > May be I need to use the command line tools for EC2. I am looking
> > into
> > > > > those right now, but if there's a better/easier way, please let me
> > > know.
> > > > >
> > > > > Thanks once again for your help.
> > > > >
> > > > >
> > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com>
> > wrote:
> > > > >
> > > > > > The ssh is installed on ec2 by default, otherwise you have no way
> > to
> > > > > login
> > > > > > to ec2
> > > > > >
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)
> "It
> > > > does
> > > > > so
> > > > > > > by
> > > > > > > connecting to the host using SSH and there executes an SCP
> server
> > > > > (scp)".
> > > > > > >
> > > > > > > So if SSH isn't working SCP won't work, either. In any case I
> > > tried
> > > > to
> > > > > > scp
> > > > > > > but getting "Permission denied (public key)".
> > > > > > >
> > > > > > > Any other ideas? Thanks.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > >
> > > > > > > > Jeff Zhang
> > > > > > > >
> > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's
> > > > > > distribution
> > > > > > > > > works well, but trying to do this myself so I can learn
> > what's
> > > > > > > happening
> > > > > > > > > behind the scene.)
> > > > > > > > >
> > > > > > > > > I have a Master & a Slave. When I start HDFS on the
> master,
> > I
> > > > get
> > > > > a
> > > > > > > > > message
> > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx
> is
> > > IP
> > > > > > > address
> > > > > > > > of
> > > > > > > > > the slave.
> > > > > > > > >
> > > > > > > > > Basic problem (I think) is that I can't ssh from the master
> > EC2
> > > > > > > instance
> > > > > > > > to
> > > > > > > > > the Slave EC2 instance. What's the best way to fix it? I
> > > think
> > > > I
> > > > > > need
> > > > > > > > the
> > > > > > > > > "Key Pair" file on my master. I have a key pair on my
> local
> > > > > machine,
> > > > > > > but
> > > > > > > > > how do I transfer it to the EC2 machine? (I know, I know,
> I
> > > > > agree..
> > > > > > I
> > > > > > > am
> > > > > > > > > dumb :) Should I FTP it?
> > > > > > > > >
> > > > > > > > > Please help. Thanks.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
Sorry.. but not sure what you mean by... "In order to achieve that you need
to add the keypair you are using to your
auhtorized_keys file in the home dir/.ssh folder."
I tried this....
cat KeyPair.pem >> /root/.ssh/authorized_keys
Is that what you meant by "add the keypair to authorized_keys"?
Still can't ssh from Slave to Master. It says..... Permission denied
(publickey).
Sorry, I am a newbie when it comes to such environment issues. Thanks for
your patience. It's greatly appreciated.
On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vp...@gmail.com>wrote:
> You need to be able to SSH without specifying the keypair from a slave to
> master (I think reverse it true too, but I am not sure).
>
> In order to achieve that you need to add the keypair you are using to your
> auhtorized_keys file in the home dir/.ssh folder.
>
> Once you setup passphraseless ssh, it should start working. Furthrmore,
> after you set this up, try to ssh manully (without key) and if it asks you
> the following message - say yes:
>
> The authenticity of host
> 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> be established.
> RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> Are you sure you want to continue connecting (yes/no)? yes
> Warning: Permanently added
> 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> (RSA) to the list of known hosts.
>
> In other words, you have to make sure that the machine you are trying to
> ssh
> is in your known_hosts file.
>
> Regards,
> Vaibhav
>
> On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Thanks, Vaibhav. That was helpful. I am able to scp from my local
> machine
> > after I opened port 22 under my security group.
> >
> > But my cluster is still not starting as expected. I mean... on my Master
> > when I run:
> >
> > ./start-all.sh
> >
> > I get messages saying...
> >
> > 10.252.xxx.xxx: Permission denied (publickey).
> >
> > where 10.252.xxx.xxx is the ip address of my Slave.
> >
> >
> > I can ssh from my Slave to master if I do this:
> >
> > ssh -i <keypairname>.pem <Master's ip address>
> >
> >
> > Any suggestions? Thanks for your help.
> >
> >
> >
> >
> > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vp...@gmail.com>
> > wrote:
> >
> > > In order to use scp from your machine to an ec2 instance you need to
> use
> > > scp
> > > -i and give it the path of your keypair.
> > >
> > > scp -i /path/my/ec2/keypair <souce> <destination>
> > >
> > > Secondly in order to configure an ec2 cluster for Hadoop, you need to
> > make
> > > a
> > > security group (let's call it hadoop) and give hadoop access to hadoop.
> > >
> > > You can read more about security groups here -
> > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > >
> > > You also need to make sure that you can ssh from all the slaves to
> master
> > > without specifying password.
> > >
> > > Regards,
> > > Vaibhav Puranik
> > > GumGum
> > >
> > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Seems like I am not explaining my problem correctly.
> > > >
> > > > My Key Pair file is on my machine at work which is behind a corp
> > > firewall.
> > > > As such I can't 'scp' from the ec2 instance to my local machine at
> > work
> > > to
> > > > 'get' the file. So I need a way to 'send' a file from my machine to
> > the
> > > > ec2
> > > > instance. I tried using 'scp' (from my machine at work) to the ec2
> > > machine
> > > > but it says "Permission denied". Does this make sense?
> > > >
> > > > May be I need to use the command line tools for EC2. I am looking
> into
> > > > those right now, but if there's a better/easier way, please let me
> > know.
> > > >
> > > > Thanks once again for your help.
> > > >
> > > >
> > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com>
> wrote:
> > > >
> > > > > The ssh is installed on ec2 by default, otherwise you have no way
> to
> > > > login
> > > > > to ec2
> > > > >
> > > > >
> > > > > Jeff Zhang
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It
> > > does
> > > > so
> > > > > > by
> > > > > > connecting to the host using SSH and there executes an SCP server
> > > > (scp)".
> > > > > >
> > > > > > So if SSH isn't working SCP won't work, either. In any case I
> > tried
> > > to
> > > > > scp
> > > > > > but getting "Permission denied (public key)".
> > > > > >
> > > > > > Any other ideas? Thanks.
> > > > > >
> > > > > >
> > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > You should scp the key-pair to EC2 machine
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's
> > > > > distribution
> > > > > > > > works well, but trying to do this myself so I can learn
> what's
> > > > > > happening
> > > > > > > > behind the scene.)
> > > > > > > >
> > > > > > > > I have a Master & a Slave. When I start HDFS on the master,
> I
> > > get
> > > > a
> > > > > > > > message
> > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is
> > IP
> > > > > > address
> > > > > > > of
> > > > > > > > the slave.
> > > > > > > >
> > > > > > > > Basic problem (I think) is that I can't ssh from the master
> EC2
> > > > > > instance
> > > > > > > to
> > > > > > > > the Slave EC2 instance. What's the best way to fix it? I
> > think
> > > I
> > > > > need
> > > > > > > the
> > > > > > > > "Key Pair" file on my master. I have a key pair on my local
> > > > machine,
> > > > > > but
> > > > > > > > how do I transfer it to the EC2 machine? (I know, I know, I
> > > > agree..
> > > > > I
> > > > > > am
> > > > > > > > dumb :) Should I FTP it?
> > > > > > > >
> > > > > > > > Please help. Thanks.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Vaibhav Puranik <vp...@gmail.com>.
You need to be able to SSH without specifying the keypair from a slave to
master (I think reverse it true too, but I am not sure).
In order to achieve that you need to add the keypair you are using to your
auhtorized_keys file in the home dir/.ssh folder.
Once you setup passphraseless ssh, it should start working. Furthrmore,
after you set this up, try to ssh manully (without key) and if it asks you
the following message - say yes:
The authenticity of host
'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
be established.
RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added
'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
(RSA) to the list of known hosts.
In other words, you have to make sure that the machine you are trying to ssh
is in your known_hosts file.
Regards,
Vaibhav
On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Thanks, Vaibhav. That was helpful. I am able to scp from my local machine
> after I opened port 22 under my security group.
>
> But my cluster is still not starting as expected. I mean... on my Master
> when I run:
>
> ./start-all.sh
>
> I get messages saying...
>
> 10.252.xxx.xxx: Permission denied (publickey).
>
> where 10.252.xxx.xxx is the ip address of my Slave.
>
>
> I can ssh from my Slave to master if I do this:
>
> ssh -i <keypairname>.pem <Master's ip address>
>
>
> Any suggestions? Thanks for your help.
>
>
>
>
> On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vp...@gmail.com>
> wrote:
>
> > In order to use scp from your machine to an ec2 instance you need to use
> > scp
> > -i and give it the path of your keypair.
> >
> > scp -i /path/my/ec2/keypair <souce> <destination>
> >
> > Secondly in order to configure an ec2 cluster for Hadoop, you need to
> make
> > a
> > security group (let's call it hadoop) and give hadoop access to hadoop.
> >
> > You can read more about security groups here -
> >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> >
> > You also need to make sure that you can ssh from all the slaves to master
> > without specifying password.
> >
> > Regards,
> > Vaibhav Puranik
> > GumGum
> >
> > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Seems like I am not explaining my problem correctly.
> > >
> > > My Key Pair file is on my machine at work which is behind a corp
> > firewall.
> > > As such I can't 'scp' from the ec2 instance to my local machine at
> work
> > to
> > > 'get' the file. So I need a way to 'send' a file from my machine to
> the
> > > ec2
> > > instance. I tried using 'scp' (from my machine at work) to the ec2
> > machine
> > > but it says "Permission denied". Does this make sense?
> > >
> > > May be I need to use the command line tools for EC2. I am looking into
> > > those right now, but if there's a better/easier way, please let me
> know.
> > >
> > > Thanks once again for your help.
> > >
> > >
> > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com> wrote:
> > >
> > > > The ssh is installed on ec2 by default, otherwise you have no way to
> > > login
> > > > to ec2
> > > >
> > > >
> > > > Jeff Zhang
> > > >
> > > >
> > > >
> > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It
> > does
> > > so
> > > > > by
> > > > > connecting to the host using SSH and there executes an SCP server
> > > (scp)".
> > > > >
> > > > > So if SSH isn't working SCP won't work, either. In any case I
> tried
> > to
> > > > scp
> > > > > but getting "Permission denied (public key)".
> > > > >
> > > > > Any other ideas? Thanks.
> > > > >
> > > > >
> > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com>
> > wrote:
> > > > >
> > > > > > You should scp the key-pair to EC2 machine
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's
> > > > distribution
> > > > > > > works well, but trying to do this myself so I can learn what's
> > > > > happening
> > > > > > > behind the scene.)
> > > > > > >
> > > > > > > I have a Master & a Slave. When I start HDFS on the master, I
> > get
> > > a
> > > > > > > message
> > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is
> IP
> > > > > address
> > > > > > of
> > > > > > > the slave.
> > > > > > >
> > > > > > > Basic problem (I think) is that I can't ssh from the master EC2
> > > > > instance
> > > > > > to
> > > > > > > the Slave EC2 instance. What's the best way to fix it? I
> think
> > I
> > > > need
> > > > > > the
> > > > > > > "Key Pair" file on my master. I have a key pair on my local
> > > machine,
> > > > > but
> > > > > > > how do I transfer it to the EC2 machine? (I know, I know, I
> > > agree..
> > > > I
> > > > > am
> > > > > > > dumb :) Should I FTP it?
> > > > > > >
> > > > > > > Please help. Thanks.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
Thanks, Vaibhav. That was helpful. I am able to scp from my local machine
after I opened port 22 under my security group.
But my cluster is still not starting as expected. I mean... on my Master
when I run:
./start-all.sh
I get messages saying...
10.252.xxx.xxx: Permission denied (publickey).
where 10.252.xxx.xxx is the ip address of my Slave.
I can ssh from my Slave to master if I do this:
ssh -i <keypairname>.pem <Master's ip address>
Any suggestions? Thanks for your help.
On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vp...@gmail.com> wrote:
> In order to use scp from your machine to an ec2 instance you need to use
> scp
> -i and give it the path of your keypair.
>
> scp -i /path/my/ec2/keypair <souce> <destination>
>
> Secondly in order to configure an ec2 cluster for Hadoop, you need to make
> a
> security group (let's call it hadoop) and give hadoop access to hadoop.
>
> You can read more about security groups here -
>
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
>
> You also need to make sure that you can ssh from all the slaves to master
> without specifying password.
>
> Regards,
> Vaibhav Puranik
> GumGum
>
> On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Seems like I am not explaining my problem correctly.
> >
> > My Key Pair file is on my machine at work which is behind a corp
> firewall.
> > As such I can't 'scp' from the ec2 instance to my local machine at work
> to
> > 'get' the file. So I need a way to 'send' a file from my machine to the
> > ec2
> > instance. I tried using 'scp' (from my machine at work) to the ec2
> machine
> > but it says "Permission denied". Does this make sense?
> >
> > May be I need to use the command line tools for EC2. I am looking into
> > those right now, but if there's a better/easier way, please let me know.
> >
> > Thanks once again for your help.
> >
> >
> > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com> wrote:
> >
> > > The ssh is installed on ec2 by default, otherwise you have no way to
> > login
> > > to ec2
> > >
> > >
> > > Jeff Zhang
> > >
> > >
> > >
> > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It
> does
> > so
> > > > by
> > > > connecting to the host using SSH and there executes an SCP server
> > (scp)".
> > > >
> > > > So if SSH isn't working SCP won't work, either. In any case I tried
> to
> > > scp
> > > > but getting "Permission denied (public key)".
> > > >
> > > > Any other ideas? Thanks.
> > > >
> > > >
> > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com>
> wrote:
> > > >
> > > > > You should scp the key-pair to EC2 machine
> > > > >
> > > > > Jeff Zhang
> > > > >
> > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's
> > > distribution
> > > > > > works well, but trying to do this myself so I can learn what's
> > > > happening
> > > > > > behind the scene.)
> > > > > >
> > > > > > I have a Master & a Slave. When I start HDFS on the master, I
> get
> > a
> > > > > > message
> > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> > > > address
> > > > > of
> > > > > > the slave.
> > > > > >
> > > > > > Basic problem (I think) is that I can't ssh from the master EC2
> > > > instance
> > > > > to
> > > > > > the Slave EC2 instance. What's the best way to fix it? I think
> I
> > > need
> > > > > the
> > > > > > "Key Pair" file on my master. I have a key pair on my local
> > machine,
> > > > but
> > > > > > how do I transfer it to the EC2 machine? (I know, I know, I
> > agree..
> > > I
> > > > am
> > > > > > dumb :) Should I FTP it?
> > > > > >
> > > > > > Please help. Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Vaibhav Puranik <vp...@gmail.com>.
In order to use scp from your machine to an ec2 instance you need to use scp
-i and give it the path of your keypair.
scp -i /path/my/ec2/keypair <souce> <destination>
Secondly in order to configure an ec2 cluster for Hadoop, you need to make a
security group (let's call it hadoop) and give hadoop access to hadoop.
You can read more about security groups here -
http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
You also need to make sure that you can ssh from all the slaves to master
without specifying password.
Regards,
Vaibhav Puranik
GumGum
On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Seems like I am not explaining my problem correctly.
>
> My Key Pair file is on my machine at work which is behind a corp firewall.
> As such I can't 'scp' from the ec2 instance to my local machine at work to
> 'get' the file. So I need a way to 'send' a file from my machine to the
> ec2
> instance. I tried using 'scp' (from my machine at work) to the ec2 machine
> but it says "Permission denied". Does this make sense?
>
> May be I need to use the command line tools for EC2. I am looking into
> those right now, but if there's a better/easier way, please let me know.
>
> Thanks once again for your help.
>
>
> On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com> wrote:
>
> > The ssh is installed on ec2 by default, otherwise you have no way to
> login
> > to ec2
> >
> >
> > Jeff Zhang
> >
> >
> >
> > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It does
> so
> > > by
> > > connecting to the host using SSH and there executes an SCP server
> (scp)".
> > >
> > > So if SSH isn't working SCP won't work, either. In any case I tried to
> > scp
> > > but getting "Permission denied (public key)".
> > >
> > > Any other ideas? Thanks.
> > >
> > >
> > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com> wrote:
> > >
> > > > You should scp the key-pair to EC2 machine
> > > >
> > > > Jeff Zhang
> > > >
> > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's
> > distribution
> > > > > works well, but trying to do this myself so I can learn what's
> > > happening
> > > > > behind the scene.)
> > > > >
> > > > > I have a Master & a Slave. When I start HDFS on the master, I get
> a
> > > > > message
> > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> > > address
> > > > of
> > > > > the slave.
> > > > >
> > > > > Basic problem (I think) is that I can't ssh from the master EC2
> > > instance
> > > > to
> > > > > the Slave EC2 instance. What's the best way to fix it? I think I
> > need
> > > > the
> > > > > "Key Pair" file on my master. I have a key pair on my local
> machine,
> > > but
> > > > > how do I transfer it to the EC2 machine? (I know, I know, I
> agree..
> > I
> > > am
> > > > > dumb :) Should I FTP it?
> > > > >
> > > > > Please help. Thanks.
> > > > >
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
Seems like I am not explaining my problem correctly.
My Key Pair file is on my machine at work which is behind a corp firewall.
As such I can't 'scp' from the ec2 instance to my local machine at work to
'get' the file. So I need a way to 'send' a file from my machine to the ec2
instance. I tried using 'scp' (from my machine at work) to the ec2 machine
but it says "Permission denied". Does this make sense?
May be I need to use the command line tools for EC2. I am looking into
those right now, but if there's a better/easier way, please let me know.
Thanks once again for your help.
On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zj...@gmail.com> wrote:
> The ssh is installed on ec2 by default, otherwise you have no way to login
> to ec2
>
>
> Jeff Zhang
>
>
>
> On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It does so
> > by
> > connecting to the host using SSH and there executes an SCP server (scp)".
> >
> > So if SSH isn't working SCP won't work, either. In any case I tried to
> scp
> > but getting "Permission denied (public key)".
> >
> > Any other ideas? Thanks.
> >
> >
> > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com> wrote:
> >
> > > You should scp the key-pair to EC2 machine
> > >
> > > Jeff Zhang
> > >
> > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's
> distribution
> > > > works well, but trying to do this myself so I can learn what's
> > happening
> > > > behind the scene.)
> > > >
> > > > I have a Master & a Slave. When I start HDFS on the master, I get a
> > > > message
> > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> > address
> > > of
> > > > the slave.
> > > >
> > > > Basic problem (I think) is that I can't ssh from the master EC2
> > instance
> > > to
> > > > the Slave EC2 instance. What's the best way to fix it? I think I
> need
> > > the
> > > > "Key Pair" file on my master. I have a key pair on my local machine,
> > but
> > > > how do I transfer it to the EC2 machine? (I know, I know, I agree..
> I
> > am
> > > > dumb :) Should I FTP it?
> > > >
> > > > Please help. Thanks.
> > > >
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Jeff Zhang <zj...@gmail.com>.
The ssh is installed on ec2 by default, otherwise you have no way to login
to ec2
Jeff Zhang
On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
mailinglists19@gmail.com> wrote:
> As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It does so
> by
> connecting to the host using SSH and there executes an SCP server (scp)".
>
> So if SSH isn't working SCP won't work, either. In any case I tried to scp
> but getting "Permission denied (public key)".
>
> Any other ideas? Thanks.
>
>
> On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
> > You should scp the key-pair to EC2 machine
> >
> > Jeff Zhang
> >
> > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's distribution
> > > works well, but trying to do this myself so I can learn what's
> happening
> > > behind the scene.)
> > >
> > > I have a Master & a Slave. When I start HDFS on the master, I get a
> > > message
> > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> address
> > of
> > > the slave.
> > >
> > > Basic problem (I think) is that I can't ssh from the master EC2
> instance
> > to
> > > the Slave EC2 instance. What's the best way to fix it? I think I need
> > the
> > > "Key Pair" file on my master. I have a key pair on my local machine,
> but
> > > how do I transfer it to the EC2 machine? (I know, I know, I agree.. I
> am
> > > dumb :) Should I FTP it?
> > >
> > > Please help. Thanks.
> > >
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Steve Loughran <st...@apache.org>.
Something Something wrote:
> As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It does so by
> connecting to the host using SSH and there executes an SCP server (scp)".
>
> So if SSH isn't working SCP won't work, either. In any case I tried to scp
> but getting "Permission denied (public key)".
>
when you create a VM on EC2 you give it the public key you want to
SCP/SSH in on.
Re: Starting Hadoop cluster on EC2
Posted by Something Something <ma...@gmail.com>.
As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy) "It does so by
connecting to the host using SSH and there executes an SCP server (scp)".
So if SSH isn't working SCP won't work, either. In any case I tried to scp
but getting "Permission denied (public key)".
Any other ideas? Thanks.
On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zj...@gmail.com> wrote:
> You should scp the key-pair to EC2 machine
>
> Jeff Zhang
>
> On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's distribution
> > works well, but trying to do this myself so I can learn what's happening
> > behind the scene.)
> >
> > I have a Master & a Slave. When I start HDFS on the master, I get a
> > message
> > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address
> of
> > the slave.
> >
> > Basic problem (I think) is that I can't ssh from the master EC2 instance
> to
> > the Slave EC2 instance. What's the best way to fix it? I think I need
> the
> > "Key Pair" file on my master. I have a key pair on my local machine, but
> > how do I transfer it to the EC2 machine? (I know, I know, I agree.. I am
> > dumb :) Should I FTP it?
> >
> > Please help. Thanks.
> >
>
Re: Starting Hadoop cluster on EC2
Posted by Jeff Zhang <zj...@gmail.com>.
You should scp the key-pair to EC2 machine
Jeff Zhang
On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's distribution
> works well, but trying to do this myself so I can learn what's happening
> behind the scene.)
>
> I have a Master & a Slave. When I start HDFS on the master, I get a
> message
> saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
> the slave.
>
> Basic problem (I think) is that I can't ssh from the master EC2 instance to
> the Slave EC2 instance. What's the best way to fix it? I think I need the
> "Key Pair" file on my master. I have a key pair on my local machine, but
> how do I transfer it to the EC2 machine? (I know, I know, I agree.. I am
> dumb :) Should I FTP it?
>
> Please help. Thanks.
>
Re: Starting Hadoop cluster on EC2
Posted by Jeff Zhang <zj...@gmail.com>.
You can scp key-pair to ec2 machine
Jeff Zhang
On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's distribution
> works well, but trying to do this myself so I can learn what's happening
> behind the scene.)
>
> I have a Master & a Slave. When I start HDFS on the master, I get a
> message
> saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
> the slave.
>
> Basic problem (I think) is that I can't ssh from the master EC2 instance to
> the Slave EC2 instance. What's the best way to fix it? I think I need the
> "Key Pair" file on my master. I have a key pair on my local machine, but
> how do I transfer it to the EC2 machine? (I know, I know, I agree.. I am
> dumb :) Should I FTP it?
>
> Please help. Thanks.
>
Re: Starting Hadoop cluster on EC2
Posted by Jeff Zhang <zj...@gmail.com>.
You can scp key-pair to ec2 machine
Jeff Zhang
On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Trying to start a Hadoop cluster on EC2. (Yes, Cloudera's distribution
> works well, but trying to do this myself so I can learn what's happening
> behind the scene.)
>
> I have a Master & a Slave. When I start HDFS on the master, I get a
> message
> saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
> the slave.
>
> Basic problem (I think) is that I can't ssh from the master EC2 instance to
> the Slave EC2 instance. What's the best way to fix it? I think I need the
> "Key Pair" file on my master. I have a key pair on my local machine, but
> how do I transfer it to the EC2 machine? (I know, I know, I agree.. I am
> dumb :) Should I FTP it?
>
> Please help. Thanks.
>