You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Yabo-Arber Xu <ar...@gmail.com> on 2009/04/22 05:56:13 UTC
How to access data node without a passphrase?
Hi there,
I setup a small cluster for testing. When I start my cluster on my master
node, I have to type the password for starting each datanode and
tasktracker. That's pretty annoying and may be hard to handle when the
cluster grows. Any graceful way to handle this?
Best,
Arber
Re: How to access data node without a passphrase?
Posted by Yabo-Arber Xu <ar...@gmail.com>.
Dear Alex,
Thanks for your suggestion. I would be very interested in try RPMs with
DEBs, and will shoot an email to Todd soon.
Best,
Arber
On Thu, Apr 23, 2009 at 2:01 AM, Alex Loddengaard <al...@cloudera.com> wrote:
> RPMs won't work on Ubuntu, but we're almost finished with DEBs, which will
> work on Ubuntu. Shoot Todd an email if you want to try out our DEBs:
>
> <to...@cloudera.com>
>
> Are you asking about choosing a Linux distribution? The problem with
> Ubuntu
> is that it changes very frequently and generally uses relatively new
> software, making it a great desktop distribution, but perhaps not as good
> of
> a stable server distribution. (Note that the LTS Ubuntu releases are
> supported longer but still user newer, possibly unstable software.) I
> think
> that the majority of Linux server people use Redhat derivatives, in
> particular RHEL and CentOS, as they're not updated frequently and use
> stable
> software (RHEL costs money; CentOS is free). That said, CentOS is annoying
> to administer if you're hoping to use a version of Python newer than 2.4.
> I'm sure that the Debian people on this list will yell at me for saying
> Redhat derivatives are the majority, but we'll see I guess.
>
> So anyway, give Todd a shout if you want to try DEBs out. Otherwise, if
> you're interested in going down the Redhat derivative route (Fedora, RHEL,
> CentOS), you can use the RPMs.
>
> Alex
>
> On Tue, Apr 21, 2009 at 10:04 PM, Yabo-Arber Xu <arber.research@gmail.com
> >wrote:
>
> > Thanks for all your help, especially Asteem's detailed instruction. It
> > works
> > now!
> >
> > Alex: I did not use RPMs, but several of my existing nodes are installed
> > with Ubuntu. Is there any diff on running Hadoop on Ubuntu? I am thinking
> > of
> > choosing one before I started scaling up the cluster, but not sure which
> > one
> > benefit from long-term, i.e. get more support etc.
> >
> > Best
> > Arber
> >
> >
> > On Wed, Apr 22, 2009 at 12:35 PM, Puri, Aseem <Aseem.Puri@honeywell.com
> > >wrote:
> >
> > > cat ~/.ssh/master-key.pub >> ~/.ssh/authorized_keys
> > >
> >
>
Re: How to access data node without a passphrase?
Posted by Alex Loddengaard <al...@cloudera.com>.
RPMs won't work on Ubuntu, but we're almost finished with DEBs, which will
work on Ubuntu. Shoot Todd an email if you want to try out our DEBs:
<to...@cloudera.com>
Are you asking about choosing a Linux distribution? The problem with Ubuntu
is that it changes very frequently and generally uses relatively new
software, making it a great desktop distribution, but perhaps not as good of
a stable server distribution. (Note that the LTS Ubuntu releases are
supported longer but still user newer, possibly unstable software.) I think
that the majority of Linux server people use Redhat derivatives, in
particular RHEL and CentOS, as they're not updated frequently and use stable
software (RHEL costs money; CentOS is free). That said, CentOS is annoying
to administer if you're hoping to use a version of Python newer than 2.4.
I'm sure that the Debian people on this list will yell at me for saying
Redhat derivatives are the majority, but we'll see I guess.
So anyway, give Todd a shout if you want to try DEBs out. Otherwise, if
you're interested in going down the Redhat derivative route (Fedora, RHEL,
CentOS), you can use the RPMs.
Alex
On Tue, Apr 21, 2009 at 10:04 PM, Yabo-Arber Xu <ar...@gmail.com>wrote:
> Thanks for all your help, especially Asteem's detailed instruction. It
> works
> now!
>
> Alex: I did not use RPMs, but several of my existing nodes are installed
> with Ubuntu. Is there any diff on running Hadoop on Ubuntu? I am thinking
> of
> choosing one before I started scaling up the cluster, but not sure which
> one
> benefit from long-term, i.e. get more support etc.
>
> Best
> Arber
>
>
> On Wed, Apr 22, 2009 at 12:35 PM, Puri, Aseem <Aseem.Puri@honeywell.com
> >wrote:
>
> > cat ~/.ssh/master-key.pub >> ~/.ssh/authorized_keys
> >
>
Re: How to access data node without a passphrase?
Posted by Yabo-Arber Xu <ar...@gmail.com>.
Thanks for all your help, especially Asteem's detailed instruction. It works
now!
Alex: I did not use RPMs, but several of my existing nodes are installed
with Ubuntu. Is there any diff on running Hadoop on Ubuntu? I am thinking of
choosing one before I started scaling up the cluster, but not sure which one
benefit from long-term, i.e. get more support etc.
Best
Arber
On Wed, Apr 22, 2009 at 12:35 PM, Puri, Aseem <As...@honeywell.com>wrote:
> cat ~/.ssh/master-key.pub >> ~/.ssh/authorized_keys
>
RE: How to access data node without a passphrase?
Posted by "Puri, Aseem" <As...@Honeywell.com>.
Arber,
A. You have to first setup authorization keys
1. Execute the following command to generate keys: "ssh-keygen"
2. When prompted for filenames and pass phrases press ENTER to accept
default values.
3. After the command has finished generating keys, enter the following
command to change into your .ssh directory: "cd ~/.ssh"
4. To register the new authorization keys enter the following command:
"cat id_rsa.pub >> authorized_keys"
B. Generate public/private key pairs on all your machines
1. Issue the following commands ($> is the command prompt):
a) ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
b) cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
2. When prompted for your password, enter it.
C. Exchange public keys
On the master issue the following command
$ scp ~/.ssh/id_dsa.pub <slaveusername>@slave:~/.ssh/master-key.pub
Enter your password when prompted. This will copy your public key file
in use on the master to the slave.
On the slave, issue the following command:
$ cat ~/.ssh/master-key.pub >> ~/.ssh/authorized_keys
This will append your public key to the set of authorized keys the slave
accepts for authentication purposes.
Hope this helps
Aseem Puri
-----Original Message-----
From: Alex Loddengaard [mailto:alex@cloudera.com]
Sent: Wednesday, April 22, 2009 9:55 AM
To: core-user@hadoop.apache.org
Subject: Re: How to access data node without a passphrase?
I would recommend installing the Hadoop RPMs and avoid the start-all
scripts
all together. The RPMs ship with init scripts, allowing you to start
and
stop daemons with /sbin/service (or with a configuration management
tool,
which I assume you'll be using as your cluster grows). Here's more info
on
the RPMs:
<http://www.cloudera.com/hadoop>
The start-all scripts are easy ways for small clusters to get started /
stopped, but they're more annoying as your cluster grows (you have to
distribute authorized_keys files, iteratively start each daemon, etc).
If
you want to stick with the tarball, you can use bin/hadoop-daemon.sh on
each
node as well, though the only thing this buys you is being able to avoid
shipping your public key around for the "hadoop" user:
bin/hadoop-daemon.sh start datanode
> bin/hadoop-daemon.sh start tasktracker
> bin/hadoop-daemon.sh start etc
>
Hope this helps.
Alex
On Tue, Apr 21, 2009 at 8:56 PM, Yabo-Arber Xu
<ar...@gmail.com>wrote:
> Hi there,
>
> I setup a small cluster for testing. When I start my cluster on my
master
> node, I have to type the password for starting each datanode and
> tasktracker. That's pretty annoying and may be hard to handle when the
> cluster grows. Any graceful way to handle this?
>
> Best,
> Arber
>
Re: How to access data node without a passphrase?
Posted by Alex Loddengaard <al...@cloudera.com>.
I would recommend installing the Hadoop RPMs and avoid the start-all scripts
all together. The RPMs ship with init scripts, allowing you to start and
stop daemons with /sbin/service (or with a configuration management tool,
which I assume you'll be using as your cluster grows). Here's more info on
the RPMs:
<http://www.cloudera.com/hadoop>
The start-all scripts are easy ways for small clusters to get started /
stopped, but they're more annoying as your cluster grows (you have to
distribute authorized_keys files, iteratively start each daemon, etc). If
you want to stick with the tarball, you can use bin/hadoop-daemon.sh on each
node as well, though the only thing this buys you is being able to avoid
shipping your public key around for the "hadoop" user:
bin/hadoop-daemon.sh start datanode
> bin/hadoop-daemon.sh start tasktracker
> bin/hadoop-daemon.sh start etc
>
Hope this helps.
Alex
On Tue, Apr 21, 2009 at 8:56 PM, Yabo-Arber Xu <ar...@gmail.com>wrote:
> Hi there,
>
> I setup a small cluster for testing. When I start my cluster on my master
> node, I have to type the password for starting each datanode and
> tasktracker. That's pretty annoying and may be hard to handle when the
> cluster grows. Any graceful way to handle this?
>
> Best,
> Arber
>
Re: How to access data node without a passphrase?
Posted by Amit Saha <am...@gmail.com>.
On Wed, Apr 22, 2009 at 9:26 AM, Yabo-Arber Xu <ar...@gmail.com> wrote:
> Hi there,
>
> I setup a small cluster for testing. When I start my cluster on my master
> node, I have to type the password for starting each datanode and
> tasktracker. That's pretty annoying and may be hard to handle when the
> cluster grows. Any graceful way to handle this?
If you are using the 'default', SSH authentication mechanism, setting
up password-less SSH should help you. This page (among others) can
help you: http://www.cs.toronto.edu/~murray/compnotes/passwordless_ssh.html
Hope this helps,
Amit
--
http://amitksaha.blogspot.com
http://amitsaha.in.googlepages.com/
*Bangalore Open Java Users Group*:http:www.bojug.in
"Recursion is the basic iteration mechanism in Scheme"
--- http://c2.com/cgi/wiki?TailRecursion