You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Grandl Robert <rg...@yahoo.com> on 2010/11/03 17:32:02 UTC
Hadoop on PlanetLab nodes
Hi all,
I am trying to set up Hadoop on PlanetLab nodes.
However, as far as I seen I need password less SSH login between master and slave nodes in Hadoop.
My
main problem is for planet lab nodes I have a privatekey I always use
when connect to planet lab nodes. I am using ssh -i ~/.ssh/privatekey
user@planet_lab_node
Also, the username on planet lab nodes is different from the user name I have on my local cluster.
Is there any possibility to use hadoop on planet lab nodes ?
Any hints, suggestions or links will be valuable for me.
Thanks,
Robert
Re: Hadoop on PlanetLab nodes
Posted by Steve Loughran <st...@apache.org>.
On 03/11/10 16:32, Grandl Robert wrote:
> Hi all,
>
> I am trying to set up Hadoop on PlanetLab nodes.
>
> However, as far as I seen I need password less SSH login between master and slave nodes in Hadoop.
> My
> main problem is for planet lab nodes I have a privatekey I always use
> when connect to planet lab nodes. I am using ssh -i ~/.ssh/privatekey
> user@planet_lab_node
>
> Also, the username on planet lab nodes is different from the user name I have on my local cluster.
>
> Is there any possibility to use hadoop on planet lab nodes ?
>
I wouldn't rush to do it as their nodes are distributed with odd
networking -you get a wide cluster, not a fast one.
Better to try for cluster time on OpenCirrus, which does run Hadoop.
Re: Hadoop on PlanetLab nodes
Posted by Ian Holsman <ha...@holsman.net>.
to answer the 2nd part of your question.. how do I log on with a different
userID.
you can create a file called
.ssh/config
and in that you can specify the userID to use as a default, as well as other
parameters
for example:
I have the following set up for all hosts on example.com
33 Host *.example.com
34 Port 22
35 Protocol 2
36 Compression no
38 User xyz
man ssh_config has more details.
Regards
Ian
On Thu, Nov 4, 2010 at 4:03 AM, Eric Sammer <es...@cloudera.com> wrote:
> Better than passwordless ssh is to use ssh-agent and load your key
> (providing the passphrase once). See 'man ssh-agent' for details,
> although it's pretty straight forward.
>
> Short version:
>
> # See if ssh-agent is already running for your user by trying to list
> # keys it knows about. That's a "dash-ell."
> ssh-add -l
>
> # Load an ssh key.
> ssh-add ~/.ssh/id_rsa
> <enter passphrase>
>
> If your public key is already on all the nodes, watch as magically
> start-all/stop-all work without requiring a password.
>
> WARNING: When working with authentication, remote login, and anything
> security related, *never* blindly take advice. Always take the time to
> understand the ramifications. See man pages for ssh, ssh-agent,
> ssh-add, and ssh_config.
>
> Hope that helps.
>
> On Wed, Nov 3, 2010 at 12:53 PM, Harsh J <qw...@gmail.com> wrote:
> > Hi,
> >
> > On Wed, Nov 3, 2010 at 10:02 PM, Grandl Robert <rg...@yahoo.com>
> wrote:
> >> Hi all,
> >>
> >> I am trying to set up Hadoop on PlanetLab nodes.
> >>
> >> However, as far as I seen I need password less SSH login between master
> and slave nodes in Hadoop.
> >
> > This is "needed" for ease of setup, but you can as well start all
> > nodes manually yourselves without requiring SSH as a dependency. Just
> > for your information (won't be the right way out, perhaps).
> >
> > For instance, you can ssh manually and issue "hadoop tasktracker" and
> > "hadoop datanode" commands in the nodes to start those daemons.
> >
> >> My
> >> main problem is for planet lab nodes I have a privatekey I always use
> >> when connect to planet lab nodes. I am using ssh -i ~/.ssh/privatekey
> >> user@planet_lab_node
> >>
> >> Also, the username on planet lab nodes is different from the user name I
> have on my local cluster.
> >>
> >> Is there any possibility to use hadoop on planet lab nodes ?
> >>
> >> Any hints, suggestions or links will be valuable for me.
> >>
> >> Thanks,
> >> Robert
> >>
> >>
> >>
> >
> >
> >
> > --
> > Harsh J
> > www.harshj.com
> >
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>
Re: Hadoop on PlanetLab nodes
Posted by Eric Sammer <es...@cloudera.com>.
Better than passwordless ssh is to use ssh-agent and load your key
(providing the passphrase once). See 'man ssh-agent' for details,
although it's pretty straight forward.
Short version:
# See if ssh-agent is already running for your user by trying to list
# keys it knows about. That's a "dash-ell."
ssh-add -l
# Load an ssh key.
ssh-add ~/.ssh/id_rsa
<enter passphrase>
If your public key is already on all the nodes, watch as magically
start-all/stop-all work without requiring a password.
WARNING: When working with authentication, remote login, and anything
security related, *never* blindly take advice. Always take the time to
understand the ramifications. See man pages for ssh, ssh-agent,
ssh-add, and ssh_config.
Hope that helps.
On Wed, Nov 3, 2010 at 12:53 PM, Harsh J <qw...@gmail.com> wrote:
> Hi,
>
> On Wed, Nov 3, 2010 at 10:02 PM, Grandl Robert <rg...@yahoo.com> wrote:
>> Hi all,
>>
>> I am trying to set up Hadoop on PlanetLab nodes.
>>
>> However, as far as I seen I need password less SSH login between master and slave nodes in Hadoop.
>
> This is "needed" for ease of setup, but you can as well start all
> nodes manually yourselves without requiring SSH as a dependency. Just
> for your information (won't be the right way out, perhaps).
>
> For instance, you can ssh manually and issue "hadoop tasktracker" and
> "hadoop datanode" commands in the nodes to start those daemons.
>
>> My
>> main problem is for planet lab nodes I have a privatekey I always use
>> when connect to planet lab nodes. I am using ssh -i ~/.ssh/privatekey
>> user@planet_lab_node
>>
>> Also, the username on planet lab nodes is different from the user name I have on my local cluster.
>>
>> Is there any possibility to use hadoop on planet lab nodes ?
>>
>> Any hints, suggestions or links will be valuable for me.
>>
>> Thanks,
>> Robert
>>
>>
>>
>
>
>
> --
> Harsh J
> www.harshj.com
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Hadoop on PlanetLab nodes
Posted by Harsh J <qw...@gmail.com>.
Hi,
On Wed, Nov 3, 2010 at 10:02 PM, Grandl Robert <rg...@yahoo.com> wrote:
> Hi all,
>
> I am trying to set up Hadoop on PlanetLab nodes.
>
> However, as far as I seen I need password less SSH login between master and slave nodes in Hadoop.
This is "needed" for ease of setup, but you can as well start all
nodes manually yourselves without requiring SSH as a dependency. Just
for your information (won't be the right way out, perhaps).
For instance, you can ssh manually and issue "hadoop tasktracker" and
"hadoop datanode" commands in the nodes to start those daemons.
> My
> main problem is for planet lab nodes I have a privatekey I always use
> when connect to planet lab nodes. I am using ssh -i ~/.ssh/privatekey
> user@planet_lab_node
>
> Also, the username on planet lab nodes is different from the user name I have on my local cluster.
>
> Is there any possibility to use hadoop on planet lab nodes ?
>
> Any hints, suggestions or links will be valuable for me.
>
> Thanks,
> Robert
>
>
>
--
Harsh J
www.harshj.com