Multiple NICs on hbase nodes - regionservers register with master using wrong address

[Marc Harris <mh...@jumptap.com>]

I am trying to set up an hbase cluster, and the nodes are unable to
communicate with each other. I have the following slightly unusual
setup:

- Each machine has two NICs:

NIC1:
- accessible outside the network
- lower bandwidth
- restricted ports
- listed in DNS

NIC2:
- accessible only inside the network
- higher bandwidth
- all ports open
- listed in /etc/hosts, not DNS (for some reason; I have no control over
this).

What I am finding is that even if I use the servers internal names in
the property dfs.datanode.dns.interface, the region servers announce
themselves to the master using the external NIC address. Even if I use
the numeric IP address of the internal NIC, it still uses the internal
IP address.

Tracing this through with a debugger, it looks like
HRegionServer.getThisIP() which calls org.apache.hadoop.net.DNS does not
handle the situation correctly. Perhaps if the external NIC was in DNS
it would work; I don't know. 

So a 2 part question:
1) Why does the code do any of this? Why not just assume that if the
user has supplied a value for dfs.datanode.dns.interface, then it is
correct, and the value to use?
2) Is there any way I can work around this problem?

Somehow datanode manages to avoid this problem. It registers to the
namenode using the correct (internal) NIC.

Thanks for any help you can provide.

RE: Hbase Distributed Operration

["Goel, Ankur" <An...@corp.aol.com>]

Thanks for the suggestion folks!
I'll try this out in couple of days to see it works or not.

-----Original Message-----
From: Jean-Daniel Cryans [mailto:jdcryans@gmail.com] 
Sent: Thursday, March 06, 2008 1:30 AM
To: hbase-user@hadoop.apache.org
Subject: Re: Hbase Distributed Operration

Oh yeah good idea.

J-D

2008/3/5, stack <st...@duboce.net>:
>
> Jean-Daniel Cryans wrote:
> > Hi Ankur,
> >
> > This is the default SSH behavior.
> >
> > 2 suggestions :
> >
> > 1) Dirty solution, you can modify the scripts in the bin folder and 
> > add
> your
> > "remoteuser" on the ssh connection line.
> >
>
>
> Would you get the same effect by uncommenting HBASE_SSH_OPTS in 
> $HBASE_HOME/conf/hbase-env.sh and adding '-l <remote-user>' (or 
> whatever the appropriate ssh opt is )?
> St.Ack
>
>

Re: Hbase Distributed Operration

[Jean-Daniel Cryans <jd...@gmail.com>]

Oh yeah good idea.

J-D

2008/3/5, stack <st...@duboce.net>:
>
> Jean-Daniel Cryans wrote:
> > Hi Ankur,
> >
> > This is the default SSH behavior.
> >
> > 2 suggestions :
> >
> > 1) Dirty solution, you can modify the scripts in the bin folder and add
> your
> > "remoteuser" on the ssh connection line.
> >
>
>
> Would you get the same effect by uncommenting HBASE_SSH_OPTS in
> $HBASE_HOME/conf/hbase-env.sh and adding '-l <remote-user>' (or whatever
> the appropriate ssh opt is )?
> St.Ack
>
>

Re: Hbase Distributed Operration

[stack <st...@duboce.net>]

Jean-Daniel Cryans wrote:
> Hi Ankur,
>
> This is the default SSH behavior.
>
> 2 suggestions :
>
> 1) Dirty solution, you can modify the scripts in the bin folder and add your
> "remoteuser" on the ssh connection line.
>   

Would you get the same effect by uncommenting HBASE_SSH_OPTS in 
$HBASE_HOME/conf/hbase-env.sh and adding '-l <remote-user>' (or whatever 
the appropriate ssh opt is )?
St.Ack

Re: Hbase Distributed Operration

[Jean-Daniel Cryans <jd...@gmail.com>]

Hi Ankur,

This is the default SSH behavior.

2 suggestions :

1) Dirty solution, you can modify the scripts in the bin folder and add your
"remoteuser" on the ssh connection line.
2) Clean solution, use the same account on each machine.

Btw, why do you want to use different accounts?

J-D

2008/3/5, Goel, Ankur <An...@corp.aol.com>:
>
>
> Hi,
>     I installed Hbase on my local machine and configured it to connect
> it to remote HDFS instance.
>
> The problem is that the user-account names are different on local
> machine (HBase)
> and remote machine (HDFS). So even though the public key/private key is
> correctly
> set-up between local/remote system, the ssh scripts try to login using
> the local
> account name that does not exist on remote machine.
>
> So instead of ssh login defaulting to "localuser@remotehost" on Hbase
> start-up,
> it should default to "remoteuser@remotehost".
>
> Can I configure scripts to use "remoteuser" for ssh login ?
>
> The help is much needed and appreciated.
>
> Thanks
>
> -Ankur
>

Hbase Distributed Operration

["Goel, Ankur" <An...@corp.aol.com>]

 
Hi,
    I installed Hbase on my local machine and configured it to connect
it to remote HDFS instance. 

The problem is that the user-account names are different on local
machine (HBase)
and remote machine (HDFS). So even though the public key/private key is
correctly
set-up between local/remote system, the ssh scripts try to login using
the local
account name that does not exist on remote machine. 

So instead of ssh login defaulting to "localuser@remotehost" on Hbase
start-up,
it should default to "remoteuser@remotehost".

Can I configure scripts to use "remoteuser" for ssh login ?

The help is much needed and appreciated.

Thanks
-Ankur