You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2011/12/15 19:24:14 UTC
Re: RegionServer unable to connect to master
Hi,
A few notes:
Remove the 127.0.1.1 lines, they usually mess things up.
The hbase.master configuration has been removed from the HBase code
more than 2 years ago, you can remove it too.
Setting hbase.master.dns.interface alone without
hbase.master.dns.nameserver doesn't do anything if I remember
correctly, also I'm pretty sure you don't have an interface named
"master", it's more likely "eth0" and whatnot.
I don't know if this will fix your DNS resolution, but at least it
will clean up a bit :)
J-D
2011/12/15 exp <ex...@163.com>:
> hi all,
>
> I am installing hbase on a small cluster of 3 machines. The RegionServer unable connect to the master. This is the log:
>
>
> 2011-12-15 13:46:43,415 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost:60000
> 2011-12-15 13:47:43,473 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> ......
>
>
>
>
> The lod shows this RegionServer connect to the master at a wrong address(localhost:60000).
>
>
> as far as I know, the RegionServer use ZooKeeper to find out the master's address. so I check the HMaster's log.
>
>
> this is what i find out:
> ...
> 2011-12-15 13:44:34,946 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2011-12-15 13:44:35,010 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=localhost:60000
> 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave2:2181,slave
> 1:2181 sessionTimeout=180000 watcher=hconnection
> ...
>
>
> The HMaster starts at localhost:60000! I think the HMaster should start at master:60000. "master" is the hosts where JobTracker runs.
>
>
> I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source code where the HMaster get the address:
>
>
> HServerAddress a = new HServerAddress(getMyAddress(this.conf));
> int numHandlers = conf.getInt("hbase.regionserver.handler.count", 10);
> this.rpcServer = HBaseRPC.getServer(this,
> new Class<?>[]{HMasterInterface.class, HMasterRegionInterface.class},
> a.getBindAddress(), a.getPort(),
> numHandlers,
> 0, // we dont use high priority handlers in master
> false, conf,
> 0); // this is a DNC w/o high priority handlers
> this.address = new HServerAddress(rpcServer.getListenerAddress());
>
>
> private static String getMyAddress(final Configuration c)
> throws UnknownHostException {
> // Find out our address up in DNS.
> String s = DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"),
> c.get("hbase.master.dns.nameserver","default"));
> s += ":" + c.get(HConstants.MASTER_PORT,
> Integer.toString(HConstants.DEFAULT_MASTER_PORT));
> return s;
> }
>
>
>
>
> As you can see in the code, the HMaster find its address in DNS! I have no idea how to setup the hbase to let the HMaster find the correct address.
>
>
> this is my hbase-site.xml:
>
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://master:9000/user/hadoop/hbase</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
>
>
> <property>
> <name>hbase.master.dns.interface</name>
> <value>master</value>
> </property>
>
>
> <property>
> <name>zookeeper.znode.master</name>
> <value>master</value>
> </property>
>
>
> <property>
> <name>hbase.master</name>
> <value>master:60000</value>
> </property>
> </configuration>
>
>
> and all the three machines are virtual machine with Ubuntu 11.04
>
>
> this is the /etc/hosts :
> master:
> 127.0.0.1 localhost ubt
> 127.0.0.1 ubt ubuntu
> 10.66.201.243 master
> 10.66.201.244 slave1
> 10.66.201.245 slave2
>
>
> slave1:
> 127.0.0.1 localhost
> 127.0.1.1 slave1 ubuntu
> 10.66.201.243 master
> 10.66.201.244 slave1
> 10.66.201.245 slave2
>
>
> slave2:
> 127.0.0.1 localhost
> 127.0.1.1 slave2 ubuntu
> 10.66.201.243 master
> 10.66.201.244 slave1
> 10.66.201.245 slave2
>
>
> any idea on this problems?
>
>
>
>
>
>
> cheers
>
>
> exception
Re: Re: Re: RegionServer unable to connect to master
Posted by shashwat shriparv <dw...@gmail.com>.
check if password less ssh is enabled for local server or not. if you have
already created it try to remove .ssh folder in /home and try recreating
.some time its problem related to this also. just a wild guess. have a look
on this.
Configuring SSH
Hadoop requires SSH access to manage its nodes, i.e. remote machines plus
your local machine if you want to use Hadoop on it (which is what we want
to do in this short tutorial). For our single-node setup of Hadoop, we
therefore need to configure SSH access to localhost for the hadoop user we
create in the previous section.
I assume that you have SSH up and running on your machine and configured it
to allow SSH public key authentication. If not, there are several
guides<http://ubuntuguide.org/>
available.
First, we have to generate an SSH key for the <tt>hadoop</tt> user.
noll@ubuntu:~$ su - hadoop
hadoop@ubuntu:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
9d:47:ab:d7:22:54:f0:f9:b9:3b:64:93:12:75:81:27 hadoop@ubuntu
hadoop@ubuntu:~$
The second line will create an RSA key pair with an empty password.
Generally, using an empty password is not recommended, but in this case it
is needed to unlock the key without your interaction (you don't want to
enter the passphrase every time Hadoop interacts with its nodes).
Second, you have to enable SSH access to your local machine with this newly
created key.
hadoop@ubuntu:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
The final step is to test the SSH setup by connecting to your local machine
with the hadoop user. The step is also needed to save your local machine's
host key fingerprint to the hadoop user'sknown_hosts file. If you have any
special SSH configuration for your local machine like a non-standard SSH
port, you can define host-specific SSH options in $HOME/.ssh/config (see
man ssh_config for more information).
hadoop@ubuntu:~$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 76:d7:61:86:ea:86:8f:31:89:9f:68:b0:75:88:52:72.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Ubuntu 7.04
...
hadoop@ubuntu:~$
If the SSH connect should fail, these general tips might help:
-
Enable debugging with ssh -vvv localhost and investigate the error in
detail.
-
Check the SSH server configuration in /etc/ssh/sshd_config, in
particular the options PubkeyAuthentication (which should be set to yes)
and AllowUsers (if this option is active, add the <tt>hadoop</tt> user
to it). If you made any changes to the SSH server configuration file, you
can force a configuration reload with sudo /etc/init.d/ssh reload.
Disabling IPv6
I have not found out yet how to configure Hadoop to listen on *all
IPv4* (again:
IPv4) network interfaces. Using 0.0.0.0 for the various networking-related
Hadoop configuration options will result in Hadoop binding to the
*IPv6* addresses
on my Ubuntu box.
As a workaround (and realizing that there's no practical point in enabling
IPv6 on a box when you are not connected to any IPv6 network), I simply
disabled IPv6 on my Ubuntu machine.
To disable IPv6 on Ubuntu Linux, open /etc/modprobe.d/blacklist in the
editor of your choice and add the following lines to the end of the file:
# disable IPv6
blacklist ipv6
You have to reboot your machine in order to make the changes take effect.
2011/12/16 exp <ex...@163.com>
> hi Mohammad Tariq,
>
>
> thanks for reply.
>
>
> I follow your instruction, change the hosts to this:
> master:
> 127.0.0.1 localhost 127.0.0.1 localhost ubuntu 10.66.201.243 master
> 10.66.201.244 slave1 10.66.201.245 slave2
>
>
> slave1:
> 127.0.0.1 localhost 127.0.0.1 slave1 ubuntu 10.66.201.243 master
> 10.66.201.244 slave1 10.66.201.245 slave2
>
> slave2:
> 127.0.0.1 localhost
> 127.0.0.1 slave2 ubuntu
> 10.66.201.243 master
> 10.66.201.244 slave1
> 10.66.201.245 slave2
>
>
> It still doesn't work. the same problem.
> The regionserver try to connect to localhost:60000. And the hmaster still
> start at localhost:60000.
>
>
>
>
>
>
>
> At 2011-12-16 15:37:53,"Mohammad Tariq" <do...@gmail.com> wrote:
> >Hi exp,
> >
> > Do not remove this line, instead make it 127.0.0.1..and copy the
> >hadoop-core-0.20.204.0.jar from your HADOOP_HOME and
> >commons-configuration-1.6.jar from the HADOOP_HOME/lib folder to the
> >HBASE_HOME/lib folder. It should work then..Please let me know if it
> >works for you.
> >
> >Regards,
> > Mohammad Tariq
> >
> >
> >
> >2011/12/16 exp <ex...@163.com>:
> >> hi,
> >>
> >>
> >> I'm using Hadoop 0.20.204.0
> >>
> >>
> >> After I remove the 127.0.1.1 lines, the HMaster cannot start. I get
> this exception:
> >>
> >>
> >> 2011-12-16 13:37:11,899 ERROR
> org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1060)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1074) Caused
> by: java.net.BindException: Cannot assign requested address at
> sun.nio.ch.Net.bind(Native Method) at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at
> org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.j
> ava
> >> :201) at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.<init>(HBaseServer.java:270)
> at org.apache.hadoop.hbase.ipc.HBaseServer.<init>(HBaseServer.java:1168) at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.<init>(HBaseRPC.java:544) at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:514) at
> org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:196) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1055)
> ... 5 more
> >>
> >>
> >> what address should the hmaster rpcserver bind to?
> >>
> >>
> >>
> >>
> >> cheers
> >> exception
> >>
> >>
> >> At 2011-12-16 02:37:43,"Mohammad Tariq" <do...@gmail.com> wrote:
> >>>I agree with J-D and Shashwat. BTW, which version of Hadoop are you
> using??
> >>>Regards,
> >>> Mohammad Tariq
> >>>
> >>>
> >>>
> >>>On Thu, Dec 15, 2011 at 11:56 PM, shashwat shriparv
> >>><dw...@gmail.com> wrote:
> >>>> make 127.0.1.1 to 127.0.0.1 that will solve lot of problems
> >>>>
> >>>> On Thu, Dec 15, 2011 at 11:54 PM, Jean-Daniel Cryans <
> jdcryans@apache.org>wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> A few notes:
> >>>>>
> >>>>> Remove the 127.0.1.1 lines, they usually mess things up.
> >>>>>
> >>>>> The hbase.master configuration has been removed from the HBase code
> >>>>> more than 2 years ago, you can remove it too.
> >>>>>
> >>>>> Setting hbase.master.dns.interface alone without
> >>>>> hbase.master.dns.nameserver doesn't do anything if I remember
> >>>>> correctly, also I'm pretty sure you don't have an interface named
> >>>>> "master", it's more likely "eth0" and whatnot.
> >>>>>
> >>>>> I don't know if this will fix your DNS resolution, but at least it
> >>>>> will clean up a bit :)
> >>>>>
> >>>>> J-D
> >>>>>
> >>>>> 2011/12/15 exp <ex...@163.com>:
> >>>>> > hi all,
> >>>>> >
> >>>>> > I am installing hbase on a small cluster of 3 machines. The
> RegionServer
> >>>>> unable connect to the master. This is the log:
> >>>>> >
> >>>>> >
> >>>>> > 2011-12-15 13:46:43,415 INFO
> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting
> connect to
> >>>>> Master server at localhost:60000
> >>>>> > 2011-12-15 13:47:43,473 WARN
> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
> connect to
> >>>>> master. Retrying. Error was:
> >>>>> > java.net.ConnectException: Connection refused
> >>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>>> > at
> >>>>>
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >>>>> > at
> >>>>>
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >>>>> > ......
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > The lod shows this RegionServer connect to the master at a wrong
> >>>>> address(localhost:60000).
> >>>>> >
> >>>>> >
> >>>>> > as far as I know, the RegionServer use ZooKeeper to find out the
> >>>>> master's address. so I check the HMaster's log.
> >>>>> >
> >>>>> >
> >>>>> > this is what i find out:
> >>>>> > ...
> >>>>> > 2011-12-15 13:44:34,946 INFO
> >>>>> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> >>>>> > 2011-12-15 13:44:35,010 INFO
> >>>>> org.apache.hadoop.hbase.master.ActiveMasterManager:
> Master=localhost:60000
> >>>>> > 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper:
> Initiating
> >>>>> client connection, connectString=master:2181,slave2:2181,slave
> >>>>> > 1:2181 sessionTimeout=180000 watcher=hconnection
> >>>>> > ...
> >>>>> >
> >>>>> >
> >>>>> > The HMaster starts at localhost:60000! I think the HMaster should
> start
> >>>>> at master:60000. "master" is the hosts where JobTracker runs.
> >>>>> >
> >>>>> >
> >>>>> > I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source
> code
> >>>>> where the HMaster get the address:
> >>>>> >
> >>>>> >
> >>>>> > HServerAddress a = new HServerAddress(getMyAddress(this.conf));
> >>>>> > int numHandlers = conf.getInt("hbase.regionserver.handler.count",
> 10);
> >>>>> > this.rpcServer = HBaseRPC.getServer(this,
> >>>>> > new Class<?>[]{HMasterInterface.class,
> >>>>> HMasterRegionInterface.class},
> >>>>> > a.getBindAddress(), a.getPort(),
> >>>>> > numHandlers,
> >>>>> > 0, // we dont use high priority handlers in master
> >>>>> > false, conf,
> >>>>> > 0); // this is a DNC w/o high priority handlers
> >>>>> > this.address = new HServerAddress(rpcServer.getListenerAddress());
> >>>>> >
> >>>>> >
> >>>>> > private static String getMyAddress(final Configuration c)
> >>>>> > throws UnknownHostException {
> >>>>> > // Find out our address up in DNS.
> >>>>> > String s =
> >>>>> DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"),
> >>>>> > c.get("hbase.master.dns.nameserver","default"));
> >>>>> > s += ":" + c.get(HConstants.MASTER_PORT,
> >>>>> > Integer.toString(HConstants.DEFAULT_MASTER_PORT));
> >>>>> > return s;
> >>>>> > }
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > As you can see in the code, the HMaster find its address in DNS! I
> have
> >>>>> no idea how to setup the hbase to let the HMaster find the correct
> address.
> >>>>> >
> >>>>> >
> >>>>> > this is my hbase-site.xml:
> >>>>> >
> >>>>> >
> >>>>> > <configuration>
> >>>>> > <property>
> >>>>> > <name>hbase.rootdir</name>
> >>>>> > <value>hdfs://master:9000/user/hadoop/hbase</value>
> >>>>> > </property>
> >>>>> > <property>
> >>>>> > <name>hbase.cluster.distributed</name>
> >>>>> > <value>true</value>
> >>>>> > </property>
> >>>>> >
> >>>>> >
> >>>>> > <property>
> >>>>> > <name>hbase.master.dns.interface</name>
> >>>>> > <value>master</value>
> >>>>> > </property>
> >>>>> >
> >>>>> >
> >>>>> > <property>
> >>>>> > <name>zookeeper.znode.master</name>
> >>>>> > <value>master</value>
> >>>>> > </property>
> >>>>> >
> >>>>> >
> >>>>> > <property>
> >>>>> > <name>hbase.master</name>
> >>>>> > <value>master:60000</value>
> >>>>> > </property>
> >>>>> > </configuration>
> >>>>> >
> >>>>> >
> >>>>> > and all the three machines are virtual machine with Ubuntu 11.04
> >>>>> >
> >>>>> >
> >>>>> > this is the /etc/hosts :
> >>>>> > master:
> >>>>> > 127.0.0.1 localhost ubt
> >>>>> > 127.0.0.1 ubt ubuntu
> >>>>> > 10.66.201.243 master
> >>>>> > 10.66.201.244 slave1
> >>>>> > 10.66.201.245 slave2
> >>>>> >
> >>>>> >
> >>>>> > slave1:
> >>>>> > 127.0.0.1 localhost
> >>>>> > 127.0.1.1 slave1 ubuntu
> >>>>> > 10.66.201.243 master
> >>>>> > 10.66.201.244 slave1
> >>>>> > 10.66.201.245 slave2
> >>>>> >
> >>>>> >
> >>>>> > slave2:
> >>>>> > 127.0.0.1 localhost
> >>>>> > 127.0.1.1 slave2 ubuntu
> >>>>> > 10.66.201.243 master
> >>>>> > 10.66.201.244 slave1
> >>>>> > 10.66.201.245 slave2
> >>>>> >
> >>>>> >
> >>>>> > any idea on this problems?
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > cheers
> >>>>> >
> >>>>> >
> >>>>> > exception
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Shashwat Shriparv
> >>>> 09900059620
> >>>> 09663531241
> >>>>
> >>>>
> >>>>
> >>>> <iframe src="
> >>>>
> http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr
> "
> >>>> width="728" height="90" scrolling="no" border="0" marginwidth="0"
> >>>> style="border:none;" frameborder="0"></iframe>
>
--
Shashwat Shriparv
09900059620
09663531241
<iframe src="
http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr"
width="728" height="90" scrolling="no" border="0" marginwidth="0"
style="border:none;" frameborder="0"></iframe>
Re:Re: Re: RegionServer unable to connect to master
Posted by exp <ex...@163.com>.
hi Mohammad Tariq,
thanks for reply.
I follow your instruction, change the hosts to this:
master:
127.0.0.1 localhost 127.0.0.1 localhost ubuntu 10.66.201.243 master 10.66.201.244 slave1 10.66.201.245 slave2
slave1:
127.0.0.1 localhost 127.0.0.1 slave1 ubuntu 10.66.201.243 master 10.66.201.244 slave1 10.66.201.245 slave2
slave2:
127.0.0.1 localhost
127.0.0.1 slave2 ubuntu
10.66.201.243 master
10.66.201.244 slave1
10.66.201.245 slave2
It still doesn't work. the same problem.
The regionserver try to connect to localhost:60000. And the hmaster still start at localhost:60000.
At 2011-12-16 15:37:53,"Mohammad Tariq" <do...@gmail.com> wrote:
>Hi exp,
>
> Do not remove this line, instead make it 127.0.0.1..and copy the
>hadoop-core-0.20.204.0.jar from your HADOOP_HOME and
>commons-configuration-1.6.jar from the HADOOP_HOME/lib folder to the
>HBASE_HOME/lib folder. It should work then..Please let me know if it
>works for you.
>
>Regards,
> Mohammad Tariq
>
>
>
>2011/12/16 exp <ex...@163.com>:
>> hi,
>>
>>
>> I'm using Hadoop 0.20.204.0
>>
>>
>> After I remove the 127.0.1.1 lines, the HMaster cannot start. I get this exception:
>>
>>
>> 2011-12-16 13:37:11,899 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1060) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1074) Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.j
ava
>> :201) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.<init>(HBaseServer.java:270) at org.apache.hadoop.hbase.ipc.HBaseServer.<init>(HBaseServer.java:1168) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.<init>(HBaseRPC.java:544) at org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:514) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1055) ... 5 more
>>
>>
>> what address should the hmaster rpcserver bind to?
>>
>>
>>
>>
>> cheers
>> exception
>>
>>
>> At 2011-12-16 02:37:43,"Mohammad Tariq" <do...@gmail.com> wrote:
>>>I agree with J-D and Shashwat. BTW, which version of Hadoop are you using??
>>>Regards,
>>> Mohammad Tariq
>>>
>>>
>>>
>>>On Thu, Dec 15, 2011 at 11:56 PM, shashwat shriparv
>>><dw...@gmail.com> wrote:
>>>> make 127.0.1.1 to 127.0.0.1 that will solve lot of problems
>>>>
>>>> On Thu, Dec 15, 2011 at 11:54 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> A few notes:
>>>>>
>>>>> Remove the 127.0.1.1 lines, they usually mess things up.
>>>>>
>>>>> The hbase.master configuration has been removed from the HBase code
>>>>> more than 2 years ago, you can remove it too.
>>>>>
>>>>> Setting hbase.master.dns.interface alone without
>>>>> hbase.master.dns.nameserver doesn't do anything if I remember
>>>>> correctly, also I'm pretty sure you don't have an interface named
>>>>> "master", it's more likely "eth0" and whatnot.
>>>>>
>>>>> I don't know if this will fix your DNS resolution, but at least it
>>>>> will clean up a bit :)
>>>>>
>>>>> J-D
>>>>>
>>>>> 2011/12/15 exp <ex...@163.com>:
>>>>> > hi all,
>>>>> >
>>>>> > I am installing hbase on a small cluster of 3 machines. The RegionServer
>>>>> unable connect to the master. This is the log:
>>>>> >
>>>>> >
>>>>> > 2011-12-15 13:46:43,415 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
>>>>> Master server at localhost:60000
>>>>> > 2011-12-15 13:47:43,473 WARN
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
>>>>> master. Retrying. Error was:
>>>>> > java.net.ConnectException: Connection refused
>>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>> > at
>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>>>>> > at
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>>>>> > ......
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > The lod shows this RegionServer connect to the master at a wrong
>>>>> address(localhost:60000).
>>>>> >
>>>>> >
>>>>> > as far as I know, the RegionServer use ZooKeeper to find out the
>>>>> master's address. so I check the HMaster's log.
>>>>> >
>>>>> >
>>>>> > this is what i find out:
>>>>> > ...
>>>>> > 2011-12-15 13:44:34,946 INFO
>>>>> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
>>>>> > 2011-12-15 13:44:35,010 INFO
>>>>> org.apache.hadoop.hbase.master.ActiveMasterManager: Master=localhost:60000
>>>>> > 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>>>> client connection, connectString=master:2181,slave2:2181,slave
>>>>> > 1:2181 sessionTimeout=180000 watcher=hconnection
>>>>> > ...
>>>>> >
>>>>> >
>>>>> > The HMaster starts at localhost:60000! I think the HMaster should start
>>>>> at master:60000. "master" is the hosts where JobTracker runs.
>>>>> >
>>>>> >
>>>>> > I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source code
>>>>> where the HMaster get the address:
>>>>> >
>>>>> >
>>>>> > HServerAddress a = new HServerAddress(getMyAddress(this.conf));
>>>>> > int numHandlers = conf.getInt("hbase.regionserver.handler.count", 10);
>>>>> > this.rpcServer = HBaseRPC.getServer(this,
>>>>> > new Class<?>[]{HMasterInterface.class,
>>>>> HMasterRegionInterface.class},
>>>>> > a.getBindAddress(), a.getPort(),
>>>>> > numHandlers,
>>>>> > 0, // we dont use high priority handlers in master
>>>>> > false, conf,
>>>>> > 0); // this is a DNC w/o high priority handlers
>>>>> > this.address = new HServerAddress(rpcServer.getListenerAddress());
>>>>> >
>>>>> >
>>>>> > private static String getMyAddress(final Configuration c)
>>>>> > throws UnknownHostException {
>>>>> > // Find out our address up in DNS.
>>>>> > String s =
>>>>> DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"),
>>>>> > c.get("hbase.master.dns.nameserver","default"));
>>>>> > s += ":" + c.get(HConstants.MASTER_PORT,
>>>>> > Integer.toString(HConstants.DEFAULT_MASTER_PORT));
>>>>> > return s;
>>>>> > }
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > As you can see in the code, the HMaster find its address in DNS! I have
>>>>> no idea how to setup the hbase to let the HMaster find the correct address.
>>>>> >
>>>>> >
>>>>> > this is my hbase-site.xml:
>>>>> >
>>>>> >
>>>>> > <configuration>
>>>>> > <property>
>>>>> > <name>hbase.rootdir</name>
>>>>> > <value>hdfs://master:9000/user/hadoop/hbase</value>
>>>>> > </property>
>>>>> > <property>
>>>>> > <name>hbase.cluster.distributed</name>
>>>>> > <value>true</value>
>>>>> > </property>
>>>>> >
>>>>> >
>>>>> > <property>
>>>>> > <name>hbase.master.dns.interface</name>
>>>>> > <value>master</value>
>>>>> > </property>
>>>>> >
>>>>> >
>>>>> > <property>
>>>>> > <name>zookeeper.znode.master</name>
>>>>> > <value>master</value>
>>>>> > </property>
>>>>> >
>>>>> >
>>>>> > <property>
>>>>> > <name>hbase.master</name>
>>>>> > <value>master:60000</value>
>>>>> > </property>
>>>>> > </configuration>
>>>>> >
>>>>> >
>>>>> > and all the three machines are virtual machine with Ubuntu 11.04
>>>>> >
>>>>> >
>>>>> > this is the /etc/hosts :
>>>>> > master:
>>>>> > 127.0.0.1 localhost ubt
>>>>> > 127.0.0.1 ubt ubuntu
>>>>> > 10.66.201.243 master
>>>>> > 10.66.201.244 slave1
>>>>> > 10.66.201.245 slave2
>>>>> >
>>>>> >
>>>>> > slave1:
>>>>> > 127.0.0.1 localhost
>>>>> > 127.0.1.1 slave1 ubuntu
>>>>> > 10.66.201.243 master
>>>>> > 10.66.201.244 slave1
>>>>> > 10.66.201.245 slave2
>>>>> >
>>>>> >
>>>>> > slave2:
>>>>> > 127.0.0.1 localhost
>>>>> > 127.0.1.1 slave2 ubuntu
>>>>> > 10.66.201.243 master
>>>>> > 10.66.201.244 slave1
>>>>> > 10.66.201.245 slave2
>>>>> >
>>>>> >
>>>>> > any idea on this problems?
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > cheers
>>>>> >
>>>>> >
>>>>> > exception
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Shashwat Shriparv
>>>> 09900059620
>>>> 09663531241
>>>>
>>>>
>>>>
>>>> <iframe src="
>>>> http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr"
>>>> width="728" height="90" scrolling="no" border="0" marginwidth="0"
>>>> style="border:none;" frameborder="0"></iframe>
Re: Re: RegionServer unable to connect to master
Posted by Mohammad Tariq <do...@gmail.com>.
Hi exp,
Do not remove this line, instead make it 127.0.0.1..and copy the
hadoop-core-0.20.204.0.jar from your HADOOP_HOME and
commons-configuration-1.6.jar from the HADOOP_HOME/lib folder to the
HBASE_HOME/lib folder. It should work then..Please let me know if it
works for you.
Regards,
Mohammad Tariq
2011/12/16 exp <ex...@163.com>:
> hi,
>
>
> I'm using Hadoop 0.20.204.0
>
>
> After I remove the 127.0.1.1 lines, the HMaster cannot start. I get this exception:
>
>
> 2011-12-16 13:37:11,899 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1060) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1074) Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.java
> :201) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.<init>(HBaseServer.java:270) at org.apache.hadoop.hbase.ipc.HBaseServer.<init>(HBaseServer.java:1168) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.<init>(HBaseRPC.java:544) at org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:514) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1055) ... 5 more
>
>
> what address should the hmaster rpcserver bind to?
>
>
>
>
> cheers
> exception
>
>
> At 2011-12-16 02:37:43,"Mohammad Tariq" <do...@gmail.com> wrote:
>>I agree with J-D and Shashwat. BTW, which version of Hadoop are you using??
>>Regards,
>> Mohammad Tariq
>>
>>
>>
>>On Thu, Dec 15, 2011 at 11:56 PM, shashwat shriparv
>><dw...@gmail.com> wrote:
>>> make 127.0.1.1 to 127.0.0.1 that will solve lot of problems
>>>
>>> On Thu, Dec 15, 2011 at 11:54 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>>>
>>>> Hi,
>>>>
>>>> A few notes:
>>>>
>>>> Remove the 127.0.1.1 lines, they usually mess things up.
>>>>
>>>> The hbase.master configuration has been removed from the HBase code
>>>> more than 2 years ago, you can remove it too.
>>>>
>>>> Setting hbase.master.dns.interface alone without
>>>> hbase.master.dns.nameserver doesn't do anything if I remember
>>>> correctly, also I'm pretty sure you don't have an interface named
>>>> "master", it's more likely "eth0" and whatnot.
>>>>
>>>> I don't know if this will fix your DNS resolution, but at least it
>>>> will clean up a bit :)
>>>>
>>>> J-D
>>>>
>>>> 2011/12/15 exp <ex...@163.com>:
>>>> > hi all,
>>>> >
>>>> > I am installing hbase on a small cluster of 3 machines. The RegionServer
>>>> unable connect to the master. This is the log:
>>>> >
>>>> >
>>>> > 2011-12-15 13:46:43,415 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
>>>> Master server at localhost:60000
>>>> > 2011-12-15 13:47:43,473 WARN
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
>>>> master. Retrying. Error was:
>>>> > java.net.ConnectException: Connection refused
>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> > at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>>>> > at
>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>>>> > ......
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > The lod shows this RegionServer connect to the master at a wrong
>>>> address(localhost:60000).
>>>> >
>>>> >
>>>> > as far as I know, the RegionServer use ZooKeeper to find out the
>>>> master's address. so I check the HMaster's log.
>>>> >
>>>> >
>>>> > this is what i find out:
>>>> > ...
>>>> > 2011-12-15 13:44:34,946 INFO
>>>> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
>>>> > 2011-12-15 13:44:35,010 INFO
>>>> org.apache.hadoop.hbase.master.ActiveMasterManager: Master=localhost:60000
>>>> > 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>>> client connection, connectString=master:2181,slave2:2181,slave
>>>> > 1:2181 sessionTimeout=180000 watcher=hconnection
>>>> > ...
>>>> >
>>>> >
>>>> > The HMaster starts at localhost:60000! I think the HMaster should start
>>>> at master:60000. "master" is the hosts where JobTracker runs.
>>>> >
>>>> >
>>>> > I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source code
>>>> where the HMaster get the address:
>>>> >
>>>> >
>>>> > HServerAddress a = new HServerAddress(getMyAddress(this.conf));
>>>> > int numHandlers = conf.getInt("hbase.regionserver.handler.count", 10);
>>>> > this.rpcServer = HBaseRPC.getServer(this,
>>>> > new Class<?>[]{HMasterInterface.class,
>>>> HMasterRegionInterface.class},
>>>> > a.getBindAddress(), a.getPort(),
>>>> > numHandlers,
>>>> > 0, // we dont use high priority handlers in master
>>>> > false, conf,
>>>> > 0); // this is a DNC w/o high priority handlers
>>>> > this.address = new HServerAddress(rpcServer.getListenerAddress());
>>>> >
>>>> >
>>>> > private static String getMyAddress(final Configuration c)
>>>> > throws UnknownHostException {
>>>> > // Find out our address up in DNS.
>>>> > String s =
>>>> DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"),
>>>> > c.get("hbase.master.dns.nameserver","default"));
>>>> > s += ":" + c.get(HConstants.MASTER_PORT,
>>>> > Integer.toString(HConstants.DEFAULT_MASTER_PORT));
>>>> > return s;
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > As you can see in the code, the HMaster find its address in DNS! I have
>>>> no idea how to setup the hbase to let the HMaster find the correct address.
>>>> >
>>>> >
>>>> > this is my hbase-site.xml:
>>>> >
>>>> >
>>>> > <configuration>
>>>> > <property>
>>>> > <name>hbase.rootdir</name>
>>>> > <value>hdfs://master:9000/user/hadoop/hbase</value>
>>>> > </property>
>>>> > <property>
>>>> > <name>hbase.cluster.distributed</name>
>>>> > <value>true</value>
>>>> > </property>
>>>> >
>>>> >
>>>> > <property>
>>>> > <name>hbase.master.dns.interface</name>
>>>> > <value>master</value>
>>>> > </property>
>>>> >
>>>> >
>>>> > <property>
>>>> > <name>zookeeper.znode.master</name>
>>>> > <value>master</value>
>>>> > </property>
>>>> >
>>>> >
>>>> > <property>
>>>> > <name>hbase.master</name>
>>>> > <value>master:60000</value>
>>>> > </property>
>>>> > </configuration>
>>>> >
>>>> >
>>>> > and all the three machines are virtual machine with Ubuntu 11.04
>>>> >
>>>> >
>>>> > this is the /etc/hosts :
>>>> > master:
>>>> > 127.0.0.1 localhost ubt
>>>> > 127.0.0.1 ubt ubuntu
>>>> > 10.66.201.243 master
>>>> > 10.66.201.244 slave1
>>>> > 10.66.201.245 slave2
>>>> >
>>>> >
>>>> > slave1:
>>>> > 127.0.0.1 localhost
>>>> > 127.0.1.1 slave1 ubuntu
>>>> > 10.66.201.243 master
>>>> > 10.66.201.244 slave1
>>>> > 10.66.201.245 slave2
>>>> >
>>>> >
>>>> > slave2:
>>>> > 127.0.0.1 localhost
>>>> > 127.0.1.1 slave2 ubuntu
>>>> > 10.66.201.243 master
>>>> > 10.66.201.244 slave1
>>>> > 10.66.201.245 slave2
>>>> >
>>>> >
>>>> > any idea on this problems?
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > cheers
>>>> >
>>>> >
>>>> > exception
>>>>
>>>
>>>
>>>
>>> --
>>> Shashwat Shriparv
>>> 09900059620
>>> 09663531241
>>>
>>>
>>>
>>> <iframe src="
>>> http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr"
>>> width="728" height="90" scrolling="no" border="0" marginwidth="0"
>>> style="border:none;" frameborder="0"></iframe>
Re:Re: RegionServer unable to connect to master
Posted by exp <ex...@163.com>.
hi,
I'm using Hadoop 0.20.204.0
After I remove the 127.0.1.1 lines, the HMaster cannot start. I get this exception:
2011-12-16 13:37:11,899 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1060) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1074) Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind(Native Method) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.java
:201) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.<init>(HBaseServer.java:270) at org.apache.hadoop.hbase.ipc.HBaseServer.<init>(HBaseServer.java:1168) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.<init>(HBaseRPC.java:544) at org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:514) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1055) ... 5 more
what address should the hmaster rpcserver bind to?
cheers
exception
At 2011-12-16 02:37:43,"Mohammad Tariq" <do...@gmail.com> wrote:
>I agree with J-D and Shashwat. BTW, which version of Hadoop are you using??
>Regards,
> Mohammad Tariq
>
>
>
>On Thu, Dec 15, 2011 at 11:56 PM, shashwat shriparv
><dw...@gmail.com> wrote:
>> make 127.0.1.1 to 127.0.0.1 that will solve lot of problems
>>
>> On Thu, Dec 15, 2011 at 11:54 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>>
>>> Hi,
>>>
>>> A few notes:
>>>
>>> Remove the 127.0.1.1 lines, they usually mess things up.
>>>
>>> The hbase.master configuration has been removed from the HBase code
>>> more than 2 years ago, you can remove it too.
>>>
>>> Setting hbase.master.dns.interface alone without
>>> hbase.master.dns.nameserver doesn't do anything if I remember
>>> correctly, also I'm pretty sure you don't have an interface named
>>> "master", it's more likely "eth0" and whatnot.
>>>
>>> I don't know if this will fix your DNS resolution, but at least it
>>> will clean up a bit :)
>>>
>>> J-D
>>>
>>> 2011/12/15 exp <ex...@163.com>:
>>> > hi all,
>>> >
>>> > I am installing hbase on a small cluster of 3 machines. The RegionServer
>>> unable connect to the master. This is the log:
>>> >
>>> >
>>> > 2011-12-15 13:46:43,415 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
>>> Master server at localhost:60000
>>> > 2011-12-15 13:47:43,473 WARN
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
>>> master. Retrying. Error was:
>>> > java.net.ConnectException: Connection refused
>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> > at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>>> > at
>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>>> > ......
>>> >
>>> >
>>> >
>>> >
>>> > The lod shows this RegionServer connect to the master at a wrong
>>> address(localhost:60000).
>>> >
>>> >
>>> > as far as I know, the RegionServer use ZooKeeper to find out the
>>> master's address. so I check the HMaster's log.
>>> >
>>> >
>>> > this is what i find out:
>>> > ...
>>> > 2011-12-15 13:44:34,946 INFO
>>> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
>>> > 2011-12-15 13:44:35,010 INFO
>>> org.apache.hadoop.hbase.master.ActiveMasterManager: Master=localhost:60000
>>> > 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>> client connection, connectString=master:2181,slave2:2181,slave
>>> > 1:2181 sessionTimeout=180000 watcher=hconnection
>>> > ...
>>> >
>>> >
>>> > The HMaster starts at localhost:60000! I think the HMaster should start
>>> at master:60000. "master" is the hosts where JobTracker runs.
>>> >
>>> >
>>> > I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source code
>>> where the HMaster get the address:
>>> >
>>> >
>>> > HServerAddress a = new HServerAddress(getMyAddress(this.conf));
>>> > int numHandlers = conf.getInt("hbase.regionserver.handler.count", 10);
>>> > this.rpcServer = HBaseRPC.getServer(this,
>>> > new Class<?>[]{HMasterInterface.class,
>>> HMasterRegionInterface.class},
>>> > a.getBindAddress(), a.getPort(),
>>> > numHandlers,
>>> > 0, // we dont use high priority handlers in master
>>> > false, conf,
>>> > 0); // this is a DNC w/o high priority handlers
>>> > this.address = new HServerAddress(rpcServer.getListenerAddress());
>>> >
>>> >
>>> > private static String getMyAddress(final Configuration c)
>>> > throws UnknownHostException {
>>> > // Find out our address up in DNS.
>>> > String s =
>>> DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"),
>>> > c.get("hbase.master.dns.nameserver","default"));
>>> > s += ":" + c.get(HConstants.MASTER_PORT,
>>> > Integer.toString(HConstants.DEFAULT_MASTER_PORT));
>>> > return s;
>>> > }
>>> >
>>> >
>>> >
>>> >
>>> > As you can see in the code, the HMaster find its address in DNS! I have
>>> no idea how to setup the hbase to let the HMaster find the correct address.
>>> >
>>> >
>>> > this is my hbase-site.xml:
>>> >
>>> >
>>> > <configuration>
>>> > <property>
>>> > <name>hbase.rootdir</name>
>>> > <value>hdfs://master:9000/user/hadoop/hbase</value>
>>> > </property>
>>> > <property>
>>> > <name>hbase.cluster.distributed</name>
>>> > <value>true</value>
>>> > </property>
>>> >
>>> >
>>> > <property>
>>> > <name>hbase.master.dns.interface</name>
>>> > <value>master</value>
>>> > </property>
>>> >
>>> >
>>> > <property>
>>> > <name>zookeeper.znode.master</name>
>>> > <value>master</value>
>>> > </property>
>>> >
>>> >
>>> > <property>
>>> > <name>hbase.master</name>
>>> > <value>master:60000</value>
>>> > </property>
>>> > </configuration>
>>> >
>>> >
>>> > and all the three machines are virtual machine with Ubuntu 11.04
>>> >
>>> >
>>> > this is the /etc/hosts :
>>> > master:
>>> > 127.0.0.1 localhost ubt
>>> > 127.0.0.1 ubt ubuntu
>>> > 10.66.201.243 master
>>> > 10.66.201.244 slave1
>>> > 10.66.201.245 slave2
>>> >
>>> >
>>> > slave1:
>>> > 127.0.0.1 localhost
>>> > 127.0.1.1 slave1 ubuntu
>>> > 10.66.201.243 master
>>> > 10.66.201.244 slave1
>>> > 10.66.201.245 slave2
>>> >
>>> >
>>> > slave2:
>>> > 127.0.0.1 localhost
>>> > 127.0.1.1 slave2 ubuntu
>>> > 10.66.201.243 master
>>> > 10.66.201.244 slave1
>>> > 10.66.201.245 slave2
>>> >
>>> >
>>> > any idea on this problems?
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > cheers
>>> >
>>> >
>>> > exception
>>>
>>
>>
>>
>> --
>> Shashwat Shriparv
>> 09900059620
>> 09663531241
>>
>>
>>
>> <iframe src="
>> http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr"
>> width="728" height="90" scrolling="no" border="0" marginwidth="0"
>> style="border:none;" frameborder="0"></iframe>
Re: RegionServer unable to connect to master
Posted by Mohammad Tariq <do...@gmail.com>.
I agree with J-D and Shashwat. BTW, which version of Hadoop are you using??
Regards,
Mohammad Tariq
On Thu, Dec 15, 2011 at 11:56 PM, shashwat shriparv
<dw...@gmail.com> wrote:
> make 127.0.1.1 to 127.0.0.1 that will solve lot of problems
>
> On Thu, Dec 15, 2011 at 11:54 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> Hi,
>>
>> A few notes:
>>
>> Remove the 127.0.1.1 lines, they usually mess things up.
>>
>> The hbase.master configuration has been removed from the HBase code
>> more than 2 years ago, you can remove it too.
>>
>> Setting hbase.master.dns.interface alone without
>> hbase.master.dns.nameserver doesn't do anything if I remember
>> correctly, also I'm pretty sure you don't have an interface named
>> "master", it's more likely "eth0" and whatnot.
>>
>> I don't know if this will fix your DNS resolution, but at least it
>> will clean up a bit :)
>>
>> J-D
>>
>> 2011/12/15 exp <ex...@163.com>:
>> > hi all,
>> >
>> > I am installing hbase on a small cluster of 3 machines. The RegionServer
>> unable connect to the master. This is the log:
>> >
>> >
>> > 2011-12-15 13:46:43,415 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
>> Master server at localhost:60000
>> > 2011-12-15 13:47:43,473 WARN
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
>> master. Retrying. Error was:
>> > java.net.ConnectException: Connection refused
>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> > at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>> > at
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> > ......
>> >
>> >
>> >
>> >
>> > The lod shows this RegionServer connect to the master at a wrong
>> address(localhost:60000).
>> >
>> >
>> > as far as I know, the RegionServer use ZooKeeper to find out the
>> master's address. so I check the HMaster's log.
>> >
>> >
>> > this is what i find out:
>> > ...
>> > 2011-12-15 13:44:34,946 INFO
>> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
>> > 2011-12-15 13:44:35,010 INFO
>> org.apache.hadoop.hbase.master.ActiveMasterManager: Master=localhost:60000
>> > 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper: Initiating
>> client connection, connectString=master:2181,slave2:2181,slave
>> > 1:2181 sessionTimeout=180000 watcher=hconnection
>> > ...
>> >
>> >
>> > The HMaster starts at localhost:60000! I think the HMaster should start
>> at master:60000. "master" is the hosts where JobTracker runs.
>> >
>> >
>> > I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source code
>> where the HMaster get the address:
>> >
>> >
>> > HServerAddress a = new HServerAddress(getMyAddress(this.conf));
>> > int numHandlers = conf.getInt("hbase.regionserver.handler.count", 10);
>> > this.rpcServer = HBaseRPC.getServer(this,
>> > new Class<?>[]{HMasterInterface.class,
>> HMasterRegionInterface.class},
>> > a.getBindAddress(), a.getPort(),
>> > numHandlers,
>> > 0, // we dont use high priority handlers in master
>> > false, conf,
>> > 0); // this is a DNC w/o high priority handlers
>> > this.address = new HServerAddress(rpcServer.getListenerAddress());
>> >
>> >
>> > private static String getMyAddress(final Configuration c)
>> > throws UnknownHostException {
>> > // Find out our address up in DNS.
>> > String s =
>> DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"),
>> > c.get("hbase.master.dns.nameserver","default"));
>> > s += ":" + c.get(HConstants.MASTER_PORT,
>> > Integer.toString(HConstants.DEFAULT_MASTER_PORT));
>> > return s;
>> > }
>> >
>> >
>> >
>> >
>> > As you can see in the code, the HMaster find its address in DNS! I have
>> no idea how to setup the hbase to let the HMaster find the correct address.
>> >
>> >
>> > this is my hbase-site.xml:
>> >
>> >
>> > <configuration>
>> > <property>
>> > <name>hbase.rootdir</name>
>> > <value>hdfs://master:9000/user/hadoop/hbase</value>
>> > </property>
>> > <property>
>> > <name>hbase.cluster.distributed</name>
>> > <value>true</value>
>> > </property>
>> >
>> >
>> > <property>
>> > <name>hbase.master.dns.interface</name>
>> > <value>master</value>
>> > </property>
>> >
>> >
>> > <property>
>> > <name>zookeeper.znode.master</name>
>> > <value>master</value>
>> > </property>
>> >
>> >
>> > <property>
>> > <name>hbase.master</name>
>> > <value>master:60000</value>
>> > </property>
>> > </configuration>
>> >
>> >
>> > and all the three machines are virtual machine with Ubuntu 11.04
>> >
>> >
>> > this is the /etc/hosts :
>> > master:
>> > 127.0.0.1 localhost ubt
>> > 127.0.0.1 ubt ubuntu
>> > 10.66.201.243 master
>> > 10.66.201.244 slave1
>> > 10.66.201.245 slave2
>> >
>> >
>> > slave1:
>> > 127.0.0.1 localhost
>> > 127.0.1.1 slave1 ubuntu
>> > 10.66.201.243 master
>> > 10.66.201.244 slave1
>> > 10.66.201.245 slave2
>> >
>> >
>> > slave2:
>> > 127.0.0.1 localhost
>> > 127.0.1.1 slave2 ubuntu
>> > 10.66.201.243 master
>> > 10.66.201.244 slave1
>> > 10.66.201.245 slave2
>> >
>> >
>> > any idea on this problems?
>> >
>> >
>> >
>> >
>> >
>> >
>> > cheers
>> >
>> >
>> > exception
>>
>
>
>
> --
> Shashwat Shriparv
> 09900059620
> 09663531241
>
>
>
> <iframe src="
> http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr"
> width="728" height="90" scrolling="no" border="0" marginwidth="0"
> style="border:none;" frameborder="0"></iframe>
Re: RegionServer unable to connect to master
Posted by shashwat shriparv <dw...@gmail.com>.
make 127.0.1.1 to 127.0.0.1 that will solve lot of problems
On Thu, Dec 15, 2011 at 11:54 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Hi,
>
> A few notes:
>
> Remove the 127.0.1.1 lines, they usually mess things up.
>
> The hbase.master configuration has been removed from the HBase code
> more than 2 years ago, you can remove it too.
>
> Setting hbase.master.dns.interface alone without
> hbase.master.dns.nameserver doesn't do anything if I remember
> correctly, also I'm pretty sure you don't have an interface named
> "master", it's more likely "eth0" and whatnot.
>
> I don't know if this will fix your DNS resolution, but at least it
> will clean up a bit :)
>
> J-D
>
> 2011/12/15 exp <ex...@163.com>:
> > hi all,
> >
> > I am installing hbase on a small cluster of 3 machines. The RegionServer
> unable connect to the master. This is the log:
> >
> >
> > 2011-12-15 13:46:43,415 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
> Master server at localhost:60000
> > 2011-12-15 13:47:43,473 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> master. Retrying. Error was:
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> > at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> > ......
> >
> >
> >
> >
> > The lod shows this RegionServer connect to the master at a wrong
> address(localhost:60000).
> >
> >
> > as far as I know, the RegionServer use ZooKeeper to find out the
> master's address. so I check the HMaster's log.
> >
> >
> > this is what i find out:
> > ...
> > 2011-12-15 13:44:34,946 INFO
> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> > 2011-12-15 13:44:35,010 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Master=localhost:60000
> > 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=master:2181,slave2:2181,slave
> > 1:2181 sessionTimeout=180000 watcher=hconnection
> > ...
> >
> >
> > The HMaster starts at localhost:60000! I think the HMaster should start
> at master:60000. "master" is the hosts where JobTracker runs.
> >
> >
> > I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source code
> where the HMaster get the address:
> >
> >
> > HServerAddress a = new HServerAddress(getMyAddress(this.conf));
> > int numHandlers = conf.getInt("hbase.regionserver.handler.count", 10);
> > this.rpcServer = HBaseRPC.getServer(this,
> > new Class<?>[]{HMasterInterface.class,
> HMasterRegionInterface.class},
> > a.getBindAddress(), a.getPort(),
> > numHandlers,
> > 0, // we dont use high priority handlers in master
> > false, conf,
> > 0); // this is a DNC w/o high priority handlers
> > this.address = new HServerAddress(rpcServer.getListenerAddress());
> >
> >
> > private static String getMyAddress(final Configuration c)
> > throws UnknownHostException {
> > // Find out our address up in DNS.
> > String s =
> DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"),
> > c.get("hbase.master.dns.nameserver","default"));
> > s += ":" + c.get(HConstants.MASTER_PORT,
> > Integer.toString(HConstants.DEFAULT_MASTER_PORT));
> > return s;
> > }
> >
> >
> >
> >
> > As you can see in the code, the HMaster find its address in DNS! I have
> no idea how to setup the hbase to let the HMaster find the correct address.
> >
> >
> > this is my hbase-site.xml:
> >
> >
> > <configuration>
> > <property>
> > <name>hbase.rootdir</name>
> > <value>hdfs://master:9000/user/hadoop/hbase</value>
> > </property>
> > <property>
> > <name>hbase.cluster.distributed</name>
> > <value>true</value>
> > </property>
> >
> >
> > <property>
> > <name>hbase.master.dns.interface</name>
> > <value>master</value>
> > </property>
> >
> >
> > <property>
> > <name>zookeeper.znode.master</name>
> > <value>master</value>
> > </property>
> >
> >
> > <property>
> > <name>hbase.master</name>
> > <value>master:60000</value>
> > </property>
> > </configuration>
> >
> >
> > and all the three machines are virtual machine with Ubuntu 11.04
> >
> >
> > this is the /etc/hosts :
> > master:
> > 127.0.0.1 localhost ubt
> > 127.0.0.1 ubt ubuntu
> > 10.66.201.243 master
> > 10.66.201.244 slave1
> > 10.66.201.245 slave2
> >
> >
> > slave1:
> > 127.0.0.1 localhost
> > 127.0.1.1 slave1 ubuntu
> > 10.66.201.243 master
> > 10.66.201.244 slave1
> > 10.66.201.245 slave2
> >
> >
> > slave2:
> > 127.0.0.1 localhost
> > 127.0.1.1 slave2 ubuntu
> > 10.66.201.243 master
> > 10.66.201.244 slave1
> > 10.66.201.245 slave2
> >
> >
> > any idea on this problems?
> >
> >
> >
> >
> >
> >
> > cheers
> >
> >
> > exception
>
--
Shashwat Shriparv
09900059620
09663531241
<iframe src="
http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr"
width="728" height="90" scrolling="no" border="0" marginwidth="0"
style="border:none;" frameborder="0"></iframe>