You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Cosmin Cătălin Sanda <co...@gmail.com> on 2014/07/09 13:34:40 UTC
HBase Master cannot be contacted by Region Servers
I have a Hadoop cluster made of 3 slaves and 1 master on top of which there
is an HBase cluster with 3 RS and 1 master respectively. Additionally there
is a Zookeeper ensemble on 3 machines.
The Hadoop cluster is functioning correctly as well as the Zookeeper
ensemble. However, the HBase cluster fails to initialize correctly.
I start HBase it by running ./bin/start-hbase.sh. This *correctly* starts
the HBase Master and the Region Servers. The hbase folder in hdfs is set-up
correctly.
*jps on master*
hduser@master:~/hbase$ jps
5694 HMaster
3934 JobHistoryServer
3786 NameNode
3873 ResourceManager
6025 Jps
*jps on slaves*
5737 Jps
5499 HRegionServer
3736 DataNode
3820 NodeManager
However, the HBase master does not register the Region Servers as it is
also apparent from looking at the logs:
*master log*
[master:master:60000] master.ServerManager: Waiting for region servers
count to settle; currently checked in 0, slept for 1511 ms, expecting
minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of
1500 ms.
*slave log*
[regionserver60020] regionserver.HRegionServer: reportForDuty to
master=master,60000,1404856451890 with port=60020,
startcode=1404856453874
[regionserver60020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException:
org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout
while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending
local=/10.0.2.15:53939 remote=master/192.168.66.60:60000]
Here are the configuration details:
*/etc/hosts on master*
192.168.66.63 slave-3 # Data Node and Region Server
192.168.66.60 master # Name Node and HBase Master
192.168.66.73 zookeeper-3 # Zookeeper node
192.168.66.71 zookeeper-1 # Zookeeper node
192.168.66.72 zookeeper-2 # Zookeeper node
192.168.66.62 slave-2 # Data Node and Region Server
192.168.66.61 slave-1 # Data Node and Region Server
*/etc/hosts on slave-1*
192.168.66.60 master
192.168.66.73 zookeeper-3
192.168.66.71 zookeeper-1
192.168.66.72 zookeeper-2
*hbase-site.xml on ALL cluster nodes*
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.tmp.dir</name>
<value>/home/hduser/hbase/tmp</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.local.dir</name>
<value>/home/hduser/hbase/local</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>6010</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>zookeeper-1,zookeeper-2,zookeeper-3,</value>
</property>
</configuration>
*regionservers file on master and slaves*
slave-3
slave-1
slave-2
*hbase-env.sh on master and slaves*
export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::"
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
export HBASE_MANAGES_ZK=false
What am I doing wrong so that the nodes cannot talk to each other?
I am using Hadoop 2.4.0 and Hbase 0.98.3 along with Zookeeper 3.4.6 on
Ubuntu Trusty Tahr x64.
Fwd: HBase Master cannot be contacted by Region Servers
Posted by Cosmin Cătălin Sanda <co...@gmail.com>.
I have a Hadoop cluster made of 3 slaves and 1 master on top of which
there is an HBase cluster with 3 RS and 1 master respectively.
Additionally there is a Zookeeper ensemble on 3 machines.
The Hadoop cluster is functioning correctly as well as the Zookeeper
ensemble. However, the HBase cluster fails to initialize correctly.
I start HBase it by running ./bin/start-hbase.sh. This correctly
starts the HBase Master and the Region Servers. The hbase folder in
hdfs is set-up correctly.
jps on master
hduser@master:~/hbase$ jps
5694 HMaster
3934 JobHistoryServer
3786 NameNode
3873 ResourceManager
6025 Jps
jps on slaves
5737 Jps
5499 HRegionServer
3736 DataNode
3820 NodeManager
However, the HBase master does not register the Region Servers as it
is also apparent from looking at the logs:
master log
[master:master:60000] master.ServerManager: Waiting for region servers
count to settle; currently checked in 0, slept for 1511 ms, expecting
minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of
1500 ms.
slave log
[regionserver60020] regionserver.HRegionServer: reportForDuty to
master=master,60000,1404856451890 with port=60020,
startcode=1404856453874
[regionserver60020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException:
org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout
while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending
local=/10.0.2.15:53939 remote=master/192.168.66.60:60000]
Here are the configuration details:
/etc/hosts on master
192.168.66.63 slave-3 # Data Node and Region Server
192.168.66.60 master # Name Node and HBase Master
192.168.66.73 zookeeper-3 # Zookeeper node
192.168.66.71 zookeeper-1 # Zookeeper node
192.168.66.72 zookeeper-2 # Zookeeper node
192.168.66.62 slave-2 # Data Node and Region Server
192.168.66.61 slave-1 # Data Node and Region Server
/etc/hosts on slave-1
192.168.66.60 master
192.168.66.73 zookeeper-3
192.168.66.71 zookeeper-1
192.168.66.72 zookeeper-2
hbase-site.xml on ALL cluster nodes
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.tmp.dir</name>
<value>/home/hduser/hbase/tmp</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.local.dir</name>
<value>/home/hduser/hbase/local</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>6010</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>zookeeper-1,zookeeper-2,zookeeper-3,</value>
</property>
</configuration>
regionservers file on master and slaves
slave-3
slave-1
slave-2
hbase-env.sh on master and slaves
export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::"
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
export HBASE_MANAGES_ZK=false
What am I doing wrong so that the nodes cannot talk to each other?
I am using Hadoop 2.4.0 and Hbase 0.98.3 along with Zookeeper 3.4.6 on
Ubuntu Trusty Tahr x64.