You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Bo Fu <bo...@uchicago.edu> on 2015/04/26 20:08:29 UTC
I can't start cluster due to zookeeper
Hi all,
I have problem starting a cluster of 1 master and 3 region server. When I started the cluster, the HMaster and HRegionserver will automatically exit.
My abase-site.xml:
<configuration>
<property>
<name>hbase.master</name>
<value>hadoopmaster:60000</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoopmaster:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/proj/ucare/bo/hadoop_data/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>pc225.emulab.net<http://pc225.emulab.net>,pc273.emulab.net<http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net></value>
<description>The directory shared by RegionServers.
</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
Log file:
2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to authenticate using SASL (unknown error)
2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection established to pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181>, initiating session
2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to authenticate using SASL (unknown error)
2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection established to pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181>, initiating session
2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc332.emulab.net/155.98.39.132:2181<http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to authenticate using SASL (unknown error)
2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 30025ms for sessionid 0x0, closing socket connection and attempting reconnect
2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=pc273.emulab.net<http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net>:2181,pc332.emulab.net<http://pc332.emulab.net>:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Can someone helps me figure out the problem? Is that because the number of region server and zookeeper nodes are incorrect?
Thanks a lot!
Bo
Re: I can't start cluster due to zookeeper
Posted by Ted Yu <yu...@gmail.com>.
Can you create maven build for your project ?
In the maven project you can add hbase dependencies.
See hbase-testing-util/pom.xml for example.
Suggest starting a new thread if you have question(s) on build.
On Mon, Apr 27, 2015 at 9:47 AM, Bo Fu <bo...@uchicago.edu> wrote:
> Hi Ted,
>
> Thanks for reminding. This problem has been solved. Now I’m facing another
> problem:
> When I compile Java class using
> javac -cp ‘bin/hbase classpath’ TestHBase.java
>
> Error appears:
> BF@node-3:/proj/ucare/bo/hbase-1.0.0 % javac -cp 'bin/hbase classpath'
> TestHBase.java
> TestHBase.java:1: error: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> TestHBase.java:2: error: package org.apache.hadoop.hbase does not exist
> import org.apache.hadoop.hbase.*;
> ^
> TestHBase.java:3: error: package org.apache.hadoop.hbase.client does not
> exist
> import org.apache.hadoop.hbase.client.*;
> ^
> TestHBase.java:4: error: package org.apache.hadoop.hbase.util does not
> exist
> import org.apache.hadoop.hbase.util.*;
> ^
> TestHBase.java:7: error: cannot find symbol
> Configuration conf = HBaseConfiguration.create();
> ^
> symbol: class Configuration
> location: class TestHBase
> TestHBase.java:7: error: cannot find symbol
> Configuration conf = HBaseConfiguration.create();
> ^
> symbol: variable HBaseConfiguration
> location: class TestHBase
> TestHBase.java:8: error: cannot find symbol
> HBaseAdmin admin = new HBaseAdmin(conf);
> ^
> symbol: class HBaseAdmin
> location: class TestHBase
> TestHBase.java:8: error: cannot find symbol
> HBaseAdmin admin = new HBaseAdmin(conf);
> ^
> symbol: class HBaseAdmin
> location: class TestHBase
> TestHBase.java:10: error: cannot find symbol
> HTable table = new HTable(conf, "test-table");
> ^
> symbol: class HTable
> location: class TestHBase
> TestHBase.java:10: error: cannot find symbol
> HTable table = new HTable(conf, "test-table");
> ^
> symbol: class HTable
> location: class TestHBase
> TestHBase.java:11: error: cannot find symbol
>
> But I have use ‘bin/hbase classpath’ to refer to jars. Do you know what’s
> the problem?
>
> Thx so much
>
> Bo
>
>
> On Apr 27, 2015, at 9:57 AM, Ted Yu <yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com>> wrote:
>
> Can you do 'ls /hbase' in zkcli and pastebin the output ?
>
> Cheers
>
> On Mon, Apr 27, 2015 at 7:45 AM, Bo Fu <bof@uchicago.edu<mailto:
> bof@uchicago.edu>> wrote:
>
> Hi Ted,
>
> I used bin/hbase zkcli to check, and the result was:
> Connecting to pc277.emulab.net<http://pc277.emulab.net><
> http://pc277.emulab.net>:2181,
> pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net>:2181,
> pc312.emulab.net<http://pc312.emulab.net><
> http://pc312.emulab.net>:2181
> 2015-04-27 08:42:30,523 INFO [main] zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
> environment:host.name=pc322.emulab.net<http://pc322.emulab.net><
> http://pc322.emulab.net>
> 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.version=1.7.0_79
> 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
> 2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client
>
> environment:java.class.path=/proj/ucare/bo/hbase-1.0.0/bin/../conf:/usr/lib/jvm/java-7-openjdk-amd64//lib/tools.jar:/proj/ucare/bo/hbase-1.0.0/bin/..:/proj/ucare/bo/hbase-1.0.0/bin/../lib/activation-1.1.jar:/pro…………..
> 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.version=3.2.0-56-generic
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.name=BF
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.home=/users/BF
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.dir=/proj/ucare/bo/hbase-1.0.0
> 2015-04-27 08:42:30,532 INFO [main] zookeeper.ZooKeeper: Initiating
> client connection, connectString=pc277.emulab.net<http://pc277.emulab.net
> ><http://pc277.emulab.net
> :2181,pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net
> >:2181,pc312.emulab.net<http://pc312.emulab.net><
> http://pc312.emulab.net>:2181 sessionTimeout=30000
> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@920d014
> Welcome to ZooKeeper!
> 2015-04-27 08:42:30,694 INFO [main-SendThread(pc277.emulab.net<
> http://pc277.emulab.net><
> http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc277.emulab.net/155.98.39.77:2181<
> http://pc277.emulab.net/155.98.39.77:2181><
> http://pc277.emulab.net/155.98.39.77:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-27 08:42:30,718 INFO [main-SendThread(pc277.emulab.net<
> http://pc277.emulab.net><
> http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc277.emulab.net/155.98.39.77:2181<
> http://pc277.emulab.net/155.98.39.77:2181><
> http://pc277.emulab.net/155.98.39.77:2181>, initiating session
> JLine support is enabled
> 2015-04-27 08:42:30,760 INFO [main-SendThread(pc277.emulab.net<
> http://pc277.emulab.net><
> http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Session
> establishment complete on server pc277.emulab.net/155.98.39.77:2181<
> http://pc277.emulab.net/155.98.39.77:2181><
> http://pc277.emulab.net/155.98.39.77:2181>, sessionid =
> 0x14cfb53f8de0000, negotiated timeout = 30000
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zk: pc277.emulab.net<http://pc277.emulab.net><http://pc277.emulab.net
> >:2181,pc273.emulab.net<http://pc273.emulab.net><
> http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net><
> http://pc312.emulab.net>:2181(CONNECTED)
> 0]
>
>
> It seems that zookeeper is running correctly?
>
> Bo
>
>
> On Apr 27, 2015, at 9:19 AM, Ted Yu <yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com><mailto:
> yuzhihong@gmail.com<ma...@gmail.com>>> wrote:
>
> zkCli.sh is provided by zookeeper.
>
> See http://hbase.apache.org/book.html#quickstart for standalone HBase
> deployment.
>
> Cheers
>
> On Mon, Apr 27, 2015 at 6:53 AM, Bo Fu <bof@uchicago.edu<mailto:
> bof@uchicago.edu><mailto:
> bof@uchicago.edu<ma...@uchicago.edu>>> wrote:
>
> Hi Ted,
>
> I can’t find zkCli.sh. I’m using the zookeeper provided by HBase.
>
> So bad. Should I use my own zookeeper?
>
> Bo
>
>
> On Apr 26, 2015, at 3:23 PM, Ted Yu <yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com><mailto:
> yuzhihong@gmail.com<ma...@gmail.com>>> wrote:
>
> Can you use zkCli.sh to check the contents under /hbase ?
>
> Please check zookeeper logs to see if quorum was running fine.
>
> bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net><
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)]
>
> Please remove the '<http://...>' from hbase.zookeeper.quorum config.
>
> Cheers
>
> On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <bof@uchicago.edu<mailto:
> bof@uchicago.edu><mailto:
> bof@uchicago.edu<ma...@uchicago.edu>>> wrote:
>
> Hi all,
>
> I have problem starting a cluster of 1 master and 3 region server. When
> I
> started the cluster, the HMaster and HRegionserver will automatically
> exit.
>
> My abase-site.xml:
> <configuration>
> <property>
> <name>hbase.master</name>
> <value>hadoopmaster:60000</value>
> </property>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://hadoopmaster:9000/hbase</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/proj/ucare/bo/hadoop_data/zookeeper</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>pc225.emulab.net<http://pc225.emulab.net><http://pc225.emulab.net
> ><http://pc225.emulab.net
> ,pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net><
> http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net><
> http://pc210.emulab.net><
> http://pc210.emulab.net
> </value>
> <description>The directory shared by RegionServers.
> </description>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
>
> Log file:
>
> 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc273.emulab.net/155.98.39.73:2181<
> http://pc273.emulab.net/155.98.39.73:2181><
> http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc273.emulab.net/155.98.39.73:2181<
> http://pc273.emulab.net/155.98.39.73:2181><
> http://pc273.emulab.net/155.98.39.73:2181>, initiating session
> 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> additional data from server sessionid 0x0, likely server has closed
> socket,
> closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net><
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc225.emulab.net/155.98.39.25:2181<
> http://pc225.emulab.net/155.98.39.25:2181><
> http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net><
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc225.emulab.net/155.98.39.25:2181<
> http://pc225.emulab.net/155.98.39.25:2181><
> http://pc225.emulab.net/155.98.39.25:2181>, initiating session
> 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net><
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> additional data from server sessionid 0x0, likely server has closed
> socket,
> closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<
> http://pc332.emulab.net><
> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc332.emulab.net/155.98.39.132:2181<
> http://pc332.emulab.net/155.98.39.132:2181><
> http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<
> http://pc332.emulab.net><
> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session
> timed out, have not heard from server in 30025ms for sessionid 0x0,
> closing
> socket connection and attempting reconnect
> 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper:
> Possibly transient ZooKeeper, quorum=pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net><
> http://pc225.emulab.net
> :2181,pc332.emulab.net<http://pc332.emulab.net><http://pc332.emulab.net
> :2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper:
> ZooKeeper create failed after 4 attempts
> 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master
> exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
> at
>
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
> at
>
>
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
> at
>
>
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
>
>
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
>
>
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
> at
>
>
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
> at
>
>
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
> at
>
>
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
> at
>
>
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
> at
>
>
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
> at
>
>
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
> at
> org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
>
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
>
>
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>
> Can someone helps me figure out the problem? Is that because the number
> of
> region server and zookeeper nodes are incorrect?
>
> Thanks a lot!
>
> Bo
>
>
>
>
>
>
>
>
>
Re: I can't start cluster due to zookeeper
Posted by Bo Fu <bo...@uchicago.edu>.
Hi Ted,
Thanks for reminding. This problem has been solved. Now I’m facing another problem:
When I compile Java class using
javac -cp ‘bin/hbase classpath’ TestHBase.java
Error appears:
BF@node-3:/proj/ucare/bo/hbase-1.0.0 % javac -cp 'bin/hbase classpath' TestHBase.java
TestHBase.java:1: error: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
TestHBase.java:2: error: package org.apache.hadoop.hbase does not exist
import org.apache.hadoop.hbase.*;
^
TestHBase.java:3: error: package org.apache.hadoop.hbase.client does not exist
import org.apache.hadoop.hbase.client.*;
^
TestHBase.java:4: error: package org.apache.hadoop.hbase.util does not exist
import org.apache.hadoop.hbase.util.*;
^
TestHBase.java:7: error: cannot find symbol
Configuration conf = HBaseConfiguration.create();
^
symbol: class Configuration
location: class TestHBase
TestHBase.java:7: error: cannot find symbol
Configuration conf = HBaseConfiguration.create();
^
symbol: variable HBaseConfiguration
location: class TestHBase
TestHBase.java:8: error: cannot find symbol
HBaseAdmin admin = new HBaseAdmin(conf);
^
symbol: class HBaseAdmin
location: class TestHBase
TestHBase.java:8: error: cannot find symbol
HBaseAdmin admin = new HBaseAdmin(conf);
^
symbol: class HBaseAdmin
location: class TestHBase
TestHBase.java:10: error: cannot find symbol
HTable table = new HTable(conf, "test-table");
^
symbol: class HTable
location: class TestHBase
TestHBase.java:10: error: cannot find symbol
HTable table = new HTable(conf, "test-table");
^
symbol: class HTable
location: class TestHBase
TestHBase.java:11: error: cannot find symbol
But I have use ‘bin/hbase classpath’ to refer to jars. Do you know what’s the problem?
Thx so much
Bo
On Apr 27, 2015, at 9:57 AM, Ted Yu <yu...@gmail.com>> wrote:
Can you do 'ls /hbase' in zkcli and pastebin the output ?
Cheers
On Mon, Apr 27, 2015 at 7:45 AM, Bo Fu <bo...@uchicago.edu>> wrote:
Hi Ted,
I used bin/hbase zkcli to check, and the result was:
Connecting to pc277.emulab.net<http://pc277.emulab.net><http://pc277.emulab.net>:2181,
pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net><
http://pc312.emulab.net>:2181
2015-04-27 08:42:30,523 INFO [main] zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
environment:host.name=pc322.emulab.net<http://pc322.emulab.net><http://pc322.emulab.net>
2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_79
2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client
environment:java.class.path=/proj/ucare/bo/hbase-1.0.0/bin/../conf:/usr/lib/jvm/java-7-openjdk-amd64//lib/tools.jar:/proj/ucare/bo/hbase-1.0.0/bin/..:/proj/ucare/bo/hbase-1.0.0/bin/../lib/activation-1.1.jar:/pro…………..
2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=3.2.0-56-generic
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
environment:user.name=BF
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=/users/BF
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
environment:user.dir=/proj/ucare/bo/hbase-1.0.0
2015-04-27 08:42:30,532 INFO [main] zookeeper.ZooKeeper: Initiating
client connection, connectString=pc277.emulab.net<http://pc277.emulab.net><http://pc277.emulab.net
:2181,pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net><
http://pc312.emulab.net>:2181 sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@920d014
Welcome to ZooKeeper!
2015-04-27 08:42:30,694 INFO [main-SendThread(pc277.emulab.net<http://pc277.emulab.net><
http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
connection to server pc277.emulab.net/155.98.39.77:2181<http://pc277.emulab.net/155.98.39.77:2181><
http://pc277.emulab.net/155.98.39.77:2181>. Will not attempt to
authenticate using SASL (unknown error)
2015-04-27 08:42:30,718 INFO [main-SendThread(pc277.emulab.net<http://pc277.emulab.net><
http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
established to pc277.emulab.net/155.98.39.77:2181<http://pc277.emulab.net/155.98.39.77:2181><
http://pc277.emulab.net/155.98.39.77:2181>, initiating session
JLine support is enabled
2015-04-27 08:42:30,760 INFO [main-SendThread(pc277.emulab.net<http://pc277.emulab.net><
http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Session
establishment complete on server pc277.emulab.net/155.98.39.77:2181<http://pc277.emulab.net/155.98.39.77:2181><
http://pc277.emulab.net/155.98.39.77:2181>, sessionid =
0x14cfb53f8de0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: pc277.emulab.net<http://pc277.emulab.net><http://pc277.emulab.net>:2181,pc273.emulab.net<http://pc273.emulab.net><
http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net><http://pc312.emulab.net>:2181(CONNECTED)
0]
It seems that zookeeper is running correctly?
Bo
On Apr 27, 2015, at 9:19 AM, Ted Yu <yu...@gmail.com><mailto:
yuzhihong@gmail.com<ma...@gmail.com>>> wrote:
zkCli.sh is provided by zookeeper.
See http://hbase.apache.org/book.html#quickstart for standalone HBase
deployment.
Cheers
On Mon, Apr 27, 2015 at 6:53 AM, Bo Fu <bo...@uchicago.edu><mailto:
bof@uchicago.edu<ma...@uchicago.edu>>> wrote:
Hi Ted,
I can’t find zkCli.sh. I’m using the zookeeper provided by HBase.
So bad. Should I use my own zookeeper?
Bo
On Apr 26, 2015, at 3:23 PM, Ted Yu <yu...@gmail.com><mailto:
yuzhihong@gmail.com<ma...@gmail.com>>> wrote:
Can you use zkCli.sh to check the contents under /hbase ?
Please check zookeeper logs to see if quorum was running fine.
bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net><
http://pc273.emulab.net>:2181)]
Please remove the '<http://...>' from hbase.zookeeper.quorum config.
Cheers
On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <bo...@uchicago.edu><mailto:
bof@uchicago.edu<ma...@uchicago.edu>>> wrote:
Hi all,
I have problem starting a cluster of 1 master and 3 region server. When
I
started the cluster, the HMaster and HRegionserver will automatically
exit.
My abase-site.xml:
<configuration>
<property>
<name>hbase.master</name>
<value>hadoopmaster:60000</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoopmaster:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/proj/ucare/bo/hadoop_data/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>pc225.emulab.net<http://pc225.emulab.net><http://pc225.emulab.net><http://pc225.emulab.net
,pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net><
http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net><http://pc210.emulab.net><
http://pc210.emulab.net
</value>
<description>The directory shared by RegionServers.
</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
Log file:
2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<
http://pc273.emulab.net><
http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
connection to server pc273.emulab.net/155.98.39.73:2181<
http://pc273.emulab.net/155.98.39.73:2181><
http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to
authenticate using SASL (unknown error)
2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<
http://pc273.emulab.net><
http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
established to pc273.emulab.net/155.98.39.73:2181<
http://pc273.emulab.net/155.98.39.73:2181><
http://pc273.emulab.net/155.98.39.73:2181>, initiating session
2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<
http://pc273.emulab.net><
http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
additional data from server sessionid 0x0, likely server has closed
socket,
closing socket connection and attempting reconnect
2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<
http://pc225.emulab.net><
http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
connection to server pc225.emulab.net/155.98.39.25:2181<
http://pc225.emulab.net/155.98.39.25:2181><
http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to
authenticate using SASL (unknown error)
2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<
http://pc225.emulab.net><
http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
established to pc225.emulab.net/155.98.39.25:2181<
http://pc225.emulab.net/155.98.39.25:2181><
http://pc225.emulab.net/155.98.39.25:2181>, initiating session
2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<
http://pc225.emulab.net><
http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
additional data from server sessionid 0x0, likely server has closed
socket,
closing socket connection and attempting reconnect
2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<
http://pc332.emulab.net><
http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
connection to server pc332.emulab.net/155.98.39.132:2181<
http://pc332.emulab.net/155.98.39.132:2181><
http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to
authenticate using SASL (unknown error)
2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<
http://pc332.emulab.net><
http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session
timed out, have not heard from server in 30025ms for sessionid 0x0,
closing
socket connection and attempting reconnect
2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper:
Possibly transient ZooKeeper, quorum=pc273.emulab.net<
http://pc273.emulab.net><
http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net><
http://pc225.emulab.net
:2181,pc332.emulab.net<http://pc332.emulab.net><http://pc332.emulab.net
:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper:
ZooKeeper create failed after 4 attempts
2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master
exiting
java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMaster
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
at
org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Can someone helps me figure out the problem? Is that because the number
of
region server and zookeeper nodes are incorrect?
Thanks a lot!
Bo
Re: I can't start cluster due to zookeeper
Posted by Ted Yu <yu...@gmail.com>.
Can you do 'ls /hbase' in zkcli and pastebin the output ?
Cheers
On Mon, Apr 27, 2015 at 7:45 AM, Bo Fu <bo...@uchicago.edu> wrote:
> Hi Ted,
>
> I used bin/hbase zkcli to check, and the result was:
> Connecting to pc277.emulab.net<http://pc277.emulab.net>:2181,
> pc273.emulab.net<http://pc273.emulab.net>:2181,pc312.emulab.net<
> http://pc312.emulab.net>:2181
> 2015-04-27 08:42:30,523 INFO [main] zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
> environment:host.name=pc322.emulab.net<http://pc322.emulab.net>
> 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.version=1.7.0_79
> 2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
> 2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.class.path=/proj/ucare/bo/hbase-1.0.0/bin/../conf:/usr/lib/jvm/java-7-openjdk-amd64//lib/tools.jar:/proj/ucare/bo/hbase-1.0.0/bin/..:/proj/ucare/bo/hbase-1.0.0/bin/../lib/activation-1.1.jar:/pro…………..
> 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.version=3.2.0-56-generic
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.name=BF
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.home=/users/BF
> 2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.dir=/proj/ucare/bo/hbase-1.0.0
> 2015-04-27 08:42:30,532 INFO [main] zookeeper.ZooKeeper: Initiating
> client connection, connectString=pc277.emulab.net<http://pc277.emulab.net
> >:2181,pc273.emulab.net<http://pc273.emulab.net>:2181,pc312.emulab.net<
> http://pc312.emulab.net>:2181 sessionTimeout=30000
> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@920d014
> Welcome to ZooKeeper!
> 2015-04-27 08:42:30,694 INFO [main-SendThread(pc277.emulab.net<
> http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc277.emulab.net/155.98.39.77:2181<
> http://pc277.emulab.net/155.98.39.77:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-27 08:42:30,718 INFO [main-SendThread(pc277.emulab.net<
> http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc277.emulab.net/155.98.39.77:2181<
> http://pc277.emulab.net/155.98.39.77:2181>, initiating session
> JLine support is enabled
> 2015-04-27 08:42:30,760 INFO [main-SendThread(pc277.emulab.net<
> http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Session
> establishment complete on server pc277.emulab.net/155.98.39.77:2181<
> http://pc277.emulab.net/155.98.39.77:2181>, sessionid =
> 0x14cfb53f8de0000, negotiated timeout = 30000
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zk: pc277.emulab.net<http://pc277.emulab.net>:2181,pc273.emulab.net<
> http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net>:2181(CONNECTED)
> 0]
>
>
> It seems that zookeeper is running correctly?
>
> Bo
>
>
> On Apr 27, 2015, at 9:19 AM, Ted Yu <yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com>> wrote:
>
> zkCli.sh is provided by zookeeper.
>
> See http://hbase.apache.org/book.html#quickstart for standalone HBase
> deployment.
>
> Cheers
>
> On Mon, Apr 27, 2015 at 6:53 AM, Bo Fu <bof@uchicago.edu<mailto:
> bof@uchicago.edu>> wrote:
>
> Hi Ted,
>
> I can’t find zkCli.sh. I’m using the zookeeper provided by HBase.
>
> So bad. Should I use my own zookeeper?
>
> Bo
>
>
> On Apr 26, 2015, at 3:23 PM, Ted Yu <yuzhihong@gmail.com<mailto:
> yuzhihong@gmail.com>> wrote:
>
> Can you use zkCli.sh to check the contents under /hbase ?
>
> Please check zookeeper logs to see if quorum was running fine.
>
> bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)]
>
> Please remove the '<http://...>' from hbase.zookeeper.quorum config.
>
> Cheers
>
> On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <bof@uchicago.edu<mailto:
> bof@uchicago.edu>> wrote:
>
> Hi all,
>
> I have problem starting a cluster of 1 master and 3 region server. When
> I
> started the cluster, the HMaster and HRegionserver will automatically
> exit.
>
> My abase-site.xml:
> <configuration>
> <property>
> <name>hbase.master</name>
> <value>hadoopmaster:60000</value>
> </property>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://hadoopmaster:9000/hbase</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/proj/ucare/bo/hadoop_data/zookeeper</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>pc225.emulab.net<http://pc225.emulab.net><http://pc225.emulab.net
> >,pc273.emulab.net<http://pc273.emulab.net><
> http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net><
> http://pc210.emulab.net
> </value>
> <description>The directory shared by RegionServers.
> </description>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
>
> Log file:
>
> 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc273.emulab.net/155.98.39.73:2181<
> http://pc273.emulab.net/155.98.39.73:2181><
> http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc273.emulab.net/155.98.39.73:2181<
> http://pc273.emulab.net/155.98.39.73:2181><
> http://pc273.emulab.net/155.98.39.73:2181>, initiating session
> 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> additional data from server sessionid 0x0, likely server has closed
> socket,
> closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net><
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc225.emulab.net/155.98.39.25:2181<
> http://pc225.emulab.net/155.98.39.25:2181><
> http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net><
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc225.emulab.net/155.98.39.25:2181<
> http://pc225.emulab.net/155.98.39.25:2181><
> http://pc225.emulab.net/155.98.39.25:2181>, initiating session
> 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net><
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> additional data from server sessionid 0x0, likely server has closed
> socket,
> closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<
> http://pc332.emulab.net><
> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc332.emulab.net/155.98.39.132:2181<
> http://pc332.emulab.net/155.98.39.132:2181><
> http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<
> http://pc332.emulab.net><
> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session
> timed out, have not heard from server in 30025ms for sessionid 0x0,
> closing
> socket connection and attempting reconnect
> 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper:
> Possibly transient ZooKeeper, quorum=pc273.emulab.net<
> http://pc273.emulab.net><
> http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net><
> http://pc225.emulab.net
> :2181,pc332.emulab.net<http://pc332.emulab.net><http://pc332.emulab.net
> >:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper:
> ZooKeeper create failed after 4 attempts
> 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master
> exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
> at
>
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
> at
>
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
> at
>
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
>
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
>
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
> at
>
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
> at
>
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
> at
>
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
> at
>
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
> at
>
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
> at
>
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
> at
> org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
>
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>
> Can someone helps me figure out the problem? Is that because the number
> of
> region server and zookeeper nodes are incorrect?
>
> Thanks a lot!
>
> Bo
>
>
>
>
>
>
>
Re: I can't start cluster due to zookeeper
Posted by Bo Fu <bo...@uchicago.edu>.
Hi Ted,
I used bin/hbase zkcli to check, and the result was:
Connecting to pc277.emulab.net<http://pc277.emulab.net>:2181,pc273.emulab.net<http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net>:2181
2015-04-27 08:42:30,523 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=pc322.emulab.net<http://pc322.emulab.net>
2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_79
2015-04-27 08:42:30,525 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-04-27 08:42:30,526 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/proj/ucare/bo/hbase-1.0.0/bin/../conf:/usr/lib/jvm/java-7-openjdk-amd64//lib/tools.jar:/proj/ucare/bo/hbase-1.0.0/bin/..:/proj/ucare/bo/hbase-1.0.0/bin/../lib/activation-1.1.jar:/pro…………..
2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2015-04-27 08:42:30,527 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.2.0-56-generic
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=BF
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/users/BF
2015-04-27 08:42:30,528 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/proj/ucare/bo/hbase-1.0.0
2015-04-27 08:42:30,532 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=pc277.emulab.net<http://pc277.emulab.net>:2181,pc273.emulab.net<http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net>:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@920d014
Welcome to ZooKeeper!
2015-04-27 08:42:30,694 INFO [main-SendThread(pc277.emulab.net<http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc277.emulab.net/155.98.39.77:2181<http://pc277.emulab.net/155.98.39.77:2181>. Will not attempt to authenticate using SASL (unknown error)
2015-04-27 08:42:30,718 INFO [main-SendThread(pc277.emulab.net<http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection established to pc277.emulab.net/155.98.39.77:2181<http://pc277.emulab.net/155.98.39.77:2181>, initiating session
JLine support is enabled
2015-04-27 08:42:30,760 INFO [main-SendThread(pc277.emulab.net<http://pc277.emulab.net>:2181)] zookeeper.ClientCnxn: Session establishment complete on server pc277.emulab.net/155.98.39.77:2181<http://pc277.emulab.net/155.98.39.77:2181>, sessionid = 0x14cfb53f8de0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: pc277.emulab.net<http://pc277.emulab.net>:2181,pc273.emulab.net<http://pc273.emulab.net>:2181,pc312.emulab.net<http://pc312.emulab.net>:2181(CONNECTED) 0]
It seems that zookeeper is running correctly?
Bo
On Apr 27, 2015, at 9:19 AM, Ted Yu <yu...@gmail.com>> wrote:
zkCli.sh is provided by zookeeper.
See http://hbase.apache.org/book.html#quickstart for standalone HBase
deployment.
Cheers
On Mon, Apr 27, 2015 at 6:53 AM, Bo Fu <bo...@uchicago.edu>> wrote:
Hi Ted,
I can’t find zkCli.sh. I’m using the zookeeper provided by HBase.
So bad. Should I use my own zookeeper?
Bo
On Apr 26, 2015, at 3:23 PM, Ted Yu <yu...@gmail.com>> wrote:
Can you use zkCli.sh to check the contents under /hbase ?
Please check zookeeper logs to see if quorum was running fine.
bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net><http://pc273.emulab.net>:2181)]
Please remove the '<http://...>' from hbase.zookeeper.quorum config.
Cheers
On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <bo...@uchicago.edu>> wrote:
Hi all,
I have problem starting a cluster of 1 master and 3 region server. When
I
started the cluster, the HMaster and HRegionserver will automatically
exit.
My abase-site.xml:
<configuration>
<property>
<name>hbase.master</name>
<value>hadoopmaster:60000</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoopmaster:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/proj/ucare/bo/hadoop_data/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>pc225.emulab.net<http://pc225.emulab.net><http://pc225.emulab.net>,pc273.emulab.net<http://pc273.emulab.net><
http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net><http://pc210.emulab.net
</value>
<description>The directory shared by RegionServers.
</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
Log file:
2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net><
http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
connection to server pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181><
http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to
authenticate using SASL (unknown error)
2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net><
http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
established to pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181><
http://pc273.emulab.net/155.98.39.73:2181>, initiating session
2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net><
http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
additional data from server sessionid 0x0, likely server has closed
socket,
closing socket connection and attempting reconnect
2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net><
http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
connection to server pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181><
http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to
authenticate using SASL (unknown error)
2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net><
http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
established to pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181><
http://pc225.emulab.net/155.98.39.25:2181>, initiating session
2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net><
http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
additional data from server sessionid 0x0, likely server has closed
socket,
closing socket connection and attempting reconnect
2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net><
http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
connection to server pc332.emulab.net/155.98.39.132:2181<http://pc332.emulab.net/155.98.39.132:2181><
http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to
authenticate using SASL (unknown error)
2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net><
http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session
timed out, have not heard from server in 30025ms for sessionid 0x0,
closing
socket connection and attempting reconnect
2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper:
Possibly transient ZooKeeper, quorum=pc273.emulab.net<http://pc273.emulab.net><
http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net><http://pc225.emulab.net
:2181,pc332.emulab.net<http://pc332.emulab.net><http://pc332.emulab.net>:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper:
ZooKeeper create failed after 4 attempts
2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master
exiting
java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMaster
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
at
org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Can someone helps me figure out the problem? Is that because the number
of
region server and zookeeper nodes are incorrect?
Thanks a lot!
Bo
Re: I can't start cluster due to zookeeper
Posted by Ted Yu <yu...@gmail.com>.
zkCli.sh is provided by zookeeper.
See http://hbase.apache.org/book.html#quickstart for standalone HBase
deployment.
Cheers
On Mon, Apr 27, 2015 at 6:53 AM, Bo Fu <bo...@uchicago.edu> wrote:
> Hi Ted,
>
> I can’t find zkCli.sh. I’m using the zookeeper provided by HBase.
>
> So bad. Should I use my own zookeeper?
>
> Bo
>
>
> > On Apr 26, 2015, at 3:23 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > Can you use zkCli.sh to check the contents under /hbase ?
> >
> > Please check zookeeper logs to see if quorum was running fine.
> >
> > bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)]
> >
> > Please remove the '<http://...>' from hbase.zookeeper.quorum config.
> >
> > Cheers
> >
> > On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <bo...@uchicago.edu> wrote:
> >
> >> Hi all,
> >>
> >> I have problem starting a cluster of 1 master and 3 region server. When
> I
> >> started the cluster, the HMaster and HRegionserver will automatically
> exit.
> >>
> >> My abase-site.xml:
> >> <configuration>
> >> <property>
> >> <name>hbase.master</name>
> >> <value>hadoopmaster:60000</value>
> >> </property>
> >> <property>
> >> <name>hbase.rootdir</name>
> >> <value>hdfs://hadoopmaster:9000/hbase</value>
> >> </property>
> >> <property>
> >> <name>hbase.zookeeper.property.dataDir</name>
> >> <value>/proj/ucare/bo/hadoop_data/zookeeper</value>
> >> </property>
> >> <property>
> >> <name>hbase.zookeeper.quorum</name>
> >> <value>pc225.emulab.net<http://pc225.emulab.net>,pc273.emulab.net<
> >> http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net
> ></value>
> >> <description>The directory shared by RegionServers.
> >> </description>
> >> </property>
> >> <property>
> >> <name>hbase.zookeeper.property.clientPort</name>
> >> <value>2181</value>
> >> </property>
> >> <property>
> >> <name>hbase.cluster.distributed</name>
> >> <value>true</value>
> >> </property>
> >> <property>
> >> <name>dfs.replication</name>
> >> <value>1</value>
> >> </property>
> >> </configuration>
> >>
> >>
> >> Log file:
> >>
> >> 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<
> >> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> >> connection to server pc273.emulab.net/155.98.39.73:2181<
> >> http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to
> >> authenticate using SASL (unknown error)
> >> 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<
> >> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> >> established to pc273.emulab.net/155.98.39.73:2181<
> >> http://pc273.emulab.net/155.98.39.73:2181>, initiating session
> >> 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<
> >> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> >> additional data from server sessionid 0x0, likely server has closed
> socket,
> >> closing socket connection and attempting reconnect
> >> 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<
> >> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> >> connection to server pc225.emulab.net/155.98.39.25:2181<
> >> http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to
> >> authenticate using SASL (unknown error)
> >> 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<
> >> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> >> established to pc225.emulab.net/155.98.39.25:2181<
> >> http://pc225.emulab.net/155.98.39.25:2181>, initiating session
> >> 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<
> >> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> >> additional data from server sessionid 0x0, likely server has closed
> socket,
> >> closing socket connection and attempting reconnect
> >> 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<
> >> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> >> connection to server pc332.emulab.net/155.98.39.132:2181<
> >> http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to
> >> authenticate using SASL (unknown error)
> >> 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<
> >> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session
> >> timed out, have not heard from server in 30025ms for sessionid 0x0,
> closing
> >> socket connection and attempting reconnect
> >> 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper:
> >> Possibly transient ZooKeeper, quorum=pc273.emulab.net<
> >> http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net
> >>> :2181,pc332.emulab.net<http://pc332.emulab.net>:2181,
> >> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss for /hbase
> >> 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper:
> >> ZooKeeper create failed after 4 attempts
> >> 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master
> >> exiting
> >> java.lang.RuntimeException: Failed construction of Master: class
> >> org.apache.hadoop.hbase.master.HMaster
> >> at
> >>
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
> >> at
> >>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
> >> at
> >>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> at
> >>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> >> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
> >> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss for /hbase
> >> at
> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >> at
> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> >> at
> >>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
> >> at
> >>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
> >> at
> >>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
> >> at
> >>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
> >> at
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
> >> at
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
> >> at
> org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >> Method)
> >> at
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >>
> >>
> >> Can someone helps me figure out the problem? Is that because the number
> of
> >> region server and zookeeper nodes are incorrect?
> >>
> >> Thanks a lot!
> >>
> >> Bo
> >>
> >>
> >>
>
>
Re: I can't start cluster due to zookeeper
Posted by Bo Fu <bo...@uchicago.edu>.
Hi Ted,
I can’t find zkCli.sh. I’m using the zookeeper provided by HBase.
So bad. Should I use my own zookeeper?
Bo
> On Apr 26, 2015, at 3:23 PM, Ted Yu <yu...@gmail.com> wrote:
>
> Can you use zkCli.sh to check the contents under /hbase ?
>
> Please check zookeeper logs to see if quorum was running fine.
>
> bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)]
>
> Please remove the '<http://...>' from hbase.zookeeper.quorum config.
>
> Cheers
>
> On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <bo...@uchicago.edu> wrote:
>
>> Hi all,
>>
>> I have problem starting a cluster of 1 master and 3 region server. When I
>> started the cluster, the HMaster and HRegionserver will automatically exit.
>>
>> My abase-site.xml:
>> <configuration>
>> <property>
>> <name>hbase.master</name>
>> <value>hadoopmaster:60000</value>
>> </property>
>> <property>
>> <name>hbase.rootdir</name>
>> <value>hdfs://hadoopmaster:9000/hbase</value>
>> </property>
>> <property>
>> <name>hbase.zookeeper.property.dataDir</name>
>> <value>/proj/ucare/bo/hadoop_data/zookeeper</value>
>> </property>
>> <property>
>> <name>hbase.zookeeper.quorum</name>
>> <value>pc225.emulab.net<http://pc225.emulab.net>,pc273.emulab.net<
>> http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net></value>
>> <description>The directory shared by RegionServers.
>> </description>
>> </property>
>> <property>
>> <name>hbase.zookeeper.property.clientPort</name>
>> <value>2181</value>
>> </property>
>> <property>
>> <name>hbase.cluster.distributed</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>> </configuration>
>>
>>
>> Log file:
>>
>> 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<
>> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
>> connection to server pc273.emulab.net/155.98.39.73:2181<
>> http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to
>> authenticate using SASL (unknown error)
>> 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<
>> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
>> established to pc273.emulab.net/155.98.39.73:2181<
>> http://pc273.emulab.net/155.98.39.73:2181>, initiating session
>> 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<
>> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
>> additional data from server sessionid 0x0, likely server has closed socket,
>> closing socket connection and attempting reconnect
>> 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<
>> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
>> connection to server pc225.emulab.net/155.98.39.25:2181<
>> http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to
>> authenticate using SASL (unknown error)
>> 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<
>> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
>> established to pc225.emulab.net/155.98.39.25:2181<
>> http://pc225.emulab.net/155.98.39.25:2181>, initiating session
>> 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<
>> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
>> additional data from server sessionid 0x0, likely server has closed socket,
>> closing socket connection and attempting reconnect
>> 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<
>> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
>> connection to server pc332.emulab.net/155.98.39.132:2181<
>> http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to
>> authenticate using SASL (unknown error)
>> 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<
>> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session
>> timed out, have not heard from server in 30025ms for sessionid 0x0, closing
>> socket connection and attempting reconnect
>> 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper:
>> Possibly transient ZooKeeper, quorum=pc273.emulab.net<
>> http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net
>>> :2181,pc332.emulab.net<http://pc332.emulab.net>:2181,
>> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase
>> 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper:
>> ZooKeeper create failed after 4 attempts
>> 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master
>> exiting
>> java.lang.RuntimeException: Failed construction of Master: class
>> org.apache.hadoop.hbase.master.HMaster
>> at
>> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
>> at
>> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
>> at
>> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at
>> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
>> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
>> at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>
>>
>> Can someone helps me figure out the problem? Is that because the number of
>> region server and zookeeper nodes are incorrect?
>>
>> Thanks a lot!
>>
>> Bo
>>
>>
>>
Re: I can't start cluster due to zookeeper
Posted by Ted Yu <yu...@gmail.com>.
Can you use zkCli.sh to check the contents under /hbase ?
Please check zookeeper logs to see if quorum was running fine.
bq. [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)]
Please remove the '<http://...>' from hbase.zookeeper.quorum config.
Cheers
On Sun, Apr 26, 2015 at 11:08 AM, Bo Fu <bo...@uchicago.edu> wrote:
> Hi all,
>
> I have problem starting a cluster of 1 master and 3 region server. When I
> started the cluster, the HMaster and HRegionserver will automatically exit.
>
> My abase-site.xml:
> <configuration>
> <property>
> <name>hbase.master</name>
> <value>hadoopmaster:60000</value>
> </property>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://hadoopmaster:9000/hbase</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/proj/ucare/bo/hadoop_data/zookeeper</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>pc225.emulab.net<http://pc225.emulab.net>,pc273.emulab.net<
> http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net></value>
> <description>The directory shared by RegionServers.
> </description>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
>
> Log file:
>
> 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc273.emulab.net/155.98.39.73:2181<
> http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc273.emulab.net/155.98.39.73:2181<
> http://pc273.emulab.net/155.98.39.73:2181>, initiating session
> 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<
> http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> additional data from server sessionid 0x0, likely server has closed socket,
> closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc225.emulab.net/155.98.39.25:2181<
> http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection
> established to pc225.emulab.net/155.98.39.25:2181<
> http://pc225.emulab.net/155.98.39.25:2181>, initiating session
> 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<
> http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read
> additional data from server sessionid 0x0, likely server has closed socket,
> closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<
> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket
> connection to server pc332.emulab.net/155.98.39.132:2181<
> http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<
> http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session
> timed out, have not heard from server in 30025ms for sessionid 0x0, closing
> socket connection and attempting reconnect
> 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper:
> Possibly transient ZooKeeper, quorum=pc273.emulab.net<
> http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net
> >:2181,pc332.emulab.net<http://pc332.emulab.net>:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper:
> ZooKeeper create failed after 4 attempts
> 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master
> exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>
> Can someone helps me figure out the problem? Is that because the number of
> region server and zookeeper nodes are incorrect?
>
> Thanks a lot!
>
> Bo
>
>
>
Re: I can't start cluster due to zookeeper
Posted by Bo Fu <bo...@uchicago.edu>.
I was using the zookeeper provided by HBase. Could you please tell me how to run zookeeper before HBase
Bo
> On Apr 27, 2015, at 3:25 AM, Wellington Chevreuil <we...@gmail.com> wrote:
>
> Hi,
>
> Have you checked if your ZK quorum is properly running, before trying to start HBase? Also, the hostnames defined for ZK quorum nodes seem quite unusual. Shouldn't these be as follows?
>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>pc225.emulab.net,pc273.emulab.net,pc210.emulab.net</value>
> <description>The directory shared by RegionServers.
> </description>
> </property>
>
>
> Regards,
> Wellington.
>
> On 26 Apr 2015, at 19:08, Bo Fu <bo...@uchicago.edu> wrote:
>
>> Hi all,
>>
>> I have problem starting a cluster of 1 master and 3 region server. When I started the cluster, the HMaster and HRegionserver will automatically exit.
>>
>> My abase-site.xml:
>> <configuration>
>> <property>
>> <name>hbase.master</name>
>> <value>hadoopmaster:60000</value>
>> </property>
>> <property>
>> <name>hbase.rootdir</name>
>> <value>hdfs://hadoopmaster:9000/hbase</value>
>> </property>
>> <property>
>> <name>hbase.zookeeper.property.dataDir</name>
>> <value>/proj/ucare/bo/hadoop_data/zookeeper</value>
>> </property>
>> <property>
>> <name>hbase.zookeeper.quorum</name>
>> <value>pc225.emulab.net<http://pc225.emulab.net>,pc273.emulab.net<http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net></value>
>> <description>The directory shared by RegionServers.
>> </description>
>> </property>
>> <property>
>> <name>hbase.zookeeper.property.clientPort</name>
>> <value>2181</value>
>> </property>
>> <property>
>> <name>hbase.cluster.distributed</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>> </configuration>
>>
>>
>> Log file:
>>
>> 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to authenticate using SASL (unknown error)
>> 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection established to pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181>, initiating session
>> 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
>> 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to authenticate using SASL (unknown error)
>> 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection established to pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181>, initiating session
>> 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
>> 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc332.emulab.net/155.98.39.132:2181<http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to authenticate using SASL (unknown error)
>> 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 30025ms for sessionid 0x0, closing socket connection and attempting reconnect
>> 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=pc273.emulab.net<http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net>:2181,pc332.emulab.net<http://pc332.emulab.net>:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
>> 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
>> 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master exiting
>> java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
>> at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
>> at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
>> at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
>> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
>> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
>> at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
>> at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
>> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
>> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
>> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>
>>
>> Can someone helps me figure out the problem? Is that because the number of region server and zookeeper nodes are incorrect?
>>
>> Thanks a lot!
>>
>> Bo
>>
>>
>
Re: I can't start cluster due to zookeeper
Posted by Wellington Chevreuil <we...@gmail.com>.
Hi,
Have you checked if your ZK quorum is properly running, before trying to start HBase? Also, the hostnames defined for ZK quorum nodes seem quite unusual. Shouldn't these be as follows?
<property>
<name>hbase.zookeeper.quorum</name>
<value>pc225.emulab.net,pc273.emulab.net,pc210.emulab.net</value>
<description>The directory shared by RegionServers.
</description>
</property>
Regards,
Wellington.
On 26 Apr 2015, at 19:08, Bo Fu <bo...@uchicago.edu> wrote:
> Hi all,
>
> I have problem starting a cluster of 1 master and 3 region server. When I started the cluster, the HMaster and HRegionserver will automatically exit.
>
> My abase-site.xml:
> <configuration>
> <property>
> <name>hbase.master</name>
> <value>hadoopmaster:60000</value>
> </property>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://hadoopmaster:9000/hbase</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/proj/ucare/bo/hadoop_data/zookeeper</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>pc225.emulab.net<http://pc225.emulab.net>,pc273.emulab.net<http://pc273.emulab.net>,pc210.emulab.net<http://pc210.emulab.net></value>
> <description>The directory shared by RegionServers.
> </description>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>2181</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
>
> Log file:
>
> 2015-04-26 11:38:45,938 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181>. Will not attempt to authenticate using SASL (unknown error)
> 2015-04-26 11:38:45,939 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection established to pc273.emulab.net/155.98.39.73:2181<http://pc273.emulab.net/155.98.39.73:2181>, initiating session
> 2015-04-26 11:38:45,940 INFO [main-SendThread(pc273.emulab.net<http://pc273.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,022 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181>. Will not attempt to authenticate using SASL (unknown error)
> 2015-04-26 11:38:47,023 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Socket connection established to pc225.emulab.net/155.98.39.25:2181<http://pc225.emulab.net/155.98.39.25:2181>, initiating session
> 2015-04-26 11:38:47,025 INFO [main-SendThread(pc225.emulab.net<http://pc225.emulab.net>:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
> 2015-04-26 11:38:47,994 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Opening socket connection to server pc332.emulab.net/155.98.39.132:2181<http://pc332.emulab.net/155.98.39.132:2181>. Will not attempt to authenticate using SASL (unknown error)
> 2015-04-26 11:39:17,150 INFO [main-SendThread(pc332.emulab.net<http://pc332.emulab.net>:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 30025ms for sessionid 0x0, closing socket connection and attempting reconnect
> 2015-04-26 11:39:17,251 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=pc273.emulab.net<http://pc273.emulab.net>:2181,pc225.emulab.net<http://pc225.emulab.net>:2181,pc332.emulab.net<http://pc332.emulab.net>:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
> 2015-04-26 11:39:17,251 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
> 2015-04-26 11:39:17,255 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
> at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1982)
> at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
> at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1996)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:512)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:491)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1252)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1230)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:531)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>
> Can someone helps me figure out the problem? Is that because the number of region server and zookeeper nodes are incorrect?
>
> Thanks a lot!
>
> Bo
>
>