You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Richard Tang <tr...@gmail.com> on 2012/09/18 18:09:04 UTC
Zookeeper have error msg "Session 0x0 for server null" when setting
distributed hbase cluster
Hi, All,
I have some problems in setting up distributed hbase cluster. My setup is a
3-node hbase cluster, (node3 for hmaster and zk, and node4/5 for
regionservers). It is attached with a 3-node hdfs cluster (node 6 for
namenode, and node7/8 for datanodes). Passphrase-less ssh have been setup
from node3 to node4 (and from node3 to node5) and from node6 to node7 (and
from node6 to node8).
When I start hbase from node3, I have following error found in hbase log
files.
@node3
2012-09-18 08:23:11,523 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x139d9c2d4a50000 type:create
cxid:0x8 zxid:0x1d3 txntype:-1 reqpath:n/a Error Path:/hbase/master
Error:KeeperErrorCode = NodeExists for /hbase/master
@node4
2012-09-18 08:23:27,328 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
2012-09-18 08:23:27,429 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
2012-09-18 08:23:27,429 ERROR
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists
failed after 3 retries
2012-09-18 08:23:27,429 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
regionserver:60020 Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:563)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:532)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:634)
at java.lang.Thread.run(Thread.java:679)
I have managed to run hbase under single-node configuration (with the same
system setup), without problems. Please let me know the possible cause or
howto for approaching the problem cause.
Regards,
Richard
Re: Zookeeper have error msg "Session 0x0 for server null" when
setting distributed hbase cluster
Posted by Aleksandr Shulman <al...@cloudera.com>.
Great! Feel free to follow up with additional questions.
Aleks S.
Cloudera
On Tue, Sep 18, 2012 at 10:08 AM, Richard Tang <tr...@gmail.com>wrote:
> Hi Aleks, thanks for the notes. I have fixed the problem, which is I didnt
> specify hbase.zookeeper.quorum explicitly in my setup. Once I fixed that,
> hbase works well. (It looks in standalone mode, this property is not
> mandatory...)
> Regards,
> Richard
>
> On Tue, Sep 18, 2012 at 12:20 PM, Aleksandr Shulman <aleks@cloudera.com
> >wrote:
>
> > Hi Richard,
> >
> > Are you running zookeeper-server? The default port on which ZK listens is
> > 2181 and so it may also be worth doing a netstat -ap | grep 2181 to see
> if
> > it is up.
> >
> > Can you also post your hbase-site.xml configs?
> >
> > Thanks,
> >
> > Aleks S.
> > Cloudera
> >
> > On Tue, Sep 18, 2012 at 9:09 AM, Richard Tang <tristartom.tech@gmail.com
> > >wrote:
> >
> > > Hi, All,
> > > I have some problems in setting up distributed hbase cluster. My setup
> > is a
> > > 3-node hbase cluster, (node3 for hmaster and zk, and node4/5 for
> > > regionservers). It is attached with a 3-node hdfs cluster (node 6 for
> > > namenode, and node7/8 for datanodes). Passphrase-less ssh have been
> setup
> > > from node3 to node4 (and from node3 to node5) and from node6 to node7
> > (and
> > > from node6 to node8).
> > >
> > > When I start hbase from node3, I have following error found in hbase
> log
> > > files.
> > >
> > > @node3
> > > 2012-09-18 08:23:11,523 INFO
> > > org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
> > > KeeperException when processing sessionid:0x139d9c2d4a50000 type:create
> > > cxid:0x8 zxid:0x1d3 txntype:-1 reqpath:n/a Error Path:/hbase/master
> > > Error:KeeperErrorCode = NodeExists for /hbase/master
> > >
> > > @node4
> > > 2012-09-18 08:23:27,328 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x0
> > > for server null, unexpected error, closing socket connection and
> > attempting
> > > reconnect
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> > > at
> > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> > > at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> > > 2012-09-18 08:23:27,429 WARN
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> > transient
> > > ZooKeeper exception:
> > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > KeeperErrorCode = ConnectionLoss for /hbase/master
> > > 2012-09-18 08:23:27,429 ERROR
> > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper
> exists
> > > failed after 3 retries
> > > 2012-09-18 08:23:27,429 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > > regionserver:60020 Unable to set watcher on znode /hbase/master
> > > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > > KeeperErrorCode = ConnectionLoss for /hbase/master
> > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:563)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:532)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:634)
> > > at java.lang.Thread.run(Thread.java:679)
> > >
> > > I have managed to run hbase under single-node configuration (with the
> > same
> > > system setup), without problems. Please let me know the possible cause
> or
> > > howto for approaching the problem cause.
> > >
> > > Regards,
> > > Richard
> > >
> >
> >
> >
> > --
> > Best Regards,
> >
> > Aleks Shulman
> > 847.814.5804
> > Cloudera
> >
>
--
Best Regards,
Aleks Shulman
847.814.5804
Cloudera
Re: Zookeeper have error msg "Session 0x0 for server null" when
setting distributed hbase cluster
Posted by Richard Tang <tr...@gmail.com>.
Hi Aleks, thanks for the notes. I have fixed the problem, which is I didnt
specify hbase.zookeeper.quorum explicitly in my setup. Once I fixed that,
hbase works well. (It looks in standalone mode, this property is not
mandatory...)
Regards,
Richard
On Tue, Sep 18, 2012 at 12:20 PM, Aleksandr Shulman <al...@cloudera.com>wrote:
> Hi Richard,
>
> Are you running zookeeper-server? The default port on which ZK listens is
> 2181 and so it may also be worth doing a netstat -ap | grep 2181 to see if
> it is up.
>
> Can you also post your hbase-site.xml configs?
>
> Thanks,
>
> Aleks S.
> Cloudera
>
> On Tue, Sep 18, 2012 at 9:09 AM, Richard Tang <tristartom.tech@gmail.com
> >wrote:
>
> > Hi, All,
> > I have some problems in setting up distributed hbase cluster. My setup
> is a
> > 3-node hbase cluster, (node3 for hmaster and zk, and node4/5 for
> > regionservers). It is attached with a 3-node hdfs cluster (node 6 for
> > namenode, and node7/8 for datanodes). Passphrase-less ssh have been setup
> > from node3 to node4 (and from node3 to node5) and from node6 to node7
> (and
> > from node6 to node8).
> >
> > When I start hbase from node3, I have following error found in hbase log
> > files.
> >
> > @node3
> > 2012-09-18 08:23:11,523 INFO
> > org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
> > KeeperException when processing sessionid:0x139d9c2d4a50000 type:create
> > cxid:0x8 zxid:0x1d3 txntype:-1 reqpath:n/a Error Path:/hbase/master
> > Error:KeeperErrorCode = NodeExists for /hbase/master
> >
> > @node4
> > 2012-09-18 08:23:27,328 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
> > for server null, unexpected error, closing socket connection and
> attempting
> > reconnect
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> > at
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> > 2012-09-18 08:23:27,429 WARN
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient
> > ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-09-18 08:23:27,429 ERROR
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists
> > failed after 3 retries
> > 2012-09-18 08:23:27,429 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> > regionserver:60020 Unable to set watcher on znode /hbase/master
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:563)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:532)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:634)
> > at java.lang.Thread.run(Thread.java:679)
> >
> > I have managed to run hbase under single-node configuration (with the
> same
> > system setup), without problems. Please let me know the possible cause or
> > howto for approaching the problem cause.
> >
> > Regards,
> > Richard
> >
>
>
>
> --
> Best Regards,
>
> Aleks Shulman
> 847.814.5804
> Cloudera
>
Re: Zookeeper have error msg "Session 0x0 for server null" when
setting distributed hbase cluster
Posted by Aleksandr Shulman <al...@cloudera.com>.
Hi Richard,
Are you running zookeeper-server? The default port on which ZK listens is
2181 and so it may also be worth doing a netstat -ap | grep 2181 to see if
it is up.
Can you also post your hbase-site.xml configs?
Thanks,
Aleks S.
Cloudera
On Tue, Sep 18, 2012 at 9:09 AM, Richard Tang <tr...@gmail.com>wrote:
> Hi, All,
> I have some problems in setting up distributed hbase cluster. My setup is a
> 3-node hbase cluster, (node3 for hmaster and zk, and node4/5 for
> regionservers). It is attached with a 3-node hdfs cluster (node 6 for
> namenode, and node7/8 for datanodes). Passphrase-less ssh have been setup
> from node3 to node4 (and from node3 to node5) and from node6 to node7 (and
> from node6 to node8).
>
> When I start hbase from node3, I have following error found in hbase log
> files.
>
> @node3
> 2012-09-18 08:23:11,523 INFO
> org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
> KeeperException when processing sessionid:0x139d9c2d4a50000 type:create
> cxid:0x8 zxid:0x1d3 txntype:-1 reqpath:n/a Error Path:/hbase/master
> Error:KeeperErrorCode = NodeExists for /hbase/master
>
> @node4
> 2012-09-18 08:23:27,328 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
> for server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> 2012-09-18 08:23:27,429 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
> ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2012-09-18 08:23:27,429 ERROR
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists
> failed after 3 retries
> 2012-09-18 08:23:27,429 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> regionserver:60020 Unable to set watcher on znode /hbase/master
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:563)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:532)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:634)
> at java.lang.Thread.run(Thread.java:679)
>
> I have managed to run hbase under single-node configuration (with the same
> system setup), without problems. Please let me know the possible cause or
> howto for approaching the problem cause.
>
> Regards,
> Richard
>
--
Best Regards,
Aleks Shulman
847.814.5804
Cloudera