You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jay Wilson <re...@circle-cross-jn.com> on 2012/07/02 03:34:50 UTC
HBASE -- HMaster Aborts after 28 minutes.
I assume a timer may need to be increased based on these messages. Any
thoughts as to which one or is it something else?
2012-07-01 18:17:16,461 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=0
2012-07-01 18:17:16,466 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s)
and gc'd 0 unreferenced parent region(s)
2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
read additional data from server sessionid 0x38444cf77f0000, likely
server has closed socket, closing socket connection and attempting reconnect
2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
read additional data from server sessionid 0x238444cf77e0000, likely
server has closed socket, closing socket connection and attempting reconnect
2012-07-01 18:20:00,384 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
hconnection-0x38444cf77f0000 Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
2012-07-01 18:20:00,386 ERROR
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
hconnection-0x38444cf77f0000 Received unexpected KeeperException,
re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
2012-07-01 18:20:00,387 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
This client just lost it's session with ZooKeeper, trying to reconnect.
2012-07-01 18:20:00,387 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Trying to reconnect to zookeeper.
2012-07-01 18:20:00,957 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server devrackA-05/172.18.0.6:2181
2012-07-01 18:20:00,958 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to devrackA-05/172.18.0.6:2181, initiating session
2012-07-01 18:20:00,961 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x238444cf77e0000 has expired,
closing socket connection
2012-07-01 18:20:00,961 FATAL org.apache.hadoop.hbase.master.HMaster:
master:60000-0x238444cf77e0000 master:60000-0x238444cf77e0000 received
expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
2012-07-01 18:20:00,961 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2012-07-01 18:20:00,961 INFO org.apache.zookeeper.ClientCnxn:
EventThread shut down
---
Jay Wilson
Re: HBASE -- HMaster Aborts after 28 minutes.
Posted by Suraj Varma <sv...@gmail.com>.
Session expired usually results from a long GC that exceeds the
zookeeper.session.timeout.
> 2012-07-01 18:20:00,961 FATAL org.apache.hadoop.hbase.master.HMaster:master:60000-0x238444cf77e0000 master:60000-0x238444cf77e0000 received
expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode
= Session expired
See http://hbase.apache.org/book.html#trouble.log.gc
--Suraj
On Sun, Jul 1, 2012 at 6:34 PM, Jay Wilson
<re...@circle-cross-jn.com> wrote:
> I assume a timer may need to be increased based on these messages. Any
> thoughts as to which one or is it something else?
>
> 2012-07-01 18:17:16,461 INFO
> org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
> servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=0
> 2012-07-01 18:17:16,466 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
> 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x38444cf77f0000, likely
> server has closed socket, closing socket connection and attempting reconnect
> 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x238444cf77e0000, likely
> server has closed socket, closing socket connection and attempting reconnect
> 2012-07-01 18:20:00,384 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> hconnection-0x38444cf77f0000 Unable to set watcher on znode /hbase/master
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> 2012-07-01 18:20:00,386 ERROR
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
> hconnection-0x38444cf77f0000 Received unexpected KeeperException,
> re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> 2012-07-01 18:20:00,387 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> This client just lost it's session with ZooKeeper, trying to reconnect.
> 2012-07-01 18:20:00,387 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Trying to reconnect to zookeeper.
> 2012-07-01 18:20:00,957 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server devrackA-05/172.18.0.6:2181
> 2012-07-01 18:20:00,958 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to devrackA-05/172.18.0.6:2181, initiating session
> 2012-07-01 18:20:00,961 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x238444cf77e0000 has expired,
> closing socket connection
> 2012-07-01 18:20:00,961 FATAL org.apache.hadoop.hbase.master.HMaster:
> master:60000-0x238444cf77e0000 master:60000-0x238444cf77e0000 received
> expired from ZooKeeper, aborting
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> 2012-07-01 18:20:00,961 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
> 2012-07-01 18:20:00,961 INFO org.apache.zookeeper.ClientCnxn:
> EventThread shut down
>
> ---
> Jay Wilson