You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Gaojinchao <ga...@huawei.com> on 2011/07/18 11:03:28 UTC

Hmaster crashed

I verified the issue HBASE-4064 and created about 100K regions . The master couldn't startup.

Logs showed zk's session was exception. Who can give me a hint?  Thanks.

Logs:
2011-07-18 16:11:15,432 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x2313bf64d1d0000 Retrieved 93 byte(s) of data from znode /hbase/unassigned/c6cb86289f04d5595dc9492e9d946efc and set watcher; region=ufdr5,1509786138,1310707945591.c6cb86289f04d5595dc9492e9d946efc., server=C4C1.site:60000, state=M_ZK_REGION_OFFLINE
2011-07-18 16:11:15,509 WARN org.apache.zookeeper.ClientCnxn: Session 0x2313bf64d1d0000 for server C4C3/157.5.100.3:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Packet len4541600 is out of range!
         at org.apache.zookeeper.ClientCnxn$SendThread.readLength(ClientCnxn.java:710)
         at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:869)
         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
2011-07-18 16:11:15,613 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x2313bf64d1d0000 Unable to list children of znode /hbase/unassigned
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/unassigned
         at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
         at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1243)
         at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:297)
         at org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:252)
         at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:225)
         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:400)
         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:281)
2011-07-18 16:11:15,613 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x2313bf64d1d0000 Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/unassigned
         at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
         at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1243)
         at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:297)
         at org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:252)
         at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:225)
         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:400)
         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:281)
2011-07-18 16:11:15,613 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/unassigned
         at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
         at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1243)
         at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:297)
         at org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:252)
         at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:225)
         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:400)
         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:281)
2011-07-18 16:11:15,614 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2011-07-18 16: