You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Vincent Bernat (JIRA)" <ji...@apache.org> on 2013/08/07 16:44:47 UTC
[jira] [Created] (ZOOKEEPER-1738) Xid out of order from a 3.4.5
client to a 3.3.5 cluster
Vincent Bernat created ZOOKEEPER-1738:
-----------------------------------------
Summary: Xid out of order from a 3.4.5 client to a 3.3.5 cluster
Key: ZOOKEEPER-1738
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1738
Project: ZooKeeper
Issue Type: Bug
Affects Versions: 3.3.5
Environment: Server: zookeeper 3.3.5+dfsg1-1ubuntu1
Client: zookeeper 3.4.5 from Cloudera 4.3.0
Reporter: Vincent Bernat
Fix For: 3.4.5
This happens in the context of HBase master nodes getting connections from HBase region server. Once an HBase region server joins the cluster, I get the following error:
{code}
2013-08-07 13:35:18,676 WARN org.apache.zookeeper.ClientCnxn: Session 0xd4058c4d7940003 for server zk-01.dev.dailymotion.com/10.194.60.13:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 56 with err -101 expected Xid 55 for a packet with details: clientPath:null serverPath:null finished:false header:: 55,14 replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@360193e5 response:: org.apache.zookeeper.MultiResponse@0
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:795)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
2013-08-07 13:35:18,676 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
2013-08-07 13:35:18,676 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper multi failed after 3 retries
2013-08-07 13:35:18,677 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Unable to ensure that the table -ROOT- will be enabled because of a ZooKeeper issue
2013-08-07 13:35:18,677 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2013-08-07 13:35:18,677 FATAL org.apache.hadoop.hbase.master.HMaster: Unable to ensure that the table -ROOT- will be enabled because of a ZooKeeper issue
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:931)
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:531)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1440)
at org.apache.hadoop.hbase.zookeeper.ZKTable.setTableState(ZKTable.java:245)
at org.apache.hadoop.hbase.zookeeper.ZKTable.setEnabledTable(ZKTable.java:325)
at org.apache.hadoop.hbase.master.AssignmentManager.setEnabledTable(AssignmentManager.java:3576)
at org.apache.hadoop.hbase.master.AssignmentManager.setEnabledTable(AssignmentManager.java:2340)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1674)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:675)
at org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:586)
at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:525)
at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransitionAndBlockUntilAssigned(AssignmentManager.java:489)
at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:679)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:583)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:395)
at java.lang.Thread.run(Thread.java:722)
2013-08-07 13:35:18,678 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2013-08-07 13:35:18,678 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Server stopped; skipping assign of -ROOT-,,0.70236052 state=OFFLINE, ts=1375881792131, server=null
2013-08-07 13:35:18,678 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Waiting on 70236052/-ROOT-
2013-08-07 13:35:18,678 INFO org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor: masternode-01.dev.dailymotion.com,60000,1375880747185.timeoutMonitor exiting
2013-08-07 13:35:18,679 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=masternode-01.dev.dailymotion.com,60000,1375880747185, region=70236052/-ROOT-, which is more than 15 seconds late
2013-08-07 13:35:18,776 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
2013-08-07 13:35:18,776 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2013-08-07 13:35:18,776 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper getData failed after 3 retries
2013-08-07 13:35:18,777 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira