You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Ted Yu (Created) (JIRA)" <ji...@apache.org> on 2011/11/26 06:39:40 UTC
[jira] [Created] (HBASE-4875) ZKLeaderManager.handleLeaderChange()
doesn't handle KeeperException$SessionExpiredException
ZKLeaderManager.handleLeaderChange() doesn't handle KeeperException$SessionExpiredException
-------------------------------------------------------------------------------------------
Key: HBASE-4875
URL: https://issues.apache.org/jira/browse/HBASE-4875
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Ted Yu
TestMasterFailover#testSimpleMasterFailover has failed twice in a row for builds 15 and 16.
>From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/16/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testSimpleMasterFailover/:
{code}
2011-11-26 01:34:49,218 ERROR [Thread-1-EventThread] zookeeper.ZooKeeperWatcher(403): master:52934-0x133dd8281300001 Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
2011-11-26 01:34:49,216 DEBUG [RegionServer:2;hemera.apache.org,44702,1322271278232-EventThread] zookeeper.ZKUtil(230): hconnection-0x133dd8281300009 /hbase/master does not exist. Watcher is set.
2011-11-26 01:34:49,215 DEBUG [Thread-1-EventThread] zookeeper.ZKUtil(230): master:44883-0x133dd8281300002 /hbase/master does not exist. Watcher is set.
2011-11-26 01:34:49,219 DEBUG [Thread-1-EventThread] master.ActiveMasterManager(104): No master available. Notifying waiting threads
2011-11-26 01:34:49,215 INFO [Master:1;hemera.apache.org,52934,1322271278115] master.HMaster(338): HMaster main thread exiting
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4875)
ZKLeaderManager.handleLeaderChange() doesn't handle
KeeperException$SessionExpiredException
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159047#comment-13159047 ]
Ted Yu commented on HBASE-4875:
-------------------------------
In HMaster, this.zooKeeper is replenished:
{code}
private boolean tryRecoveringExpiredZKSession() throws InterruptedException,
IOException, KeeperException {
this.zooKeeper = new ZooKeeperWatcher(conf, MASTER + ":"
+ this.serverName.getPort(), this, true);
{code}
To handle expired session in a general way, we need to update the ZooKeeperWatcher of the caller.
> ZKLeaderManager.handleLeaderChange() doesn't handle KeeperException$SessionExpiredException
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-4875
> URL: https://issues.apache.org/jira/browse/HBASE-4875
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Ted Yu
>
> TestMasterFailover#testSimpleMasterFailover has failed twice in a row for builds 15 and 16.
> From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/16/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testSimpleMasterFailover/:
> {code}
> 2011-11-26 01:34:49,217 DEBUG [RegionServer:0;hemera.apache.org,57516,1322271278190-EventThread] zookeeper.ZooKeeperWatcher(257): regionserver:57516-0x133dd8281300003 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/tokenauth/keymaster
> 2011-11-26 01:34:49,217 WARN [Thread-1-EventThread] zookeeper.ZKUtil(234): master:52934-0x133dd8281300001 Unable to set watcher on znode /hbase/tokenauth/keymaster
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,218 ERROR [Thread-1-EventThread] zookeeper.ZooKeeperWatcher(403): master:52934-0x133dd8281300001 Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,216 DEBUG [RegionServer:2;hemera.apache.org,44702,1322271278232-EventThread] zookeeper.ZKUtil(230): hconnection-0x133dd8281300009 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,215 DEBUG [Thread-1-EventThread] zookeeper.ZKUtil(230): master:44883-0x133dd8281300002 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,219 DEBUG [Thread-1-EventThread] master.ActiveMasterManager(104): No master available. Notifying waiting threads
> 2011-11-26 01:34:49,215 INFO [Master:1;hemera.apache.org,52934,1322271278115] master.HMaster(338): HMaster main thread exiting
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4875)
ZKLeaderManager.handleLeaderChange() doesn't handle
KeeperException$SessionExpiredException
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157376#comment-13157376 ]
Ted Yu commented on HBASE-4875:
-------------------------------
I think we can follow the example from Master dealing with SessionExpiredException:
{code}
if (t != null && t instanceof KeeperException.SessionExpiredException) {
try {
LOG.info("Primary Master trying to recover from ZooKeeper session " +
"expiry.");
return !tryRecoveringExpiredZKSession();
{code}
> ZKLeaderManager.handleLeaderChange() doesn't handle KeeperException$SessionExpiredException
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-4875
> URL: https://issues.apache.org/jira/browse/HBASE-4875
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Ted Yu
>
> TestMasterFailover#testSimpleMasterFailover has failed twice in a row for builds 15 and 16.
> From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/16/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testSimpleMasterFailover/:
> {code}
> 2011-11-26 01:34:49,217 DEBUG [RegionServer:0;hemera.apache.org,57516,1322271278190-EventThread] zookeeper.ZooKeeperWatcher(257): regionserver:57516-0x133dd8281300003 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/tokenauth/keymaster
> 2011-11-26 01:34:49,217 WARN [Thread-1-EventThread] zookeeper.ZKUtil(234): master:52934-0x133dd8281300001 Unable to set watcher on znode /hbase/tokenauth/keymaster
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,218 ERROR [Thread-1-EventThread] zookeeper.ZooKeeperWatcher(403): master:52934-0x133dd8281300001 Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,216 DEBUG [RegionServer:2;hemera.apache.org,44702,1322271278232-EventThread] zookeeper.ZKUtil(230): hconnection-0x133dd8281300009 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,215 DEBUG [Thread-1-EventThread] zookeeper.ZKUtil(230): master:44883-0x133dd8281300002 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,219 DEBUG [Thread-1-EventThread] master.ActiveMasterManager(104): No master available. Notifying waiting threads
> 2011-11-26 01:34:49,215 INFO [Master:1;hemera.apache.org,52934,1322271278115] master.HMaster(338): HMaster main thread exiting
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4875)
ZKLeaderManager.handleLeaderChange() doesn't handle
KeeperException$SessionExpiredException
Posted by "Jimmy Xiang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159017#comment-13159017 ]
Jimmy Xiang commented on HBASE-4875:
------------------------------------
Can we just retry if it is session expired exception? Currently, it just retry in case of (RecoverableZooKeeper):
case CONNECTIONLOSS:
case OPERATIONTIMEOUT:
> ZKLeaderManager.handleLeaderChange() doesn't handle KeeperException$SessionExpiredException
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-4875
> URL: https://issues.apache.org/jira/browse/HBASE-4875
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Ted Yu
>
> TestMasterFailover#testSimpleMasterFailover has failed twice in a row for builds 15 and 16.
> From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/16/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testSimpleMasterFailover/:
> {code}
> 2011-11-26 01:34:49,217 DEBUG [RegionServer:0;hemera.apache.org,57516,1322271278190-EventThread] zookeeper.ZooKeeperWatcher(257): regionserver:57516-0x133dd8281300003 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/tokenauth/keymaster
> 2011-11-26 01:34:49,217 WARN [Thread-1-EventThread] zookeeper.ZKUtil(234): master:52934-0x133dd8281300001 Unable to set watcher on znode /hbase/tokenauth/keymaster
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,218 ERROR [Thread-1-EventThread] zookeeper.ZooKeeperWatcher(403): master:52934-0x133dd8281300001 Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,216 DEBUG [RegionServer:2;hemera.apache.org,44702,1322271278232-EventThread] zookeeper.ZKUtil(230): hconnection-0x133dd8281300009 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,215 DEBUG [Thread-1-EventThread] zookeeper.ZKUtil(230): master:44883-0x133dd8281300002 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,219 DEBUG [Thread-1-EventThread] master.ActiveMasterManager(104): No master available. Notifying waiting threads
> 2011-11-26 01:34:49,215 INFO [Master:1;hemera.apache.org,52934,1322271278115] master.HMaster(338): HMaster main thread exiting
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4875) ZKLeaderManager.handleLeaderChange()
doesn't handle KeeperException$SessionExpiredException
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4875:
--------------------------
Description:
TestMasterFailover#testSimpleMasterFailover has failed twice in a row for builds 15 and 16.
>From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/16/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testSimpleMasterFailover/:
{code}
2011-11-26 01:34:49,217 DEBUG [RegionServer:0;hemera.apache.org,57516,1322271278190-EventThread] zookeeper.ZooKeeperWatcher(257): regionserver:57516-0x133dd8281300003 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/tokenauth/keymaster
2011-11-26 01:34:49,217 WARN [Thread-1-EventThread] zookeeper.ZKUtil(234): master:52934-0x133dd8281300001 Unable to set watcher on znode /hbase/tokenauth/keymaster
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
2011-11-26 01:34:49,218 ERROR [Thread-1-EventThread] zookeeper.ZooKeeperWatcher(403): master:52934-0x133dd8281300001 Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
2011-11-26 01:34:49,216 DEBUG [RegionServer:2;hemera.apache.org,44702,1322271278232-EventThread] zookeeper.ZKUtil(230): hconnection-0x133dd8281300009 /hbase/master does not exist. Watcher is set.
2011-11-26 01:34:49,215 DEBUG [Thread-1-EventThread] zookeeper.ZKUtil(230): master:44883-0x133dd8281300002 /hbase/master does not exist. Watcher is set.
2011-11-26 01:34:49,219 DEBUG [Thread-1-EventThread] master.ActiveMasterManager(104): No master available. Notifying waiting threads
2011-11-26 01:34:49,215 INFO [Master:1;hemera.apache.org,52934,1322271278115] master.HMaster(338): HMaster main thread exiting
{code}
was:
TestMasterFailover#testSimpleMasterFailover has failed twice in a row for builds 15 and 16.
>From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/16/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testSimpleMasterFailover/:
{code}
2011-11-26 01:34:49,218 ERROR [Thread-1-EventThread] zookeeper.ZooKeeperWatcher(403): master:52934-0x133dd8281300001 Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
2011-11-26 01:34:49,216 DEBUG [RegionServer:2;hemera.apache.org,44702,1322271278232-EventThread] zookeeper.ZKUtil(230): hconnection-0x133dd8281300009 /hbase/master does not exist. Watcher is set.
2011-11-26 01:34:49,215 DEBUG [Thread-1-EventThread] zookeeper.ZKUtil(230): master:44883-0x133dd8281300002 /hbase/master does not exist. Watcher is set.
2011-11-26 01:34:49,219 DEBUG [Thread-1-EventThread] master.ActiveMasterManager(104): No master available. Notifying waiting threads
2011-11-26 01:34:49,215 INFO [Master:1;hemera.apache.org,52934,1322271278115] master.HMaster(338): HMaster main thread exiting
{code}
> ZKLeaderManager.handleLeaderChange() doesn't handle KeeperException$SessionExpiredException
> -------------------------------------------------------------------------------------------
>
> Key: HBASE-4875
> URL: https://issues.apache.org/jira/browse/HBASE-4875
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Ted Yu
>
> TestMasterFailover#testSimpleMasterFailover has failed twice in a row for builds 15 and 16.
> From https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/16/testReport/org.apache.hadoop.hbase.master/TestMasterFailover/testSimpleMasterFailover/:
> {code}
> 2011-11-26 01:34:49,217 DEBUG [RegionServer:0;hemera.apache.org,57516,1322271278190-EventThread] zookeeper.ZooKeeperWatcher(257): regionserver:57516-0x133dd8281300003 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/tokenauth/keymaster
> 2011-11-26 01:34:49,217 WARN [Thread-1-EventThread] zookeeper.ZKUtil(234): master:52934-0x133dd8281300001 Unable to set watcher on znode /hbase/tokenauth/keymaster
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,218 ERROR [Thread-1-EventThread] zookeeper.ZooKeeperWatcher(403): master:52934-0x133dd8281300001 Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/tokenauth/keymaster
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
> at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:225)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.handleLeaderChange(ZKLeaderManager.java:85)
> at org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.nodeDeleted(ZKLeaderManager.java:78)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> 2011-11-26 01:34:49,216 DEBUG [RegionServer:2;hemera.apache.org,44702,1322271278232-EventThread] zookeeper.ZKUtil(230): hconnection-0x133dd8281300009 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,215 DEBUG [Thread-1-EventThread] zookeeper.ZKUtil(230): master:44883-0x133dd8281300002 /hbase/master does not exist. Watcher is set.
> 2011-11-26 01:34:49,219 DEBUG [Thread-1-EventThread] master.ActiveMasterManager(104): No master available. Notifying waiting threads
> 2011-11-26 01:34:49,215 INFO [Master:1;hemera.apache.org,52934,1322271278115] master.HMaster(338): HMaster main thread exiting
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira