You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (Created) (JIRA)" <ji...@apache.org> on 2012/03/09 13:20:57 UTC
[jira] [Created] (HBASE-5549) Master can fail if ZooKeeper session
expires
Master can fail if ZooKeeper session expires
--------------------------------------------
Key: HBASE-5549
URL: https://issues.apache.org/jira/browse/HBASE-5549
Project: HBase
Issue Type: Bug
Components: master, zookeeper
Affects Versions: 0.96.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
This can happen in real life, it can happen when:
- master & zookeeper starts
- zookeeper connection is cut
- master enters the retry loop
- in the meantime the session expires
- the network comes back, the session is recreated
- the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231217#comment-13231217 ]
Hadoop QA commented on HBASE-5549:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518664/5549.v8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.TestZooKeeper
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1201//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1201//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1201//console
This message is automatically generated.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231029#comment-13231029 ]
Hadoop QA commented on HBASE-5549:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518639/nochange.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1199//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1199//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1199//console
This message is automatically generated.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5549:
-------------------------
Attachment: 5549_092.txt
Backport of the trunk patch to 0.92
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549_092.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Patch Available (was: Open)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440097#comment-13440097 ]
Hudson commented on HBASE-5549:
-------------------------------
Integrated in HBase-0.94 #417 (See [https://builds.apache.org/job/HBase-0.94/417/])
HBASE-5549 Master can fail if ZooKeeper session expires (Revision 1376374)
Result = SUCCESS
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231305#comment-13231305 ]
nkeywal commented on HBASE-5549:
--------------------------------
Can be committed. Hopefully this is the end of the "ZooKeeper expiry" flakiness.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406977#comment-13406977 ]
nkeywal commented on HBASE-5549:
--------------------------------
It should be portable to previous branches. As it impacts multiple files there could be some conflict but this jira is quite self contained.
For the test part, reading the code you mentioned, yeah it could be :-). I don't see why it's failing however. Making zookeeper session expire is a kind of black art...
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Open (was: Patch Available)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408770#comment-13408770 ]
Himanshu Vashishtha commented on HBASE-5549:
--------------------------------------------
>From the code (and javadoc), it seems we are not 100% sure of the zookeeper close session event.
And, we avoid test failures based on that (either by re-creating zkw as in TestZookeeper#testClientSessionExpired, or remove asserts altogether, TestReplicationPeer.
Is it ok to impose a hard wait on the session timeout (basic idea is to have this in HBaseTestingUtility#expireSession):
{code}
+ final boolean[] isClosed = new boolean[]{false} ;
+ ZooKeeper monitorWatcher = new ZooKeeper(quorumServers, sessionTimeout, new Watcher() {
+ @Override
+ public void process(WatchedEvent event) {
+ LOG.info("Closed in the monitor.");
+ isClosed[0] = true ;
+ }
+ }, sessionID, password);
monitorWatcher.close();
+ while(!isClosed[0]){
+ // sleep;
+ Thread.sleep(sessionTimeout);
+ }
{code}
And, remove the two handler approach.
This way, we are sure that the session has indeed expired, and clean up the tests. The downside is we sleep, until we actually have expired the session (or we can have some increasing sleep time duration and then fail the process after a hard limit).
Good to know what others think.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440153#comment-13440153 ]
Hudson commented on HBASE-5549:
-------------------------------
Integrated in HBase-0.92-security #116 (See [https://builds.apache.org/job/HBase-0.92-security/116/])
HBASE-5549 Master can fail if ZooKeeper session expires (Revision 1376369)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231242#comment-13231242 ]
Hadoop QA commented on HBASE-5549:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518665/5549.v9.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestImportTsv
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1202//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1202//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1202//console
This message is automatically generated.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231159#comment-13231159 ]
Hadoop QA commented on HBASE-5549:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518660/5549.v7.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1200//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1200//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1200//console
This message is automatically generated.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Patch Available (was: Open)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231444#comment-13231444 ]
nkeywal commented on HBASE-5549:
--------------------------------
Can't create a review, I got error 500 as well...
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "Zhihong Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5549:
------------------------------
Fix Version/s: 0.96.0
Hadoop Flags: Reviewed
Integrated to TRUNK.
Thanks for the patch, N.
Thanks for the review, Stack.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Open (was: Patch Available)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Patch Available (was: Open)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440081#comment-13440081 ]
Lars Hofhansl commented on HBASE-5549:
--------------------------------------
Thanks Stack! I was going to do the 0.94 patch tomorrow... But this is better :)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Attachment: 5549.v9.patch
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Open (was: Patch Available)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Attachment: 5549.v6.patch
Contains the patch for HBASE-5572
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231453#comment-13231453 ]
nkeywal commented on HBASE-5549:
--------------------------------
Now it works: https://reviews.apache.org/r/4391/
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439993#comment-13439993 ]
Himanshu Vashishtha commented on HBASE-5549:
--------------------------------------------
by "hang", I mean "hangs occasionally".
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549_092.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231676#comment-13231676 ]
Lars Hofhansl commented on HBASE-5549:
--------------------------------------
Hey N., is this dependent on HBASE-5399. If not, it might be a good candidate for 0.94.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439753#comment-13439753 ]
stack commented on HBASE-5549:
------------------------------
@Himanshu What JIRA did you upload your fix too?
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408782#comment-13408782 ]
Himanshu Vashishtha commented on HBASE-5549:
--------------------------------------------
Thanks for looking into this.
Since the update is happening inside an inner class, it should be final (volatile is not possible). I will wait/notify on isClosed array object, and upload the patch in a new jira.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Attachment: 5549.v8.patch
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408772#comment-13408772 ]
nkeywal commented on HBASE-5549:
--------------------------------
It seems to be a good idea. For the implementation, you should replace the sleep by a notify, plus 'isClosed' should be also be a simple boolean instead of an array, tagged as volatile as it's used between thread.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5549:
-------------------------
Fix Version/s: 0.92.2
Applied to 0.92 branch. Working on patch for 0.94 now.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0
>
> Attachments: 5549_092.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231598#comment-13231598 ]
nkeywal commented on HBASE-5549:
--------------------------------
v11 with the comments taken into account... Thank you for the review.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Open (was: Patch Available)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Patch Available (was: Open)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440069#comment-13440069 ]
stack commented on HBASE-5549:
------------------------------
I tested the 0.92 patch on two machines and it didn't seem to break anything. I got a fail though it ran 1142 tests. A repeat ran all tests but it failed TestMasterReplication. Will keep an eye on it.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0
>
> Attachments: 5549_092.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Attachment: 5549.v11.patch
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231299#comment-13231299 ]
Hadoop QA commented on HBASE-5549:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518676/5549.v10.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1204//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1204//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1204//console
This message is automatically generated.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440214#comment-13440214 ]
Hudson commented on HBASE-5549:
-------------------------------
Integrated in HBase-0.94-security #49 (See [https://builds.apache.org/job/HBase-0.94-security/49/])
HBASE-5549 Master can fail if ZooKeeper session expires (Revision 1376374)
Result = SUCCESS
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231491#comment-13231491 ]
stack commented on HBASE-5549:
------------------------------
@N LGTM. Change reconnectAfterExpiry to reconnectAfterExpiration and post patch here. Will run it through hadoopqa then commit if all good.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Attachment: 5549.v7.patch
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439992#comment-13439992 ]
Himanshu Vashishtha commented on HBASE-5549:
--------------------------------------------
@Stack: The patch is at HBase-6354. It is based on the approach as mentioned in the above comment. It works as per current approach, but when I tried to catch explicitly the "Expired" event in the process method of the watcher, it hangs as if it never received that event. In that case, the only event it received is SynConnected, which is when you create a new zk instance.
Basically, this:
{code}
ZooKeeper monitorWatcher = new ZooKeeper(quorumServers, 1000,
new Watcher() {
@Override
public void process(WatchedEvent event) {
KeeperState state = event.getState();
switch (state) {
case Expired:
LOG.info("processing SessionExpired event");
sessionClosed.set(true);
synchronized (sessionClosed) {
sessionClosed.notifyAll();
}
break;
default:
LOG.info("processing default event" +state);
}
}
}, sessionID, password);
{code}
Sorry for the late reply.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549_092.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231418#comment-13231418 ]
Zhihong Yu commented on HBASE-5549:
-----------------------------------
TestMasterZKSessionRecovery is removed. Is it covered in other tests now ?
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440044#comment-13440044 ]
Lars Hofhansl commented on HBASE-5549:
--------------------------------------
@Stack: are you planning to integrate the patch into 0.92? If so, let's mark this issue with fix version of 0.92 (and 0.94)... Should have a 0.94 patch as well.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549_092.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440074#comment-13440074 ]
stack commented on HBASE-5549:
------------------------------
@Himanshu A thread dump when its hung? Might tell us something?
@Lars Done I'm triggering and watching 0.94 builds... will make sure I've not added a regression.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440378#comment-13440378 ]
stack commented on HBASE-5549:
------------------------------
@Himanshu Where is the issue which adds waiting on expire if its not being done over in hbase-6354?
I backported this patch because it removed a test purportedly 'useless' that was messing up a subsequent test. I also backported it to get the other cleanups around zk interaction that this patch adds and to make it so there is parity here from 0.92 through to trunk. If there is an issue w/ fix for wait on expire, I can backport that too.
Suggest you look at Alex's countdown latch over in his watcher in the this patch https://reviews.facebook.net/D4605 It does something like what you paste above for the expiration monitor.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448300#comment-13448300 ]
Hudson commented on HBASE-5549:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/])
HBASE-5549 Master can fail if ZooKeeper session expires (Revision 1376374)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Patch Available (was: Open)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Open (was: Patch Available)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Open (was: Patch Available)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Attachment: nochange.patch
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231478#comment-13231478 ]
Zhihong Yu commented on HBASE-5549:
-----------------------------------
Commented on review board.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406763#comment-13406763 ]
Himanshu Vashishtha commented on HBASE-5549:
--------------------------------------------
This seems to fix the HBaseTestingUtility#expiresession method as it introduced a new logic of creating a monitor and then expiring the session. But it seems this fix needs some more work? For example, TestReplicationPeer occasionally fails even with this change on the trunk, citing improper session termination.
{code}
testResetZooKeeperSession(org.apache.hadoop.hbase.replication.TestReplicationPeer): ReplicationPeer ZooKeeper session was not properly expired.
{code}
On another note, I wonder whether this patch can be backported to 0.92/0.94?
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231695#comment-13231695 ]
Hudson commented on HBASE-5549:
-------------------------------
Integrated in HBase-TRUNK #2686 (See [https://builds.apache.org/job/HBase-TRUNK/2686/])
HBASE-5549 HBASE-5572 Master can fail if ZooKeeper session expires (N Keywal) (Revision 1301775)
Result = SUCCESS
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Patch Available (was: Open)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231405#comment-13231405 ]
Zhihong Yu commented on HBASE-5549:
-----------------------------------
Minor comments:
For ActiveMasterManager.java, the second line should be indented to the left.
{code}
- String backupZNode = ZKUtil.joinZNode(
+ ClusterStatusTracker clusterStatusTracker) {
{code}
I tried to create a review request on review board but got error 500.
The patch is of decent size so review board usage is desirable. Please leave out Bugs field so that this JIRA is not flooded.
Nice work.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231858#comment-13231858 ]
Hudson commented on HBASE-5549:
-------------------------------
Integrated in HBase-TRUNK-security #140 (See [https://builds.apache.org/job/HBase-TRUNK-security/140/])
HBASE-5549 HBASE-5572 Master can fail if ZooKeeper session expires (N Keywal) (Revision 1301775)
Result = SUCCESS
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterZKSessionRecovery.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5549:
-------------------------
Fix Version/s: 0.94.2
Applied to 0.94 branch.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232005#comment-13232005 ]
nkeywal commented on HBASE-5549:
--------------------------------
@lars: there is no theoretical dependency. In practice, I think that both impacted some same tests so it may require some minor changes when applying the patch.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Status: Patch Available (was: Open)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-5549:
---------------------------
Attachment: 5549.v10.patch
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "Zhihong Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihong Yu updated HBASE-5549:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231441#comment-13231441 ]
nkeywal commented on HBASE-5549:
--------------------------------
Yes, see HBASE-5572 for the reasons...
On Fri, Mar 16, 2012 at 6:41 PM, Zhihong Yu (Commented) (JIRA) <
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440351#comment-13440351 ]
Himanshu Vashishtha commented on HBASE-5549:
--------------------------------------------
@Stack: So, currently (in this patch also), we don't wait for the Expired event, which is not correct.
In 6354, v2 patch also doesn't look for it.
I added this check (as mentioned earlier) but sometimes, the only event received is synConnected, and not the Expired event. So, it waits on the AtomicBoolean sessionClosed, to become true, which will happen only when the watcher received the Expired event. Thinking today, it looks like we should invoke the close method only *after* the monitorwatcher is properly initialized. Hmmm, I will look at this today again.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231646#comment-13231646 ]
Hadoop QA commented on HBASE-5549:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518730/5549.v11.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1210//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1210//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1210//console
This message is automatically generated.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5549) Master can fail if ZooKeeper session
expires
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5549:
-------------------------
Attachment: 5549_094.txt
Patch for 0.94. Doesn't seem to have broken anything obvious. Will apply and keep an eye out.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 5549_092.txt, 5549_094.txt, 5549.v10.patch, 5549.v11.patch, 5549.v6.patch, 5549.v7.patch, 5549.v8.patch, 5549.v9.patch, nochange.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5549) Master can fail if ZooKeeper
session expires
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230457#comment-13230457 ]
Hadoop QA commented on HBASE-5549:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518500/5549.v6.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 161 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.client.TestMultiParallel
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1193//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1193//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1193//console
This message is automatically generated.
> Master can fail if ZooKeeper session expires
> --------------------------------------------
>
> Key: HBASE-5549
> URL: https://issues.apache.org/jira/browse/HBASE-5549
> Project: HBase
> Issue Type: Bug
> Components: master, zookeeper
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 5549.v6.patch
>
>
> There is a retry mechanism in RecoverableZooKeeper, but when the session expires, the whole ZooKeeperWatcher is recreated, hence the retry mechanism does not work in this case. This is why a sleep is needed in TestZooKeeper#testMasterSessionExpired: we need to wait for ZooKeeperWatcher to be recreated before using the connection.
> This can happen in real life, it can happen when:
> - master & zookeeper starts
> - zookeeper connection is cut
> - master enters the retry loop
> - in the meantime the session expires
> - the network comes back, the session is recreated
> - the retries continues, but on the wrong object, hence fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira