You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Alexander Shraer (JIRA)" <ji...@apache.org> on 2011/09/20 02:11:08 UTC
[jira] [Created] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Two possible race conditions during leader establishment
--------------------------------------------------------
Key: ZOOKEEPER-1194
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.4.0
Reporter: Alexander Shraer
Assignee: Alexander Shraer
Fix For: 3.4.0
Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108333#comment-13108333 ]
Hadoop QA commented on ZOOKEEPER-1194:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12495184/zookeeper-1194.patch
against trunk revision 1172406.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/565//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/565//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/565//console
This message is automatically generated.
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0
>
> Attachments: zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-1194:
-------------------------------------
Fix Version/s: 3.4.0
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0, 3.5.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194-ver2.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-1194:
------------------------------------
Fix Version/s: 3.5.0
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0, 3.5.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Alexander Shraer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112886#comment-13112886 ]
Alexander Shraer commented on ZOOKEEPER-1194:
---------------------------------------------
thanks, ben, I fixed these.
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.5.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194-ver2.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Alexander Shraer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Shraer updated ZOOKEEPER-1194:
----------------------------------------
Attachment: zookeeper-1194-ver1.patch
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Alexander Shraer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Shraer updated ZOOKEEPER-1194:
----------------------------------------
Attachment: zookeeper-1194-ver2.patch
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0, 3.5.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194-ver2.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112908#comment-13112908 ]
Hadoop QA commented on ZOOKEEPER-1194:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496174/zookeeper-1194-ver2.patch
against trunk revision 1173949.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/583//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/583//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/583//console
This message is automatically generated.
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.5.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194-ver2.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Thomas Koch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109734#comment-13109734 ]
Thomas Koch commented on ZOOKEEPER-1194:
----------------------------------------
Could you please open a review request?
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Alexander Shraer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108336#comment-13108336 ]
Alexander Shraer commented on ZOOKEEPER-1194:
---------------------------------------------
this patch subsumes zookeeper-1192-ver1.patch (both issues touched the same two files).
I added two new tests to Zab1_0Test.java that test exactly for the two error scenarios described above in the bug report.
The tests fail without the patch.
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0
>
> Attachments: zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Alexander Shraer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Shraer updated ZOOKEEPER-1194:
----------------------------------------
Affects Version/s: (was: 3.4.0)
Fix Version/s: (was: 3.4.0)
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.5.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194-ver2.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112149#comment-13112149 ]
Benjamin Reed commented on ZOOKEEPER-1194:
------------------------------------------
can we break this out from 1192? most of the work in this patch is in the tests anyway. i think it keeps this review simpler.
a couple of small comments:
1) you misspell follwer
2) you should comment testLeaderInConnectingFollwers the importance of the joins after the starts
3) in the Mock Threads, when you detect an error, set it to a string that describes the error so that you can print it on the assert
4) testLeaderInElectingFollwers can be simplified by not starting the leader and making sure the followers fail. this also removes the need for the sleep
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108346#comment-13108346 ]
Hadoop QA commented on ZOOKEEPER-1194:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12495185/zookeeper-1194-ver1.patch
against trunk revision 1172406.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/566//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/566//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/566//console
This message is automatically generated.
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0
>
> Attachments: zookeeper-1194-ver1.patch, zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1194) Two possible race conditions
during leader establishment
Posted by "Alexander Shraer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Shraer updated ZOOKEEPER-1194:
----------------------------------------
Attachment: zookeeper-1194.patch
> Two possible race conditions during leader establishment
> --------------------------------------------------------
>
> Key: ZOOKEEPER-1194
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1194
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.4.0
> Reporter: Alexander Shraer
> Assignee: Alexander Shraer
> Fix For: 3.4.0
>
> Attachments: zookeeper-1194.patch
>
>
> Leader.getEpochToPropose() and Leader.waitForNewEpoch() act as barriers - they make sure that a leader/follower can return from calling the method only once connectingFollowers (or electingFollowers) contain a quorum. But these methods don't make sure that the leader itself is in connectingFollowers/electingFollowers. So the leader didn't necessarily reach the barrier when followers pass it. This can cause the following problems:
> 1. If the leader is not in connectingFollowers when a LearnerHandler returns from getEpochToPropose(), then the epoch sent by the leader to the follower might be smaller than the leader's own last accepted epoch.
> 2. If the leader is not in electingFollowers when LearnerHandler returns from waitForNewEpoch() then the leader will send a NEWLEADER message to followers, and the followers will respond, but it is possible that the NEWLEADER message is not in outstandingProposals when these NEWLEADER acks arrive, which will cause the NEWLEADER acks to be dropped.
> To fix this I propose to explicitly check that the leader is in connectingFollowers/electingFollowers before anyone can pass these barriers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira