You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Henry Robinson (JIRA)" <ji...@apache.org> on 2009/11/06 20:48:33 UTC
[jira] Created: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Failure of elected leader can lead to never-ending leader election
------------------------------------------------------------------
Key: ZOOKEEPER-569
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
Project: Zookeeper
Issue Type: Bug
Reporter: Henry Robinson
Assignee: Henry Robinson
It is possible for basic LeaderElection to enter a situation where it never terminates.
As an example, consider a three node cluster A, B and C.
1. In the first round, A votes for A, B votes for B and C votes for C
2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
3. A, B vote for C, but C fails.
4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Attachment: zookeeper-569.patch
Here's a patch with tests that appears to fix the issue (test fails without fix, test succeeds with). All tests pass for me with this patch on my laptop.
I have replaced one kludge with another here. QuorumPeer.electionAlg is set to null when electionType==0 until the election is actually run. This causes problems if you want to retrieve the electionAlg object via getElectionAlg() beforehand for tests.
I've set it up so that makeLEStrategy always creates a new LeaderElection if electionType == 0, but also that createElectionAlgorithm sets electionAlg=new LeaderElection(this) instead of null, so that as long as startLeaderElection has been called, getElectionAlg() won't return null.
I've checked to see if this will cause any obvious problems for the call sites of getElectionAlg and couldn't find anything that expected null. It seems more consistent to me this way. The question I have is over why LeaderElection needs re-instantiating each time when FLE does not.
If this sounds confusing, it's because the code really is! The interaction of createElectionAlgorithm, startLeaderElection and makeLEStrategy is hard to discern.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836082#action_12836082 ]
Hadoop QA commented on ZOOKEEPER-569:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435629/zookeeper-569.patch
against trunk revision 912052.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/68/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/68/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/68/console
This message is automatically generated.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829326#action_12829326 ]
Hadoop QA commented on ZOOKEEPER-569:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12434729/zookeeper-569.patch
against trunk revision 903483.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/65/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/65/console
This message is automatically generated.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flavio Paiva Junqueira updated ZOOKEEPER-569:
---------------------------------------------
Status: Open (was: Patch Available)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Attachment: zookeeper-569.patch
I'm making really heavy weather of this :) Flavio, thanks, you're completely right. You also exposed a weird issue that the listener was actually causing the test to succeed no matter what messages were sent - but if the fix for this issue was not in place the test would fail. This is fixed by using the right port for the DatagramSocket.
New patch hopefully has the DatagramSocket listening on the right port, with everything else in order.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830521#action_12830521 ]
Flavio Paiva Junqueira commented on ZOOKEEPER-569:
--------------------------------------------------
Thanks, Henry, it looks good. I agree with your comment on the confusion between LE between instantiated every time it is used, and FLE behaving differently. We should really just have one model.
One comment on the patch is that I don't think you need to instantiate QuorumCnxManager in mockServer() on the new test. The conditional block that checks the listener can also be removed.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799669#action_12799669 ]
Flavio Paiva Junqueira commented on ZOOKEEPER-569:
--------------------------------------------------
One way to implement the test is to implement a mock server to force the particular message interleaving that triggers the bug. No claim it is the best way, but it seemed to be a good idea for FLELostMessageTest.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831203#action_12831203 ]
Mahadev konar commented on ZOOKEEPER-569:
-----------------------------------------
henry, are you working on a new patch adressing flavio's comments?
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Status: Patch Available (was: Open)
Running this by Hudson to verify the build, despite not including new tests.
(It will probably +1 if tests pass because I have slightly edited LETest)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831213#action_12831213 ]
Henry Robinson commented on ZOOKEEPER-569:
------------------------------------------
Yes, hoping to get it out this week.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836009#action_12836009 ]
Henry Robinson commented on ZOOKEEPER-569:
------------------------------------------
The most recent patch I submitted is the right patch - it includes Flavio's suggestions.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Status: Patch Available (was: Open)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-569:
-----------------------------------
Status: Open (was: Patch Available)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829217#action_12829217 ]
Benjamin Reed commented on ZOOKEEPER-569:
-----------------------------------------
i'm also wondering about the heardFrom == 0. in your case A and B will still be up, so heardFrom will not be zero. don't you really want to check whether or not you heard from guy that you think is the leader?
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828823#action_12828823 ]
Flavio Paiva Junqueira commented on ZOOKEEPER-569:
--------------------------------------------------
Henry, I was taking a look at the patch, and I'm slightly confused about how it goes, so I was wondering if you could give me a hand on understanding it.
It seems to me that in the situation you describe, heardFrom won't be empty, so the checking for heardFrom == 0 wouldn't work. Instead, I think you have to call countVotes and check if there is any vote left after it returns, no?
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Status: Open (was: Patch Available)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836356#action_12836356 ]
Hudson commented on ZOOKEEPER-569:
----------------------------------
Integrated in ZooKeeper-trunk #703 (See [http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/703/])
. Failure of elected leader can lead to never-ending leader
election (henry via flavio)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Attachment: zookeeper-569.patch
Here's a functional patch for this issue. Looking into making a mock test now - would like to use Mockito or similar, but will probably have to use hand-rolled mocks for now.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835970#action_12835970 ]
Patrick Hunt commented on ZOOKEEPER-569:
----------------------------------------
Henry, there are two patches, please highlight which one the review should review. thx
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Status: Patch Available (was: Open)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-569:
-----------------------------------
Status: Open (was: Patch Available)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-569:
-----------------------------------
Status: Patch Available (was: Open)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-569:
-----------------------------------
Fix Version/s: 3.3.0
Status: Patch Available (was: Open)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flavio Paiva Junqueira updated ZOOKEEPER-569:
---------------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
+1, thanks Henry! I have just committed it (Committed revision 912119).
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flavio Paiva Junqueira updated ZOOKEEPER-569:
---------------------------------------------
Attachment: ZOOKEEPER-569.patch
Henry, you do need an instance of QCM to get the listener, but you don't need the listener because you're using LE.
I'm uploading a patch in which I have commented out some lines of the new test you wrote. This is how I think it should look like. If you agree, please remove those lines and submit a new patch.
Note that I have also changed the leader election parameter in the new instance of QuorumPeer in mockServer to 0.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: ZOOKEEPER-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Attachment: zookeeper-569.patch
Forgot --no-prefix from git. Hopefully this will apply.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828860#action_12828860 ]
Hadoop QA commented on ZOOKEEPER-569:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12434555/zookeeper-569.patch
against trunk revision 903483.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/62/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/62/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/62/console
This message is automatically generated.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828708#action_12828708 ]
Hadoop QA commented on ZOOKEEPER-569:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12434553/zookeeper-569.patch
against trunk revision 903483.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h7.grid.sp2.yahoo.net/61/console
This message is automatically generated.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Attachment: zookeeper-569.patch
Slightly updated patch - Flavio, I believe you do need to instantiate QuorumCnxManager as QCM.Listener is not static and therefore can't be instantiated without an enclosing instance. I have removed the check for listener == null and set the thread timeout to 15s to tally with LETest.java.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828764#action_12828764 ]
Patrick Hunt commented on ZOOKEEPER-569:
----------------------------------------
Mockito looks good. If someone wants to include as a testing option please enter a JIRA/patch. :-)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-569) Failure of elected leader can
lead to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829220#action_12829220 ]
Henry Robinson commented on ZOOKEEPER-569:
------------------------------------------
Yes, you're both right! I misread my own notes on the bug :/
I'm writing tests for a *real* fix now. Thanks both for pointing this out.
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Status: Patch Available (was: Open)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-569) Failure of elected leader can lead
to never-ending leader election
Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated ZOOKEEPER-569:
-------------------------------------
Status: Open (was: Patch Available)
> Failure of elected leader can lead to never-ending leader election
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-569
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-569
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: zookeeper-569.patch, zookeeper-569.patch, zookeeper-569.patch
>
>
> It is possible for basic LeaderElection to enter a situation where it never terminates.
> As an example, consider a three node cluster A, B and C.
> 1. In the first round, A votes for A, B votes for B and C votes for C
> 2. Since C > B > A, all nodes resolve to vote for C in the second round as there is no first round winner
> 3. A, B vote for C, but C fails.
> 4. C is not elected because neither A nor B hear from it, and so votes for it are discarded
> 5. A and B never reset their votes, despite not hearing from C, so continue to vote for it ad infinitum.
> Step 5 is the bug. If A and B reset their votes to themselves in the case where the heard-from vote set is empty, leader election will continue.
> I do not know if this affects running ZK clusters, as it is possible that the out-of-band failure detection protocols may cause leader election to be restarted anyhow, but I've certainly seen this in tests.
> I have a trivial patch which fixes it, but it needs a test (and tests for race conditions are hard to write!)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.