You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Benjamin Reed (JIRA)" <ji...@apache.org> on 2008/09/02 19:27:44 UTC
[jira] Created: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Old leader election can elect a dead leader over and over again
---------------------------------------------------------------
Key: ZOOKEEPER-131
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
Project: Zookeeper
Issue Type: Bug
Components: leaderElection
Reporter: Benjamin Reed
I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
1) Leader dies
2) Followers start looking for a new leader before all Followers have abandoned the Leader
3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
4) The dead Leader gets reelected.
For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-131:
------------------------------------
Attachment: ZOOKEEPER-131.patch
here is a patch that changes e.printstacktrace to log.err.... other than that the patch looks good. +1
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Assignee: Benjamin Reed
> Attachments: ZOOKEEPER-131.patch, ZOOKEEPER-131.patch, ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-131:
------------------------------------
Attachment: ZOOKEEPER-131.patch
forgot to add the test file.
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Assignee: Benjamin Reed
> Attachments: ZOOKEEPER-131.patch, ZOOKEEPER-131.patch, ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated ZOOKEEPER-131:
------------------------------------
Status: Patch Available (was: Open)
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Attachments: ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632016#action_12632016 ]
Flavio Paiva Junqueira commented on ZOOKEEPER-131:
--------------------------------------------------
+1 for the patch. The only suggestion I have is to replace e.printStackTrace() statements with LOG("msg", e) statements. I received this comment on another patch, so I'm just passing the message along. :-)
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Assignee: Benjamin Reed
> Attachments: ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632933#action_12632933 ]
Hudson commented on ZOOKEEPER-131:
----------------------------------
Integrated in ZooKeeper-trunk #89 (See [http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/89/])
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Assignee: Benjamin Reed
> Fix For: 3.0.0
>
> Attachments: ZOOKEEPER-131.patch, ZOOKEEPER-131.patch, ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-131:
-----------------------------------
Assignee: Benjamin Reed
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Assignee: Benjamin Reed
> Attachments: ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Austin Shoemaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632120#action_12632120 ]
Austin Shoemaker commented on ZOOKEEPER-131:
--------------------------------------------
This patch appears to solve the problem for algorithm 0- our unit test completed successfully 16 times.
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Assignee: Benjamin Reed
> Attachments: ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated ZOOKEEPER-131:
------------------------------------
Attachment: ZOOKEEPER-131.patch
This patch adds a test case that reproduces the problem and a fix (that passes the test).
The fix simply removes votes for hosts that a peer has not heard from before the vote is counted.
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Attachments: ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-131) Old leader election can elect a
dead leader over and over again
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-131:
------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks Ben.
> Old leader election can elect a dead leader over and over again
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-131
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-131
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Benjamin Reed
> Assignee: Benjamin Reed
> Fix For: 3.0.0
>
> Attachments: ZOOKEEPER-131.patch, ZOOKEEPER-131.patch, ZOOKEEPER-131.patch
>
>
> I think there is a race condition that is probably easy to get into with the old leader election and a large number of servers:
> 1) Leader dies
> 2) Followers start looking for a new leader before all Followers have abandoned the Leader
> 3) The Followers looking for a new leader see votes of Followers still following the (now dead) Leader and start voting for the dead Leader
> 4) The dead Leader gets reelected.
> For the old leader election a server should not vote for another server that is not nominating himself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.