You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Vishal Kathuria (JIRA)" <ji...@apache.org> on 2011/05/25 23:41:47 UTC
[jira] [Created] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
--------------------------------------------------------------------------------------------------------
Key: ZOOKEEPER-1075
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
Project: ZooKeeper
Issue Type: Bug
Components: leaderElection
Affects Versions: 3.3.2
Environment: Windows 7
Reporter: Vishal Kathuria
Here is the sequence of steps that reproduces the problem.
On a 3 server ensemble,
1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
2. Bring down 2
3. Bring up 2.
4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Vishal Kathuria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039429#comment-13039429 ]
Vishal Kathuria commented on ZOOKEEPER-1075:
--------------------------------------------
I am syncing from the trunk and trying it out.
This issue looks different from the one in 975 - in this issue, we never reach convergence.
I lost the logs as I reran the server after implementing my fix. I'll upload the logs after rerunning it.
Thanks!
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039784#comment-13039784 ]
Flavio Junqueira commented on ZOOKEEPER-1075:
---------------------------------------------
Could you post your configuration here, please? It sounds awkward that your leader was hanging there for over an hour without support for a quorum. It should have timed out.
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Vishal Kathuria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vishal Kathuria resolved ZOOKEEPER-1075.
----------------------------------------
Resolution: Not A Problem
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Attachments: zk1075.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Vishal K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039394#comment-13039394 ]
Vishal K commented on ZOOKEEPER-1075:
-------------------------------------
Hi Vishal,
Can you please attach logs? Also, it might be good to use the trunk since there have been some relevant code changes (https://issues.apache.org/jira/browse/ZOOKEEPER-975).
Thanks,
Vishal :-)
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039554#comment-13039554 ]
Flavio Junqueira commented on ZOOKEEPER-1075:
---------------------------------------------
Hi Vishal, In the scenario you're describing, it sounds like 1 will eventually release leadership (it doesn't have enough supporters), and in that case both 1 and 2 are looking for a leader and a leader is elected regularly, no?
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Vishal Kathuria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vishal Kathuria updated ZOOKEEPER-1075:
---------------------------------------
Attachment: zk1075.txt
This is a simple patch to optimize the case when a server that would restore the quorum is coming up.
Instead of leader giving up and leader election happening, this change will make the new server join the existing ensemble.
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Attachments: zk1075.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Vishal Kathuria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039364#comment-13039364 ]
Vishal Kathuria commented on ZOOKEEPER-1075:
--------------------------------------------
The issue I think is this code below in FastLeaderElection.java
/**
* Before joining an established ensemble, verify that
* a majority are following the same leader.
*/
outofelection.put(n.sid, new Vote(n.leader, n.zxid,
n.epoch, n.state));
if (termPredicate(outofelection, new Vote(n.leader,
n.zxid, n.epoch, n.state))
&& checkLeader(outofelection, n.leader, n.epoch)) {
In the case above, there is only one entry in outofelection that does not constitute the majority. What we really need to check is whether outofelection.size() + 1(this server) forms a majority because once this server accepts the leader, the leader will have a majority of followers.
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040099#comment-13040099 ]
Flavio Junqueira commented on ZOOKEEPER-1075:
---------------------------------------------
Heh, no worries, Vishal. It is good already that you considered contributing.
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Attachments: zk1075.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Vishal Kathuria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039762#comment-13039762 ]
Vishal Kathuria commented on ZOOKEEPER-1075:
--------------------------------------------
I did not see 1 relinquish the leadership when I ran the repro. 1 continued to think it was the leader and 2 continued to refuse accepting that and they stayed in this state indefinitely (over an hour I think)
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1075) Zookeeper Server cannot join an
existing ensemble if the existing ensemble doesn't already have a quorum
Posted by "Vishal Kathuria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039844#comment-13039844 ]
Vishal Kathuria commented on ZOOKEEPER-1075:
--------------------------------------------
ah, that makes sense. I was using high timeouts because I was debugging some other issue. It doesn't occur with low timeouts (and I am embarrassed it didn't occur to me :).
I am attaching the patch anyway if you folks think it is a nice optimization - this way the node can join right away instead of having to wait for the timeout and start a new election.
I'll close the Jira.
> Zookeeper Server cannot join an existing ensemble if the existing ensemble doesn't already have a quorum
> --------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-1075
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1075
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.3.2
> Environment: Windows 7
> Reporter: Vishal Kathuria
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Here is the sequence of steps that reproduces the problem.
> On a 3 server ensemble,
> 1. Bring up two servers (say 1 and 2). Lets say 1 is leading.
> 2. Bring down 2
> 3. Bring up 2.
> 4. 2 gets a notification from 1 that it is leading but 2 doesn't accept it as a leader since it cannot find one other node that thinks 1 is the leader.
> So the ensemble gets stuck where 2 isn't following. If at this point, 3 comes up, then one of 2 & 3 will become a leader and 1 will keep thinking it is the leader.
> I am working on a patch to fix this issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira