You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Jeremy Stribling (Updated) (JIRA)" <ji...@apache.org> on 2012/03/16 01:10:39 UTC
[jira] [Updated] (ZOOKEEPER-1419) Leader election never settles for
a 5-node cluster
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Stribling updated ZOOKEEPER-1419:
----------------------------------------
Attachment: ZOOKEEPER-1419.tgz
Contains logs for each node in each run, and the data directories for all nodes.
> Leader election never settles for a 5-node cluster
> --------------------------------------------------
>
> Key: ZOOKEEPER-1419
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1419
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.4.3
> Environment: 64-bit Linux, all nodes running on the same machine (different ports)
> Reporter: Jeremy Stribling
> Attachments: ZOOKEEPER-1419.tgz
>
>
> We have a situation where it seems to my untrained eye that leader election never finishes for a 5-node cluster. In this test, all nodes are ZK 3.4.3 and running on the same server (listening on different ports, of course). The nodes have server IDs of 0, 1, 2, 3, 4. The test brings up the cluster in different configurations, adding in a new node each time. We embed ZK in our application, so when we shut a node down and restart it with a new configuration, it all happens in a single JVM process. Here's our server startup code (for the case where there's more than one node in the cluster):
> {code}
> if (servers.size() > 1) {
> _log.debug("Starting Zookeeper server in quorum server mode");
> _quorum_peer = new QuorumPeer();
> synchronized(_quorum_peer) {
> _quorum_peer.setClientPortAddress(clientAddr);
> _quorum_peer.setTxnFactory(log);
> _quorum_peer.setQuorumPeers(servers);
> _quorum_peer.setElectionType(_election_alg);
> _quorum_peer.setMyid(_server_id);
> _quorum_peer.setTickTime(_tick_time);
> _quorum_peer.setInitLimit(_init_limit);
> _quorum_peer.setSyncLimit(_sync_limit);
> QuorumVerifier quorumVerifier =
> new QuorumMaj(servers.size());
> _quorum_peer.setQuorumVerifier(quorumVerifier);
> _quorum_peer.setCnxnFactory(_cnxn_factory);
> _quorum_peer.setZKDatabase(new ZKDatabase(log));
> _quorum_peer.start();
> }
> } else {
> _log.debug("Starting Zookeeper server in single server mode");
> _zk_server = new ZooKeeperServer();
> _zk_server.setTxnLogFactory(log);
> _zk_server.setTickTime(_tick_time);
> _cnxn_factory.startup(_zk_server);
> }
> {code}
> And here's our shutdown code:
> {code}
> if (_quorum_peer != null) {
> synchronized(_quorum_peer) {
> _quorum_peer.shutdown();
> FastLeaderElection fle =
> (FastLeaderElection) _quorum_peer.getElectionAlg();
> fle.shutdown();
> try {
> _quorum_peer.getTxnFactory().commit();
> } catch (java.nio.channels.ClosedChannelException e) {
> // ignore
> }
> }
> } else {
> _cnxn_factory.shutdown();
> _zk_server.getTxnLogFactory().commit();
> }
> {code}
> The test steps through the following scenarios in quick succession:
> Run 1: Start a 1-node cluster, servers=[0]
> Run 2: Start a 2-node cluster, servers=[0,3]
> Run 3: Start a 3-node cluster, servers=[0,1,3]
> Run 4: Start a 4-node cluster, servers=[0,1,2,3]
> Run 5: Start a 5-node cluster, servers=[0,1,2,3,4]
> It appears that run 5 never elects a leader -- the nodes just keep spewing messages like this (example from node 0):
> {noformat}
> 2012-03-14 16:23:12,775 13308 [WorkerSender[myid=0]] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - There is a connection already for server 2
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 3 (recipient), 0 (myid), 0x2 (n.peerEpoch)
> 2012-03-14 16:23:12,776 13309 [WorkerSender[myid=0]] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - There is a connection already for server 3
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - Sending Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), 4 (recipient), 0 (myid), 0x2 (n.peerEpoch)
> 2012-03-14 16:23:12,776 13309 [WorkerSender[myid=0]] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - There is a connection already for server 4
> 2012-03-14 16:23:12,776 13309 [WorkerReceiver[myid=0]] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - Receive new notification message. My id = 0
> 2012-03-14 16:23:12,776 13309 [WorkerReceiver[myid=0]] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
> 2012-03-14 16:23:12,776 13309 [WorkerReceiver[myid=0]] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - Receive new notification message. My id = 0
> 2012-03-14 16:23:12,776 13309 [WorkerReceiver[myid=0]] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - Adding vote: from=1, proposed leader=3, proposed zxid=0x0, proposed election epoch=0x1
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - id: 3, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - id: 3, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - id: 3, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - id: 4, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
> 2012-03-14 16:23:12,776 13309 [QuorumPeer[myid=0]/127.0.0.1:2900] DEBUG org.apache.zookeeper.server.quorum.FastLeaderElection - id: 4, proposed id: 3, zxid: 0x0, proposed zxid: 0x0
> {noformat}
> I'm guessing this means that nodes 3 and 4 are fighting over leadership, but I don't know enough about the leader election code to debug this any further. Attaching a tarball with the logs for each run and the data directories for each node (though I don't think any data is being written to ZK during the test).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira