You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Mate Szalay-Beko (Jira)" <ji...@apache.org> on 2020/04/01 06:58:00 UTC
[jira] [Commented] (ZOOKEEPER-3776) Cluster stuck not forming up
quorum
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072446#comment-17072446 ]
Mate Szalay-Beko commented on ZOOKEEPER-3776:
---------------------------------------------
Hi Philip,
As far as I can tell by the first look the logs suggest that all nodes think that they are the leader. This can be either a bug or a config issue.
Are you using 0.0.0.0 in the config files? I think this (combined with some network config) can cause the issue. The fix is already made but not released yet. But there are workarounds: here is an example how to change the config: https://issues.apache.org/jira/browse/ZOOKEEPER-3725?focusedCommentId=17041124&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17041124
If 0.0.0.0 is not the case, then I can look into it deeper. Please attach all the three log files (DEBUG log level could also help a lot), and please attach the zoo.cfg or all the three zoo.cfg config files (if they are different).
Are you using real hosts or some dockerized environment?
Can you also share the JDK and OS versions?
Thanks!
> Cluster stuck not forming up quorum
> ------------------------------------
>
> Key: ZOOKEEPER-3776
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3776
> Project: ZooKeeper
> Issue Type: Bug
> Reporter: Philipp
> Assignee: Mate Szalay-Beko
> Priority: Major
>
> We had working Zookeeper 3.4.8 that came with Kafka. 3 machines cluster. Now I'm trying to upgrade it to 3.6.0. I didn't touch the data directory, just set up 3.6.0 Zookeeper in the new directory and updated config. Stopped all Zookeepers. Now I've started eache Zk in turn with some delay. And they are not forming up quorum, stuck seemingly forever, sometimes trying to elect and sometimes throwing nonsensical errors.
> For example, on node 1 (sorry don't know how to format this properly in jira):
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,133 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,133 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,134 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,335 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@857] - Peer state changed: following
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1453] - FOLLOWING
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1246] - minSessionTimeout set to 6000
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1255] - maxSessionTimeout set to 60000
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ResponseCache@45] - Response cache size is initialized with value 400.
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@111] - zookeeper.pathStats.slotCapacity = 60
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@112] - zookeeper.pathStats.slotDuration = 15
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@113] - zookeeper.pathStats.maxDepth = 6
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@114] - zookeeper.pathStats.initialDelay = 5
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@115] - zookeeper.pathStats.delay = 5
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@116] - zookeeper.pathStats.enabled = false
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1470] - The max bytes for all large requests are set to 104857600
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1484] - The large request threshold is set to -1
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@329] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir /kafka/kafka_zookeeper_data/logdir/version-2 snapdir /kafka/kafka_zookeeper_data/version-2
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@75] - FOLLOWING - LEADER ELECTION TOOK - 92790 MS
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: following - discovery
> Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,736 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:FOLLOWING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,538 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:FOLLOWING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - WARN [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@129] - Exception when following the leader
> Mar 31 20:05:22 java[24055]: java.lang.IllegalArgumentException
> Mar 31 20:05:22 java[24055]: at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307)
> Mar 31 20:05:22 java[24055]: at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1195)
> Mar 31 20:05:22 java[24055]: at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:89)
> Mar 31 20:05:22 java[24055]: at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:275)
> Mar 31 20:05:22 java[24055]: at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87)
> Mar 31 20:05:22 java[24055]: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1455)
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@292] - shutdown Follower
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: looking
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - WARN [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1501] - PeerState set to LOOKING
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1371] - LOOKING
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@931] - New election. My id = 1, proposed zxid=0x10002b653
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,542 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,544 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,744 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,744 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 400
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,745 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,746 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0{code}
> ----------
> On node 2 in the same time:
> Mar 31 20:04:39 java[23556]: 2020-03-31 20:04:39,764 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,130 [myid:2] - WARN [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1471] - Unexpected exception
> Mar 31 20:05:21 java[23556]: java.lang.InterruptedException: Timeout while waiting for epoch from quorum
> Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader.getEpochToPropose(Leader.java:1425)
> Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:599)
> Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1468)
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@792] - Shutting down
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@798] - Shutdown called. For the reason Forcing shutdown
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: looking
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - WARN [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1501] - PeerState set to LOOKING
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1371] - LOOKING
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - WARN [LearnerCnxAcceptorHandler-/172.30.2.252:2888:Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler@523] - Exception while shutting down acceptor.
> Mar 31 20:05:21 java[23556]: java.net.SocketException: Socket closed
> Mar 31 20:05:21 java[23556]: at java.net.PlainSocketImpl.socketAccept(Native Method)
> Mar 31 20:05:21 java[23556]: at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> Mar 31 20:05:21 java[23556]: at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> Mar 31 20:05:21 java[23556]: at java.net.ServerSocket.accept(ServerSocket.java:513)
> Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler.acceptConnections(Leader.java:510)
> Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler.run(Leader.java:493)
> Mar 31 20:05:21 java[23556]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> Mar 31 20:05:21 java[23556]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Mar 31 20:05:21 java[23556]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> Mar 31 20:05:21 java[23556]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> Mar 31 20:05:21 java[23556]: at java.lang.Thread.run(Thread.java:745)
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@931] - New election. My id = 2, proposed zxid=0x10002b653
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,133 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,134 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,134 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,335 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 400
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,335 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,336 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,736 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 800
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,736 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,737 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,737 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,538 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 1600
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,538 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@857] - Peer state changed: leading
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1465] - LEADING
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1246] - minSessionTimeout set to 6000
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1255] - maxSessionTimeout set to 60000
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ResponseCache@45] - Response cache size is initialized with value 400.
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@111] - zookeeper.pathStats.slotCapacity = 60
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@112] - zookeeper.pathStats.slotDuration = 15
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@113] - zookeeper.pathStats.maxDepth = 6
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,540 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,540 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@114] - zookeeper.pathStats.initialDelay = 5
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@115] - zookeeper.pathStats.delay = 5
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@116] - zookeeper.pathStats.enabled = false
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1470] - The max bytes for all large requests are set to 104857600
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1484] - The large request threshold is set to -1
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@329] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir /kafka/kafka_zookeeper_data/logdir/version-2 snapdir /kafka/kafka_zookeeper_data/version-2
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@581] - LEADING - LEADER ELECTION TOOK - 1410 MS
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: leading - discovery
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FileTxnSnapLog@470] - Snapshotting: 0x10002b653 to /kafka/kafka_zookeeper_data/version-2/snapshot.10002b653
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,568 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@519] - Snapshot taken in 26 ms
> ----------------------
> node 3:
> Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,133 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,134 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,335 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,736 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,540 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,544 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,745 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
> Mar 31 20:05:23 java[20867]: 2020-03-31 20:05:23,147 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)