You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Sebastian Schmitz (Jira)" <ji...@apache.org> on 2019/10/29 19:17:40 UTC
[jira] [Comment Edited] (ZOOKEEPER-2164) fast leader election keeps
failing
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962332#comment-16962332 ]
Sebastian Schmitz edited comment on ZOOKEEPER-2164 at 10/29/19 7:16 PM:
------------------------------------------------------------------------
Hello,
also attaching my logs here....
When I start node in order 1-2-3 it works. If I start it like 3-2-1 the election times out for one node and everything running in circles...
Starting 1-2-3:
{code:java}
// Oct 30 07:31:07 zookeeper_node_1: setting dataDir=/mnt/zk_dataOct 30 07:31:07 zookeeper_node_1: setting dataDir=/mnt/zk_dataOct 30 07:31:07 zookeeper_node_1: setting dataLogDir=/mnt/zk_data_logOct 30 07:31:07 zookeeper_node_1: setting server.1=0.0.0.0:2888:3888Oct 30 07:31:07 zookeeper_node_1: setting server.2=node02.kafka.net:2888:3888Oct 30 07:31:07 zookeeper_node_1: setting server.3=node03.kafka.net:2888:3888Oct 30 07:31:07 zookeeper_node_1: echo 1 > /mnt/zk_data/myidOct 30 07:31:07 zookeeper_node_1: setting ZOO_LOG4J_PROP=INFO,CONSOLEOct 30 07:31:07 zookeeper_node_1: ZooKeeper JMX enabled by defaultOct 30 07:31:07 zookeeper_node_1: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:31:09 zookeeper_node_1: Oct 29, 2019 18:31:08 +0000 [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"Oct 30 07:31:09 zookeeper_node_1: Oct 29, 2019 18:31:09 +0000 [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.comOct 30 07:31:09 zookeeper_node_1: Oct 29, 2019 18:31:09 +0000 [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.logOct 30 07:31:14 zookeeper_node_2: setting dataDir=/mnt/zk_dataOct 30 07:31:14 zookeeper_node_2: setting dataLogDir=/mnt/zk_data_logOct 30 07:31:14 zookeeper_node_2: setting server.1=node01.kafka.net:2888:3888Oct 30 07:31:14 zookeeper_node_2: setting server.2=0.0.0.0:2888:3888Oct 30 07:31:14 zookeeper_node_2: setting server.3=node03.kafka.net:2888:3888Oct 30 07:31:14 zookeeper_node_2: echo 2 > /mnt/zk_data/myidOct 30 07:31:14 zookeeper_node_2: setting ZOO_LOG4J_PROP=INFO,CONSOLEOct 30 07:31:14 zookeeper_node_2: ZooKeeper JMX enabled by defaultOct 30 07:31:14 zookeeper_node_2: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:31:16 zookeeper_node_2: Oct 29, 2019 18:31:16 +0000 [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"Oct 30 07:31:16 zookeeper_node_2: Oct 29, 2019 18:31:16 +0000 [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.comOct 30 07:31:16 zookeeper_node_2: Oct 29, 2019 18:31:16 +0000 [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.logOct 30 07:31:19 zookeeper_node_1: 2019-10-29 18:31:19,751 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:31:19 zookeeper_node_1: 2019-10-29 18:31:19,774 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181Oct 30 07:31:19 zookeeper_node_1: 2019-10-29 18:31:19,774 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not setOct 30 07:31:19 zookeeper_node_1: 2019-10-29 18:31:19,830 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3Oct 30 07:31:19 zookeeper_node_1: 2019-10-29 18:31:19,831 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0Oct 30 07:31:19 zookeeper_node_1: 2019-10-29 18:31:19,832 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.Oct 30 07:31:19 zookeeper_node_1: 2019-10-29 18:31:19,833 [myid:1] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.Oct 30 07:31:20 zookeeper_node_1: 2019-10-29 18:31:19,897 [myid:1] - INFO [main:QuorumPeerMain@141] - Starting quorum peerOct 30 07:31:20 zookeeper_node_3: setting dataDir=/mnt/zk_dataOct 30 07:31:20 zookeeper_node_3: setting dataLogDir=/mnt/zk_data_logOct 30 07:31:20 zookeeper_node_3: setting server.1=node01.kafka.net:2888:3888Oct 30 07:31:20 zookeeper_node_3: setting server.2=node02.kafka.net:2888:3888Oct 30 07:31:20 zookeeper_node_3: setting server.3=0.0.0.0:2888:3888Oct 30 07:31:20 zookeeper_node_3: echo 3 > /mnt/zk_data/myidOct 30 07:31:20 zookeeper_node_3: setting ZOO_LOG4J_PROP=INFO,CONSOLEOct 30 07:31:20 zookeeper_node_3: ZooKeeper JMX enabled by defaultOct 30 07:31:20 zookeeper_node_3: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:31:20 zookeeper_node_1: 2019-10-29 18:31:20,298 [myid:1] - INFO [main:Log@193] - Logging initialized @12725ms to org.eclipse.jetty.util.log.Slf4jLogOct 30 07:31:20 zookeeper_node_1: 2019-10-29 18:31:20,820 [myid:1] - WARN [main:ContextHandler@1588] - o.e.j.s.ServletContextHandler@1dba4e06{/,null,UNAVAILABLE} contextPath ends with /*Oct 30 07:31:20 zookeeper_node_1: 2019-10-29 18:31:20,821 [myid:1] - WARN [main:ContextHandler@1599] - Empty contextPathOct 30 07:31:20 zookeeper_node_3: Oct 29, 2019 18:31:20 +0000 [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,108 [myid:1] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiationOct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,126 [myid:1] - INFO [main:FileTxnSnapLog@103] - zookeeper.snapshot.trust.empty : falseOct 30 07:31:21 zookeeper_node_3: Oct 29, 2019 18:31:21 +0000 [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.comOct 30 07:31:21 zookeeper_node_3: Oct 29, 2019 18:31:21 +0000 [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.logOct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,781 [myid:1] - INFO [main:QuorumPeer@1488] - Local sessions disabledOct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,782 [myid:1] - INFO [main:QuorumPeer@1499] - Local session upgrading disabledOct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,782 [myid:1] - INFO [main:QuorumPeer@1466] - tickTime set to 2000Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,782 [myid:1] - INFO [main:QuorumPeer@1510] - minSessionTimeout set to 4000Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,782 [myid:1] - INFO [main:QuorumPeer@1521] - maxSessionTimeout set to 40000Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,782 [myid:1] - INFO [main:QuorumPeer@1536] - initLimit set to 10Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,804 [myid:1] - INFO [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,807 [myid:1] - INFO [main:QuorumPeer@1781] - Using insecure (non-TLS) quorum communicationOct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,807 [myid:1] - INFO [main:QuorumPeer@1787] - Port unification disabledOct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,807 [myid:1] - INFO [main:QuorumPeer@2154] - QuorumPeer communication is not secured! (SASL auth disabled)Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,807 [myid:1] - INFO [main:QuorumPeer@2183] - quorum.cnxn.threads.size set to 20Oct 30 07:31:21 zookeeper_node_1: 2019-10-29 18:31:21,866 [myid:1] - INFO [main:FileSnap@83] - Reading snapshot /mnt/zk_data/version-2/snapshot.6e005ab0a6Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,230 [myid:1] - INFO [main:Server@370] - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 1.8.0_232-b09Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,381 [myid:1] - INFO [main:DefaultSessionIdManager@365] - DefaultSessionIdManager workerName=node0Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,381 [myid:1] - INFO [main:DefaultSessionIdManager@370] - No SessionScavenger set, using defaultsOct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,385 [myid:1] - INFO [main:HouseKeeper@149] - node0 Scavenging every 600000msOct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,504 [myid:1] - INFO [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@1dba4e06{/,null,AVAILABLE}Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,539 [myid:1] - INFO [main:AbstractConnector@292] - Started ServerConnector@165e389b{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,539 [myid:1] - INFO [main:Server@410] - Started @16967msOct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,539 [myid:1] - INFO [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commandsOct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,568 [myid:1] - INFO [main:QuorumCnxManager$Listener@861] - Election port bind maximum retries is 3Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,582 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@911] - My election bind port: /0.0.0.0:3888Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,614 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1193] - LOOKINGOct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,616 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@885] - New election. My id = 1, proposed zxid=0x7100000f25Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,633 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:31:24 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:24 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:24 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:24 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:31:24 zookeeper_node_1: at java.lang.Thread.run(Thread.java:748)Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,651 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@679] - Cannot open channel to 3 at election address node03.kafka.net/10.24.8.29:3888Oct 30 07:31:24 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:24 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:24 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:24 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:31:24 zookeeper_node_1: at java.lang.Thread.run(Thread.java:748)Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,652 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 1 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x71 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,856 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:31:24 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:24 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:24 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:24 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,859 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 3 at election address node03.kafka.net/10.24.8.29:3888Oct 30 07:31:24 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:24 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:24 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:24 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:24 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:24 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:24 zookeeper_node_1: 2019-10-29 18:31:24,860 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 400Oct 30 07:31:25 zookeeper_node_1: 2019-10-29 18:31:25,262 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:31:25 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:25 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:25 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:25 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:25 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:25 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:25 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:25 zookeeper_node_1: 2019-10-29 18:31:25,264 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 3 at election address node03.kafka.net/10.24.8.29:3888Oct 30 07:31:25 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:25 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:25 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:25 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:25 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:25 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:25 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:25 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:25 zookeeper_node_1: 2019-10-29 18:31:25,265 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 800Oct 30 07:31:26 zookeeper_node_1: 2019-10-29 18:31:26,066 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:31:26 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:26 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:26 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:26 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:26 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:26 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:26 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:26 zookeeper_node_1: 2019-10-29 18:31:26,073 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 3 at election address node03.kafka.net/10.24.8.29:3888Oct 30 07:31:26 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:26 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:26 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:26 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:26 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:26 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:26 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:26 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:26 zookeeper_node_1: 2019-10-29 18:31:26,073 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 1600Oct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,386 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,408 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181Oct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,409 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not setOct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,477 [myid:2] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3Oct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,478 [myid:2] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0Oct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,478 [myid:2] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.Oct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,479 [myid:2] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.Oct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,529 [myid:2] - INFO [main:QuorumPeerMain@141] - Starting quorum peerOct 30 07:31:27 zookeeper_node_1: 2019-10-29 18:31:27,675 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:31:27 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:27 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:27 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:27 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:27 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:27 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:27 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:27 zookeeper_node_1: 2019-10-29 18:31:27,677 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 3 at election address node03.kafka.net/10.24.8.29:3888Oct 30 07:31:27 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:27 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:27 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:27 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:27 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:27 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:27 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:31:27 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:31:27 zookeeper_node_1: 2019-10-29 18:31:27,678 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 3200Oct 30 07:31:27 zookeeper_node_2: 2019-10-29 18:31:27,893 [myid:2] - INFO [main:Log@193] - Logging initialized @13666ms to org.eclipse.jetty.util.log.Slf4jLogOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,285 [myid:2] - WARN [main:ContextHandler@1588] - o.e.j.s.ServletContextHandler@65bdd558{/,null,UNAVAILABLE} contextPath ends with /*Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,285 [myid:2] - WARN [main:ContextHandler@1599] - Empty contextPathOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,544 [myid:2] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiationOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,559 [myid:2] - INFO [main:FileTxnSnapLog@103] - zookeeper.snapshot.trust.empty : falseOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,636 [myid:2] - INFO [main:QuorumPeer@1488] - Local sessions disabledOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,637 [myid:2] - INFO [main:QuorumPeer@1499] - Local session upgrading disabledOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,637 [myid:2] - INFO [main:QuorumPeer@1466] - tickTime set to 2000Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,637 [myid:2] - INFO [main:QuorumPeer@1510] - minSessionTimeout set to 4000Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,637 [myid:2] - INFO [main:QuorumPeer@1521] - maxSessionTimeout set to 40000Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,637 [myid:2] - INFO [main:QuorumPeer@1536] - initLimit set to 10Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,733 [myid:2] - INFO [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,747 [myid:2] - INFO [main:QuorumPeer@1781] - Using insecure (non-TLS) quorum communicationOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,748 [myid:2] - INFO [main:QuorumPeer@1787] - Port unification disabledOct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,748 [myid:2] - INFO [main:QuorumPeer@2154] - QuorumPeer communication is not secured! (SASL auth disabled)Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,748 [myid:2] - INFO [main:QuorumPeer@2183] - quorum.cnxn.threads.size set to 20Oct 30 07:31:28 zookeeper_node_2: 2019-10-29 18:31:28,792 [myid:2] - INFO [main:FileSnap@83] - Reading snapshot /mnt/zk_data/version-2/snapshot.7100000f25Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,237 [myid:2] - INFO [main:Server@370] - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 1.8.0_232-b09Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,356 [myid:2] - INFO [main:DefaultSessionIdManager@365] - DefaultSessionIdManager workerName=node0Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,357 [myid:2] - INFO [main:DefaultSessionIdManager@370] - No SessionScavenger set, using defaultsOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,361 [myid:2] - INFO [main:HouseKeeper@149] - node0 Scavenging every 600000msOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,523 [myid:2] - INFO [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@65bdd558{/,null,AVAILABLE}Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,555 [myid:2] - INFO [main:AbstractConnector@292] - Started ServerConnector@20a05b32{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,555 [myid:2] - INFO [main:Server@410] - Started @16328msOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,556 [myid:2] - INFO [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commandsOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,572 [myid:2] - INFO [main:QuorumCnxManager$Listener@861] - Election port bind maximum retries is 3Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,585 [myid:2] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@911] - My election bind port: /0.0.0.0:3888Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,627 [myid:1] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:39402Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,618 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1193] - LOOKINGOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,622 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@885] - New election. My id = 2, proposed zxid=0x7100000f25Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,643 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x72 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,645 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@679] - Cannot open channel to 3 at election address node03.kafka.net/10.24.8.29:3888Oct 30 07:31:30 zookeeper_node_2: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:30 zookeeper_node_2: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:30 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:30 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:30 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:30 zookeeper_node_2: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:30 zookeeper_node_2: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:30 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:30 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:30 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:31:30 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,647 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@679] - Cannot open channel to 3 at election address node03.kafka.net/10.24.8.29:3888Oct 30 07:31:30 zookeeper_node_1: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:31:30 zookeeper_node_1: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:31:30 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:31:30 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:31:30 zookeeper_node_2: at java.lang.Thread.run(Thread.java:748)Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,652 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x72 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,664 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 1 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x71 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,665 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x72 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:30 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:31:30 zookeeper_node_1: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:31:30 zookeeper_node_1: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:31:30 zookeeper_node_1: at java.net.Socket.connect(Socket.java:607)Oct 30 07:31:30 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:31:30 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:31:30 zookeeper_node_1: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:31:30 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:31:30 zookeeper_node_1: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:31:30 zookeeper_node_1: at java.lang.Thread.run(Thread.java:748)Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,649 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x72 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,854 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1269] - FOLLOWINGOct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,881 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@91] - TCP NoDelay set to: trueOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,868 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1281] - LEADINGOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,874 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Leader@66] - TCP NoDelay set to: trueOct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,874 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Leader@86] - zookeeper.leader.maxConcurrentSnapshots = 10Oct 30 07:31:30 zookeeper_node_2: 2019-10-29 18:31:30,875 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Leader@88] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,898 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMTOct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,898 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:host.name=zookeeper_node_1Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,901 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_232Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,901 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle CorporationOct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,901 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/local/openjdk-8Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,901 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-jute-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-unix-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-epoll-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-resolver-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-handler-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-codec-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-buffer-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jline-2.11.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-util-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-servlet-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-server-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-security-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-io-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-http-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-databind-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-core-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-annotations-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf::/opt/zookeeper-cluster/newrelic/newrelic.jarOct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,901 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/libOct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,901 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmpOct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,902 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,902 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.name=LinuxOct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,902 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64Oct 30 07:31:30 zookeeper_node_1: 2019-10-29 18:31:30,902 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.version=3.10.0-862.2.3.el7.x86_64Oct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,902 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.name=rootOct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,912 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/rootOct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,912 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/Oct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,912 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=231MBOct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,913 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=889MBOct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,913 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=461MBOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,910 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMTOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,910 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:host.name=zookeeper_node_2Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,910 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_232Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,910 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle CorporationOct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,917 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000Oct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,918 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000Oct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,918 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /mnt/zk_data_log/version-2 snapdir /mnt/zk_data/version-2Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,911 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/local/openjdk-8Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,911 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-jute-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-unix-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-epoll-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-resolver-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-handler-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-codec-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-buffer-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jline-2.11.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-util-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-servlet-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-server-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-security-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-io-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-http-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-databind-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-core-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-annotations-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf::/opt/zookeeper-cluster/newrelic/newrelic.jarOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,911 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/libOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmpOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.name=LinuxOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.version=3.10.0-693.21.1.el7.x86_64Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.name=rootOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/rootOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,912 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=304MBOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,913 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=889MBOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,913 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=484MBOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,924 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,924 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,927 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /mnt/zk_data_log/version-2 snapdir /mnt/zk_data/version-2Oct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,929 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Leader@464] - LEADING - LEADER ELECTION TOOK - 61 MSOct 30 07:31:31 zookeeper_node_2: 2019-10-29 18:31:30,933 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FileTxnSnapLog@384] - Snapshotting: 0x7100000f25 to /mnt/zk_data/version-2/snapshot.7100000f25Oct 30 07:31:31 zookeeper_node_1: 2019-10-29 18:31:30,919 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 65 MSOct 30 07:31:32 zookeeper_node_2: 2019-10-29 18:31:31,951 [myid:2] - INFO [LearnerHandler-/10.24.8.27:34708:LearnerHandler@406] - Follower sid: 1 : info : node01.kafka.net:2888:3888:participantOct 30 07:31:33 zookeeper_node_1: 2019-10-29 18:31:33,119 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@391] - Getting a diff from the leader 0x7100000f25Oct 30 07:31:33 zookeeper_node_1: 2019-10-29 18:31:33,127 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@546] - Learner received NEWLEADER messageOct 30 07:31:33 zookeeper_node_2: 2019-10-29 18:31:33,114 [myid:2] - INFO [LearnerHandler-/10.24.8.27:34708:ZKDatabase@295] - On disk txn sync enabled with snapshotSizeFactor 0.33Oct 30 07:31:33 zookeeper_node_2: 2019-10-29 18:31:33,115 [myid:2] - INFO [LearnerHandler-/10.24.8.27:34708:LearnerHandler@708] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x7100000f25 peerLastZxid=0x7100000f25Oct 30 07:31:33 zookeeper_node_2: 2019-10-29 18:31:33,115 [myid:2] - INFO [LearnerHandler-/10.24.8.27:34708:LearnerHandler@752] - Sending DIFF zxid=0x7100000f25 for peer sid: 1Oct 30 07:31:33 zookeeper_node_2: 2019-10-29 18:31:33,183 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):Leader@1296] - Have quorum of supporters, sids: [ [1, 2],[1, 2] ]; starting up and setting last processed zxid: 0x7300000000Oct 30 07:31:33 zookeeper_node_2: 2019-10-29 18:31:33,219 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 4 worker threads.Oct 30 07:31:33 zookeeper_node_2: 2019-10-29 18:31:33,308 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000Oct 30 07:31:33 zookeeper_node_1: 2019-10-29 18:31:33,337 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@529] - Learner received UPTODATE messageOct 30 07:31:33 zookeeper_node_1: 2019-10-29 18:31:33,356 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 4 worker threads.Oct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,626 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,643 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181Oct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,643 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not setOct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,690 [myid:3] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3Oct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,692 [myid:3] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0Oct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,692 [myid:3] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.Oct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,693 [myid:3] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.Oct 30 07:31:33 zookeeper_node_3: 2019-10-29 18:31:33,741 [myid:3] - INFO [main:QuorumPeerMain@141] - Starting quorum peerOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,131 [myid:3] - INFO [main:Log@193] - Logging initialized @15446ms to org.eclipse.jetty.util.log.Slf4jLogOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,504 [myid:3] - WARN [main:ContextHandler@1588] - o.e.j.s.ServletContextHandler@65bdd558{/,null,UNAVAILABLE} contextPath ends with /*Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,504 [myid:3] - WARN [main:ContextHandler@1599] - Empty contextPathOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,726 [myid:3] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiationOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,736 [myid:3] - INFO [main:FileTxnSnapLog@103] - zookeeper.snapshot.trust.empty : falseOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,784 [myid:3] - INFO [main:QuorumPeer@1488] - Local sessions disabledOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,785 [myid:3] - INFO [main:QuorumPeer@1499] - Local session upgrading disabledOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,785 [myid:3] - INFO [main:QuorumPeer@1466] - tickTime set to 2000Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,785 [myid:3] - INFO [main:QuorumPeer@1510] - minSessionTimeout set to 4000Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,785 [myid:3] - INFO [main:QuorumPeer@1521] - maxSessionTimeout set to 40000Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,786 [myid:3] - INFO [main:QuorumPeer@1536] - initLimit set to 10Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,824 [myid:3] - INFO [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,832 [myid:3] - INFO [main:QuorumPeer@1781] - Using insecure (non-TLS) quorum communicationOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,833 [myid:3] - INFO [main:QuorumPeer@1787] - Port unification disabledOct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,833 [myid:3] - INFO [main:QuorumPeer@2154] - QuorumPeer communication is not secured! (SASL auth disabled)Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,833 [myid:3] - INFO [main:QuorumPeer@2183] - quorum.cnxn.threads.size set to 20Oct 30 07:31:34 zookeeper_node_3: 2019-10-29 18:31:34,881 [myid:3] - INFO [main:FileSnap@83] - Reading snapshot /mnt/zk_data/version-2/snapshot.7100000f25Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,023 [myid:3] - INFO [main:Server@370] - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 1.8.0_232-b09Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,206 [myid:3] - INFO [main:DefaultSessionIdManager@365] - DefaultSessionIdManager workerName=node0Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,207 [myid:3] - INFO [main:DefaultSessionIdManager@370] - No SessionScavenger set, using defaultsOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,215 [myid:3] - INFO [main:HouseKeeper@149] - node0 Scavenging every 600000msOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,348 [myid:3] - INFO [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@65bdd558{/,null,AVAILABLE}Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,373 [myid:3] - INFO [main:AbstractConnector@292] - Started ServerConnector@20a05b32{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,374 [myid:3] - INFO [main:Server@410] - Started @17689msOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,374 [myid:3] - INFO [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commandsOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,388 [myid:3] - INFO [main:QuorumCnxManager$Listener@861] - Election port bind maximum retries is 3Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,393 [myid:3] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@911] - My election bind port: /0.0.0.0:3888Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,412 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1193] - LOOKINGOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,418 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@885] - New election. My id = 3, proposed zxid=0x7100000f25Oct 30 07:31:36 zookeeper_node_1: 2019-10-29 18:31:36,427 [myid:1] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.29:40716Oct 30 07:31:36 zookeeper_node_1: 2019-10-29 18:31:36,436 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x72 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)Oct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,440 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.29:47080Oct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,444 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x72 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,458 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x72 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,458 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x72 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,459 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,459 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x72 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,466 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LEADING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,472 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1269] - FOLLOWINGOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,481 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Learner@91] - TCP NoDelay set to: trueOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,492 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMTOct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,508 [myid:2] - INFO [LearnerHandler-/10.24.8.29:35546:LearnerHandler@406] - Follower sid: 3 : info : node03.kafka.net:2888:3888:participantOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,492 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:host.name=zookeeper_node_3Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,492 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_232Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,492 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle CorporationOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,492 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/local/openjdk-8Oct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,550 [myid:2] - INFO [LearnerHandler-/10.24.8.29:35546:ZKDatabase@295] - On disk txn sync enabled with snapshotSizeFactor 0.33Oct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,551 [myid:2] - INFO [LearnerHandler-/10.24.8.29:35546:LearnerHandler@708] - Synchronizing with Follower sid: 3 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x7300000000 peerLastZxid=0x7100000f25Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,492 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-jute-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-unix-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-epoll-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-resolver-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-handler-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-codec-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-buffer-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jline-2.11.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-util-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-servlet-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-server-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-security-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-io-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-http-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-databind-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-core-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-annotations-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf::/opt/zookeeper-cluster/newrelic/newrelic.jarOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/libOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmpOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.name=LinuxOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.version=3.10.0-693.11.6.el7.x86_64Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.name=rootOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/rootOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,493 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,494 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=298MBOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,494 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=889MBOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,494 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=474MBOct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,497 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,497 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,498 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /mnt/zk_data_log/version-2 snapdir /mnt/zk_data/version-2Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,498 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 26 MSOct 30 07:31:36 zookeeper_node_1: 2019-10-29 18:31:36,572 [myid:1] - INFO [NIOWorkerThread-4:ZooKeeperServer@1032] - Refusing session request for client /10.24.8.28:56662 as it has seen zxid 0x7200000000 our last zxid is 0x7100000f25 client must try another serverOct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,594 [myid:2] - INFO [LearnerHandler-/10.24.8.29:35546:LearnerHandler@784] - Use txnlog and committedLog for peer sid: 3Oct 30 07:31:36 zookeeper_node_3: 2019-10-29 18:31:36,691 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Learner@395] - Getting a snapshot from leader 0x7300000000Oct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,687 [myid:2] - ERROR [LearnerHandler-/10.24.8.29:35546:LearnerHandler@812] - Unhandled scenario for peer sid: 3 fall back to use snapshotOct 30 07:31:36 zookeeper_node_2: 2019-10-29 18:31:36,689 [myid:2] - INFO [LearnerHandler-/10.24.8.29:35546:LearnerHandler@463] - Sending snapshot last zxid of peer is 0x7100000f25, zxid of leader is 0x7300000000, send zxid of db as 0x7300000000, 1 concurrent snapshots, snapshot was exempt from throttleOct 30 07:31:37 zookeeper_node_1: 2019-10-29 18:31:36,979 [myid:1] - INFO [NIOWorkerThread-5:ZooKeeperServer@1032] - Refusing session request for client /10.24.8.29:36698 as it has seen zxid 0x7200000000 our last zxid is 0x7100000f25 client must try another serverOct 30 07:31:38 zookeeper_node_3: 2019-10-29 18:31:38,039 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Learner@546] - Learner received NEWLEADER messageOct 30 07:31:38 zookeeper_node_3: 2019-10-29 18:31:38,113 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FileTxnSnapLog@384] - Snapshotting: 0x7300000000 to /mnt/zk_data/version-2/snapshot.7300000000Oct 30 07:31:39 zookeeper_node_3: 2019-10-29 18:31:39,658 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Learner@529] - Learner received UPTODATE messageOct 30 07:31:39 zookeeper_node_3: 2019-10-29 18:31:39,671 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 4 worker threads.
{code}
Starting 3-2-1:
{code:java}
// Oct 30 07:34:22 zookeeper_node_3: setting dataLogDir=/mnt/zk_data_logOct 30 07:34:22 zookeeper_node_3: setting dataLogDir=/mnt/zk_data_logOct 30 07:34:22 zookeeper_node_3: setting server.1= node01.kafka.net:2888:3888Oct 30 07:34:22 zookeeper_node_3: setting server.2= node02.kafka.net:2888:3888Oct 30 07:34:22 zookeeper_node_3: setting server.3=0.0.0.0:2888:3888Oct 30 07:34:22 zookeeper_node_3: echo 3 > /mnt/zk_data/myidOct 30 07:34:22 zookeeper_node_3: setting ZOO_LOG4J_PROP=INFO,CONSOLEOct 30 07:34:22 zookeeper_node_3: ZooKeeper JMX enabled by defaultOct 30 07:34:22 zookeeper_node_3: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:34:23 zookeeper_node_3: Oct 29, 2019 18:34:22 +0000 [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"Oct 30 07:34:23 zookeeper_node_3: Oct 29, 2019 18:34:23 +0000 [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.comOct 30 07:34:23 zookeeper_node_3: Oct 29, 2019 18:34:23 +0000 [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.logOct 30 07:34:29 zookeeper_node_2: setting dataDir=/mnt/zk_dataOct 30 07:34:29 zookeeper_node_2: setting dataLogDir=/mnt/zk_data_logOct 30 07:34:29 zookeeper_node_2: setting server.1= node01.kafka.net:2888:3888Oct 30 07:34:29 zookeeper_node_2: setting server.2=0.0.0.0:2888:3888Oct 30 07:34:29 zookeeper_node_2: setting server.3= node03.kafka.net:2888:3888Oct 30 07:34:29 zookeeper_node_2: echo 2 > /mnt/zk_data/myidOct 30 07:34:29 zookeeper_node_2: setting ZOO_LOG4J_PROP=INFO,CONSOLEOct 30 07:34:29 zookeeper_node_2: ZooKeeper JMX enabled by defaultOct 30 07:34:29 zookeeper_node_2: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:34:30 zookeeper_node_2: Oct 29, 2019 18:34:30 +0000 [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"Oct 30 07:34:31 zookeeper_node_2: Oct 29, 2019 18:34:31 +0000 [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.comOct 30 07:34:31 zookeeper_node_2: Oct 29, 2019 18:34:31 +0000 [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.logOct 30 07:34:34 zookeeper_node_1: setting dataLogDir=/mnt/zk_data_logOct 30 07:34:34 zookeeper_node_1: setting server.1=0.0.0.0:2888:3888Oct 30 07:34:34 zookeeper_node_1: setting server.2= node02.kafka.net:2888:3888Oct 30 07:34:34 zookeeper_node_1: setting server.3= node03.kafka.net:2888:3888Oct 30 07:34:34 zookeeper_node_1: echo 1 > /mnt/zk_data/myidOct 30 07:34:34 zookeeper_node_1: setting ZOO_LOG4J_PROP=INFO,CONSOLEOct 30 07:34:34 zookeeper_node_1: ZooKeeper JMX enabled by defaultOct 30 07:34:34 zookeeper_node_1: Using config: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,180 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,194 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181Oct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,195 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not setOct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,244 [myid:3] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3Oct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,246 [myid:3] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0Oct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,246 [myid:3] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.Oct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,248 [myid:3] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.Oct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,294 [myid:3] - INFO [main:QuorumPeerMain@141] - Starting quorum peerOct 30 07:34:35 zookeeper_node_3: 2019-10-29 18:34:35,625 [myid:3] - INFO [main:Log@193] - Logging initialized @14808ms to org.eclipse.jetty.util.log.Slf4jLogOct 30 07:34:35 zookeeper_node_1: Oct 29, 2019 18:34:35 +0000 [1 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/zookeeper-cluster/newrelic/./newrelic.yml"Oct 30 07:34:36 zookeeper_node_1: Oct 29, 2019 18:34:36 +0000 [1 1] com.newrelic INFO: Using default collector host: collector.newrelic.comOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,031 [myid:3] - WARN [main:ContextHandler@1588] - o.e.j.s.ServletContextHandler@4a734c04{/,null,UNAVAILABLE} contextPath ends with /*Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,032 [myid:3] - WARN [main:ContextHandler@1599] - Empty contextPathOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,222 [myid:3] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiationOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,232 [myid:3] - INFO [main:FileTxnSnapLog@103] - zookeeper.snapshot.trust.empty : falseOct 30 07:34:36 zookeeper_node_1: Oct 29, 2019 18:34:36 +0000 [1 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/zookeeper-cluster/newrelic/logs/newrelic_agent.logOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,267 [myid:3] - INFO [main:QuorumPeer@1488] - Local sessions disabledOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,267 [myid:3] - INFO [main:QuorumPeer@1499] - Local session upgrading disabledOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,267 [myid:3] - INFO [main:QuorumPeer@1466] - tickTime set to 2000Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,268 [myid:3] - INFO [main:QuorumPeer@1510] - minSessionTimeout set to 4000Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,268 [myid:3] - INFO [main:QuorumPeer@1521] - maxSessionTimeout set to 40000Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,268 [myid:3] - INFO [main:QuorumPeer@1536] - initLimit set to 10Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,296 [myid:3] - INFO [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,310 [myid:3] - INFO [main:QuorumPeer@1781] - Using insecure (non-TLS) quorum communicationOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,311 [myid:3] - INFO [main:QuorumPeer@1787] - Port unification disabledOct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,311 [myid:3] - INFO [main:QuorumPeer@2154] - QuorumPeer communication is not secured! (SASL auth disabled)Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,311 [myid:3] - INFO [main:QuorumPeer@2183] - quorum.cnxn.threads.size set to 20Oct 30 07:34:36 zookeeper_node_3: 2019-10-29 18:34:36,373 [myid:3] - INFO [main:FileSnap@83] - Reading snapshot /mnt/zk_data/version-2/snapshot.7300000000Oct 30 07:34:37 zookeeper_node_3: 2019-10-29 18:34:37,768 [myid:3] - INFO [main:Server@370] - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 1.8.0_232-b09Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,022 [myid:3] - INFO [main:DefaultSessionIdManager@365] - DefaultSessionIdManager workerName=node0Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,023 [myid:3] - INFO [main:DefaultSessionIdManager@370] - No SessionScavenger set, using defaultsOct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,032 [myid:3] - INFO [main:HouseKeeper@149] - node0 Scavenging every 660000msOct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,272 [myid:3] - INFO [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@4a734c04{/,null,AVAILABLE}Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,309 [myid:3] - INFO [main:AbstractConnector@292] - Started ServerConnector@33d53216{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,310 [myid:3] - INFO [main:Server@410] - Started @17494msOct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,311 [myid:3] - INFO [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commandsOct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,349 [myid:3] - INFO [main:QuorumCnxManager$Listener@861] - Election port bind maximum retries is 3Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,380 [myid:3] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@911] - My election bind port: /0.0.0.0:3888Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,602 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1193] - LOOKINGOct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,605 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@885] - New election. My id = 3, proposed zxid=0x7300000000Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,618 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:38 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:38 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:38 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:38 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:34:38 zookeeper_node_3: at java.lang.Thread.run(Thread.java:748)Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,632 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:34:38 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:38 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:38 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:38 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:34:38 zookeeper_node_3: at java.lang.Thread.run(Thread.java:748)Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,636 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,849 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:38 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:38 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:38 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:38 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,851 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:34:38 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:38 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:38 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:38 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:38 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:38 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:38 zookeeper_node_3: 2019-10-29 18:34:38,851 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 400Oct 30 07:34:39 zookeeper_node_3: 2019-10-29 18:34:39,259 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:39 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:39 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:39 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:39 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:39 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:39 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:39 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:39 zookeeper_node_3: 2019-10-29 18:34:39,261 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:34:39 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:39 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:39 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:39 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:39 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:39 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:39 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:39 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:39 zookeeper_node_3: 2019-10-29 18:34:39,262 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 800Oct 30 07:34:40 zookeeper_node_3: 2019-10-29 18:34:40,063 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:40 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:40 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:40 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:40 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:40 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:40 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:40 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:40 zookeeper_node_3: 2019-10-29 18:34:40,067 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:34:40 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:40 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:40 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:40 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:40 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:40 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:40 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:40 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:40 zookeeper_node_3: 2019-10-29 18:34:40,067 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 1600Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,068 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,079 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,080 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not setOct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,140 [myid:2] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,141 [myid:2] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,142 [myid:2] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,143 [myid:2] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,200 [myid:2] - INFO [main:QuorumPeerMain@141] - Starting quorum peerOct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,429 [myid:2] - INFO [main:Log@193] - Logging initialized @12246ms to org.eclipse.jetty.util.log.Slf4jLogOct 30 07:34:41 zookeeper_node_3: 2019-10-29 18:34:41,672 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:41 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:41 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:41 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:41 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:41 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:41 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:41 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,741 [myid:2] - WARN [main:ContextHandler@1588] - o.e.j.s.ServletContextHandler@1dba4e06{/,null,UNAVAILABLE} contextPath ends with /*Oct 30 07:34:41 zookeeper_node_2: 2019-10-29 18:34:41,742 [myid:2] - WARN [main:ContextHandler@1599] - Empty contextPathOct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:41 zookeeper_node_3: 2019-10-29 18:34:41,675 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 2 at election address node02.kafka.net/10.24.8.28:3888Oct 30 07:34:41 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:41 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:41 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:41 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:41 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:41 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:41 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:41 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:41 zookeeper_node_3: 2019-10-29 18:34:41,675 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 3200Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:41,966 [myid:2] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiationOct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:41,975 [myid:2] - INFO [main:FileTxnSnapLog@103] - zookeeper.snapshot.trust.empty : falseOct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,032 [myid:2] - INFO [main:QuorumPeer@1488] - Local sessions disabledOct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,033 [myid:2] - INFO [main:QuorumPeer@1499] - Local session upgrading disabledOct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,033 [myid:2] - INFO [main:QuorumPeer@1466] - tickTime set to 2000Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,033 [myid:2] - INFO [main:QuorumPeer@1510] - minSessionTimeout set to 4000Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,033 [myid:2] - INFO [main:QuorumPeer@1521] - maxSessionTimeout set to 40000Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,033 [myid:2] - INFO [main:QuorumPeer@1536] - initLimit set to 10Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,076 [myid:2] - INFO [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,102 [myid:2] - INFO [main:QuorumPeer@1781] - Using insecure (non-TLS) quorum communicationOct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,102 [myid:2] - INFO [main:QuorumPeer@1787] - Port unification disabledOct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,102 [myid:2] - INFO [main:QuorumPeer@2154] - QuorumPeer communication is not secured! (SASL auth disabled)Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,102 [myid:2] - INFO [main:QuorumPeer@2183] - quorum.cnxn.threads.size set to 20Oct 30 07:34:42 zookeeper_node_2: 2019-10-29 18:34:42,155 [myid:2] - INFO [main:FileSnap@83] - Reading snapshot /mnt/zk_data/version-2/snapshot.7100000f25Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,360 [myid:2] - INFO [main:Server@370] - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 1.8.0_232-b09Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,552 [myid:2] - INFO [main:DefaultSessionIdManager@365] - DefaultSessionIdManager workerName=node0Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,552 [myid:2] - INFO [main:DefaultSessionIdManager@370] - No SessionScavenger set, using defaultsOct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,558 [myid:2] - INFO [main:HouseKeeper@149] - node0 Scavenging every 660000msOct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,722 [myid:2] - INFO [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@1dba4e06{/,null,AVAILABLE}Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,750 [myid:2] - INFO [main:AbstractConnector@292] - Started ServerConnector@165e389b{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,754 [myid:2] - INFO [main:Server@410] - Started @14567msOct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,755 [myid:2] - INFO [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commandsOct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,773 [myid:2] - INFO [main:QuorumCnxManager$Listener@861] - Election port bind maximum retries is 3Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,792 [myid:2] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@911] - My election bind port: /0.0.0.0:3888Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,822 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1193] - LOOKINGOct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,825 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@885] - New election. My id = 2, proposed zxid=0x7100000f25Oct 30 07:34:43 zookeeper_node_3: 2019-10-29 18:34:43,846 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42012Oct 30 07:34:43 zookeeper_node_2: 2019-10-29 18:34:43,830 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:43 zookeeper_node_2: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:43 zookeeper_node_2: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:43 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:43 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:43 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:43 zookeeper_node_3: 2019-10-29 18:34:43,865 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45584Oct 30 07:34:43 zookeeper_node_3: 2019-10-29 18:34:43,873 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:44 zookeeper_node_2: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:44 zookeeper_node_2: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:34:44 zookeeper_node_2: at java.lang.Thread.run(Thread.java:748)Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:43,843 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:43,845 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,053 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42022Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,055 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:44 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,056 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:44,050 [myid:2] - WARN [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:44 zookeeper_node_2: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:44 zookeeper_node_2: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:44 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:44 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:44 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:44 zookeeper_node_2: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:44 zookeeper_node_2: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:44,055 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:44,055 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 400Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,057 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:34:44 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:44 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:44 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:44 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,057 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:44 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:44 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,058 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,057 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,059 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45586Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,066 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,066 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:44 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,067 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:44,457 [myid:2] - WARN [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:44 zookeeper_node_2: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:44 zookeeper_node_2: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:44 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:44 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,459 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42042Oct 30 07:34:44 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:44 zookeeper_node_2: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:44 zookeeper_node_2: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:44 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,459 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:44 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:44,459 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:34:44 zookeeper_node_2: 2019-10-29 18:34:44,459 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 800Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,460 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,460 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:34:44 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:44 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:44 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:44 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:44 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,461 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,459 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:44 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:44 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,462 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,464 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:44 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:44 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:44 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,464 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45592Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,470 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:44 zookeeper_node_3: 2019-10-29 18:34:44,475 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,265 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42048Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,266 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:45 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:45 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:45 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:45 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,266 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,267 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:34:45 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:45 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:45 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:45 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:45 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:45 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:45 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:45 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:45 zookeeper_node_2: 2019-10-29 18:34:45,263 [myid:2] - WARN [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:45 zookeeper_node_2: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:45 zookeeper_node_2: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:45 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:45 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:45 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:45 zookeeper_node_2: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:45 zookeeper_node_2: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:45 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:45 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,267 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,267 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:45 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:45 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,268 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,269 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45596Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,269 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:45 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:45 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:45 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:45 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:45 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,270 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:45 zookeeper_node_3: 2019-10-29 18:34:45,273 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:45 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:45 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:45 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:45 zookeeper_node_2: 2019-10-29 18:34:45,265 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:34:45 zookeeper_node_2: 2019-10-29 18:34:45,265 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 1600Oct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,279 [myid:] - INFO [main:QuorumPeerConfig@133] - Reading configuration from: /opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfgOct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,297 [myid:] - INFO [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181Oct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,297 [myid:] - INFO [main:QuorumPeerConfig@389] - secureClientPort is not setOct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,346 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3Oct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,347 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0Oct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,347 [myid:1] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.Oct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,349 [myid:1] - INFO [main:ManagedUtil@46] - Log4j found with jmx enabled.Oct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,383 [myid:1] - INFO [main:QuorumPeerMain@141] - Starting quorum peerOct 30 07:34:46 zookeeper_node_1: 2019-10-29 18:34:46,764 [myid:1] - INFO [main:Log@193] - Logging initialized @12609ms to org.eclipse.jetty.util.log.Slf4jLogOct 30 07:34:46 zookeeper_node_3: 2019-10-29 18:34:46,869 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42064Oct 30 07:34:46 zookeeper_node_2: 2019-10-29 18:34:46,867 [myid:2] - WARN [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:46 zookeeper_node_2: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:46 zookeeper_node_2: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:46 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:46 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:46 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:46 zookeeper_node_2: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:46 zookeeper_node_2: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:46 zookeeper_node_3: 2019-10-29 18:34:46,870 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:46 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:46 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:46 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:46 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:46 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:46 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:46 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:46 zookeeper_node_3: 2019-10-29 18:34:46,870 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:34:47 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:47 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:47 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:47 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:47 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:47 zookeeper_node_2: 2019-10-29 18:34:46,868 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:34:47 zookeeper_node_2: 2019-10-29 18:34:46,869 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 3200Oct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,870 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:34:47 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:47 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:47 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:47 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:47 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:47 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:47 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:47 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:47 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,871 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:47 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:47 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:47 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,872 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,871 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,873 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45620Oct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,874 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:47 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:47 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:47 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:47 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:47 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:47 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:47 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,876 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:47 zookeeper_node_3: 2019-10-29 18:34:46,878 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:47 zookeeper_node_1: 2019-10-29 18:34:47,217 [myid:1] - WARN [main:ContextHandler@1588] - o.e.j.s.ServletContextHandler@1dba4e06{/,null,UNAVAILABLE} contextPath ends with /*Oct 30 07:34:47 zookeeper_node_1: 2019-10-29 18:34:47,217 [myid:1] - WARN [main:ContextHandler@1599] - Empty contextPathOct 30 07:34:47 zookeeper_node_1: 2019-10-29 18:34:47,533 [myid:1] - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiationOct 30 07:34:47 zookeeper_node_1: 2019-10-29 18:34:47,543 [myid:1] - INFO [main:FileTxnSnapLog@103] - zookeeper.snapshot.trust.empty : falseOct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,203 [myid:1] - INFO [main:QuorumPeer@1488] - Local sessions disabledOct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,203 [myid:1] - INFO [main:QuorumPeer@1499] - Local session upgrading disabledOct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,203 [myid:1] - INFO [main:QuorumPeer@1466] - tickTime set to 2000Oct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,203 [myid:1] - INFO [main:QuorumPeer@1510] - minSessionTimeout set to 4000Oct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,203 [myid:1] - INFO [main:QuorumPeer@1521] - maxSessionTimeout set to 40000Oct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,204 [myid:1] - INFO [main:QuorumPeer@1536] - initLimit set to 10Oct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,237 [myid:1] - INFO [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33Oct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,259 [myid:1] - INFO [main:QuorumPeer@1781] - Using insecure (non-TLS) quorum communicationOct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,259 [myid:1] - INFO [main:QuorumPeer@1787] - Port unification disabledOct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,259 [myid:1] - INFO [main:QuorumPeer@2154] - QuorumPeer communication is not secured! (SASL auth disabled)Oct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,259 [myid:1] - INFO [main:QuorumPeer@2183] - quorum.cnxn.threads.size set to 20Oct 30 07:34:48 zookeeper_node_1: 2019-10-29 18:34:48,318 [myid:1] - INFO [main:FileSnap@83] - Reading snapshot /mnt/zk_data/version-2/snapshot.6e005ab0a6Oct 30 07:34:50 zookeeper_node_2: 2019-10-29 18:34:50,072 [myid:2] - WARN [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:50 zookeeper_node_2: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:50 zookeeper_node_2: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:50 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:50 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:50 zookeeper_node_2: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:50 zookeeper_node_2: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:50 zookeeper_node_2: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:50 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:50 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,074 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42086Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,075 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:50 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:50 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:50 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:50 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,076 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:34:50 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735)Oct 30 07:34:50 zookeeper_node_2: at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)Oct 30 07:34:50 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)Oct 30 07:34:50 zookeeper_node_2: 2019-10-29 18:34:50,073 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:34:50 zookeeper_node_2: 2019-10-29 18:34:50,074 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 6400Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,076 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:34:50 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:50 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:50 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:50 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:50 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:50 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:50 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:50 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,076 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,077 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:50 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:50 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,077 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,078 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45640Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,082 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 6400Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,091 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@679] - Cannot open channel to 1 at election address node01.kafka.net/10.24.8.27:3888Oct 30 07:34:50 zookeeper_node_3: java.net.ConnectException: Connection refused (Connection refused)Oct 30 07:34:50 zookeeper_node_3: at java.net.PlainSocketImpl.socketConnect(Native Method)Oct 30 07:34:50 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)Oct 30 07:34:50 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)Oct 30 07:34:50 zookeeper_node_3: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)Oct 30 07:34:50 zookeeper_node_3: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)Oct 30 07:34:50 zookeeper_node_3: at java.net.Socket.connect(Socket.java:607)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)Oct 30 07:34:50 zookeeper_node_3: at java.lang.Thread.run(Thread.java:748)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,092 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,094 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:50 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:50 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:50 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:50 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:50 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,095 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,097 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:50 zookeeper_node_3: 2019-10-29 18:34:50,097 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:50 zookeeper_node_1: 2019-10-29 18:34:50,794 [myid:1] - INFO [main:Server@370] - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 1.8.0_232-b09Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:50,945 [myid:1] - INFO [main:DefaultSessionIdManager@365] - DefaultSessionIdManager workerName=node0Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:50,945 [myid:1] - INFO [main:DefaultSessionIdManager@370] - No SessionScavenger set, using defaultsOct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:50,950 [myid:1] - INFO [main:HouseKeeper@149] - node0 Scavenging every 600000msOct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,049 [myid:1] - INFO [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@1dba4e06{/,null,AVAILABLE}Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,076 [myid:1] - INFO [main:AbstractConnector@292] - Started ServerConnector@165e389b{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,077 [myid:1] - INFO [main:Server@410] - Started @16922msOct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,082 [myid:1] - INFO [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commandsOct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,091 [myid:1] - INFO [main:QuorumCnxManager$Listener@861] - Election port bind maximum retries is 3Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,097 [myid:1] - INFO [QuorumPeerListener:QuorumCnxManager$Listener@911] - My election bind port: /0.0.0.0:3888Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,121 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1193] - LOOKINGOct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,132 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44288Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,123 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@885] - New election. My id = 1, proposed zxid=0x7100000f25Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,132 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48622Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,134 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45646Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,135 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:51 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:51 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:51 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:51 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,136 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,136 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,129 [myid:1] - INFO [WorkerSender[myid=1]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,131 [myid:1] - INFO [WorkerSender[myid=1]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,132 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 1 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,163 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57476Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,181 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:51 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,136 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:34:51 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:51 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,136 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,137 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,137 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,139 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,144 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,336 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48630Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,343 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44296Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,342 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:34:51 zookeeper_node_2: java.io.EOFExceptionOct 30 07:34:51 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,342 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,340 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,348 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 3, error = Oct 30 07:34:51 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:51 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:51 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:51 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,349 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,349 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,349 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 3Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,350 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:51 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:51 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,346 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,347 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 400Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,348 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57478Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,348 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,349 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,350 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:34:51 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:51 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:51 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:51 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,350 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,350 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,350 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,350 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45648Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,359 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,359 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,360 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,351 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,368 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,748 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,750 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44300Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,750 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:34:51 zookeeper_node_1: 2019-10-29 18:34:51,750 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 800Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,751 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,751 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 3Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,752 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45654Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,751 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 3, error = Oct 30 07:34:51 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:51 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:51 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:51 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:51 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,753 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,753 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,751 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:51 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:51 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:51 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_3: 2019-10-29 18:34:51,758 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,750 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48634Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,751 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:34:51 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:51 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:51 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:51 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:51 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,751 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,752 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,752 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,752 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:34:51 zookeeper_node_2: java.io.EOFExceptionOct 30 07:34:51 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,752 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,761 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57480Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,761 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:51 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:51 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:51 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,767 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:34:51 zookeeper_node_2: 2019-10-29 18:34:51,769 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:51,757 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:51,760 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,555 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44310Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,556 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 3, error = Oct 30 07:34:52 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:52 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:52 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:52 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:52 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:52 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:52 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:52 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,557 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,557 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:52 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:52 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:52 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:52 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:52 zookeeper_node_1: 2019-10-29 18:34:52,553 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,553 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48644Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,554 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:34:52 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:34:52 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:52 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:52 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:52 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:52 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:52 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:52 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,554 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,554 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:34:52 zookeeper_node_2: java.io.EOFExceptionOct 30 07:34:52 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,554 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,554 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:52 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:52 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:52 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:52 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,555 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,556 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57484Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,556 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:52 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:52 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:52 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:52 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:52 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,559 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:34:52 zookeeper_node_2: 2019-10-29 18:34:52,560 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:52 zookeeper_node_1: 2019-10-29 18:34:52,554 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:34:52 zookeeper_node_1: 2019-10-29 18:34:52,554 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 1600Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,557 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 3Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,560 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:52 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:52 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,560 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,560 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45676Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,561 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:52 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:52 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:52 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:52 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:52 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,568 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:52 zookeeper_node_3: 2019-10-29 18:34:52,569 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:54 zookeeper_node_1: 2019-10-29 18:34:54,156 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,158 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44332Oct 30 07:34:54 zookeeper_node_1: 2019-10-29 18:34:54,158 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:34:54 zookeeper_node_1: 2019-10-29 18:34:54,158 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 3200Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,159 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:54 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:54 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:54 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:54 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,159 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:54 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:54 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,160 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,157 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48666Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,158 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:34:54 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:34:54 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:54 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:54 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:54 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:54 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:54 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:54 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,159 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 3, error = Oct 30 07:34:54 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:54 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:54 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:54 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:54 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:54 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:54 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:54 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,160 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,159 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 3Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,168 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45684Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,170 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:54 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:54 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:54 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:54 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:54 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,172 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:54 zookeeper_node_3: 2019-10-29 18:34:54,174 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,158 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:54 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:54 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:54 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:54 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,158 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,158 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:34:54 zookeeper_node_2: java.io.EOFExceptionOct 30 07:34:54 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,159 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,158 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,159 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57498Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,167 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,168 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:54 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:54 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:54 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:54 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:54 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:54 zookeeper_node_2: 2019-10-29 18:34:54,168 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,362 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44372Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,363 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:57 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:57 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:57 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:57 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:57 zookeeper_node_1: 2019-10-29 18:34:57,360 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:34:57 zookeeper_node_1: 2019-10-29 18:34:57,361 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:34:57 zookeeper_node_1: 2019-10-29 18:34:57,362 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 6400Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,361 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48706Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,361 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:57 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:57 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:57 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:57 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,361 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,363 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 3, error = Oct 30 07:34:57 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:34:57 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:57 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:57 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:57 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:57 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:57 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:57 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,363 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,363 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:34:57 zookeeper_node_3: java.io.EOFExceptionOct 30 07:34:57 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,364 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,363 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 3Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,367 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:57 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:34:57 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:57 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:57 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:57 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,368 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,364 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45712Oct 30 07:34:57 zookeeper_node_3: 2019-10-29 18:34:57,372 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,362 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:34:57 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:34:57 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:34:57 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:34:57 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:34:57 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:34:57 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:34:57 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:34:57 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,362 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,362 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:34:57 zookeeper_node_2: java.io.EOFExceptionOct 30 07:34:57 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,363 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,365 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57524Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,366 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:34:57 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:34:57 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:34:57 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:34:57 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:34:57 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,367 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:34:57 zookeeper_node_2: 2019-10-29 18:34:57,368 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:03 zookeeper_node_1: 2019-10-29 18:35:03,765 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,765 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48750Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,767 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:35:03 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:35:03 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:35:03 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:35:03 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:35:03 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:35:03 zookeeper_node_1: 2019-10-29 18:35:03,766 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:35:03 zookeeper_node_1: 2019-10-29 18:35:03,766 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 12800Oct 30 07:35:03 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:35:03 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:35:03 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,768 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:03 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:35:03 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:03 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:03 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,768 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,768 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 12800Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,768 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,767 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:35:03 zookeeper_node_2: java.io.EOFExceptionOct 30 07:35:03 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,769 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,770 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57594Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,770 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:03 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:35:03 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:03 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:03 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:03 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,777 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,778 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,780 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.29:48566Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,783 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,784 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:03 zookeeper_node_2: 2019-10-29 18:35:03,790 [myid:2] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 2, error = Oct 30 07:35:03 zookeeper_node_3: 2019-10-29 18:35:03,767 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44416Oct 30 07:35:03 zookeeper_node_3: 2019-10-29 18:35:03,769 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 3, error = Oct 30 07:35:03 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:35:03 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:35:03 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:35:03 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:35:03 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:35:03 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:35:03 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:35:03 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,770 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,770 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:04 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,770 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 3Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,770 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:35:04 zookeeper_node_2: java.io.EOFExceptionOct 30 07:35:04 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:04 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:04 zookeeper_node_2: 2019-10-29 18:35:03,790 [myid:2] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:04 zookeeper_node_2: 2019-10-29 18:35:03,791 [myid:2] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:04 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:35:04 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:04 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:04 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:04 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:04 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:04 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:04 zookeeper_node_2: 2019-10-29 18:35:03,791 [myid:2] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 2Oct 30 07:35:04 zookeeper_node_3: java.io.EOFExceptionOct 30 07:35:04 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,771 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,771 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45782Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,772 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:04 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,772 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42230Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,773 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 12800Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,774 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,783 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,783 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,784 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,792 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45788Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,792 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:35:04 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:35:04 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:35:04 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:35:04 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:35:04 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:35:04 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:35:04 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:35:04 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,793 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,793 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:04 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,793 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 3, error = Oct 30 07:35:04 zookeeper_node_3: java.io.EOFExceptionOct 30 07:35:04 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,793 [myid:3] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,795 [myid:3] - ERROR [SendWorker:2:QuorumCnxManager$SendWorker@1113] - Failed to send last message. Shutting down thread.Oct 30 07:35:04 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:35:04 zookeeper_node_3: at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)Oct 30 07:35:04 zookeeper_node_3: at java.net.SocketOutputStream.write(SocketOutputStream.java:134)Oct 30 07:35:04 zookeeper_node_3: at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:1082)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1109)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,795 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,798 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,799 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,799 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:04 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:04 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:04 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,801 [myid:3] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 3Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,821 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:04 zookeeper_node_3: 2019-10-29 18:35:03,822 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,585 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 25600Oct 30 07:35:16 zookeeper_node_1: 2019-10-29 18:35:16,598 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,599 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48850Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,599 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:35:16 zookeeper_node_2: java.io.EOFExceptionOct 30 07:35:16 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,600 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:16 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,617 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42328Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,618 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:35:16 zookeeper_node_3: java.io.EOFExceptionOct 30 07:35:16 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,619 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,619 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:16 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:16 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:16 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:16 zookeeper_node_1: 2019-10-29 18:35:16,623 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 1)Oct 30 07:35:16 zookeeper_node_1: 2019-10-29 18:35:16,624 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 25600Oct 30 07:35:16 zookeeper_node_1: 2019-10-29 18:35:16,646 [myid:1] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.29:42314Oct 30 07:35:16 zookeeper_node_1: 2019-10-29 18:35:16,689 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_1: 2019-10-29 18:35:16,696 [myid:1] - INFO [WorkerSender[myid=1]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (2, 1)Oct 30 07:35:16 zookeeper_node_1: 2019-10-29 18:35:16,699 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,600 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,600 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,600 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:35:16 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:35:16 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:35:16 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:35:16 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,601 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,602 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:16 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,602 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,614 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,615 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,615 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,616 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57696Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,618 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,693 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:48856Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,697 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 1, my id = 2, error = Oct 30 07:35:16 zookeeper_node_2: java.net.SocketException: Socket closedOct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:35:16 zookeeper_node_2: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:35:16 zookeeper_node_2: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:35:16 zookeeper_node_2: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:35:16 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,698 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:16 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,698 [myid:2] - WARN [SendWorker:1:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 1 my id = 2Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,698 [myid:2] - WARN [RecvWorker:1:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,697 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 2, error = Oct 30 07:35:16 zookeeper_node_2: java.io.EOFExceptionOct 30 07:35:16 zookeeper_node_2: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,698 [myid:2] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,699 [myid:2] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.4:57698Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,699 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:16 zookeeper_node_2: java.lang.InterruptedExceptionOct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:16 zookeeper_node_2: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:16 zookeeper_node_2: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,702 [myid:2] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 2Oct 30 07:35:16 zookeeper_node_2: 2019-10-29 18:35:16,704 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,621 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,621 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:45892Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,619 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:35:16 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:35:16 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:35:16 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:35:16 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:35:16 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:35:16 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:35:16 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:35:16 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,621 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,622 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:16 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:16 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:16 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:16 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:16 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,629 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,626 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 25600Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,625 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.27:44516Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,658 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,659 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,659 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,669 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 1 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,699 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,900 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1281] - LEADINGOct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,906 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Leader@66] - TCP NoDelay set to: trueOct 30 07:35:16 zookeeper_node_3: 2019-10-29 18:35:16,906 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Leader@86] - zookeeper.leader.maxConcurrentSnapshots = 10Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,902 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer@1269] - FOLLOWINGOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,906 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Leader@88] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,918 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@91] - TCP NoDelay set to: trueOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,932 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMTOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,932 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:host.name=zookeeper_node_1Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,932 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_232Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,932 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle CorporationOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,932 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/local/openjdk-8Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-jute-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-unix-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-epoll-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-resolver-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-handler-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-codec-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-buffer-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jline-2.11.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-util-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-servlet-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-server-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-security-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-io-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-http-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-databind-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-core-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-annotations-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf::/opt/zookeeper-cluster/newrelic/newrelic.jarOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/libOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmpOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.name=LinuxOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.version=3.10.0-862.2.3.el7.x86_64Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,933 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.name=rootOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,934 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/rootOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,934 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,934 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=218MBOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,925 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMTOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,926 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:host.name=zookeeper_node_3Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,926 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_232Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,926 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle CorporationOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,926 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/local/openjdk-8Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,926 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/classes:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/target/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../build/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-jute-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/zookeeper-3.5.6.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-unix-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-native-epoll-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-transport-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-resolver-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-handler-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-common-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-codec-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/netty-buffer-4.1.42.Final.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/json-simple-1.1.1.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jline-2.11.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-util-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-servlet-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-server-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-security-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-io-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jetty-http-9.4.17.v20190418.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-databind-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-core-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/jackson-annotations-2.9.10.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/zookeeper-cluster/apache-zookeeper-3.5.6-bin/bin/../conf::/opt/zookeeper-cluster/newrelic/newrelic.jarOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,926 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/libOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,926 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmpOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,927 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,927 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.name=LinuxOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,927 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,927 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.version=3.10.0-693.11.6.el7.x86_64Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,927 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.name=rootOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,927 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/rootOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,927 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,928 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=251MBOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,928 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=889MBOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,928 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=440MBOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,930 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,931 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,934 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /mnt/zk_data_log/version-2 snapdir /mnt/zk_data/version-2Oct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,937 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Leader@464] - LEADING - LEADER ELECTION TOOK - 37 MSOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:16,941 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):FileTxnSnapLog@384] - Snapshotting: 0x7300000000 to /mnt/zk_data/version-2/snapshot.7300000000Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,934 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=889MBOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,936 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=457MBOct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,941 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,941 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,942 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /mnt/zk_data_log/version-2 snapdir /mnt/zk_data/version-2Oct 30 07:35:17 zookeeper_node_1: 2019-10-29 18:35:16,942 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 40 MSOct 30 07:35:17 zookeeper_node_3: 2019-10-29 18:35:17,672 [myid:3] - INFO [LearnerHandler-/10.24.8.27:51354:LearnerHandler@406] - Follower sid: 1 : info : node01.kafka.net:2888:3888:participantOct 30 07:35:18 zookeeper_node_3: 2019-10-29 18:35:18,212 [myid:3] - INFO [LearnerHandler-/10.24.8.27:51354:ZKDatabase@295] - On disk txn sync enabled with snapshotSizeFactor 0.33Oct 30 07:35:18 zookeeper_node_3: 2019-10-29 18:35:18,212 [myid:3] - INFO [LearnerHandler-/10.24.8.27:51354:LearnerHandler@708] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x7300000000 peerLastZxid=0x7100000f25Oct 30 07:35:18 zookeeper_node_3: 2019-10-29 18:35:18,254 [myid:3] - INFO [LearnerHandler-/10.24.8.27:51354:LearnerHandler@784] - Use txnlog and committedLog for peer sid: 1Oct 30 07:35:18 zookeeper_node_3: 2019-10-29 18:35:18,284 [myid:3] - INFO [LearnerHandler-/10.24.8.27:51354:LearnerHandler@859] - Sending DIFF zxid=0x7300000000 for peer sid: 1Oct 30 07:35:18 zookeeper_node_1: 2019-10-29 18:35:18,290 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@391] - Getting a diff from the leader 0x7300000000Oct 30 07:35:18 zookeeper_node_1: 2019-10-29 18:35:18,298 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@546] - Learner received NEWLEADER messageOct 30 07:35:18 zookeeper_node_3: 2019-10-29 18:35:18,366 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):Leader@1296] - Have quorum of supporters, sids: [ [1, 3],[1, 3] ]; starting up and setting last processed zxid: 0x7400000000Oct 30 07:35:19 zookeeper_node_3: 2019-10-29 18:35:18,987 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 4 worker threads.Oct 30 07:35:19 zookeeper_node_3: 2019-10-29 18:35:19,026 [myid:3] - INFO [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000Oct 30 07:35:19 zookeeper_node_1: 2019-10-29 18:35:19,029 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Learner@529] - Learner received UPTODATE messageOct 30 07:35:19 zookeeper_node_1: 2019-10-29 18:35:19,047 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 4 worker threads.Oct 30 07:35:19 zookeeper_node_1: 2019-10-29 18:35:19,355 [myid:1] - INFO [NIOWorkerThread-2:Learner@113] - Revalidating client: 0x1027e6ba8b10000Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,307 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42498Oct 30 07:35:42 zookeeper_node_2: 2019-10-29 18:35:42,305 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 51200Oct 30 07:35:42 zookeeper_node_2: 2019-10-29 18:35:42,307 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:35:42 zookeeper_node_2: 2019-10-29 18:35:42,308 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:42 zookeeper_node_2: 2019-10-29 18:35:42,309 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,310 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:35:42 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:35:42 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:35:42 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:35:42 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:35:42 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:35:42 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:35:42 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:35:42 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,310 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,310 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:35:42 zookeeper_node_3: java.io.EOFExceptionOct 30 07:35:42 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,311 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,311 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:42 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:42 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:42 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:42 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,311 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,312 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:46054Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,312 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:35:42 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:35:42 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:35:42 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:35:42 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:35:42 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,314 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,317 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:35:42 zookeeper_node_3: 2019-10-29 18:35:42,317 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LEADING (n.state), 3 (n.sid), 0x74 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:36:33 zookeeper_node_2: 2019-10-29 18:36:33,509 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 60000Oct 30 07:36:33 zookeeper_node_2: 2019-10-29 18:36:33,521 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:36:33 zookeeper_node_2: 2019-10-29 18:36:33,521 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:36:33 zookeeper_node_2: 2019-10-29 18:36:33,522 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,521 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:42798Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,522 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:36:33 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:36:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:36:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:36:33 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,522 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,523 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:36:33 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:36:33 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:36:33 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:36:33 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:36:33 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:36:33 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:36:33 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:36:33 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,523 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,524 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:36:33 zookeeper_node_3: java.io.EOFExceptionOct 30 07:36:33 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,524 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,526 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:46344Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,527 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:36:33 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:36:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:36:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:36:33 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:36:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,540 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,573 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:36:33 zookeeper_node_3: 2019-10-29 18:36:33,574 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LEADING (n.state), 3 (n.sid), 0x74 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:37:33 zookeeper_node_2: 2019-10-29 18:37:33,523 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 60000Oct 30 07:37:33 zookeeper_node_2: 2019-10-29 18:37:33,536 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:37:33 zookeeper_node_2: 2019-10-29 18:37:33,536 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,537 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:43158Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,540 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:37:33 zookeeper_node_3: java.io.EOFExceptionOct 30 07:37:33 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,540 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:37:33 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:37:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:37:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:37:33 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:37:33 zookeeper_node_2: 2019-10-29 18:37:33,536 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,540 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,540 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,540 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:37:33 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:37:33 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:37:33 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:37:33 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:37:33 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:37:33 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:37:33 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:37:33 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,546 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,546 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:37:33 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:37:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:37:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:37:33 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:37:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,562 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,541 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:46702Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,587 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:37:33 zookeeper_node_3: 2019-10-29 18:37:33,587 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LEADING (n.state), 3 (n.sid), 0x74 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:38:33 zookeeper_node_2: 2019-10-29 18:38:33,537 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 60000Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,549 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 10.24.8.28:43496Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,550 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 2, my id = 3, error = Oct 30 07:38:33 zookeeper_node_3: java.net.SocketException: Socket closedOct 30 07:38:33 zookeeper_node_3: at java.net.SocketInputStream.socketRead0(Native Method)Oct 30 07:38:33 zookeeper_node_2: 2019-10-29 18:38:33,549 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@432] - Have smaller server identifier, so dropping the connection: (3, 2)Oct 30 07:38:33 zookeeper_node_2: 2019-10-29 18:38:33,549 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:38:33 zookeeper_node_3: at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)Oct 30 07:38:33 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:171)Oct 30 07:38:33 zookeeper_node_3: at java.net.SocketInputStream.read(SocketInputStream.java:141)Oct 30 07:38:33 zookeeper_node_3: at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)Oct 30 07:38:33 zookeeper_node_3: at java.io.BufferedInputStream.read(BufferedInputStream.java:265)Oct 30 07:38:33 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:387)Oct 30 07:38:33 zookeeper_node_2: 2019-10-29 18:38:33,549 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x7100000f25 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x73 (n.peerEPoch), LOOKING (my state)0 (n.config version)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,551 [myid:3] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,551 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:38:33 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:38:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:38:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:38:33 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,551 [myid:3] - WARN [SendWorker:2:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 2 my id = 3Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,551 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1221] - Connection broken for id 3, my id = 3, error = Oct 30 07:38:33 zookeeper_node_3: java.io.EOFExceptionOct 30 07:38:33 zookeeper_node_3: at java.io.DataInputStream.readInt(DataInputStream.java:392)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206)Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,554 [myid:3] - WARN [RecvWorker:3:QuorumCnxManager$RecvWorker@1224] - Interrupting SendWorkerOct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,554 [myid:3] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener@918] - Received connection request 192.168.2.3:47038Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,555 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1137] - Interrupted while waiting for message on queueOct 30 07:38:33 zookeeper_node_3: java.lang.InterruptedExceptionOct 30 07:38:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)Oct 30 07:38:33 zookeeper_node_3: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)Oct 30 07:38:33 zookeeper_node_3: at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1288)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:80)Oct 30 07:38:33 zookeeper_node_3: at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1125)Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,555 [myid:3] - WARN [SendWorker:3:QuorumCnxManager$SendWorker@1147] - Send worker leaving thread id 3 my id = 3Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,588 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x73 (n.peerEPoch), LEADING (my state)0 (n.config version)Oct 30 07:38:33 zookeeper_node_3: 2019-10-29 18:38:33,588 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x7300000000 (n.zxid), 0x1 (n.round), LEADING (n.state), 3 (n.sid), 0x74 (n.peerEPoch), LEADING (my state)0 (n.config version)
{code}
Best regards
Sebastian
was (Author: paxi):
Hello,
also attaching my logs here....
When I start node in order 1-2-3 it works. If I start it like 3-2-1 the election times out for one node and everything running in circles...
See attached logs for both versions of the start.
Best regards
Sebastian
> fast leader election keeps failing
> ----------------------------------
>
> Key: ZOOKEEPER-2164
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2164
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection
> Affects Versions: 3.4.5
> Reporter: Michi Mutsuzaki
> Priority: Major
> Fix For: 3.6.0, 3.5.7
>
>
> I have a 3-node cluster with sids 1, 2 and 3. Originally 2 is the leader. When I shut down 2, 1 and 3 keep going back to leader election. Here is what seems to be happening.
> - Both 1 and 3 elect 3 as the leader.
> - 1 receives votes from 3 and itself, and starts trying to connect to 3 as a follower.
> - 3 doesn't receive votes for 5 seconds because connectOne() to 2 doesn't timeout for 5 seconds: https://github.com/apache/zookeeper/blob/41c9fcb3ca09cd3d05e59fe47f08ecf0b85532c8/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java#L346
> - By the time 3 receives votes, 1 has given up trying to connect to 3: https://github.com/apache/zookeeper/blob/41c9fcb3ca09cd3d05e59fe47f08ecf0b85532c8/src/java/main/org/apache/zookeeper/server/quorum/Learner.java#L247
> I'm using 3.4.5, but it looks like this part of the code hasn't changed for a while, so I'm guessing later versions have the same issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)