You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@zookeeper.apache.org by Onno Zweers <on...@surf.nl> on 2022/10/20 13:19:39 UTC

2 out of 5 Zookeepers down with "Exception when following the leader"

Hi all,

I’m new to this list so please forgive me if I’m asking a question that has been discussed before or if I’m asking it in the wrong place.

I sometimes see 1 or 2 out of our 5 zookeepers going down. If it’s two going down, it appears to happen at almost the same millisecond. They stay down for ~16 minutes. This scares the hell out of me because it means that during this time we don’t have any redundancy left.

Our 5 zookeepers are at version 3.5.8, running in libvirt VMs with Centos 8 and OpenJDK 1.8.0_322. They are all on a separate VM host. I checked for network issues (with ping, iperf3) but found nothing. It’s not always the same zookeeper VM(s) that suffers from this problem. It does not happen at a regular time so there does not seem to be a cron job affecting this behavior.

I have some logging from the last time this happened; I’ll add it below.

I know we’re running an old version of Zookeeper. I have looked through the release notes but I couldn’t find a fix for this.

I’d welcome any ideas. How could we prevent this? Based on the SocketTimeoutException, I’d think there might be a network hiccup; but why is it so difficult for Zookeeper to recover from this? Is it a bug in Zookeeper? Should I report it on https://issues.apache.org?

Kind regards,
Onno

[root@feszoo4 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' messages-20221016
Oct 11 16:01:43 feszoo4 zookeeper[1199]: 2022-10-11 16:01:43,837 - WARN  [QuorumPeer[myid=4](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96] - Exception when following the leader
Oct 11 16:01:43 feszoo4 zookeeper[1199]: java.net.SocketTimeoutException: Read timed out
...


[root@feszoo3 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' messages-20221016
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,838 - WARN  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96] - Exception when following the leader
Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.net.SocketTimeoutException: Read timed out
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.socketRead0(Native Method)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.read(SocketInputStream.java:171)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.net.SocketInputStream.read(SocketInputStream.java:141)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at java.io.DataInputStream.readInt(DataInputStream.java:387)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,839 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@201] - shutdown called
Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.lang.Exception: shutdown Follower
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1257)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,866 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 1 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):LearnerZooKeeperServer@165] - Shutting down
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@558] - shutting down
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FollowerRequestProcessor@139] - Shutting down
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@362] - Shutting down
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO  [FollowerRequestProcessor:3:FollowerRequestProcessor@110] - FollowerRequestProcessor exited loop!
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO  [CommitProcessor:3:CommitProcessor@195] - CommitProcessor exited loop!
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,888 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FinalRequestProcessor@514] - shutdown of request processor complete
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):SyncRequestProcessor@191] - Shutting down
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO  [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited!
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - WARN  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1300] - PeerState set to LOOKING
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1175] - LOOKING
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@903] - New election. My id =  3, proposed zxid=0x8a00010a28
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,892 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 3 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,938 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,939 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,940 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,941 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,004 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,019 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,046 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,053 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,057 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,075 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,086 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,090 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,095 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,101 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,113 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,122 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 400
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,143 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,149 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,152 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,158 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,159 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,162 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,167 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,179 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,186 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,198 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,217 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,228 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,234 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,235 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,241 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,242 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,246 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,250 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,260 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,265 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,276 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,280 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,288 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,300 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,307 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,312 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,318 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,325 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,344 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,351 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,375 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,377 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,381 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,394 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,395 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,408 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,424 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,432 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,433 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,439 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,444 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,476 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,482 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,486 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,490 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,495 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,525 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,526 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,535 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,537 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,540 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,541 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,544 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 800
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,553 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,556 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,562 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,580 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,596 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,597 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,607 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,625 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,632 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,638 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,646 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,648 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,667 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,673 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,680 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,690 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,693 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,697 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,708 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,712 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,716 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,723 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,727 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,734 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,740 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,744 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,752 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,765 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,774 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,778 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,803 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,804 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,821 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,832 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,836 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,837 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,854 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,862 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,867 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,872 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,874 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,883 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,886 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,888 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,892 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,895 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,909 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,913 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,934 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,938 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,947 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,951 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,956 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,964 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,976 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,980 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,012 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,027 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,036 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,070 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,090 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,127 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,197 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,348 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 1600
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,349 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,351 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,356 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,385 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,451 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,561 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,563 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,576 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,595 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,639 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,654 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,685 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,703 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,744 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,753 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,014 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,045 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,062 - WARN  [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,121 - WARN  [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,141 - WARN  [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,151 - WARN  [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,224 - WARN  [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,254 - WARN  [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,322 - WARN  [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,398 - WARN  [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,435 - WARN  [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,522 - WARN  [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,541 - WARN  [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,817 - WARN  [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,873 - WARN  [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,897 - WARN  [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,914 - WARN  [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,951 - INFO  [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937] - Notification time out: 3200
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,952 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,955 - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)
Oct 11 16:01:47 feszoo3 zookeeper[1187]: 2022-10-11 16:01:47,582 - WARN  [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running


[root@feszoo3 ~]# cat /etc/zookeeper/zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# The directory where the snapshot is stored.
dataDir=/var/lib/zookeeper/data
# The port at which the clients will connect
clientPort=2181
skipACL=1
# 4-letter words: which commands are allowed
4lw.commands.whitelist=*
# Cluster topo
server.1=feszoo1:2888:3888
server.2=feszoo2:2888:3888
server.3=feszoo3:2888:3888
server.4=feszoo4:2888:3888
server.5=feszoo5:2888:3888

Re: 2 out of 5 Zookeepers down with "Exception when following the leader"

Posted by Onno Zweers <on...@surf.nl>.

Hi Máté,

Thanks for your reply.

I checked the clock status and they seem to be nicely synced by Chrony:


[root@feszoo3 ~]# chronyc tracking

Reference ID    : 91641D92 (dns1.osd.surfsara.nl)

Stratum         : 3

Ref time (UTC)  : Fri Oct 21 08:31:51 2022

System time     : 0.000006580 seconds fast of NTP time

Last offset     : -0.000011125 seconds

RMS offset      : 0.000024293 seconds

Frequency       : 23.810 ppm slow

Residual freq   : -0.000 ppm

Skew            : 0.019 ppm

Root delay      : 0.001364297 seconds

Root dispersion : 0.029461518 seconds

Update interval : 1041.2 seconds

Leap status     : Normal

Good point to look at the leader’s log at the time of the last incident. I’ll include it below.

It looks to be like the leader (feszoo5 in this case) got SocketTimeoutExceptions for all 4 followers. Not sure though why two of the followers seem to recover immediately and the other two don’t. I’ll also include a log from a follower that recovered. So here’s the overview:


[root@feszoo3 ~]# for host in feszoo{1..5} ; do host $host ; done

feszoo1.grid.surfsara.nl has address 145.100.32.86   # recovered

feszoo2.grid.surfsara.nl has address 145.100.32.87   # recovered

feszoo3.grid.surfsara.nl has address 145.100.32.88   # did not recover

feszoo4.grid.surfsara.nl has address 145.100.32.89   # did not recover

feszoo5.grid.surfsara.nl has address 145.100.32.90   # leader


The leader’s log:

[root@feszoo5 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' messages-20221016
Oct 11 16:01:41 feszoo5 zookeeper[137097]: 2022-10-11 16:01:41,089 - INFO  [NIOWorkerThread-10:NIOServerCnxn@518] - Processing stat command from /0:0:0:0:0:0:0:1:48040
Oct 11 16:01:41 feszoo5 zookeeper[137097]: 2022-10-11 16:01:41,089 - INFO  [NIOWorkerThread-10:StatCommand@53] - Stat command output
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,828 - ERROR [LearnerHandler-/145.100.32.87:52454:LearnerHandler@629] - Unexpected exception causing shutdown while sock still open
Oct 11 16:01:43 feszoo5 zookeeper[137097]: java.net.SocketTimeoutException: Read timed out
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead0(Native Method)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:171)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:141)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.DataInputStream.readInt(DataInputStream.java:387)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:539)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,829 - WARN  [LearnerHandler-/145.100.32.87:52454:LearnerHandler@644] - ******* GOODBYE /145.100.32.87:52454 ********
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,834 - ERROR [LearnerHandler-/145.100.32.86:42688:LearnerHandler@629] - Unexpected exception causing shutdown while sock still open
Oct 11 16:01:43 feszoo5 zookeeper[137097]: java.net.SocketTimeoutException: Read timed out
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead0(Native Method)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:171)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:141)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.DataInputStream.readInt(DataInputStream.java:387)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:539)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,835 - WARN  [LearnerHandler-/145.100.32.86:42688:LearnerHandler@644] - ******* GOODBYE /145.100.32.86:42688 ********
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,837 - ERROR [LearnerHandler-/145.100.32.88:49168:LearnerHandler@629] - Unexpected exception causing shutdown while sock still open
Oct 11 16:01:43 feszoo5 zookeeper[137097]: java.net.SocketTimeoutException: Read timed out
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead0(Native Method)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:171)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:141)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.DataInputStream.readInt(DataInputStream.java:387)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:539)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,837 - ERROR [LearnerHandler-/145.100.32.89:59626:LearnerHandler@629] - Unexpected exception causing shutdown while sock still open
Oct 11 16:01:43 feszoo5 zookeeper[137097]: java.net.SocketTimeoutException: Read timed out
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead0(Native Method)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:171)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.net.SocketInputStream.read(SocketInputStream.java:141)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at java.io.DataInputStream.readInt(DataInputStream.java:387)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: #011at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:539)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,838 - WARN  [LearnerHandler-/145.100.32.88:49168:LearnerHandler@644] - ******* GOODBYE /145.100.32.88:49168 ********
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,838 - WARN  [LearnerHandler-/145.100.32.89:59626:LearnerHandler@644] - ******* GOODBYE /145.100.32.89:59626 ********
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,867 - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LEADING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,870 - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@697] - Notification: 2 (message format version), 1 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LEADING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,870 - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LEADING (my state)0 (n.config version)
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,877 - INFO  [LearnerHandler-/145.100.32.87:59226:LearnerHandler@406] - Follower sid: 2 : info : feszoo2:2888:3888:participant
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,877 - INFO  [LearnerHandler-/145.100.32.86:56642:LearnerHandler@406] - Follower sid: 1 : info : feszoo1:2888:3888:participant
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,878 - INFO  [LearnerHandler-/145.100.32.87:59226:ZKDatabase@295] - On disk txn sync enabled with snapshotSizeFactor 0.33
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,878 - INFO  [LearnerHandler-/145.100.32.86:56642:ZKDatabase@295] - On disk txn sync enabled with snapshotSizeFactor 0.33
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,879 - INFO  [LearnerHandler-/145.100.32.87:59226:LearnerHandler@708] - Synchronizing with Follower sid: 2 maxCommittedLog=0x8a00010a28 minCommittedLog=0x8a00010834 lastProcessedZxid=0x8a00010a28 peerLastZxid=0x8a00010a28
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,879 - INFO  [LearnerHandler-/145.100.32.87:59226:LearnerHandler@752] - Sending DIFF zxid=0x8a00010a28 for peer sid: 2
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,879 - INFO  [LearnerHandler-/145.100.32.86:56642:LearnerHandler@708] - Synchronizing with Follower sid: 1 maxCommittedLog=0x8a00010a28 minCommittedLog=0x8a00010834 lastProcessedZxid=0x8a00010a28 peerLastZxid=0x8a00010a28
Oct 11 16:01:43 feszoo5 zookeeper[137097]: 2022-10-11 16:01:43,879 - INFO  [LearnerHandler-/145.100.32.86:56642:LearnerHandler@752] - Sending DIFF zxid=0x8a00010a28 for peer sid: 1



A follower that recovered:


[root@feszoo1 /var/log]# grep 'Oct 11 16:01:4.*zookeeper' messages-20221016

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,840 - WARN  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96] - Exception when following the leader

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: java.net.SocketTimeoutException: Read timed out

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at java.net.SocketInputStream.socketRead0(Native Method)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at java.net.SocketInputStream.read(SocketInputStream.java:171)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at java.net.SocketInputStream.read(SocketInputStream.java:141)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at java.io.DataInputStream.readInt(DataInputStream.java:387)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,848 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@201] - shutdown called

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: java.lang.Exception: shutdown Follower

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: #011at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1257)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,848 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):LearnerZooKeeperServer@165] - Shutting down

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,848 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@558] - shutting down

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,848 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FollowerRequestProcessor@139] - Shutting down

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,848 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@362] - Shutting down

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,849 - INFO  [CommitProcessor:1:CommitProcessor@195] - CommitProcessor exited loop!

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,849 - INFO  [FollowerRequestProcessor:1:FollowerRequestProcessor@110] - FollowerRequestProcessor exited loop!

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,851 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FinalRequestProcessor@514] - shutdown of request processor complete

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,863 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):SyncRequestProcessor@191] - Shutting down

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,863 - INFO  [SyncThread:1:SyncRequestProcessor@169] - SyncRequestProcessor exited!

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,867 - WARN  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1300] - PeerState set to LOOKING

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,867 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1175] - LOOKING

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,867 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,868 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@903] - New election. My id =  1, proposed zxid=0x8a00010a28

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,868 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 1 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,869 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52 (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,870 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,871 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,871 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), LEADING (n.state), 5 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,872 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), LEADING (n.state), 5 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,872 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,872 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1251] - FOLLOWING

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,872 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,872 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,873 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/lib/zookeeper/data/version-2 snapdir /var/lib/zookeeper/data/version-2

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,873 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 0 MS

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,874 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@697] - Notification: 2 (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51 (n.round), FOLLOWING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my state)0 (n.config version)

Oct 11 16:01:43 feszoo1 zookeeper[2315850]: 2022-10-11 16:01:43,884 - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@391] - Getting a diff from the leader 0x8a00010a28





Van: Szalay-Beko Máté <sy...@apache.org>
Datum: donderdag, 20 oktober 2022 om 19:24
Aan: user@zookeeper.apache.org <us...@zookeeper.apache.org>
Onderwerp: Re: 2 out of 5 Zookeepers down with "Exception when following the leader"
Hello!

once I saw something similar, when the problem was that the clocks were not
in sync on the ZooKeeper VMs. And when the ntp server updated the clock, it
screwed up the kerberos token renewal threads in the current leader,
causing SASL related unhandled exceptions during authentication on leader
election ports which basically made it impossible for Peers to connect to
the Leader for some time.

Although I'm not sure if you use quorum authentication. (I see you don't
really use ACLs, at least "skipACL=1" suggests that in your zoo.cfg, so
maybe you have a different problem)

Anyway, maybe you can check the logs from the leader during this time,
maybe you see something interesting there.

Also you might look for error messages that would indicate that some
threads died in the leader / in other servers.

E.g.
"ERROR org.apache.zookeeper.server.NIOServerCnxnFactory: Thread
Thread[QuorumConnectionThread-[myid=3]-43,5,main] died"

In the leader, there are separate threads responsible to handle leader
election of other quorum messages. For each peer, there is a separate
thread. And it is possible some times that these threads die, and the
Leader is still leading but can not accept new quorum members. Several bugs
were fixed regarding this in the past. (but AFAIK all of them are present
in 3.5.8)

Kind regards,
Máté

On Thu, Oct 20, 2022 at 3:20 PM Onno Zweers <on...@surf.nl> wrote:

> Hi all,
>
> I’m new to this list so please forgive me if I’m asking a question that
> has been discussed before or if I’m asking it in the wrong place.
>
> I sometimes see 1 or 2 out of our 5 zookeepers going down. If it’s two
> going down, it appears to happen at almost the same millisecond. They stay
> down for ~16 minutes. This scares the hell out of me because it means that
> during this time we don’t have any redundancy left.
>
> Our 5 zookeepers are at version 3.5.8, running in libvirt VMs with Centos
> 8 and OpenJDK 1.8.0_322. They are all on a separate VM host. I checked for
> network issues (with ping, iperf3) but found nothing. It’s not always the
> same zookeeper VM(s) that suffers from this problem. It does not happen at
> a regular time so there does not seem to be a cron job affecting this
> behavior.
>
> I have some logging from the last time this happened; I’ll add it below.
>
> I know we’re running an old version of Zookeeper. I have looked through
> the release notes but I couldn’t find a fix for this.
>
> I’d welcome any ideas. How could we prevent this? Based on the
> SocketTimeoutException, I’d think there might be a network hiccup; but why
> is it so difficult for Zookeeper to recover from this? Is it a bug in
> Zookeeper? Should I report it on https://issues.apache.org?
>
> Kind regards,
> Onno
>
> [root@feszoo4 /var/log]# grep 'Oct 11 16:01:4.*zookeeper'
> messages-20221016
> Oct 11 16:01:43 feszoo4 zookeeper[1199]: 2022-10-11 16:01:43,837 - WARN
> [QuorumPeer[myid=4](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96]
> - Exception when following the leader
> Oct 11 16:01:43 feszoo4 zookeeper[1199]: java.net.SocketTimeoutException:
> Read timed out
> ...
>
>
> [root@feszoo3 /var/log]# grep 'Oct 11 16:01:4.*zookeeper'
> messages-20221016
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,838 - WARN
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96]
> - Exception when following the leader
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.net.SocketTimeoutException:
> Read timed out
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.socketRead0(Native Method)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.read(SocketInputStream.java:171)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.read(SocketInputStream.java:141)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.io.DataInputStream.readInt(DataInputStream.java:387)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,839 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@201]
> - shutdown called
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.lang.Exception: shutdown
> Follower
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1257)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,866 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 1 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):LearnerZooKeeperServer@165]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@558]
> - shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FollowerRequestProcessor@139]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@362]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO
> [FollowerRequestProcessor:3:FollowerRequestProcessor@110] -
> FollowerRequestProcessor exited loop!
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO
> [CommitProcessor:3:CommitProcessor@195] - CommitProcessor exited loop!
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,888 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FinalRequestProcessor@514]
> - shutdown of request processor complete
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):SyncRequestProcessor@191]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO
> [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited!
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - WARN
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1300]
> - PeerState set to LOOKING
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1175]
> - LOOKING
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@903]
> - New election. My id =  3, proposed zxid=0x8a00010a28
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,892 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 3 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,938 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,939 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,940 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,941 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,004 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,019 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,046 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,053 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,057 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,075 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,086 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,090 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,095 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,101 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,113 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,122 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 400
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,143 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,149 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,152 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,158 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,159 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,162 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,167 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,179 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,186 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,198 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,217 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,228 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,234 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,235 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,241 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,242 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,246 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,250 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,260 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,265 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,276 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,280 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,288 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,300 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,307 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,312 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,318 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,325 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,344 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,351 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,375 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,377 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,381 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,394 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,395 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,408 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,424 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,432 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,433 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,439 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,444 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,476 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,482 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,486 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,490 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,495 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,525 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,526 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,535 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,537 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,540 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,541 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,544 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 800
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,553 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,556 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,562 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,580 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,596 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,597 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,607 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,625 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,632 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,638 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,646 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,648 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,667 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,673 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,680 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,690 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,693 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,697 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,708 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,712 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,716 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,723 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,727 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,734 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,740 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,744 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,752 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,765 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,774 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,778 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,803 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,804 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,821 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,832 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,836 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,837 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,854 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,862 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,867 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,872 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,874 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,883 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,886 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,888 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,892 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,895 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,909 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,913 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,934 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,938 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,947 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,951 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,956 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,964 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,976 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,980 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,012 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,027 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,036 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,070 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,090 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,127 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,197 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,348 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 1600
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,349 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,351 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,356 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,385 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,451 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,561 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,563 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,576 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,595 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,639 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,654 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,685 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,703 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,744 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,753 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,014 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,045 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,062 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,121 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,141 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,151 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,224 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,254 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,322 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,398 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,435 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,522 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,541 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,817 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,873 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,897 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,914 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,951 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 3200
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,952 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,955 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:47 feszoo3 zookeeper[1187]: 2022-10-11 16:01:47,582 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
>
>
> [root@feszoo3 ~]# cat /etc/zookeeper/zoo.cfg
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # The directory where the snapshot is stored.
> dataDir=/var/lib/zookeeper/data
> # The port at which the clients will connect
> clientPort=2181
> skipACL=1
> # 4-letter words: which commands are allowed
> 4lw.commands.whitelist=*
> # Cluster topo
> server.1=feszoo1:2888:3888
> server.2=feszoo2:2888:3888
> server.3=feszoo3:2888:3888
> server.4=feszoo4:2888:3888
> server.5=feszoo5:2888:3888
>
>

Re: 2 out of 5 Zookeepers down with "Exception when following the leader"

Posted by Szalay-Bekő Máté <sy...@apache.org>.

Hello!

once I saw something similar, when the problem was that the clocks were not
in sync on the ZooKeeper VMs. And when the ntp server updated the clock, it
screwed up the kerberos token renewal threads in the current leader,
causing SASL related unhandled exceptions during authentication on leader
election ports which basically made it impossible for Peers to connect to
the Leader for some time.

Although I'm not sure if you use quorum authentication. (I see you don't
really use ACLs, at least "skipACL=1" suggests that in your zoo.cfg, so
maybe you have a different problem)

Anyway, maybe you can check the logs from the leader during this time,
maybe you see something interesting there.

Also you might look for error messages that would indicate that some
threads died in the leader / in other servers.

E.g.
"ERROR org.apache.zookeeper.server.NIOServerCnxnFactory: Thread
Thread[QuorumConnectionThread-[myid=3]-43,5,main] died"

In the leader, there are separate threads responsible to handle leader
election of other quorum messages. For each peer, there is a separate
thread. And it is possible some times that these threads die, and the
Leader is still leading but can not accept new quorum members. Several bugs
were fixed regarding this in the past. (but AFAIK all of them are present
in 3.5.8)

Kind regards,
Máté

On Thu, Oct 20, 2022 at 3:20 PM Onno Zweers <on...@surf.nl> wrote:

> Hi all,
>
> I’m new to this list so please forgive me if I’m asking a question that
> has been discussed before or if I’m asking it in the wrong place.
>
> I sometimes see 1 or 2 out of our 5 zookeepers going down. If it’s two
> going down, it appears to happen at almost the same millisecond. They stay
> down for ~16 minutes. This scares the hell out of me because it means that
> during this time we don’t have any redundancy left.
>
> Our 5 zookeepers are at version 3.5.8, running in libvirt VMs with Centos
> 8 and OpenJDK 1.8.0_322. They are all on a separate VM host. I checked for
> network issues (with ping, iperf3) but found nothing. It’s not always the
> same zookeeper VM(s) that suffers from this problem. It does not happen at
> a regular time so there does not seem to be a cron job affecting this
> behavior.
>
> I have some logging from the last time this happened; I’ll add it below.
>
> I know we’re running an old version of Zookeeper. I have looked through
> the release notes but I couldn’t find a fix for this.
>
> I’d welcome any ideas. How could we prevent this? Based on the
> SocketTimeoutException, I’d think there might be a network hiccup; but why
> is it so difficult for Zookeeper to recover from this? Is it a bug in
> Zookeeper? Should I report it on https://issues.apache.org?
>
> Kind regards,
> Onno
>
> [root@feszoo4 /var/log]# grep 'Oct 11 16:01:4.*zookeeper'
> messages-20221016
> Oct 11 16:01:43 feszoo4 zookeeper[1199]: 2022-10-11 16:01:43,837 - WARN
> [QuorumPeer[myid=4](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96]
> - Exception when following the leader
> Oct 11 16:01:43 feszoo4 zookeeper[1199]: java.net.SocketTimeoutException:
> Read timed out
> ...
>
>
> [root@feszoo3 /var/log]# grep 'Oct 11 16:01:4.*zookeeper'
> messages-20221016
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,838 - WARN
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@96]
> - Exception when following the leader
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.net.SocketTimeoutException:
> Read timed out
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.socketRead0(Native Method)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.read(SocketInputStream.java:171)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.net.SocketInputStream.read(SocketInputStream.java:141)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> java.io.DataInputStream.readInt(DataInputStream.java:387)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,839 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@201]
> - shutdown called
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: java.lang.Exception: shutdown
> Follower
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: #011at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1257)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,866 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 1 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,869 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 2 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), FOLLOWING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):LearnerZooKeeperServer@165]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@558]
> - shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FollowerRequestProcessor@139]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,885 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@362]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO
> [FollowerRequestProcessor:3:FollowerRequestProcessor@110] -
> FollowerRequestProcessor exited loop!
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,887 - INFO
> [CommitProcessor:3:CommitProcessor@195] - CommitProcessor exited loop!
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,888 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FinalRequestProcessor@514]
> - shutdown of request processor complete
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):SyncRequestProcessor@191]
> - Shutting down
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,890 - INFO
> [SyncThread:3:SyncRequestProcessor@169] - SyncRequestProcessor exited!
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - WARN
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1300]
> - PeerState set to LOOKING
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1175]
> - LOOKING
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,891 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@903]
> - New election. My id =  3, proposed zxid=0x8a00010a28
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,892 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 3 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,893 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,938 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,939 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,940 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:43 feszoo3 zookeeper[1187]: 2022-10-11 16:01:43,941 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,004 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,019 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,046 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,053 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,057 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,075 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,086 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,090 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,095 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,101 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,113 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,122 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 400
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,142 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,143 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,144 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,149 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,152 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,158 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,159 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,162 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,167 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,179 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,186 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,198 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,217 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,228 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,234 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,235 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,241 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,242 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,246 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,250 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,255 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,260 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,265 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,276 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,280 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,288 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,300 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,307 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,312 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,318 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,322 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,325 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,344 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,351 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,375 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,377 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,379 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,381 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,394 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,395 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,408 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,419 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,424 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,432 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,433 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,439 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,444 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,476 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,482 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,486 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,490 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,495 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,525 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,526 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,535 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,537 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,540 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,541 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,544 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,545 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 800
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,546 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,547 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,553 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,556 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,562 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,580 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,596 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,597 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,607 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,625 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,632 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,638 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,646 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,648 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,667 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,673 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,680 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,690 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,693 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,697 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,708 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,712 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,716 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,723 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,727 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,734 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,740 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,744 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,752 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,765 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,774 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,778 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,803 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,804 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,821 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,832 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,836 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,837 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,854 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,862 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,867 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,872 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,874 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,883 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,886 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,888 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,892 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,895 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,909 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,913 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,920 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,934 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,938 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,939 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,947 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,951 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,956 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,964 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,976 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:44 feszoo3 zookeeper[1187]: 2022-10-11 16:01:44,980 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,012 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,027 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,036 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,070 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,090 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,127 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,197 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,348 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 1600
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,349 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,350 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,351 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,356 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,385 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,451 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,561 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,563 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,576 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,595 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,639 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,654 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,685 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,703 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,744 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:45 feszoo3 zookeeper[1187]: 2022-10-11 16:01:45,753 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,014 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,045 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,062 - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,121 - WARN
> [NIOWorkerThread-11:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,141 - WARN
> [NIOWorkerThread-13:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,151 - WARN
> [NIOWorkerThread-9:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,224 - WARN
> [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,254 - WARN
> [NIOWorkerThread-5:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,322 - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,398 - WARN
> [NIOWorkerThread-12:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,435 - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,522 - WARN
> [NIOWorkerThread-16:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,541 - WARN
> [NIOWorkerThread-14:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,817 - WARN
> [NIOWorkerThread-4:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,873 - WARN
> [NIOWorkerThread-8:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,897 - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,914 - WARN
> [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,951 - INFO
> [QuorumPeer[myid=3](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FastLeaderElection@937]
> - Notification time out: 3200
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,952 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 3 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 4 (n.leader), 0x8a00010a28 (n.zxid), 0x52
> (n.round), LOOKING (n.state), 4 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,954 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 2 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:46 feszoo3 zookeeper[1187]: 2022-10-11 16:01:46,955 - INFO
> [WorkerReceiver[myid=3]:FastLeaderElection@697] - Notification: 2
> (message format version), 5 (n.leader), 0x8900004ede (n.zxid), 0x51
> (n.round), FOLLOWING (n.state), 1 (n.sid), 0x8a (n.peerEPoch), LOOKING (my
> state)0 (n.config version)
> Oct 11 16:01:47 feszoo3 zookeeper[1187]: 2022-10-11 16:01:47,582 - WARN
> [NIOWorkerThread-15:NIOServerCnxn@370] - Exception causing close of
> session 0x0: ZooKeeperServer not running
>
>
> [root@feszoo3 ~]# cat /etc/zookeeper/zoo.cfg
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # The directory where the snapshot is stored.
> dataDir=/var/lib/zookeeper/data
> # The port at which the clients will connect
> clientPort=2181
> skipACL=1
> # 4-letter words: which commands are allowed
> 4lw.commands.whitelist=*
> # Cluster topo
> server.1=feszoo1:2888:3888
> server.2=feszoo2:2888:3888
> server.3=feszoo3:2888:3888
> server.4=feszoo4:2888:3888
> server.5=feszoo5:2888:3888
>
>