You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Benjamin Jaton <be...@gmail.com> on 2018/07/04 02:16:52 UTC
Follower never recovers and keeps saying ZooKeeperServer not running
Hello,
I'm wondering what can cause a ZK follower to check out like this:
2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/10.0.0.248:40282
:LearnerHandler@620] - Unexpected exception causing shutdown while sock
still open
java.io.EOFException: null
at java.io.DataInputStream.readInt(DataInputStream.java:392)
~[?:1.8.0_171]
at
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
at
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
~[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
at
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
[zookeeper-3.5.0-alpha.jar:?]
2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/10.0.0.248:40282
:LearnerHandler@635] - ******* GOODBYE /10.0.0.248:40282 ********
2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/10.0.0.237:54408
:LearnerHandler@620] - Unexpected exception causing shutdown while sock
still open
java.io.EOFException: null
at java.io.DataInputStream.readInt(DataInputStream.java:392)
~[?:1.8.0_171]
at
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
at
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
~[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
at
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
[zookeeper-3.5.0-alpha.jar:?]
2018-07-03T13:43:28,814 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:46754
2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/10.0.0.237:54408
:LearnerHandler@635] - ******* GOODBYE /10.0.0.237:54408 ********
2018-07-03T13:43:28,814 [myid:] - INFO
[ConnnectionExpirer:MBeanRegistry@128] - Unregister MBean
[org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.111,name5=0x200005f37660000]
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33198
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:46780
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33268
2018-07-03T13:43:28,815 [myid:] - INFO
[ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.111:39162 which had sessionid 0x200005f37660000
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:46810
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:46824
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:46826
2018-07-03T13:43:28,815 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@613]
- Shutting down
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33316
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33548
2018-07-03T13:43:28,815 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@619]
- Shutdown called
java.lang.Exception: shutdown Leader! reason: Not sufficient followers
synced, only synced with sids: [ [2] ]
at
org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:619)
[zookeeper-3.5.0-alpha.jar:?]
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
[zookeeper-3.5.0-alpha.jar:?]
2018-07-03T13:43:28,815 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:33198 (no session
established for client)
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128]
- Unregister MBean
[org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.237,name5=0x20000629130001b]
2018-07-03T13:43:28,815 [myid:] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message
format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x0 (n.round), LOOKING
(n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
(n.config version)
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.237:45602 which had sessionid
0x20000629130001b
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128]
- Unregister MBean
[org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a0008]
2018-07-03T13:43:28,816 [myid:] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message
format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round), LOOKING
(n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
(n.config version)
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:53764 which had sessionid
0x2000005cf2a0008
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:33268 (no session
established for client)
2018-07-03T13:43:28,816 [myid:] - WARN
[NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128]
- Unregister MBean
[org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a000a]
2018-07-03T13:43:28,816 [myid:] - INFO
[NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.237:46826 (no session established for client)
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:53758 which had sessionid
0x2000005cf2a000a
2018-07-03T13:43:28,816 [myid:] - INFO
[LearnerCnxAcceptor-0.0.0.0/0.0.0.0:2888:Leader$LearnerCnxAcceptor@373] -
exception while shutting down acceptor: java.net.SocketException: Socket
closed
2018-07-03T13:43:28,816 [myid:] - WARN
[NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T13:43:28,816 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128]
- Unregister MBean
[org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x30000209a080002]
2018-07-03T13:43:28,816 [myid:] - INFO
[NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.237:46810 (no session established for client)
2018-07-03T13:43:28,817 [myid:] - WARN [NIOWorkerThread-6:NIOServerCnxn@369]
- Exception causing close of session 0x0: ZooKeeperServer not running
2018-07-03T13:43:28,817 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:53760 which had sessionid
0x30000209a080002
2018-07-03T13:43:28,817 [myid:] - INFO [NIOWorkerThread-6:NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.237:46780 (no session
established for client)
2018-07-03T13:43:28,817 [myid:] - WARN [NIOWorkerThread-4:NIOServerCnxn@369]
- Exception causing close of session 0x0: ZooKeeperServer not running
2018-07-03T13:43:28,817 [myid:] - INFO [SessionTracker:ZooKeeperServer@384]
- Expiring session 0x300006dbd91000b, timeout of 15000ms exceeded
2018-07-03T13:43:28,817 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498]
- shutting down
2018-07-03T13:43:28,817 [myid:] - INFO [NIOWorkerThread-4:NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:33548 (no session
established for client)
2018-07-03T13:43:28,817 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SessionTrackerImpl@232]
- Shutting down
2018-07-03T13:43:28,817 [myid:] - INFO
[SessionTracker:QuorumZooKeeperServer@132] - Submitting global closeSession
request for session 0x300006dbd91000b
2018-07-03T13:43:28,817 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LeaderRequestProcessor@77]
- Shutting down
2018-07-03T13:43:28,817 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@967]
- Shutting down
2018-07-03T13:43:28,817 [myid:] - INFO
[SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
2018-07-03T13:43:28,817 [myid:] - INFO [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
2018-07-03T13:43:28,815 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:46908
2018-07-03T13:43:28,817 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33786
2018-07-03T13:43:28,817 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:46954
2018-07-03T13:43:28,817 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33992
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47010
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34240
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47084
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47142
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47248
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34532
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47314
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47342
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47384
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34706
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47452
2018-07-03T13:43:28,818 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47496
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47512
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34878
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47580
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47620
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35086
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47700
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47748
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47772
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47886
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47894
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47920
2018-07-03T13:43:28,819 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:47998
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48018
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48020
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48136
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48138
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48140
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48172
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35898
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48294
2018-07-03T13:43:28,820 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48296
2018-07-03T13:43:28,817 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ProposalRequestProcessor@88]
- Shutting down
2018-07-03T13:43:28,816 [myid:] - INFO
[WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message
format version), 1 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round), LOOKING
(n.state), 1 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
(n.config version)
2018-07-03T13:43:28,817 [myid:] - WARN
[NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T13:43:28,821 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48298
2018-07-03T13:43:28,821 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358]
- Shutting down
2018-07-03T13:43:28,821 [myid:] - INFO
[NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.237:46754 (no session established for client)
2018-07-03T13:43:28,821 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:48334
2018-07-03T13:43:28,821 [myid:] - INFO
[CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
2018-07-03T13:43:28,821 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:36162
2018-07-03T13:43:28,821 [myid:] - WARN
[NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T13:43:28,821 [myid:] - WARN
[NIOWorkerThread-16:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T13:43:28,821 [myid:] - WARN
[NIOWorkerThread-31:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T13:43:28,821 [myid:] - WARN
[NIOWorkerThread-11:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
The rest are more failed clients connections attempts and "ZooKeeperServer
not running" messages for hours.
There was an event before that that could be linked:
2018-07-03T12:42:27,639 [myid:] - INFO [NIOWorkerThread-7:NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.237:35536 (no session
established for client)
2018-07-03T12:47:28,864 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33256
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:35568
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33544
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:35672
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:33788
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:35756
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34014
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:35840
2018-07-03T12:47:28,865 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34244
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:35924
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34466
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:35994
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34730
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36086
2018-07-03T12:47:28,866 [myid:] - WARN
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@93]
- Exception when following the leader
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
~[?:1.8.0_171]
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
~[?:1.8.0_171]
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
~[?:1.8.0_171]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[?:1.8.0_171]
at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[?:1.8.0_171]
at
org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
~[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:609)
~[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:116)
~[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:90)
[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
[zookeeper-3.5.0-alpha.jar:?]
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34944
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36166
2018-07-03T12:47:28,866 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128]
- Unregister MBean
[org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:34952
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36172
2018-07-03T12:47:28,866 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198]
- shutdown called
java.lang.Exception: shutdown Follower
at
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)
[zookeeper-3.5.0-alpha.jar:?]
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35178
2018-07-03T12:47:28,866 [myid:] - INFO
[NIOWorkerThread-14:ZooKeeperServer@969] - Client attempting to establish
new session at /10.0.0.248:33256
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36262
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35244
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36276
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35396
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36334
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35472
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36368
2018-07-03T12:47:28,867 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35708
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36446
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:35930
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36530
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:36150
2018-07-03T12:47:28,867 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:33544 (no session
established for client)
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36608
2018-07-03T12:47:28,868 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606]
- Closed socket connection for client /10.0.0.248:33256 which had sessionid
0x200003aa13c0010
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36692
2018-07-03T12:47:28,868 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LearnerZooKeeperServer@167]
- Shutting down
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.248:36418
2018-07-03T12:47:28,868 [myid:] - ERROR
[FollowerRequestProcessor:2:ZooKeeperCriticalThread@48] - Severe
unrecoverable error, from thread : FollowerRequestProcessor:2
java.net.SocketException: Socket closed
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
~[?:1.8.0_171]
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
~[?:1.8.0_171]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[?:1.8.0_171]
at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[?:1.8.0_171]
at
org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
~[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.Learner.request(Learner.java:190)
~[zookeeper-3.5.0-alpha.jar:?]
at
org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:101)
[zookeeper-3.5.0-alpha.jar:?]
2018-07-03T12:47:28,868 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498]
- shutting down
2018-07-03T12:47:28,868 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FollowerRequestProcessor@138]
- Shutting down
2018-07-03T12:47:28,868 [myid:] - INFO
[FollowerRequestProcessor:2:ZooKeeperServer$ZooKeeperServerListenerImpl@475]
- Thread FollowerRequestProcessor:2 exits, error code 1
2018-07-03T12:47:28,868 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358]
- Shutting down
2018-07-03T12:47:28,869 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FinalRequestProcessor@492]
- shutdown of request processor complete
2018-07-03T12:47:28,869 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128]
- Unregister MBean
[org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
2018-07-03T12:47:28,870 [myid:] - INFO
[QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SyncRequestProcessor@191]
- Shutting down
2018-07-03T12:47:28,870 [myid:] - WARN
[NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T12:47:28,870 [myid:] - INFO
[NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.248:34952 (no session established for client)
2018-07-03T12:47:28,870 [myid:] - WARN
[NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T12:47:28,870 [myid:] - INFO
[NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.248:34014 (no session established for client)
2018-07-03T12:47:28,870 [myid:] - WARN
[NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T12:47:28,868 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /
10.0.0.237:36766
2018-07-03T12:47:28,871 [myid:] - INFO
[NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.248:36150 (no session established for client)
2018-07-03T12:47:28,871 [myid:] - WARN
[NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T12:47:28,871 [myid:] - WARN
[NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T12:47:28,871 [myid:] - INFO
[NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.248:33788 (no session established for client)
2018-07-03T12:47:28,871 [myid:] - INFO
[NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
client /10.0.0.248:35930 (no session established for client)
2018-07-03T12:47:28,871 [myid:] - WARN
[NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T12:47:28,871 [myid:] - WARN
[NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of session
0x0: ZooKeeperServer not running
2018-07-03T12:47:28,871 [myid:] - INFO
[CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
In particular I don't like the sound of this "Severe unrecoverable error,
from thread : FollowerRequestProcessor" from "ZooKeeperCriticalThread".
Is this a fatal event for the whole ZK server?
Thanks
Benjamin
Re: Follower never recovers and keeps saying ZooKeeperServer not running
Posted by Benjamin Jaton <be...@gmail.com>.
bump
On Wed, Jul 4, 2018 at 9:31 AM, Steph van Schalkwyk <svanschalkwyk@gmail.com
> wrote:
> ill one of the devs answer. my issue was on solr restart. i incrwased znode
> siE on all and restarted
>
> On Wed, Jul 4, 2018, 10:37 AM Benjamin Jaton <be...@gmail.com>
> wrote:
>
> > Yes - can increasing the ZK node size cause crashes like this?
> >
> > On Wed, Jul 4, 2018 at 8:11 AM Steph van Schalkwyk <
> > svanschalkwyk@gmail.com>
> > wrote:
> >
> > > think default is one m
> > >
> > > On Wed, Jul 4, 2018, 10:04 AM Benjamin Jaton <benjamin.jaton@gmail.com
> >
> > > wrote:
> > >
> > > > Ah interestingly we did increase the znode size to ~ 40m. Think this
> > can
> > > be
> > > > linked?
> > > >
> > > > On Tue, Jul 3, 2018 at 10:55 PM Steph van Schalkwyk <
> > > > svanschalkwyk@gmail.com>
> > > > wrote:
> > > >
> > > > > I've seen this bur can't remember what caused it.
> > > > > What's your zNode size?
> > > > > steph
> > > > >
> > > > > On Tue, Jul 3, 2018, 9:17 PM Benjamin Jaton <
> > benjamin.jaton@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I'm wondering what can cause a ZK follower to check out like
> this:
> > > > > >
> > > > > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > > > > 10.0.0.248:40282
> > > > > > :LearnerHandler@620] - Unexpected exception causing shutdown
> while
> > > > sock
> > > > > > still open
> > > > > > java.io.EOFException: null
> > > > > > at
> > java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > >
> > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.zookeeper.server.quorum.QuorumPacket.
> deserialize(QuorumPacket.java:83)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > >
> > org.apache.jute.BinaryInputArchive.readRecord(
> BinaryInputArchive.java:99)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.zookeeper.server.quorum.LearnerHandler.run(
> LearnerHandler.java:514)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > > > > 10.0.0.248:40282
> > > > > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.248:40282
> ********
> > > > > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > > > > 10.0.0.237:54408
> > > > > > :LearnerHandler@620] - Unexpected exception causing shutdown
> while
> > > > sock
> > > > > > still open
> > > > > > java.io.EOFException: null
> > > > > > at
> > java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > >
> > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.zookeeper.server.quorum.QuorumPacket.
> deserialize(QuorumPacket.java:83)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > >
> > org.apache.jute.BinaryInputArchive.readRecord(
> BinaryInputArchive.java:99)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.zookeeper.server.quorum.LearnerHandler.run(
> LearnerHandler.java:514)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:46754
> > > > > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > > > > 10.0.0.237:54408
> > > > > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.237:54408
> ********
> > > > > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > > > > [ConnnectionExpirer:MBeanRegistry@128] - Unregister MBean
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,
> name1=replica.2,name2=Leader,name3=Connections,name4=10.0.
> 0.111,name5=0x200005f37660000]
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33198
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:46780
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33268
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.111:39162 which had sessionid 0x200005f37660000
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:46810
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:46824
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:46826
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):Leader@613
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33316
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33548
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):Leader@619
> > > > > > ]
> > > > > > - Shutdown called
> > > > > > java.lang.Exception: shutdown Leader! reason: Not sufficient
> > > followers
> > > > > > synced, only synced with sids: [ [2] ]
> > > > > > at
> > > > > > org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.
> java:619)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > org.apache.zookeeper.server.quorum.QuorumPeer.run(
> QuorumPeer.java:1077)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.248:33198 (no
> > session
> > > > > > established for client)
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):MBeanRegistry@128
> > > > > > ]
> > > > > > - Unregister MBean
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,
> name1=replica.2,name2=Leader,name3=Connections,name4=10.0.
> 0.237,name5=0x20000629130001b]
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification:
> 2
> > > > > (message
> > > > > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x0
> (n.round),
> > > > > LOOKING
> > > > > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my
> > state)300000066
> > > > > > (n.config version)
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.237:45602 which
> had
> > > > > > sessionid
> > > > > > 0x20000629130001b
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):MBeanRegistry@128
> > > > > > ]
> > > > > > - Unregister MBean
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,
> name1=replica.2,name2=Leader,name3=Connections,name4=10.0.
> 0.248,name5=0x2000005cf2a0008]
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification:
> 2
> > > > > (message
> > > > > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x7
> (n.round),
> > > > > LOOKING
> > > > > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my
> > state)300000066
> > > > > > (n.config version)
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.248:53764 which
> had
> > > > > > sessionid
> > > > > > 0x2000005cf2a0008
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.248:33268 (no
> > session
> > > > > > established for client)
> > > > > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):MBeanRegistry@128
> > > > > > ]
> > > > > > - Unregister MBean
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,
> name1=replica.2,name2=Leader,name3=Connections,name4=10.0.
> 0.248,name5=0x2000005cf2a000a]
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.237:46826 (no session established for client)
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.248:53758 which
> had
> > > > > > sessionid
> > > > > > 0x2000005cf2a000a
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > > [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:2888
> > > :Leader$LearnerCnxAcceptor@373
> > > > ]
> > > > > -
> > > > > > exception while shutting down acceptor: java.net.SocketException:
> > > > Socket
> > > > > > closed
> > > > > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):MBeanRegistry@128
> > > > > > ]
> > > > > > - Unregister MBean
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,
> name1=replica.2,name2=Leader,name3=Connections,name4=10.0.
> 0.248,name5=0x30000209a080002]
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.237:46810 (no session established for client)
> > > > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > > > [NIOWorkerThread-6:NIOServerCnxn@369]
> > > > > > - Exception causing close of session 0x0: ZooKeeperServer not
> > running
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.248:53760 which
> had
> > > > > > sessionid
> > > > > > 0x30000209a080002
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > > [NIOWorkerThread-6:NIOServerCnxn@606]
> > > > > > - Closed socket connection for client /10.0.0.237:46780 (no
> > session
> > > > > > established for client)
> > > > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > > > [NIOWorkerThread-4:NIOServerCnxn@369]
> > > > > > - Exception causing close of session 0x0: ZooKeeperServer not
> > running
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [SessionTracker:ZooKeeperServer@384
> > > > > > ]
> > > > > > - Expiring session 0x300006dbd91000b, timeout of 15000ms exceeded
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):ZooKeeperServer@498
> > > > > > ]
> > > > > > - shutting down
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > > [NIOWorkerThread-4:NIOServerCnxn@606]
> > > > > > - Closed socket connection for client /10.0.0.248:33548 (no
> > session
> > > > > > established for client)
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):SessionTrackerImpl@232
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > > [SessionTracker:QuorumZooKeeperServer@132] - Submitting global
> > > > > > closeSession
> > > > > > request for session 0x300006dbd91000b
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):
> LeaderRequestProcessor@77
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):
> PrepRequestProcessor@967
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > > [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl
> > exited
> > > > > loop!
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO [ProcessThread(sid:2
> > > > > > cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor
> exited
> > > > loop!
> > > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:46908
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33786
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:46954
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33992
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47010
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34240
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47084
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47142
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47248
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34532
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47314
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47342
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47384
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34706
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47452
> > > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47496
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47512
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34878
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47580
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47620
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35086
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47700
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47748
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47772
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47886
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47894
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47920
> > > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:47998
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48018
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48020
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48136
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48138
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48140
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48172
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35898
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48294
> > > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48296
> > > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):
> ProposalRequestProcessor@88
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification:
> 2
> > > > > (message
> > > > > > format version), 1 (n.leader), 0x9000001cb (n.zxid), 0x7
> (n.round),
> > > > > LOOKING
> > > > > > (n.state), 1 (n.sid), 0x9 (n.peerEPoch), LEADING (my
> > state)300000066
> > > > > > (n.config version)
> > > > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48298
> > > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):CommitProcessor@358
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.237:46754 (no session established for client)
> > > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:48334
> > > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited
> > > loop!
> > > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:36162
> > > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > > [NIOWorkerThread-16:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > > [NIOWorkerThread-31:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > > [NIOWorkerThread-11:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > >
> > > > > > The rest are more failed clients connections attempts and
> > > > > "ZooKeeperServer
> > > > > > not running" messages for hours.
> > > > > >
> > > > > >
> > > > > > There was an event before that that could be linked:
> > > > > >
> > > > > > 2018-07-03T12:42:27,639 [myid:] - INFO
> > > > > > [NIOWorkerThread-7:NIOServerCnxn@606]
> > > > > > - Closed socket connection for client /10.0.0.237:35536 (no
> > session
> > > > > > established for client)
> > > > > > 2018-07-03T12:47:28,864 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33256
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:35568
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33544
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:35672
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:33788
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:35756
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34014
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:35840
> > > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34244
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:35924
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34466
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:35994
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34730
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36086
> > > > > > 2018-07-03T12:47:28,866 [myid:] - WARN
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):Follower@93
> > > > > > ]
> > > > > > - Exception when following the leader
> > > > > > java.net.SocketException: Broken pipe (Write failed)
> > > > > > at java.net.SocketOutputStream.socketWrite0(Native
> Method)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > >
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > >
> > > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > > java.io.BufferedOutputStream.flush(BufferedOutputStream.
> java:140)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > >
> > > >
> > org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > > org.apache.zookeeper.server.quorum.Learner.ping(Learner.
> java:609)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.zookeeper.server.quorum.Follower.processPacket(
> Follower.java:116)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.zookeeper.server.quorum.Follower.followLeader(
> Follower.java:90)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > org.apache.zookeeper.server.quorum.QuorumPeer.run(
> QuorumPeer.java:1064)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34944
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36166
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):MBeanRegistry@128
> > > > > > ]
> > > > > > - Unregister MBean
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,
> name1=replica.2,name2=Follower]
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:34952
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36172
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):Follower@198
> > > > > > ]
> > > > > > - shutdown called
> > > > > > java.lang.Exception: shutdown Follower
> > > > > > at
> > > > > >
> > > org.apache.zookeeper.server.quorum.Follower.shutdown(
> Follower.java:198)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > org.apache.zookeeper.server.quorum.QuorumPeer.run(
> QuorumPeer.java:1068)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35178
> > > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > > [NIOWorkerThread-14:ZooKeeperServer@969] - Client attempting to
> > > > > establish
> > > > > > new session at /10.0.0.248:33256
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36262
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35244
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36276
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35396
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36334
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35472
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36368
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35708
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36446
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:35930
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36530
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:36150
> > > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.248:33544 (no
> > session
> > > > > > established for client)
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36608
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):NIOServerCnxn@606
> > > > > > ]
> > > > > > - Closed socket connection for client /10.0.0.248:33256 which
> had
> > > > > > sessionid
> > > > > > 0x200003aa13c0010
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36692
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):
> LearnerZooKeeperServer@167
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.248:36418
> > > > > > 2018-07-03T12:47:28,868 [myid:] - ERROR
> > > > > > [FollowerRequestProcessor:2:ZooKeeperCriticalThread@48] - Severe
> > > > > > unrecoverable error, from thread : FollowerRequestProcessor:2
> > > > > > java.net.SocketException: Socket closed
> > > > > > at
> > > > > >
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > >
> > > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > > java.io.BufferedOutputStream.flush(BufferedOutputStream.
> java:140)
> > > > > > ~[?:1.8.0_171]
> > > > > > at
> > > > > >
> > > >
> > org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > org.apache.zookeeper.server.quorum.Learner.request(Learner.java:190)
> > > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(
> FollowerRequestProcessor.java:101)
> > > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):ZooKeeperServer@498
> > > > > > ]
> > > > > > - shutting down
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):
> FollowerRequestProcessor@138
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > >
> > > > >
> > > >
> > >
> > [FollowerRequestProcessor:2:ZooKeeperServer$ZooKeeperServerListenerImpl@
> 475
> > > > > > ]
> > > > > > - Thread FollowerRequestProcessor:2 exits, error code 1
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):CommitProcessor@358
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):
> FinalRequestProcessor@492
> > > > > > ]
> > > > > > - shutdown of request processor complete
> > > > > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=
> disabled):MBeanRegistry@128
> > > > > > ]
> > > > > > - Unregister MBean
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,
> name1=replica.2,name2=Follower,name3=InMemoryDataTree]
> > > > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):
> SyncRequestProcessor@191
> > > > > > ]
> > > > > > - Shutting down
> > > > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.248:34952 (no session established for client)
> > > > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.248:34014 (no session established for client)
> > > > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> > connection
> > > > > from
> > > > > > /
> > > > > > 10.0.0.237:36766
> > > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.248:36150 (no session established for client)
> > > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.248:33788 (no session established for client)
> > > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket
> connection
> > > for
> > > > > > client /10.0.0.248:35930 (no session established for client)
> > > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> > of
> > > > > > session
> > > > > > 0x0: ZooKeeperServer not running
> > > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited
> > > loop!
> > > > > >
> > > > > > In particular I don't like the sound of this "Severe
> unrecoverable
> > > > error,
> > > > > > from thread : FollowerRequestProcessor" from
> > > "ZooKeeperCriticalThread".
> > > > > >
> > > > > > Is this a fatal event for the whole ZK server?
> > > > > >
> > > > > > Thanks
> > > > > > Benjamin
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Follower never recovers and keeps saying ZooKeeperServer not running
Posted by Steph van Schalkwyk <sv...@gmail.com>.
ill one of the devs answer. my issue was on solr restart. i incrwased znode
siE on all and restarted
On Wed, Jul 4, 2018, 10:37 AM Benjamin Jaton <be...@gmail.com>
wrote:
> Yes - can increasing the ZK node size cause crashes like this?
>
> On Wed, Jul 4, 2018 at 8:11 AM Steph van Schalkwyk <
> svanschalkwyk@gmail.com>
> wrote:
>
> > think default is one m
> >
> > On Wed, Jul 4, 2018, 10:04 AM Benjamin Jaton <be...@gmail.com>
> > wrote:
> >
> > > Ah interestingly we did increase the znode size to ~ 40m. Think this
> can
> > be
> > > linked?
> > >
> > > On Tue, Jul 3, 2018 at 10:55 PM Steph van Schalkwyk <
> > > svanschalkwyk@gmail.com>
> > > wrote:
> > >
> > > > I've seen this bur can't remember what caused it.
> > > > What's your zNode size?
> > > > steph
> > > >
> > > > On Tue, Jul 3, 2018, 9:17 PM Benjamin Jaton <
> benjamin.jaton@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I'm wondering what can cause a ZK follower to check out like this:
> > > > >
> > > > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > > > 10.0.0.248:40282
> > > > > :LearnerHandler@620] - Unexpected exception causing shutdown while
> > > sock
> > > > > still open
> > > > > java.io.EOFException: null
> > > > > at
> java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > >
> > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > >
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > > > 10.0.0.248:40282
> > > > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.248:40282 ********
> > > > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > > > 10.0.0.237:54408
> > > > > :LearnerHandler@620] - Unexpected exception causing shutdown while
> > > sock
> > > > > still open
> > > > > java.io.EOFException: null
> > > > > at
> java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > >
> > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > >
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:46754
> > > > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > > > 10.0.0.237:54408
> > > > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.237:54408 ********
> > > > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > > > [ConnnectionExpirer:MBeanRegistry@128] - Unregister MBean
> > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.111,name5=0x200005f37660000]
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33198
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:46780
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33268
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.111:39162 which had sessionid 0x200005f37660000
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:46810
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:46824
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:46826
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@613
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33316
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33548
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@619
> > > > > ]
> > > > > - Shutdown called
> > > > > java.lang.Exception: shutdown Leader! reason: Not sufficient
> > followers
> > > > > synced, only synced with sids: [ [2] ]
> > > > > at
> > > > > org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:619)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.248:33198 (no
> session
> > > > > established for client)
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > > ]
> > > > > - Unregister MBean
> > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.237,name5=0x20000629130001b]
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > > > (message
> > > > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x0 (n.round),
> > > > LOOKING
> > > > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my
> state)300000066
> > > > > (n.config version)
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.237:45602 which had
> > > > > sessionid
> > > > > 0x20000629130001b
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > > ]
> > > > > - Unregister MBean
> > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a0008]
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > > > (message
> > > > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> > > > LOOKING
> > > > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my
> state)300000066
> > > > > (n.config version)
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.248:53764 which had
> > > > > sessionid
> > > > > 0x2000005cf2a0008
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.248:33268 (no
> session
> > > > > established for client)
> > > > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > > ]
> > > > > - Unregister MBean
> > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a000a]
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.237:46826 (no session established for client)
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.248:53758 which had
> > > > > sessionid
> > > > > 0x2000005cf2a000a
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:2888
> > :Leader$LearnerCnxAcceptor@373
> > > ]
> > > > -
> > > > > exception while shutting down acceptor: java.net.SocketException:
> > > Socket
> > > > > closed
> > > > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > > ]
> > > > > - Unregister MBean
> > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x30000209a080002]
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.237:46810 (no session established for client)
> > > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > > [NIOWorkerThread-6:NIOServerCnxn@369]
> > > > > - Exception causing close of session 0x0: ZooKeeperServer not
> running
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.248:53760 which had
> > > > > sessionid
> > > > > 0x30000209a080002
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [NIOWorkerThread-6:NIOServerCnxn@606]
> > > > > - Closed socket connection for client /10.0.0.237:46780 (no
> session
> > > > > established for client)
> > > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > > [NIOWorkerThread-4:NIOServerCnxn@369]
> > > > > - Exception causing close of session 0x0: ZooKeeperServer not
> running
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [SessionTracker:ZooKeeperServer@384
> > > > > ]
> > > > > - Expiring session 0x300006dbd91000b, timeout of 15000ms exceeded
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > > > > ]
> > > > > - shutting down
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [NIOWorkerThread-4:NIOServerCnxn@606]
> > > > > - Closed socket connection for client /10.0.0.248:33548 (no
> session
> > > > > established for client)
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SessionTrackerImpl@232
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [SessionTracker:QuorumZooKeeperServer@132] - Submitting global
> > > > > closeSession
> > > > > request for session 0x300006dbd91000b
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LeaderRequestProcessor@77
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@967
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl
> exited
> > > > loop!
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO [ProcessThread(sid:2
> > > > > cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited
> > > loop!
> > > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:46908
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33786
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:46954
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33992
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47010
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34240
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47084
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47142
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47248
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34532
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47314
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47342
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47384
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34706
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47452
> > > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47496
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47512
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34878
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47580
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47620
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35086
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47700
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47748
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47772
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47886
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47894
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47920
> > > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:47998
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48018
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48020
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48136
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48138
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48140
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48172
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35898
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48294
> > > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48296
> > > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ProposalRequestProcessor@88
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > > > (message
> > > > > format version), 1 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> > > > LOOKING
> > > > > (n.state), 1 (n.sid), 0x9 (n.peerEPoch), LEADING (my
> state)300000066
> > > > > (n.config version)
> > > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48298
> > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.237:46754 (no session established for client)
> > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:48334
> > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited
> > loop!
> > > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:36162
> > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > [NIOWorkerThread-16:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > [NIOWorkerThread-31:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > > [NIOWorkerThread-11:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > >
> > > > > The rest are more failed clients connections attempts and
> > > > "ZooKeeperServer
> > > > > not running" messages for hours.
> > > > >
> > > > >
> > > > > There was an event before that that could be linked:
> > > > >
> > > > > 2018-07-03T12:42:27,639 [myid:] - INFO
> > > > > [NIOWorkerThread-7:NIOServerCnxn@606]
> > > > > - Closed socket connection for client /10.0.0.237:35536 (no
> session
> > > > > established for client)
> > > > > 2018-07-03T12:47:28,864 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33256
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:35568
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33544
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:35672
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:33788
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:35756
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34014
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:35840
> > > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34244
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:35924
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34466
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:35994
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34730
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36086
> > > > > 2018-07-03T12:47:28,866 [myid:] - WARN
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@93
> > > > > ]
> > > > > - Exception when following the leader
> > > > > java.net.SocketException: Broken pipe (Write failed)
> > > > > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > >
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > >
> > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > >
> > >
> org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > > org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:609)
> > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:116)
> > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:90)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34944
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36166
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > > ]
> > > > > - Unregister MBean
> > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:34952
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36172
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198
> > > > > ]
> > > > > - shutdown called
> > > > > java.lang.Exception: shutdown Follower
> > > > > at
> > > > >
> > org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35178
> > > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > > [NIOWorkerThread-14:ZooKeeperServer@969] - Client attempting to
> > > > establish
> > > > > new session at /10.0.0.248:33256
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36262
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35244
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36276
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35396
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36334
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35472
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36368
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35708
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36446
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:35930
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36530
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:36150
> > > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.248:33544 (no
> session
> > > > > established for client)
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36608
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > > ]
> > > > > - Closed socket connection for client /10.0.0.248:33256 which had
> > > > > sessionid
> > > > > 0x200003aa13c0010
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36692
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LearnerZooKeeperServer@167
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.248:36418
> > > > > 2018-07-03T12:47:28,868 [myid:] - ERROR
> > > > > [FollowerRequestProcessor:2:ZooKeeperCriticalThread@48] - Severe
> > > > > unrecoverable error, from thread : FollowerRequestProcessor:2
> > > > > java.net.SocketException: Socket closed
> > > > > at
> > > > >
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > >
> > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > > > > ~[?:1.8.0_171]
> > > > > at
> > > > >
> > >
> org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> org.apache.zookeeper.server.quorum.Learner.request(Learner.java:190)
> > > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:101)
> > > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > > > > ]
> > > > > - shutting down
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FollowerRequestProcessor@138
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > >
> > > >
> > >
> >
> [FollowerRequestProcessor:2:ZooKeeperServer$ZooKeeperServerListenerImpl@475
> > > > > ]
> > > > > - Thread FollowerRequestProcessor:2 exits, error code 1
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FinalRequestProcessor@492
> > > > > ]
> > > > > - shutdown of request processor complete
> > > > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > > ]
> > > > > - Unregister MBean
> > > > >
> > > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
> > > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > >
> > > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SyncRequestProcessor@191
> > > > > ]
> > > > > - Shutting down
> > > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.248:34952 (no session established for client)
> > > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.248:34014 (no session established for client)
> > > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket
> connection
> > > > from
> > > > > /
> > > > > 10.0.0.237:36766
> > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.248:36150 (no session established for client)
> > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.248:33788 (no session established for client)
> > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection
> > for
> > > > > client /10.0.0.248:35930 (no session established for client)
> > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close
> of
> > > > > session
> > > > > 0x0: ZooKeeperServer not running
> > > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited
> > loop!
> > > > >
> > > > > In particular I don't like the sound of this "Severe unrecoverable
> > > error,
> > > > > from thread : FollowerRequestProcessor" from
> > "ZooKeeperCriticalThread".
> > > > >
> > > > > Is this a fatal event for the whole ZK server?
> > > > >
> > > > > Thanks
> > > > > Benjamin
> > > > >
> > > >
> > >
> >
>
Re: Follower never recovers and keeps saying ZooKeeperServer not running
Posted by Benjamin Jaton <be...@gmail.com>.
Yes - can increasing the ZK node size cause crashes like this?
On Wed, Jul 4, 2018 at 8:11 AM Steph van Schalkwyk <sv...@gmail.com>
wrote:
> think default is one m
>
> On Wed, Jul 4, 2018, 10:04 AM Benjamin Jaton <be...@gmail.com>
> wrote:
>
> > Ah interestingly we did increase the znode size to ~ 40m. Think this can
> be
> > linked?
> >
> > On Tue, Jul 3, 2018 at 10:55 PM Steph van Schalkwyk <
> > svanschalkwyk@gmail.com>
> > wrote:
> >
> > > I've seen this bur can't remember what caused it.
> > > What's your zNode size?
> > > steph
> > >
> > > On Tue, Jul 3, 2018, 9:17 PM Benjamin Jaton <be...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I'm wondering what can cause a ZK follower to check out like this:
> > > >
> > > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > > 10.0.0.248:40282
> > > > :LearnerHandler@620] - Unexpected exception causing shutdown while
> > sock
> > > > still open
> > > > java.io.EOFException: null
> > > > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > > ~[?:1.8.0_171]
> > > > at
> > > >
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > > 10.0.0.248:40282
> > > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.248:40282 ********
> > > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > > 10.0.0.237:54408
> > > > :LearnerHandler@620] - Unexpected exception causing shutdown while
> > sock
> > > > still open
> > > > java.io.EOFException: null
> > > > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > > ~[?:1.8.0_171]
> > > > at
> > > >
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:46754
> > > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > > 10.0.0.237:54408
> > > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.237:54408 ********
> > > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > > [ConnnectionExpirer:MBeanRegistry@128] - Unregister MBean
> > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.111,name5=0x200005f37660000]
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33198
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:46780
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33268
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.111:39162 which had sessionid 0x200005f37660000
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:46810
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:46824
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:46826
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@613
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33316
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33548
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@619
> > > > ]
> > > > - Shutdown called
> > > > java.lang.Exception: shutdown Leader! reason: Not sufficient
> followers
> > > > synced, only synced with sids: [ [2] ]
> > > > at
> > > > org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:619)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.248:33198 (no session
> > > > established for client)
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > ]
> > > > - Unregister MBean
> > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.237,name5=0x20000629130001b]
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > > (message
> > > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x0 (n.round),
> > > LOOKING
> > > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > > > (n.config version)
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.237:45602 which had
> > > > sessionid
> > > > 0x20000629130001b
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > ]
> > > > - Unregister MBean
> > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a0008]
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > > (message
> > > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> > > LOOKING
> > > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > > > (n.config version)
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.248:53764 which had
> > > > sessionid
> > > > 0x2000005cf2a0008
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.248:33268 (no session
> > > > established for client)
> > > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > ]
> > > > - Unregister MBean
> > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a000a]
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.237:46826 (no session established for client)
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.248:53758 which had
> > > > sessionid
> > > > 0x2000005cf2a000a
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:2888
> :Leader$LearnerCnxAcceptor@373
> > ]
> > > -
> > > > exception while shutting down acceptor: java.net.SocketException:
> > Socket
> > > > closed
> > > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > ]
> > > > - Unregister MBean
> > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x30000209a080002]
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.237:46810 (no session established for client)
> > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > [NIOWorkerThread-6:NIOServerCnxn@369]
> > > > - Exception causing close of session 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.248:53760 which had
> > > > sessionid
> > > > 0x30000209a080002
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [NIOWorkerThread-6:NIOServerCnxn@606]
> > > > - Closed socket connection for client /10.0.0.237:46780 (no session
> > > > established for client)
> > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > [NIOWorkerThread-4:NIOServerCnxn@369]
> > > > - Exception causing close of session 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [SessionTracker:ZooKeeperServer@384
> > > > ]
> > > > - Expiring session 0x300006dbd91000b, timeout of 15000ms exceeded
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > > > ]
> > > > - shutting down
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [NIOWorkerThread-4:NIOServerCnxn@606]
> > > > - Closed socket connection for client /10.0.0.248:33548 (no session
> > > > established for client)
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SessionTrackerImpl@232
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [SessionTracker:QuorumZooKeeperServer@132] - Submitting global
> > > > closeSession
> > > > request for session 0x300006dbd91000b
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LeaderRequestProcessor@77
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@967
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited
> > > loop!
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO [ProcessThread(sid:2
> > > > cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited
> > loop!
> > > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:46908
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33786
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:46954
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33992
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47010
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34240
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47084
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47142
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47248
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34532
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47314
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47342
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47384
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34706
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47452
> > > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47496
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47512
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34878
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47580
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47620
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35086
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47700
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47748
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47772
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47886
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47894
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47920
> > > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:47998
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48018
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48020
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48136
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48138
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48140
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48172
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35898
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48294
> > > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48296
> > > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ProposalRequestProcessor@88
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > > (message
> > > > format version), 1 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> > > LOOKING
> > > > (n.state), 1 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > > > (n.config version)
> > > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48298
> > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.237:46754 (no session established for client)
> > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:48334
> > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited
> loop!
> > > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:36162
> > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > [NIOWorkerThread-16:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > [NIOWorkerThread-31:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > > [NIOWorkerThread-11:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > >
> > > > The rest are more failed clients connections attempts and
> > > "ZooKeeperServer
> > > > not running" messages for hours.
> > > >
> > > >
> > > > There was an event before that that could be linked:
> > > >
> > > > 2018-07-03T12:42:27,639 [myid:] - INFO
> > > > [NIOWorkerThread-7:NIOServerCnxn@606]
> > > > - Closed socket connection for client /10.0.0.237:35536 (no session
> > > > established for client)
> > > > 2018-07-03T12:47:28,864 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33256
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:35568
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33544
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:35672
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:33788
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:35756
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34014
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:35840
> > > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34244
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:35924
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34466
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:35994
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34730
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36086
> > > > 2018-07-03T12:47:28,866 [myid:] - WARN
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@93
> > > > ]
> > > > - Exception when following the leader
> > > > java.net.SocketException: Broken pipe (Write failed)
> > > > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > > > ~[?:1.8.0_171]
> > > > at
> > > > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> > > > ~[?:1.8.0_171]
> > > > at
> > java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > > ~[?:1.8.0_171]
> > > > at
> > > >
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > > ~[?:1.8.0_171]
> > > > at
> > > > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > > > ~[?:1.8.0_171]
> > > > at
> > > >
> > org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > > org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:609)
> > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:116)
> > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:90)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34944
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36166
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > ]
> > > > - Unregister MBean
> > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:34952
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36172
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198
> > > > ]
> > > > - shutdown called
> > > > java.lang.Exception: shutdown Follower
> > > > at
> > > >
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35178
> > > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > > [NIOWorkerThread-14:ZooKeeperServer@969] - Client attempting to
> > > establish
> > > > new session at /10.0.0.248:33256
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36262
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35244
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36276
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35396
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36334
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35472
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36368
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35708
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36446
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:35930
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36530
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:36150
> > > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.248:33544 (no session
> > > > established for client)
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36608
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > > ]
> > > > - Closed socket connection for client /10.0.0.248:33256 which had
> > > > sessionid
> > > > 0x200003aa13c0010
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36692
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LearnerZooKeeperServer@167
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.248:36418
> > > > 2018-07-03T12:47:28,868 [myid:] - ERROR
> > > > [FollowerRequestProcessor:2:ZooKeeperCriticalThread@48] - Severe
> > > > unrecoverable error, from thread : FollowerRequestProcessor:2
> > > > java.net.SocketException: Socket closed
> > > > at
> > > > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> > > > ~[?:1.8.0_171]
> > > > at
> > java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > > ~[?:1.8.0_171]
> > > > at
> > > >
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > > ~[?:1.8.0_171]
> > > > at
> > > > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > > > ~[?:1.8.0_171]
> > > > at
> > > >
> > org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > > org.apache.zookeeper.server.quorum.Learner.request(Learner.java:190)
> > > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:101)
> > > > [zookeeper-3.5.0-alpha.jar:?]
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > > > ]
> > > > - shutting down
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FollowerRequestProcessor@138
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > >
> > >
> >
> [FollowerRequestProcessor:2:ZooKeeperServer$ZooKeeperServerListenerImpl@475
> > > > ]
> > > > - Thread FollowerRequestProcessor:2 exits, error code 1
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FinalRequestProcessor@492
> > > > ]
> > > > - shutdown of request processor complete
> > > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > > ]
> > > > - Unregister MBean
> > > >
> > > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
> > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > >
> > > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SyncRequestProcessor@191
> > > > ]
> > > > - Shutting down
> > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.248:34952 (no session established for client)
> > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.248:34014 (no session established for client)
> > > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > > from
> > > > /
> > > > 10.0.0.237:36766
> > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.248:36150 (no session established for client)
> > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.248:33788 (no session established for client)
> > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection
> for
> > > > client /10.0.0.248:35930 (no session established for client)
> > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > > session
> > > > 0x0: ZooKeeperServer not running
> > > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited
> loop!
> > > >
> > > > In particular I don't like the sound of this "Severe unrecoverable
> > error,
> > > > from thread : FollowerRequestProcessor" from
> "ZooKeeperCriticalThread".
> > > >
> > > > Is this a fatal event for the whole ZK server?
> > > >
> > > > Thanks
> > > > Benjamin
> > > >
> > >
> >
>
Re: Follower never recovers and keeps saying ZooKeeperServer not running
Posted by Steph van Schalkwyk <sv...@gmail.com>.
think default is one m
On Wed, Jul 4, 2018, 10:04 AM Benjamin Jaton <be...@gmail.com>
wrote:
> Ah interestingly we did increase the znode size to ~ 40m. Think this can be
> linked?
>
> On Tue, Jul 3, 2018 at 10:55 PM Steph van Schalkwyk <
> svanschalkwyk@gmail.com>
> wrote:
>
> > I've seen this bur can't remember what caused it.
> > What's your zNode size?
> > steph
> >
> > On Tue, Jul 3, 2018, 9:17 PM Benjamin Jaton <be...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I'm wondering what can cause a ZK follower to check out like this:
> > >
> > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > 10.0.0.248:40282
> > > :LearnerHandler@620] - Unexpected exception causing shutdown while
> sock
> > > still open
> > > java.io.EOFException: null
> > > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > ~[?:1.8.0_171]
> > > at
> > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > at
> > >
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > 10.0.0.248:40282
> > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.248:40282 ********
> > > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> > 10.0.0.237:54408
> > > :LearnerHandler@620] - Unexpected exception causing shutdown while
> sock
> > > still open
> > > java.io.EOFException: null
> > > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > ~[?:1.8.0_171]
> > > at
> > > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > at
> > >
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:46754
> > > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> > 10.0.0.237:54408
> > > :LearnerHandler@635] - ******* GOODBYE /10.0.0.237:54408 ********
> > > 2018-07-03T13:43:28,814 [myid:] - INFO
> > > [ConnnectionExpirer:MBeanRegistry@128] - Unregister MBean
> > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.111,name5=0x200005f37660000]
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33198
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:46780
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33268
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.111:39162 which had sessionid 0x200005f37660000
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:46810
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:46824
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:46826
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@613
> > > ]
> > > - Shutting down
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33316
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33548
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@619
> > > ]
> > > - Shutdown called
> > > java.lang.Exception: shutdown Leader! reason: Not sufficient followers
> > > synced, only synced with sids: [ [2] ]
> > > at
> > > org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:619)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > at
> > org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > at
> > > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.248:33198 (no session
> > > established for client)
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > ]
> > > - Unregister MBean
> > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.237,name5=0x20000629130001b]
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > (message
> > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x0 (n.round),
> > LOOKING
> > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > > (n.config version)
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.237:45602 which had
> > > sessionid
> > > 0x20000629130001b
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > ]
> > > - Unregister MBean
> > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a0008]
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > (message
> > > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> > LOOKING
> > > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > > (n.config version)
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.248:53764 which had
> > > sessionid
> > > 0x2000005cf2a0008
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.248:33268 (no session
> > > established for client)
> > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > ]
> > > - Unregister MBean
> > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a000a]
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.237:46826 (no session established for client)
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.248:53758 which had
> > > sessionid
> > > 0x2000005cf2a000a
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:2888:Leader$LearnerCnxAcceptor@373
> ]
> > -
> > > exception while shutting down acceptor: java.net.SocketException:
> Socket
> > > closed
> > > 2018-07-03T13:43:28,816 [myid:] - WARN
> > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > ]
> > > - Unregister MBean
> > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x30000209a080002]
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.237:46810 (no session established for client)
> > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > [NIOWorkerThread-6:NIOServerCnxn@369]
> > > - Exception causing close of session 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.248:53760 which had
> > > sessionid
> > > 0x30000209a080002
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [NIOWorkerThread-6:NIOServerCnxn@606]
> > > - Closed socket connection for client /10.0.0.237:46780 (no session
> > > established for client)
> > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > [NIOWorkerThread-4:NIOServerCnxn@369]
> > > - Exception causing close of session 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [SessionTracker:ZooKeeperServer@384
> > > ]
> > > - Expiring session 0x300006dbd91000b, timeout of 15000ms exceeded
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > > ]
> > > - shutting down
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [NIOWorkerThread-4:NIOServerCnxn@606]
> > > - Closed socket connection for client /10.0.0.248:33548 (no session
> > > established for client)
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SessionTrackerImpl@232
> > > ]
> > > - Shutting down
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [SessionTracker:QuorumZooKeeperServer@132] - Submitting global
> > > closeSession
> > > request for session 0x300006dbd91000b
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LeaderRequestProcessor@77
> > > ]
> > > - Shutting down
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@967
> > > ]
> > > - Shutting down
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited
> > loop!
> > > 2018-07-03T13:43:28,817 [myid:] - INFO [ProcessThread(sid:2
> > > cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited
> loop!
> > > 2018-07-03T13:43:28,815 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:46908
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33786
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:46954
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33992
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47010
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34240
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47084
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47142
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47248
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34532
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47314
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47342
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47384
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34706
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47452
> > > 2018-07-03T13:43:28,818 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47496
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47512
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34878
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47580
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47620
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35086
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47700
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47748
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47772
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47886
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47894
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47920
> > > 2018-07-03T13:43:28,819 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:47998
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48018
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48020
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48136
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48138
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48140
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48172
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35898
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48294
> > > 2018-07-03T13:43:28,820 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48296
> > > 2018-07-03T13:43:28,817 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ProposalRequestProcessor@88
> > > ]
> > > - Shutting down
> > > 2018-07-03T13:43:28,816 [myid:] - INFO
> > > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> > (message
> > > format version), 1 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> > LOOKING
> > > (n.state), 1 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > > (n.config version)
> > > 2018-07-03T13:43:28,817 [myid:] - WARN
> > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48298
> > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > > ]
> > > - Shutting down
> > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.237:46754 (no session established for client)
> > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:48334
> > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
> > > 2018-07-03T13:43:28,821 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:36162
> > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > [NIOWorkerThread-16:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > [NIOWorkerThread-31:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T13:43:28,821 [myid:] - WARN
> > > [NIOWorkerThread-11:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > >
> > > The rest are more failed clients connections attempts and
> > "ZooKeeperServer
> > > not running" messages for hours.
> > >
> > >
> > > There was an event before that that could be linked:
> > >
> > > 2018-07-03T12:42:27,639 [myid:] - INFO
> > > [NIOWorkerThread-7:NIOServerCnxn@606]
> > > - Closed socket connection for client /10.0.0.237:35536 (no session
> > > established for client)
> > > 2018-07-03T12:47:28,864 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33256
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:35568
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33544
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:35672
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:33788
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:35756
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34014
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:35840
> > > 2018-07-03T12:47:28,865 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34244
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:35924
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34466
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:35994
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34730
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36086
> > > 2018-07-03T12:47:28,866 [myid:] - WARN
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@93
> > > ]
> > > - Exception when following the leader
> > > java.net.SocketException: Broken pipe (Write failed)
> > > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > > ~[?:1.8.0_171]
> > > at
> > > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> > > ~[?:1.8.0_171]
> > > at
> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > ~[?:1.8.0_171]
> > > at
> > > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > ~[?:1.8.0_171]
> > > at
> > > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > > ~[?:1.8.0_171]
> > > at
> > >
> org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > at
> > > org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:609)
> > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:116)
> > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > at
> > >
> >
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:90)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > at
> > > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34944
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36166
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > ]
> > > - Unregister MBean
> > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:34952
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36172
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198
> > > ]
> > > - shutdown called
> > > java.lang.Exception: shutdown Follower
> > > at
> > > org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > at
> > > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35178
> > > 2018-07-03T12:47:28,866 [myid:] - INFO
> > > [NIOWorkerThread-14:ZooKeeperServer@969] - Client attempting to
> > establish
> > > new session at /10.0.0.248:33256
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36262
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35244
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36276
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35396
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36334
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35472
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36368
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35708
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36446
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:35930
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36530
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:36150
> > > 2018-07-03T12:47:28,867 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.248:33544 (no session
> > > established for client)
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36608
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > > ]
> > > - Closed socket connection for client /10.0.0.248:33256 which had
> > > sessionid
> > > 0x200003aa13c0010
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36692
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LearnerZooKeeperServer@167
> > > ]
> > > - Shutting down
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.248:36418
> > > 2018-07-03T12:47:28,868 [myid:] - ERROR
> > > [FollowerRequestProcessor:2:ZooKeeperCriticalThread@48] - Severe
> > > unrecoverable error, from thread : FollowerRequestProcessor:2
> > > java.net.SocketException: Socket closed
> > > at
> > > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> > > ~[?:1.8.0_171]
> > > at
> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > > ~[?:1.8.0_171]
> > > at
> > > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > ~[?:1.8.0_171]
> > > at
> > > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > > ~[?:1.8.0_171]
> > > at
> > >
> org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > at
> > > org.apache.zookeeper.server.quorum.Learner.request(Learner.java:190)
> > > ~[zookeeper-3.5.0-alpha.jar:?]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:101)
> > > [zookeeper-3.5.0-alpha.jar:?]
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > > ]
> > > - shutting down
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FollowerRequestProcessor@138
> > > ]
> > > - Shutting down
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > >
> >
> [FollowerRequestProcessor:2:ZooKeeperServer$ZooKeeperServerListenerImpl@475
> > > ]
> > > - Thread FollowerRequestProcessor:2 exits, error code 1
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > > ]
> > > - Shutting down
> > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FinalRequestProcessor@492
> > > ]
> > > - shutdown of request processor complete
> > > 2018-07-03T12:47:28,869 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > > ]
> > > - Unregister MBean
> > >
> > >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
> > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > >
> > >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SyncRequestProcessor@191
> > > ]
> > > - Shutting down
> > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.248:34952 (no session established for client)
> > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T12:47:28,870 [myid:] - INFO
> > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.248:34014 (no session established for client)
> > > 2018-07-03T12:47:28,870 [myid:] - WARN
> > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T12:47:28,868 [myid:] - INFO
> > > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> > from
> > > /
> > > 10.0.0.237:36766
> > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.248:36150 (no session established for client)
> > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.248:33788 (no session established for client)
> > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> > > client /10.0.0.248:35930 (no session established for client)
> > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T12:47:28,871 [myid:] - WARN
> > > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > > session
> > > 0x0: ZooKeeperServer not running
> > > 2018-07-03T12:47:28,871 [myid:] - INFO
> > > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
> > >
> > > In particular I don't like the sound of this "Severe unrecoverable
> error,
> > > from thread : FollowerRequestProcessor" from "ZooKeeperCriticalThread".
> > >
> > > Is this a fatal event for the whole ZK server?
> > >
> > > Thanks
> > > Benjamin
> > >
> >
>
Re: Follower never recovers and keeps saying ZooKeeperServer not running
Posted by Benjamin Jaton <be...@gmail.com>.
Ah interestingly we did increase the znode size to ~ 40m. Think this can be
linked?
On Tue, Jul 3, 2018 at 10:55 PM Steph van Schalkwyk <sv...@gmail.com>
wrote:
> I've seen this bur can't remember what caused it.
> What's your zNode size?
> steph
>
> On Tue, Jul 3, 2018, 9:17 PM Benjamin Jaton <be...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I'm wondering what can cause a ZK follower to check out like this:
> >
> > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> 10.0.0.248:40282
> > :LearnerHandler@620] - Unexpected exception causing shutdown while sock
> > still open
> > java.io.EOFException: null
> > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > ~[?:1.8.0_171]
> > at
> > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > ~[zookeeper-3.5.0-alpha.jar:?]
> > at
> > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > at
> >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > [zookeeper-3.5.0-alpha.jar:?]
> > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> 10.0.0.248:40282
> > :LearnerHandler@635] - ******* GOODBYE /10.0.0.248:40282 ********
> > 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/
> 10.0.0.237:54408
> > :LearnerHandler@620] - Unexpected exception causing shutdown while sock
> > still open
> > java.io.EOFException: null
> > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > ~[?:1.8.0_171]
> > at
> > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > ~[zookeeper-3.5.0-alpha.jar:?]
> > at
> > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> > ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> > at
> >
> >
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> > [zookeeper-3.5.0-alpha.jar:?]
> > 2018-07-03T13:43:28,814 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:46754
> > 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/
> 10.0.0.237:54408
> > :LearnerHandler@635] - ******* GOODBYE /10.0.0.237:54408 ********
> > 2018-07-03T13:43:28,814 [myid:] - INFO
> > [ConnnectionExpirer:MBeanRegistry@128] - Unregister MBean
> >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.111,name5=0x200005f37660000]
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33198
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:46780
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33268
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.111:39162 which had sessionid 0x200005f37660000
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:46810
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:46824
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:46826
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@613
> > ]
> > - Shutting down
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33316
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33548
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@619
> > ]
> > - Shutdown called
> > java.lang.Exception: shutdown Leader! reason: Not sufficient followers
> > synced, only synced with sids: [ [2] ]
> > at
> > org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:619)
> > [zookeeper-3.5.0-alpha.jar:?]
> > at
> org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
> > [zookeeper-3.5.0-alpha.jar:?]
> > at
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
> > [zookeeper-3.5.0-alpha.jar:?]
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.248:33198 (no session
> > established for client)
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > ]
> > - Unregister MBean
> >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.237,name5=0x20000629130001b]
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> (message
> > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x0 (n.round),
> LOOKING
> > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > (n.config version)
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.237:45602 which had
> > sessionid
> > 0x20000629130001b
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > ]
> > - Unregister MBean
> >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a0008]
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> (message
> > format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> LOOKING
> > (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > (n.config version)
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.248:53764 which had
> > sessionid
> > 0x2000005cf2a0008
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.248:33268 (no session
> > established for client)
> > 2018-07-03T13:43:28,816 [myid:] - WARN
> > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > ]
> > - Unregister MBean
> >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a000a]
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.237:46826 (no session established for client)
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.248:53758 which had
> > sessionid
> > 0x2000005cf2a000a
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> > [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:2888:Leader$LearnerCnxAcceptor@373]
> -
> > exception while shutting down acceptor: java.net.SocketException: Socket
> > closed
> > 2018-07-03T13:43:28,816 [myid:] - WARN
> > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > ]
> > - Unregister MBean
> >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x30000209a080002]
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.237:46810 (no session established for client)
> > 2018-07-03T13:43:28,817 [myid:] - WARN
> > [NIOWorkerThread-6:NIOServerCnxn@369]
> > - Exception causing close of session 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.248:53760 which had
> > sessionid
> > 0x30000209a080002
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [NIOWorkerThread-6:NIOServerCnxn@606]
> > - Closed socket connection for client /10.0.0.237:46780 (no session
> > established for client)
> > 2018-07-03T13:43:28,817 [myid:] - WARN
> > [NIOWorkerThread-4:NIOServerCnxn@369]
> > - Exception causing close of session 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> [SessionTracker:ZooKeeperServer@384
> > ]
> > - Expiring session 0x300006dbd91000b, timeout of 15000ms exceeded
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > ]
> > - shutting down
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [NIOWorkerThread-4:NIOServerCnxn@606]
> > - Closed socket connection for client /10.0.0.248:33548 (no session
> > established for client)
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SessionTrackerImpl@232
> > ]
> > - Shutting down
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [SessionTracker:QuorumZooKeeperServer@132] - Submitting global
> > closeSession
> > request for session 0x300006dbd91000b
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LeaderRequestProcessor@77
> > ]
> > - Shutting down
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@967
> > ]
> > - Shutting down
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited
> loop!
> > 2018-07-03T13:43:28,817 [myid:] - INFO [ProcessThread(sid:2
> > cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
> > 2018-07-03T13:43:28,815 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:46908
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33786
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:46954
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33992
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47010
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34240
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47084
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47142
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47248
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34532
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47314
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47342
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47384
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34706
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47452
> > 2018-07-03T13:43:28,818 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47496
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47512
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34878
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47580
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47620
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35086
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47700
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47748
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47772
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47886
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47894
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47920
> > 2018-07-03T13:43:28,819 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:47998
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48018
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48020
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48136
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48138
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48140
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48172
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35898
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48294
> > 2018-07-03T13:43:28,820 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48296
> > 2018-07-03T13:43:28,817 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ProposalRequestProcessor@88
> > ]
> > - Shutting down
> > 2018-07-03T13:43:28,816 [myid:] - INFO
> > [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2
> (message
> > format version), 1 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round),
> LOOKING
> > (n.state), 1 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> > (n.config version)
> > 2018-07-03T13:43:28,817 [myid:] - WARN
> > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,821 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48298
> > 2018-07-03T13:43:28,821 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > ]
> > - Shutting down
> > 2018-07-03T13:43:28,821 [myid:] - INFO
> > [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.237:46754 (no session established for client)
> > 2018-07-03T13:43:28,821 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:48334
> > 2018-07-03T13:43:28,821 [myid:] - INFO
> > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
> > 2018-07-03T13:43:28,821 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:36162
> > 2018-07-03T13:43:28,821 [myid:] - WARN
> > [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,821 [myid:] - WARN
> > [NIOWorkerThread-16:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,821 [myid:] - WARN
> > [NIOWorkerThread-31:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T13:43:28,821 [myid:] - WARN
> > [NIOWorkerThread-11:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> >
> > The rest are more failed clients connections attempts and
> "ZooKeeperServer
> > not running" messages for hours.
> >
> >
> > There was an event before that that could be linked:
> >
> > 2018-07-03T12:42:27,639 [myid:] - INFO
> > [NIOWorkerThread-7:NIOServerCnxn@606]
> > - Closed socket connection for client /10.0.0.237:35536 (no session
> > established for client)
> > 2018-07-03T12:47:28,864 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33256
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:35568
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33544
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:35672
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:33788
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:35756
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34014
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:35840
> > 2018-07-03T12:47:28,865 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34244
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:35924
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34466
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:35994
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34730
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36086
> > 2018-07-03T12:47:28,866 [myid:] - WARN
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@93
> > ]
> > - Exception when following the leader
> > java.net.SocketException: Broken pipe (Write failed)
> > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > ~[?:1.8.0_171]
> > at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> > ~[?:1.8.0_171]
> > at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > ~[?:1.8.0_171]
> > at
> > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > ~[?:1.8.0_171]
> > at
> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > ~[?:1.8.0_171]
> > at
> > org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > ~[zookeeper-3.5.0-alpha.jar:?]
> > at
> > org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:609)
> > ~[zookeeper-3.5.0-alpha.jar:?]
> > at
> >
> >
> org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:116)
> > ~[zookeeper-3.5.0-alpha.jar:?]
> > at
> >
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:90)
> > [zookeeper-3.5.0-alpha.jar:?]
> > at
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
> > [zookeeper-3.5.0-alpha.jar:?]
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34944
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36166
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > ]
> > - Unregister MBean
> >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:34952
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36172
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198
> > ]
> > - shutdown called
> > java.lang.Exception: shutdown Follower
> > at
> > org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
> > [zookeeper-3.5.0-alpha.jar:?]
> > at
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)
> > [zookeeper-3.5.0-alpha.jar:?]
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35178
> > 2018-07-03T12:47:28,866 [myid:] - INFO
> > [NIOWorkerThread-14:ZooKeeperServer@969] - Client attempting to
> establish
> > new session at /10.0.0.248:33256
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36262
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35244
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36276
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35396
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36334
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35472
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36368
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35708
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36446
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:35930
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36530
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:36150
> > 2018-07-03T12:47:28,867 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.248:33544 (no session
> > established for client)
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36608
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> > ]
> > - Closed socket connection for client /10.0.0.248:33256 which had
> > sessionid
> > 0x200003aa13c0010
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36692
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LearnerZooKeeperServer@167
> > ]
> > - Shutting down
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.248:36418
> > 2018-07-03T12:47:28,868 [myid:] - ERROR
> > [FollowerRequestProcessor:2:ZooKeeperCriticalThread@48] - Severe
> > unrecoverable error, from thread : FollowerRequestProcessor:2
> > java.net.SocketException: Socket closed
> > at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> > ~[?:1.8.0_171]
> > at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > ~[?:1.8.0_171]
> > at
> > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > ~[?:1.8.0_171]
> > at
> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > ~[?:1.8.0_171]
> > at
> > org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> > ~[zookeeper-3.5.0-alpha.jar:?]
> > at
> > org.apache.zookeeper.server.quorum.Learner.request(Learner.java:190)
> > ~[zookeeper-3.5.0-alpha.jar:?]
> > at
> >
> >
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:101)
> > [zookeeper-3.5.0-alpha.jar:?]
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> > ]
> > - shutting down
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FollowerRequestProcessor@138
> > ]
> > - Shutting down
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> >
> [FollowerRequestProcessor:2:ZooKeeperServer$ZooKeeperServerListenerImpl@475
> > ]
> > - Thread FollowerRequestProcessor:2 exits, error code 1
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> > ]
> > - Shutting down
> > 2018-07-03T12:47:28,869 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FinalRequestProcessor@492
> > ]
> > - shutdown of request processor complete
> > 2018-07-03T12:47:28,869 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> > ]
> > - Unregister MBean
> >
> >
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
> > 2018-07-03T12:47:28,870 [myid:] - INFO
> >
> >
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SyncRequestProcessor@191
> > ]
> > - Shutting down
> > 2018-07-03T12:47:28,870 [myid:] - WARN
> > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T12:47:28,870 [myid:] - INFO
> > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.248:34952 (no session established for client)
> > 2018-07-03T12:47:28,870 [myid:] - WARN
> > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T12:47:28,870 [myid:] - INFO
> > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.248:34014 (no session established for client)
> > 2018-07-03T12:47:28,870 [myid:] - WARN
> > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T12:47:28,868 [myid:] - INFO
> > [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> > :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection
> from
> > /
> > 10.0.0.237:36766
> > 2018-07-03T12:47:28,871 [myid:] - INFO
> > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.248:36150 (no session established for client)
> > 2018-07-03T12:47:28,871 [myid:] - WARN
> > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T12:47:28,871 [myid:] - WARN
> > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T12:47:28,871 [myid:] - INFO
> > [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.248:33788 (no session established for client)
> > 2018-07-03T12:47:28,871 [myid:] - INFO
> > [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> > client /10.0.0.248:35930 (no session established for client)
> > 2018-07-03T12:47:28,871 [myid:] - WARN
> > [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T12:47:28,871 [myid:] - WARN
> > [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> > session
> > 0x0: ZooKeeperServer not running
> > 2018-07-03T12:47:28,871 [myid:] - INFO
> > [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
> >
> > In particular I don't like the sound of this "Severe unrecoverable error,
> > from thread : FollowerRequestProcessor" from "ZooKeeperCriticalThread".
> >
> > Is this a fatal event for the whole ZK server?
> >
> > Thanks
> > Benjamin
> >
>
Re: Follower never recovers and keeps saying ZooKeeperServer not running
Posted by Steph van Schalkwyk <sv...@gmail.com>.
I've seen this bur can't remember what caused it.
What's your zNode size?
steph
On Tue, Jul 3, 2018, 9:17 PM Benjamin Jaton <be...@gmail.com>
wrote:
> Hello,
>
> I'm wondering what can cause a ZK follower to check out like this:
>
> 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/10.0.0.248:40282
> :LearnerHandler@620] - Unexpected exception causing shutdown while sock
> still open
> java.io.EOFException: null
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> ~[?:1.8.0_171]
> at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> at
>
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> ~[zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> at
>
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> [zookeeper-3.5.0-alpha.jar:?]
> 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/10.0.0.248:40282
> :LearnerHandler@635] - ******* GOODBYE /10.0.0.248:40282 ********
> 2018-07-03T13:43:28,814 [myid:] - ERROR [LearnerHandler-/10.0.0.237:54408
> :LearnerHandler@620] - Unexpected exception causing shutdown while sock
> still open
> java.io.EOFException: null
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> ~[?:1.8.0_171]
> at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> at
>
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> ~[zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
> ~[zookeeper-3.5.0-alpha.jar:3.5.2-alpha-SNAPSHOT-1702518]
> at
>
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:514)
> [zookeeper-3.5.0-alpha.jar:?]
> 2018-07-03T13:43:28,814 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:46754
> 2018-07-03T13:43:28,814 [myid:] - WARN [LearnerHandler-/10.0.0.237:54408
> :LearnerHandler@635] - ******* GOODBYE /10.0.0.237:54408 ********
> 2018-07-03T13:43:28,814 [myid:] - INFO
> [ConnnectionExpirer:MBeanRegistry@128] - Unregister MBean
>
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.111,name5=0x200005f37660000]
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33198
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:46780
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33268
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [ConnnectionExpirer:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.111:39162 which had sessionid 0x200005f37660000
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:46810
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:46824
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:46826
> 2018-07-03T13:43:28,815 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@613
> ]
> - Shutting down
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33316
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33548
> 2018-07-03T13:43:28,815 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@619
> ]
> - Shutdown called
> java.lang.Exception: shutdown Leader! reason: Not sufficient followers
> synced, only synced with sids: [ [2] ]
> at
> org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:619)
> [zookeeper-3.5.0-alpha.jar:?]
> at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
> [zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
> [zookeeper-3.5.0-alpha.jar:?]
> 2018-07-03T13:43:28,815 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.248:33198 (no session
> established for client)
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> ]
> - Unregister MBean
>
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.237,name5=0x20000629130001b]
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message
> format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x0 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> (n.config version)
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.237:45602 which had
> sessionid
> 0x20000629130001b
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> ]
> - Unregister MBean
>
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a0008]
> 2018-07-03T13:43:28,816 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message
> format version), 3 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round), LOOKING
> (n.state), 3 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> (n.config version)
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.248:53764 which had
> sessionid
> 0x2000005cf2a0008
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.248:33268 (no session
> established for client)
> 2018-07-03T13:43:28,816 [myid:] - WARN
> [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> ]
> - Unregister MBean
>
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x2000005cf2a000a]
> 2018-07-03T13:43:28,816 [myid:] - INFO
> [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.237:46826 (no session established for client)
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.248:53758 which had
> sessionid
> 0x2000005cf2a000a
> 2018-07-03T13:43:28,816 [myid:] - INFO
> [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:2888:Leader$LearnerCnxAcceptor@373] -
> exception while shutting down acceptor: java.net.SocketException: Socket
> closed
> 2018-07-03T13:43:28,816 [myid:] - WARN
> [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,816 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> ]
> - Unregister MBean
>
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader,name3=Connections,name4=10.0.0.248,name5=0x30000209a080002]
> 2018-07-03T13:43:28,816 [myid:] - INFO
> [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.237:46810 (no session established for client)
> 2018-07-03T13:43:28,817 [myid:] - WARN
> [NIOWorkerThread-6:NIOServerCnxn@369]
> - Exception causing close of session 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,817 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.248:53760 which had
> sessionid
> 0x30000209a080002
> 2018-07-03T13:43:28,817 [myid:] - INFO
> [NIOWorkerThread-6:NIOServerCnxn@606]
> - Closed socket connection for client /10.0.0.237:46780 (no session
> established for client)
> 2018-07-03T13:43:28,817 [myid:] - WARN
> [NIOWorkerThread-4:NIOServerCnxn@369]
> - Exception causing close of session 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,817 [myid:] - INFO [SessionTracker:ZooKeeperServer@384
> ]
> - Expiring session 0x300006dbd91000b, timeout of 15000ms exceeded
> 2018-07-03T13:43:28,817 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> ]
> - shutting down
> 2018-07-03T13:43:28,817 [myid:] - INFO
> [NIOWorkerThread-4:NIOServerCnxn@606]
> - Closed socket connection for client /10.0.0.248:33548 (no session
> established for client)
> 2018-07-03T13:43:28,817 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SessionTrackerImpl@232
> ]
> - Shutting down
> 2018-07-03T13:43:28,817 [myid:] - INFO
> [SessionTracker:QuorumZooKeeperServer@132] - Submitting global
> closeSession
> request for session 0x300006dbd91000b
> 2018-07-03T13:43:28,817 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LeaderRequestProcessor@77
> ]
> - Shutting down
> 2018-07-03T13:43:28,817 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@967
> ]
> - Shutting down
> 2018-07-03T13:43:28,817 [myid:] - INFO
> [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
> 2018-07-03T13:43:28,817 [myid:] - INFO [ProcessThread(sid:2
> cport:-1)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
> 2018-07-03T13:43:28,815 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:46908
> 2018-07-03T13:43:28,817 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33786
> 2018-07-03T13:43:28,817 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:46954
> 2018-07-03T13:43:28,817 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33992
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47010
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34240
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47084
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47142
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47248
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34532
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47314
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47342
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47384
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34706
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47452
> 2018-07-03T13:43:28,818 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47496
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47512
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34878
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47580
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47620
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35086
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47700
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47748
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47772
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47886
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47894
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47920
> 2018-07-03T13:43:28,819 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:47998
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48018
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48020
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48136
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48138
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48140
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48172
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35898
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48294
> 2018-07-03T13:43:28,820 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48296
> 2018-07-03T13:43:28,817 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ProposalRequestProcessor@88
> ]
> - Shutting down
> 2018-07-03T13:43:28,816 [myid:] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@688] - Notification: 2 (message
> format version), 1 (n.leader), 0x9000001cb (n.zxid), 0x7 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x9 (n.peerEPoch), LEADING (my state)300000066
> (n.config version)
> 2018-07-03T13:43:28,817 [myid:] - WARN
> [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,821 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48298
> 2018-07-03T13:43:28,821 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> ]
> - Shutting down
> 2018-07-03T13:43:28,821 [myid:] - INFO
> [NIOWorkerThread-20:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.237:46754 (no session established for client)
> 2018-07-03T13:43:28,821 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:48334
> 2018-07-03T13:43:28,821 [myid:] - INFO
> [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
> 2018-07-03T13:43:28,821 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:36162
> 2018-07-03T13:43:28,821 [myid:] - WARN
> [NIOWorkerThread-20:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,821 [myid:] - WARN
> [NIOWorkerThread-16:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,821 [myid:] - WARN
> [NIOWorkerThread-31:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T13:43:28,821 [myid:] - WARN
> [NIOWorkerThread-11:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
>
> The rest are more failed clients connections attempts and "ZooKeeperServer
> not running" messages for hours.
>
>
> There was an event before that that could be linked:
>
> 2018-07-03T12:42:27,639 [myid:] - INFO
> [NIOWorkerThread-7:NIOServerCnxn@606]
> - Closed socket connection for client /10.0.0.237:35536 (no session
> established for client)
> 2018-07-03T12:47:28,864 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33256
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:35568
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33544
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:35672
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:33788
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:35756
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34014
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:35840
> 2018-07-03T12:47:28,865 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34244
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:35924
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34466
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:35994
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34730
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36086
> 2018-07-03T12:47:28,866 [myid:] - WARN
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@93
> ]
> - Exception when following the leader
> java.net.SocketException: Broken pipe (Write failed)
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> ~[?:1.8.0_171]
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> ~[?:1.8.0_171]
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> ~[?:1.8.0_171]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[?:1.8.0_171]
> at
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[?:1.8.0_171]
> at
> org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> ~[zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:609)
> ~[zookeeper-3.5.0-alpha.jar:?]
> at
>
> org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:116)
> ~[zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:90)
> [zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
> [zookeeper-3.5.0-alpha.jar:?]
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34944
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36166
> 2018-07-03T12:47:28,866 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> ]
> - Unregister MBean
>
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower]
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:34952
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36172
> 2018-07-03T12:47:28,866 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198
> ]
> - shutdown called
> java.lang.Exception: shutdown Follower
> at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
> [zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)
> [zookeeper-3.5.0-alpha.jar:?]
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35178
> 2018-07-03T12:47:28,866 [myid:] - INFO
> [NIOWorkerThread-14:ZooKeeperServer@969] - Client attempting to establish
> new session at /10.0.0.248:33256
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36262
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35244
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36276
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35396
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36334
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35472
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36368
> 2018-07-03T12:47:28,867 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35708
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36446
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:35930
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36530
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:36150
> 2018-07-03T12:47:28,867 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.248:33544 (no session
> established for client)
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36608
> 2018-07-03T12:47:28,868 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):NIOServerCnxn@606
> ]
> - Closed socket connection for client /10.0.0.248:33256 which had
> sessionid
> 0x200003aa13c0010
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36692
> 2018-07-03T12:47:28,868 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):LearnerZooKeeperServer@167
> ]
> - Shutting down
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.248:36418
> 2018-07-03T12:47:28,868 [myid:] - ERROR
> [FollowerRequestProcessor:2:ZooKeeperCriticalThread@48] - Severe
> unrecoverable error, from thread : FollowerRequestProcessor:2
> java.net.SocketException: Socket closed
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
> ~[?:1.8.0_171]
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> ~[?:1.8.0_171]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[?:1.8.0_171]
> at
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[?:1.8.0_171]
> at
> org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:141)
> ~[zookeeper-3.5.0-alpha.jar:?]
> at
> org.apache.zookeeper.server.quorum.Learner.request(Learner.java:190)
> ~[zookeeper-3.5.0-alpha.jar:?]
> at
>
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:101)
> [zookeeper-3.5.0-alpha.jar:?]
> 2018-07-03T12:47:28,868 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@498
> ]
> - shutting down
> 2018-07-03T12:47:28,868 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FollowerRequestProcessor@138
> ]
> - Shutting down
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [FollowerRequestProcessor:2:ZooKeeperServer$ZooKeeperServerListenerImpl@475
> ]
> - Thread FollowerRequestProcessor:2 exits, error code 1
> 2018-07-03T12:47:28,868 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@358
> ]
> - Shutting down
> 2018-07-03T12:47:28,869 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FinalRequestProcessor@492
> ]
> - shutdown of request processor complete
> 2018-07-03T12:47:28,869 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128
> ]
> - Unregister MBean
>
> [org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Follower,name3=InMemoryDataTree]
> 2018-07-03T12:47:28,870 [myid:] - INFO
>
> [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):SyncRequestProcessor@191
> ]
> - Shutting down
> 2018-07-03T12:47:28,870 [myid:] - WARN
> [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T12:47:28,870 [myid:] - INFO
> [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.248:34952 (no session established for client)
> 2018-07-03T12:47:28,870 [myid:] - WARN
> [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T12:47:28,870 [myid:] - INFO
> [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.248:34014 (no session established for client)
> 2018-07-03T12:47:28,870 [myid:] - WARN
> [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T12:47:28,868 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181
> :NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from
> /
> 10.0.0.237:36766
> 2018-07-03T12:47:28,871 [myid:] - INFO
> [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.248:36150 (no session established for client)
> 2018-07-03T12:47:28,871 [myid:] - WARN
> [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T12:47:28,871 [myid:] - WARN
> [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T12:47:28,871 [myid:] - INFO
> [NIOWorkerThread-32:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.248:33788 (no session established for client)
> 2018-07-03T12:47:28,871 [myid:] - INFO
> [NIOWorkerThread-17:NIOServerCnxn@606] - Closed socket connection for
> client /10.0.0.248:35930 (no session established for client)
> 2018-07-03T12:47:28,871 [myid:] - WARN
> [NIOWorkerThread-32:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T12:47:28,871 [myid:] - WARN
> [NIOWorkerThread-17:NIOServerCnxn@369] - Exception causing close of
> session
> 0x0: ZooKeeperServer not running
> 2018-07-03T12:47:28,871 [myid:] - INFO
> [CommitProcessor:2:CommitProcessor@193] - CommitProcessor exited loop!
>
> In particular I don't like the sound of this "Severe unrecoverable error,
> from thread : FollowerRequestProcessor" from "ZooKeeperCriticalThread".
>
> Is this a fatal event for the whole ZK server?
>
> Thanks
> Benjamin
>