You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Filip Deleersnijder <fi...@motum.be> on 2015/06/24 15:05:26 UTC
Leader election problems
Hi,
Let’s start with some description of our system :
- We our using a Zookeeper cluster with 8 participants for an application with mobile nodes ( connected over Wifi ).
( Ip of the different nodes are according to the following structure : Node X has IP : 172.17.35.1X )
- It is not that unusual to have a node being shut-down or restarted
- We haven’t benchmarked the number of write operations yet, but I would estimate that it would be less than 10 writes / second
The problem we are having however is that sometimes(*), some instances seem to be having problems with leader election.
Under the header “Attachment 1” below, you can find the leader election times that were needed over 24h ( from 1 node ). One average it took more than 1 minute !
I assume that this is not normal behaviour ? ( If somebody could confirm that in a 8-node cluster, these are not normal leader election times, that would be nice )
In attachement 2 : I included an extract from the logging during a leader election that took 101874ms for 1 node ( server 2 ).
Any help is greatly appreciated.
If further or more specific logging is required, please ask !
Thanks in advance,
Filip
(*) I do realise that ’sometimes’ is a bad description for a bug, but we haven’t found any logic behind when it does or does not happen.
Attachment 1 : Leader Election Times :
2015-06-22 12:02:06,259 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 5271
2015-06-22 12:06:44,196 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 16372
2015-06-22 12:11:14,479 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 46290
2015-06-22 12:28:08,206 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 105280
2015-06-22 13:46:21,276 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 19581
2015-06-22 13:48:20,213 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 64922
2015-06-22 14:17:35,252 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 43082
2015-06-22 16:09:31,170 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 75684
2015-06-22 16:10:41,035 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 60767
2015-06-22 16:12:20,739 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 90639
2015-06-22 16:14:01,358 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 91528
2015-06-22 16:15:31,268 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 80838
2015-06-22 16:17:11,315 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 90844
2015-06-22 16:18:30,782 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 49429
2015-06-22 16:20:31,567 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 111656
2015-06-22 16:22:21,644 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 101009
2015-06-22 16:37:05,703 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 18015
2015-06-22 17:08:44,839 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 6742
2015-06-22 17:16:46,716 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 34605
2015-06-22 18:04:05,160 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 35413
2015-06-22 18:05:17,887 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 63633
2015-06-22 18:35:24,398 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 34765
2015-06-22 18:36:28,311 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 20867
2015-06-23 05:01:34,198 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 20475
2015-06-23 07:42:14,764 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 42494
2015-06-23 07:43:31,374 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 32582
2015-06-23 08:23:06,522 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 5079
2015-06-23 09:37:31,120 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 65102
2015-06-23 09:40:57,895 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 117194
2015-06-23 09:42:31,903 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 39996
2015-06-23 10:00:31,130 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 97710
2015-06-23 10:35:02,798 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 30920
2015-06-23 11:56:54,484 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 19387
2015-06-23 12:08:52,736 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 388674
2015-06-23 13:03:30,263 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 128415
2015-06-23 13:04:11,071 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 30345
2015-06-23 14:58:09,944 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 101874
Attachement 2 : Leader election logging
2015-06-23 14:56:28,055 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@89][] - Exception when following the leader
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)
2015-06-23 14:56:28,061 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@166][] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790)
2015-06-23 14:56:28,064 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55322 which had sessionid 0x24e201711a20000
2015-06-23 14:56:28,067 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FollowerZooKeeperServer@139][] - Shutting down
2015-06-23 14:56:28,067 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@441][] - shutting down
2015-06-23 14:56:28,067 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FollowerRequestProcessor@105][] - Shutting down
2015-06-23 14:56:28,067 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor@181][] - Shutting down
2015-06-23 14:56:28,068 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@415][] - shutdown of request processor complete
2015-06-23 14:56:28,068 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:SyncRequestProcessor@209][] - Shutting down
2015-06-23 14:56:28,068 [myid:2] - INFO [CommitProcessor:2:CommitProcessor@150][] - CommitProcessor exited loop!
2015-06-23 14:56:28,068 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x0 type:notification cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:56:28,069 [myid:2] - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@95][] - FollowerRequestProcessor exited loop!
2015-06-23 14:56:28,068 [myid:2] - INFO [SyncThread:2:SyncRequestProcessor@187][] - SyncRequestProcessor exited!
2015-06-23 14:56:28,069 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumPeer@714][] - LOOKING
2015-06-23 14:56:28,069 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumPeer@645][] - Initializing leader election protocol...
2015-06-23 14:56:28,071 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FileSnap@83][] - Reading snapshot c:\motum\config\MASS\ZK\version-2\snapshot.1a20000000a
2015-06-23 14:56:28,071 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - loadData --- session in archive: 94047701500755968 with timeout: 4000
2015-06-23 14:56:28,072 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - loadData --- session in archive: 238163113551855616 with timeout: 4000
2015-06-23 14:56:28,072 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - loadData --- session in archive: 598449186428682241 with timeout: 4000
2015-06-23 14:56:28,072 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - loadData --- session in archive: 382278301628432384 with timeout: 4000
2015-06-23 14:56:28,072 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - loadData --- session in archive: 526393489704222720 with timeout: 4000
2015-06-23 14:56:28,074 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FileTxnLog$FileTxnIterator@593][] - Created new input stream c:\motum\config\MASS\ZK\version-2\log.19a00000001
2015-06-23 14:56:28,074 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FileTxnLog$FileTxnIterator@596][] - Created new input archive c:\motum\config\MASS\ZK\version-2\log.19a00000001
2015-06-23 14:56:28,092 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- create session in log: 0x24e201711a20000 with timeout: 4000
2015-06-23 14:56:28,094 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- create session in log: 0x44e201711af0000 with timeout: 4000
2015-06-23 14:56:28,199 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- close session in log: 0x14e1fe2dfe50000
2015-06-23 14:56:28,199 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- create session in log: 0x14e201705e10000 with timeout: 4000
2015-06-23 14:56:28,234 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- close session in log: 0x44e201711af0000
2015-06-23 14:56:28,234 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- create session in log: 0x44e201711af0001 with timeout: 4000
2015-06-23 14:56:28,234 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- close session in log: 0x54e201705e70000
2015-06-23 14:56:28,234 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - playLog --- close session in log: 0x74e201705e60000
2015-06-23 14:56:28,234 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FileTxnLog$FileTxnIterator@636][] - EOF excepton java.io.EOFException: Failed to read c:\motum\config\MASS\ZK\version-2\log.19a00000001
2015-06-23 14:56:28,235 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@716][] - Updating proposal: 2 (newleader), 0x1a200000623 (newzxid), 8 (oldleader), 0x1a1000002a2 (oldzxid)
2015-06-23 14:56:28,235 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@815][] - New election. My id = 2, proposed zxid=0x1a200000623
2015-06-23 14:56:28,235 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,235 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,236 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,236 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,236 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,236 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,236 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,236 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,236 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:56:28,236 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:56:28,237 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:56:28,237 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:56:28,237 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:56:28,237 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:28,238 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:28,238 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 2, proposed id: 2, zxid: 0x1a200000623, proposed zxid: 0x1a200000623
2015-06-23 14:56:28,238 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=2, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@412][] - Queue size: 0
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,439 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,440 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849][] - Notification time out: 400
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@412][] - Queue size: 0
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,841 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:28,844 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849][] - Notification time out: 800
2015-06-23 14:56:29,008 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55373
2015-06-23 14:56:29,009 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:29,009 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:29,009 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55373 (no session established for client)
2015-06-23 14:56:29,644 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@412][] - Queue size: 0
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,645 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:29,646 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849][] - Notification time out: 1600
2015-06-23 14:56:29,729 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55386
2015-06-23 14:56:29,729 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:29,729 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:29,730 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55386 (no session established for client)
2015-06-23 14:56:31,248 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@412][] - Queue size: 0
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:31,249 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849][] - Notification time out: 3200
2015-06-23 14:56:31,328 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55413
2015-06-23 14:56:31,331 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:31,331 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:31,331 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55413 (no session established for client)
2015-06-23 14:56:31,910 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55424
2015-06-23 14:56:31,911 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:31,911 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:31,912 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55424 (no session established for client)
2015-06-23 14:56:33,238 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:56:33,239 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:56:33,239 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:56:33,239 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:56:33,240 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:56:33,240 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:56:33,240 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:56:33,240 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:56:33,241 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:33,241 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:33,248 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 2, proposed id: 2, zxid: 0x1a200000623, proposed zxid: 0x1a200000623
2015-06-23 14:56:33,248 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=2, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:33,695 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55455
2015-06-23 14:56:33,695 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:33,695 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:33,696 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55455 (no session established for client)
2015-06-23 14:56:33,849 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55460
2015-06-23 14:56:33,849 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:33,850 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:33,850 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55460 (no session established for client)
2015-06-23 14:56:35,077 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55481
2015-06-23 14:56:35,077 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:35,077 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:35,078 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55481 (no session established for client)
2015-06-23 14:56:35,919 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55495
2015-06-23 14:56:35,919 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:35,919 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:35,920 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55495 (no session established for client)
2015-06-23 14:56:36,455 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@412][] - Queue size: 0
2015-06-23 14:56:36,455 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,455 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,455 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,455 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,456 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,456 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,456 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,456 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:36,456 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849][] - Notification time out: 6400
2015-06-23 14:56:37,995 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55530
2015-06-23 14:56:37,995 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:37,995 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:37,996 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55530 (no session established for client)
2015-06-23 14:56:38,242 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:56:38,246 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:56:38,246 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:56:38,246 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:56:38,246 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:56:38,246 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:56:38,246 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:56:38,247 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:56:38,247 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:38,248 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:38,248 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 2, proposed id: 2, zxid: 0x1a200000623, proposed zxid: 0x1a200000623
2015-06-23 14:56:38,248 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=2, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:38,947 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55549
2015-06-23 14:56:38,947 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:38,948 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:38,948 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55549 (no session established for client)
2015-06-23 14:56:38,970 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:38,970 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:38,971 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 5, proposed id: 2, zxid: 0x1a200000623, proposed zxid: 0x1a200000623
2015-06-23 14:56:38,971 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@716][] - Updating proposal: 5 (newleader), 0x1a200000623 (newzxid), 2 (oldleader), 0x1a200000623 (oldzxid)
2015-06-23 14:56:38,971 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,971 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,971 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,971 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,971 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,972 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,972 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,972 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:38,972 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=5, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:40,092 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55569
2015-06-23 14:56:40,092 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:40,092 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:40,093 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55569 (no session established for client)
2015-06-23 14:56:40,327 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55572
2015-06-23 14:56:40,327 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:40,327 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:40,328 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55572 (no session established for client)
2015-06-23 14:56:41,483 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55593
2015-06-23 14:56:41,483 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:41,483 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:41,483 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55593 (no session established for client)
2015-06-23 14:56:41,748 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55599
2015-06-23 14:56:41,748 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:41,748 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:41,749 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55599 (no session established for client)
2015-06-23 14:56:43,247 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:56:43,247 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:56:43,247 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:56:43,248 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:56:43,248 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:56:43,248 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:56:43,248 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:56:43,248 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:56:43,249 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:43,249 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:43,249 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 2, proposed id: 5, zxid: 0x1a200000623, proposed zxid: 0x1a200000623
2015-06-23 14:56:43,250 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=2, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:43,473 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55629
2015-06-23 14:56:43,473 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:43,473 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:43,474 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55629 (no session established for client)
2015-06-23 14:56:43,908 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55638
2015-06-23 14:56:43,909 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:43,909 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:43,909 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55638 (no session established for client)
2015-06-23 14:56:44,264 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:44,265 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:44,265 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:44,265 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:44,265 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 5, zxid: 0x1a200000627, proposed zxid: 0x1a200000623
2015-06-23 14:56:44,265 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@716][] - Updating proposal: 3 (newleader), 0x1a200000627 (newzxid), 5 (oldleader), 0x1a200000623 (oldzxid)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:56:44,266 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:56:44,267 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:56:44,267 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:56:45,250 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55661
2015-06-23 14:56:45,250 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:45,250 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:45,251 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55661 (no session established for client)
2015-06-23 14:56:45,628 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55668
2015-06-23 14:56:45,628 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:45,628 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:45,629 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55668 (no session established for client)
2015-06-23 14:56:46,848 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55689
2015-06-23 14:56:46,848 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:46,848 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:46,848 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55689 (no session established for client)
2015-06-23 14:56:47,449 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55699
2015-06-23 14:56:47,450 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:47,450 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:47,450 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55699 (no session established for client)
2015-06-23 14:56:48,250 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:56:48,250 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:56:48,251 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:56:48,251 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:56:48,251 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:56:48,251 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:56:48,251 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:56:48,251 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:56:48,252 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:48,252 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:48,253 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 2, proposed id: 3, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:56:48,253 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=2, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:49,200 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55730
2015-06-23 14:56:49,200 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:49,200 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:49,201 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55730 (no session established for client)
2015-06-23 14:56:50,140 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55746
2015-06-23 14:56:50,140 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:50,140 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:50,141 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55746 (no session established for client)
2015-06-23 14:56:51,456 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55769
2015-06-23 14:56:51,456 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:51,456 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:51,457 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55769 (no session established for client)
2015-06-23 14:56:51,583 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55773
2015-06-23 14:56:51,583 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:51,583 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:51,583 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55773 (no session established for client)
2015-06-23 14:56:52,818 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55792
2015-06-23 14:56:52,818 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:52,818 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:52,818 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55792 (no session established for client)
2015-06-23 14:56:53,255 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:56:53,256 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:56:53,256 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:56:53,256 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:56:53,256 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:56:53,256 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:56:53,256 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:56:53,257 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:56:53,257 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:53,258 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 2 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:53,258 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 2, proposed id: 3, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:56:53,258 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=2, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:53,677 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55808
2015-06-23 14:56:53,677 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:53,677 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:53,677 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55808 (no session established for client)
2015-06-23 14:56:55,463 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55838
2015-06-23 14:56:55,463 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:55,464 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:55,464 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55838 (no session established for client)
2015-06-23 14:56:55,643 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55842
2015-06-23 14:56:55,643 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:55,643 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:55,644 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55842 (no session established for client)
2015-06-23 14:56:57,149 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55870
2015-06-23 14:56:57,149 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:57,150 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:57,150 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55870 (no session established for client)
2015-06-23 14:56:58,243 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55888
2015-06-23 14:56:58,243 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:58,243 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:58,244 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55888 (no session established for client)
2015-06-23 14:56:58,258 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:56:58,259 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:56:58,259 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:56:58,259 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:56:58,259 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:56:58,259 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:56:58,259 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:56:58,259 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:56:58,260 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:56:58,260 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:56:58,261 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 5, proposed id: 3, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:56:58,261 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=5, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:56:59,496 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55911
2015-06-23 14:56:59,496 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:56:59,496 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:56:59,496 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55911 (no session established for client)
2015-06-23 14:57:00,394 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55928
2015-06-23 14:57:00,394 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:00,394 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:00,395 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55928 (no session established for client)
2015-06-23 14:57:01,580 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55949
2015-06-23 14:57:01,581 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:01,581 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:01,581 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55949 (no session established for client)
2015-06-23 14:57:02,027 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55956
2015-06-23 14:57:02,027 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:02,027 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:02,028 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55956 (no session established for client)
2015-06-23 14:57:03,260 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:03,260 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:03,261 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:03,261 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:57:03,261 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:57:03,261 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:57:03,261 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:57:03,261 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:57:03,262 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:03,262 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:03,263 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:03,263 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:03,299 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:55979
2015-06-23 14:57:03,299 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:03,299 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:03,300 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:55979 (no session established for client)
2015-06-23 14:57:04,314 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:55997
2015-06-23 14:57:04,316 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:04,316 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:04,317 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:55997 (no session established for client)
2015-06-23 14:57:05,481 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56017
2015-06-23 14:57:05,482 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:05,482 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:05,482 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56017 (no session established for client)
2015-06-23 14:57:06,281 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56031
2015-06-23 14:57:06,281 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:06,283 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:06,283 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56031 (no session established for client)
2015-06-23 14:57:08,263 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:08,263 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:08,264 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:08,377 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56066
2015-06-23 14:57:08,377 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:08,377 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:08,377 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56066 (no session established for client)
2015-06-23 14:57:09,068 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56080
2015-06-23 14:57:09,068 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:09,068 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:09,069 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56080 (no session established for client)
2015-06-23 14:57:09,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@412][] - Queue size: 0
2015-06-23 14:57:09,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,665 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:57:09,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,665 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:09,665 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:09,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:09,666 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849][] - Notification time out: 12800
2015-06-23 14:57:09,667 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:09,667 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:09,666 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:57:09,667 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:57:09,667 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:57:09,667 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:57:10,710 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56109
2015-06-23 14:57:10,711 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:10,712 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:10,712 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56109 (no session established for client)
2015-06-23 14:57:10,878 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56113
2015-06-23 14:57:10,878 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:10,878 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:10,879 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56113 (no session established for client)
2015-06-23 14:57:12,815 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56145
2015-06-23 14:57:12,815 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:12,815 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:12,816 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56145 (no session established for client)
2015-06-23 14:57:13,044 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,044 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,051 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,052 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,056 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,062 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,064 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,064 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,064 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,064 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,053 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,062 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56151
2015-06-23 14:57:13,065 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:13,065 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:13,065 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56151 (no session established for client)
2015-06-23 14:57:13,064 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,070 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,070 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,070 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@716][] - Updating proposal: 8 (newleader), 0x1a200000627 (newzxid), 3 (oldleader), 0x1a200000627 (oldzxid)
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,071 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,072 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:13,073 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,073 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,073 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,073 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,073 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,682 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,682 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 5 (n.leader), 0x1a200000623 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,683 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,683 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,683 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,683 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,683 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,684 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,684 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,684 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,684 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,684 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,684 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:13,684 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:13,683 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 5, proposed id: 8, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=5, proposed zxid=0x1a200000623, proposed election epoch=0x11
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,685 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,686 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,686 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,686 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:13,686 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:13,686 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:14,434 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56174
2015-06-23 14:57:14,434 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:14,434 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:14,435 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56174 (no session established for client)
2015-06-23 14:57:14,668 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:14,668 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:14,668 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:14,669 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:57:14,669 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:57:14,669 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:57:14,669 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:57:14,669 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:57:14,670 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:14,670 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:14,670 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:14,670 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:14,705 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56181
2015-06-23 14:57:14,705 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:14,705 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:14,705 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56181 (no session established for client)
2015-06-23 14:57:16,511 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56210
2015-06-23 14:57:16,511 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:16,511 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:16,511 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56210 (no session established for client)
2015-06-23 14:57:17,538 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56227
2015-06-23 14:57:17,539 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:17,539 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:17,539 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56227 (no session established for client)
2015-06-23 14:57:19,290 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56259
2015-06-23 14:57:19,290 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:19,290 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:19,291 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56259 (no session established for client)
2015-06-23 14:57:19,671 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:19,671 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:19,671 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:19,874 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56270
2015-06-23 14:57:19,874 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:19,875 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:19,875 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56270 (no session established for client)
2015-06-23 14:57:19,907 [myid:2] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@780][] - Connection broken for id 4, my id = 2, error =
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:765)
2015-06-23 14:57:19,907 [myid:2] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker@783][] - Interrupting SendWorker
2015-06-23 14:57:19,907 [myid:2] - DEBUG [RecvWorker:4:QuorumCnxManager$SendWorker@606][] - Calling finish for 4
2015-06-23 14:57:19,907 [myid:2] - DEBUG [RecvWorker:4:QuorumCnxManager$SendWorker@626][] - Removing entry from senderWorkerMap sid=4
2015-06-23 14:57:19,908 [myid:2] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@697][] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:849)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:64)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:685)
2015-06-23 14:57:19,908 [myid:2] - DEBUG [SendWorker:4:QuorumCnxManager$SendWorker@606][] - Calling finish for 4
2015-06-23 14:57:19,908 [myid:2] - WARN [SendWorker:4:QuorumCnxManager$SendWorker@706][] - Send worker leaving thread
2015-06-23 14:57:21,465 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56299
2015-06-23 14:57:21,465 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:21,465 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:21,466 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56299 (no session established for client)
2015-06-23 14:57:22,099 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56311
2015-06-23 14:57:22,099 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:22,099 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:22,100 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56311 (no session established for client)
2015-06-23 14:57:23,473 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56334
2015-06-23 14:57:23,473 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:23,473 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:23,474 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56334 (no session established for client)
2015-06-23 14:57:24,155 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56348
2015-06-23 14:57:24,155 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:24,155 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:24,156 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56348 (no session established for client)
2015-06-23 14:57:25,613 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56372
2015-06-23 14:57:25,613 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:25,613 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:25,614 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56372 (no session established for client)
2015-06-23 14:57:26,069 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56380
2015-06-23 14:57:26,070 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:26,070 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:26,071 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56380 (no session established for client)
2015-06-23 14:57:27,473 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@412][] - Queue size: 0
2015-06-23 14:57:27,473 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,473 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:57:27,473 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,473 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:27,474 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:27,474 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,474 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,474 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,474 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,474 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,475 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:27,475 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849][] - Notification time out: 25600
2015-06-23 14:57:27,475 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:27,475 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:27,475 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:57:27,475 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 4
2015-06-23 14:57:27,751 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56409
2015-06-23 14:57:27,751 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:27,751 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:27,752 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56409 (no session established for client)
2015-06-23 14:57:28,741 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56425
2015-06-23 14:57:28,741 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:28,741 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:28,742 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56425 (no session established for client)
2015-06-23 14:57:30,675 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56459
2015-06-23 14:57:30,675 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:30,676 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:30,676 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56459 (no session established for client)
2015-06-23 14:57:30,951 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:30,952 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:30,952 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:30,952 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:30,952 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:30,952 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:30,952 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:30,953 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:31,339 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56471
2015-06-23 14:57:31,339 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:31,339 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:31,339 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56471 (no session established for client)
2015-06-23 14:57:32,481 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 4 at election address /172.17.35.14:3888
2015-06-23 14:57:32,481 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:57:32,482 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:57:32,687 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56496
2015-06-23 14:57:32,687 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:32,687 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:32,687 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56496 (no session established for client)
2015-06-23 14:57:32,940 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,940 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 1 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,940 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,940 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,941 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,941 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,941 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,941 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,941 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,941 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,942 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,942 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,943 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,943 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,943 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,943 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,943 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,943 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,944 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,944 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,944 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,944 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,944 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,944 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,940 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 1, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=1, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:32,945 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:32,946 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:32,946 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:32,946 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:32,946 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x11
2015-06-23 14:57:32,972 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,972 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,976 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 1, proposed id: 2, zxid: 0x1a200000627, proposed zxid: 0x1a200000623
2015-06-23 14:57:32,978 [myid:2] - INFO [/172.17.35.12:3888:QuorumCnxManager$Listener@511][] - Received connection request /172.17.35.14:55375
2015-06-23 14:57:32,981 [myid:2] - DEBUG [/172.17.35.12:3888:QuorumCnxManager$SendWorker@588][] - Address of remote peer: 4
2015-06-23 14:57:32,980 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@716][] - Updating proposal: 1 (newleader), 0x1a200000627 (newzxid), 8 (oldleader), 0x1a200000627 (oldzxid)
2015-06-23 14:57:32,982 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,983 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,983 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,983 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,984 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,984 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,984 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,984 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:32,992 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=1, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:32,998 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:32,998 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), LOOKING (n.state), 4 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:32,998 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@873][] - Notification election epoch is smaller than logicalclock. n.electionEpoch = 0x11, logicalclock=0x12
2015-06-23 14:57:33,316 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:33,317 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 5 (n.leader), 0x1a200000623 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:33,317 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 5, proposed id: 1, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:57:33,317 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=5, proposed zxid=0x1a200000623, proposed election epoch=0x12
2015-06-23 14:57:33,492 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56509
2015-06-23 14:57:33,492 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:33,492 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:33,493 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56509 (no session established for client)
2015-06-23 14:57:34,750 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,751 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,754 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,754 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,754 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,754 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,754 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,754 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,755 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,755 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,755 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,755 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x11 (n.round), FOLLOWING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,755 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,755 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,756 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:34,756 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:34,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 1, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:34,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@716][] - Updating proposal: 3 (newleader), 0x1a200000627 (newzxid), 1 (oldleader), 0x1a200000627 (oldzxid)
2015-06-23 14:57:34,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,757 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,757 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,757 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,757 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:34,757 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:34,757 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:34,757 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:34,894 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56540
2015-06-23 14:57:34,895 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:34,895 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:34,895 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56540 (no session established for client)
2015-06-23 14:57:35,407 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56551
2015-06-23 14:57:35,407 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:35,407 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:35,408 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56551 (no session established for client)
2015-06-23 14:57:36,781 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56610
2015-06-23 14:57:36,781 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:36,782 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:36,782 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56610 (no session established for client)
2015-06-23 14:57:37,156 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56639
2015-06-23 14:57:37,156 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:37,156 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:37,157 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56639 (no session established for client)
2015-06-23 14:57:37,483 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:37,483 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:37,483 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:37,484 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:57:37,484 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:57:37,484 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:57:37,484 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:57:37,484 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:57:37,485 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:37,485 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:37,485 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 1, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:37,485 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=1, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:37,661 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:37,662 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:37,662 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 1, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:37,662 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=1, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:37,752 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:37,752 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:37,752 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:37,752 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:37,887 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:37,887 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 4 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:37,887 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 1, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:37,887 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=4, proposed leader=1, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:38,097 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:38,097 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 5 (n.leader), 0x1a200000623 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:38,098 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 5, proposed id: 3, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:57:38,098 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=5, proposed zxid=0x1a200000623, proposed election epoch=0x12
2015-06-23 14:57:38,593 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56706
2015-06-23 14:57:38,593 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:38,596 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:38,596 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56706 (no session established for client)
2015-06-23 14:57:39,565 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56763
2015-06-23 14:57:39,566 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:39,566 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:39,566 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56763 (no session established for client)
2015-06-23 14:57:41,328 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56810
2015-06-23 14:57:41,328 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:41,328 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:41,329 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56810 (no session established for client)
2015-06-23 14:57:41,786 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56825
2015-06-23 14:57:41,786 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:41,786 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:41,787 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56825 (no session established for client)
2015-06-23 14:57:42,485 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:42,485 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:42,486 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:42,486 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:57:42,486 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:57:42,486 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:57:42,486 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:57:42,486 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:57:42,486 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:57:42,487 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:42,487 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:42,488 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:42,488 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:42,664 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:42,664 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:42,664 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 3, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:42,664 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@716][] - Updating proposal: 8 (newleader), 0x1a200000627 (newzxid), 3 (oldleader), 0x1a200000627 (oldzxid)
2015-06-23 14:57:42,664 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 1 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 2 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 3 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 4 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 5 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 6 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 7 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@586][] - Sending Notification: 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), 8 (recipient), 2 (myid), 0x1a2 (n.peerEpoch)
2015-06-23 14:57:42,665 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:42,753 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:42,753 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:42,753 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:42,753 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:42,908 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:42,908 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 4 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:42,908 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:42,908 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=4, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:43,102 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:43,102 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 5 (n.leader), 0x1a200000623 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:43,102 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 5, proposed id: 8, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:57:43,102 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=5, proposed zxid=0x1a200000623, proposed election epoch=0x12
2015-06-23 14:57:43,361 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56888
2015-06-23 14:57:43,361 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:43,361 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:43,362 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56888 (no session established for client)
2015-06-23 14:57:43,658 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56896
2015-06-23 14:57:43,658 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:43,658 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:43,658 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56896 (no session established for client)
2015-06-23 14:57:45,743 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:56968
2015-06-23 14:57:45,743 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:45,744 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:45,744 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:56968 (no session established for client)
2015-06-23 14:57:46,541 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:56991
2015-06-23 14:57:46,542 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:46,542 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:46,542 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:56991 (no session established for client)
2015-06-23 14:57:47,487 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:47,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:47,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:47,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 1
2015-06-23 14:57:47,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 3
2015-06-23 14:57:47,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 4
2015-06-23 14:57:47,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:57:47,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@364][] - Opening channel to server 6
2015-06-23 14:57:47,489 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:47,489 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 2 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:47,490 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:47,490 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=2, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:47,676 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:47,676 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:47,676 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:47,677 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:47,753 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:47,754 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:47,754 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:47,754 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:47,927 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:47,927 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 4 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:47,927 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:47,927 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=4, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:48,101 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:48,102 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 5 (n.leader), 0x1a200000623 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:48,102 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 5, proposed id: 8, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:57:48,102 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=5, proposed zxid=0x1a200000623, proposed election epoch=0x12
2015-06-23 14:57:48,402 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57031
2015-06-23 14:57:48,402 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:48,402 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:48,402 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57031 (no session established for client)
2015-06-23 14:57:49,376 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57051
2015-06-23 14:57:49,376 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:49,377 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:49,377 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57051 (no session established for client)
2015-06-23 14:57:51,326 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57086
2015-06-23 14:57:51,326 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:51,326 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:51,326 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57086 (no session established for client)
2015-06-23 14:57:51,606 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57093
2015-06-23 14:57:51,684 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:51,684 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:51,684 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57093 (no session established for client)
2015-06-23 14:57:52,488 [myid:2] - WARN [WorkerSender[myid=2]:QuorumCnxManager@382][] - Cannot open channel to 6 at election address /172.17.35.16:3888
2015-06-23 14:57:52,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 7
2015-06-23 14:57:52,488 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 8
2015-06-23 14:57:52,666 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:52,666 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:52,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:52,666 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:52,755 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:52,755 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:52,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:52,756 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:53,109 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:53,110 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 7 (n.leader), 0x1a200000623 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:53,111 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 7, proposed id: 8, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:57:53,111 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=7, proposed zxid=0x1a200000623, proposed election epoch=0x12
2015-06-23 14:57:53,716 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57133
2015-06-23 14:57:53,716 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:53,717 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:53,717 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57133 (no session established for client)
2015-06-23 14:57:53,842 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57138
2015-06-23 14:57:53,842 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:53,842 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:53,842 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57138 (no session established for client)
2015-06-23 14:57:55,891 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57176
2015-06-23 14:57:55,891 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:55,891 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:55,891 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57176 (no session established for client)
2015-06-23 14:57:56,021 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57178
2015-06-23 14:57:56,021 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:56,021 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:56,021 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57178 (no session established for client)
2015-06-23 14:57:57,652 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57210
2015-06-23 14:57:57,653 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:57,653 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:57,653 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57210 (no session established for client)
2015-06-23 14:57:57,671 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:57,672 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 1 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:57,672 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:57,672 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=1, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:57,761 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:57,761 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:57,761 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:57:57,761 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:57:58,100 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:57:58,100 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 7 (n.leader), 0x1a200000623 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:57:58,101 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 7, proposed id: 8, zxid: 0x1a200000623, proposed zxid: 0x1a200000627
2015-06-23 14:57:58,101 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=7, proposed zxid=0x1a200000623, proposed election epoch=0x12
2015-06-23 14:57:58,387 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57224
2015-06-23 14:57:58,388 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:57:58,389 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:57:58,391 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57224 (no session established for client)
2015-06-23 14:58:00,081 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57257
2015-06-23 14:58:00,082 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:00,082 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:00,083 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57257 (no session established for client)
2015-06-23 14:58:00,464 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57265
2015-06-23 14:58:00,464 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:00,464 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:00,465 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57265 (no session established for client)
2015-06-23 14:58:01,842 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57291
2015-06-23 14:58:01,843 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:01,843 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:01,843 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57291 (no session established for client)
2015-06-23 14:58:02,619 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57306
2015-06-23 14:58:02,619 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:02,619 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:02,620 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57306 (no session established for client)
2015-06-23 14:58:02,757 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:58:02,758 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 3 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:58:02,758 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:58:02,758 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=3, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:58:03,107 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:58:03,107 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 1 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:58:03,107 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 1, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:58:03,107 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=1, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:58:03,746 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57326
2015-06-23 14:58:03,747 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:03,747 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:03,747 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57326 (no session established for client)
2015-06-23 14:58:04,709 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57346
2015-06-23 14:58:04,709 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:04,709 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:04,710 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57346 (no session established for client)
2015-06-23 14:58:06,781 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57385
2015-06-23 14:58:06,781 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:06,781 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:06,781 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57385 (no session established for client)
2015-06-23 14:58:07,060 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57390
2015-06-23 14:58:07,060 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:07,060 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:07,061 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57390 (no session established for client)
2015-06-23 14:58:07,697 [myid:2] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@780][] - Connection broken for id 7, my id = 2, error =
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.net.SocketInputStream.read(SocketInputStream.java:203)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:765)
2015-06-23 14:58:07,698 [myid:2] - WARN [RecvWorker:7:QuorumCnxManager$RecvWorker@783][] - Interrupting SendWorker
2015-06-23 14:58:07,698 [myid:2] - DEBUG [RecvWorker:7:QuorumCnxManager$SendWorker@606][] - Calling finish for 7
2015-06-23 14:58:07,698 [myid:2] - DEBUG [RecvWorker:7:QuorumCnxManager$SendWorker@626][] - Removing entry from senderWorkerMap sid=7
2015-06-23 14:58:07,698 [myid:2] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@697][] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:849)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:64)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:685)
2015-06-23 14:58:07,699 [myid:2] - DEBUG [SendWorker:7:QuorumCnxManager$SendWorker@606][] - Calling finish for 7
2015-06-23 14:58:07,699 [myid:2] - WARN [SendWorker:7:QuorumCnxManager$SendWorker@706][] - Send worker leaving thread
2015-06-23 14:58:08,107 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:58:08,107 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 3 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:58:08,107 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 3, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:58:08,107 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=5, proposed leader=3, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:58:09,113 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57430
2015-06-23 14:58:09,113 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:09,113 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:09,114 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57430 (no session established for client)
2015-06-23 14:58:09,726 [myid:2] - INFO [/172.17.35.12:3888:QuorumCnxManager$Listener@511][] - Received connection request /172.17.35.17:57115
2015-06-23 14:58:09,727 [myid:2] - DEBUG [/172.17.35.12:3888:QuorumCnxManager$SendWorker@588][] - Address of remote peer: 7
2015-06-23 14:58:09,728 [myid:2] - DEBUG [SendWorker:7:QuorumCnxManager$SendWorker@668][] - Attempting to send lastMessage to sid=7
2015-06-23 14:58:09,742 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:58:09,742 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 7 (n.sid), 0x1a2 (n.peerEpoch) LOOKING (my state)
2015-06-23 14:58:09,742 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609][] - id: 8, proposed id: 8, zxid: 0x1a200000627, proposed zxid: 0x1a200000627
2015-06-23 14:58:09,742 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885][] - Adding vote: from=7, proposed leader=8, proposed zxid=0x1a200000627, proposed election epoch=0x12
2015-06-23 14:58:09,904 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /0:0:0:0:0:0:0:1:57445
2015-06-23 14:58:09,905 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362][] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-23 14:58:09,905 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366][] - IOException stack trace
2015-06-23 14:58:09,905 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /0:0:0:0:0:0:0:1:57445 (no session established for client)
2015-06-23 14:58:09,944 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@738][] - I'm a participant: 2
2015-06-23 14:58:09,944 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@548][] - About to leave FLE instance: leader=8, zxid=0x1a200000627, my id=2, my state=FOLLOWING
2015-06-23 14:58:09,944 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:QuorumPeer@784][] - FOLLOWING
2015-06-23 14:58:09,944 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@162][] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir c:\motum\config\MASS\ZK\version-2 snapdir c:\motum\config\MASS\ZK\version-2
2015-06-23 14:58:09,944 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@63][] - FOLLOWING - LEADER ELECTION TOOK - 101874
2015-06-23 14:58:09,966 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Learner@323][] - Getting a diff from the leader 0x1a30000000e
2015-06-23 14:58:09,970 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Learner@374][] - Got zxid 0x1a200000624 expected 0x1
2015-06-23 14:58:09,970 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Learner@374][] - Got zxid 0x1a300000001 expected 0x1a200000628
2015-06-23 14:58:09,973 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@240][] - Snapshotting: 0x1a30000000e to c:\motum\config\MASS\ZK\version-2\snapshot.1a30000000e
2015-06-23 14:58:10,184 [myid:2] - WARN [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Follower@118][] - Got zxid 0x1a30000000f expected 0x1
2015-06-23 14:58:10,193 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor@161][] - Committing request:: sessionid:0x74e207f30bb0000 type:multi cxid:0x3 zxid:0x1a30000000f txntype:14 reqpath:n/a
2015-06-23 14:58:10,193 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x74e207f30bb0000 type:multi cxid:0x3 zxid:0x1a30000000f txntype:14 reqpath:n/a
2015-06-23 14:58:10,193 [myid:2] - TRACE [CommitProcessor:2:ZooTrace@90][] - :Esessionid:0x74e207f30bb0000 type:multi cxid:0x3 zxid:0x1a30000000f txntype:14 reqpath:n/a
2015-06-23 14:58:11,158 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor@161][] - Committing request:: sessionid:0x14e207f300f0000 type:multi cxid:0x1 zxid:0x1a300000010 txntype:14 reqpath:n/a
2015-06-23 14:58:11,159 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x14e207f300f0000 type:multi cxid:0x1 zxid:0x1a300000010 txntype:14 reqpath:n/a
2015-06-23 14:58:11,159 [myid:2] - TRACE [CommitProcessor:2:ZooTrace@90][] - :Esessionid:0x14e207f300f0000 type:multi cxid:0x1 zxid:0x1a300000010 txntype:14 reqpath:n/a
2015-06-23 14:58:11,186 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor@161][] - Committing request:: sessionid:0x14e207f300f0000 type:multi cxid:0x7 zxid:0x1a300000011 txntype:14 reqpath:n/a
2015-06-23 14:58:11,186 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x14e207f300f0000 type:multi cxid:0x7 zxid:0x1a300000011 txntype:14 reqpath:n/a
2015-06-23 14:58:11,186 [myid:2] - TRACE [CommitProcessor:2:ZooTrace@90][] - :Esessionid:0x14e207f300f0000 type:multi cxid:0x7 zxid:0x1a300000011 txntype:14 reqpath:n/a
2015-06-23 14:58:11,598 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57479
2015-06-23 14:58:11,600 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@810][] - Session establishment request from client /127.0.0.1:57479 client's lastZxid is 0x1a200000621
2015-06-23 14:58:11,600 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861][] - Client attempting to renew session 0x24e201711a20000 at /127.0.0.1:57479
2015-06-23 14:58:11,600 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108][] - Revalidating client: 0x24e201711a20000
2015-06-23 14:58:11,600 [myid:2] - TRACE [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooTrace@71][] - To validate session 0x24e201711a20000
2015-06-23 14:58:11,606 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@610][] - Invalid session 0x24e201711a20000 for client /127.0.0.1:57479, probably expired
2015-06-23 14:58:11,608 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@168][] - Add a buffer to outgoingBuffers, sk sun.nio.ch.SelectionKeyImpl@11a6650 is valid: true
2015-06-23 14:58:11,609 [myid:2] - TRACE [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooTrace@71][] - Session 0x24e201711a20000 is valid: false
2015-06-23 14:58:11,609 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007][] - Closed socket connection for client /127.0.0.1:57479 which had sessionid 0x24e201711a20000
2015-06-23 14:58:11,721 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197][] - Accepted socket connection from /127.0.0.1:57483
2015-06-23 14:58:11,722 [myid:2] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@810][] - Session establishment request from client /127.0.0.1:57483 client's lastZxid is 0x0
2015-06-23 14:58:11,722 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868][] - Client attempting to establish new session at /127.0.0.1:57483
2015-06-23 14:58:11,722 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x24e207f4aae0000 type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:11,723 [myid:2] - DEBUG [FollowerRequestProcessor:2:CommitProcessor@171][] - Processing request:: sessionid:0x24e207f4aae0000 type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:11,740 [myid:2] - DEBUG [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor@161][] - Committing request:: sessionid:0x24e207f4aae0000 type:createSession cxid:0x0 zxid:0x1a300000012 txntype:-10 reqpath:n/a
2015-06-23 14:58:11,740 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x24e207f4aae0000 type:createSession cxid:0x0 zxid:0x1a300000012 txntype:-10 reqpath:n/a
2015-06-23 14:58:11,740 [myid:2] - TRACE [CommitProcessor:2:ZooTrace@90][] - :Esessionid:0x24e207f4aae0000 type:createSession cxid:0x0 zxid:0x1a300000012 txntype:-10 reqpath:n/a
2015-06-23 14:58:11,740 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@160][] - sessionid:0x24e207f4aae0000 type:createSession cxid:0x0 zxid:0x1a300000012 txntype:-10 reqpath:n/a
2015-06-23 14:58:11,742 [myid:2] - INFO [CommitProcessor:2:ZooKeeperServer@617][] - Established session 0x24e207f4aae0000 with negotiated timeout 4000 for client /127.0.0.1:57483
2015-06-23 14:58:13,054 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:13,054 [myid:2] - DEBUG [FollowerRequestProcessor:2:CommitProcessor@171][] - Processing request:: sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:13,054 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:13,054 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@160][] - sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:13,102 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x24e207f4aae0000 type:getData cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,103 [myid:2] - DEBUG [FollowerRequestProcessor:2:CommitProcessor@171][] - Processing request:: sessionid:0x24e207f4aae0000 type:getData cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,103 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x24e207f4aae0000 type:getData cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,103 [myid:2] - TRACE [CommitProcessor:2:ZooTrace@90][] - :Esessionid:0x24e207f4aae0000 type:getData cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,103 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@160][] - sessionid:0x24e207f4aae0000 type:getData cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,106 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x24e207f4aae0000 type:getData cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,106 [myid:2] - DEBUG [FollowerRequestProcessor:2:CommitProcessor@171][] - Processing request:: sessionid:0x24e207f4aae0000 type:getData cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,114 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:58:13,114 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), LOOKING (n.state), 5 (n.sid), 0x1a2 (n.peerEpoch) FOLLOWING (my state)
2015-06-23 14:58:13,115 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@367][] - Sending new notification. My id = 2 recipient=5 zxid=0x1a200000627 leader=8
2015-06-23 14:58:13,114 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x24e207f4aae0000 type:getData cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,115 [myid:2] - TRACE [CommitProcessor:2:ZooTrace@90][] - :Esessionid:0x24e207f4aae0000 type:getData cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,115 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@160][] - sessionid:0x24e207f4aae0000 type:getData cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:/Orderfiltering
2015-06-23 14:58:13,117 [myid:2] - DEBUG [WorkerSender[myid=2]:QuorumCnxManager@387][] - There is a connection already for server 5
2015-06-23 14:58:14,441 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:14,441 [myid:2] - DEBUG [FollowerRequestProcessor:2:CommitProcessor@171][] - Processing request:: sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:14,441 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:14,441 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@160][] - sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:14,736 [myid:2] - DEBUG [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@270][] - Receive new notification message. My id = 2
2015-06-23 14:58:14,736 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@597][] - Notification: 1 (message format version), 8 (n.leader), 0x1a200000627 (n.zxid), 0x12 (n.round), FOLLOWING (n.state), 7 (n.sid), 0x1a3 (n.peerEpoch) FOLLOWING (my state)
2015-06-23 14:58:15,773 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:15,773 [myid:2] - DEBUG [FollowerRequestProcessor:2:CommitProcessor@171][] - Processing request:: sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:15,774 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@88][] - Processing request:: sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:15,774 [myid:2] - DEBUG [CommitProcessor:2:FinalRequestProcessor@160][] - sessionid:0x24e207f4aae0000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
2015-06-23 14:58:15,807 [myid:2] - TRACE [FollowerRequestProcessor:2:ZooTrace@90][] - :Fsessionid:0x24e207f4aae0000 type:multi cxid:0x3 zxid:0xfffffffffffffffe txntype:unknown reqpath:
Re: Leader election problems
Posted by Raúl Gutiérrez Segalés <rg...@itevenworks.net>.
On 25 June 2015 at 19:23, Alexander Shraer <sh...@gmail.com> wrote:
>
> A Group Communication / Virtual Synchrony may be better suited here. In
> Paxos-based systems
> you need a majority of the servers to remain connected to have progress.
It
> tolerates a minority being
> offline but stops working once you have more failures / disconnects. You
> could reconfigure
> to remove someone from the cluster when he is suspected (otherwise you
risk
> remaining without a quorum).
> With group communication / virtual synchrony reconfiguration happens
> automatically when a server
> leaves and it gives you primitives to communicate with members of the
> current group, such as atomic broadcast
> (equivalent to consensus). You can achieve the same with ZK, but it may be
> more work (I'm not sure).
>
> Check out Isis2 from (Ken Birman's project), Ensemble group communication,
> JGroups and others.
Or, if you are running 3.5 (or later) you could run this command [0] from
your shutdown
scripts to remove each server as they go away:
zk-shell --run-once 'reconfig remove 7' localhost
-rgs
[0] https://github.com/rgs1/zk_shell
Re: Leader election problems
Posted by Alexander Shraer <sh...@gmail.com>.
A Group Communication / Virtual Synchrony may be better suited here. In
Paxos-based systems
you need a majority of the servers to remain connected to have progress. It
tolerates a minority being
offline but stops working once you have more failures / disconnects. You
could reconfigure
to remove someone from the cluster when he is suspected (otherwise you risk
remaining without a quorum).
With group communication / virtual synchrony reconfiguration happens
automatically when a server
leaves and it gives you primitives to communicate with members of the
current group, such as atomic broadcast
(equivalent to consensus). You can achieve the same with ZK, but it may be
more work (I'm not sure).
Check out Isis2 from (Ken Birman's project), Ensemble group communication,
JGroups and others.
On Wed, Jun 24, 2015 at 11:39 PM, Filip Deleersnijder <fi...@motum.be>
wrote:
> Hi,
>
> Thanks for your response.
>
> Our application consists of 8 automatic vehicles in a warehouse setting.
> Those vehicles need some consensus decisions, and that is what we use
> Zookeeper for.
> Because vehicles can come and go at random, we installed a ZK participant
> on every vehicle. The ZK client is some other piece of software that is
> also running on the vehicles.
>
> Therefor :
> - We can not choose the number of ZK-participants because it just
> depends on the number of vehicles.
> - The participants communicate over Wifi
> - The client is running on the same machine, so it communicates
> over the local network
>
> We are running Zookeeper version 3.4.6
>
> Our zoo.cfg can be found below this e-mail.
>
> Thanks in advance !
>
> Filip
>
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=c:/motum/config/MASS/ZK
> # the port at which the clients will connect
> clientPort=2181
>
> server.1=172.17.35.11:2888:3888
> server.2=172.17.35.12:2888:3888
> server.3=172.17.35.13:2888:3888
> server.4=172.17.35.14:2888:3888
> server.5=172.17.35.15:2888:3888
> server.6=172.17.35.16:2888:3888
> server.7=172.17.35.17:2888:3888
> server.8=172.17.35.18:2888:3888
>
> # The number of snapshots to retain in dataDir
> # Purge task interval in hours
> # Set to "0" to disable auto purge feature
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=1
>
>
>
> > On 24 Jun 2015, at 18:54, Raúl Gutiérrez Segalés <rg...@itevenworks.net>
> wrote:
> >
> > Hi,
> >
> > On 24 June 2015 at 06:05, Filip Deleersnijder <fi...@motum.be> wrote:
> >
> >> Hi,
> >>
> >> Let’s start with some description of our system :
> >>
> >> - We our using a Zookeeper cluster with 8 participants for an
> application
> >> with mobile nodes ( connected over Wifi ).
> >>
> >
> > You mean the participants talk over wifi or the clients?
> >
> >
> >> ( Ip of the different nodes are according to the following structure :
> >> Node X has IP : 172.17.35.1X )
> >>
> >
> > Why 8 and not an odd number of machines (i.e.:
> >
> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
> > )?
> >
> > - It is not that unusual to have a node being shut-down or restarted
> >> - We haven’t benchmarked the number of write operations yet, but I would
> >> estimate that it would be less than 10 writes / second
> >>
> >
> > What version of ZK are you using?
> >
> >
> >>
> >> The problem we are having however is that sometimes(*), some instances
> >> seem to be having problems with leader election.
> >> Under the header “Attachment 1” below, you can find the leader election
> >> times that were needed over 24h ( from 1 node ). One average it took
> more
> >> than 1 minute !
> >> I assume that this is not normal behaviour ? ( If somebody could confirm
> >> that in a 8-node cluster, these are not normal leader election times,
> that
> >> would be nice )
> >>
> >> In attachement 2 : I included an extract from the logging during a
> leader
> >> election that took 101874ms for 1 node ( server 2 ).
> >>
> >> Any help is greatly appreciated.
> >> If further or more specific logging is required, please ask !
> >>
> >>
> > Do you mind sharing a copy of your config file (zoo.cfg)? Thanks!
> >
> >
> > -rgs
>
>
Re: Leader election problems
Posted by Raúl Gutiérrez Segalés <rg...@itevenworks.net>.
On 25 June 2015 at 08:12, Filip Deleersnijder <fi...@motum.be> wrote:
>
> Some more information that we just got from our installation.
>
> - To check the EOF theory, we cleared all the version-2 data directories
> and restarted all vehicles
> - 2 of the 8 vehicles were unavailable at that time
> - The remaining 6 vehicles did not succeed in finishing leader election
> phase !
>
After how long?
>
> A log extract can be found at the end of this e-mail.
>
> - We then removed the 2 vehicles from the configuration just for testing
> purposes
> - Now leader election succeed without any problems !
>
How long did it take?
Having the logs for all servers (for both cases: for when the elections
fails and for when it succeeds) would be helpful. Please paste the logs on
gist.github.com or something similar since reading them in-line is a bit
hard.
I don't think the unclean shutdowns should be a problem, as long as quorum
can be formed (i.e.: they'll just be ignored, since ZooKeeper is meant to
be crash safe) (also, a Windows shutdown is probably not very different
then a shutdown through the init scripts).
-rgs
Re: Leader election problems
Posted by Filip Deleersnijder <fi...@motum.be>.
Hi,
Some more information that we just got from our installation.
- To check the EOF theory, we cleared all the version-2 data directories and restarted all vehicles
- 2 of the 8 vehicles were unavailable at that time
- The remaining 6 vehicles did not succeed in finishing leader election phase !
A log extract can be found at the end of this e-mail.
- We then removed the 2 vehicles from the configuration just for testing purposes
- Now leader election succeed without any problems !
Any insights are very helpful !
Filip
Logging during failing leader election phase :
2015-06-25 16:29:30,965 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65133 (no session established for client)
2015-06-25 16:29:31,171 [myid:8] - WARN [WorkerSender[myid=8]:QuorumCnxManager@382] - Cannot open channel to 5 at election address /172.17.35.15:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:31,189 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@364] - Opening channel to server 6
2015-06-25 16:29:31,464 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65145
2015-06-25 16:29:31,466 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:31,467 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:31,467 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65145 (no session established for client)
2015-06-25 16:29:32,370 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65163
2015-06-25 16:29:32,372 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:32,374 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:32,374 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65163 (no session established for client)
2015-06-25 16:29:32,585 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65167
2015-06-25 16:29:32,587 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:32,588 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:32,589 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65167 (no session established for client)
2015-06-25 16:29:32,800 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65173
2015-06-25 16:29:32,805 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:32,809 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:32,810 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65173 (no session established for client)
2015-06-25 16:29:33,543 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65190
2015-06-25 16:29:33,544 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:33,546 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:33,546 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65190 (no session established for client)
2015-06-25 16:29:33,983 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65200
2015-06-25 16:29:33,985 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:33,986 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:33,986 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65200 (no session established for client)
2015-06-25 16:29:34,527 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65212
2015-06-25 16:29:34,529 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:34,530 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:34,530 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65212 (no session established for client)
2015-06-25 16:29:35,108 [myid:8] - DEBUG [WorkerReceiver[myid=8]:FastLeaderElection$Messenger$WorkerReceiver@270] - Receive new notification message. My id = 8
2015-06-25 16:29:35,109 [myid:8] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x1ae0000019c (n.zxid), 0x2e (n.round), LOOKING (n.state), 4 (n.sid), 0x1ae (n.peerEpoch) LOOKING (my state)
2015-06-25 16:29:35,111 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609] - id: 2, proposed id: 8, zxid: 0x1ae0000019c, proposed zxid: 0x1ae0000019c
2015-06-25 16:29:35,113 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885] - Adding vote: from=4, proposed leader=2, proposed zxid=0x1ae0000019c, proposed election epoch=0x2e
2015-06-25 16:29:35,113 [myid:8] - DEBUG [WorkerReceiver[myid=8]:FastLeaderElection$Messenger$WorkerReceiver@270] - Receive new notification message. My id = 8
2015-06-25 16:29:35,113 [myid:8] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@597] - Notification: 1 (message format version), 8 (n.leader), 0x1ae0000019c (n.zxid), 0x2e (n.round), LOOKING (n.state), 2 (n.sid), 0x1ae (n.peerEpoch) LOOKING (my state)
2015-06-25 16:29:35,115 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609] - id: 8, proposed id: 8, zxid: 0x1ae0000019c, proposed zxid: 0x1ae0000019c
2015-06-25 16:29:35,115 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885] - Adding vote: from=2, proposed leader=8, proposed zxid=0x1ae0000019c, proposed election epoch=0x2e
2015-06-25 16:29:35,333 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65229
2015-06-25 16:29:35,335 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:35,336 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:35,336 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65229 (no session established for client)
2015-06-25 16:29:35,369 [myid:8] - DEBUG [WorkerReceiver[myid=8]:FastLeaderElection$Messenger$WorkerReceiver@270] - Receive new notification message. My id = 8
2015-06-25 16:29:35,369 [myid:8] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x1ae0000019c (n.zxid), 0x2e (n.round), LOOKING (n.state), 1 (n.sid), 0x1ae (n.peerEpoch) LOOKING (my state)
2015-06-25 16:29:35,371 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609] - id: 2, proposed id: 8, zxid: 0x1ae0000019c, proposed zxid: 0x1ae0000019c
2015-06-25 16:29:35,371 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885] - Adding vote: from=1, proposed leader=2, proposed zxid=0x1ae0000019c, proposed election epoch=0x2e
2015-06-25 16:29:35,776 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65239
2015-06-25 16:29:35,779 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:35,780 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:35,781 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65239 (no session established for client)
2015-06-25 16:29:36,052 [myid:8] - DEBUG [WorkerReceiver[myid=8]:FastLeaderElection$Messenger$WorkerReceiver@270] - Receive new notification message. My id = 8
2015-06-25 16:29:36,052 [myid:8] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@597] - Notification: 1 (message format version), 8 (n.leader), 0x1ae0000019c (n.zxid), 0x2e (n.round), LOOKING (n.state), 3 (n.sid), 0x1ae (n.peerEpoch) LOOKING (my state)
2015-06-25 16:29:36,054 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609] - id: 8, proposed id: 8, zxid: 0x1ae0000019c, proposed zxid: 0x1ae0000019c
2015-06-25 16:29:36,054 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885] - Adding vote: from=3, proposed leader=8, proposed zxid=0x1ae0000019c, proposed election epoch=0x2e
2015-06-25 16:29:36,064 [myid:8] - DEBUG [WorkerReceiver[myid=8]:FastLeaderElection$Messenger$WorkerReceiver@270] - Receive new notification message. My id = 8
2015-06-25 16:29:36,064 [myid:8] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@597] - Notification: 1 (message format version), 7 (n.leader), 0x1ae00000196 (n.zxid), 0x2e (n.round), LOOKING (n.state), 7 (n.sid), 0x1ae (n.peerEpoch) LOOKING (my state)
2015-06-25 16:29:36,066 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609] - id: 7, proposed id: 8, zxid: 0x1ae00000196, proposed zxid: 0x1ae0000019c
2015-06-25 16:29:36,066 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885] - Adding vote: from=7, proposed leader=7, proposed zxid=0x1ae00000196, proposed election epoch=0x2e
2015-06-25 16:29:36,189 [myid:8] - WARN [WorkerSender[myid=8]:QuorumCnxManager@382] - Cannot open channel to 6 at election address /172.17.35.16:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:36,198 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@387] - There is a connection already for server 7
2015-06-25 16:29:36,198 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@387] - There is a connection already for server 1
2015-06-25 16:29:36,198 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@387] - There is a connection already for server 2
2015-06-25 16:29:36,198 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@387] - There is a connection already for server 3
2015-06-25 16:29:36,198 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@387] - There is a connection already for server 4
2015-06-25 16:29:36,198 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@364] - Opening channel to server 5
2015-06-25 16:29:36,200 [myid:8] - DEBUG [WorkerReceiver[myid=8]:FastLeaderElection$Messenger$WorkerReceiver@270] - Receive new notification message. My id = 8
2015-06-25 16:29:36,201 [myid:8] - INFO [WorkerReceiver[myid=8]:FastLeaderElection@597] - Notification: 1 (message format version), 8 (n.leader), 0x1ae0000019c (n.zxid), 0x2e (n.round), LOOKING (n.state), 8 (n.sid), 0x1ae (n.peerEpoch) LOOKING (my state)
2015-06-25 16:29:36,215 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@609] - id: 8, proposed id: 8, zxid: 0x1ae0000019c, proposed zxid: 0x1ae0000019c
2015-06-25 16:29:36,215 [myid:8] - DEBUG [QuorumPeer[myid=8]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@885] - Adding vote: from=8, proposed leader=8, proposed zxid=0x1ae0000019c, proposed election epoch=0x2e
2015-06-25 16:29:36,309 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65251
2015-06-25 16:29:36,315 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:36,344 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:36,345 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65251 (no session established for client)
2015-06-25 16:29:36,369 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65254
2015-06-25 16:29:36,371 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:36,372 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:36,373 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65254 (no session established for client)
2015-06-25 16:29:38,053 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65288
2015-06-25 16:29:38,056 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:38,057 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:38,058 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65288 (no session established for client)
2015-06-25 16:29:38,328 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65294
2015-06-25 16:29:38,330 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:38,331 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:38,332 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65294 (no session established for client)
2015-06-25 16:29:38,679 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65302
2015-06-25 16:29:38,686 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:38,688 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:38,689 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65302 (no session established for client)
2015-06-25 16:29:39,385 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65318
2015-06-25 16:29:39,393 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:39,400 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:39,401 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65318 (no session established for client)
2015-06-25 16:29:39,973 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65331
2015-06-25 16:29:39,975 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:39,977 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:39,977 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65331 (no session established for client)
2015-06-25 16:29:40,510 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65343
2015-06-25 16:29:40,512 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:40,514 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:40,514 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:65343 (no session established for client)
2015-06-25 16:29:40,973 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65352
2015-06-25 16:29:40,975 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:40,977 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:40,977 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65352 (no session established for client)
2015-06-25 16:29:41,200 [myid:8] - WARN [WorkerSender[myid=8]:QuorumCnxManager@382] - Cannot open channel to 5 at election address /172.17.35.15:3888
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:41,211 [myid:8] - DEBUG [WorkerSender[myid=8]:QuorumCnxManager@364] - Opening channel to server 6
2015-06-25 16:29:41,564 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /0:0:0:0:0:0:0:1:65364
2015-06-25 16:29:41,566 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:41,567 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
java.io.IOException: ZooKeeperServer not running
at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:931)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:237)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2015-06-25 16:29:41,567 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:65364 (no session established for client)
2015-06-25 16:29:42,448 [myid:8] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:65384
2015-06-25 16:29:42,465 [myid:8] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2015-06-25 16:29:42,467 [myid:8] - DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@366] - IOException stack trace
> On 25 Jun 2015, at 16:34, Filip Deleersnijder <fi...@motum.be> wrote:
>
> Hi,
>
> I can see that all of our logs contain the following log-statements pretty often.
>
> 2015-06-22 12:02:00,752 [myid:2] - DEBUG [main:DataTree@949][] - Ignoring processTxn failure hdr: -1 : error: -2
> 2015-06-22 12:02:00,753 [myid:2] - DEBUG [main:DataTree@949][] - Ignoring processTxn failure hdr: 14 : error: -101
>
> 2015-06-25 14:02:39,505 [myid:3] - DEBUG [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FileTxnLog$FileTxnIterator@636] - EOF excepton java.io.EOFException: Failed to read c:\motum\config\MASS\ZK\version-2\log.1aa00000001
> Since we don’t properly shut the ZK process down ( we just shutdown windows ), this properly can cause corruption of files.
>
> Is there somebody that has a clear idea about whether the “EOF” or the “Ignoring processTxn” problems could cause frequent and long during Leader Elections ?
>
> Any help is greatly appreciated,
>
> Filip
>
>
>
>> On 25 Jun 2015, at 11:51, Guy Moshkowich <guy.moshkowich@gmail.com <ma...@gmail.com>> wrote:
>>
>> Are you using ZK client on your vehicles or ZK servers?
>> You mentioned below 8 vehicles and i see 8 servers defined in the config.
>> I would expect you have 8 client(running on your vehicles) communicating
>> against 1 or 3 ZK servers as this will be more than enough for 8 clients.
>> Guy
>>
>> On Thursday, June 25, 2015, Filip Deleersnijder <filip@motum.be <ma...@motum.be>> wrote:
>>
>>> Hi,
>>>
>>> Thanks for your response.
>>>
>>> Our application consists of 8 automatic vehicles in a warehouse setting.
>>> Those vehicles need some consensus decisions, and that is what we use
>>> Zookeeper for.
>>> Because vehicles can come and go at random, we installed a ZK participant
>>> on every vehicle. The ZK client is some other piece of software that is
>>> also running on the vehicles.
>>>
>>> Therefor :
>>> - We can not choose the number of ZK-participants because it just
>>> depends on the number of vehicles.
>>> - The participants communicate over Wifi
>>> - The client is running on the same machine, so it communicates
>>> over the local network
>>>
>>> We are running Zookeeper version 3.4.6
>>>
>>> Our zoo.cfg can be found below this e-mail.
>>>
>>> Thanks in advance !
>>>
>>> Filip
>>>
>>> # The number of milliseconds of each tick
>>> tickTime=2000
>>> # The number of ticks that the initial
>>> # synchronization phase can take
>>> initLimit=10
>>> # The number of ticks that can pass between
>>> # sending a request and getting an acknowledgement
>>> syncLimit=5
>>> # the directory where the snapshot is stored.
>>> # do not use /tmp for storage, /tmp here is just
>>> # example sakes.
>>> dataDir=c:/motum/config/MASS/ZK
>>> # the port at which the clients will connect
>>> clientPort=2181
>>>
>>> server.1=172.17.35.11:2888:3888
>>> server.2=172.17.35.12:2888:3888
>>> server.3=172.17.35.13:2888:3888
>>> server.4=172.17.35.14:2888:3888
>>> server.5=172.17.35.15:2888:3888
>>> server.6=172.17.35.16:2888:3888
>>> server.7=172.17.35.17:2888:3888
>>> server.8=172.17.35.18:2888:3888
>>>
>>> # The number of snapshots to retain in dataDir
>>> # Purge task interval in hours
>>> # Set to "0" to disable auto purge feature
>>> autopurge.snapRetainCount=3
>>> autopurge.purgeInterval=1
>>>
>>>
>>>
>>>> On 24 Jun 2015, at 18:54, Raúl Gutiérrez Segalés <rgs@itevenworks.net <ma...@itevenworks.net>
>>> <javascript:;>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 24 June 2015 at 06:05, Filip Deleersnijder <filip@motum.be <ma...@motum.be>
>>> <javascript:;>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Let’s start with some description of our system :
>>>>>
>>>>> - We our using a Zookeeper cluster with 8 participants for an
>>> application
>>>>> with mobile nodes ( connected over Wifi ).
>>>>>
>>>>
>>>> You mean the participants talk over wifi or the clients?
>>>>
>>>>
>>>>> ( Ip of the different nodes are according to the following structure :
>>>>> Node X has IP : 172.17.35.1X )
>>>>>
>>>>
>>>> Why 8 and not an odd number of machines (i.e.:
>>>>
>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup <http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup>
>>>> )?
>>>>
>>>> - It is not that unusual to have a node being shut-down or restarted
>>>>> - We haven’t benchmarked the number of write operations yet, but I would
>>>>> estimate that it would be less than 10 writes / second
>>>>>
>>>>
>>>> What version of ZK are you using?
>>>>
>>>>
>>>>>
>>>>> The problem we are having however is that sometimes(*), some instances
>>>>> seem to be having problems with leader election.
>>>>> Under the header “Attachment 1” below, you can find the leader election
>>>>> times that were needed over 24h ( from 1 node ). One average it took
>>> more
>>>>> than 1 minute !
>>>>> I assume that this is not normal behaviour ? ( If somebody could confirm
>>>>> that in a 8-node cluster, these are not normal leader election times,
>>> that
>>>>> would be nice )
>>>>>
>>>>> In attachement 2 : I included an extract from the logging during a
>>> leader
>>>>> election that took 101874ms for 1 node ( server 2 ).
>>>>>
>>>>> Any help is greatly appreciated.
>>>>> If further or more specific logging is required, please ask !
>>>>>
>>>>>
>>>> Do you mind sharing a copy of your config file (zoo.cfg)? Thanks!
>>>>
>>>>
>>>> -rgs
>
Re: Leader election problems
Posted by Filip Deleersnijder <fi...@motum.be>.
Hi,
I can see that all of our logs contain the following log-statements pretty often.
2015-06-22 12:02:00,752 [myid:2] - DEBUG [main:DataTree@949][] - Ignoring processTxn failure hdr: -1 : error: -2
2015-06-22 12:02:00,753 [myid:2] - DEBUG [main:DataTree@949][] - Ignoring processTxn failure hdr: 14 : error: -101
2015-06-25 14:02:39,505 [myid:3] - DEBUG [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FileTxnLog$FileTxnIterator@636] - EOF excepton java.io.EOFException: Failed to read c:\motum\config\MASS\ZK\version-2\log.1aa00000001
Since we don’t properly shut the ZK process down ( we just shutdown windows ), this properly can cause corruption of files.
Is there somebody that has a clear idea about whether the “EOF” or the “Ignoring processTxn” problems could cause frequent and long during Leader Elections ?
Any help is greatly appreciated,
Filip
> On 25 Jun 2015, at 11:51, Guy Moshkowich <gu...@gmail.com> wrote:
>
> Are you using ZK client on your vehicles or ZK servers?
> You mentioned below 8 vehicles and i see 8 servers defined in the config.
> I would expect you have 8 client(running on your vehicles) communicating
> against 1 or 3 ZK servers as this will be more than enough for 8 clients.
> Guy
>
> On Thursday, June 25, 2015, Filip Deleersnijder <filip@motum.be <ma...@motum.be>> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> Our application consists of 8 automatic vehicles in a warehouse setting.
>> Those vehicles need some consensus decisions, and that is what we use
>> Zookeeper for.
>> Because vehicles can come and go at random, we installed a ZK participant
>> on every vehicle. The ZK client is some other piece of software that is
>> also running on the vehicles.
>>
>> Therefor :
>> - We can not choose the number of ZK-participants because it just
>> depends on the number of vehicles.
>> - The participants communicate over Wifi
>> - The client is running on the same machine, so it communicates
>> over the local network
>>
>> We are running Zookeeper version 3.4.6
>>
>> Our zoo.cfg can be found below this e-mail.
>>
>> Thanks in advance !
>>
>> Filip
>>
>> # The number of milliseconds of each tick
>> tickTime=2000
>> # The number of ticks that the initial
>> # synchronization phase can take
>> initLimit=10
>> # The number of ticks that can pass between
>> # sending a request and getting an acknowledgement
>> syncLimit=5
>> # the directory where the snapshot is stored.
>> # do not use /tmp for storage, /tmp here is just
>> # example sakes.
>> dataDir=c:/motum/config/MASS/ZK
>> # the port at which the clients will connect
>> clientPort=2181
>>
>> server.1=172.17.35.11:2888:3888
>> server.2=172.17.35.12:2888:3888
>> server.3=172.17.35.13:2888:3888
>> server.4=172.17.35.14:2888:3888
>> server.5=172.17.35.15:2888:3888
>> server.6=172.17.35.16:2888:3888
>> server.7=172.17.35.17:2888:3888
>> server.8=172.17.35.18:2888:3888
>>
>> # The number of snapshots to retain in dataDir
>> # Purge task interval in hours
>> # Set to "0" to disable auto purge feature
>> autopurge.snapRetainCount=3
>> autopurge.purgeInterval=1
>>
>>
>>
>>> On 24 Jun 2015, at 18:54, Raúl Gutiérrez Segalés <rgs@itevenworks.net
>> <javascript:;>> wrote:
>>>
>>> Hi,
>>>
>>> On 24 June 2015 at 06:05, Filip Deleersnijder <filip@motum.be <ma...@motum.be>
>> <javascript:;>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Let’s start with some description of our system :
>>>>
>>>> - We our using a Zookeeper cluster with 8 participants for an
>> application
>>>> with mobile nodes ( connected over Wifi ).
>>>>
>>>
>>> You mean the participants talk over wifi or the clients?
>>>
>>>
>>>> ( Ip of the different nodes are according to the following structure :
>>>> Node X has IP : 172.17.35.1X )
>>>>
>>>
>>> Why 8 and not an odd number of machines (i.e.:
>>>
>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup <http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup>
>>> )?
>>>
>>> - It is not that unusual to have a node being shut-down or restarted
>>>> - We haven’t benchmarked the number of write operations yet, but I would
>>>> estimate that it would be less than 10 writes / second
>>>>
>>>
>>> What version of ZK are you using?
>>>
>>>
>>>>
>>>> The problem we are having however is that sometimes(*), some instances
>>>> seem to be having problems with leader election.
>>>> Under the header “Attachment 1” below, you can find the leader election
>>>> times that were needed over 24h ( from 1 node ). One average it took
>> more
>>>> than 1 minute !
>>>> I assume that this is not normal behaviour ? ( If somebody could confirm
>>>> that in a 8-node cluster, these are not normal leader election times,
>> that
>>>> would be nice )
>>>>
>>>> In attachement 2 : I included an extract from the logging during a
>> leader
>>>> election that took 101874ms for 1 node ( server 2 ).
>>>>
>>>> Any help is greatly appreciated.
>>>> If further or more specific logging is required, please ask !
>>>>
>>>>
>>> Do you mind sharing a copy of your config file (zoo.cfg)? Thanks!
>>>
>>>
>>> -rgs
Re: Leader election problems
Posted by Filip Deleersnijder <fi...@motum.be>.
Hi,
The reason we are not using the architecture as proposed by you is because of our application requirements.
We have the following hard requirements in our application :
- No static hardware / servers can be used, beside the hardware that is on-board in the vehicles
- It must be allowed to shut any ( non-majority ) subset of vehicles off without the system stopping to work
Therefor, we have 1 ZK-server per vehicle, and every vehicle is also running the Client code that is connected to it’s “local” ZK-server
Filip
> On 25 Jun 2015, at 11:51, Guy Moshkowich <gu...@gmail.com> wrote:
>
> Are you using ZK client on your vehicles or ZK servers?
> You mentioned below 8 vehicles and i see 8 servers defined in the config.
> I would expect you have 8 client(running on your vehicles) communicating
> against 1 or 3 ZK servers as this will be more than enough for 8 clients.
> Guy
>
> On Thursday, June 25, 2015, Filip Deleersnijder <filip@motum.be <ma...@motum.be>> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> Our application consists of 8 automatic vehicles in a warehouse setting.
>> Those vehicles need some consensus decisions, and that is what we use
>> Zookeeper for.
>> Because vehicles can come and go at random, we installed a ZK participant
>> on every vehicle. The ZK client is some other piece of software that is
>> also running on the vehicles.
>>
>> Therefor :
>> - We can not choose the number of ZK-participants because it just
>> depends on the number of vehicles.
>> - The participants communicate over Wifi
>> - The client is running on the same machine, so it communicates
>> over the local network
>>
>> We are running Zookeeper version 3.4.6
>>
>> Our zoo.cfg can be found below this e-mail.
>>
>> Thanks in advance !
>>
>> Filip
>>
>> # The number of milliseconds of each tick
>> tickTime=2000
>> # The number of ticks that the initial
>> # synchronization phase can take
>> initLimit=10
>> # The number of ticks that can pass between
>> # sending a request and getting an acknowledgement
>> syncLimit=5
>> # the directory where the snapshot is stored.
>> # do not use /tmp for storage, /tmp here is just
>> # example sakes.
>> dataDir=c:/motum/config/MASS/ZK
>> # the port at which the clients will connect
>> clientPort=2181
>>
>> server.1=172.17.35.11:2888:3888
>> server.2=172.17.35.12:2888:3888
>> server.3=172.17.35.13:2888:3888
>> server.4=172.17.35.14:2888:3888
>> server.5=172.17.35.15:2888:3888
>> server.6=172.17.35.16:2888:3888
>> server.7=172.17.35.17:2888:3888
>> server.8=172.17.35.18:2888:3888
>>
>> # The number of snapshots to retain in dataDir
>> # Purge task interval in hours
>> # Set to "0" to disable auto purge feature
>> autopurge.snapRetainCount=3
>> autopurge.purgeInterval=1
>>
>>
>>
>>> On 24 Jun 2015, at 18:54, Raúl Gutiérrez Segalés <rgs@itevenworks.net
>> <javascript:;>> wrote:
>>>
>>> Hi,
>>>
>>> On 24 June 2015 at 06:05, Filip Deleersnijder <filip@motum.be <ma...@motum.be>
>> <javascript:;>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Let’s start with some description of our system :
>>>>
>>>> - We our using a Zookeeper cluster with 8 participants for an
>> application
>>>> with mobile nodes ( connected over Wifi ).
>>>>
>>>
>>> You mean the participants talk over wifi or the clients?
>>>
>>>
>>>> ( Ip of the different nodes are according to the following structure :
>>>> Node X has IP : 172.17.35.1X )
>>>>
>>>
>>> Why 8 and not an odd number of machines (i.e.:
>>>
>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup <http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup>
>>> )?
>>>
>>> - It is not that unusual to have a node being shut-down or restarted
>>>> - We haven’t benchmarked the number of write operations yet, but I would
>>>> estimate that it would be less than 10 writes / second
>>>>
>>>
>>> What version of ZK are you using?
>>>
>>>
>>>>
>>>> The problem we are having however is that sometimes(*), some instances
>>>> seem to be having problems with leader election.
>>>> Under the header “Attachment 1” below, you can find the leader election
>>>> times that were needed over 24h ( from 1 node ). One average it took
>> more
>>>> than 1 minute !
>>>> I assume that this is not normal behaviour ? ( If somebody could confirm
>>>> that in a 8-node cluster, these are not normal leader election times,
>> that
>>>> would be nice )
>>>>
>>>> In attachement 2 : I included an extract from the logging during a
>> leader
>>>> election that took 101874ms for 1 node ( server 2 ).
>>>>
>>>> Any help is greatly appreciated.
>>>> If further or more specific logging is required, please ask !
>>>>
>>>>
>>> Do you mind sharing a copy of your config file (zoo.cfg)? Thanks!
>>>
>>>
>>> -rgs
Re: Leader election problems
Posted by Guy Moshkowich <gu...@gmail.com>.
Are you using ZK client on your vehicles or ZK servers?
You mentioned below 8 vehicles and i see 8 servers defined in the config.
I would expect you have 8 client(running on your vehicles) communicating
against 1 or 3 ZK servers as this will be more than enough for 8 clients.
Guy
On Thursday, June 25, 2015, Filip Deleersnijder <fi...@motum.be> wrote:
> Hi,
>
> Thanks for your response.
>
> Our application consists of 8 automatic vehicles in a warehouse setting.
> Those vehicles need some consensus decisions, and that is what we use
> Zookeeper for.
> Because vehicles can come and go at random, we installed a ZK participant
> on every vehicle. The ZK client is some other piece of software that is
> also running on the vehicles.
>
> Therefor :
> - We can not choose the number of ZK-participants because it just
> depends on the number of vehicles.
> - The participants communicate over Wifi
> - The client is running on the same machine, so it communicates
> over the local network
>
> We are running Zookeeper version 3.4.6
>
> Our zoo.cfg can be found below this e-mail.
>
> Thanks in advance !
>
> Filip
>
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=c:/motum/config/MASS/ZK
> # the port at which the clients will connect
> clientPort=2181
>
> server.1=172.17.35.11:2888:3888
> server.2=172.17.35.12:2888:3888
> server.3=172.17.35.13:2888:3888
> server.4=172.17.35.14:2888:3888
> server.5=172.17.35.15:2888:3888
> server.6=172.17.35.16:2888:3888
> server.7=172.17.35.17:2888:3888
> server.8=172.17.35.18:2888:3888
>
> # The number of snapshots to retain in dataDir
> # Purge task interval in hours
> # Set to "0" to disable auto purge feature
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=1
>
>
>
> > On 24 Jun 2015, at 18:54, Raúl Gutiérrez Segalés <rgs@itevenworks.net
> <javascript:;>> wrote:
> >
> > Hi,
> >
> > On 24 June 2015 at 06:05, Filip Deleersnijder <filip@motum.be
> <javascript:;>> wrote:
> >
> >> Hi,
> >>
> >> Let’s start with some description of our system :
> >>
> >> - We our using a Zookeeper cluster with 8 participants for an
> application
> >> with mobile nodes ( connected over Wifi ).
> >>
> >
> > You mean the participants talk over wifi or the clients?
> >
> >
> >> ( Ip of the different nodes are according to the following structure :
> >> Node X has IP : 172.17.35.1X )
> >>
> >
> > Why 8 and not an odd number of machines (i.e.:
> >
> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
> > )?
> >
> > - It is not that unusual to have a node being shut-down or restarted
> >> - We haven’t benchmarked the number of write operations yet, but I would
> >> estimate that it would be less than 10 writes / second
> >>
> >
> > What version of ZK are you using?
> >
> >
> >>
> >> The problem we are having however is that sometimes(*), some instances
> >> seem to be having problems with leader election.
> >> Under the header “Attachment 1” below, you can find the leader election
> >> times that were needed over 24h ( from 1 node ). One average it took
> more
> >> than 1 minute !
> >> I assume that this is not normal behaviour ? ( If somebody could confirm
> >> that in a 8-node cluster, these are not normal leader election times,
> that
> >> would be nice )
> >>
> >> In attachement 2 : I included an extract from the logging during a
> leader
> >> election that took 101874ms for 1 node ( server 2 ).
> >>
> >> Any help is greatly appreciated.
> >> If further or more specific logging is required, please ask !
> >>
> >>
> > Do you mind sharing a copy of your config file (zoo.cfg)? Thanks!
> >
> >
> > -rgs
>
>
Re: Leader election problems
Posted by Filip Deleersnijder <fi...@motum.be>.
Hi,
Thanks for your response.
Our application consists of 8 automatic vehicles in a warehouse setting. Those vehicles need some consensus decisions, and that is what we use Zookeeper for.
Because vehicles can come and go at random, we installed a ZK participant on every vehicle. The ZK client is some other piece of software that is also running on the vehicles.
Therefor :
- We can not choose the number of ZK-participants because it just depends on the number of vehicles.
- The participants communicate over Wifi
- The client is running on the same machine, so it communicates over the local network
We are running Zookeeper version 3.4.6
Our zoo.cfg can be found below this e-mail.
Thanks in advance !
Filip
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=c:/motum/config/MASS/ZK
# the port at which the clients will connect
clientPort=2181
server.1=172.17.35.11:2888:3888
server.2=172.17.35.12:2888:3888
server.3=172.17.35.13:2888:3888
server.4=172.17.35.14:2888:3888
server.5=172.17.35.15:2888:3888
server.6=172.17.35.16:2888:3888
server.7=172.17.35.17:2888:3888
server.8=172.17.35.18:2888:3888
# The number of snapshots to retain in dataDir
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
> On 24 Jun 2015, at 18:54, Raúl Gutiérrez Segalés <rg...@itevenworks.net> wrote:
>
> Hi,
>
> On 24 June 2015 at 06:05, Filip Deleersnijder <fi...@motum.be> wrote:
>
>> Hi,
>>
>> Let’s start with some description of our system :
>>
>> - We our using a Zookeeper cluster with 8 participants for an application
>> with mobile nodes ( connected over Wifi ).
>>
>
> You mean the participants talk over wifi or the clients?
>
>
>> ( Ip of the different nodes are according to the following structure :
>> Node X has IP : 172.17.35.1X )
>>
>
> Why 8 and not an odd number of machines (i.e.:
> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
> )?
>
> - It is not that unusual to have a node being shut-down or restarted
>> - We haven’t benchmarked the number of write operations yet, but I would
>> estimate that it would be less than 10 writes / second
>>
>
> What version of ZK are you using?
>
>
>>
>> The problem we are having however is that sometimes(*), some instances
>> seem to be having problems with leader election.
>> Under the header “Attachment 1” below, you can find the leader election
>> times that were needed over 24h ( from 1 node ). One average it took more
>> than 1 minute !
>> I assume that this is not normal behaviour ? ( If somebody could confirm
>> that in a 8-node cluster, these are not normal leader election times, that
>> would be nice )
>>
>> In attachement 2 : I included an extract from the logging during a leader
>> election that took 101874ms for 1 node ( server 2 ).
>>
>> Any help is greatly appreciated.
>> If further or more specific logging is required, please ask !
>>
>>
> Do you mind sharing a copy of your config file (zoo.cfg)? Thanks!
>
>
> -rgs
Re: Leader election problems
Posted by Raúl Gutiérrez Segalés <rg...@itevenworks.net>.
Hi,
On 24 June 2015 at 06:05, Filip Deleersnijder <fi...@motum.be> wrote:
> Hi,
>
> Let’s start with some description of our system :
>
> - We our using a Zookeeper cluster with 8 participants for an application
> with mobile nodes ( connected over Wifi ).
>
You mean the participants talk over wifi or the clients?
> ( Ip of the different nodes are according to the following structure :
> Node X has IP : 172.17.35.1X )
>
Why 8 and not an odd number of machines (i.e.:
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
)?
- It is not that unusual to have a node being shut-down or restarted
> - We haven’t benchmarked the number of write operations yet, but I would
> estimate that it would be less than 10 writes / second
>
What version of ZK are you using?
>
> The problem we are having however is that sometimes(*), some instances
> seem to be having problems with leader election.
> Under the header “Attachment 1” below, you can find the leader election
> times that were needed over 24h ( from 1 node ). One average it took more
> than 1 minute !
> I assume that this is not normal behaviour ? ( If somebody could confirm
> that in a 8-node cluster, these are not normal leader election times, that
> would be nice )
>
> In attachement 2 : I included an extract from the logging during a leader
> election that took 101874ms for 1 node ( server 2 ).
>
> Any help is greatly appreciated.
> If further or more specific logging is required, please ask !
>
>
Do you mind sharing a copy of your config file (zoo.cfg)? Thanks!
-rgs