You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "liuli (JIRA)" <ji...@apache.org> on 2012/07/24 14:54:33 UTC
[jira] [Created] (ZOOKEEPER-1517) zookeeper follower closed
liuli created ZOOKEEPER-1517:
--------------------------------
Summary: zookeeper follower closed
Key: ZOOKEEPER-1517
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1517
Project: ZooKeeper
Issue Type: Bug
Components: quorum
Affects Versions: 3.3.5
Environment: zookeeper version 3.3.5
Hadoop version 0.20.205.0
Reporter: liuli
I have Hadoop and Zookeeper installed
the zoo.cfg is :
tickTime=2000
dataDir=/home/hduser/zookeeper/conf
clientPort=2181
initLimit=10
syncLimit=5
server.1=rsmm-master:2888:3888
server.2=rsmm-slave-1:2888:3888
server.3=rsmm-slave-2:2888:3888
server.4=rsmm-slave-3:2888:3888
server.5=rsmm-slave-4:2888:3888
=====================================
I tried to start zookeeper,
./zkServer.sh start
./zkServer.sh status
JMX enabled by default
Using config: /home/hduser/zookeeper/bin/../conf/zoo.cfg
Mode: follower
The follower (rsmm-slave-4) logs complain:
012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Thread.java:679)
2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
2012-07-24 20:30:37,768 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
2012-07-24 20:30:37,774 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
2012-07-24 20:30:37,792 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
2012-07-24 20:30:37,820 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@819] - tickTime set to 2000
2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@856] - initLimit set to 10
2012-07-24 20:30:37,863 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.0
2012-07-24 20:30:37,895 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
2012-07-24 20:30:37,909 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
2012-07-24 20:30:37,912 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 0
2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,925 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
2012-07-24 20:30:37,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,931 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,934 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,940 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:30:38,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
2012-07-24 20:30:38,150 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-4
2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
2012-07-24 20:30:38,161 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
2012-07-24 20:30:38,162 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
2012-07-24 20:30:38,175 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
2012-07-24 20:30:38,179 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
2012-07-24 20:30:38,180 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 200000000
2012-07-24 20:30:46,564 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41116
2012-07-24 20:30:46,569 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41116
2012-07-24 20:30:46,573 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41116 (no session established for client)
2012-07-24 20:33:27,407 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41118
2012-07-24 20:33:27,408 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41118
2012-07-24 20:33:27,411 - INFO [Thread-11:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41118 (no session established for client)
2012-07-24 20:47:21,659 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41126
2012-07-24 20:47:21,660 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41126
2012-07-24 20:47:21,663 - INFO [Thread-12:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41126 (no session established for client)
==================================
while the leader 's log shows
2012-07-24 20:22:33,769 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
2012-07-24 20:22:33,776 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
2012-07-24 20:22:33,795 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
2012-07-24 20:22:33,827 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@819] - tickTime set to 2000
2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@856] - initLimit set to 10
2012-07-24 20:22:33,874 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
2012-07-24 20:22:33,905 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
2012-07-24 20:22:33,923 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
2012-07-24 20:22:33,926 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
2012-07-24 20:22:33,935 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 2)
2012-07-24 20:22:33,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,936 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (4, 2)
2012-07-24 20:22:33,937 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (5, 2)
2012-07-24 20:22:33,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,943 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), LOOKING (my state)
2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), FOLLOWING (my state)
2012-07-24 20:22:33,946 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
2012-07-24 20:22:33,952 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
2012-07-24 20:22:33,959 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-1
2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
2012-07-24 20:22:33,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
2012-07-24 20:22:33,977 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@291] - Getting a diff from the leader 0x100000000
2012-07-24 20:22:33,981 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 1
2012-07-24 20:22:33,983 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
2012-07-24 20:22:40,102 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41400
2012-07-24 20:22:40,106 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41400
2012-07-24 20:22:40,109 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41400 (no session established for client)
2012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Thread.java:679)
2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
2012-07-24 20:49:19,788 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41403
2012-07-24 20:49:19,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41403
2012-07-24 20:49:19,791 - INFO [Thread-18:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41403 (no session established for client)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1517) zookeeper follower closed
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426816#comment-13426816 ]
Patrick Hunt commented on ZOOKEEPER-1517:
-----------------------------------------
Looks like your myid for slave 4 is wrong?
you mention it's rsmm-slave-4, with a config of
server.5=rsmm-slave-4:2888:3888
however in the logs you list it says:
2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
should be myid = 5, no?
> zookeeper follower closed
> -------------------------
>
> Key: ZOOKEEPER-1517
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1517
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
> Affects Versions: 3.3.5
> Environment: zookeeper version 3.3.5
> Hadoop version 0.20.205.0
> Reporter: liuli
>
> I have Hadoop and Zookeeper installed
> the zoo.cfg is :
> tickTime=2000
> dataDir=/home/hduser/zookeeper/conf
> clientPort=2181
> initLimit=10
> syncLimit=5
> server.1=rsmm-master:2888:3888
> server.2=rsmm-slave-1:2888:3888
> server.3=rsmm-slave-2:2888:3888
> server.4=rsmm-slave-3:2888:3888
> server.5=rsmm-slave-4:2888:3888
> =====================================
> I tried to start zookeeper,
> ./zkServer.sh start
> ./zkServer.sh status
> JMX enabled by default
> Using config: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> Mode: follower
> The follower (rsmm-slave-4) logs complain:
> 012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:30:37,768 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:30:37,774 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:30:37,792 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:30:37,820 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:30:37,863 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.0
> 2012-07-24 20:30:37,895 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:30:37,909 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:30:37,912 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 0
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,925 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
> 2012-07-24 20:30:37,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,931 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,934 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,940 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:38,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:30:38,150 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-4
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:30:38,161 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:30:38,162 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:30:38,175 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
> 2012-07-24 20:30:38,179 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
> 2012-07-24 20:30:38,180 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 200000000
> 2012-07-24 20:30:46,564 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41116
> 2012-07-24 20:30:46,569 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41116
> 2012-07-24 20:30:46,573 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41116 (no session established for client)
> 2012-07-24 20:33:27,407 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41118
> 2012-07-24 20:33:27,408 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41118
> 2012-07-24 20:33:27,411 - INFO [Thread-11:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41118 (no session established for client)
> 2012-07-24 20:47:21,659 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41126
> 2012-07-24 20:47:21,660 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41126
> 2012-07-24 20:47:21,663 - INFO [Thread-12:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41126 (no session established for client)
> ==================================
> while the leader 's log shows
> 2012-07-24 20:22:33,769 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:22:33,776 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:22:33,795 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:22:33,827 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:22:33,874 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:22:33,905 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:22:33,923 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:22:33,926 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:22:33,935 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 2)
> 2012-07-24 20:22:33,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,936 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2012-07-24 20:22:33,937 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (5, 2)
> 2012-07-24 20:22:33,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,943 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:22:33,946 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:22:33,952 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:22:33,959 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-1
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:22:33,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:22:33,977 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@291] - Getting a diff from the leader 0x100000000
> 2012-07-24 20:22:33,981 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 1
> 2012-07-24 20:22:33,983 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:22:40,102 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41400
> 2012-07-24 20:22:40,106 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41400
> 2012-07-24 20:22:40,109 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41400 (no session established for client)
> 2012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:49:19,788 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41403
> 2012-07-24 20:49:19,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41403
> 2012-07-24 20:49:19,791 - INFO [Thread-18:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41403 (no session established for client)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1517) zookeeper follower closed
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427771#comment-13427771 ]
Patrick Hunt commented on ZOOKEEPER-1517:
-----------------------------------------
No problem whatsoever liuli. Happy we could help. Good luck.
> zookeeper follower closed
> -------------------------
>
> Key: ZOOKEEPER-1517
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1517
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
> Affects Versions: 3.3.5
> Environment: zookeeper version 3.3.5
> Hadoop version 0.20.205.0
> Reporter: liuli
> Fix For: 3.3.5
>
>
> I have Hadoop and Zookeeper installed
> the zoo.cfg is :
> tickTime=2000
> dataDir=/home/hduser/zookeeper/conf
> clientPort=2181
> initLimit=10
> syncLimit=5
> server.1=rsmm-master:2888:3888
> server.2=rsmm-slave-1:2888:3888
> server.3=rsmm-slave-2:2888:3888
> server.4=rsmm-slave-3:2888:3888
> server.5=rsmm-slave-4:2888:3888
> =====================================
> I tried to start zookeeper,
> ./zkServer.sh start
> ./zkServer.sh status
> JMX enabled by default
> Using config: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> Mode: follower
> The follower (rsmm-slave-4) logs complain:
> 012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:30:37,768 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:30:37,774 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:30:37,792 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:30:37,820 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:30:37,863 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.0
> 2012-07-24 20:30:37,895 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:30:37,909 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:30:37,912 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 0
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,925 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
> 2012-07-24 20:30:37,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,931 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,934 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,940 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:38,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:30:38,150 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-4
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:30:38,161 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:30:38,162 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:30:38,175 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
> 2012-07-24 20:30:38,179 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
> 2012-07-24 20:30:38,180 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 200000000
> 2012-07-24 20:30:46,564 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41116
> 2012-07-24 20:30:46,569 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41116
> 2012-07-24 20:30:46,573 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41116 (no session established for client)
> 2012-07-24 20:33:27,407 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41118
> 2012-07-24 20:33:27,408 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41118
> 2012-07-24 20:33:27,411 - INFO [Thread-11:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41118 (no session established for client)
> 2012-07-24 20:47:21,659 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41126
> 2012-07-24 20:47:21,660 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41126
> 2012-07-24 20:47:21,663 - INFO [Thread-12:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41126 (no session established for client)
> ==================================
> while the leader 's log shows
> 2012-07-24 20:22:33,769 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:22:33,776 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:22:33,795 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:22:33,827 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:22:33,874 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:22:33,905 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:22:33,923 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:22:33,926 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:22:33,935 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 2)
> 2012-07-24 20:22:33,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,936 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2012-07-24 20:22:33,937 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (5, 2)
> 2012-07-24 20:22:33,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,943 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:22:33,946 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:22:33,952 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:22:33,959 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-1
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:22:33,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:22:33,977 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@291] - Getting a diff from the leader 0x100000000
> 2012-07-24 20:22:33,981 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 1
> 2012-07-24 20:22:33,983 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:22:40,102 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41400
> 2012-07-24 20:22:40,106 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41400
> 2012-07-24 20:22:40,109 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41400 (no session established for client)
> 2012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:49:19,788 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41403
> 2012-07-24 20:49:19,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41403
> 2012-07-24 20:49:19,791 - INFO [Thread-18:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41403 (no session established for client)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (ZOOKEEPER-1517) zookeeper follower closed
Posted by "liuli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liuli resolved ZOOKEEPER-1517.
------------------------------
Resolution: Invalid
Fix Version/s: 3.3.5
Release Note: I will close this one since zookeeper start normally
> zookeeper follower closed
> -------------------------
>
> Key: ZOOKEEPER-1517
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1517
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
> Affects Versions: 3.3.5
> Environment: zookeeper version 3.3.5
> Hadoop version 0.20.205.0
> Reporter: liuli
> Fix For: 3.3.5
>
>
> I have Hadoop and Zookeeper installed
> the zoo.cfg is :
> tickTime=2000
> dataDir=/home/hduser/zookeeper/conf
> clientPort=2181
> initLimit=10
> syncLimit=5
> server.1=rsmm-master:2888:3888
> server.2=rsmm-slave-1:2888:3888
> server.3=rsmm-slave-2:2888:3888
> server.4=rsmm-slave-3:2888:3888
> server.5=rsmm-slave-4:2888:3888
> =====================================
> I tried to start zookeeper,
> ./zkServer.sh start
> ./zkServer.sh status
> JMX enabled by default
> Using config: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> Mode: follower
> The follower (rsmm-slave-4) logs complain:
> 012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:30:37,768 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:30:37,774 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:30:37,792 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:30:37,820 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:30:37,863 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.0
> 2012-07-24 20:30:37,895 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:30:37,909 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:30:37,912 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 0
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,925 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
> 2012-07-24 20:30:37,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,931 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,934 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,940 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:38,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:30:38,150 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-4
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:30:38,161 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:30:38,162 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:30:38,175 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
> 2012-07-24 20:30:38,179 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
> 2012-07-24 20:30:38,180 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 200000000
> 2012-07-24 20:30:46,564 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41116
> 2012-07-24 20:30:46,569 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41116
> 2012-07-24 20:30:46,573 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41116 (no session established for client)
> 2012-07-24 20:33:27,407 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41118
> 2012-07-24 20:33:27,408 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41118
> 2012-07-24 20:33:27,411 - INFO [Thread-11:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41118 (no session established for client)
> 2012-07-24 20:47:21,659 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41126
> 2012-07-24 20:47:21,660 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41126
> 2012-07-24 20:47:21,663 - INFO [Thread-12:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41126 (no session established for client)
> ==================================
> while the leader 's log shows
> 2012-07-24 20:22:33,769 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:22:33,776 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:22:33,795 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:22:33,827 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:22:33,874 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:22:33,905 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:22:33,923 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:22:33,926 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:22:33,935 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 2)
> 2012-07-24 20:22:33,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,936 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2012-07-24 20:22:33,937 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (5, 2)
> 2012-07-24 20:22:33,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,943 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:22:33,946 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:22:33,952 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:22:33,959 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-1
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:22:33,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:22:33,977 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@291] - Getting a diff from the leader 0x100000000
> 2012-07-24 20:22:33,981 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 1
> 2012-07-24 20:22:33,983 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:22:40,102 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41400
> 2012-07-24 20:22:40,106 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41400
> 2012-07-24 20:22:40,109 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41400 (no session established for client)
> 2012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:49:19,788 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41403
> 2012-07-24 20:49:19,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41403
> 2012-07-24 20:49:19,791 - INFO [Thread-18:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41403 (no session established for client)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1517) zookeeper follower closed
Posted by "liuli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427189#comment-13427189 ]
liuli commented on ZOOKEEPER-1517:
----------------------------------
In the last log, yes.
But even the My id is correct, zookeeper is stil complaining:
2028-12-12 13:16:06,385 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
2028-12-12 13:16:06,392 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
2028-12-12 13:16:06,412 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
2028-12-12 13:16:06,443 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
2028-12-12 13:16:06,468 - INFO [main:QuorumPeer@819] - tickTime set to 2000
2028-12-12 13:16:06,468 - INFO [main:QuorumPeer@830] - minSessionTimeout set to 10000
2028-12-12 13:16:06,469 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to 20000
2028-12-12 13:16:06,469 - INFO [main:QuorumPeer@856] - initLimit set to 10
2028-12-12 13:16:06,497 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
2028-12-12 13:16:06,529 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
2028-12-12 13:16:06,531 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 1, Proposed zxid = 0
2028-12-12 13:16:06,534 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2028-12-12 13:16:06,539 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 2 at election address rsmm-slave-1.example.com/109.123.121.24:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Unknown Source)
2028-12-12 13:16:06,540 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 3 at election address rsmm-slave-2.example.com/109.123.121.25:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Unknown Source)
2028-12-12 13:16:06,541 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 4 at election address rsmm-slave-3.example.com/109.123.121.26:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Unknown Source)
2028-12-12 13:16:06,543 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4.example.com/109.123.121.27:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Unknown Source)
2028-12-12 13:16:06,737 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4.example.com/109.123.121.27:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:06,737 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address rsmm-slave-1.example.com/109.123.121.24:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:06,738 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address rsmm-slave-2.example.com/109.123.121.25:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:06,739 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 4 at election address rsmm-slave-3.example.com/109.123.121.26:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:06,740 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 400
2028-12-12 13:16:07,141 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4.example.com/109.123.121.27:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,142 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address rsmm-slave-1.example.com/109.123.121.24:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,143 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address rsmm-slave-2.example.com/109.123.121.25:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,144 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 4 at election address rsmm-slave-3.example.com/109.123.121.26:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,144 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 800
2028-12-12 13:16:07,945 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4.example.com/109.123.121.27:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,946 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address rsmm-slave-1.example.com/109.123.121.24:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,948 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address rsmm-slave-2.example.com/109.123.121.25:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,948 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 4 at election address rsmm-slave-3.example.com/109.123.121.26:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)
2028-12-12 13:16:07,949 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 1600
2028-12-12 13:16:08,655 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2028-12-12 13:16:08,655 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
2028-12-12 13:16:08,656 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2028-12-12 13:16:08,656 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 2 at election address rsmm-slave-1.example.com/109.123.121.24:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Unknown Source)
2028-12-12 13:16:08,657 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 4 at election address rsmm-slave-3.example.com/109.123.121.26:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Unknown Source)
2028-12-12 13:16:08,658 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4.example.com/109.123.121.27:3888
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
at java.lang.Thread.run(Unknown Source)
2028-12-12 13:16:08,862 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2028-12-12 13:16:09,264 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
2028-12-12 13:16:09,988 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
2028-12-12 13:16:09,997 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
2028-12-12 13:16:10,198 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
2028-12-12 13:16:10,205 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
2028-12-12 13:16:10,213 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
2028-12-12 13:16:10,214 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-master.example.com
2028-12-12 13:16:10,214 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_33
2028-12-12 13:16:10,214 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
2028-12-12 13:16:10,215 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/java/jre1.6.0_33
2028-12-12 13:16:10,215 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf::/usr/java/jre.6.0_33/lib
2028-12-12 13:16:10,215 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/java/jre1.6.0_33/lib/i386/client:/usr/java/jre1.6.0_33/lib/i386:/usr/java/jre1.6.0_33/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
2028-12-12 13:16:10,215 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
2028-12-12 13:16:10,216 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
2028-12-12 13:16:10,216 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
2028-12-12 13:16:10,216 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
2028-12-12 13:16:10,216 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-24-generic
2028-12-12 13:16:10,217 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
2028-12-12 13:16:10,217 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
2028-12-12 13:16:10,217 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/hbase/bin
2028-12-12 13:16:10,219 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 10000 maxSessionTimeout 20000 datadir /home/hduser/zookeeper/logs/version-2 snapdir /home/hduser/zookeeper/conf/version-2
2028-12-12 13:16:10,272 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
2028-12-12 13:16:10,284 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
2028-12-12 13:16:10,284 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 1000000f9
2028-12-12 13:16:11,408 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), FOLLOWING (my state)
2028-12-12 13:16:11,414 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), FOLLOWING (my state)
2028-12-12 13:16:11,418 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), FOLLOWING (my state)
2028-12-12 13:16:11,427 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), FOLLOWING (my state)
2028-12-12 13:16:13,338 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), FOLLOWING (my state)
2028-12-12 13:16:13,346 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967545 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), FOLLOWING (my state)
2028-12-12 13:16:50,876 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@116] - Got zxid 0x200000001 expected 0x1
2028-12-12 13:16:50,876 - INFO [SyncThread:1:FileTxnLog@199] - Creating new log file: log.200000001
2028-12-12 13:18:22,480 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:51342
2028-12-12 13:18:22,484 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:51342
2028-12-12 13:18:22,488 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:51342 (no session established for client)
2028-12-12 13:20:44,203 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /109.123.121.23:60926
2028-12-12 13:20:44,206 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /109.123.121.23:60926
2028-12-12 13:20:44,232 - INFO [CommitProcessor:1:NIOServerCnxn@1580] - Established session 0xffb11d36ceac0000 with negotiated timeout 20000 for client /109.123.121.23:60926
2028-12-12 13:20:46,325 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /109.123.121.24:59071
2028-12-12 13:20:46,326 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /109.123.121.24:59071
2028-12-12 13:20:46,336 - INFO [CommitProcessor:1:NIOServerCnxn@1580] - Established session 0xffb11d36ceac0001 with negotiated timeout 20000 for client /109.123.121.24:59071
2028-12-12 13:23:31,249 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /109.123.121.24:59071 which had sessionid 0xffb11d36ceac0001
2028-12-12 13:23:31,405 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /109.123.121.23:60926 which had sessionid 0xffb11d36ceac0000
2028-12-12 13:24:28,985 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /109.123.121.23:60987
2028-12-12 13:24:28,993 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /109.123.121.23:60987
2028-12-12 13:24:29,005 - INFO [CommitProcessor:1:NIOServerCnxn@1580] - Established session 0xffb11d36ceac0002 with negotiated timeout 20000 for client /109.123.121.23:60987
2028-12-12 13:24:29,714 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - EndOfStreamException: Unable to read additional data from client sessionid 0xffb11d36ceac0002, likely client has closed socket
2028-12-12 13:24:29,716 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /109.123.121.23:60987 which had sessionid 0xffb11d36ceac0002
> zookeeper follower closed
> -------------------------
>
> Key: ZOOKEEPER-1517
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1517
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
> Affects Versions: 3.3.5
> Environment: zookeeper version 3.3.5
> Hadoop version 0.20.205.0
> Reporter: liuli
>
> I have Hadoop and Zookeeper installed
> the zoo.cfg is :
> tickTime=2000
> dataDir=/home/hduser/zookeeper/conf
> clientPort=2181
> initLimit=10
> syncLimit=5
> server.1=rsmm-master:2888:3888
> server.2=rsmm-slave-1:2888:3888
> server.3=rsmm-slave-2:2888:3888
> server.4=rsmm-slave-3:2888:3888
> server.5=rsmm-slave-4:2888:3888
> =====================================
> I tried to start zookeeper,
> ./zkServer.sh start
> ./zkServer.sh status
> JMX enabled by default
> Using config: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> Mode: follower
> The follower (rsmm-slave-4) logs complain:
> 012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:30:37,768 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:30:37,774 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:30:37,792 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:30:37,820 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:30:37,863 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.0
> 2012-07-24 20:30:37,895 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:30:37,909 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:30:37,912 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 0
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,925 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
> 2012-07-24 20:30:37,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,931 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,934 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,940 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:38,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:30:38,150 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-4
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:30:38,161 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:30:38,162 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:30:38,175 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
> 2012-07-24 20:30:38,179 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
> 2012-07-24 20:30:38,180 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 200000000
> 2012-07-24 20:30:46,564 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41116
> 2012-07-24 20:30:46,569 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41116
> 2012-07-24 20:30:46,573 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41116 (no session established for client)
> 2012-07-24 20:33:27,407 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41118
> 2012-07-24 20:33:27,408 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41118
> 2012-07-24 20:33:27,411 - INFO [Thread-11:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41118 (no session established for client)
> 2012-07-24 20:47:21,659 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41126
> 2012-07-24 20:47:21,660 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41126
> 2012-07-24 20:47:21,663 - INFO [Thread-12:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41126 (no session established for client)
> ==================================
> while the leader 's log shows
> 2012-07-24 20:22:33,769 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:22:33,776 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:22:33,795 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:22:33,827 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:22:33,874 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:22:33,905 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:22:33,923 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:22:33,926 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:22:33,935 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 2)
> 2012-07-24 20:22:33,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,936 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2012-07-24 20:22:33,937 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (5, 2)
> 2012-07-24 20:22:33,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,943 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:22:33,946 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:22:33,952 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:22:33,959 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-1
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:22:33,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:22:33,977 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@291] - Getting a diff from the leader 0x100000000
> 2012-07-24 20:22:33,981 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 1
> 2012-07-24 20:22:33,983 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:22:40,102 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41400
> 2012-07-24 20:22:40,106 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41400
> 2012-07-24 20:22:40,109 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41400 (no session established for client)
> 2012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:49:19,788 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41403
> 2012-07-24 20:49:19,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41403
> 2012-07-24 20:49:19,791 - INFO [Thread-18:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41403 (no session established for client)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1517) zookeeper follower closed
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427618#comment-13427618 ]
Patrick Hunt commented on ZOOKEEPER-1517:
-----------------------------------------
I'm not sure I see the problem (in the latest log you attached). At 13:16:50 the server joins the quorum successfully and starts serving.
> zookeeper follower closed
> -------------------------
>
> Key: ZOOKEEPER-1517
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1517
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
> Affects Versions: 3.3.5
> Environment: zookeeper version 3.3.5
> Hadoop version 0.20.205.0
> Reporter: liuli
>
> I have Hadoop and Zookeeper installed
> the zoo.cfg is :
> tickTime=2000
> dataDir=/home/hduser/zookeeper/conf
> clientPort=2181
> initLimit=10
> syncLimit=5
> server.1=rsmm-master:2888:3888
> server.2=rsmm-slave-1:2888:3888
> server.3=rsmm-slave-2:2888:3888
> server.4=rsmm-slave-3:2888:3888
> server.5=rsmm-slave-4:2888:3888
> =====================================
> I tried to start zookeeper,
> ./zkServer.sh start
> ./zkServer.sh status
> JMX enabled by default
> Using config: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> Mode: follower
> The follower (rsmm-slave-4) logs complain:
> 012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:30:37,768 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:30:37,774 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:30:37,792 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:30:37,820 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:30:37,863 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.0
> 2012-07-24 20:30:37,895 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:30:37,909 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:30:37,912 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 0
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,925 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
> 2012-07-24 20:30:37,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,931 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,934 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,940 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:38,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:30:38,150 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-4
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:30:38,161 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:30:38,162 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:30:38,175 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
> 2012-07-24 20:30:38,179 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
> 2012-07-24 20:30:38,180 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 200000000
> 2012-07-24 20:30:46,564 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41116
> 2012-07-24 20:30:46,569 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41116
> 2012-07-24 20:30:46,573 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41116 (no session established for client)
> 2012-07-24 20:33:27,407 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41118
> 2012-07-24 20:33:27,408 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41118
> 2012-07-24 20:33:27,411 - INFO [Thread-11:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41118 (no session established for client)
> 2012-07-24 20:47:21,659 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41126
> 2012-07-24 20:47:21,660 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41126
> 2012-07-24 20:47:21,663 - INFO [Thread-12:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41126 (no session established for client)
> ==================================
> while the leader 's log shows
> 2012-07-24 20:22:33,769 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:22:33,776 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:22:33,795 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:22:33,827 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:22:33,874 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:22:33,905 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:22:33,923 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:22:33,926 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:22:33,935 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 2)
> 2012-07-24 20:22:33,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,936 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2012-07-24 20:22:33,937 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (5, 2)
> 2012-07-24 20:22:33,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,943 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:22:33,946 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:22:33,952 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:22:33,959 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-1
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:22:33,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:22:33,977 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@291] - Getting a diff from the leader 0x100000000
> 2012-07-24 20:22:33,981 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 1
> 2012-07-24 20:22:33,983 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:22:40,102 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41400
> 2012-07-24 20:22:40,106 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41400
> 2012-07-24 20:22:40,109 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41400 (no session established for client)
> 2012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:49:19,788 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41403
> 2012-07-24 20:49:19,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41403
> 2012-07-24 20:49:19,791 - INFO [Thread-18:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41403 (no session established for client)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1517) zookeeper follower closed
Posted by "liuli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427762#comment-13427762 ]
liuli commented on ZOOKEEPER-1517:
----------------------------------
Mr.Hunt,Thanks for your apply.
I'm a dummy and not so understand the log flow (which means right, which means wrong)
Your answer is a great relief to me.
> zookeeper follower closed
> -------------------------
>
> Key: ZOOKEEPER-1517
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1517
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
> Affects Versions: 3.3.5
> Environment: zookeeper version 3.3.5
> Hadoop version 0.20.205.0
> Reporter: liuli
>
> I have Hadoop and Zookeeper installed
> the zoo.cfg is :
> tickTime=2000
> dataDir=/home/hduser/zookeeper/conf
> clientPort=2181
> initLimit=10
> syncLimit=5
> server.1=rsmm-master:2888:3888
> server.2=rsmm-slave-1:2888:3888
> server.3=rsmm-slave-2:2888:3888
> server.4=rsmm-slave-3:2888:3888
> server.5=rsmm-slave-4:2888:3888
> =====================================
> I tried to start zookeeper,
> ./zkServer.sh start
> ./zkServer.sh status
> JMX enabled by default
> Using config: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> Mode: follower
> The follower (rsmm-slave-4) logs complain:
> 012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:30:37,768 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:30:37,774 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:30:37,792 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:30:37,820 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:30:37,845 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:30:37,846 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:30:37,863 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.0
> 2012-07-24 20:30:37,895 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:30:37,909 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:30:37,912 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 0
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,923 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,924 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,925 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal
> 2012-07-24 20:30:37,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,929 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,931 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,933 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,934 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,937 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LEADING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,940 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:37,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:30:38,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:30:38,150 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:30:38,157 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-4
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:30:38,158 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:30:38,159 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:30:38,160 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:30:38,161 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:30:38,162 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:30:38,175 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader
> 2012-07-24 20:30:38,179 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 2
> 2012-07-24 20:30:38,180 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 200000000
> 2012-07-24 20:30:46,564 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41116
> 2012-07-24 20:30:46,569 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41116
> 2012-07-24 20:30:46,573 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41116 (no session established for client)
> 2012-07-24 20:33:27,407 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41118
> 2012-07-24 20:33:27,408 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41118
> 2012-07-24 20:33:27,411 - INFO [Thread-11:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41118 (no session established for client)
> 2012-07-24 20:47:21,659 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41126
> 2012-07-24 20:47:21,660 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41126
> 2012-07-24 20:47:21,663 - INFO [Thread-12:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41126 (no session established for client)
> ==================================
> while the leader 's log shows
> 2012-07-24 20:22:33,769 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /home/hduser/zookeeper/bin/../conf/zoo.cfg
> 2012-07-24 20:22:33,776 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums
> 2012-07-24 20:22:33,795 - INFO [main:QuorumPeerMain@119] - Starting quorum peer
> 2012-07-24 20:22:33,827 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-07-24 20:22:33,854 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1
> 2012-07-24 20:22:33,855 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-07-24 20:22:33,874 - INFO [main:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:22:33,905 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888
> 2012-07-24 20:22:33,923 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:22:33,926 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:22:33,935 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 2)
> 2012-07-24 20:22:33,935 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,936 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2012-07-24 20:22:33,937 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (5, 2)
> 2012-07-24 20:22:33,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,941 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,942 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,943 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), FOLLOWING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), LOOKING (my state)
> 2012-07-24 20:22:33,945 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:22:33,946 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING
> 2012-07-24 20:22:33,952 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true
> 2012-07-24 20:22:33,959 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.5-1301095, built on 03/15/2012 19:48 GMT
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=rsmm-slave-1
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_23
> 2012-07-24 20:22:33,960 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hduser/zookeeper/bin/../build/classes:/home/hduser/zookeeper/bin/../build/lib/*.jar:/home/hduser/zookeeper/bin/../zookeeper-3.3.5.jar:/home/hduser/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hduser/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hduser/zookeeper/bin/../src/java/lib/*.jar:/home/hduser/zookeeper/bin/../conf:
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2012-07-24 20:22:33,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA>
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-12-generic
> 2012-07-24 20:22:33,962 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hduser
> 2012-07-24 20:22:33,963 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hduser/zookeeper/bin
> 2012-07-24 20:22:33,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:22:33,977 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@291] - Getting a diff from the leader 0x100000000
> 2012-07-24 20:22:33,981 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@326] - Setting leader epoch 1
> 2012-07-24 20:22:33,983 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:22:40,102 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41400
> 2012-07-24 20:22:40,106 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41400
> 2012-07-24 20:22:40,109 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41400 (no session established for client)
> 2012-07-24 20:29:35,903 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 2, error = java.io.IOException: Channel eof
> 2012-07-24 20:29:35,904 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker
> 2012-07-24 20:29:35,905 - WARN [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@82] - Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:148)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)
> 2012-07-24 20:29:35,905 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622)
> 2012-07-24 20:29:35,907 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread
> 2012-07-24 20:29:35,907 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Follower@165] - shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:165)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:649)
> 2012-07-24 20:29:35,913 - INFO [FollowerRequestProcessor:2:FollowerRequestProcessor@93] - FollowerRequestProcessor exited loop!
> 2012-07-24 20:29:35,914 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FinalRequestProcessor@370] - shutdown of request processor complete
> 2012-07-24 20:29:35,914 - INFO [CommitProcessor:2:CommitProcessor@148] - CommitProcessor exited loop!
> 2012-07-24 20:29:35,915 - INFO [SyncThread:2:SyncRequestProcessor@151] - SyncRequestProcessor exited!
> 2012-07-24 20:29:35,916 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), FOLLOWING (my state)
> 2012-07-24 20:29:35,916 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING
> 2012-07-24 20:29:35,918 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:35,919 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 2, Proposed zxid = 4294967296
> 2012-07-24 20:29:35,919 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,920 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address rsmm-slave-4/109.123.121.27:3888
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)
> at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)
> at java.lang.Thread.run(Thread.java:679)
> 2012-07-24 20:29:35,920 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,922 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,926 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 4 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,928 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 4 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,932 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:35,936 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), LOOKING (my state)
> 2012-07-24 20:29:36,137 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING
> 2012-07-24 20:29:36,141 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true
> 2012-07-24 20:29:36,143 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hduser/zookeeper/conf/version-2 snapdir /home/hduser/zookeeper/conf/version-2
> 2012-07-24 20:29:36,147 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileSnap@82] - Reading snapshot /home/hduser/zookeeper/conf/version-2/snapshot.100000000
> 2012-07-24 20:29:36,148 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 100000000
> 2012-07-24 20:29:37,149 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@249] - Follower sid: 4 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1c74f37
> 2012-07-24 20:29:37,150 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@273] - Synchronizing with Follower sid: 4 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,151 - INFO [LearnerHandler-/109.123.121.26:34087:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,152 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@249] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@a17083
> 2012-07-24 20:29:37,153 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@273] - Synchronizing with Follower sid: 1 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 100000000
> 2012-07-24 20:29:37,154 - INFO [LearnerHandler-/109.123.121.23:41659:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x100000000 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@249] - Follower sid: 3 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@16fe0f4
> 2012-07-24 20:29:37,156 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@273] - Synchronizing with Follower sid: 3 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:29:37,157 - INFO [LearnerHandler-/109.123.121.25:54707:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x100000000
> 2012-07-24 20:29:37,159 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@492] - Commiting zxid 0x200000000 from /109.123.121.24:2888 not first!
> 2012-07-24 20:29:37,160 - WARN [LearnerHandler-/109.123.121.26:34087:Leader@494] - First is 0
> 2012-07-24 20:29:37,172 - INFO [LearnerHandler-/109.123.121.26:34087:Leader@518] - Have quorum of supporters; starting up and setting last processed zxid: 8589934592
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,397 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,398 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,400 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 4294967296 (n.zxid), 2 (n.round), LOOKING (n.state), 5 (n.sid), LEADING (my state)
> 2012-07-24 20:30:40,641 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@249] - Follower sid: 5 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@15663a2
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@273] - Synchronizing with Follower sid: 5 maxCommittedLog =0 minCommittedLog = 0 peerLastZxid = 0
> 2012-07-24 20:30:40,642 - INFO [LearnerHandler-/109.123.121.27:34526:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x0 zxid of leader is 0x200000000sent zxid of db as 0x200000000
> 2012-07-24 20:49:19,788 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:41403
> 2012-07-24 20:49:19,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:41403
> 2012-07-24 20:49:19,791 - INFO [Thread-18:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:41403 (no session established for client)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira