You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Preeti Bhat <pr...@shoregrp.com> on 2016/10/26 13:24:13 UTC

Zookeeper fails to connect in cluster while using DNS 

Hi All,

I am getting the below messages while trying to form the zookeeper cluster in zookeeper.out file. The zookeeper is setup in AWS EC2 RHEL linux servers. The configuration works when we are trying to Public DNS of AWS, but when trying to use the specific DNS created for these instances using RouteS3 we are getting the below error.
I have tried stopping the servers, clearing out version-2 folder and restarting with no result.
The DNS for the specific server is added to /etc/hosts file as well.
Could someone please advise on this.


2016-10-26 09:03:09,991 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /root/zookeeper-3.4.8/bin/../conf/zoo.cfg
2016-10-26 09:03:10,054 [myid:] - INFO  [main:QuorumPeerConfig@331] - Defaulting to majority quorums
2016-10-26 09:03:10,057 [myid:2] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-10-26 09:03:10,057 [myid:2] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-10-26 09:03:10,057 [myid:2] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-10-26 09:03:10,067 [myid:2] - INFO  [main:QuorumPeerMain@127] - Starting quorum peer
2016-10-26 09:03:10,078 [myid:2] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:80
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1019] - tickTime set to 40000
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1039] - minSessionTimeout set to 120000
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1050] - maxSessionTimeout set to 240000
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1065] - initLimit set to 10
2016-10-26 09:03:10,099 [myid:2] - INFO  [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: xxx.com/127.0.0.1:3888
2016-10-26 09:03:10,108 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:80:QuorumPeer@774] - LOOKING
2016-10-26 09:03:10,109 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:80:FastLeaderElection@818] - New election. My id =  2, proposed zxid=0x0
2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 2)
2016-10-26 09:03:10,117 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:10,117 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
2016-10-26 09:03:10,118 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:10,120 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 2)
2016-10-26 09:03:10,318 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:80:FastLeaderElection@852] - Notification time out: 400
2016-10-26 09:03:10,320 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:10,321 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 2)
2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 2)
2016-10-26 09:03:10,720 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:80:FastLeaderElection@852] - Notification time out: 800
2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:10,722 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:10,723 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 2)
2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 2)
2016-10-26 09:03:11,522 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:80:FastLeaderElection@852] - Notification time out: 1600
2016-10-26 09:03:11,524 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:11,524 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:11,525 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:11,525 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 2)
2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:11,526 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
2016-10-26 09:03:11,527 [myid:2] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 2)
2016-10-26 09:03:11,555 [myid:2] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:80:NIOServerCnxnFactory@192] - Accepted socket connection from /172.31.18.151:36082
2016-10-26 09:03:11,609 [myid:2] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:80:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:745)


Thanks and Regards,
Preeti Bhat



NOTICE TO RECIPIENTS: This communication may contain confidential and/or privileged information. If you are not the intended recipient (or have received this communication in error) please notify the sender and it-support@shoregrp.com immediately, and destroy this communication. Any unauthorized copying, disclosure or distribution of the material in this communication is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

Re: Zookeeper fails to connect in cluster while using DNS

Posted by Will Martin <wm...@outlook.com>.
server side. doh. make sure your bind of zkserver to port 80 is working. (really? port 80?)


# netstat -npl


On 10/28/2016 4:13 AM, Preeti Bhat wrote:

Hi Michael,

The client side log is as below. For Route S3, we have associated single IP to single DNS.

java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:181)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:115)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:105)
        at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:188)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
        at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:235)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:173)
        ... 3 more

Thanks and Regards,
Preeti Bhat

-----Original Message-----
From: Michael Han [mailto:hanm@cloudera.com]
Sent: Thursday, October 27, 2016 9:51 AM
To: UserZooKeeper
Subject: Re: Zookeeper fails to connect in cluster while using DNS

This looks like server logs - since the problem is ZK client fail to connect to server, could you also post client logs?

For route 53, if you associate multiple IP addresses to a single DNS name configured in ZK ensemble and for some reasons one of the IP address does not have ZK server process running, it could lead to client fail to connect, but I am not sure if that is your case.

On Wed, Oct 26, 2016 at 6:24 AM, Preeti Bhat <pr...@shoregrp.com>
wrote:



Hi All,

I am getting the below messages while trying to form the zookeeper
cluster in zookeeper.out file. The zookeeper is setup in AWS EC2 RHEL
linux servers. The configuration works when we are trying to Public
DNS of AWS, but when trying to use the specific DNS created for these
instances using
RouteS3 we are getting the below error.
I have tried stopping the servers, clearing out version-2 folder and
restarting with no result.
The DNS for the specific server is added to /etc/hosts file as well.
Could someone please advise on this.


2016-10-26 09:03:09,991 [myid:] - INFO  [main:QuorumPeerConfig@103] -
Reading configuration from: /root/zookeeper-3.4.8/bin/../conf/zoo.cfg
2016-10-26 09:03:10,054 [myid:] - INFO  [main:QuorumPeerConfig@331] -
Defaulting to majority quorums
2016-10-26 09:03:10,057 [myid:2] - INFO
[main:DatadirCleanupManager@78]
- autopurge.snapRetainCount set to 3
2016-10-26 09:03:10,057 [myid:2] - INFO
[main:DatadirCleanupManager@79]
- autopurge.purgeInterval set to 0
2016-10-26 09:03:10,057 [myid:2] - INFO
[main:DatadirCleanupManager@101]
- Purge task is not scheduled.
2016-10-26 09:03:10,067 [myid:2] - INFO  [main:QuorumPeerMain@127] -
Starting quorum peer
2016-10-26 09:03:10,078 [myid:2] - INFO
[main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:80
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1019] -
tickTime set to 40000
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1039] -
minSessionTimeout set to 120000
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1050] -
maxSessionTimeout set to 240000
2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1065] -
initLimit set to 10
2016-10-26 09:03:10,099 [myid:2] - INFO  [ListenerThread:
QuorumCnxManager$Listener@534] - My election bind port:
xxx.com/127.0.0.1:3888
2016-10-26 09:03:10,108 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
0:0:0:80:QuorumPeer@774] - LOOKING
2016-10-26 09:03:10,109 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
0:0:0:80:FastLeaderElection@818] - New election. My id =  2, proposed
zxid=0x0
2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerReceiver[myid=2]:
FastLeaderElection@600] - Notification: 1 (message format version), 2
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
0x0
(n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (3, 2)
2016-10-26 09:03:10,117 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
2, error = java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:10,117 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (4, 2)
2016-10-26 09:03:10,118 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
message on queue java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(
ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:10,120 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (5, 2)
2016-10-26 09:03:10,318 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
0:0:0:80:FastLeaderElection@852] - Notification time out: 400
2016-10-26 09:03:10,320 [myid:2] - INFO  [WorkerReceiver[myid=2]:
FastLeaderElection@600] - Notification: 1 (message format version), 2
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
0x0
(n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:10,321 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (3, 2)
2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
2, error = java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
message on queue java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(
ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (4, 2)
2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (5, 2)
2016-10-26 09:03:10,720 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
0:0:0:80:FastLeaderElection@852] - Notification time out: 800
2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerReceiver[myid=2]:
FastLeaderElection@600] - Notification: 1 (message format version), 2
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
0x0
(n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:10,722 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
2, error = java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:10,723 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (3, 2)
2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
message on queue java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(
ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (4, 2)
2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (5, 2)
2016-10-26 09:03:11,522 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
0:0:0:80:FastLeaderElection@852] - Notification time out: 1600
2016-10-26 09:03:11,524 [myid:2] - INFO  [WorkerReceiver[myid=2]:
FastLeaderElection@600] - Notification: 1 (message format version), 2
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
0x0
(n.peerEpoch) LOOKING (my state)
2016-10-26 09:03:11,524 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
2, error = java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
RecvWorker.run(QuorumCnxManager.java:795)
2016-10-26 09:03:11,525 [myid:2] - WARN  [RecvWorker:1:
QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
2016-10-26 09:03:11,525 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (3, 2)
2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
message on queue java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
java:2014)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
        at java.util.concurrent.ArrayBlockingQueue.poll(
ArrayBlockingQueue.java:418)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
pollSendQueue(QuorumCnxManager.java:879)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.
access$500(QuorumCnxManager.java:65)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$
SendWorker.run(QuorumCnxManager.java:715)
2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
QuorumCnxManager$SendWorker@736] - Send worker leaving thread
2016-10-26 09:03:11,526 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (4, 2)
2016-10-26 09:03:11,527 [myid:2] - INFO  [WorkerSender[myid=2]:
QuorumCnxManager@199] - Have smaller server identifier, so dropping
the
connection: (5, 2)
2016-10-26 09:03:11,555 [myid:2] - INFO
[NIOServerCxn.Factory:0.0.0.0/ 0.0.0.0:80:NIOServerCnxnFactory@192] -
Accepted socket connection from /
172.31.18.151:36082
2016-10-26 09:03:11,609 [myid:2] - WARN
[NIOServerCxn.Factory:0.0.0.0/ 0.0.0.0:80:NIOServerCnxn@357] - caught
end of stream exception
EndOfStreamException: Unable to read additional data from client
sessionid 0x0, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(
NIOServerCnxn.java:230)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:745)


Thanks and Regards,
Preeti Bhat



NOTICE TO RECIPIENTS: This communication may contain confidential
and/or privileged information. If you are not the intended recipient
(or have received this communication in error) please notify the
sender and it-support@shoregrp.com<ma...@shoregrp.com> immediately, and destroy this
communication. Any unauthorized copying, disclosure or distribution of
the material in this communication is strictly forbidden. Any views or
opinions presented in this email are solely those of the author and do
not necessarily represent those of the company. Finally, the recipient
should check this email and any attachments for the presence of
viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.







--
Cheers
Michael.

NOTICE TO RECIPIENTS: This communication may contain confidential and/or privileged information. If you are not the intended recipient (or have received this communication in error) please notify the sender and it-support@shoregrp.com<ma...@shoregrp.com> immediately, and destroy this communication. Any unauthorized copying, disclosure or distribution of the material in this communication is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

Re: Zookeeper fails to connect in cluster while using DNS

Posted by Will Martin <wm...@outlook.com>.
Try configuring solr zk with IP's. That will tell you, 1) the connect 
error is an invisible IP, or 2) the error is in the DNS setup in 
Route53;  [ref'd here the simplest version 
http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-ec2-instance.html. 


If neither of those conditions seem to apply you may want to look at 
your security groups?

-will martin

On 10/28/2016 4:13 AM, Preeti Bhat wrote:
> Hi Michael,
>
> The client side log is as below. For Route S3, we have associated single IP to single DNS.
>
> java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
>          at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:181)
>          at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:115)
>          at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:105)
>          at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:188)
> Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
>          at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:235)
>          at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:173)
>          ... 3 more
>
> Thanks and Regards,
> Preeti Bhat
>
> -----Original Message-----
> From: Michael Han [mailto:hanm@cloudera.com]
> Sent: Thursday, October 27, 2016 9:51 AM
> To: UserZooKeeper
> Subject: Re: Zookeeper fails to connect in cluster while using DNS
>
> This looks like server logs - since the problem is ZK client fail to connect to server, could you also post client logs?
>
> For route 53, if you associate multiple IP addresses to a single DNS name configured in ZK ensemble and for some reasons one of the IP address does not have ZK server process running, it could lead to client fail to connect, but I am not sure if that is your case.
>
> On Wed, Oct 26, 2016 at 6:24 AM, Preeti Bhat <pr...@shoregrp.com>
> wrote:
>
>> Hi All,
>>
>> I am getting the below messages while trying to form the zookeeper
>> cluster in zookeeper.out file. The zookeeper is setup in AWS EC2 RHEL
>> linux servers. The configuration works when we are trying to Public
>> DNS of AWS, but when trying to use the specific DNS created for these
>> instances using
>> RouteS3 we are getting the below error.
>> I have tried stopping the servers, clearing out version-2 folder and
>> restarting with no result.
>> The DNS for the specific server is added to /etc/hosts file as well.
>> Could someone please advise on this.
>>
>>
>> 2016-10-26 09:03:09,991 [myid:] - INFO  [main:QuorumPeerConfig@103] -
>> Reading configuration from: /root/zookeeper-3.4.8/bin/../conf/zoo.cfg
>> 2016-10-26 09:03:10,054 [myid:] - INFO  [main:QuorumPeerConfig@331] -
>> Defaulting to majority quorums
>> 2016-10-26 09:03:10,057 [myid:2] - INFO
>> [main:DatadirCleanupManager@78]
>> - autopurge.snapRetainCount set to 3
>> 2016-10-26 09:03:10,057 [myid:2] - INFO
>> [main:DatadirCleanupManager@79]
>> - autopurge.purgeInterval set to 0
>> 2016-10-26 09:03:10,057 [myid:2] - INFO
>> [main:DatadirCleanupManager@101]
>> - Purge task is not scheduled.
>> 2016-10-26 09:03:10,067 [myid:2] - INFO  [main:QuorumPeerMain@127] -
>> Starting quorum peer
>> 2016-10-26 09:03:10,078 [myid:2] - INFO
>> [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:80
>> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1019] -
>> tickTime set to 40000
>> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1039] -
>> minSessionTimeout set to 120000
>> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1050] -
>> maxSessionTimeout set to 240000
>> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1065] -
>> initLimit set to 10
>> 2016-10-26 09:03:10,099 [myid:2] - INFO  [ListenerThread:
>> QuorumCnxManager$Listener@534] - My election bind port:
>> xxx.com/127.0.0.1:3888
>> 2016-10-26 09:03:10,108 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
>> 0:0:0:80:QuorumPeer@774] - LOOKING
>> 2016-10-26 09:03:10,109 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
>> 0:0:0:80:FastLeaderElection@818] - New election. My id =  2, proposed
>> zxid=0x0
>> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerReceiver[myid=2]:
>> FastLeaderElection@600] - Notification: 1 (message format version), 2
>> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
>> 0x0
>> (n.peerEpoch) LOOKING (my state)
>> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (3, 2)
>> 2016-10-26 09:03:10,117 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
>> 2, error = java.io.EOFException
>>          at java.io.DataInputStream.readInt(DataInputStream.java:392)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> RecvWorker.run(QuorumCnxManager.java:795)
>> 2016-10-26 09:03:10,117 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (4, 2)
>> 2016-10-26 09:03:10,118 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
>> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
>> message on queue java.lang.InterruptedException
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
>> java:2014)
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>          at java.util.concurrent.ArrayBlockingQueue.poll(
>> ArrayBlockingQueue.java:418)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> pollSendQueue(QuorumCnxManager.java:879)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> access$500(QuorumCnxManager.java:65)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> SendWorker.run(QuorumCnxManager.java:715)
>> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
>> 2016-10-26 09:03:10,120 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (5, 2)
>> 2016-10-26 09:03:10,318 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
>> 0:0:0:80:FastLeaderElection@852] - Notification time out: 400
>> 2016-10-26 09:03:10,320 [myid:2] - INFO  [WorkerReceiver[myid=2]:
>> FastLeaderElection@600] - Notification: 1 (message format version), 2
>> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
>> 0x0
>> (n.peerEpoch) LOOKING (my state)
>> 2016-10-26 09:03:10,321 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (3, 2)
>> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
>> 2, error = java.io.EOFException
>>          at java.io.DataInputStream.readInt(DataInputStream.java:392)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> RecvWorker.run(QuorumCnxManager.java:795)
>> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
>> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
>> message on queue java.lang.InterruptedException
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
>> java:2014)
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>          at java.util.concurrent.ArrayBlockingQueue.poll(
>> ArrayBlockingQueue.java:418)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> pollSendQueue(QuorumCnxManager.java:879)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> access$500(QuorumCnxManager.java:65)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> SendWorker.run(QuorumCnxManager.java:715)
>> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
>> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (4, 2)
>> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (5, 2)
>> 2016-10-26 09:03:10,720 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
>> 0:0:0:80:FastLeaderElection@852] - Notification time out: 800
>> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerReceiver[myid=2]:
>> FastLeaderElection@600] - Notification: 1 (message format version), 2
>> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
>> 0x0
>> (n.peerEpoch) LOOKING (my state)
>> 2016-10-26 09:03:10,722 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
>> 2, error = java.io.EOFException
>>          at java.io.DataInputStream.readInt(DataInputStream.java:392)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> RecvWorker.run(QuorumCnxManager.java:795)
>> 2016-10-26 09:03:10,723 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
>> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (3, 2)
>> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
>> message on queue java.lang.InterruptedException
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
>> java:2014)
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>          at java.util.concurrent.ArrayBlockingQueue.poll(
>> ArrayBlockingQueue.java:418)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> pollSendQueue(QuorumCnxManager.java:879)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> access$500(QuorumCnxManager.java:65)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> SendWorker.run(QuorumCnxManager.java:715)
>> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
>> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (4, 2)
>> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (5, 2)
>> 2016-10-26 09:03:11,522 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
>> 0:0:0:80:FastLeaderElection@852] - Notification time out: 1600
>> 2016-10-26 09:03:11,524 [myid:2] - INFO  [WorkerReceiver[myid=2]:
>> FastLeaderElection@600] - Notification: 1 (message format version), 2
>> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
>> 0x0
>> (n.peerEpoch) LOOKING (my state)
>> 2016-10-26 09:03:11,524 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
>> 2, error = java.io.EOFException
>>          at java.io.DataInputStream.readInt(DataInputStream.java:392)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> RecvWorker.run(QuorumCnxManager.java:795)
>> 2016-10-26 09:03:11,525 [myid:2] - WARN  [RecvWorker:1:
>> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
>> 2016-10-26 09:03:11,525 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (3, 2)
>> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
>> message on queue java.lang.InterruptedException
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
>> java:2014)
>>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>          at java.util.concurrent.ArrayBlockingQueue.poll(
>> ArrayBlockingQueue.java:418)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> pollSendQueue(QuorumCnxManager.java:879)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager.
>> access$500(QuorumCnxManager.java:65)
>>          at org.apache.zookeeper.server.quorum.QuorumCnxManager$
>> SendWorker.run(QuorumCnxManager.java:715)
>> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
>> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
>> 2016-10-26 09:03:11,526 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (4, 2)
>> 2016-10-26 09:03:11,527 [myid:2] - INFO  [WorkerSender[myid=2]:
>> QuorumCnxManager@199] - Have smaller server identifier, so dropping
>> the
>> connection: (5, 2)
>> 2016-10-26 09:03:11,555 [myid:2] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/ 0.0.0.0:80:NIOServerCnxnFactory@192] -
>> Accepted socket connection from /
>> 172.31.18.151:36082
>> 2016-10-26 09:03:11,609 [myid:2] - WARN
>> [NIOServerCxn.Factory:0.0.0.0/ 0.0.0.0:80:NIOServerCnxn@357] - caught
>> end of stream exception
>> EndOfStreamException: Unable to read additional data from client
>> sessionid 0x0, likely client has closed socket
>>          at org.apache.zookeeper.server.NIOServerCnxn.doIO(
>> NIOServerCnxn.java:230)
>>          at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
>> NIOServerCnxnFactory.java:203)
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Thanks and Regards,
>> Preeti Bhat
>>
>>
>>
>> NOTICE TO RECIPIENTS: This communication may contain confidential
>> and/or privileged information. If you are not the intended recipient
>> (or have received this communication in error) please notify the
>> sender and it-support@shoregrp.com immediately, and destroy this
>> communication. Any unauthorized copying, disclosure or distribution of
>> the material in this communication is strictly forbidden. Any views or
>> opinions presented in this email are solely those of the author and do
>> not necessarily represent those of the company. Finally, the recipient
>> should check this email and any attachments for the presence of
>> viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
>>
>>
>>
>
> --
> Cheers
> Michael.
>
> NOTICE TO RECIPIENTS: This communication may contain confidential and/or privileged information. If you are not the intended recipient (or have received this communication in error) please notify the sender and it-support@shoregrp.com immediately, and destroy this communication. Any unauthorized copying, disclosure or distribution of the material in this communication is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
>
>

RE: Zookeeper fails to connect in cluster while using DNS

Posted by William Hargrove <Wi...@ig.com>.
Unsubscribe

-----Original Message-----
From: Preeti Bhat [mailto:preeti.bhat@shoregrp.com]
Sent: 28 October 2016 09:13
To: user@zookeeper.apache.org
Subject: RE: Zookeeper fails to connect in cluster while using DNS

Hi Michael,

The client side log is as below. For Route S3, we have associated single IP to single DNS.

java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:181)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:115)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:105)
        at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:188)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
        at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:235)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:173)
        ... 3 more

Thanks and Regards,
Preeti Bhat

-----Original Message-----
From: Michael Han [mailto:hanm@cloudera.com]
Sent: Thursday, October 27, 2016 9:51 AM
To: UserZooKeeper
Subject: Re: Zookeeper fails to connect in cluster while using DNS

This looks like server logs - since the problem is ZK client fail to connect to server, could you also post client logs?

For route 53, if you associate multiple IP addresses to a single DNS name configured in ZK ensemble and for some reasons one of the IP address does not have ZK server process running, it could lead to client fail to connect, but I am not sure if that is your case.

On Wed, Oct 26, 2016 at 6:24 AM, Preeti Bhat <pr...@shoregrp.com>
wrote:

> Hi All,
>
> I am getting the below messages while trying to form the zookeeper
> cluster in zookeeper.out file. The zookeeper is setup in AWS EC2 RHEL
> linux servers. The configuration works when we are trying to Public
> DNS of AWS, but when trying to use the specific DNS created for these
> instances using
> RouteS3 we are getting the below error.
> I have tried stopping the servers, clearing out version-2 folder and
> restarting with no result.
> The DNS for the specific server is added to /etc/hosts file as well.
> Could someone please advise on this.
>
>
> 2016-10-26 09:03:09,991 [myid:] - INFO  [main:QuorumPeerConfig@103] -
> Reading configuration from: /root/zookeeper-3.4.8/bin/../conf/zoo.cfg
> 2016-10-26 09:03:10,054 [myid:] - INFO  [main:QuorumPeerConfig@331] -
> Defaulting to majority quorums
> 2016-10-26 09:03:10,057 [myid:2] - INFO
> [main:DatadirCleanupManager@78]
> - autopurge.snapRetainCount set to 3
> 2016-10-26 09:03:10,057 [myid:2] - INFO
> [main:DatadirCleanupManager@79]
> - autopurge.purgeInterval set to 0
> 2016-10-26 09:03:10,057 [myid:2] - INFO
> [main:DatadirCleanupManager@101]
> - Purge task is not scheduled.
> 2016-10-26 09:03:10,067 [myid:2] - INFO  [main:QuorumPeerMain@127] -
> Starting quorum peer
> 2016-10-26 09:03:10,078 [myid:2] - INFO [main:NIOServerCnxnFactory@89]
> - binding to port 0.0.0.0/0.0.0.0:80
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1019] -
> tickTime set to 40000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1039] -
> minSessionTimeout set to 120000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1050] -
> maxSessionTimeout set to 240000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1065] -
> initLimit set to 10
> 2016-10-26 09:03:10,099 [myid:2] - INFO  [ListenerThread:
> QuorumCnxManager$Listener@534] - My election bind port:
> xxx.com/127.0.0.1:3888
> 2016-10-26 09:03:10,108 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:QuorumPeer@774] - LOOKING
> 2016-10-26 09:03:10,109 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@818] - New election. My id =  2, proposed
> zxid=0x0
> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:10,117 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,117 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:10,118 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,120 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:10,318 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 400
> 2016-10-26 09:03:10,320 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,321 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:10,720 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 800
> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,722 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:11,522 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 1600
> 2016-10-26 09:03:11,524 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:11,524 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:11,525 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:11,526 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:11,527 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:11,555 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/
> 0.0.0.0:80:NIOServerCnxnFactory@192] - Accepted socket connection from
> /
> 172.31.18.151:36082
> 2016-10-26 09:03:11,609 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/
> 0.0.0.0:80:NIOServerCnxn@357] - caught end of stream exception
> EndOfStreamException: Unable to read additional data from client
> sessionid 0x0, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:230)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:203)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks and Regards,
> Preeti Bhat
>
>
>
> NOTICE TO RECIPIENTS: This communication may contain confidential
> and/or privileged information. If you are not the intended recipient
> (or have received this communication in error) please notify the
> sender and it-support@shoregrp.com immediately, and destroy this
> communication. Any unauthorized copying, disclosure or distribution of
> the material in this communication is strictly forbidden. Any views or
> opinions presented in this email are solely those of the author and do
> not necessarily represent those of the company. Finally, the recipient
> should check this email and any attachments for the presence of
> viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
>
>
>


--
Cheers
Michael.

NOTICE TO RECIPIENTS: This communication may contain confidential and/or privileged information. If you are not the intended recipient (or have received this communication in error) please notify the sender and it-support@shoregrp.com immediately, and destroy this communication. Any unauthorized copying, disclosure or distribution of the material in this communication is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

RE: Zookeeper fails to connect in cluster while using DNS

Posted by Preeti Bhat <pr...@shoregrp.com>.
Hi Michael,

The client side log is as below. For Route S3, we have associated single IP to single DNS.

java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:181)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:115)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:105)
        at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:188)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper xxx.xxx.xxxx.com:80, xxx.xxx.xxxx.com:80 within 30000 ms
        at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:235)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:173)
        ... 3 more

Thanks and Regards,
Preeti Bhat

-----Original Message-----
From: Michael Han [mailto:hanm@cloudera.com]
Sent: Thursday, October 27, 2016 9:51 AM
To: UserZooKeeper
Subject: Re: Zookeeper fails to connect in cluster while using DNS

This looks like server logs - since the problem is ZK client fail to connect to server, could you also post client logs?

For route 53, if you associate multiple IP addresses to a single DNS name configured in ZK ensemble and for some reasons one of the IP address does not have ZK server process running, it could lead to client fail to connect, but I am not sure if that is your case.

On Wed, Oct 26, 2016 at 6:24 AM, Preeti Bhat <pr...@shoregrp.com>
wrote:

> Hi All,
>
> I am getting the below messages while trying to form the zookeeper
> cluster in zookeeper.out file. The zookeeper is setup in AWS EC2 RHEL
> linux servers. The configuration works when we are trying to Public
> DNS of AWS, but when trying to use the specific DNS created for these
> instances using
> RouteS3 we are getting the below error.
> I have tried stopping the servers, clearing out version-2 folder and
> restarting with no result.
> The DNS for the specific server is added to /etc/hosts file as well.
> Could someone please advise on this.
>
>
> 2016-10-26 09:03:09,991 [myid:] - INFO  [main:QuorumPeerConfig@103] -
> Reading configuration from: /root/zookeeper-3.4.8/bin/../conf/zoo.cfg
> 2016-10-26 09:03:10,054 [myid:] - INFO  [main:QuorumPeerConfig@331] -
> Defaulting to majority quorums
> 2016-10-26 09:03:10,057 [myid:2] - INFO
> [main:DatadirCleanupManager@78]
> - autopurge.snapRetainCount set to 3
> 2016-10-26 09:03:10,057 [myid:2] - INFO
> [main:DatadirCleanupManager@79]
> - autopurge.purgeInterval set to 0
> 2016-10-26 09:03:10,057 [myid:2] - INFO
> [main:DatadirCleanupManager@101]
> - Purge task is not scheduled.
> 2016-10-26 09:03:10,067 [myid:2] - INFO  [main:QuorumPeerMain@127] -
> Starting quorum peer
> 2016-10-26 09:03:10,078 [myid:2] - INFO
> [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:80
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1019] -
> tickTime set to 40000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1039] -
> minSessionTimeout set to 120000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1050] -
> maxSessionTimeout set to 240000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1065] -
> initLimit set to 10
> 2016-10-26 09:03:10,099 [myid:2] - INFO  [ListenerThread:
> QuorumCnxManager$Listener@534] - My election bind port:
> xxx.com/127.0.0.1:3888
> 2016-10-26 09:03:10,108 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:QuorumPeer@774] - LOOKING
> 2016-10-26 09:03:10,109 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@818] - New election. My id =  2, proposed
> zxid=0x0
> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:10,117 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,117 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:10,118 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,120 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:10,318 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 400
> 2016-10-26 09:03:10,320 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,321 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:10,720 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 800
> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,722 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:11,522 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 1600
> 2016-10-26 09:03:11,524 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid),
> 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:11,524 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id =
> 2, error = java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:11,525 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (3, 2)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for
> message on queue java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:11,526 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (4, 2)
> 2016-10-26 09:03:11,527 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping
> the
> connection: (5, 2)
> 2016-10-26 09:03:11,555 [myid:2] - INFO
> [NIOServerCxn.Factory:0.0.0.0/ 0.0.0.0:80:NIOServerCnxnFactory@192] -
> Accepted socket connection from /
> 172.31.18.151:36082
> 2016-10-26 09:03:11,609 [myid:2] - WARN
> [NIOServerCxn.Factory:0.0.0.0/ 0.0.0.0:80:NIOServerCnxn@357] - caught
> end of stream exception
> EndOfStreamException: Unable to read additional data from client
> sessionid 0x0, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:230)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:203)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks and Regards,
> Preeti Bhat
>
>
>
> NOTICE TO RECIPIENTS: This communication may contain confidential
> and/or privileged information. If you are not the intended recipient
> (or have received this communication in error) please notify the
> sender and it-support@shoregrp.com immediately, and destroy this
> communication. Any unauthorized copying, disclosure or distribution of
> the material in this communication is strictly forbidden. Any views or
> opinions presented in this email are solely those of the author and do
> not necessarily represent those of the company. Finally, the recipient
> should check this email and any attachments for the presence of
> viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
>
>
>


--
Cheers
Michael.

NOTICE TO RECIPIENTS: This communication may contain confidential and/or privileged information. If you are not the intended recipient (or have received this communication in error) please notify the sender and it-support@shoregrp.com immediately, and destroy this communication. Any unauthorized copying, disclosure or distribution of the material in this communication is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

Re: Zookeeper fails to connect in cluster while using DNS

Posted by Michael Han <ha...@cloudera.com>.
This looks like server logs - since the problem is ZK client fail to
connect to server, could you also post client logs?

For route 53, if you associate multiple IP addresses to a single DNS name
configured in ZK ensemble and for some reasons one of the IP address does
not have ZK server process running, it could lead to client fail to
connect, but I am not sure if that is your case.

On Wed, Oct 26, 2016 at 6:24 AM, Preeti Bhat <pr...@shoregrp.com>
wrote:

> Hi All,
>
> I am getting the below messages while trying to form the zookeeper cluster
> in zookeeper.out file. The zookeeper is setup in AWS EC2 RHEL linux
> servers. The configuration works when we are trying to Public DNS of AWS,
> but when trying to use the specific DNS created for these instances using
> RouteS3 we are getting the below error.
> I have tried stopping the servers, clearing out version-2 folder and
> restarting with no result.
> The DNS for the specific server is added to /etc/hosts file as well.
> Could someone please advise on this.
>
>
> 2016-10-26 09:03:09,991 [myid:] - INFO  [main:QuorumPeerConfig@103] -
> Reading configuration from: /root/zookeeper-3.4.8/bin/../conf/zoo.cfg
> 2016-10-26 09:03:10,054 [myid:] - INFO  [main:QuorumPeerConfig@331] -
> Defaulting to majority quorums
> 2016-10-26 09:03:10,057 [myid:2] - INFO  [main:DatadirCleanupManager@78]
> - autopurge.snapRetainCount set to 3
> 2016-10-26 09:03:10,057 [myid:2] - INFO  [main:DatadirCleanupManager@79]
> - autopurge.purgeInterval set to 0
> 2016-10-26 09:03:10,057 [myid:2] - INFO  [main:DatadirCleanupManager@101]
> - Purge task is not scheduled.
> 2016-10-26 09:03:10,067 [myid:2] - INFO  [main:QuorumPeerMain@127] -
> Starting quorum peer
> 2016-10-26 09:03:10,078 [myid:2] - INFO  [main:NIOServerCnxnFactory@89] -
> binding to port 0.0.0.0/0.0.0.0:80
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1019] -
> tickTime set to 40000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1039] -
> minSessionTimeout set to 120000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1050] -
> maxSessionTimeout set to 240000
> 2016-10-26 09:03:10,085 [myid:2] - INFO  [main:QuorumPeer@1065] -
> initLimit set to 10
> 2016-10-26 09:03:10,099 [myid:2] - INFO  [ListenerThread:
> QuorumCnxManager$Listener@534] - My election bind port:
> xxx.com/127.0.0.1:3888
> 2016-10-26 09:03:10,108 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:QuorumPeer@774] - LOOKING
> 2016-10-26 09:03:10,109 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@818] - New election. My id =  2, proposed
> zxid=0x0
> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,116 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (3, 2)
> 2016-10-26 09:03:10,117 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2,
> error =
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,117 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (4, 2)
> 2016-10-26 09:03:10,118 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message
> on queue
> java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,119 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,120 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (5, 2)
> 2016-10-26 09:03:10,318 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 400
> 2016-10-26 09:03:10,320 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,321 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (3, 2)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2,
> error =
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message
> on queue
> java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,321 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (4, 2)
> 2016-10-26 09:03:10,322 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (5, 2)
> 2016-10-26 09:03:10,720 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 800
> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:10,722 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2,
> error =
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:10,722 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (3, 2)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message
> on queue
> java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:10,723 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (4, 2)
> 2016-10-26 09:03:10,724 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (5, 2)
> 2016-10-26 09:03:11,522 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> 0:0:0:80:FastLeaderElection@852] - Notification time out: 1600
> 2016-10-26 09:03:11,524 [myid:2] - INFO  [WorkerReceiver[myid=2]:
> FastLeaderElection@600] - Notification: 1 (message format version), 2
> (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0
> (n.peerEpoch) LOOKING (my state)
> 2016-10-26 09:03:11,524 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 2,
> error =
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:795)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [RecvWorker:1:
> QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-10-26 09:03:11,525 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (3, 2)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message
> on queue
> java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:879)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:65)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:715)
> 2016-10-26 09:03:11,525 [myid:2] - WARN  [SendWorker:1:
> QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-10-26 09:03:11,526 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (4, 2)
> 2016-10-26 09:03:11,527 [myid:2] - INFO  [WorkerSender[myid=2]:
> QuorumCnxManager@199] - Have smaller server identifier, so dropping the
> connection: (5, 2)
> 2016-10-26 09:03:11,555 [myid:2] - INFO  [NIOServerCxn.Factory:0.0.0.0/
> 0.0.0.0:80:NIOServerCnxnFactory@192] - Accepted socket connection from /
> 172.31.18.151:36082
> 2016-10-26 09:03:11,609 [myid:2] - WARN  [NIOServerCxn.Factory:0.0.0.0/
> 0.0.0.0:80:NIOServerCnxn@357] - caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x0, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:230)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:203)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks and Regards,
> Preeti Bhat
>
>
>
> NOTICE TO RECIPIENTS: This communication may contain confidential and/or
> privileged information. If you are not the intended recipient (or have
> received this communication in error) please notify the sender and
> it-support@shoregrp.com immediately, and destroy this communication. Any
> unauthorized copying, disclosure or distribution of the material in this
> communication is strictly forbidden. Any views or opinions presented in
> this email are solely those of the author and do not necessarily represent
> those of the company. Finally, the recipient should check this email and
> any attachments for the presence of viruses. The company accepts no
> liability for any damage caused by any virus transmitted by this email.
>
>
>


-- 
Cheers
Michael.