You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by hkwan <hk...@centerfield.com> on 2018/01/10 01:16:51 UTC

Unable to connect node to ensemble after restart of node zookeeper 3.4.6

I have a 3 node ensemble in production and after restarting one node it can
no longer connect to the ensemble.  I am getting this error below:

2018-01-10 00:49:32,492 [myid:2] - INFO 
[WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
identifier, so dropping the connection: (3, 2)
2018-01-10 00:50:20,342 [myid:2] - WARN 
[RecvWorker:1:QuorumCnxManager$RecvWorker@780] - Connection broken for id 1,
my id = 2, error =
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:197)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:211)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:765)
2018-01-10 00:50:20,343 [myid:2] - WARN 
[RecvWorker:1:QuorumCnxManager$RecvWorker@783] - Interrupting SendWorker
2018-01-10 00:50:20,343 [myid:2] - WARN 
[SendWorker:1:QuorumCnxManager$SendWorker@697] - Interrupted while waiting
for message on queue
java.lang.InterruptedException
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:849)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:64)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:685)
2018-01-10 00:50:20,343 [myid:2] - WARN 
[SendWorker:1:QuorumCnxManager$SendWorker@706] - Send worker leaving thread
2018-01-10 00:50:32,491 [myid:2] - INFO 
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] -
Notification time out: 60000
2018-01-10 00:50:32,493 [myid:2] - INFO 
[WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message
format version), 2 (n.leader), 0x707e3e9a9 (n.zxid), 0x1 (n.round), LOOKING
(n.state), 2 (n.sid), 0x7 (n.peerEpoch) LOOKING (my state)
2018-01-10 00:50:32,495 [myid:2] - INFO 
[WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
identifier, so dropping the connection: (3, 2)
2018-01-10 00:51:32,494 [myid:2] - INFO 
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] -
Notification time out: 60000
2018-01-10 00:51:32,494 [myid:2] - INFO 
[WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message
format version), 2 (n.leader), 0x707e3e9a9 (n.zxid), 0x1 (n.round), LOOKING
(n.state), 2 (n.sid), 0x7 (n.peerEpoch) LOOKING (my state)
2018-01-10 00:51:32,496 [myid:2] - INFO 
[WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
identifier, so dropping the connection: (3, 2)
2018-01-10 00:52:19,126 [myid:2] - WARN 
[RecvWorker:1:QuorumCnxManager$RecvWorker@780] - Connection broken for id 1,
my id = 2, error =
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:197)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:211)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:765)
2018-01-10 00:52:19,127 [myid:2] - WARN 
[RecvWorker:1:QuorumCnxManager$RecvWorker@783] - Interrupting SendWorker
2018-01-10 00:52:19,127 [myid:2] - WARN 
[SendWorker:1:QuorumCnxManager$SendWorker@697] - Interrupted while waiting
for message on queue
java.lang.InterruptedException
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
        at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:849)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:64)
        at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:685)
2018-01-10 00:52:19,128 [myid:2] - WARN 
[SendWorker:1:QuorumCnxManager$SendWorker@706] - Send worker leaving thread
2018-01-10 00:52:32,495 [myid:2] - INFO 
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] -
Notification time out: 60000
2018-01-10 00:52:32,497 [myid:2] - INFO 
[WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message
format version), 2 (n.leader), 0x707e3e9a9 (n.zxid), 0x1 (n.round), LOOKING
(n.state), 2 (n.sid), 0x7 (n.peerEpoch) LOOKING (my state)
2018-01-10 00:52:32,499 [myid:2] - INFO 
[WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
identifier, so dropping the connection: (3, 2)


my configuration on all three servers are:

clientPort=2181
dataDir=/var/opt/zookeeper/data
tickTime=2000
autopurge.purgeInterval=24
initLimit=10
syncLimit=5
server.1=10.1.0.122:2888:3888
server.2=10.1.1.75:2888:3888
server.3=10.1.2.221:2888:3888

server 3 is currently leader
server 1 is currently follower
server 2 currently cannot rejoin the ensemble

myid files are correctly configured for all three servers.  this is a
production cluster so I would like to know if there was a way to force the
node back into the cluster without anything drastic that would cause the
quorum to be lost.





--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Unable to connect node to ensemble after restart of node zookeeper 3.4.6

Posted by hkwan <hk...@centerfield.com>.
I am running it on AWS ec2 instances.  It is a pretty standard setup, the
only thing of note is that it has been running for a year without any need
for a reboot and without any issues.  this is the first time i've tried to
reboot one of the nodes.



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Unable to connect node to ensemble after restart of node zookeeper 3.4.6

Posted by Andor Molnar <an...@cloudera.com>.
Yes, you shouldn't experience any data loss, only a short outage in your
service until clients are able to reconnect the ensemble.
Unfortunately I'm unable to repro your issue with ZK 3.4.6. Is there any
specific in your ensemble that I need to know? Your config looks to be
quite ordinary.
Are u running on bare metal or cloud provider?

Andor


On Wed, Jan 10, 2018 at 7:16 PM, hkwan <hk...@centerfield.com> wrote:

> At this point would it be safe to say that I would have to reboot the
> entire
> ensemble to get that node back into the cluster?
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: Unable to connect node to ensemble after restart of node zookeeper 3.4.6

Posted by hkwan <hk...@centerfield.com>.
At this point would it be safe to say that I would have to reboot the entire
ensemble to get that node back into the cluster?



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Unable to connect node to ensemble after restart of node zookeeper 3.4.6

Posted by Andor Molnar <an...@cloudera.com>.
​It's very likely related to this:

https://issues.apache.org/jira/browse/ZOOKEEPER-2938

​

On Wed, Jan 10, 2018 at 11:52 AM, Andor Molnar <an...@cloudera.com> wrote:

> [WorkerSender[myid=2]:QuorumCnxManager@193] - *Have smaller server **identifier,
> so dropping the connection: (3, 2)*
>
> Sorry, the peer explicitly drops the connection during leader election for
> some weird reason. The error message indicates that it believes that
> another connection is already established between the two servers.
> Not sure why.
>
> Andor
>
>
>
> On Wed, Jan 10, 2018 at 11:06 AM, hkwan <hk...@centerfield.com> wrote:
>
>> I am able to connect from server 2 to the leader using zkcli.
>>
>>
>>
>> --
>> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>>
>
>

Re: Unable to connect node to ensemble after restart of node zookeeper 3.4.6

Posted by Andor Molnar <an...@cloudera.com>.
[WorkerSender[myid=2]:QuorumCnxManager@193] - *Have smaller server
**identifier,
so dropping the connection: (3, 2)*

Sorry, the peer explicitly drops the connection during leader election for
some weird reason. The error message indicates that it believes that
another connection is already established between the two servers.
Not sure why.

Andor



On Wed, Jan 10, 2018 at 11:06 AM, hkwan <hk...@centerfield.com> wrote:

> I am able to connect from server 2 to the leader using zkcli.
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Re: Unable to connect node to ensemble after restart of node zookeeper 3.4.6

Posted by hkwan <hk...@centerfield.com>.
I am able to connect from server 2 to the leader using zkcli.



--
Sent from: http://zookeeper-user.578899.n2.nabble.com/

Re: Unable to connect node to ensemble after restart of node zookeeper 3.4.6

Posted by Andor Molnar <an...@cloudera.com>.
Hi hkwan,

java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:197)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.net.SocketInputStream.read(SocketInputStream.java:211)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)

This looks like a network issue to me.
Have you tried connecting a client from server 2 to the leader?

Regards,
Andor



On Wed, Jan 10, 2018 at 2:16 AM, hkwan <hk...@centerfield.com> wrote:

> I have a 3 node ensemble in production and after restarting one node it can
> no longer connect to the ensemble.  I am getting this error below:
>
> 2018-01-10 00:49:32,492 [myid:2] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
> identifier, so dropping the connection: (3, 2)
> 2018-01-10 00:50:20,342 [myid:2] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@780] - Connection broken for id
> 1,
> my id = 2, error =
> java.net.SocketException: Connection reset
>         at java.net.SocketInputStream.read(SocketInputStream.java:197)
>         at java.net.SocketInputStream.read(SocketInputStream.java:122)
>         at java.net.SocketInputStream.read(SocketInputStream.java:211)
>         at java.io.DataInputStream.readInt(DataInputStream.java:387)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(
> QuorumCnxManager.java:765)
> 2018-01-10 00:50:20,343 [myid:2] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@783] - Interrupting SendWorker
> 2018-01-10 00:50:20,343 [myid:2] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@697] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
> reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
>         at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(
> QuorumCnxManager.java:849)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:64)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(
> QuorumCnxManager.java:685)
> 2018-01-10 00:50:20,343 [myid:2] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@706] - Send worker leaving
> thread
> 2018-01-10 00:50:32,491 [myid:2] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] -
> Notification time out: 60000
> 2018-01-10 00:50:32,493 [myid:2] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message
> format version), 2 (n.leader), 0x707e3e9a9 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x7 (n.peerEpoch) LOOKING (my state)
> 2018-01-10 00:50:32,495 [myid:2] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
> identifier, so dropping the connection: (3, 2)
> 2018-01-10 00:51:32,494 [myid:2] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] -
> Notification time out: 60000
> 2018-01-10 00:51:32,494 [myid:2] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message
> format version), 2 (n.leader), 0x707e3e9a9 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x7 (n.peerEpoch) LOOKING (my state)
> 2018-01-10 00:51:32,496 [myid:2] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
> identifier, so dropping the connection: (3, 2)
> 2018-01-10 00:52:19,126 [myid:2] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@780] - Connection broken for id
> 1,
> my id = 2, error =
> java.net.SocketException: Connection reset
>         at java.net.SocketInputStream.read(SocketInputStream.java:197)
>         at java.net.SocketInputStream.read(SocketInputStream.java:122)
>         at java.net.SocketInputStream.read(SocketInputStream.java:211)
>         at java.io.DataInputStream.readInt(DataInputStream.java:387)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(
> QuorumCnxManager.java:765)
> 2018-01-10 00:52:19,127 [myid:2] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@783] - Interrupting SendWorker
> 2018-01-10 00:52:19,127 [myid:2] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@697] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
> reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
>         at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(
> QuorumCnxManager.java:849)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:64)
>         at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(
> QuorumCnxManager.java:685)
> 2018-01-10 00:52:19,128 [myid:2] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@706] - Send worker leaving
> thread
> 2018-01-10 00:52:32,495 [myid:2] - INFO
> [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] -
> Notification time out: 60000
> 2018-01-10 00:52:32,497 [myid:2] - INFO
> [WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message
> format version), 2 (n.leader), 0x707e3e9a9 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 2 (n.sid), 0x7 (n.peerEpoch) LOOKING (my state)
> 2018-01-10 00:52:32,499 [myid:2] - INFO
> [WorkerSender[myid=2]:QuorumCnxManager@193] - Have smaller server
> identifier, so dropping the connection: (3, 2)
>
>
> my configuration on all three servers are:
>
> clientPort=2181
> dataDir=/var/opt/zookeeper/data
> tickTime=2000
> autopurge.purgeInterval=24
> initLimit=10
> syncLimit=5
> server.1=10.1.0.122:2888:3888
> server.2=10.1.1.75:2888:3888
> server.3=10.1.2.221:2888:3888
>
> server 3 is currently leader
> server 1 is currently follower
> server 2 currently cannot rejoin the ensemble
>
> myid files are correctly configured for all three servers.  this is a
> production cluster so I would like to know if there was a way to force the
> node back into the cluster without anything drastic that would cause the
> quorum to be lost.
>
>
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>