[jira] [Resolved] (ZOOKEEPER-3149) Unreachable node can prevent remaining nodes from gaining quorum

["Andrew January (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew January resolved ZOOKEEPER-3149.
---------------------------------------
    Resolution: Duplicate

> Unreachable node can prevent remaining nodes from gaining quorum
> ----------------------------------------------------------------
>
>                 Key: ZOOKEEPER-3149
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3149
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: leaderElection
>    Affects Versions: 3.4.12
>            Reporter: Andrew January
>            Priority: Minor
>
> Steps to reproduce:
>  # Have a 3 node cluster set up, with node 2 as the leader, and node 3 zxid ahead of node 1 such that node 3 will be the new leader when node 2 disappears.
>  # Shut down node 2 such that it is unreachable and attempts to connect to it yield a socket timeout.
>  # Have the remaining two nodes get "Connection refused" responses almost immediately if one tries to connect to the other on a port that isn't open.
> Expected behaviour:
> The remaining nodes reach quorum.
> Actual behaviour:
> The remaining nodes repeatedly fail to reach quorum, spinning and holding elections until node 2 is brought back.
>  
> This is because:
>  # An election for a new leader starts.
>  # Both nodes broadcast notifications to all the other nodes
>  # The notifications are sent to node 1 quickly, then it tries to send it to node 2, which takes cnxTimeout (default 5s) before timing out, then sends it to node 3. This results in all the notifications to node 3 taking 5 seconds to arrive.
>  # Despite the delays, node 1 and node 3 agree that node 3 should be leader.
>  # node 1 sends the message that it will follow node 3, then immediately tries to connect to it as leader.
>  # Because of the delay, node 3 hasn't yet received the notification that node 1 is following it, so doesn't start accepting requests.
>  # This causes the requests from node 1 to fail quickly with "Connection refused".
>  # It retries 5 times (pausing a second between each)
>  # Because these connection refused are happening at 1/5th of cnxTimeout, node 1 gives up trying to follow node 3 and starts a new election.
>  # Node 3 times out waiting for node 1 to acknowledge it as leader, and starts a new election.
>  
> We can work around the issue by decreasing cnxTimeout to be less than 5. However, it seems like a bad idea to rely on tweaking a value based on network performance, especially as the value is only configurable via JVM args rather than the conf files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)