You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Vikramark (Jira)" <ji...@apache.org> on 2020/11/07 23:46:00 UTC
[jira] [Created] (ZOOKEEPER-3997) Why Zookeeper 3.5.8 leader
shutdown makes the follower not allow new sessions?
Vikramark created ZOOKEEPER-3997:
------------------------------------
Summary: Why Zookeeper 3.5.8 leader shutdown makes the follower not allow new sessions?
Key: ZOOKEEPER-3997
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3997
Project: ZooKeeper
Issue Type: Bug
Reporter: Vikramark
Fix For: 3.5.8
There is a behavior difference between zookeeper 3.4.14 and 3.5.8. In 3.5.8 given a cluster of 3 nodes, when the leader node is shutdown, the follower does not seem to be in sync with the new leader. The clients are not able to establish session and requests gets queued up as outstanding requests. We don't see this issue in case of 3.4.14 version.
Below are the details of how to recreate the issue:
*With version 3.5.8:*
Initial fresh setup of 3 node cluster:
|Zoo1|Zoo2|Zoo3|
|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 0/0/0
Received: 3
Sent: 2
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: follower
Node count: 5|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 0/0/0
Received: 3
Sent: 2
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: leader
Node count: 5
Proposal sizes last/min/max: -1/-1/-1|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: follower
Node count: 5|
After starting one session using zkCli.sh on Zoo1 node:
|Zoo1|Zoo2|Zoo3|
|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 1/9/23
Received: 7
Sent: 6
Connections: 2
Outstanding: 0
Zxid: 0x100000001
Mode: follower
Node count: 5|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x100000001
Mode: leader
Node count: 5
Proposal sizes last/min/max: 36/36/36|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 0/0/0
Received: 3
Sent: 2
Connections: 1
Outstanding: 0
Zxid: 0x100000001
Mode: follower
Node count: 5|
+Note: We can see that Zxid is now consistent across all nodes.+
After shutting down leader node zoo2. We can see ZOO3 became the Leader. For some reason the ZXID is not same between zoo1 and zoo3.
Start a new zkCli.sh session on same node (zoo1). *The session was not established, the cli client just keeps retrying and created many outstanding requests on zoo1.*
*Expected behavior:* The zoo1 should have same zxid as zoo3. The client session should be allowed to be created. No outstanding requests should be added.
|Zoo1|Zoo2|Zoo3|
|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 0/0/2
Received: 50
Sent: 43
Connections: 2
{color:#de350b}Outstanding: 6{color}
{color:#de350b}Zxid: 0x100000001{color}
Mode: follower
Node count: 5|down|Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x200000000
Mode: leader
Node count: 5
Proposal sizes last/min/max: -1/-1/-1|
*With version 3.4.14*
First initial setup:
|Zoo1|Zoo2|Zoo3|
|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: follower
Node count: 4|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: leader
Node count: 4
Proposal sizes last/min/max: -1/-1/-1|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: follower
Node count: 4|
After connecting with zkCli on ZOO1.
After connecting with zkCli on ZOO1.
|Zoo1|Zoo2|Zoo3|
|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/14/33
Received: 5
Sent: 4
Connections: 2
Outstanding: 0
Zxid: 0x100000001
Mode: follower
Node count: 4|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000001
Mode: leader
Node count: 4
Proposal sizes last/min/max: 36/36/36|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000001
Mode: follower
Node count: 4|
Note: The zkid is now same for all the nodes.
After shutting down leader node zoo2. We can see ZOO3 became the Leader. For some reason the ZXID is not same between zoo1 and zoo3 initially. ZOO3 has new zkid as a new epoch was created but zoo1 still had old zkid.
Now closed the existing zxcli and started a new zkCli.sh session on same node (zoo1). *This time session was established!.*
|Zoo1|Zoo2|Zoo3|
|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/1/4
Received: 8
Sent: 7
Connections: 2
{color:#00875a}Outstanding: 0{color}
{color:#00875a}Zxid: 0x200000001{color}
Mode: follower
Node count: 4|down
|Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 3
Sent: 2
Connections: 1
Outstanding: 0
Zxid: 0x200000001
Mode: leader
Node count: 4
Proposal sizes last/min/max: 36/36/36|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)