You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Rounak Kakkar (rkakkar)" <rk...@cisco.com.INVALID> on 2023/07/06 06:30:43 UTC
Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Rounak Kakkar (rkakkar)" <rk...@cisco.com.INVALID>.
Thanks Steph.
We have changed the 0.0.0.0 to 10.x.x.x IP, and now config is same across all the ZK nodes in the cluster.
After this change we don’t see any issue.
Thanks,
Rounak
From: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Date: Friday, 7 July 2023 at 14:25
To: Steph van Schalkwyk <sv...@gmail.com>, user@zookeeper.apache.org <us...@zookeeper.apache.org>, Rounak Kakkar (rkakkar) <rk...@cisco.com>
Cc: dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Subject: Re: Zookeeper server doesn't list all the participants
+Rounak
From: Steph van Schalkwyk <sv...@gmail.com>
Date: Thursday, 6 July 2023 at 9:38 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>
Cc: dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Took a quick glance at your issue.
I see a 169.xxx.... IP address in your diags, which means that NIC cannot get an address reservation.
Check your ZK configs and don't use 0.0.0.0 for an IP as it binds to all your NICS, use a pingable IP address as in 10.0.xxx. Make sure your
server,0, server.1 server.2 settings are the same across all three ZK nodes. No 0.0.0.0.
Steph
On Thu, Jul 6, 2023 at 10:40 AM Rounak Kakkar (rkakkar) <rk...@cisco.com.invalid> wrote:
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Rounak Kakkar (rkakkar)" <rk...@cisco.com.INVALID>.
Thanks Steph.
We have changed the 0.0.0.0 to 10.x.x.x IP, and now config is same across all the ZK nodes in the cluster.
After this change we don’t see any issue.
Thanks,
Rounak
From: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Date: Friday, 7 July 2023 at 14:25
To: Steph van Schalkwyk <sv...@gmail.com>, user@zookeeper.apache.org <us...@zookeeper.apache.org>, Rounak Kakkar (rkakkar) <rk...@cisco.com>
Cc: dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Subject: Re: Zookeeper server doesn't list all the participants
+Rounak
From: Steph van Schalkwyk <sv...@gmail.com>
Date: Thursday, 6 July 2023 at 9:38 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>
Cc: dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Took a quick glance at your issue.
I see a 169.xxx.... IP address in your diags, which means that NIC cannot get an address reservation.
Check your ZK configs and don't use 0.0.0.0 for an IP as it binds to all your NICS, use a pingable IP address as in 10.0.xxx. Make sure your
server,0, server.1 server.2 settings are the same across all three ZK nodes. No 0.0.0.0.
Steph
On Thu, Jul 6, 2023 at 10:40 AM Rounak Kakkar (rkakkar) <rk...@cisco.com.invalid> wrote:
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Jitesh Mulchandani (jmulchan)" <jm...@cisco.com.INVALID>.
+Rounak
From: Steph van Schalkwyk <sv...@gmail.com>
Date: Thursday, 6 July 2023 at 9:38 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>
Cc: dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Took a quick glance at your issue.
I see a 169.xxx.... IP address in your diags, which means that NIC cannot get an address reservation.
Check your ZK configs and don't use 0.0.0.0 for an IP as it binds to all your NICS, use a pingable IP address as in 10.0.xxx. Make sure your
server,0, server.1 server.2 settings are the same across all three ZK nodes. No 0.0.0.0.
Steph
On Thu, Jul 6, 2023 at 10:40 AM Rounak Kakkar (rkakkar) <rk...@cisco.com.invalid> wrote:
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Jitesh Mulchandani (jmulchan)" <jm...@cisco.com.INVALID>.
+Rounak
From: Steph van Schalkwyk <sv...@gmail.com>
Date: Thursday, 6 July 2023 at 9:38 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>
Cc: dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Took a quick glance at your issue.
I see a 169.xxx.... IP address in your diags, which means that NIC cannot get an address reservation.
Check your ZK configs and don't use 0.0.0.0 for an IP as it binds to all your NICS, use a pingable IP address as in 10.0.xxx. Make sure your
server,0, server.1 server.2 settings are the same across all three ZK nodes. No 0.0.0.0.
Steph
On Thu, Jul 6, 2023 at 10:40 AM Rounak Kakkar (rkakkar) <rk...@cisco.com.invalid> wrote:
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=10.0.105.39:2888<http://10.0.105.39:2888>:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888<http://10.0.105.32:2888>:3888:participant
server.1=10.0.105.38:2888<http://10.0.105.38:2888>:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by Steph van Schalkwyk <sv...@gmail.com>.
Took a quick glance at your issue.
I see a 169.xxx.... IP address in your diags, which means that NIC cannot
get an address reservation.
Check your ZK configs and don't use 0.0.0.0 for an IP as it binds to all
your NICS, use a pingable IP address as in 10.0.xxx. Make sure your
server,0, server.1 server.2 settings are the same across all three ZK
nodes. No 0.0.0.0.
Steph
On Thu, Jul 6, 2023 at 10:40 AM Rounak Kakkar (rkakkar)
<rk...@cisco.com.invalid> wrote:
> Hello Team,
>
> We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18
> and vm19).
> We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in
> the cluster, ZK doesn’t list all the participants.
> The node on which the issue is seen is a leader and the other two nodes
> (vm12 and vm19) are followers and they are connected to the leader node
> (vm18).
>
> Could you please help us to debug this issue?
>
> ZK Version:
>
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
>
>
> Node on which issue is seen (vm18). It shows only 2 participants.
>
> vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh
> -server localhost config | grep participant
> server.0=10.0.105.32:2888:3888:participant
> server.1=0.0.0.0:2888:3888:participant
> vm18:~#
>
>
> We can see in config there are nodes.
>
> vm18:~# echo conf | nc 10.0.105.38 2181
> clientPort=2181
> secureClientPort=-1
> dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
> dataDirSize=201387396
> dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
> dataLogSize=201387396
> tickTime=2000
> maxClientCnxns=0
> minSessionTimeout=4000
> maxSessionTimeout=40000
> clientPortListenBacklog=-1
> serverId=1
> initLimit=5
> syncLimit=2
> electionAlg=3
> electionPort=3888
> quorumPort=2888
> peerType=0
> membership:
> server.0=10.0.105.32:2888:3888:participant
> server.1=0.0.0.0:2888:3888:participant
> server.2=10.0.105.39:2888:3888:participant
> version=0
>
>
> And this node is a leader.
>
> vm18:~# echo stat | nc 10.0.105.38 2181
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built
> on 09/04/2020 12:44 GMT
> Clients:
> /10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
> /10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
> /169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
> /10.0.105.38:50552[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0.2586/1626
> Received: 296333
> Sent: 296451
> Connections: 4
> Outstanding: 0
> Zxid: 0x5000015b9
> Mode: leader
> Node count: 381
> Proposal
>
> Other two nodes where participants all three participants are seen.
>
> VM12:
>
> vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh
> -server localhost config | grep participant
> server.0=0.0.0.0:2888:3888:participant
> server.1=10.0.105.38:2888:3888:participant
> server.2=10.0.105.39:2888:3888:participant
> vm12:~#
>
>
>
> vm12:~# echo stat | nc 10.0.105.32 2181
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built
> on 09/04/2020 12:44 GMT
> Clients:
> /10.0.105.32:41494[0](queued=0,recved=1,sent=0)
> /169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
> /10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
> /10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
> /10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
> /10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
> /10.0.105.39:53858[0](queued=0,recved=1,sent=0)
> /10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
>
> Latency min/avg/max: 0/0.4517/350
> Received: 361365
> Sent: 361728
> Connections: 8
> Outstanding: 0
> Zxid: 0x5000015d4
> Mode: follower
> Node count: 384
> vm12:~#
>
>
> VM19:
>
> vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh
> -server localhost config | grep participant
> server.0=10.0.105.32:2888:3888:participant
> server.1=10.0.105.38:2888:3888:participant
> server.2=0.0.0.0:2888:3888:participant
> vm19:~#
>
>
>
> vm19:~# echo stat | nc 10.0.105.39 2181
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built
> on 09/04/2020 12:44 GMT
> Clients:
> /169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
> /10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
> /169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
> /10.0.105.39:41558[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0.3002/276
> Received: 238713
> Sent: 238881
> Connections: 4
> Outstanding: 0
> Zxid: 0x5000015dc
> Mode: follower
> Node count: 386
> vm19:~#
>
> Thanks,
> Rounak
>
Re: Zookeeper server doesn't list all the participants
Posted by Steph van Schalkwyk <sv...@gmail.com>.
Took a quick glance at your issue.
I see a 169.xxx.... IP address in your diags, which means that NIC cannot
get an address reservation.
Check your ZK configs and don't use 0.0.0.0 for an IP as it binds to all
your NICS, use a pingable IP address as in 10.0.xxx. Make sure your
server,0, server.1 server.2 settings are the same across all three ZK
nodes. No 0.0.0.0.
Steph
On Thu, Jul 6, 2023 at 10:40 AM Rounak Kakkar (rkakkar)
<rk...@cisco.com.invalid> wrote:
> Hello Team,
>
> We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18
> and vm19).
> We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in
> the cluster, ZK doesn’t list all the participants.
> The node on which the issue is seen is a leader and the other two nodes
> (vm12 and vm19) are followers and they are connected to the leader node
> (vm18).
>
> Could you please help us to debug this issue?
>
> ZK Version:
>
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
>
>
> Node on which issue is seen (vm18). It shows only 2 participants.
>
> vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh
> -server localhost config | grep participant
> server.0=10.0.105.32:2888:3888:participant
> server.1=0.0.0.0:2888:3888:participant
> vm18:~#
>
>
> We can see in config there are nodes.
>
> vm18:~# echo conf | nc 10.0.105.38 2181
> clientPort=2181
> secureClientPort=-1
> dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
> dataDirSize=201387396
> dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
> dataLogSize=201387396
> tickTime=2000
> maxClientCnxns=0
> minSessionTimeout=4000
> maxSessionTimeout=40000
> clientPortListenBacklog=-1
> serverId=1
> initLimit=5
> syncLimit=2
> electionAlg=3
> electionPort=3888
> quorumPort=2888
> peerType=0
> membership:
> server.0=10.0.105.32:2888:3888:participant
> server.1=0.0.0.0:2888:3888:participant
> server.2=10.0.105.39:2888:3888:participant
> version=0
>
>
> And this node is a leader.
>
> vm18:~# echo stat | nc 10.0.105.38 2181
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built
> on 09/04/2020 12:44 GMT
> Clients:
> /10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
> /10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
> /169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
> /10.0.105.38:50552[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0.2586/1626
> Received: 296333
> Sent: 296451
> Connections: 4
> Outstanding: 0
> Zxid: 0x5000015b9
> Mode: leader
> Node count: 381
> Proposal
>
> Other two nodes where participants all three participants are seen.
>
> VM12:
>
> vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh
> -server localhost config | grep participant
> server.0=0.0.0.0:2888:3888:participant
> server.1=10.0.105.38:2888:3888:participant
> server.2=10.0.105.39:2888:3888:participant
> vm12:~#
>
>
>
> vm12:~# echo stat | nc 10.0.105.32 2181
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built
> on 09/04/2020 12:44 GMT
> Clients:
> /10.0.105.32:41494[0](queued=0,recved=1,sent=0)
> /169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
> /10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
> /10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
> /10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
> /10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
> /10.0.105.39:53858[0](queued=0,recved=1,sent=0)
> /10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
>
> Latency min/avg/max: 0/0.4517/350
> Received: 361365
> Sent: 361728
> Connections: 8
> Outstanding: 0
> Zxid: 0x5000015d4
> Mode: follower
> Node count: 384
> vm12:~#
>
>
> VM19:
>
> vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh
> -server localhost config | grep participant
> server.0=10.0.105.32:2888:3888:participant
> server.1=10.0.105.38:2888:3888:participant
> server.2=0.0.0.0:2888:3888:participant
> vm19:~#
>
>
>
> vm19:~# echo stat | nc 10.0.105.39 2181
> Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built
> on 09/04/2020 12:44 GMT
> Clients:
> /169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
> /10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
> /169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
> /10.0.105.39:41558[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0.3002/276
> Received: 238713
> Sent: 238881
> Connections: 4
> Outstanding: 0
> Zxid: 0x5000015dc
> Mode: follower
> Node count: 386
> vm19:~#
>
> Thanks,
> Rounak
>
Re: Zookeeper server doesn't list all the participants
Posted by Josef Roehrl <Jo...@fuseforward.com.INVALID>.
Good to hear. Thanks.
Glad it's working.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[https://fuseforward.atlassian.net/wiki/download/attachments/512327681/image001.png?version=1&modificationDate=1537397840684&cacheVersion=1&api=v2]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com>
Sent: Tuesday, July 11, 2023 8:12 PM
To: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>; Josef Roehrl <Jo...@fuseforward.com>; user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Subject: Re: Zookeeper server doesn't list all the participants
Thanks Josef.
We have changed the 0.0.0.0 to 10.x.x.x IP, and now config is same across all the ZK nodes in the cluster.
After this change we don’t see any issue.
Thanks,
Rounak
From: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Date: Friday, 7 July 2023 at 14:25
To: Josef Roehrl <Jo...@fuseforward.com>, user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Rounak Kakkar (rkakkar) <rk...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
+Rounak
From: Josef Roehrl <Jo...@fuseforward.com>
Date: Thursday, 6 July 2023 at 11:55 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[Image removed by sender.]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by Josef Roehrl <Jo...@fuseforward.com.INVALID>.
Good to hear. Thanks.
Glad it's working.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[https://fuseforward.atlassian.net/wiki/download/attachments/512327681/image001.png?version=1&modificationDate=1537397840684&cacheVersion=1&api=v2]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com>
Sent: Tuesday, July 11, 2023 8:12 PM
To: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>; Josef Roehrl <Jo...@fuseforward.com>; user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Subject: Re: Zookeeper server doesn't list all the participants
Thanks Josef.
We have changed the 0.0.0.0 to 10.x.x.x IP, and now config is same across all the ZK nodes in the cluster.
After this change we don’t see any issue.
Thanks,
Rounak
From: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Date: Friday, 7 July 2023 at 14:25
To: Josef Roehrl <Jo...@fuseforward.com>, user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Rounak Kakkar (rkakkar) <rk...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
+Rounak
From: Josef Roehrl <Jo...@fuseforward.com>
Date: Thursday, 6 July 2023 at 11:55 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[Image removed by sender.]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Rounak Kakkar (rkakkar)" <rk...@cisco.com.INVALID>.
Thanks Josef.
We have changed the 0.0.0.0 to 10.x.x.x IP, and now config is same across all the ZK nodes in the cluster.
After this change we don’t see any issue.
Thanks,
Rounak
From: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Date: Friday, 7 July 2023 at 14:25
To: Josef Roehrl <Jo...@fuseforward.com>, user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Rounak Kakkar (rkakkar) <rk...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
+Rounak
From: Josef Roehrl <Jo...@fuseforward.com>
Date: Thursday, 6 July 2023 at 11:55 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[Image removed by sender.]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Rounak Kakkar (rkakkar)" <rk...@cisco.com.INVALID>.
Thanks Josef.
We have changed the 0.0.0.0 to 10.x.x.x IP, and now config is same across all the ZK nodes in the cluster.
After this change we don’t see any issue.
Thanks,
Rounak
From: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Date: Friday, 7 July 2023 at 14:25
To: Josef Roehrl <Jo...@fuseforward.com>, user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>, Rounak Kakkar (rkakkar) <rk...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
+Rounak
From: Josef Roehrl <Jo...@fuseforward.com>
Date: Thursday, 6 July 2023 at 11:55 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[Image removed by sender.]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Jitesh Mulchandani (jmulchan)" <jm...@cisco.com.INVALID>.
+Rounak
From: Josef Roehrl <Jo...@fuseforward.com>
Date: Thursday, 6 July 2023 at 11:55 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[Image removed by sender.]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by "Jitesh Mulchandani (jmulchan)" <jm...@cisco.com.INVALID>.
+Rounak
From: Josef Roehrl <Jo...@fuseforward.com>
Date: Thursday, 6 July 2023 at 11:55 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>, dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Re: Zookeeper server doesn't list all the participants
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[Image removed by sender.]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by Josef Roehrl <Jo...@fuseforward.com.INVALID>.
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[https://fuseforward.atlassian.net/wiki/download/attachments/512327681/image001.png?version=1&modificationDate=1537397840684&cacheVersion=1&api=v2]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak
Re: Zookeeper server doesn't list all the participants
Posted by Josef Roehrl <Jo...@fuseforward.com.INVALID>.
Hi Rounak,
Your server definitions should be the same for each config file in each instance (they are not).
One of your servers is 0.0.0.0; it should be the appropriate IP address.
Let me know if that helps.
Josef Roehrl
FuseForward | Senior Architect - Professional Services
[https://fuseforward.atlassian.net/wiki/download/attachments/512327681/image001.png?version=1&modificationDate=1537397840684&cacheVersion=1&api=v2]
Website<https://fuseforward.com/?utm_source=Email%20Signature&utm_medium=email%20signature&utm_campaign=email%20signature> | Newsletter<https://fuseforward.com/subscribe-to-our-newsletter/?utm_source=Email%20Signature&utm_medium=Email%20Signature&utm_campaign=Email%20Signature> | Twitter<https://twitter.com/fuseforward> | LinkedIn<https://www.linkedin.com/company/fuseforward/?originalSubdomain=ca>
________________________________
From: Rounak Kakkar (rkakkar) <rk...@cisco.com.INVALID>
Sent: Wednesday, July 5, 2023 11:30 PM
To: user@zookeeper.apache.org <us...@zookeeper.apache.org>; dev@zookeeper.apache.org <de...@zookeeper.apache.org>
Cc: Jitesh Mulchandani (jmulchan) <jm...@cisco.com>
Subject: Zookeeper server doesn't list all the participants
Hello Team,
We are using a ZK cluster with 3 nodes in the cluster (namely: vm12, vm18 and vm19).
We are seeing a issue with ZK cluster, where on one of the nodes (vm18) in the cluster, ZK doesn’t list all the participants.
The node on which the issue is seen is a leader and the other two nodes (vm12 and vm19) are followers and they are connected to the leader node (vm18).
Could you please help us to debug this issue?
ZK Version:
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715
Node on which issue is seen (vm18). It shows only 2 participants.
vm18:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
vm18:~#
We can see in config there are nodes.
vm18:~# echo conf | nc 10.0.105.38 2181
clientPort=2181
secureClientPort=-1
dataDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataDirSize=201387396
dataLogDir=/data/6356d4ed-ed79-4481-9913-e1c470ec2e79/version-2
dataLogSize=201387396
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=1
initLimit=5
syncLimit=2
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership:
server.0=10.0.105.32:2888:3888:participant
server.1=0.0.0.0:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
version=0
And this node is a leader.
vm18:~# echo stat | nc 10.0.105.38 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:33086[1](queued=0,recved=229788,sent=229790)
/10.0.105.38:52350[1](queued=0,recved=55027,sent=55034)
/169.254.8.3:33050[1](queued=0,recved=3337,sent=3341)
/10.0.105.38:50552[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.2586/1626
Received: 296333
Sent: 296451
Connections: 4
Outstanding: 0
Zxid: 0x5000015b9
Mode: leader
Node count: 381
Proposal
Other two nodes where participants all three participants are seen.
VM12:
vm12:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=0.0.0.0:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=10.0.105.39:2888:3888:participant
vm12:~#
vm12:~# echo stat | nc 10.0.105.32 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/10.0.105.32:41494[0](queued=0,recved=1,sent=0)
/169.254.8.3:55894[1](queued=0,recved=3037,sent=3038)
/10.0.105.38:35292[1](queued=0,recved=231472,sent=231474)
/10.0.105.32:39984[1](queued=0,recved=55138,sent=55148)
/10.0.105.39:52614[1](queued=0,recved=5568,sent=5721)
/10.0.105.32:38996[1](queued=0,recved=6219,sent=6400)
/10.0.105.39:53858[0](queued=0,recved=1,sent=0)
/10.0.105.39:53880[1](queued=0,recved=54945,sent=54949)
Latency min/avg/max: 0/0.4517/350
Received: 361365
Sent: 361728
Connections: 8
Outstanding: 0
Zxid: 0x5000015d4
Mode: follower
Node count: 384
vm12:~#
VM19:
vm19:~# docker exec -i coordination-server /var/lib/zookeeper/bin/zkCli.sh -server localhost config | grep participant
server.0=10.0.105.32:2888:3888:participant
server.1=10.0.105.38:2888:3888:participant
server.2=0.0.0.0:2888:3888:participant
vm19:~#
vm19:~# echo stat | nc 10.0.105.39 2181
Zookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
Clients:
/169.254.8.3:58810[1](queued=0,recved=3246,sent=3250)
/10.0.105.38:43338[1](queued=0,recved=4097,sent=4258)
/169.254.8.1:53464[1](queued=0,recved=228908,sent=228908)
/10.0.105.39:41558[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0.3002/276
Received: 238713
Sent: 238881
Connections: 4
Outstanding: 0
Zxid: 0x5000015dc
Mode: follower
Node count: 386
vm19:~#
Thanks,
Rounak