You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jason Grammenos <ja...@agilitypr.com.INVALID> on 2022/02/23 12:30:36 UTC
Help With zookeeper follower connections
Hello,
I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out if the inter node connection behaviour is normal.
I should have 1 zookeeper leader and 4 followers, and I expect that all 4 followers would open connections exclusively to the leader.
So that "echo stat | nc localhost 2181" on the leader should show me 6 connections: 4 from the followers and 2 from itself (one via proper ip, one from localhost)
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
Clients:
/host02:48610[1](queued=0,recved=1455831,sent=1455833)
/host03:36244[1](queued=0,recved=1459769,sent=1459793)
/host05:53680[1](queued=0,recved=484,sent=484)
/host01:48978[1](queued=0,recved=226,sent=226)
/host04:44810[1](queued=0,recved=52,sent=52)
/127.0.0.1:41434[0](queued=0,recved=1,sent=0)
But instead I have ended up with a few connections on each node, including the followers (as seen in below output). Some followers are even connecting to themselves? The cluster looks stable, with only 1 leader and 4 followers, but I just do not understand this connection behaviour and would like to understand if this is normal behaviour, mis configuration, a bug or something else.
user@host01:~$<mailto:jason.grammenos@ca02vleventstream01:~$> echo stat | nc localhost 2181
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
Clients:
/host01:50768[1](queued=0,recved=6156,sent=6156)
/127.0.0.1:35508[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/3
Received: 9855
Sent: 9854
Connections: 2
Outstanding: 0
Zxid: 0xb00000068
Mode: follower
Node count: 226
user@host02:~$<mailto:jason.grammenos@ca02vleventstream02:~$> echo stat | nc localhost 2181
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
Clients:
/127.0.0.1:49624[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 3699
Sent: 3698
Connections: 1
Outstanding: 0
Zxid: 0xb00000068
Mode: follower
Node count: 226
usaer@host03:~$<mailto:jason.grammenos@ca02vleventstream03:~$> echo stat | nc localhost 2181
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
Clients:
/host03:50674[1](queued=0,recved=6136,sent=6136)
/127.0.0.1:54554[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/22
Received: 9795
Sent: 9794
Connections: 2
Outstanding: 0
Zxid: 0xb00000068
Mode: follower
Node count: 226
user@host04:~$<mailto:jason.grammenos@ca04vleventstream04:~$> echo stat | nc localhost 2181
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
Clients:
/host02:42874[1](queued=0,recved=143,sent=143)
/host04:38222[1](queued=0,recved=89,sent=89)
/127.0.0.1:58796[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/27
Received: 287
Sent: 286
Connections: 3
Outstanding: 0
Zxid: 0xb00000056
Mode: follower
Node count: 226
user@host05:~$<mailto:jason.grammenos@ca04vleventstream05:~$> echo stat | nc localhost 2181
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
Clients:
/127.0.0.1:38680[0](queued=0,recved=1,sent=0)
/host01:41198[1](queued=0,recved=6339,sent=6340)
Latency min/avg/max: 0/0/8
Received: 10041
Sent: 10041
Connections: 2
Outstanding: 0
Zxid: 0xb00000068
Mode: leader
Node count: 226
Proposal sizes last/min/max: 220/36/392
Thank you
Jason
[cid:image001.png@01D82885.92BDE6F0]
Jason Grammenos | Operations and Infrastructure Analyst
Pronouns: he/him
P: 613.232.7797 x1131
Toll-free: 866.545.3745 x1131
jason.grammenos@agilitypr.com
agilitypr.com<https://agilitypr.com/>
Learn new PR tips from our free resources<https://www.agilitypr.com/resources/>.
RE: Help With zookeeper follower connections
Posted by Jason Grammenos <ja...@agilitypr.com.INVALID>.
So I ran "ss -plunt" (netstat equivalent) and could not find any matches between the output of the "stat" command and the port output in ss, but then realized that the port reported via "stat" changes incredibly frequently, so by the time I have run ss, it has changed.
I appreciate the help, but am a little confused about how this line of investigation will tell if all the zookeeper followers should connect to leader or to each other (as they are doing).
Thank you
Jason
Jason Grammenos | Operations and Infrastructure Analyst
Pronouns: he/him
P: 613.232.7797 x1131
Toll-free: 866.545.3745 x1131
jason.grammenos@agilitypr.com
agilitypr.com
Learn new PR tips from our free resources.
-----Original Message-----
From: Enrico Olivelli <eo...@gmail.com>
Sent: February 23, 2022 8:30 AM
To: UserZooKeeper <us...@zookeeper.apache.org>
Subject: Re: Help With zookeeper follower connections
Are you able to verify, using netstat, if those connections are from the ZooKeeper process?
You can compare the TCP port
Enrico
Il Mer 23 Feb 2022, 14:15 Jason Grammenos <ja...@agilitypr.com.invalid> ha scritto:
> I do have tools that could be opening connections. I have akhq
> (akhq.io) running on two of the hosts [host01, host04 (two nodes for
> redundancy reasons)], connecting to the kafka cluster. I also have
> telegraph running the jolokia2 agent plugin to pull monitoring/stats
> data from the hosts jmx (for both kafka and zookeeper)
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.grammenos@agilitypr.com
> agilitypr.com
> Learn new PR tips from our free resources.
>
> -----Original Message-----
> From: Enrico Olivelli <eo...@gmail.com>
> Sent: February 23, 2022 8:08 AM
> To: jason.grammenos@agilitypr.com.invalid
> Cc: UserZooKeeper <us...@zookeeper.apache.org>
> Subject: Re: Help With zookeeper follower connections
>
> Jason,
> Do you have other tools on those machines that could open client
> connections?
>
> Enrico
>
> Il Mer 23 Feb 2022, 13:30 Jason Grammenos
> <ja...@agilitypr.com.invalid> ha scritto:
>
> > Hello,
> >
> >
> >
> > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out
> > if the inter node connection behaviour is normal.
> >
> > I should have 1 zookeeper leader and 4 followers, and I expect that
> > all 4 followers would open connections exclusively to the leader.
> >
> > So that “echo stat | nc localhost 2181” on the leader should show me
> > 6
> > connections: 4 from the followers and 2 from itself (one via proper
> > ip, one from localhost)
> >
> >
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host02:48610[1](queued=0,recved=1455831,sent=1455833)
> >
> > /host03:36244[1](queued=0,recved=1459769,sent=1459793)
> >
> > /host05:53680[1](queued=0,recved=484,sent=484)
> >
> > /host01:48978[1](queued=0,recved=226,sent=226)
> >
> > /host04:44810[1](queued=0,recved=52,sent=52)
> >
> > /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > But instead I have ended up with a few connections on each node,
> > including the followers (as seen in below output). Some followers
> > are even connecting to themselves? The cluster looks stable, with
> > only 1 leader and 4 followers, but I just do not understand this
> > connection behaviour and would like to understand if this is normal
> > behaviour, mis configuration, a bug or something else.
> >
> >
> >
> > user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host01:50768[1](queued=0,recved=6156,sent=6156)
> >
> > /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/3
> >
> > Received: 9855
> >
> > Sent: 9854
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/0
> >
> > Received: 3699
> >
> > Sent: 3698
> >
> > Connections: 1
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host03:50674[1](queued=0,recved=6136,sent=6136)
> >
> > /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/22
> >
> > Received: 9795
> >
> > Sent: 9794
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host02:42874[1](queued=0,recved=143,sent=143)
> >
> > /host04:38222[1](queued=0,recved=89,sent=89)
> >
> > /127.0.0.1:58796[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/0/27
> >
> > Received: 287
> >
> > Sent: 286
> >
> > Connections: 3
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000056
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /127.0.0.1:38680[0](queued=0,recved=1,sent=0)
> >
> > /host01:41198[1](queued=0,recved=6339,sent=6340)
> >
> >
> >
> > Latency min/avg/max: 0/0/8
> >
> > Received: 10041
> >
> > Sent: 10041
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: leader
> >
> > Node count: 226
> >
> > Proposal sizes last/min/max: 220/36/392
> >
> >
> >
> >
> >
> > Thank you
> >
> > Jason
> >
> >
> >
> > Jason Grammenos | Operations and Infrastructure Analyst
> >
> > Pronouns: he/him
> > P: 613.232.7797 x1131
> > Toll-free: 866.545.3745 x1131
> > jason.grammenos@agilitypr.com
> > agilitypr.com
> >
> > Learn new PR tips from our free resources
> > <https://www.agilitypr.com/resources/>.
> >
> >
> >
>
Re: Help With zookeeper follower connections
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
I think the "stat" command should list all ZooKeeper client connections
that are handled by the given ZooKeeper server.
I'm not sure if the ZooKeeper server-server connections are listed there.
(it is based on the result of the ServerCnxFactory.getAllConnectionInfo()
method, which should return client connection info)
But I didn't dig deep, maybe missed something.
On Wed, Feb 23, 2022 at 3:23 PM Jason Grammenos
<ja...@agilitypr.com.invalid> wrote:
> Hello,
>
> I did some more digging after Enrico's last reply (netstat) and it appears
> I may be miss interpreting what the "stat" command output is telling me.
> Does the stat command list all connections or only connections from
> followers/leaders?
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.grammenos@agilitypr.com
> agilitypr.com
> Learn new PR tips from our free resources.
>
> -----Original Message-----
> From: Enrico Olivelli <eo...@gmail.com>
> Sent: February 23, 2022 8:30 AM
> To: UserZooKeeper <us...@zookeeper.apache.org>
> Subject: Re: Help With zookeeper follower connections
>
> Are you able to verify, using netstat, if those connections are from the
> ZooKeeper process?
> You can compare the TCP port
>
> Enrico
>
> Il Mer 23 Feb 2022, 14:15 Jason Grammenos
> <ja...@agilitypr.com.invalid> ha scritto:
>
> > I do have tools that could be opening connections. I have akhq
> > (akhq.io) running on two of the hosts [host01, host04 (two nodes for
> > redundancy reasons)], connecting to the kafka cluster. I also have
> > telegraph running the jolokia2 agent plugin to pull monitoring/stats
> > data from the hosts jmx (for both kafka and zookeeper)
> >
> >
> > Jason Grammenos | Operations and Infrastructure Analyst
> > Pronouns: he/him
> > P: 613.232.7797 x1131
> > Toll-free: 866.545.3745 x1131
> > jason.grammenos@agilitypr.com
> > agilitypr.com
> > Learn new PR tips from our free resources.
> >
> > -----Original Message-----
> > From: Enrico Olivelli <eo...@gmail.com>
> > Sent: February 23, 2022 8:08 AM
> > To: jason.grammenos@agilitypr.com.invalid
> > Cc: UserZooKeeper <us...@zookeeper.apache.org>
> > Subject: Re: Help With zookeeper follower connections
> >
> > Jason,
> > Do you have other tools on those machines that could open client
> > connections?
> >
> > Enrico
> >
> > Il Mer 23 Feb 2022, 13:30 Jason Grammenos
> > <ja...@agilitypr.com.invalid> ha scritto:
> >
> > > Hello,
> > >
> > >
> > >
> > > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out
> > > if the inter node connection behaviour is normal.
> > >
> > > I should have 1 zookeeper leader and 4 followers, and I expect that
> > > all 4 followers would open connections exclusively to the leader.
> > >
> > > So that “echo stat | nc localhost 2181” on the leader should show me
> > > 6
> > > connections: 4 from the followers and 2 from itself (one via proper
> > > ip, one from localhost)
> > >
> > >
> > >
> > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > > built on 05/04/2020 15:53 GMT
> > >
> > > Clients:
> > >
> > > /host02:48610[1](queued=0,recved=1455831,sent=1455833)
> > >
> > > /host03:36244[1](queued=0,recved=1459769,sent=1459793)
> > >
> > > /host05:53680[1](queued=0,recved=484,sent=484)
> > >
> > > /host01:48978[1](queued=0,recved=226,sent=226)
> > >
> > > /host04:44810[1](queued=0,recved=52,sent=52)
> > >
> > > /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
> > >
> > >
> > >
> > > But instead I have ended up with a few connections on each node,
> > > including the followers (as seen in below output). Some followers
> > > are even connecting to themselves? The cluster looks stable, with
> > > only 1 leader and 4 followers, but I just do not understand this
> > > connection behaviour and would like to understand if this is normal
> > > behaviour, mis configuration, a bug or something else.
> > >
> > >
> > >
> > > user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat |
> > > nc localhost 2181
> > >
> > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > > built on 05/04/2020 15:53 GMT
> > >
> > > Clients:
> > >
> > > /host01:50768[1](queued=0,recved=6156,sent=6156)
> > >
> > > /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
> > >
> > >
> > >
> > > Latency min/avg/max: 0/0/3
> > >
> > > Received: 9855
> > >
> > > Sent: 9854
> > >
> > > Connections: 2
> > >
> > > Outstanding: 0
> > >
> > > Zxid: 0xb00000068
> > >
> > > Mode: follower
> > >
> > > Node count: 226
> > >
> > >
> > >
> > > user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat |
> > > nc localhost 2181
> > >
> > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > > built on 05/04/2020 15:53 GMT
> > >
> > > Clients:
> > >
> > > /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
> > >
> > >
> > >
> > > Latency min/avg/max: 0/0/0
> > >
> > > Received: 3699
> > >
> > > Sent: 3698
> > >
> > > Connections: 1
> > >
> > > Outstanding: 0
> > >
> > > Zxid: 0xb00000068
> > >
> > > Mode: follower
> > >
> > > Node count: 226
> > >
> > >
> > >
> > > usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat |
> > > nc localhost 2181
> > >
> > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > > built on 05/04/2020 15:53 GMT
> > >
> > > Clients:
> > >
> > > /host03:50674[1](queued=0,recved=6136,sent=6136)
> > >
> > > /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
> > >
> > >
> > >
> > > Latency min/avg/max: 0/0/22
> > >
> > > Received: 9795
> > >
> > > Sent: 9794
> > >
> > > Connections: 2
> > >
> > > Outstanding: 0
> > >
> > > Zxid: 0xb00000068
> > >
> > > Mode: follower
> > >
> > > Node count: 226
> > >
> > >
> > >
> > > user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat |
> > > nc localhost 2181
> > >
> > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > > built on 05/04/2020 15:53 GMT
> > >
> > > Clients:
> > >
> > > /host02:42874[1](queued=0,recved=143,sent=143)
> > >
> > > /host04:38222[1](queued=0,recved=89,sent=89)
> > >
> > > /127.0.0.1:58796[0](queued=0,recved=1,sent=0)
> > >
> > > Latency min/avg/max: 0/0/27
> > >
> > > Received: 287
> > >
> > > Sent: 286
> > >
> > > Connections: 3
> > >
> > > Outstanding: 0
> > >
> > > Zxid: 0xb00000056
> > >
> > > Mode: follower
> > >
> > > Node count: 226
> > >
> > >
> > >
> > > user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat |
> > > nc localhost 2181
> > >
> > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > > built on 05/04/2020 15:53 GMT
> > >
> > > Clients:
> > >
> > > /127.0.0.1:38680[0](queued=0,recved=1,sent=0)
> > >
> > > /host01:41198[1](queued=0,recved=6339,sent=6340)
> > >
> > >
> > >
> > > Latency min/avg/max: 0/0/8
> > >
> > > Received: 10041
> > >
> > > Sent: 10041
> > >
> > > Connections: 2
> > >
> > > Outstanding: 0
> > >
> > > Zxid: 0xb00000068
> > >
> > > Mode: leader
> > >
> > > Node count: 226
> > >
> > > Proposal sizes last/min/max: 220/36/392
> > >
> > >
> > >
> > >
> > >
> > > Thank you
> > >
> > > Jason
> > >
> > >
> > >
> > > Jason Grammenos | Operations and Infrastructure Analyst
> > >
> > > Pronouns: he/him
> > > P: 613.232.7797 x1131
> > > Toll-free: 866.545.3745 x1131
> > > jason.grammenos@agilitypr.com
> > > agilitypr.com
> > >
> > > Learn new PR tips from our free resources
> > > <https://www.agilitypr.com/resources/>.
> > >
> > >
> > >
> >
>
RE: Help With zookeeper follower connections
Posted by Jason Grammenos <ja...@agilitypr.com.INVALID>.
Hello,
I did some more digging after Enrico's last reply (netstat) and it appears I may be miss interpreting what the "stat" command output is telling me.
Does the stat command list all connections or only connections from followers/leaders?
Jason Grammenos | Operations and Infrastructure Analyst
Pronouns: he/him
P: 613.232.7797 x1131
Toll-free: 866.545.3745 x1131
jason.grammenos@agilitypr.com
agilitypr.com
Learn new PR tips from our free resources.
-----Original Message-----
From: Enrico Olivelli <eo...@gmail.com>
Sent: February 23, 2022 8:30 AM
To: UserZooKeeper <us...@zookeeper.apache.org>
Subject: Re: Help With zookeeper follower connections
Are you able to verify, using netstat, if those connections are from the ZooKeeper process?
You can compare the TCP port
Enrico
Il Mer 23 Feb 2022, 14:15 Jason Grammenos <ja...@agilitypr.com.invalid> ha scritto:
> I do have tools that could be opening connections. I have akhq
> (akhq.io) running on two of the hosts [host01, host04 (two nodes for
> redundancy reasons)], connecting to the kafka cluster. I also have
> telegraph running the jolokia2 agent plugin to pull monitoring/stats
> data from the hosts jmx (for both kafka and zookeeper)
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.grammenos@agilitypr.com
> agilitypr.com
> Learn new PR tips from our free resources.
>
> -----Original Message-----
> From: Enrico Olivelli <eo...@gmail.com>
> Sent: February 23, 2022 8:08 AM
> To: jason.grammenos@agilitypr.com.invalid
> Cc: UserZooKeeper <us...@zookeeper.apache.org>
> Subject: Re: Help With zookeeper follower connections
>
> Jason,
> Do you have other tools on those machines that could open client
> connections?
>
> Enrico
>
> Il Mer 23 Feb 2022, 13:30 Jason Grammenos
> <ja...@agilitypr.com.invalid> ha scritto:
>
> > Hello,
> >
> >
> >
> > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out
> > if the inter node connection behaviour is normal.
> >
> > I should have 1 zookeeper leader and 4 followers, and I expect that
> > all 4 followers would open connections exclusively to the leader.
> >
> > So that “echo stat | nc localhost 2181” on the leader should show me
> > 6
> > connections: 4 from the followers and 2 from itself (one via proper
> > ip, one from localhost)
> >
> >
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host02:48610[1](queued=0,recved=1455831,sent=1455833)
> >
> > /host03:36244[1](queued=0,recved=1459769,sent=1459793)
> >
> > /host05:53680[1](queued=0,recved=484,sent=484)
> >
> > /host01:48978[1](queued=0,recved=226,sent=226)
> >
> > /host04:44810[1](queued=0,recved=52,sent=52)
> >
> > /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > But instead I have ended up with a few connections on each node,
> > including the followers (as seen in below output). Some followers
> > are even connecting to themselves? The cluster looks stable, with
> > only 1 leader and 4 followers, but I just do not understand this
> > connection behaviour and would like to understand if this is normal
> > behaviour, mis configuration, a bug or something else.
> >
> >
> >
> > user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host01:50768[1](queued=0,recved=6156,sent=6156)
> >
> > /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/3
> >
> > Received: 9855
> >
> > Sent: 9854
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/0
> >
> > Received: 3699
> >
> > Sent: 3698
> >
> > Connections: 1
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host03:50674[1](queued=0,recved=6136,sent=6136)
> >
> > /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/22
> >
> > Received: 9795
> >
> > Sent: 9794
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host02:42874[1](queued=0,recved=143,sent=143)
> >
> > /host04:38222[1](queued=0,recved=89,sent=89)
> >
> > /127.0.0.1:58796[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/0/27
> >
> > Received: 287
> >
> > Sent: 286
> >
> > Connections: 3
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000056
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /127.0.0.1:38680[0](queued=0,recved=1,sent=0)
> >
> > /host01:41198[1](queued=0,recved=6339,sent=6340)
> >
> >
> >
> > Latency min/avg/max: 0/0/8
> >
> > Received: 10041
> >
> > Sent: 10041
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: leader
> >
> > Node count: 226
> >
> > Proposal sizes last/min/max: 220/36/392
> >
> >
> >
> >
> >
> > Thank you
> >
> > Jason
> >
> >
> >
> > Jason Grammenos | Operations and Infrastructure Analyst
> >
> > Pronouns: he/him
> > P: 613.232.7797 x1131
> > Toll-free: 866.545.3745 x1131
> > jason.grammenos@agilitypr.com
> > agilitypr.com
> >
> > Learn new PR tips from our free resources
> > <https://www.agilitypr.com/resources/>.
> >
> >
> >
>
Re: Help With zookeeper follower connections
Posted by Enrico Olivelli <eo...@gmail.com>.
Are you able to verify, using netstat, if those connections are from the
ZooKeeper process?
You can compare the TCP port
Enrico
Il Mer 23 Feb 2022, 14:15 Jason Grammenos
<ja...@agilitypr.com.invalid> ha scritto:
> I do have tools that could be opening connections. I have akhq (akhq.io)
> running on two of the hosts [host01, host04 (two nodes for redundancy
> reasons)], connecting to the kafka cluster. I also have telegraph running
> the jolokia2 agent plugin to pull monitoring/stats data from the hosts jmx
> (for both kafka and zookeeper)
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.grammenos@agilitypr.com
> agilitypr.com
> Learn new PR tips from our free resources.
>
> -----Original Message-----
> From: Enrico Olivelli <eo...@gmail.com>
> Sent: February 23, 2022 8:08 AM
> To: jason.grammenos@agilitypr.com.invalid
> Cc: UserZooKeeper <us...@zookeeper.apache.org>
> Subject: Re: Help With zookeeper follower connections
>
> Jason,
> Do you have other tools on those machines that could open client
> connections?
>
> Enrico
>
> Il Mer 23 Feb 2022, 13:30 Jason Grammenos
> <ja...@agilitypr.com.invalid> ha scritto:
>
> > Hello,
> >
> >
> >
> > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out
> > if the inter node connection behaviour is normal.
> >
> > I should have 1 zookeeper leader and 4 followers, and I expect that
> > all 4 followers would open connections exclusively to the leader.
> >
> > So that “echo stat | nc localhost 2181” on the leader should show me 6
> > connections: 4 from the followers and 2 from itself (one via proper
> > ip, one from localhost)
> >
> >
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host02:48610[1](queued=0,recved=1455831,sent=1455833)
> >
> > /host03:36244[1](queued=0,recved=1459769,sent=1459793)
> >
> > /host05:53680[1](queued=0,recved=484,sent=484)
> >
> > /host01:48978[1](queued=0,recved=226,sent=226)
> >
> > /host04:44810[1](queued=0,recved=52,sent=52)
> >
> > /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > But instead I have ended up with a few connections on each node,
> > including the followers (as seen in below output). Some followers are
> > even connecting to themselves? The cluster looks stable, with only 1
> > leader and 4 followers, but I just do not understand this connection
> > behaviour and would like to understand if this is normal behaviour,
> > mis configuration, a bug or something else.
> >
> >
> >
> > user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat | nc
> > localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host01:50768[1](queued=0,recved=6156,sent=6156)
> >
> > /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/3
> >
> > Received: 9855
> >
> > Sent: 9854
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat | nc
> > localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/0
> >
> > Received: 3699
> >
> > Sent: 3698
> >
> > Connections: 1
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat |
> > nc localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host03:50674[1](queued=0,recved=6136,sent=6136)
> >
> > /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
> >
> >
> >
> > Latency min/avg/max: 0/0/22
> >
> > Received: 9795
> >
> > Sent: 9794
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat | nc
> > localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /host02:42874[1](queued=0,recved=143,sent=143)
> >
> > /host04:38222[1](queued=0,recved=89,sent=89)
> >
> > /127.0.0.1:58796[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/0/27
> >
> > Received: 287
> >
> > Sent: 286
> >
> > Connections: 3
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000056
> >
> > Mode: follower
> >
> > Node count: 226
> >
> >
> >
> > user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat | nc
> > localhost 2181
> >
> > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> > built on 05/04/2020 15:53 GMT
> >
> > Clients:
> >
> > /127.0.0.1:38680[0](queued=0,recved=1,sent=0)
> >
> > /host01:41198[1](queued=0,recved=6339,sent=6340)
> >
> >
> >
> > Latency min/avg/max: 0/0/8
> >
> > Received: 10041
> >
> > Sent: 10041
> >
> > Connections: 2
> >
> > Outstanding: 0
> >
> > Zxid: 0xb00000068
> >
> > Mode: leader
> >
> > Node count: 226
> >
> > Proposal sizes last/min/max: 220/36/392
> >
> >
> >
> >
> >
> > Thank you
> >
> > Jason
> >
> >
> >
> > Jason Grammenos | Operations and Infrastructure Analyst
> >
> > Pronouns: he/him
> > P: 613.232.7797 x1131
> > Toll-free: 866.545.3745 x1131
> > jason.grammenos@agilitypr.com
> > agilitypr.com
> >
> > Learn new PR tips from our free resources
> > <https://www.agilitypr.com/resources/>.
> >
> >
> >
>
RE: Help With zookeeper follower connections
Posted by Jason Grammenos <ja...@agilitypr.com.INVALID>.
I do have tools that could be opening connections. I have akhq (akhq.io) running on two of the hosts [host01, host04 (two nodes for redundancy reasons)], connecting to the kafka cluster. I also have telegraph running the jolokia2 agent plugin to pull monitoring/stats data from the hosts jmx (for both kafka and zookeeper)
Jason Grammenos | Operations and Infrastructure Analyst
Pronouns: he/him
P: 613.232.7797 x1131
Toll-free: 866.545.3745 x1131
jason.grammenos@agilitypr.com
agilitypr.com
Learn new PR tips from our free resources.
-----Original Message-----
From: Enrico Olivelli <eo...@gmail.com>
Sent: February 23, 2022 8:08 AM
To: jason.grammenos@agilitypr.com.invalid
Cc: UserZooKeeper <us...@zookeeper.apache.org>
Subject: Re: Help With zookeeper follower connections
Jason,
Do you have other tools on those machines that could open client connections?
Enrico
Il Mer 23 Feb 2022, 13:30 Jason Grammenos <ja...@agilitypr.com.invalid> ha scritto:
> Hello,
>
>
>
> I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out
> if the inter node connection behaviour is normal.
>
> I should have 1 zookeeper leader and 4 followers, and I expect that
> all 4 followers would open connections exclusively to the leader.
>
> So that “echo stat | nc localhost 2181” on the leader should show me 6
> connections: 4 from the followers and 2 from itself (one via proper
> ip, one from localhost)
>
>
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> built on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host02:48610[1](queued=0,recved=1455831,sent=1455833)
>
> /host03:36244[1](queued=0,recved=1459769,sent=1459793)
>
> /host05:53680[1](queued=0,recved=484,sent=484)
>
> /host01:48978[1](queued=0,recved=226,sent=226)
>
> /host04:44810[1](queued=0,recved=52,sent=52)
>
> /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
>
>
>
> But instead I have ended up with a few connections on each node,
> including the followers (as seen in below output). Some followers are
> even connecting to themselves? The cluster looks stable, with only 1
> leader and 4 followers, but I just do not understand this connection
> behaviour and would like to understand if this is normal behaviour,
> mis configuration, a bug or something else.
>
>
>
> user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> built on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host01:50768[1](queued=0,recved=6156,sent=6156)
>
> /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/3
>
> Received: 9855
>
> Sent: 9854
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> built on 05/04/2020 15:53 GMT
>
> Clients:
>
> /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/0
>
> Received: 3699
>
> Sent: 3698
>
> Connections: 1
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: follower
>
> Node count: 226
>
>
>
> usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat |
> nc localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> built on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host03:50674[1](queued=0,recved=6136,sent=6136)
>
> /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/22
>
> Received: 9795
>
> Sent: 9794
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> built on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host02:42874[1](queued=0,recved=143,sent=143)
>
> /host04:38222[1](queued=0,recved=89,sent=89)
>
> /127.0.0.1:58796[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/27
>
> Received: 287
>
> Sent: 286
>
> Connections: 3
>
> Outstanding: 0
>
> Zxid: 0xb00000056
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315,
> built on 05/04/2020 15:53 GMT
>
> Clients:
>
> /127.0.0.1:38680[0](queued=0,recved=1,sent=0)
>
> /host01:41198[1](queued=0,recved=6339,sent=6340)
>
>
>
> Latency min/avg/max: 0/0/8
>
> Received: 10041
>
> Sent: 10041
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: leader
>
> Node count: 226
>
> Proposal sizes last/min/max: 220/36/392
>
>
>
>
>
> Thank you
>
> Jason
>
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
>
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.grammenos@agilitypr.com
> agilitypr.com
>
> Learn new PR tips from our free resources
> <https://www.agilitypr.com/resources/>.
>
>
>
Re: Help With zookeeper follower connections
Posted by Enrico Olivelli <eo...@gmail.com>.
Jason,
Do you have other tools on those machines that could open client
connections?
Enrico
Il Mer 23 Feb 2022, 13:30 Jason Grammenos
<ja...@agilitypr.com.invalid> ha scritto:
> Hello,
>
>
>
> I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out if
> the inter node connection behaviour is normal.
>
> I should have 1 zookeeper leader and 4 followers, and I expect that all 4
> followers would open connections exclusively to the leader.
>
> So that “echo stat | nc localhost 2181” on the leader should show me 6
> connections: 4 from the followers and 2 from itself (one via proper ip, one
> from localhost)
>
>
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host02:48610[1](queued=0,recved=1455831,sent=1455833)
>
> /host03:36244[1](queued=0,recved=1459769,sent=1459793)
>
> /host05:53680[1](queued=0,recved=484,sent=484)
>
> /host01:48978[1](queued=0,recved=226,sent=226)
>
> /host04:44810[1](queued=0,recved=52,sent=52)
>
> /127.0.0.1:41434[0](queued=0,recved=1,sent=0)
>
>
>
> But instead I have ended up with a few connections on each node, including
> the followers (as seen in below output). Some followers are even connecting
> to themselves? The cluster looks stable, with only 1 leader and 4
> followers, but I just do not understand this connection behaviour and would
> like to understand if this is normal behaviour, mis configuration, a bug or
> something else.
>
>
>
> user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host01:50768[1](queued=0,recved=6156,sent=6156)
>
> /127.0.0.1:35508[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/3
>
> Received: 9855
>
> Sent: 9854
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /127.0.0.1:49624[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/0
>
> Received: 3699
>
> Sent: 3698
>
> Connections: 1
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: follower
>
> Node count: 226
>
>
>
> usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host03:50674[1](queued=0,recved=6136,sent=6136)
>
> /127.0.0.1:54554[0](queued=0,recved=1,sent=0)
>
>
>
> Latency min/avg/max: 0/0/22
>
> Received: 9795
>
> Sent: 9794
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /host02:42874[1](queued=0,recved=143,sent=143)
>
> /host04:38222[1](queued=0,recved=89,sent=89)
>
> /127.0.0.1:58796[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/27
>
> Received: 287
>
> Sent: 286
>
> Connections: 3
>
> Outstanding: 0
>
> Zxid: 0xb00000056
>
> Mode: follower
>
> Node count: 226
>
>
>
> user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat | nc
> localhost 2181
>
> Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built
> on 05/04/2020 15:53 GMT
>
> Clients:
>
> /127.0.0.1:38680[0](queued=0,recved=1,sent=0)
>
> /host01:41198[1](queued=0,recved=6339,sent=6340)
>
>
>
> Latency min/avg/max: 0/0/8
>
> Received: 10041
>
> Sent: 10041
>
> Connections: 2
>
> Outstanding: 0
>
> Zxid: 0xb00000068
>
> Mode: leader
>
> Node count: 226
>
> Proposal sizes last/min/max: 220/36/392
>
>
>
>
>
> Thank you
>
> Jason
>
>
>
> Jason Grammenos | Operations and Infrastructure Analyst
>
> Pronouns: he/him
> P: 613.232.7797 x1131
> Toll-free: 866.545.3745 x1131
> jason.grammenos@agilitypr.com
> agilitypr.com
>
> Learn new PR tips from our free resources
> <https://www.agilitypr.com/resources/>.
>
>
>