You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jai Bheemsen Rao Dhanwada <ja...@gmail.com> on 2020/04/14 21:11:54 UTC
Understanding "nodetool netstats" on a multi region cluster
Hello,
I have 3 region Cassandra clusters and I am trying to look at the "nodetool
netstats" output, I see two different outputs in 1,2 regions vs region 3.
There is a Bootstrap message in Region 3 and where are it's not present in
Region 1 and Region 2, though all the nodes were bootstrapped.
*Region 1 and Region 2:*
nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 589020
> Mismatch (Blocking): 8203
> Mismatch (Background): 715
> Pool Name Active Pending Completed Dropped
> Large messages n/a 0 10018 4
> Small messages n/a 0 23865593 4
> Gossip messages n/a 0 19911320 143
*Region 3:*
Mode: NORMAL
> *Bootstrap 24359390-4443-11ea-af19-1fbf341b76a0*
> Read Repair Statistics:
> Attempted: 1866447
> Mismatch (Blocking): 356
> Mismatch (Background): 56
> Pool Name Active Pending Completed Dropped
> Large messages n/a 0 25696 4
> Small messages n/a 0 52632109 4
> Gossip messages n/a 0 21123598 313
C* Version: 3.11.3
# of nodes: 3 * 3 Regions = 9
Can someone please let me know why there is a bootstrap message in netstats
output?
Re: Understanding "nodetool netstats" on a multi region cluster
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
This is not causing any issues and with the restart it goes away. But I
would like understand why this is happening only on certain DC but not all
DC.
I am depending on this output to find whether a node is doing any streams
or not.
On Fri, Apr 17, 2020 at 2:37 AM Erick Ramirez <er...@datastax.com>
wrote:
> The node which shows that bootstrap session from January 31 won't go away
> until you restart it. But you don't need to restart unless it's affecting
> the node's operation. Cheers!
>
>>
Re: Understanding "nodetool netstats" on a multi region cluster
Posted by Erick Ramirez <er...@datastax.com>.
The node which shows that bootstrap session from January 31 won't go away
until you restart it. But you don't need to restart unless it's affecting
the node's operation. Cheers!
>
Re: Understanding "nodetool netstats" on a multi region cluster
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
Hello Erick,
I can see the bootstrap message "Bootstrap
24359390-4443-11ea-af19-1fbf341b76a0" every time, I spin up a 3rd
datacenter. I can confirm that there are no network issues and cluster is
not overloaded, as this is the new cluster without any data. 1 and 2
regions don't have any Bootstrap message in "nodetool netstats", only 3rd
region have it and no they bootstrapped successfully and there were no
issues.
On Wed, Apr 15, 2020 at 9:22 PM Erick Ramirez <er...@datastax.com>
wrote:
> I could re-produce this behavior all the times in 3rd datacenter and there
>> is network connectivity issues. Also cluster is not overloaded as this is
>> brand new cluster.
>
>
> I don't quite understand what you mean. What can you re-produce? It would
> be good if you could elaborate. Cheers!
>
Re: Understanding "nodetool netstats" on a multi region cluster
Posted by Erick Ramirez <er...@datastax.com>.
>
> I could re-produce this behavior all the times in 3rd datacenter and there
> is network connectivity issues. Also cluster is not overloaded as this is
> brand new cluster.
I don't quite understand what you mean. What can you re-produce? It would
be good if you could elaborate. Cheers!
Re: Understanding "nodetool netstats" on a multi region cluster
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
I could re-produce this behavior all the times in 3rd datacenter and there
is network connectivity issues. Also cluster is not overloaded as this is
brand new cluster.
On Wednesday, April 15, 2020, Erick Ramirez <er...@datastax.com>
wrote:
> *Bootstrap 24359390-4443-11ea-af19-1fbf341b76a0*
>>
>>
> That bootstrap session ID is from January 31 8:02am Pacific time. I'm
> going to speculate that you attempted to bootstrap a node and encountered
> issues. The stream likely got orphaned as it seems you have some issues
> with your cluster considering the number of dropped messages --
> particularly the high number of dropped gossip. It's indicative of nodes
> not able to communicate with each other (overloaded, unresponsive or
> whatever reason) that's caused the bootstrap to fail (maybe, conjecture on
> my part). Cheers!
>
> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
> have answers! Share your expertise on https://community.datastax.com/.
>
Re: Understanding "nodetool netstats" on a multi region cluster
Posted by Erick Ramirez <er...@datastax.com>.
>
> *Bootstrap 24359390-4443-11ea-af19-1fbf341b76a0*
>
>
That bootstrap session ID is from January 31 8:02am Pacific time. I'm
going to speculate that you attempted to bootstrap a node and encountered
issues. The stream likely got orphaned as it seems you have some issues
with your cluster considering the number of dropped messages --
particularly the high number of dropped gossip. It's indicative of nodes
not able to communicate with each other (overloaded, unresponsive or
whatever reason) that's caused the bootstrap to fail (maybe, conjecture on
my part). Cheers!
GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
have answers! Share your expertise on https://community.datastax.com/.