You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Tadevos Papoyan <tp...@vmware.com> on 2020/05/24 07:08:12 UTC

nodetool status shows wrong cluster topology

Hi all,

I have cassandra cluster (v3.11.6 with GossipingPropertyFileSnitch) with 2 datacenters named Zone1 and Zone2 each having 2 nodes.  When all nodes are up "nodetool status" shows correct cluster topology. When I bring down all nodes except one, remaining node still shows correct topology. However, when I restart last node (others remain down), when it comes up "nodetool status" shows all down nodes under "DC1" no matter how long I wait. When I check system.peers table it shows correct topology. Is this a bug or it's expected?

Regards,
Tadevos Papoyan

Re: nodetool status shows wrong cluster topology

Posted by Tadevos Papoyan <tp...@vmware.com>.

Thanks Erick,

Issue indeed was caused by cassandra-topology.properties file. Though we don't use it we still had it in our config directory. After removing it "nodetool status" returns correct topology.

Regards,
Tadevos Papoyan
________________________________
From: Erick Ramirez <er...@datastax.com>
Sent: Wednesday, May 27, 2020 6:12 AM
To: user@cassandra.apache.org <us...@cassandra.apache.org>
Subject: Re: nodetool status shows wrong cluster topology

This isn't expected at all and there's definitely something wrong.

However, when I restart last node (others remain down), when it comes up "nodetool status" shows all down nodes under "DC1" no matter how long I wait.

This sounds like the node can't gossip with other nodes. My hypothesis is that cassandra-topology.properties gets loaded on startup. The file is used for PropertyFileSnitch and it's been known to cause intermittent gossip issues with GPFS. That's where the incorrect "DC1" comes from because the node doesn't know which DC the other nodes belong to. We haven't been able to replicate the issue so I don't know how it gets triggered. I've explained it in detail in a previous post [1].

If my hypothesis proves true, you can workaround the problem by deleting cassandra-topology.properties. When you restart Cassandra, the node should be back to normal. Cheers!

[1] https://community.datastax.com/questions/4621/<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.datastax.com%2Fquestions%2F4621%2F&data=02%7C01%7Ctpapoyan%40vmware.com%7Caf44156ea94d4de9fbd408d801e368a6%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637261423529136193&sdata=Vg8rwP6TF06l%2Fst3aQgez%2BD1fC4hFo%2BG9jy1iJ1hZ58%3D&reserved=0>

Re: nodetool status shows wrong cluster topology

Posted by Erick Ramirez <er...@datastax.com>.

This isn't expected at all and there's definitely something wrong.

However, when I restart last node (others remain down), when it comes up
> "nodetool status" shows all down nodes under "DC1" no matter how long I
> wait.
>

This sounds like the node can't gossip with other nodes. My hypothesis is
that cassandra-topology.properties gets loaded on startup. The file is used
for PropertyFileSnitch and it's been known to cause intermittent gossip
issues with GPFS. That's where the incorrect "DC1" comes from because the
node doesn't know which DC the other nodes belong to. We haven't been able
to replicate the issue so I don't know how it gets triggered. I've
explained it in detail in a previous post [1].

If my hypothesis proves true, you can workaround the problem by deleting
cassandra-topology.properties. When you restart Cassandra, the node should
be back to normal. Cheers!

[1] https://community.datastax.com/questions/4621/