You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by pabbireddy avinash <pa...@gmail.com> on 2017/06/12 19:15:33 UTC
Decommissioned nodes show as DOWN in Cassandra version 3.10
Hi
In the Cassandra version 3.10, after we decommission a node or datacenter,
we observe the decommissioned nodes marked as DOWN in the cluster when you
do a "nodetool describecluster". The nodes however do not show up in the
"nodetool status" command.
The decommissioned node also does not show up in the "system_peers" table
on the nodes.
The workaround we follow is rolling restart of the cluster, which removes
the decommissioned nodes from the "UNREACHABLE STATE", and shows the actual
state of the cluster. The workaround is tedious for huge clusters.
as anybody in the community observed similar issue?
Below are the observed logs
2017-06-12 18:23:29,209 [RMI TCP Connection(8)-127.0.0.1] INFO
StorageService.java:3938 - Announcing that I have left the ring for 30000ms
2017-06-12 18:23:59,210 [RMI TCP Connection(8)-127.0.0.1] INFO
ThriftServer.java:139 - Stop listening to thrift clients
2017-06-12 18:23:59,215 [RMI TCP Connection(8)-127.0.0.1] INFO
Server.java:176 - Stop listening for CQL clients
2017-06-12 18:23:59,216 [RMI TCP Connection(8)-127.0.0.1] WARN
Gossiper.java:1514 - No local state, state is in silent shutdown, or node
hasn't joined, not announcing shutdown
2017-06-12 18:23:59,216 [RMI TCP Connection(8)-127.0.0.1] INFO
MessagingService.java:964 - Waiting for messaging service to quiesce
2017-06-12 18:23:59,217 [ACCEPT-/96.115.209.228] INFO
MessagingService.java:1314 - MessagingService has terminated the accept()
thread
2017-06-12 18:23:59,263 [RMI TCP Connection(8)-127.0.0.1] INFO
StorageService.java:1435 - DECOMMISSIONED
Regards,
Avinash.
Re: Decommissioned nodes show as DOWN in Cassandra version 3.10
Posted by Vladimir Yudovin <vl...@winguzone.com>.
Hi,
you can use
http://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRemoveNode.html
or if this doesn't work ("It is a last resort tool if you cannot successfully use nodetool removenode.")
http://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsAssassinate.html
Best regards, Vladimir Yudovin,
Winguzone - Cloud Cassandra Hosting
---- On Mon, 12 Jun 2017 15:15:33 -0400 pabbireddy avinash <pabbireddyavinash@gmail.com> wrote ----
Hi
In the Cassandra version 3.10, after we decommission a node or datacenter, we observe the decommissioned nodes marked as DOWN in the cluster when you do a "nodetool describecluster". The nodes however do not show up in the "nodetool status" command.
The decommissioned node also does not show up in the "system_peers" table on the nodes.
The workaround we follow is rolling restart of the cluster, which removes the decommissioned nodes from the "UNREACHABLE STATE", and shows the actual state of the cluster. The workaround is tedious for huge clusters.
as anybody in the community observed similar issue?
Below are the observed logs
2017-06-12 18:23:29,209 [RMI TCP Connection(8)-127.0.0.1] INFO StorageService.java:3938 - Announcing that I have left the ring for 30000ms
2017-06-12 18:23:59,210 [RMI TCP Connection(8)-127.0.0.1] INFO ThriftServer.java:139 - Stop listening to thrift clients
2017-06-12 18:23:59,215 [RMI TCP Connection(8)-127.0.0.1] INFO Server.java:176 - Stop listening for CQL clients
2017-06-12 18:23:59,216 [RMI TCP Connection(8)-127.0.0.1] WARN Gossiper.java:1514 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
2017-06-12 18:23:59,216 [RMI TCP Connection(8)-127.0.0.1] INFO MessagingService.java:964 - Waiting for messaging service to quiesce
2017-06-12 18:23:59,217 [ACCEPT-/96.115.209.228] INFO MessagingService.java:1314 - MessagingService has terminated the accept() thread
2017-06-12 18:23:59,263 [RMI TCP Connection(8)-127.0.0.1] INFO StorageService.java:1435 - DECOMMISSIONED
Regards,
Avinash.