You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Marc Hoppins <ma...@eset.com> on 2022/08/24 07:06:13 UTC
Erroneous node. - node is not a member of the
Hi all,
I added a node but forgot to specify the correct rack so I stopped the join and removed it. When I tried adding it again it was taking a LONG time to join. I tried draining before stopping the service but that failed. I killed the process and cleared the directories but the cluster status still shows this node as UJ despite the Cassandra service not running on that node.
I tried removenode force but got this message back: node to be removed is not a member of the token ring
I tried assassinate but the node is still UJ according to nodetool status even though the service is not running on the affected node.
I read an article on datastax but that was concerning gossip purge and referred to removing data from the Cassandra/data/system/peers-UUID directory. However, I saw no directory with anything which resembled the UUID of this anomalous node on any of the nodes in the cluster.
How can I remove this from the cluster/ring so that I can re-add?
Marc
RE: Erroneous node. - node is not a member of the
Posted by Marc Hoppins <ma...@eset.com>.
Update again:
The dc1-cass14 node stopped accepting/bootstrapping/streaming early on and now there is just a bunch of
WARN [OptionalTasks:1] 2022-08-24 13:10:16,761 CassandraRoleManager.java:344 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2022-08-24 13:10:16,762 CassandraRoleManager.java:370 - Setup task failed with error, rescheduling
WARN [OptionalTasks:1] 2022-08-24 13:10:26,763 CassandraRoleManager.java:344 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2022-08-24 13:10:26,763 CassandraRoleManager.java:370 - Setup task failed with error, rescheduling
WARN [OptionalTasks:1] 2022-08-24 13:10:36,764 CassandraRoleManager.java:344 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2022-08-24 13:10:36,765 CassandraRoleManager.java:370 - Setup task failed with error, rescheduling
Am I going to have to remove this node and try again?
-----Original Message-----
From: Marc Hoppins <ma...@eset.com>
Sent: Wednesday, August 24, 2022 11:59 AM
To: user@cassandra.apache.org
Subject: RE: Erroneous node. - node is not a member of the
Update:
I shut the server down and the node finally disappeared from the status.
I then restarted the server on the similarly named node (dc1-cass14) and it came up...however, it is UJ. Was this due to the amount of time spent unavailable?
M
-----Original Message-----
From: Marc Hoppins <ma...@eset.com>
Sent: Wednesday, August 24, 2022 9:35 AM
To: user@cassandra.apache.org
Subject: RE: Erroneous node. - node is not a member of the
Also, I just had some changes made to the cass.yml config so thought that is I rolling restart the nodes it might help the problem.
Now I have a startup problem with an existing node...with a similar name Original problem node = dc2-cass14 Existing node = dc1-cass14
and am getting:
ERROR [main] 2022-08-24 09:30:15,822 CassandraDaemon.java:911 - Exception encountered during startup
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:685)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:935)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:785)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:730)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
INFO [StorageServiceShutdownHook] 2022-08-24 09:30:15,842 HintsService.java:222 - Paused hints dispatch WARN [StorageServiceShutdownHook] 2022-08-24 09:30:15,846 Gossiper.java:2032 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown INFO [StorageServiceShutdownHook] 2022-08-24 09:30:15,847 MessagingService.java:441 - Waiting for messaging service to quiesce
-----Original Message-----
From: Marc Hoppins <ma...@eset.com>
Sent: Wednesday, August 24, 2022 9:06 AM
To: user@cassandra.apache.org
Subject: Erroneous node. - node is not a member of the
Hi all,
I added a node but forgot to specify the correct rack so I stopped the join and removed it. When I tried adding it again it was taking a LONG time to join. I tried draining before stopping the service but that failed. I killed the process and cleared the directories but the cluster status still shows this node as UJ despite the Cassandra service not running on that node.
I tried removenode force but got this message back: node to be removed is not a member of the token ring
I tried assassinate but the node is still UJ according to nodetool status even though the service is not running on the affected node.
I read an article on datastax but that was concerning gossip purge and referred to removing data from the Cassandra/data/system/peers-UUID directory. However, I saw no directory with anything which resembled the UUID of this anomalous node on any of the nodes in the cluster.
How can I remove this from the cluster/ring so that I can re-add?
Marc
RE: Erroneous node. - node is not a member of the
Posted by Marc Hoppins <ma...@eset.com>.
Update:
I shut the server down and the node finally disappeared from the status.
I then restarted the server on the similarly named node (dc1-cass14) and it came up...however, it is UJ. Was this due to the amount of time spent unavailable?
M
-----Original Message-----
From: Marc Hoppins <ma...@eset.com>
Sent: Wednesday, August 24, 2022 9:35 AM
To: user@cassandra.apache.org
Subject: RE: Erroneous node. - node is not a member of the
Also, I just had some changes made to the cass.yml config so thought that is I rolling restart the nodes it might help the problem.
Now I have a startup problem with an existing node...with a similar name Original problem node = dc2-cass14 Existing node = dc1-cass14
and am getting:
ERROR [main] 2022-08-24 09:30:15,822 CassandraDaemon.java:911 - Exception encountered during startup
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:685)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:935)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:785)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:730)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
INFO [StorageServiceShutdownHook] 2022-08-24 09:30:15,842 HintsService.java:222 - Paused hints dispatch WARN [StorageServiceShutdownHook] 2022-08-24 09:30:15,846 Gossiper.java:2032 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown INFO [StorageServiceShutdownHook] 2022-08-24 09:30:15,847 MessagingService.java:441 - Waiting for messaging service to quiesce
-----Original Message-----
From: Marc Hoppins <ma...@eset.com>
Sent: Wednesday, August 24, 2022 9:06 AM
To: user@cassandra.apache.org
Subject: Erroneous node. - node is not a member of the
Hi all,
I added a node but forgot to specify the correct rack so I stopped the join and removed it. When I tried adding it again it was taking a LONG time to join. I tried draining before stopping the service but that failed. I killed the process and cleared the directories but the cluster status still shows this node as UJ despite the Cassandra service not running on that node.
I tried removenode force but got this message back: node to be removed is not a member of the token ring
I tried assassinate but the node is still UJ according to nodetool status even though the service is not running on the affected node.
I read an article on datastax but that was concerning gossip purge and referred to removing data from the Cassandra/data/system/peers-UUID directory. However, I saw no directory with anything which resembled the UUID of this anomalous node on any of the nodes in the cluster.
How can I remove this from the cluster/ring so that I can re-add?
Marc
RE: Erroneous node. - node is not a member of the
Posted by Marc Hoppins <ma...@eset.com>.
Also, I just had some changes made to the cass.yml config so thought that is I rolling restart the nodes it might help the problem.
Now I have a startup problem with an existing node...with a similar name
Original problem node = dc2-cass14
Existing node = dc1-cass14
and am getting:
ERROR [main] 2022-08-24 09:30:15,822 CassandraDaemon.java:911 - Exception encountered during startup
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:685)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:935)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:785)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:730)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
INFO [StorageServiceShutdownHook] 2022-08-24 09:30:15,842 HintsService.java:222 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2022-08-24 09:30:15,846 Gossiper.java:2032 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO [StorageServiceShutdownHook] 2022-08-24 09:30:15,847 MessagingService.java:441 - Waiting for messaging service to quiesce
-----Original Message-----
From: Marc Hoppins <ma...@eset.com>
Sent: Wednesday, August 24, 2022 9:06 AM
To: user@cassandra.apache.org
Subject: Erroneous node. - node is not a member of the
Hi all,
I added a node but forgot to specify the correct rack so I stopped the join and removed it. When I tried adding it again it was taking a LONG time to join. I tried draining before stopping the service but that failed. I killed the process and cleared the directories but the cluster status still shows this node as UJ despite the Cassandra service not running on that node.
I tried removenode force but got this message back: node to be removed is not a member of the token ring
I tried assassinate but the node is still UJ according to nodetool status even though the service is not running on the affected node.
I read an article on datastax but that was concerning gossip purge and referred to removing data from the Cassandra/data/system/peers-UUID directory. However, I saw no directory with anything which resembled the UUID of this anomalous node on any of the nodes in the cluster.
How can I remove this from the cluster/ring so that I can re-add?
Marc