You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Robert Wille <rw...@fold3.com> on 2015/10/08 17:27:19 UTC
Node won't go away
We had some problems with a node, so we decided to rebootstrap it. My IT guy screwed up, and when he added -Dcassandra.replace_address to cassandra-env.sh, he forgot the closing quote. The node bootstrapped, and then refused to join the cluster. We shut it down, and then noticed that nodetool status no longer showed that node, and the “Owns” column had increased from ~10% per node to ~11% (we originally had 10 nodes). I don’t know why Cassandra decided to automatically remove the node from the cluster, but it did. We figured it would be best to make sure the node was completely forgotten, and then add it back into the cluster as a new node. Problem is, it won’t completely go away.
nodetool status doesn’t list it, but its still in system.peers, and OpsCenter still shows it. When I run nodetool removenode, it says that it can’t find the node.
How do I completely get rid of it?
Thanks in advance
Robert
Re: Node won't go away
Posted by Carlos Alonso <in...@mrcalonso.com>.
So if the idea is to completely remove it, just deleting the corresponding
entry from system.peers should do it.
Some versions of Cassandra have a bug that leaves the entry in the
system.peers table after decommissioning, and the fix is just to delete it.
Here is the link to the JIRA:
https://issues.apache.org/jira/browse/CASSANDRA-6053
Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
On 8 October 2015 at 19:24, sai krishnam raju potturi <ps...@gmail.com>
wrote:
> the below solution should work.
>
> For each node in the cluster :
> a : Stop cassandra service on the node.
> b : manually delete data under $data_directory/system/peers/ directory.
> c : In cassandra-env.sh file, add the line JVM_OPTS="$JVM_OPTS
> -Dcassandra.load_ring_state=false".
> d : Restart service on the node.
> e : delete the added line in cassandra-env.sh JVM_OPTS="$JVM_OPTS
> -Dcassandra.load_ring_state=false".
>
> thanks
> Sai Potturi
>
>
>
> On Thu, Oct 8, 2015 at 11:27 AM, Robert Wille <rw...@fold3.com> wrote:
>
>> We had some problems with a node, so we decided to rebootstrap it. My IT
>> guy screwed up, and when he added -Dcassandra.replace_address to
>> cassandra-env.sh, he forgot the closing quote. The node bootstrapped, and
>> then refused to join the cluster. We shut it down, and then noticed that
>> nodetool status no longer showed that node, and the “Owns” column had
>> increased from ~10% per node to ~11% (we originally had 10 nodes). I don’t
>> know why Cassandra decided to automatically remove the node from the
>> cluster, but it did. We figured it would be best to make sure the node was
>> completely forgotten, and then add it back into the cluster as a new node.
>> Problem is, it won’t completely go away.
>>
>> nodetool status doesn’t list it, but its still in system.peers, and
>> OpsCenter still shows it. When I run nodetool removenode, it says that it
>> can’t find the node.
>>
>> How do I completely get rid of it?
>>
>> Thanks in advance
>>
>> Robert
>>
>>
>
Re: Node won't go away
Posted by sai krishnam raju potturi <ps...@gmail.com>.
the below solution should work.
For each node in the cluster :
a : Stop cassandra service on the node.
b : manually delete data under $data_directory/system/peers/ directory.
c : In cassandra-env.sh file, add the line JVM_OPTS="$JVM_OPTS
-Dcassandra.load_ring_state=false".
d : Restart service on the node.
e : delete the added line in cassandra-env.sh JVM_OPTS="$JVM_OPTS
-Dcassandra.load_ring_state=false".
thanks
Sai Potturi
On Thu, Oct 8, 2015 at 11:27 AM, Robert Wille <rw...@fold3.com> wrote:
> We had some problems with a node, so we decided to rebootstrap it. My IT
> guy screwed up, and when he added -Dcassandra.replace_address to
> cassandra-env.sh, he forgot the closing quote. The node bootstrapped, and
> then refused to join the cluster. We shut it down, and then noticed that
> nodetool status no longer showed that node, and the “Owns” column had
> increased from ~10% per node to ~11% (we originally had 10 nodes). I don’t
> know why Cassandra decided to automatically remove the node from the
> cluster, but it did. We figured it would be best to make sure the node was
> completely forgotten, and then add it back into the cluster as a new node.
> Problem is, it won’t completely go away.
>
> nodetool status doesn’t list it, but its still in system.peers, and
> OpsCenter still shows it. When I run nodetool removenode, it says that it
> can’t find the node.
>
> How do I completely get rid of it?
>
> Thanks in advance
>
> Robert
>
>
Re: Node won't go away
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 10/08/2015 10:27 AM, Robert Wille wrote:
> nodetool status doesn’t list it, but its still in system.peers, and
> OpsCenter still shows it. When I run nodetool removenode, it says
> that it can’t find the node.
>
> How do I completely get rid of it?
https://twitter.com/faltering/status/559845791741657088
"Don't forget the hierarchy of node removal in #cassandra: decommission,
removenode, removenode force, assassinate. Escalate in that order."
--
Michael