You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Morten A. Iversen" <mo...@mnemonic.no.INVALID> on 2019/06/29 12:35:57 UTC
Difficulties after nodetool removenode
Hi,
We had a hardware issue with one node in a Cassandra cluster and had to use the "nodetool removenode UUID" command from a different node. This seems to be running fine, but one node was restarted after the "nodetool removenode" command was run, and now it seems all streams going from that node have stopped.
On most nodes I can see both "Receiving X files, Y bytes total. Already received Z files, Q bytes total" and "Sending X files, Y bytes total. Already sent Z files, Q bytes total" messages when running nodetool netstats.
Nodes are starting to complete this process, but for the node that was restarted after the "nodetool removenode" command I can only see the "receiving" messages, and on the other nodes the progress from that node seems to have stopped. Is there some way to restart the process on only the node that was restarted?
Regards
Morten Iversen
Re: Difficulties after nodetool removenode
Posted by Jay Zhuang <jz...@apache.org>.
Hi Morten, it might be a bug, which C* version are you using? To guarantee
consistency, it's recommended to run repair on all nodes after removeNode
(for NetworkTopologyStrategy, it could be all nodes in that specific
datacenter).
On Thu, Jul 4, 2019 at 8:30 AM Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Hello,
>
> Just for one node, and if you have a strong consistency 'Read CL + Write
> CL > RF', you can:
>
> - force the node out with 'nodetool removenode force' if it's still around
> - run a repair (just on that node, but full repair).
> OR
> - force the node out with 'nodetool removenode force' if it's still around
> - wipe this node and replace it by itself (if you are missing a lot of
> data or are not comfortable with repairs). *If you just lost a node, this
> might not be safe.* Repair is safer if ran with the right options/tool.
> OR
> - If the node is still there you can also re-run the 'nodetool
> removenode'. Data will be streamed again (to all nodes) and compacted in
> the future eventually.
>
> C*heers,
> -----------------------
> Alain Rodriguez - alain@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> Le sam. 29 juin 2019 à 14:36, Morten A. Iversen
> <mo...@mnemonic.no.invalid> a écrit :
>
>> Hi,
>>
>>
>> We had a hardware issue with one node in a Cassandra cluster and had to
>> use the "nodetool removenode UUID" command from a different node. This
>> seems to be running fine, but one node was restarted after the "nodetool
>> removenode" command was run, and now it seems all streams going from that
>> node have stopped.
>>
>>
>> On most nodes I can see both "Receiving X files, Y bytes total. Already
>> received Z files, Q bytes total" and "Sending X files, Y bytes total.
>> Already sent Z files, Q bytes total" messages when running nodetool
>> netstats.
>>
>>
>> Nodes are starting to complete this process, but for the node that was
>> restarted after the "nodetool removenode" command I can only see the
>> "receiving" messages, and on the other nodes the progress from that node
>> seems to have stopped. Is there some way to restart the process on only the
>> node that was restarted?
>>
>>
>> Regards
>>
>> Morten Iversen
>>
>>
Re: Difficulties after nodetool removenode
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hello,
Just for one node, and if you have a strong consistency 'Read CL + Write CL
> RF', you can:
- force the node out with 'nodetool removenode force' if it's still around
- run a repair (just on that node, but full repair).
OR
- force the node out with 'nodetool removenode force' if it's still around
- wipe this node and replace it by itself (if you are missing a lot of data
or are not comfortable with repairs). *If you just lost a node, this might
not be safe.* Repair is safer if ran with the right options/tool.
OR
- If the node is still there you can also re-run the 'nodetool removenode'.
Data will be streamed again (to all nodes) and compacted in the future
eventually.
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
Le sam. 29 juin 2019 à 14:36, Morten A. Iversen <mo...@mnemonic.no.invalid>
a écrit :
> Hi,
>
>
> We had a hardware issue with one node in a Cassandra cluster and had to
> use the "nodetool removenode UUID" command from a different node. This
> seems to be running fine, but one node was restarted after the "nodetool
> removenode" command was run, and now it seems all streams going from that
> node have stopped.
>
>
> On most nodes I can see both "Receiving X files, Y bytes total. Already
> received Z files, Q bytes total" and "Sending X files, Y bytes total.
> Already sent Z files, Q bytes total" messages when running nodetool
> netstats.
>
>
> Nodes are starting to complete this process, but for the node that was
> restarted after the "nodetool removenode" command I can only see the
> "receiving" messages, and on the other nodes the progress from that node
> seems to have stopped. Is there some way to restart the process on only the
> node that was restarted?
>
>
> Regards
>
> Morten Iversen
>
>