You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Shuo Chen <ch...@gmail.com> on 2015/12/18 01:44:36 UTC
Would data be lost by nodetool removenode force
Hi,
I have a 4 node cluster with status 3 UN and 1 DN. I am trying to add a
new node into the cluster but it is also dead. So the cluster is now 3 UN
and 2 DN. However I didnot run nodetool cleanup on any nodes. And just
several KBs of data is migrated to this newly added node.
All the keyspaces are replicated for 3. I am trying to remove this newly
add node using nodetool removenode. But it hangs for 24 hours with remove
node status.
RemovalStatus: Removing token (-9104065154580789913). Waiting for
replication confirmation from [/192.168.148.29,/192.168.148.24,/
192.168.148.23].
Is it safe to use removenode force to remove this newly added node? Will
any data be lost? Thanks!
--
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com
Re: Would data be lost by nodetool removenode force
Posted by Carlos Alonso <in...@mrcalonso.com>.
Why is the old node not able to restart?
If you're about to bring a new one to replace the old dead one, it may be
simpler to just replace it
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
Hope it helps.
Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
On 18 December 2015 at 02:00, Shuo Chen <ch...@gmail.com> wrote:
> The version is 2.0.7
>
> Why we want to add a new node is that somebody creates an index on certain
> table and the cluster becomes slow, so that guy tries to restart the node.
> However only 3 out of 4 nodes can restart and 1 node cannot restart . So
> we intend to add a new node however it shows error that cannot find certain
> cf-id and cannot be started.
>
> So we decided to remove this newly added node by nodetool removenode. Here
> is the thing... We have already started remove this newly added node. Only
> a few data (80kb) has been migrated to this newly added node. The process
> of removenode seems hanging
>
> So how to resolve this situation, can we add another node? or using remove
> force to this newly added node?
>
> On Fri, Dec 18, 2015 at 9:09 AM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Thu, Dec 17, 2015 at 4:44 PM, Shuo Chen <ch...@gmail.com> wrote:
>>
>>> I have a 4 node cluster with status 3 UN and 1 DN. I am trying to add
>>> a new node into the cluster but it is also dead. So the cluster is now 3 UN
>>> and 2 DN. However I didnot run nodetool cleanup on any nodes. And just
>>> several KBs of data is migrated to this newly added node.
>>>
>>> All the keyspaces are replicated for 3. I am trying to remove this newly
>>> add node using nodetool removenode. But it hangs for 24 hours with remove
>>> node status.
>>>
>>> RemovalStatus: Removing token (-9104065154580789913). Waiting for
>>> replication confirmation from [/192.168.148.29,/192.168.148.24,/
>>> 192.168.148.23].
>>>
>>> Is it safe to use removenode force to remove this newly added node? Will
>>> any data be lost? Thanks!
>>>
>>
>> In general it is not "safe" to use "removenode" to remove a node from a
>> cluster, because removenode by definition reduces distinct replica count.
>>
>> That's why a departing node should always use "decommission," so it
>> streams the data it has, including any data it might be the only node it
>> has, to the new nodes responsible for the range.
>>
>> Your email seems to prompt questions.... It doesn't seem like your "real"
>> question is about removenode at all?
>>
>> Why, for example, is your new node you are adding into the cluster "dead"?
>>
>> Are you trying to replace the original DN with your new node? If so, use
>> "replace_address"?
>>
>> =Rob
>> PS - also, what version of cassandra?
>>
>
>
>
> --
> *陈硕* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>
Re: Would data be lost by nodetool removenode force
Posted by Shuo Chen <ch...@gmail.com>.
The version is 2.0.7
Why we want to add a new node is that somebody creates an index on certain
table and the cluster becomes slow, so that guy tries to restart the node.
However only 3 out of 4 nodes can restart and 1 node cannot restart . So
we intend to add a new node however it shows error that cannot find certain
cf-id and cannot be started.
So we decided to remove this newly added node by nodetool removenode. Here
is the thing... We have already started remove this newly added node. Only
a few data (80kb) has been migrated to this newly added node. The process
of removenode seems hanging
So how to resolve this situation, can we add another node? or using remove
force to this newly added node?
On Fri, Dec 18, 2015 at 9:09 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Dec 17, 2015 at 4:44 PM, Shuo Chen <ch...@gmail.com> wrote:
>
>> I have a 4 node cluster with status 3 UN and 1 DN. I am trying to add
>> a new node into the cluster but it is also dead. So the cluster is now 3 UN
>> and 2 DN. However I didnot run nodetool cleanup on any nodes. And just
>> several KBs of data is migrated to this newly added node.
>>
>> All the keyspaces are replicated for 3. I am trying to remove this newly
>> add node using nodetool removenode. But it hangs for 24 hours with remove
>> node status.
>>
>> RemovalStatus: Removing token (-9104065154580789913). Waiting for
>> replication confirmation from [/192.168.148.29,/192.168.148.24,/
>> 192.168.148.23].
>>
>> Is it safe to use removenode force to remove this newly added node? Will
>> any data be lost? Thanks!
>>
>
> In general it is not "safe" to use "removenode" to remove a node from a
> cluster, because removenode by definition reduces distinct replica count.
>
> That's why a departing node should always use "decommission," so it
> streams the data it has, including any data it might be the only node it
> has, to the new nodes responsible for the range.
>
> Your email seems to prompt questions.... It doesn't seem like your "real"
> question is about removenode at all?
>
> Why, for example, is your new node you are adding into the cluster "dead"?
>
> Are you trying to replace the original DN with your new node? If so, use
> "replace_address"?
>
> =Rob
> PS - also, what version of cassandra?
>
--
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com
Re: Would data be lost by nodetool removenode force
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Dec 17, 2015 at 4:44 PM, Shuo Chen <ch...@gmail.com> wrote:
> I have a 4 node cluster with status 3 UN and 1 DN. I am trying to add a
> new node into the cluster but it is also dead. So the cluster is now 3 UN
> and 2 DN. However I didnot run nodetool cleanup on any nodes. And just
> several KBs of data is migrated to this newly added node.
>
> All the keyspaces are replicated for 3. I am trying to remove this newly
> add node using nodetool removenode. But it hangs for 24 hours with remove
> node status.
>
> RemovalStatus: Removing token (-9104065154580789913). Waiting for
> replication confirmation from [/192.168.148.29,/192.168.148.24,/
> 192.168.148.23].
>
> Is it safe to use removenode force to remove this newly added node? Will
> any data be lost? Thanks!
>
In general it is not "safe" to use "removenode" to remove a node from a
cluster, because removenode by definition reduces distinct replica count.
That's why a departing node should always use "decommission," so it streams
the data it has, including any data it might be the only node it has, to
the new nodes responsible for the range.
Your email seems to prompt questions.... It doesn't seem like your "real"
question is about removenode at all?
Why, for example, is your new node you are adding into the cluster "dead"?
Are you trying to replace the original DN with your new node? If so, use
"replace_address"?
=Rob
PS - also, what version of cassandra?