decommission too slow

["qihuang.zheng" <qi...@fraudmetrix.cn>]

Recently we want to delete some c* nodes for data migration. C* verision:2.0.15
we use nodetooldecommission with nohup: nohup nodetool decommission -h xxx
After execute 3 days already, seems this process did’t finished yet!
This decommissioning Node data isnearly 400G.


1. I check jps -lm, and NodeCmd decommission process still there.
2. and nohup.out file is empty always.
3. I check opscenter, the node status is Leaving: 1 running task.
4. the running task is Compaction, I use nodetool stop compaction, but after sometime later compaction happend again.
5. thenodetool netstats show it’s leaving:
Mode: LEAVING
Unbootstrap 928e6be0-7950-11e5-9cfb-910d8a1425c3
….
Read Repair Statistics:
Attempted: 4746100
Mismatch (Blocking): 950
Mismatch (Background): 100746
Pool Name          Active  Pending   Completed
Commands            n/a     0   1275402208
Responses            n/a     0   1034430957


I don’t know when decommission will finished. Or does something wrong inside? 
just 400G data takes 3 days(and still unfinished) seems abnormal. 


Tks, qihuang.zheng

Re: decommission too slow

[Robert Coli <rc...@eventbrite.com>]

On Mon, Oct 26, 2015 at 7:58 PM, qihuang.zheng <qihuang.zheng@fraudmetrix.cn
> wrote:

> Recently we want to delete some c* nodes for data migration.  C*
> verision:2.0.15
>
> we use *nodetool decommission* with *nohup: nohup nodetool decommission
> -h xxx *
>
> After execute 3 days already, seems this process did’t finished yet!
>

Your streams are probably hanging indefinitely.

IOW, the problem is not "decommission is slow" but "decommission will never
complete."

https://issues.apache.org/jira/browse/CASSANDRA-8611

in 2.1.10 gives a non-infinite timeout for such streams... until then your
only option is to restart the node and restart the decomission.

FWIW, you don't really need to nohup, because having nodetool connecting to
JMX has no effect on whether the decommission completes or doesn't...

=Rob