New node insertion methods

[Vsevolod Filaretov <vs...@gmail.com>]

Hello everyone!

We have a cluster of 4 nodes, 4.5 tb/data per node, and are in the middle
of adding 4 more nodes to the cluster.
Joining a new node based on official guidelines/helps (setup cassandra on a
new node, start cassandra instance, wait until node goes from JOINING state
to NORMAL, perform nodetool cleanup) takes approximately 7 days in our
case, per single node.

Questions:
1)
Are there any workarounds to speed up the process? (e.g. doing cleanup only
after all 4 new nodes joined cluster), or inserting multiple nodes
simultaneously with specific settings?
2)
How do people with tens of Cassandra nodes perform insertion/extraction of
new/failed nodes? What's the usual routine in case you have 20+ nodes, and
need to decommission 4 nodes and insert 4 new ones instead?
Links to blog posts or mail threads much appreciated!

Thank you all in advance,
Vsevolod Filaretov.

Re: New node insertion methods

[Kyrylo Lebediev <kl...@conductor.com>]

Hi Vsevolod,

> Are there any workarounds to speed up the process? (e.g. doing cleanup only after all 4 new nodes joined cluster), or inserting multiple nodes simultaneously with specific settings?
e.g. doing cleanup only after all 4 new nodes joined cluster === allowed
inserting multiple nodes simultaneously with specific settings ==== generally, possible, but not recommended. Add/remove nodes one by one if needed. See http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html for details
also you may play with stream_throughput_outbound_megabits_per_sec parameter in cassandra.yaml

> How do people with tens of Cassandra nodes perform insertion/extraction of new/failed nodes? What's the usual routine in case you have 20+ nodes, and need to decommission 4 nodes and insert 4 new ones instead?
It’s better to replace existing node w/o streaming, just copying data by rsync / attaching EBS volume to new server etc http://thelastpickle.com/blog/2018/02/21/replace-node-without-bootstrapping.html

Regards,
Kyrill


From: Vsevolod Filaretov <vs...@gmail.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Tuesday, March 12, 2019 at 11:01
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: New node insertion methods

Hello everyone!

We have a cluster of 4 nodes, 4.5 tb/data per node, and are in the middle of adding 4 more nodes to the cluster.
Joining a new node based on official guidelines/helps (setup cassandra on a new node, start cassandra instance, wait until node goes from JOINING state to NORMAL, perform nodetool cleanup) takes approximately 7 days in our case, per single node.

Questions:
1)
Are there any workarounds to speed up the process? (e.g. doing cleanup only after all 4 new nodes joined cluster), or inserting multiple nodes simultaneously with specific settings?
2)
How do people with tens of Cassandra nodes perform insertion/extraction of new/failed nodes? What's the usual routine in case you have 20+ nodes, and need to decommission 4 nodes and insert 4 new ones instead?
Links to blog posts or mail threads much appreciated!

Thank you all in advance,
Vsevolod Filaretov.