Expanding from 1 to 2 datacenters

[Voytek Jarnot <vo...@gmail.com>]

I started an higher-level thread years ago about moving a cluster by
expanding from 1 to 2 datacenters, replicating over, then decommissioning
the original DC. Corporate plans being what they are, we're finally getting
into this; I'm largely following the writeup here:
https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html
,
but have a few more-specific questions:

current setup: 1 DC, 4 nodes, RF=3, 1 keyspace
new DC will be 4 nodes as well, RF=3

1) We currently have 2 seed nodes, I'd like to confirm that the correct
course is to make 1-2 (let's say 2) of the new-DC nodes seeds as well, and
update all nodes in both DCs to point at all 4 seeds before I get into
altering the keyspace.

2) Prior to altering the replication on my keyspace to include the new DC,
I do not need/want to create the keyspace in the new DC, correct?

3) The datastax docs mention the auto_bootstrap=false setting, but don't go
into much detail - I'm leaning toward setting it to false on all the new
nodes, sound reasonable?

4) One of the three environments in which this'll happen is slightly more
complicated due to the existing DC living in AWS, whereas the new DC will
be in a different AZ. Do I need to get into switching
from GossipingPropertyFileSnitch to Ec2MultiRegionSnitch? If so, could
someone shed a bit of light on that process, and the associated changes
needed for listen_address and broadcast_address?

Thanks for getting this far,
Voytek Jarnot

Re: Expanding from 1 to 2 datacenters

[Voytek Jarnot <vo...@gmail.com>]

Expanding on number 4 below:

I'm not quite sure what the easiest course of action might be. Currently
the 4 existing nodes listen on private IPs and seeds are set to internal
DNS names. Is setting broadcast_address to their public IPs and restarting
a viable solution?

I'm trying to minimize changes to the existing DC, if that means Amazon
charges me for transferring data via public IPs, so be it.


On Wed, Jun 26, 2019, 10:19 AM Voytek Jarnot <vo...@gmail.com>
wrote:

> I started an higher-level thread years ago about moving a cluster by
> expanding from 1 to 2 datacenters, replicating over, then decommissioning
> the original DC. Corporate plans being what they are, we're finally getting
> into this; I'm largely following the writeup here:
> https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html ,
> but have a few more-specific questions:
>
> current setup: 1 DC, 4 nodes, RF=3, 1 keyspace
> new DC will be 4 nodes as well, RF=3
>
> 1) We currently have 2 seed nodes, I'd like to confirm that the correct
> course is to make 1-2 (let's say 2) of the new-DC nodes seeds as well, and
> update all nodes in both DCs to point at all 4 seeds before I get into
> altering the keyspace.
>
> 2) Prior to altering the replication on my keyspace to include the new DC,
> I do not need/want to create the keyspace in the new DC, correct?
>
> 3) The datastax docs mention the auto_bootstrap=false setting, but don't
> go into much detail - I'm leaning toward setting it to false on all the new
> nodes, sound reasonable?
>
> 4) One of the three environments in which this'll happen is slightly more
> complicated due to the existing DC living in AWS, whereas the new DC will
> be in a different AZ. Do I need to get into switching
> from GossipingPropertyFileSnitch to Ec2MultiRegionSnitch? If so, could
> someone shed a bit of light on that process, and the associated changes
> needed for listen_address and broadcast_address?
>
> Thanks for getting this far,
> Voytek Jarnot
>

Re: Expanding from 1 to 2 datacenters

[Paul Chandler <pa...@redshots.com>]

Hi Voytek,

We moved a large number of clusters from Rackspace to Google, doing it in a similar way to you are suggesting, so to answer your questions:

1) We created the nodes first with seeds from just the original datacenters, then updated the seed list to include seeds from the new DC afterwards.

2) Don’t create the keyspace again, it will be there already, but with no data until you change the replication factor for the keyspace. Once you have done this you will need to do a nodetool rebuild on the new nodes to stream the data to the new DC.

3) We kept auto_bootstrap to true, but only changed the replication factor of the system keyspaces before creating the new nodes, therefore the bootstrapping was quick, and the real data was streamed later during the nodetool rebuild.

4) We just kept the GossipingPropertyFileSnitch, this allows you the most flexibility, and allows you to change cloud providers later.

We presented how we did this at Datastax Accelerate earlier in the year, and although that presentation is not yet online, I also created a series of blogs posts that shows our steps in detail, I recommend reading these as it is the combined learnings of moving 90+ clusters: http://www.redshots.com/accelerate/ <http://www.redshots.com/accelerate/>

Happy to answer any more questions.

Regards 

Paul Chandler
www.redshots.com

> On 26 Jun 2019, at 16:19, Voytek Jarnot <vo...@gmail.com> wrote:
> 
> I started an higher-level thread years ago about moving a cluster by expanding from 1 to 2 datacenters, replicating over, then decommissioning the original DC. Corporate plans being what they are, we're finally getting into this; I'm largely following the writeup here: https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html <https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html> , but have a few more-specific questions:
> 
> current setup: 1 DC, 4 nodes, RF=3, 1 keyspace
> new DC will be 4 nodes as well, RF=3
> 
> 1) We currently have 2 seed nodes, I'd like to confirm that the correct course is to make 1-2 (let's say 2) of the new-DC nodes seeds as well, and update all nodes in both DCs to point at all 4 seeds before I get into altering the keyspace.
> 
> 2) Prior to altering the replication on my keyspace to include the new DC, I do not need/want to create the keyspace in the new DC, correct?
> 
> 3) The datastax docs mention the auto_bootstrap=false setting, but don't go into much detail - I'm leaning toward setting it to false on all the new nodes, sound reasonable?
> 
> 4) One of the three environments in which this'll happen is slightly more complicated due to the existing DC living in AWS, whereas the new DC will be in a different AZ. Do I need to get into switching from GossipingPropertyFileSnitch to Ec2MultiRegionSnitch? If so, could someone shed a bit of light on that process, and the associated changes needed for listen_address and broadcast_address?
> 
> Thanks for getting this far,
> Voytek Jarnot