Moving a cluster between networks.

[Tim Wintle <ti...@gmail.com>]

Hi,

Suppose we have two networks:

10.1.0.0/16 and 10.2.0.0/16.

It is not possible to route packets between the two networks, but all
nodes have interfaces on both networks, so any node can communicate with
any address on either network.

We are currently running our all nodes on one network, but would like to
move to the other network.

What would the best way to achieve this? (We can tolerate a fairly short
period of downtime).

I have previously attempted to add a new node to the second network,
hoping that (as it can communicate with all nodes via some NIC) this
would work, and we could decommission each node one by one before
bringing them back up using their addresses on the secondary network -
however adding a node in the second network to the cluster was not
successful.

We are running C* 1.0.12 if this makes a significant difference.

Tim

Re: Moving a cluster between networks.

[Robert Coli <rc...@eventbrite.com>]

On Fri, Aug 23, 2013 at 9:39 AM, Tim Wintle <ti...@gmail.com> wrote:

> I tried a quick test on local VMs before an it appeared to work, but I'm
> still a little worried if the old ip addresses would appear through some
> process that only kicks in in a realistic use.
>
> I'll try to set up a more realistic simulation to test before going
> ahead.
>

Did this work out for you on the real cluster?

=Rob

Re: Moving a cluster between networks.

[Tim Wintle <ti...@gmail.com>]

On Wed, 2013-08-21 at 10:42 -0700, Robert Coli wrote:
> On Wed, Aug 21, 2013 at 3:58 AM, Tim Wintle <ti...@gmail.com> wrote:
> 
> > What would the best way to achieve this? (We can tolerate a fairly short
> > period of downtime).
> >
> 
> I think this would work, but may require a full cluster shutdown.
> 
> 1) stop nodes on old network
> 2) set auto_bootstrap to false in the conf file (it's not in there, you
> will have to add it to set it to false)
> 3) change the listen_address/seed lists/etc. in cassandra.yaml to be the
> new ips
> 4) start nodes, seed nodes first

Thank you,

I tried a quick test on local VMs before an it appeared to work, but I'm
still a little worried if the old ip addresses would appear through some
process that only kicks in in a realistic use.

I'll try to set up a more realistic simulation to test before going
ahead.

Tim

> 
> Basically I think the nodes will join, announce their new ip, not
> bootstrap, and eventually the entire cluster will coalesce on new ips.
> 
> If I were you, I would probably try to set up a QA or test cluster with
> similar setup.
> 
> =Rob

Re: Moving a cluster between networks.

[Robert Coli <rc...@eventbrite.com>]

On Wed, Aug 21, 2013 at 3:58 AM, Tim Wintle <ti...@gmail.com> wrote:

> What would the best way to achieve this? (We can tolerate a fairly short
> period of downtime).
>

I think this would work, but may require a full cluster shutdown.

1) stop nodes on old network
2) set auto_bootstrap to false in the conf file (it's not in there, you
will have to add it to set it to false)
3) change the listen_address/seed lists/etc. in cassandra.yaml to be the
new ips
4) start nodes, seed nodes first

Basically I think the nodes will join, announce their new ip, not
bootstrap, and eventually the entire cluster will coalesce on new ips.

If I were you, I would probably try to set up a QA or test cluster with
similar setup.

=Rob