You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Voytek Jarnot <vo...@gmail.com> on 2017/05/02 01:02:03 UTC
Migrating a cluster
Have a scenario where it's necessary to migrate a cluster to a different
set of hardware with minimal downtime. Setup is:
Current cluster: 4 nodes, RF 3
New cluster: 6 nodes, RF 3
My initial inclination is to follow this writeup on setting up the 6 new
nodes as a new DC:
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html
Basically, set up new DC, nodetool rebuild on new nodes to instruct
Cassandra to migrate data, change client to hit new DC, kill original DC.
First question - is this the recommended way to migrate an in-use cluster
to new hardware?
Secondly, on the assumption that it is: That link gives the impression that
DC-aware clients will not hit the "remote" DC - is that the case for the
Java driver? We don't currently explicitly set PoolingOptions
ConnectionsPerHost for HostDistance.REMOTE to 0 - seems like that would be
an important thing to do?
Thank you.
Re: Migrating a cluster
Posted by Bhuvan Rawal <bh...@gmail.com>.
+1 to Justin's answer!
As an additional step it's always good to run a full repair before deleting
data on existing nodes, as there is a possibility of ioexceptions during
rebuild. (Things like https://issues.apache.org/jira/browse/CASSANDRA-12830)
Also if you are on 3.8+ , you may go for CDC approach and instead of adding
a dc you can instead create a new cluster. Though this will involve some
downtime. Probable steps in that case:
1. Create cluster with new hardware machines
2. Migrate existing sstables
3. Bring app down & load CDC data into new cluster for the time elapsed
during step 2
4. Bring app up
On 02-May-2017 7:08 AM, "Justin Cameron" <ju...@instaclustr.com> wrote:
Yes - this is the recommended way to migrate to another DC.
Before you start the migration you'll need to ensure
1. that the replication strategy of all your keyspaces is
NetworkTopologyStrategy (if not, change it to this using ALTER KEYSPACE),
and
2. that each of your clients is using the DcAwareRoundRobinPolicy load
balancing policy, and that the localDc parameter is set to the name of your
existing data centre. https://github.com/datastax/java-driver/tree/3.x/
manual/load_balancing#dcawareroundrobinpolicy
In addition to points 1&2, in order to ensure that your clients do not
contact nodes in the new data centre, you will also need to use a LOCAL
consistency level for all your queries (e.g. LOCAL_QUORUM instead of QUORUM)
Cheers,
Justin
On Tue, 2 May 2017 at 11:02 Voytek Jarnot <vo...@gmail.com> wrote:
> Have a scenario where it's necessary to migrate a cluster to a different
> set of hardware with minimal downtime. Setup is:
>
> Current cluster: 4 nodes, RF 3
> New cluster: 6 nodes, RF 3
>
> My initial inclination is to follow this writeup on setting up the 6 new
> nodes as a new DC: https://docs.datastax.com/en/cassandra/3.0/cassandra/
> operations/opsAddDCToCluster.html
>
> Basically, set up new DC, nodetool rebuild on new nodes to instruct
> Cassandra to migrate data, change client to hit new DC, kill original DC.
>
> First question - is this the recommended way to migrate an in-use cluster
> to new hardware?
>
> Secondly, on the assumption that it is: That link gives the impression
> that DC-aware clients will not hit the "remote" DC - is that the case for
> the Java driver? We don't currently explicitly set PoolingOptions
> ConnectionsPerHost for HostDistance.REMOTE to 0 - seems like that would be
> an important thing to do?
>
> Thank you.
>
--
*Justin Cameron*Senior Software Engineer
<https://www.instaclustr.com/>
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.
Re: Migrating a cluster
Posted by Justin Cameron <ju...@instaclustr.com>.
Yes - this is the recommended way to migrate to another DC.
Before you start the migration you'll need to ensure
1. that the replication strategy of all your keyspaces is
NetworkTopologyStrategy (if not, change it to this using ALTER KEYSPACE),
and
2. that each of your clients is using the DcAwareRoundRobinPolicy load
balancing policy, and that the localDc parameter is set to the name of your
existing data centre.
https://github.com/datastax/java-driver/tree/3.x/manual/load_balancing#dcawareroundrobinpolicy
In addition to points 1&2, in order to ensure that your clients do not
contact nodes in the new data centre, you will also need to use a LOCAL
consistency level for all your queries (e.g. LOCAL_QUORUM instead of QUORUM)
Cheers,
Justin
On Tue, 2 May 2017 at 11:02 Voytek Jarnot <vo...@gmail.com> wrote:
> Have a scenario where it's necessary to migrate a cluster to a different
> set of hardware with minimal downtime. Setup is:
>
> Current cluster: 4 nodes, RF 3
> New cluster: 6 nodes, RF 3
>
> My initial inclination is to follow this writeup on setting up the 6 new
> nodes as a new DC:
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html
>
> Basically, set up new DC, nodetool rebuild on new nodes to instruct
> Cassandra to migrate data, change client to hit new DC, kill original DC.
>
> First question - is this the recommended way to migrate an in-use cluster
> to new hardware?
>
> Secondly, on the assumption that it is: That link gives the impression
> that DC-aware clients will not hit the "remote" DC - is that the case for
> the Java driver? We don't currently explicitly set PoolingOptions
> ConnectionsPerHost for HostDistance.REMOTE to 0 - seems like that would be
> an important thing to do?
>
> Thank you.
>
--
*Justin Cameron*Senior Software Engineer
<https://www.instaclustr.com/>
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.