You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Brandon McCauslin <bm...@gmail.com> on 2014/04/07 18:25:28 UTC
Migrating to new datacenter
We're currently running a small 5 node 2.0.5 cluster in a single datacenter
using the SimpleStrategy replication strategy with replication factor of 3.
We want to migrate our data from our current datacenter to a new
datacenter, without incurring any downtime or data loss. There is no plan
to maintain or keep the old (current) datacenter after the migration is
complete. I'm wondering how best to go about this migration.
*Shoud We:*
1. add a 2nd datacenter as described here - "Adding a datacenter to
cluster<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html>",
then decommission the 1st
datacenter<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_decomission_dc_t.html>.
Can we even do this given we're currently setup with SimpleStrategy
replication?
2. or should we just start adding new nodes from the new datacenter to
our existing cluster as described
here<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html>?
Then once we've added all the new nodes, start decommissioning all the old
nodes from the old/current datacenter as described
here<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html>
.
Thanks for your help and time!
Re: Migrating to new datacenter
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Apr 7, 2014 at 1:41 PM, Brandon McCauslin <bm...@gmail.com> wrote:
> If I read your response in that 1st URL correctly, it seems changing both
> the snitch and replication strategy at the same time is not advisable and
> could lead to partial data loss. Is your suggestion of dumping an
> reloading the data into the new cluster still recommended for these
> situations?
>
The snitch and the replication strategy are interrelated, in some cases you
must change both to change one.
It is possible to do this task in such a way that you eliminate the
potential of data loss. I know, because I've done it. The methodology
described with getendpoints is certainly more tractable if you are not
using vnodes... but it's still possible with vnodes.
It is simpler to do the dump and reload.
=Rob
Re: Migrating to new datacenter
Posted by Brandon McCauslin <bm...@gmail.com>.
Rob,
If I read your response in that 1st URL correctly, it seems changing both
the snitch and replication strategy at the same time is not advisable and
could lead to partial data loss. Is your suggestion of dumping an
reloading the data into the new cluster still recommended for these
situations?
On Mon, Apr 7, 2014 at 2:09 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Apr 7, 2014 at 10:48 AM, Brandon McCauslin <bm...@gmail.com>wrote:
>
>> Thanks for the confirmation on the approach. The new dc is not yet
>> ready, but while I'm waiting I was thinking about updating the existing
>> dc's replication strategy from "SimpleStrategy" to
>> "NetworkTopologyStrategy". I also assume I'll need to update my snitch
>> from the current SimpleSnitch to PropertyFileSnitch. Here's the steps I'm
>> thinking:
>>
>
> Here's a brief explication of how this process works and can be made
> safe/testable :
>
>
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Ec2Snitch-amp-Other-snitches-td6196494.html
>
> Consider whether you actually want a rack aware Snitch :
>
> https://issues.apache.org/jira/browse/CASSANDRA-3810
>
> If you do, consider using the Gossiping Property File Snitch :
>
> https://issues.apache.org/jira/browse/CASSANDRA-1974
>
> =Rob
>
>
Re: Migrating to new datacenter
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Apr 7, 2014 at 10:48 AM, Brandon McCauslin <bm...@gmail.com> wrote:
> Thanks for the confirmation on the approach. The new dc is not yet ready,
> but while I'm waiting I was thinking about updating the existing dc's
> replication strategy from "SimpleStrategy" to "NetworkTopologyStrategy". I
> also assume I'll need to update my snitch from the current SimpleSnitch to
> PropertyFileSnitch. Here's the steps I'm thinking:
>
Here's a brief explication of how this process works and can be made
safe/testable :
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Ec2Snitch-amp-Other-snitches-td6196494.html
Consider whether you actually want a rack aware Snitch :
https://issues.apache.org/jira/browse/CASSANDRA-3810
If you do, consider using the Gossiping Property File Snitch :
https://issues.apache.org/jira/browse/CASSANDRA-1974
=Rob
Re: Migrating to new datacenter
Posted by Brandon McCauslin <bm...@gmail.com>.
Thanks for the confirmation on the approach. The new dc is not yet ready,
but while I'm waiting I was thinking about updating the existing dc's
replication strategy from "SimpleStrategy" to "NetworkTopologyStrategy". I
also assume I'll need to update my snitch from the current SimpleSnitch to
PropertyFileSnitch. Here's the steps I'm thinking:
1. update snitch from SimpleSnitch to PropertyFileSnitch
- edit/create cassandra-topology.properties to enumerate all nodes
in current dc
- change snitch in cassandra.yaml on all nodes
- restart all nodes??
- run full repair on keyspace
2. update replication from "SimpleStrategy" to "NetworkTopologyStrategy"
a. ALTER KEYSPACE cfs
WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc1'
: 3};
b. nodetool repair cfs
Let me know if anything here sounds wrong. Thanks!
would like to start preparing the existing datacenter
wondering if I could start updating the existing datacenter's replication
staconfiguration
On Mon, Apr 7, 2014 at 12:45 PM, Mark Reddy <ma...@boxever.com> wrote:
> I would go with option 1.
>
> I think it is the safer of the two options, involves less work and if
> something were to go wrong mid migration you can remove the second DC from
> your keyspace replication and have a clean break. SimpleStrategy will work
> across DCs. It is generally advised to not use it across DCs as it is hard
> to maintain and can induce latency on writes depending on DC distance,
> however as you are just migrating and then killing the original DC you
> won't face these issues.
>
>
> Mark
>
>
> On Mon, Apr 7, 2014 at 5:25 PM, Brandon McCauslin <bm...@gmail.com>wrote:
>
>> We're currently running a small 5 node 2.0.5 cluster in a single
>> datacenter using the SimpleStrategy replication strategy with replication
>> factor of 3. We want to migrate our data from our current datacenter to a
>> new datacenter, without incurring any downtime or data loss. There is no
>> plan to maintain or keep the old (current) datacenter after the migration
>> is complete. I'm wondering how best to go about this migration.
>>
>> *Shoud We:*
>>
>> 1. add a 2nd datacenter as described here - "Adding a datacenter to
>> cluster<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html>",
>> then decommission the 1st datacenter<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_decomission_dc_t.html>.
>> Can we even do this given we're currently setup with SimpleStrategy
>> replication?
>> 2. or should we just start adding new nodes from the new datacenter
>> to our existing cluster as described here<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html>?
>> Then once we've added all the new nodes, start decommissioning all the old
>> nodes from the old/current datacenter as described here<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html>
>> .
>>
>> Thanks for your help and time!
>>
>
>
Re: Migrating to new datacenter
Posted by Mark Reddy <ma...@boxever.com>.
I would go with option 1.
I think it is the safer of the two options, involves less work and if
something were to go wrong mid migration you can remove the second DC from
your keyspace replication and have a clean break. SimpleStrategy will work
across DCs. It is generally advised to not use it across DCs as it is hard
to maintain and can induce latency on writes depending on DC distance,
however as you are just migrating and then killing the original DC you
won't face these issues.
Mark
On Mon, Apr 7, 2014 at 5:25 PM, Brandon McCauslin <bm...@gmail.com> wrote:
> We're currently running a small 5 node 2.0.5 cluster in a single
> datacenter using the SimpleStrategy replication strategy with replication
> factor of 3. We want to migrate our data from our current datacenter to a
> new datacenter, without incurring any downtime or data loss. There is no
> plan to maintain or keep the old (current) datacenter after the migration
> is complete. I'm wondering how best to go about this migration.
>
> *Shoud We:*
>
> 1. add a 2nd datacenter as described here - "Adding a datacenter to
> cluster<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html>",
> then decommission the 1st datacenter<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_decomission_dc_t.html>.
> Can we even do this given we're currently setup with SimpleStrategy
> replication?
> 2. or should we just start adding new nodes from the new datacenter to
> our existing cluster as described here<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html>?
> Then once we've added all the new nodes, start decommissioning all the old
> nodes from the old/current datacenter as described here<http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html>
> .
>
> Thanks for your help and time!
>