Best way to "split" cluster online

[Philippe <wa...@gmail.com>]

I have a cluster with 5 keyspaces. I would like to move one of the
keyspaces to a separate cluster because it has very different usage
patterns that can be optimized on different hardware.

What would be the best way to do that online ie. without interrupting reads
& writes. The keyspace is about 350GB on each of the 3 nodes.

Running cassandra 1.1.12, not upgrading to 1.2.x for now.

Thanks

Re: Best way to "split" cluster online

[Robert Coli <rc...@eventbrite.com>]

On Fri, Aug 2, 2013 at 12:46 PM, Philippe <wa...@gmail.com> wrote:

> I have a cluster with 5 keyspaces. I would like to move one of the
> keyspaces to a separate cluster because it has very different usage
> patterns that can be optimized on different hardware.
>
> What would be the best way to do that online ie. without interrupting
> reads & writes. The keyspace is about 350GB on each of the 3 nodes.
>

I don't know that I can assert that it is the "best" way but the "easiest"
way is as follows :

1) create a new cluster with same tokens and replication factor and
strategy as the old cluster (and same number of nodes)
2) create keyspace on new cluster
3) fork writes so that they go to both old and to new cluster
 4) flush/snapshot and rsync on-disk SSTables from source nodes (can be
done with source nodes live) to new nodes, BUT NOT INTO THE DATA DIRECTORY
[1], renaming them by inflating the numeric part of their filenames by
LARGE_NUMBER (f/e "1000") such that there is no chance of name collision
with the new node's newly flushed SSTables.

for each new node, one at a time :
5) drain and stop new node
6) move SSTables into data directory on new node
8) start new node

and finally :
9) de-fork writes

In theory you could probably "get away" with not restarting the new nodes
and using nodetool refresh, but it's more complicated to do so and there is
a nonzero risk of problems.

If you can interrupt writes while serving reads from the old cluster, the
process becomes significantly simpler. You just interrupt writes, flush,
copy this files to the new cluster and start. You don't have to avoid name
collision, because your new cluster is not taking write and is therefore
not flushing SSTables.

=Rob
[1] If you copy the files into the data directory in 4), especially without
inflating the numeric part of their name, they may be overwritten silently
by newly flushed ones.