Create a smaller cluster based on snapshost

[John Wong <go...@gmail.com>]

Hi.

Supposed I have a 6-node cluster running and I want to build a 3-node
cluster based on that 6-node cluster. What is the recommended way to
quickly build such cluster? Each node is about 120Gb and we have RF=3. We
are on Cassandra 1.2.19 and we are not using vnode.

My initial research shows it can either be done with sstableloader or
restore using snapshot and fix the token range.

In the case of sstableloader, given it is streaming, and we are restoring
from a live server, this seems to be a slow process if we throttle the
traffic. Even if I take this route, do I just pick 3 node out of the 6
nodes in any random order?

In the case of restoring from snapshots I have restored a 6-node replica
with just copying snapshot files (along with schema files), run nodetool
refresh, and should be able to complete in a few hours. But now with
smaller replica, do I again just pick snapshots from any 3 nodes? What and
why do I need to fix token range (from what I read)?

Any feedback is appreciated.

Thanks.

John

Re: Create a smaller cluster based on snapshost

[Robert Coli <rc...@eventbrite.com>]

On Sun, Jun 21, 2015 at 8:25 PM, John Wong <go...@gmail.com> wrote:

> In the case of restoring from snapshots I have restored a 6-node replica
> with just copying snapshot files (along with schema files), run nodetool
> refresh, and should be able to complete in a few hours. But now with
> smaller replica, do I again just pick snapshots from any 3 nodes? What and
> why do I need to fix token range (from what I read)?
>

Unless you have large data sizes, the easiest way to do this process
generically is :

1) copy all sstables from all source nodes to all target nodes, avoiding
namespace collision between files from different nodes.
2) start target nodes with data in place and appropriate tokens for the new
cluster (don't bother using refresh, you can just : a) start target cluster
with no data b) create schema c) stop target cluster nodes d) start target
cluster nodes)

As a small caveat, this process assumes that you can deal with the on-disk
bloat in the target cluster until compaction merges. The bloat comes from
having multiple replicas of the same data collapsed onto the same node.

Otherwise, the simplest is to :

1) not use vnodes
2) cut your cluster in half

You accomplish this by copying the data from two source nodes to one target
node, avoiding name collisions.

A B C D E F
AB CD EF

AB has the token of A, CD has the token of C, EF has the token of E.

Because no remaining nodes lose ranges, one does not need to run cleanup.

=Rob

Re: Create a smaller cluster based on snapshost

[Vitalii Skakun <vi...@gmail.com>]

Hi,

I'm aware of at least 2 ways:
1. create 6 node cluster, restore snapshot to it from original cluster; and
reduce this second cluster to 3 nodes.
in this case, you should be careful in picking nodes to be decommissioned
to avoid additional rebalance of cluster and changing tokens.

2. prepare the original cluster for multi-DC configuration; add second DC
with 3 nodes.
in this case, you can also switch to vnodes in 2nd, new DC, which is good
idea

hope this helps.



2015-06-22 6:25 GMT+03:00 John Wong <go...@gmail.com>:

> Hi.
>
> Supposed I have a 6-node cluster running and I want to build a 3-node
> cluster based on that 6-node cluster. What is the recommended way to
> quickly build such cluster? Each node is about 120Gb and we have RF=3. We
> are on Cassandra 1.2.19 and we are not using vnode.
>
> My initial research shows it can either be done with sstableloader or
> restore using snapshot and fix the token range.
>
> In the case of sstableloader, given it is streaming, and we are restoring
> from a live server, this seems to be a slow process if we throttle the
> traffic. Even if I take this route, do I just pick 3 node out of the 6
> nodes in any random order?
>
> In the case of restoring from snapshots I have restored a 6-node replica
> with just copying snapshot files (along with schema files), run nodetool
> refresh, and should be able to complete in a few hours. But now with
> smaller replica, do I again just pick snapshots from any 3 nodes? What and
> why do I need to fix token range (from what I read)?
>
> Any feedback is appreciated.
>
> Thanks.
>
> John
>



-- 

----
with best regards,
Vitalii Skakun