Property file snitch and unbalanced cluster

[Robert Sicoie <ro...@gmail.com>]

Hi guys,

I have a cluster with 5 nodes, cassandra 3.0.5, RF=2, PropertyFileSnitch.
Three of the nodes were added initially, DC=PRD. The other two were added
more recently.

The default for unknown nodes default=DC1:r1 (which is bad in my case, as
it would have been nice to have PRD as default)

As the three initial do not have the information about the later two in the
properties file, they see node4 and node5 as in DC1:r1.

I assume that:

   - 1-2-3 have mode data than 4 and 5 because all writes that come to
   1-2-3 are replicated only on 1-2-3 (as there is no schema for DC1) and this
   is why 1,2,3 have more data and the cluster is unbalanced.
   - all writes that come to 4 and 5 are replicated on 1,2,3,4 and 5 as the
   property file snitch config has info about the entire cluster on 4 and 5.

Is it save if I just update the network topology file on 1, 2, 3 and add
lines about node 4 and 5, and then roll restart 4 and 5? Will the cluster
balance itself? Currently 1, 2, 3 have about 16GB each and 4,5 about 6 GB
each.

Thanks,

Robert

Re: Property file snitch and unbalanced cluster

[Carlos Alonso <in...@mrcalonso.com>]

Hi Robert,

What has happened is that you have now two datacenters in your cluster. The
way they replicate information will depend on your keyspace settings.

Regarding your process I don't think it is safe to do it that way. I'd
start off by decommissioning nodes 4 and 5 so that your cluster is back to
1 datacenter with 3 nodes and then add them sequentially again making sure
the configuration in the Snitch is the proper one.

Regards.

Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>

On 28 October 2016 at 10:06, Robert Sicoie <ro...@gmail.com> wrote:

> Hi guys,
>
> I have a cluster with 5 nodes, cassandra 3.0.5, RF=2, PropertyFileSnitch.
> Three of the nodes were added initially, DC=PRD. The other two were added
> more recently.
>
> The default for unknown nodes default=DC1:r1 (which is bad in my case, as
> it would have been nice to have PRD as default)
>
> As the three initial do not have the information about the later two in
> the properties file, they see node4 and node5 as in DC1:r1.
>
> I assume that:
>
>    - 1-2-3 have mode data than 4 and 5 because all writes that come to
>    1-2-3 are replicated only on 1-2-3 (as there is no schema for DC1) and this
>    is why 1,2,3 have more data and the cluster is unbalanced.
>    - all writes that come to 4 and 5 are replicated on 1,2,3,4 and 5 as
>    the property file snitch config has info about the entire cluster on 4 and
>    5.
>
> Is it save if I just update the network topology file on 1, 2, 3 and add
> lines about node 4 and 5, and then roll restart 4 and 5? Will the cluster
> balance itself? Currently 1, 2, 3 have about 16GB each and 4,5 about 6 GB
> each.
>
> Thanks,
>
> Robert
>