Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.

[Francisco Nogueira Calmon Sobral <fs...@igcorp.com.br>]

We also run into the same problem when trying to load data from a 8 node cluster (C* 1.2.1, Vnodes and same rack) into a 9 node cluster (C* 1.2.11, Vnodes and different racks) using sstableloader.

We observed that a Key that clearly belonged to 3 nodes (using nodetool getendpoints) has not been sent to any of them. Why?

Best regards,
Francisco.



On Sep 24, 2013, at 7:12 PM, Suruchi Deodhar <su...@generalsentiment.com> wrote:

> As an update to this thread, we conducted several tests with Cassandra-1.2.9, varying parameters such as partitioner (Murmur3Partitioner/RandomParttioner), using NetworkToplogyStrategy (with Ec2Snitch) / SimpleStrategy (with SimpleSnitch) across 2 Availability zones and 1 AZ. We also tested the configurations separately with vnodes and without vnodes. 
> 
> Every time before each test, we wiped the cassandra cluster data and commitlog folders and restarted with an empty cassandra db. However, in all the cases using 1.2.9 we continued to see very heavy imbalance across the nodes as reported in this thread. 
> 
> We then tested the same exports with cassandra 1.2.5 version that we had been testing previously (without vnodes across 2 AZs) and the data was balanced across the nodes of the cluster. The output from bin/nodetool status is attached.
> 
> Was there some change from 1.2.5 to 1.2.9 that could be responsible for the imbalance or is there some parameter setting that we may have completely missed in our configuration wrt 1.2.9? Has anyone else experienced such an imbalance issue?
> 
> Also,  we were contemplating on using vnodes with NetworkTopologyStrategy (We want to replicate data across 2 AZs)
> We came across the following links that mention that vnodes with NetworkToplogyStrategy may create hotspots and the issue is marked as Open. Does that mean using vnodes with NetworkToplogyStrategy is a bad idea?
> 
> [ https://issues.apache.org/jira/browse/CASSANDRA-4658 , https://issues.apache.org/jira/browse/CASSANDRA-3810 , https://issues.apache.org/jira/browse/CASSANDRA-4123 ] .
> 
> Thanks again for all your replies. 
> 
> Suruchi
> 
>   
> 
> 
> 
> On Fri, Sep 20, 2013 at 7:04 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <su...@generalsentiment.com> wrote:
> Using the nodes in the same availability zone(us-east-1b), we still get a highly imbalanced cluster. The nodetool status and ring output is attached. Even after running repairs, the cluster does not seem to balance.
> 
> If your cluster doesn't experience exceptions when loading and/or store a lot of hints, repair is almost certainly just wasting your and your CPU's time.
> 
> =Rob 
> 
> <status_1.2.5_wo_vnodes_lz4.txt>