You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jayadev Jayaraman <jd...@gmail.com> on 2013/09/19 03:06:13 UTC
Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Basic configuration :
version=1.2.9
partitioner=Murmur3Partitioner
compaction strategy = LeveledCompactionStrategy
Cluster = 24 m1.xlarge nodes on EC2 ( 1.7 TB ephemeral raid0 per node )
replication factor = 2
snitch = EC2Snitch
placement_strategy = NetworkTopologyStrategy ( 12 nodes each in zones
us-east-1a , us-east-1b )
We use vnodes with num_tokens = 256 ( 256 tokens per node ) . After loading
some data with sstableloader , we find that the cluster is heavily
imbalanced :
$ bin/nodetool -h localhost status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.238.133.174 452.24 KB 256 8.1%
f2dc02a5-f9a9-43b4-85d6-dbf3c87b044b 1a
UN 10.238.133.97 485.75 KB 256 5.9%
b881777c-121d-4afb-adf5-0328386e901e 1a
UN 10.151.86.146 120.78 KB 256 0.5%
34d83902-d4e3-4245-a20c-4e829bb4020a 1a
UN 10.138.10.9 355.19 KB 256 5.4%
6f9868bf-fc87-4c0e-8e73-f02cf49f6ebe 1a
UN 10.87.87.240 393.25 KB 256 7.2%
4eb6f0c9-f30a-4615-ab89-e86e0ab84bf9 1b
UN 10.93.5.157 3.39 MB 256 12.6%
4f913ba4-e318-4080-b76f-2ad262ad5c7a 1b
UN 10.238.137.250 3.81 MB 256 27.5%
7d0c8eca-d0ed-40dd-aa5f-2b798a8225d3 1a
UN 10.92.231.170 196.5 KB 256 2.2%
4fe0ca5b-d58c-4376-8894-01ed4282bad8 1b
UN 10.93.31.44 162.36 KB 256 0.3%
83e1a28d-1080-4efe-9110-7f2ee4ac8128 1b
UN 10.138.2.20 375.86 KB 256 8.5%
a55f4480-3e17-4759-bcfc-04c04a71aa6b 1a
UN 10.93.91.139 760.45 KB 256 17.1%
ee373e42-63e4-4789-8532-cdd49dae5998 1b
UN 10.236.138.169 303.75 KB 256 5.5%
1fd7e817-1d4b-4abd-89cf-956f295679e6 1a
UN 10.137.7.90 4.27 GB 256 18.4%
a8d26ff0-8d41-406e-b724-8ddef29511b2 1a
UN 10.93.77.166 1.32 MB 256 27.8%
7bc043a7-863b-4230-b4fb-e6765bafcda3 1b
UN 10.120.249.140 4.27 GB 256 7.8%
10bb54e6-f597-4100-9408-070c45fdd449 1b
UN 10.123.95.248 608.91 KB 256 8.6%
8ed80942-9fb9-4c2f-b951-5c7648998a8a 1b
UN 10.90.246.128 401.34 KB 256 6.0%
485d3c01-e9b2-4172-9bbc-f0af86962570 1b
UN 10.136.11.40 447.6 KB 256 9.0%
3adb2d0a-810c-475c-bb2f-89a15a53a234 1a
UN 10.87.90.42 241.55 KB 256 3.8%
749730a2-121a-4d48-bef4-588c7df234ab 1b
UN 10.87.75.147 339.66 KB 256 4.6%
031baedf-e22d-41a9-b196-af64b55129ba 1b
UN 10.151.49.88 438.69 KB 256 4.2%
365d2dd3-d72c-4dd2-a833-db766e183fcc 1a
UN 10.87.83.107 170.49 KB 256 2.2%
d42a015b-e3db-42a5-9895-66d915a3e4c7 1b
UN 10.238.170.159 491.37 KB 256 6.5%
fd979cc7-d1d5-4c75-9608-6de861b0761b 1a
UN 10.137.20.183 163.06 KB 256 0.5%
361749d8-6a37-4f65-91cd-70f3222a762e 1a
Some nodes claim to own 27.5% of the data, while some nodes claim less than
1% . Also, the loads on each node don't tally with the ownership
percentages ( e.g. 10.137.7.90 has a load of 4.27 GB and owns 18.4% while
10.238.137.250 has a load of 3.81 MB and owns 27.5% ) .
Can someone help me figure out if I'm missing out on something ? Even if
the data distribution isn't perfectly even, it definitely shouldn't be this
far off.
I've attached the output of the "nodetool ring" command in a file here if
it helps.
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Richard Low <ri...@wentnet.com>.
On 19 September 2013 20:36, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> Thanks for your replies. I wiped out my data from the cluster and also
> cleared the commitlog before restarting it with num_tokens=256. I then
> uploaded data using sstableloader.
>
> However, I am still not able to see a uniform distribution of data across
> nodes of the clusters.
>
> The output of the bin/nodetool -h localhost status commands looks like
> follows. Some nodes have data as low as 1.12MB while some have as high as
> 912.57 MB.
>
Now the 'Owns (effective)' column is showing the tokens are roughly
balanced. So now the problem is the data isn't uniform - either you have
some rows much larger than others or some nodes are missing data that could
be replicated by running repair.
Richard.
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Mohit Anchlia <mo...@gmail.com>.
Did you start out your cluster after wiping all the sstables and commit
logs?
On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> We have been trying to resolve this issue to find a stable configuration
> that can give us a balanced cluster with equally distributed data across
> the nodes.
>
> We tried running a smaller 12 node cluster with following parameters:
>
> placement_strategy = NetworkTopologyStrategy ( 12 nodes in us-east-1b )
> partitioner=Murmur3Partitioner
> compaction strategy = LeveledCompactionStrategy
> replication factor = 2
> snitch = EC2Snitch
> vnodes with num_tokens = 256 ( 256 tokens per node )
>
> Using the nodes in the same availability zone(us-east-1b), we still get a
> highly imbalanced cluster. The nodetool status and ring output is attached.
> Even after running repairs, the cluster does not seem to balance.
>
> Datacenter: us-east
>> ===================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN 10.120.249.140 641.11 KB 256 16.7%
>> 90320ebc-cd03-4303-9f91-a8eb534ecaa4 1b
>> UN 10.90.246.128 1.04 MB 256 16.4%
>> 14352484-07c2-4cf0-9c12-e53d9e19f23b 1b
>> UN 10.123.95.248 4.21 GB 256 17.7%
>> 3d520c2a-f9a0-480a-83d8-675ed3f46c7d 1b
>> UN 10.87.87.240 337.46 KB 256 16.6%
>> 11bfebe6-f40c-4d4e-a76e-f6287a27fa53 1b
>> UN 10.93.5.157 4.21 GB 256 17.0%
>> 5a20ca7b-37fa-4e5b-86d4-c47c80e3d803 1b
>> UN 10.92.231.170 1021.72 KB 256 15.5%
>> e4835262-e27c-4da3-b45d-758a860f506e 1b
>> UN 10.93.31.44 653.58 KB 256 16.5%
>> 19d24eda-b9bf-42cc-8463-5be71d28da9d 1b
>> UN 10.93.91.139 298.81 KB 256 17.8%
>> 6d8d5879-4340-4d7c-960d-a12a80a9fac3 1b
>> UN 10.87.90.42 3.75 MB 256 16.8%
>> ff64cfc0-d614-48b9-8e96-aee99114c6ff 1b
>> UN 10.87.75.147 3.55 MB 256 16.6%
>> ec1f182d-bb01-4546-a523-2a42df158ee0 1b
>> UN 10.93.77.166 333.88 KB 256 16.0%
>> 0964902c-736d-411a-bed5-341dbc96813f 1b
>> UN 10.87.83.107 543.2 KB 256 16.4%
>> 924fa50c-d253-41e8-b2fb-d1be2942f2c8 1b
>>
>> After repairs- status
>>
>
> Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 10.120.249.140 82.26 MB 256 16.7%
> 90320ebc-cd03-4303-9f91-a8eb534ecaa4 1b
> UN 10.90.246.128 97.61 MB 256 16.4%
> 14352484-07c2-4cf0-9c12-e53d9e19f23b 1b
> UN 10.123.95.248 4.25 GB 256 17.7%
> 3d520c2a-f9a0-480a-83d8-675ed3f46c7d 1b
> UN 10.87.87.240 95.05 MB 256 16.6%
> 11bfebe6-f40c-4d4e-a76e-f6287a27fa53 1b
> UN 10.93.5.157 4.25 GB 256 17.0%
> 5a20ca7b-37fa-4e5b-86d4-c47c80e3d803 1b
> UN 10.92.231.170 86.54 MB 256 15.5%
> e4835262-e27c-4da3-b45d-758a860f506e 1b
> UN 10.93.31.44 102.42 MB 256 16.5%
> 19d24eda-b9bf-42cc-8463-5be71d28da9d 1b
> UN 10.93.91.139 82.85 MB 256 17.8%
> 6d8d5879-4340-4d7c-960d-a12a80a9fac3 1b
> UN 10.87.90.42 134.76 MB 256 16.8%
> ff64cfc0-d614-48b9-8e96-aee99114c6ff 1b
> UN 10.87.75.147 101.15 MB 256 16.6%
> ec1f182d-bb01-4546-a523-2a42df158ee0 1b
> UN 10.93.77.166 73.66 MB 256 16.0%
> 0964902c-736d-411a-bed5-341dbc96813f 1b
> UN 10.87.83.107 93.73 MB 256 16.4%
> 924fa50c-d253-41e8-b2fb-d1be2942f2c8 1b
>
>
> So essentially, using NetworkTopology strategy even with a single AZ is
> not giving us a balanced cluster. Is there any alternative that we can try
> for a stable cluster? We want to use vnodes.
>
> Thanks,
> Suruchi
>
>
> On Fri, Sep 20, 2013 at 1:31 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman <jd...@gmail.com>wrote:
>>
>>> As a follow-up, is operating a Cassandra cluster with machines on
>>> multiple racks and vnodes bound to cause load imbalance ? Shouldn't
>>> token-ranges assigned to individual machines via their vnodes be
>>> approximately balanced ? We're otherwise unable to explain why this
>>> imbalance occurs. ( it shouldn't be the fault of the Murmur3 partitioner
>>> which guarantees a uniform distribution of keys across token-ranges
>>> according to the doc. )
>>>
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-4658
>> https://issues.apache.org/jira/browse/CASSANDRA-4123
>>
>> tl;dr : Yes, vnodes makes the imbalance factor slightly greater, but in
>> practice it "shouldn't" be significant.
>>
>> If you are experiencing significant imbalance, one or both of those
>> tickets may be an appropriate forum to describe your experience.
>>
>> =Rob
>>
>>
>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Francisco Nogueira Calmon Sobral <fs...@igcorp.com.br>.
We also run into the same problem when trying to load data from a 8 node cluster (C* 1.2.1, Vnodes and same rack) into a 9 node cluster (C* 1.2.11, Vnodes and different racks) using sstableloader.
We observed that a Key that clearly belonged to 3 nodes (using nodetool getendpoints) has not been sent to any of them. Why?
Best regards,
Francisco.
On Sep 24, 2013, at 7:12 PM, Suruchi Deodhar <su...@generalsentiment.com> wrote:
> As an update to this thread, we conducted several tests with Cassandra-1.2.9, varying parameters such as partitioner (Murmur3Partitioner/RandomParttioner), using NetworkToplogyStrategy (with Ec2Snitch) / SimpleStrategy (with SimpleSnitch) across 2 Availability zones and 1 AZ. We also tested the configurations separately with vnodes and without vnodes.
>
> Every time before each test, we wiped the cassandra cluster data and commitlog folders and restarted with an empty cassandra db. However, in all the cases using 1.2.9 we continued to see very heavy imbalance across the nodes as reported in this thread.
>
> We then tested the same exports with cassandra 1.2.5 version that we had been testing previously (without vnodes across 2 AZs) and the data was balanced across the nodes of the cluster. The output from bin/nodetool status is attached.
>
> Was there some change from 1.2.5 to 1.2.9 that could be responsible for the imbalance or is there some parameter setting that we may have completely missed in our configuration wrt 1.2.9? Has anyone else experienced such an imbalance issue?
>
> Also, we were contemplating on using vnodes with NetworkTopologyStrategy (We want to replicate data across 2 AZs)
> We came across the following links that mention that vnodes with NetworkToplogyStrategy may create hotspots and the issue is marked as Open. Does that mean using vnodes with NetworkToplogyStrategy is a bad idea?
>
> [ https://issues.apache.org/jira/browse/CASSANDRA-4658 , https://issues.apache.org/jira/browse/CASSANDRA-3810 , https://issues.apache.org/jira/browse/CASSANDRA-4123 ] .
>
> Thanks again for all your replies.
>
> Suruchi
>
>
>
>
>
> On Fri, Sep 20, 2013 at 7:04 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <su...@generalsentiment.com> wrote:
> Using the nodes in the same availability zone(us-east-1b), we still get a highly imbalanced cluster. The nodetool status and ring output is attached. Even after running repairs, the cluster does not seem to balance.
>
> If your cluster doesn't experience exceptions when loading and/or store a lot of hints, repair is almost certainly just wasting your and your CPU's time.
>
> =Rob
>
> <status_1.2.5_wo_vnodes_lz4.txt>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Suruchi Deodhar <su...@generalsentiment.com>.
As an update to this thread, we conducted several tests with
Cassandra-1.2.9, varying parameters such as partitioner
(Murmur3Partitioner/RandomParttioner), using NetworkToplogyStrategy (with
Ec2Snitch) / SimpleStrategy (with SimpleSnitch) across 2 Availability zones
and 1 AZ. We also tested the configurations separately with vnodes and
without vnodes.
Every time before each test, we wiped the cassandra cluster data and
commitlog folders and restarted with an empty cassandra db. However, in all
the cases using 1.2.9 we continued to see very heavy imbalance across the
nodes as reported in this thread.
We then tested the same exports with cassandra 1.2.5 version that we had
been testing previously (without vnodes across 2 AZs) and the data was
balanced across the nodes of the cluster. The output from bin/nodetool
status is attached.
Was there some change from 1.2.5 to 1.2.9 that could be responsible for the
imbalance or is there some parameter setting that we may have completely
missed in our configuration wrt 1.2.9? Has anyone else experienced such an
imbalance issue?
Also, we were contemplating on using vnodes with NetworkTopologyStrategy
(We want to replicate data across 2 AZs)
We came across the following links that mention that vnodes with
NetworkToplogyStrategy may create hotspots and the issue is marked as Open.
Does that mean using vnodes with NetworkToplogyStrategy is a bad idea?
[ https://issues.apache.org/jira/browse/CASSANDRA-4658 ,
https://issues.apache.org/jira/browse/CASSANDRA-3810 ,
https://issues.apache.org/jira/browse/CASSANDRA-4123 ] .
Thanks again for all your replies.
Suruchi
On Fri, Sep 20, 2013 at 7:04 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <
> suruchi.deodhar@generalsentiment.com> wrote:
>
>> Using the nodes in the same availability zone(us-east-1b), we still get a
>> highly imbalanced cluster. The nodetool status and ring output is attached.
>> Even after running repairs, the cluster does not seem to balance.
>>
>
> If your cluster doesn't experience exceptions when loading and/or store a
> lot of hints, repair is almost certainly just wasting your and your CPU's
> time.
>
> =Rob
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> Using the nodes in the same availability zone(us-east-1b), we still get a
> highly imbalanced cluster. The nodetool status and ring output is attached.
> Even after running repairs, the cluster does not seem to balance.
>
If your cluster doesn't experience exceptions when loading and/or store a
lot of hints, repair is almost certainly just wasting your and your CPU's
time.
=Rob
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Suruchi Deodhar <su...@generalsentiment.com>.
We have been trying to resolve this issue to find a stable configuration
that can give us a balanced cluster with equally distributed data across
the nodes.
We tried running a smaller 12 node cluster with following parameters:
placement_strategy = NetworkTopologyStrategy ( 12 nodes in us-east-1b )
partitioner=Murmur3Partitioner
compaction strategy = LeveledCompactionStrategy
replication factor = 2
snitch = EC2Snitch
vnodes with num_tokens = 256 ( 256 tokens per node )
Using the nodes in the same availability zone(us-east-1b), we still get a
highly imbalanced cluster. The nodetool status and ring output is attached.
Even after running repairs, the cluster does not seem to balance.
Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 10.120.249.140 641.11 KB 256 16.7%
> 90320ebc-cd03-4303-9f91-a8eb534ecaa4 1b
> UN 10.90.246.128 1.04 MB 256 16.4%
> 14352484-07c2-4cf0-9c12-e53d9e19f23b 1b
> UN 10.123.95.248 4.21 GB 256 17.7%
> 3d520c2a-f9a0-480a-83d8-675ed3f46c7d 1b
> UN 10.87.87.240 337.46 KB 256 16.6%
> 11bfebe6-f40c-4d4e-a76e-f6287a27fa53 1b
> UN 10.93.5.157 4.21 GB 256 17.0%
> 5a20ca7b-37fa-4e5b-86d4-c47c80e3d803 1b
> UN 10.92.231.170 1021.72 KB 256 15.5%
> e4835262-e27c-4da3-b45d-758a860f506e 1b
> UN 10.93.31.44 653.58 KB 256 16.5%
> 19d24eda-b9bf-42cc-8463-5be71d28da9d 1b
> UN 10.93.91.139 298.81 KB 256 17.8%
> 6d8d5879-4340-4d7c-960d-a12a80a9fac3 1b
> UN 10.87.90.42 3.75 MB 256 16.8%
> ff64cfc0-d614-48b9-8e96-aee99114c6ff 1b
> UN 10.87.75.147 3.55 MB 256 16.6%
> ec1f182d-bb01-4546-a523-2a42df158ee0 1b
> UN 10.93.77.166 333.88 KB 256 16.0%
> 0964902c-736d-411a-bed5-341dbc96813f 1b
> UN 10.87.83.107 543.2 KB 256 16.4%
> 924fa50c-d253-41e8-b2fb-d1be2942f2c8 1b
>
> After repairs- status
>
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.120.249.140 82.26 MB 256 16.7%
90320ebc-cd03-4303-9f91-a8eb534ecaa4 1b
UN 10.90.246.128 97.61 MB 256 16.4%
14352484-07c2-4cf0-9c12-e53d9e19f23b 1b
UN 10.123.95.248 4.25 GB 256 17.7%
3d520c2a-f9a0-480a-83d8-675ed3f46c7d 1b
UN 10.87.87.240 95.05 MB 256 16.6%
11bfebe6-f40c-4d4e-a76e-f6287a27fa53 1b
UN 10.93.5.157 4.25 GB 256 17.0%
5a20ca7b-37fa-4e5b-86d4-c47c80e3d803 1b
UN 10.92.231.170 86.54 MB 256 15.5%
e4835262-e27c-4da3-b45d-758a860f506e 1b
UN 10.93.31.44 102.42 MB 256 16.5%
19d24eda-b9bf-42cc-8463-5be71d28da9d 1b
UN 10.93.91.139 82.85 MB 256 17.8%
6d8d5879-4340-4d7c-960d-a12a80a9fac3 1b
UN 10.87.90.42 134.76 MB 256 16.8%
ff64cfc0-d614-48b9-8e96-aee99114c6ff 1b
UN 10.87.75.147 101.15 MB 256 16.6%
ec1f182d-bb01-4546-a523-2a42df158ee0 1b
UN 10.93.77.166 73.66 MB 256 16.0%
0964902c-736d-411a-bed5-341dbc96813f 1b
UN 10.87.83.107 93.73 MB 256 16.4%
924fa50c-d253-41e8-b2fb-d1be2942f2c8 1b
So essentially, using NetworkTopology strategy even with a single AZ is not
giving us a balanced cluster. Is there any alternative that we can try for
a stable cluster? We want to use vnodes.
Thanks,
Suruchi
On Fri, Sep 20, 2013 at 1:31 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman <jd...@gmail.com>wrote:
>
>> As a follow-up, is operating a Cassandra cluster with machines on
>> multiple racks and vnodes bound to cause load imbalance ? Shouldn't
>> token-ranges assigned to individual machines via their vnodes be
>> approximately balanced ? We're otherwise unable to explain why this
>> imbalance occurs. ( it shouldn't be the fault of the Murmur3 partitioner
>> which guarantees a uniform distribution of keys across token-ranges
>> according to the doc. )
>>
>
> https://issues.apache.org/jira/browse/CASSANDRA-4658
> https://issues.apache.org/jira/browse/CASSANDRA-4123
>
> tl;dr : Yes, vnodes makes the imbalance factor slightly greater, but in
> practice it "shouldn't" be significant.
>
> If you are experiencing significant imbalance, one or both of those
> tickets may be an appropriate forum to describe your experience.
>
> =Rob
>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman <jd...@gmail.com>wrote:
> As a follow-up, is operating a Cassandra cluster with machines on multiple
> racks and vnodes bound to cause load imbalance ? Shouldn't token-ranges
> assigned to individual machines via their vnodes be approximately balanced
> ? We're otherwise unable to explain why this imbalance occurs. ( it
> shouldn't be the fault of the Murmur3 partitioner which guarantees a
> uniform distribution of keys across token-ranges according to the doc. )
>
https://issues.apache.org/jira/browse/CASSANDRA-4658
https://issues.apache.org/jira/browse/CASSANDRA-4123
tl;dr : Yes, vnodes makes the imbalance factor slightly greater, but in
practice it "shouldn't" be significant.
If you are experiencing significant imbalance, one or both of those tickets
may be an appropriate forum to describe your experience.
=Rob
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Mohit Anchlia <mo...@gmail.com>.
Like I said in my previous reply that I am not sure if that is the
problem and that's why I thought it would be a good test to do your test
with cluster in one RACK only.
I'll take a look at your ring output today. Did you also post cfstats
output?
On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman <jd...@gmail.com>wrote:
> As a follow-up, is operating a Cassandra cluster with machines on multiple
> racks and vnodes bound to cause load imbalance ? Shouldn't token-ranges
> assigned to individual machines via their vnodes be approximately balanced
> ? We're otherwise unable to explain why this imbalance occurs. ( it
> shouldn't be the fault of the Murmur3 partitioner which guarantees a
> uniform distribution of keys across token-ranges according to the doc. )
>
>
> On Thu, Sep 19, 2013 at 8:56 PM, Jayadev Jayaraman <jd...@gmail.com>wrote:
>
>> We're using NetworkTopologyStrategy for our placement_strategy and our
>> primary motivation for placing nodes on more than 1 rack is to prevent
>> showstoppers caused by localized faults leading to multiple EC2 instances
>> in the same availability zone (rack) becoming unreachable (which we have
>> encountered before).
>>
>> I've attached the output of "nodetool ring" here.
>>
>>
>> On Thu, Sep 19, 2013 at 8:35 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>
>>> Other thing I noticed is that you are using mutiple RACKS and that might
>>> be contributing factor to it. However, I am not sure.
>>>
>>> Can you paste the output of nodetool cfstats and ring? Is it possible to
>>> run the same test but keeping all the nodes in one rack?
>>>
>>> I think you should open a JIRA if you are able to reproduce this.
>>>
>>> On Thu, Sep 19, 2013 at 4:41 PM, Jayadev Jayaraman <jd...@gmail.com>wrote:
>>>
>>>> We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted
>>>> cassandra and this is what we get for nodetool status :
>>>>
>>>> bin/nodetool -h localhost status
>>>> Datacenter: us-east
>>>> ===================
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> -- Address Load Tokens Owns (effective) Host ID
>>>> Rack
>>>> UN 10.238.133.174 885.36 MB 256 8.4%
>>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>>> UN 10.238.133.97 468.66 MB 256 7.7%
>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>>> UN 10.151.86.146 1.08 GB 256 8.0%
>>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>>> UN 10.138.10.9 941.44 MB 256 8.6%
>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>>> UN 10.87.87.240 99.69 MB 256 8.6%
>>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>>> UN 10.93.5.157 87.44 MB 256 7.6%
>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>>> UN 10.238.137.250 561.42 MB 256 7.8%
>>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>>> UN 10.92.231.170 893.75 MB 256 9.3%
>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>>> UN 10.138.2.20 31.89 MB 256 7.9%
>>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>>> UN 10.93.31.44 312.52 MB 256 7.8%
>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>>> UN 10.93.91.139 30.46 MB 256 8.1%
>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>>> UN 10.236.138.169 260.15 MB 256 9.1%
>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>>> UN 10.137.7.90 38.45 MB 256 7.4%
>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>>> UN 10.93.77.166 867.15 MB 256 8.8%
>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>>> UN 10.120.249.140 863.98 MB 256 9.4%
>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>>> UN 10.90.246.128 242.63 MB 256 8.4%
>>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>>> UN 10.123.95.248 171.51 MB 256 7.2%
>>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>>> UN 10.136.11.40 33.8 MB 256 8.5%
>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>>> UN 10.87.90.42 38.01 MB 256 8.0%
>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>>> UN 10.87.75.147 579.29 MB 256 8.3%
>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>>> UN 10.151.49.88 151.06 MB 256 8.9%
>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>>> UN 10.87.83.107 512.91 MB 256 8.3%
>>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>>> UN 10.238.170.159 85.04 MB 256 9.4%
>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>>> UN 10.137.20.183 167.41 MB 256 8.4%
>>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>>>
>>>> It doesn't seem to have changed by much. The loads are still highly
>>>> uneven.
>>>>
>>>> As for the number of keys in each node's CFs : the largest node now
>>>> has 5589120 keys for the column-family that had 6527744 keys before
>>>> (load is now 1.08 GB as compares to 1.05 GB before), while the smallest
>>>> node now has 71808 keys as compared to 3840 keys before (load is now
>>>> 31.89 MB as compares to 1.12 MB before).
>>>>
>>>>
>>>> On Thu, Sep 19, 2013 at 5:18 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>>>
>>>>> Can you run nodetool repair on all the nodes first and look at the
>>>>> keys?
>>>>>
>>>>>
>>>>> On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>
>>>>>> Yes, the key distribution does vary across the nodes. For example, on
>>>>>> the node with the highest data, Number of Keys (estimate) is 6527744 for a
>>>>>> particular column family, whereas for the same column family on the node
>>>>>> with least data, Number of Keys (estimate) = 3840.
>>>>>>
>>>>>> Is there a way to control this distribution by setting some parameter
>>>>>> of cassandra.
>>>>>>
>>>>>> I am using the Murmur3 partitioner with NetworkTopologyStrategy.
>>>>>>
>>>>>> Thanks,
>>>>>> Suruchi
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <
>>>>>> mohitanchlia@gmail.com> wrote:
>>>>>>
>>>>>>> Can you check cfstats to see number of keys per node?
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
>>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>>
>>>>>>>> Thanks for your replies. I wiped out my data from the cluster and
>>>>>>>> also cleared the commitlog before restarting it with num_tokens=256. I then
>>>>>>>> uploaded data using sstableloader.
>>>>>>>>
>>>>>>>> However, I am still not able to see a uniform distribution of data
>>>>>>>> across nodes of the clusters.
>>>>>>>>
>>>>>>>> The output of the bin/nodetool -h localhost status commands looks
>>>>>>>> like follows. Some nodes have data as low as 1.12MB while some have as high
>>>>>>>> as 912.57 MB.
>>>>>>>>
>>>>>>>> Datacenter: us-east
>>>>>>>> ===================
>>>>>>>> Status=Up/Down
>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>> -- Address Load Tokens Owns (effective) Host
>>>>>>>> ID Rack
>>>>>>>> UN 10.238.133.174 856.66 MB 256 8.4%
>>>>>>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>>>>>>> UN 10.238.133.97 439.02 MB 256 7.7%
>>>>>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>>>>>>> UN 10.151.86.146 1.05 GB 256 8.0%
>>>>>>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>>>>>>> UN 10.138.10.9 912.57 MB 256 8.6%
>>>>>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>>>>>>> UN 10.87.87.240 70.85 MB 256 8.6%
>>>>>>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>>>>>>> UN 10.93.5.157 60.56 MB 256 7.6%
>>>>>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>>>>>>> UN 10.92.231.170 866.73 MB 256 9.3%
>>>>>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>>>>>>> UN 10.238.137.250 533.77 MB 256 7.8%
>>>>>>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>>>>>>> UN 10.93.91.139 478.45 KB 256 8.1%
>>>>>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>>>>>>> UN 10.138.2.20 1.12 MB 256 7.9%
>>>>>>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>>>>>>> UN 10.93.31.44 282.65 MB 256 7.8%
>>>>>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>>>>>>> UN 10.236.138.169 223.66 MB 256 9.1%
>>>>>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>>>>>>> UN 10.137.7.90 11.36 MB 256 7.4%
>>>>>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>>>>>>> UN 10.93.77.166 837.64 MB 256 8.8%
>>>>>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>>>>>>> UN 10.120.249.140 838.59 MB 256 9.4%
>>>>>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>>>>>>> UN 10.90.246.128 216.75 MB 256 8.4%
>>>>>>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>>>>>>> UN 10.123.95.248 147.1 MB 256 7.2%
>>>>>>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>>>>>>> UN 10.136.11.40 4.24 MB 256 8.5%
>>>>>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>>>>>>> UN 10.87.90.42 11.56 MB 256 8.0%
>>>>>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>>>>>>> UN 10.87.75.147 549 MB 256 8.3%
>>>>>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>>>>>>> UN 10.151.49.88 119.86 MB 256 8.9%
>>>>>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>>>>>>> UN 10.87.83.107 484.3 MB 256 8.3%
>>>>>>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>>>>>>> UN 10.137.20.183 137.67 MB 256 8.4%
>>>>>>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>>>>>>> UN 10.238.170.159 49.17 MB 256 9.4%
>>>>>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>>>>>>>
>>>>>>>> Is there something else that I should be doing differently?
>>>>>>>>
>>>>>>>> Thanks for your help!
>>>>>>>>
>>>>>>>> Suruchi
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com>wrote:
>>>>>>>>
>>>>>>>>> The only thing you need to guarantee is that Cassandra doesn't
>>>>>>>>> start with num_tokens=1 (the default in 1.2.x) or, if it does, that you
>>>>>>>>> wipe all the data before starting it with higher num_tokens.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com>wrote:
>>>>>>>>>
>>>>>>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>>>>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Do you suggest I should try with some other installation
>>>>>>>>>>> mechanism? Are there any known problems with the tar installation of
>>>>>>>>>>> cassandra 1.2.9 that I should be aware of?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I was asking in the context of this JIRA :
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>>>>>>>
>>>>>>>>>> Which does not seem to apply in your case!
>>>>>>>>>>
>>>>>>>>>> =Rob
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Jayadev Jayaraman <jd...@gmail.com>.
As a follow-up, is operating a Cassandra cluster with machines on multiple
racks and vnodes bound to cause load imbalance ? Shouldn't token-ranges
assigned to individual machines via their vnodes be approximately balanced
? We're otherwise unable to explain why this imbalance occurs. ( it
shouldn't be the fault of the Murmur3 partitioner which guarantees a
uniform distribution of keys across token-ranges according to the doc. )
On Thu, Sep 19, 2013 at 8:56 PM, Jayadev Jayaraman <jd...@gmail.com>wrote:
> We're using NetworkTopologyStrategy for our placement_strategy and our
> primary motivation for placing nodes on more than 1 rack is to prevent
> showstoppers caused by localized faults leading to multiple EC2 instances
> in the same availability zone (rack) becoming unreachable (which we have
> encountered before).
>
> I've attached the output of "nodetool ring" here.
>
>
> On Thu, Sep 19, 2013 at 8:35 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>
>> Other thing I noticed is that you are using mutiple RACKS and that might
>> be contributing factor to it. However, I am not sure.
>>
>> Can you paste the output of nodetool cfstats and ring? Is it possible to
>> run the same test but keeping all the nodes in one rack?
>>
>> I think you should open a JIRA if you are able to reproduce this.
>>
>> On Thu, Sep 19, 2013 at 4:41 PM, Jayadev Jayaraman <jd...@gmail.com>wrote:
>>
>>> We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted
>>> cassandra and this is what we get for nodetool status :
>>>
>>> bin/nodetool -h localhost status
>>> Datacenter: us-east
>>> ===================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns (effective) Host ID
>>> Rack
>>> UN 10.238.133.174 885.36 MB 256 8.4%
>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>> UN 10.238.133.97 468.66 MB 256 7.7%
>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>> UN 10.151.86.146 1.08 GB 256 8.0%
>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>> UN 10.138.10.9 941.44 MB 256 8.6%
>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>> UN 10.87.87.240 99.69 MB 256 8.6%
>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>> UN 10.93.5.157 87.44 MB 256 7.6%
>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>> UN 10.238.137.250 561.42 MB 256 7.8%
>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>> UN 10.92.231.170 893.75 MB 256 9.3%
>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>> UN 10.138.2.20 31.89 MB 256 7.9%
>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>> UN 10.93.31.44 312.52 MB 256 7.8%
>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>> UN 10.93.91.139 30.46 MB 256 8.1%
>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>> UN 10.236.138.169 260.15 MB 256 9.1%
>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>> UN 10.137.7.90 38.45 MB 256 7.4%
>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>> UN 10.93.77.166 867.15 MB 256 8.8%
>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>> UN 10.120.249.140 863.98 MB 256 9.4%
>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>> UN 10.90.246.128 242.63 MB 256 8.4%
>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>> UN 10.123.95.248 171.51 MB 256 7.2%
>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>> UN 10.136.11.40 33.8 MB 256 8.5%
>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>> UN 10.87.90.42 38.01 MB 256 8.0%
>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>> UN 10.87.75.147 579.29 MB 256 8.3%
>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>> UN 10.151.49.88 151.06 MB 256 8.9%
>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>> UN 10.87.83.107 512.91 MB 256 8.3%
>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>> UN 10.238.170.159 85.04 MB 256 9.4%
>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>> UN 10.137.20.183 167.41 MB 256 8.4%
>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>>
>>> It doesn't seem to have changed by much. The loads are still highly
>>> uneven.
>>>
>>> As for the number of keys in each node's CFs : the largest node now
>>> has 5589120 keys for the column-family that had 6527744 keys before
>>> (load is now 1.08 GB as compares to 1.05 GB before), while the smallest
>>> node now has 71808 keys as compared to 3840 keys before (load is now
>>> 31.89 MB as compares to 1.12 MB before).
>>>
>>>
>>> On Thu, Sep 19, 2013 at 5:18 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>>
>>>> Can you run nodetool repair on all the nodes first and look at the keys?
>>>>
>>>>
>>>> On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>
>>>>> Yes, the key distribution does vary across the nodes. For example, on
>>>>> the node with the highest data, Number of Keys (estimate) is 6527744 for a
>>>>> particular column family, whereas for the same column family on the node
>>>>> with least data, Number of Keys (estimate) = 3840.
>>>>>
>>>>> Is there a way to control this distribution by setting some parameter
>>>>> of cassandra.
>>>>>
>>>>> I am using the Murmur3 partitioner with NetworkTopologyStrategy.
>>>>>
>>>>> Thanks,
>>>>> Suruchi
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <mohitanchlia@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Can you check cfstats to see number of keys per node?
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>
>>>>>>> Thanks for your replies. I wiped out my data from the cluster and
>>>>>>> also cleared the commitlog before restarting it with num_tokens=256. I then
>>>>>>> uploaded data using sstableloader.
>>>>>>>
>>>>>>> However, I am still not able to see a uniform distribution of data
>>>>>>> across nodes of the clusters.
>>>>>>>
>>>>>>> The output of the bin/nodetool -h localhost status commands looks
>>>>>>> like follows. Some nodes have data as low as 1.12MB while some have as high
>>>>>>> as 912.57 MB.
>>>>>>>
>>>>>>> Datacenter: us-east
>>>>>>> ===================
>>>>>>> Status=Up/Down
>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>> -- Address Load Tokens Owns (effective) Host
>>>>>>> ID Rack
>>>>>>> UN 10.238.133.174 856.66 MB 256 8.4%
>>>>>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>>>>>> UN 10.238.133.97 439.02 MB 256 7.7%
>>>>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>>>>>> UN 10.151.86.146 1.05 GB 256 8.0%
>>>>>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>>>>>> UN 10.138.10.9 912.57 MB 256 8.6%
>>>>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>>>>>> UN 10.87.87.240 70.85 MB 256 8.6%
>>>>>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>>>>>> UN 10.93.5.157 60.56 MB 256 7.6%
>>>>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>>>>>> UN 10.92.231.170 866.73 MB 256 9.3%
>>>>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>>>>>> UN 10.238.137.250 533.77 MB 256 7.8%
>>>>>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>>>>>> UN 10.93.91.139 478.45 KB 256 8.1%
>>>>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>>>>>> UN 10.138.2.20 1.12 MB 256 7.9%
>>>>>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>>>>>> UN 10.93.31.44 282.65 MB 256 7.8%
>>>>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>>>>>> UN 10.236.138.169 223.66 MB 256 9.1%
>>>>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>>>>>> UN 10.137.7.90 11.36 MB 256 7.4%
>>>>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>>>>>> UN 10.93.77.166 837.64 MB 256 8.8%
>>>>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>>>>>> UN 10.120.249.140 838.59 MB 256 9.4%
>>>>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>>>>>> UN 10.90.246.128 216.75 MB 256 8.4%
>>>>>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>>>>>> UN 10.123.95.248 147.1 MB 256 7.2%
>>>>>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>>>>>> UN 10.136.11.40 4.24 MB 256 8.5%
>>>>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>>>>>> UN 10.87.90.42 11.56 MB 256 8.0%
>>>>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>>>>>> UN 10.87.75.147 549 MB 256 8.3%
>>>>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>>>>>> UN 10.151.49.88 119.86 MB 256 8.9%
>>>>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>>>>>> UN 10.87.83.107 484.3 MB 256 8.3%
>>>>>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>>>>>> UN 10.137.20.183 137.67 MB 256 8.4%
>>>>>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>>>>>> UN 10.238.170.159 49.17 MB 256 9.4%
>>>>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>>>>>>
>>>>>>> Is there something else that I should be doing differently?
>>>>>>>
>>>>>>> Thanks for your help!
>>>>>>>
>>>>>>> Suruchi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com>wrote:
>>>>>>>
>>>>>>>> The only thing you need to guarantee is that Cassandra doesn't
>>>>>>>> start with num_tokens=1 (the default in 1.2.x) or, if it does, that you
>>>>>>>> wipe all the data before starting it with higher num_tokens.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com>wrote:
>>>>>>>>
>>>>>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>>>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>>>>
>>>>>>>>>> Do you suggest I should try with some other installation
>>>>>>>>>> mechanism? Are there any known problems with the tar installation of
>>>>>>>>>> cassandra 1.2.9 that I should be aware of?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was asking in the context of this JIRA :
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>>>>>>
>>>>>>>>> Which does not seem to apply in your case!
>>>>>>>>>
>>>>>>>>> =Rob
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Jayadev Jayaraman <jd...@gmail.com>.
We're using NetworkTopologyStrategy for our placement_strategy and our
primary motivation for placing nodes on more than 1 rack is to prevent
showstoppers caused by localized faults leading to multiple EC2 instances
in the same availability zone (rack) becoming unreachable (which we have
encountered before).
I've attached the output of "nodetool ring" here.
On Thu, Sep 19, 2013 at 8:35 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> Other thing I noticed is that you are using mutiple RACKS and that might
> be contributing factor to it. However, I am not sure.
>
> Can you paste the output of nodetool cfstats and ring? Is it possible to
> run the same test but keeping all the nodes in one rack?
>
> I think you should open a JIRA if you are able to reproduce this.
>
> On Thu, Sep 19, 2013 at 4:41 PM, Jayadev Jayaraman <jd...@gmail.com>wrote:
>
>> We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted
>> cassandra and this is what we get for nodetool status :
>>
>> bin/nodetool -h localhost status
>> Datacenter: us-east
>> ===================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UN 10.238.133.174 885.36 MB 256 8.4%
>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>> UN 10.238.133.97 468.66 MB 256 7.7%
>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>> UN 10.151.86.146 1.08 GB 256 8.0%
>> 8952645d-4a27-4670-afb2-65061c205734 1a
>> UN 10.138.10.9 941.44 MB 256 8.6%
>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>> UN 10.87.87.240 99.69 MB 256 8.6%
>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>> UN 10.93.5.157 87.44 MB 256 7.6%
>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>> UN 10.238.137.250 561.42 MB 256 7.8%
>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>> UN 10.92.231.170 893.75 MB 256 9.3%
>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>> UN 10.138.2.20 31.89 MB 256 7.9%
>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>> UN 10.93.31.44 312.52 MB 256 7.8%
>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>> UN 10.93.91.139 30.46 MB 256 8.1%
>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>> UN 10.236.138.169 260.15 MB 256 9.1%
>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>> UN 10.137.7.90 38.45 MB 256 7.4%
>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>> UN 10.93.77.166 867.15 MB 256 8.8%
>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>> UN 10.120.249.140 863.98 MB 256 9.4%
>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>> UN 10.90.246.128 242.63 MB 256 8.4%
>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>> UN 10.123.95.248 171.51 MB 256 7.2%
>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>> UN 10.136.11.40 33.8 MB 256 8.5%
>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>> UN 10.87.90.42 38.01 MB 256 8.0%
>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>> UN 10.87.75.147 579.29 MB 256 8.3%
>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>> UN 10.151.49.88 151.06 MB 256 8.9%
>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>> UN 10.87.83.107 512.91 MB 256 8.3%
>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>> UN 10.238.170.159 85.04 MB 256 9.4%
>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>> UN 10.137.20.183 167.41 MB 256 8.4%
>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>
>> It doesn't seem to have changed by much. The loads are still highly
>> uneven.
>>
>> As for the number of keys in each node's CFs : the largest node now
>> has 5589120 keys for the column-family that had 6527744 keys before
>> (load is now 1.08 GB as compares to 1.05 GB before), while the smallest
>> node now has 71808 keys as compared to 3840 keys before (load is now
>> 31.89 MB as compares to 1.12 MB before).
>>
>>
>> On Thu, Sep 19, 2013 at 5:18 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>
>>> Can you run nodetool repair on all the nodes first and look at the keys?
>>>
>>>
>>> On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>
>>>> Yes, the key distribution does vary across the nodes. For example, on
>>>> the node with the highest data, Number of Keys (estimate) is 6527744 for a
>>>> particular column family, whereas for the same column family on the node
>>>> with least data, Number of Keys (estimate) = 3840.
>>>>
>>>> Is there a way to control this distribution by setting some parameter
>>>> of cassandra.
>>>>
>>>> I am using the Murmur3 partitioner with NetworkTopologyStrategy.
>>>>
>>>> Thanks,
>>>> Suruchi
>>>>
>>>>
>>>>
>>>> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>>>
>>>>> Can you check cfstats to see number of keys per node?
>>>>>
>>>>>
>>>>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>
>>>>>> Thanks for your replies. I wiped out my data from the cluster and
>>>>>> also cleared the commitlog before restarting it with num_tokens=256. I then
>>>>>> uploaded data using sstableloader.
>>>>>>
>>>>>> However, I am still not able to see a uniform distribution of data
>>>>>> across nodes of the clusters.
>>>>>>
>>>>>> The output of the bin/nodetool -h localhost status commands looks
>>>>>> like follows. Some nodes have data as low as 1.12MB while some have as high
>>>>>> as 912.57 MB.
>>>>>>
>>>>>> Datacenter: us-east
>>>>>> ===================
>>>>>> Status=Up/Down
>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>> -- Address Load Tokens Owns (effective) Host
>>>>>> ID Rack
>>>>>> UN 10.238.133.174 856.66 MB 256 8.4%
>>>>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>>>>> UN 10.238.133.97 439.02 MB 256 7.7%
>>>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>>>>> UN 10.151.86.146 1.05 GB 256 8.0%
>>>>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>>>>> UN 10.138.10.9 912.57 MB 256 8.6%
>>>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>>>>> UN 10.87.87.240 70.85 MB 256 8.6%
>>>>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>>>>> UN 10.93.5.157 60.56 MB 256 7.6%
>>>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>>>>> UN 10.92.231.170 866.73 MB 256 9.3%
>>>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>>>>> UN 10.238.137.250 533.77 MB 256 7.8%
>>>>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>>>>> UN 10.93.91.139 478.45 KB 256 8.1%
>>>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>>>>> UN 10.138.2.20 1.12 MB 256 7.9%
>>>>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>>>>> UN 10.93.31.44 282.65 MB 256 7.8%
>>>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>>>>> UN 10.236.138.169 223.66 MB 256 9.1%
>>>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>>>>> UN 10.137.7.90 11.36 MB 256 7.4%
>>>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>>>>> UN 10.93.77.166 837.64 MB 256 8.8%
>>>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>>>>> UN 10.120.249.140 838.59 MB 256 9.4%
>>>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>>>>> UN 10.90.246.128 216.75 MB 256 8.4%
>>>>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>>>>> UN 10.123.95.248 147.1 MB 256 7.2%
>>>>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>>>>> UN 10.136.11.40 4.24 MB 256 8.5%
>>>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>>>>> UN 10.87.90.42 11.56 MB 256 8.0%
>>>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>>>>> UN 10.87.75.147 549 MB 256 8.3%
>>>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>>>>> UN 10.151.49.88 119.86 MB 256 8.9%
>>>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>>>>> UN 10.87.83.107 484.3 MB 256 8.3%
>>>>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>>>>> UN 10.137.20.183 137.67 MB 256 8.4%
>>>>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>>>>> UN 10.238.170.159 49.17 MB 256 9.4%
>>>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>>>>>
>>>>>> Is there something else that I should be doing differently?
>>>>>>
>>>>>> Thanks for your help!
>>>>>>
>>>>>> Suruchi
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com>wrote:
>>>>>>
>>>>>>> The only thing you need to guarantee is that Cassandra doesn't start
>>>>>>> with num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all
>>>>>>> the data before starting it with higher num_tokens.
>>>>>>>
>>>>>>>
>>>>>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com>wrote:
>>>>>>>
>>>>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>>>
>>>>>>>>> Do you suggest I should try with some other installation
>>>>>>>>> mechanism? Are there any known problems with the tar installation of
>>>>>>>>> cassandra 1.2.9 that I should be aware of?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I was asking in the context of this JIRA :
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>>>>>
>>>>>>>> Which does not seem to apply in your case!
>>>>>>>>
>>>>>>>> =Rob
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Mohit Anchlia <mo...@gmail.com>.
Other thing I noticed is that you are using mutiple RACKS and that might be
contributing factor to it. However, I am not sure.
Can you paste the output of nodetool cfstats and ring? Is it possible to
run the same test but keeping all the nodes in one rack?
I think you should open a JIRA if you are able to reproduce this.
On Thu, Sep 19, 2013 at 4:41 PM, Jayadev Jayaraman <jd...@gmail.com>wrote:
> We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted
> cassandra and this is what we get for nodetool status :
>
> bin/nodetool -h localhost status
> Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.238.133.174 885.36 MB 256 8.4%
> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
> UN 10.238.133.97 468.66 MB 256 7.7%
> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
> UN 10.151.86.146 1.08 GB 256 8.0%
> 8952645d-4a27-4670-afb2-65061c205734 1a
> UN 10.138.10.9 941.44 MB 256 8.6%
> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
> UN 10.87.87.240 99.69 MB 256 8.6%
> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
> UN 10.93.5.157 87.44 MB 256 7.6%
> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
> UN 10.238.137.250 561.42 MB 256 7.8%
> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
> UN 10.92.231.170 893.75 MB 256 9.3%
> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
> UN 10.138.2.20 31.89 MB 256 7.9%
> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
> UN 10.93.31.44 312.52 MB 256 7.8%
> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
> UN 10.93.91.139 30.46 MB 256 8.1%
> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
> UN 10.236.138.169 260.15 MB 256 9.1%
> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
> UN 10.137.7.90 38.45 MB 256 7.4%
> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
> UN 10.93.77.166 867.15 MB 256 8.8%
> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
> UN 10.120.249.140 863.98 MB 256 9.4%
> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
> UN 10.90.246.128 242.63 MB 256 8.4%
> 054911ec-969d-43d9-aea1-db445706e4d2 1b
> UN 10.123.95.248 171.51 MB 256 7.2%
> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
> UN 10.136.11.40 33.8 MB 256 8.5%
> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
> UN 10.87.90.42 38.01 MB 256 8.0%
> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
> UN 10.87.75.147 579.29 MB 256 8.3%
> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
> UN 10.151.49.88 151.06 MB 256 8.9%
> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
> UN 10.87.83.107 512.91 MB 256 8.3%
> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
> UN 10.238.170.159 85.04 MB 256 9.4%
> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
> UN 10.137.20.183 167.41 MB 256 8.4%
> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>
> It doesn't seem to have changed by much. The loads are still highly
> uneven.
>
> As for the number of keys in each node's CFs : the largest node now
> has 5589120 keys for the column-family that had 6527744 keys before (load
> is now 1.08 GB as compares to 1.05 GB before), while the smallest node now
> has 71808 keys as compared to 3840 keys before (load is now 31.89 MB as
> compares to 1.12 MB before).
>
>
> On Thu, Sep 19, 2013 at 5:18 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>
>> Can you run nodetool repair on all the nodes first and look at the keys?
>>
>>
>> On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
>> suruchi.deodhar@generalsentiment.com> wrote:
>>
>>> Yes, the key distribution does vary across the nodes. For example, on
>>> the node with the highest data, Number of Keys (estimate) is 6527744 for a
>>> particular column family, whereas for the same column family on the node
>>> with least data, Number of Keys (estimate) = 3840.
>>>
>>> Is there a way to control this distribution by setting some parameter of
>>> cassandra.
>>>
>>> I am using the Murmur3 partitioner with NetworkTopologyStrategy.
>>>
>>> Thanks,
>>> Suruchi
>>>
>>>
>>>
>>> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>>
>>>> Can you check cfstats to see number of keys per node?
>>>>
>>>>
>>>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>
>>>>> Thanks for your replies. I wiped out my data from the cluster and also
>>>>> cleared the commitlog before restarting it with num_tokens=256. I then
>>>>> uploaded data using sstableloader.
>>>>>
>>>>> However, I am still not able to see a uniform distribution of data
>>>>> across nodes of the clusters.
>>>>>
>>>>> The output of the bin/nodetool -h localhost status commands looks like
>>>>> follows. Some nodes have data as low as 1.12MB while some have as high as
>>>>> 912.57 MB.
>>>>>
>>>>> Datacenter: us-east
>>>>> ===================
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> -- Address Load Tokens Owns (effective) Host
>>>>> ID Rack
>>>>> UN 10.238.133.174 856.66 MB 256 8.4%
>>>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>>>> UN 10.238.133.97 439.02 MB 256 7.7%
>>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>>>> UN 10.151.86.146 1.05 GB 256 8.0%
>>>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>>>> UN 10.138.10.9 912.57 MB 256 8.6%
>>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>>>> UN 10.87.87.240 70.85 MB 256 8.6%
>>>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>>>> UN 10.93.5.157 60.56 MB 256 7.6%
>>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>>>> UN 10.92.231.170 866.73 MB 256 9.3%
>>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>>>> UN 10.238.137.250 533.77 MB 256 7.8%
>>>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>>>> UN 10.93.91.139 478.45 KB 256 8.1%
>>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>>>> UN 10.138.2.20 1.12 MB 256 7.9%
>>>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>>>> UN 10.93.31.44 282.65 MB 256 7.8%
>>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>>>> UN 10.236.138.169 223.66 MB 256 9.1%
>>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>>>> UN 10.137.7.90 11.36 MB 256 7.4%
>>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>>>> UN 10.93.77.166 837.64 MB 256 8.8%
>>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>>>> UN 10.120.249.140 838.59 MB 256 9.4%
>>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>>>> UN 10.90.246.128 216.75 MB 256 8.4%
>>>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>>>> UN 10.123.95.248 147.1 MB 256 7.2%
>>>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>>>> UN 10.136.11.40 4.24 MB 256 8.5%
>>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>>>> UN 10.87.90.42 11.56 MB 256 8.0%
>>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>>>> UN 10.87.75.147 549 MB 256 8.3%
>>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>>>> UN 10.151.49.88 119.86 MB 256 8.9%
>>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>>>> UN 10.87.83.107 484.3 MB 256 8.3%
>>>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>>>> UN 10.137.20.183 137.67 MB 256 8.4%
>>>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>>>> UN 10.238.170.159 49.17 MB 256 9.4%
>>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>>>>
>>>>> Is there something else that I should be doing differently?
>>>>>
>>>>> Thanks for your help!
>>>>>
>>>>> Suruchi
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com>wrote:
>>>>>
>>>>>> The only thing you need to guarantee is that Cassandra doesn't start
>>>>>> with num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all
>>>>>> the data before starting it with higher num_tokens.
>>>>>>
>>>>>>
>>>>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com> wrote:
>>>>>>
>>>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>>
>>>>>>>> Do you suggest I should try with some other installation mechanism?
>>>>>>>> Are there any known problems with the tar installation of cassandra 1.2.9
>>>>>>>> that I should be aware of?
>>>>>>>>
>>>>>>>
>>>>>>> I was asking in the context of this JIRA :
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>>>>
>>>>>>> Which does not seem to apply in your case!
>>>>>>>
>>>>>>> =Rob
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Jayadev Jayaraman <jd...@gmail.com>.
We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted
cassandra and this is what we get for nodetool status :
bin/nodetool -h localhost status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.238.133.174 885.36 MB 256 8.4%
e41d8863-ce37-4d5c-a428-bfacea432a35 1a
UN 10.238.133.97 468.66 MB 256 7.7%
1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
UN 10.151.86.146 1.08 GB 256 8.0%
8952645d-4a27-4670-afb2-65061c205734 1a
UN 10.138.10.9 941.44 MB 256 8.6%
25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
UN 10.87.87.240 99.69 MB 256 8.6%
ea066827-83bc-458c-83e8-bd15b7fc783c 1b
UN 10.93.5.157 87.44 MB 256 7.6%
4ab9111c-39b4-4d15-9401-359d9d853c16 1b
UN 10.238.137.250 561.42 MB 256 7.8%
84301648-afff-4f06-aa0b-4be421e0d08f 1a
UN 10.92.231.170 893.75 MB 256 9.3%
a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
UN 10.138.2.20 31.89 MB 256 7.9%
a6d4672a-0915-4c64-ba47-9f190abbf951 1a
UN 10.93.31.44 312.52 MB 256 7.8%
67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
UN 10.93.91.139 30.46 MB 256 8.1%
682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
UN 10.236.138.169 260.15 MB 256 9.1%
cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
UN 10.137.7.90 38.45 MB 256 7.4%
17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
UN 10.93.77.166 867.15 MB 256 8.8%
9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
UN 10.120.249.140 863.98 MB 256 9.4%
e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
UN 10.90.246.128 242.63 MB 256 8.4%
054911ec-969d-43d9-aea1-db445706e4d2 1b
UN 10.123.95.248 171.51 MB 256 7.2%
a17deca1-9644-4520-9e62-ac66fc6fef60 1b
UN 10.136.11.40 33.8 MB 256 8.5%
66be1173-b822-40b5-b650-cb38ae3c7a51 1a
UN 10.87.90.42 38.01 MB 256 8.0%
dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
UN 10.87.75.147 579.29 MB 256 8.3%
ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
UN 10.151.49.88 151.06 MB 256 8.9%
57043573-ab1b-4e3c-8044-58376f7ce08f 1a
UN 10.87.83.107 512.91 MB 256 8.3%
0019439b-9f8a-4965-91b8-7108bbb55593 1b
UN 10.238.170.159 85.04 MB 256 9.4%
32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
UN 10.137.20.183 167.41 MB 256 8.4%
15951592-8ab2-473d-920a-da6e9d99507d 1a
It doesn't seem to have changed by much. The loads are still highly uneven.
As for the number of keys in each node's CFs : the largest node now
has 5589120 keys for the column-family that had 6527744 keys before (load
is now 1.08 GB as compares to 1.05 GB before), while the smallest node now
has 71808 keys as compared to 3840 keys before (load is now 31.89 MB as
compares to 1.12 MB before).
On Thu, Sep 19, 2013 at 5:18 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> Can you run nodetool repair on all the nodes first and look at the keys?
>
>
> On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
> suruchi.deodhar@generalsentiment.com> wrote:
>
>> Yes, the key distribution does vary across the nodes. For example, on the
>> node with the highest data, Number of Keys (estimate) is 6527744 for a
>> particular column family, whereas for the same column family on the node
>> with least data, Number of Keys (estimate) = 3840.
>>
>> Is there a way to control this distribution by setting some parameter of
>> cassandra.
>>
>> I am using the Murmur3 partitioner with NetworkTopologyStrategy.
>>
>> Thanks,
>> Suruchi
>>
>>
>>
>> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>
>>> Can you check cfstats to see number of keys per node?
>>>
>>>
>>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>
>>>> Thanks for your replies. I wiped out my data from the cluster and also
>>>> cleared the commitlog before restarting it with num_tokens=256. I then
>>>> uploaded data using sstableloader.
>>>>
>>>> However, I am still not able to see a uniform distribution of data
>>>> across nodes of the clusters.
>>>>
>>>> The output of the bin/nodetool -h localhost status commands looks like
>>>> follows. Some nodes have data as low as 1.12MB while some have as high as
>>>> 912.57 MB.
>>>>
>>>> Datacenter: us-east
>>>> ===================
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> -- Address Load Tokens Owns (effective) Host
>>>> ID Rack
>>>> UN 10.238.133.174 856.66 MB 256 8.4%
>>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>>> UN 10.238.133.97 439.02 MB 256 7.7%
>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>>> UN 10.151.86.146 1.05 GB 256 8.0%
>>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>>> UN 10.138.10.9 912.57 MB 256 8.6%
>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>>> UN 10.87.87.240 70.85 MB 256 8.6%
>>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>>> UN 10.93.5.157 60.56 MB 256 7.6%
>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>>> UN 10.92.231.170 866.73 MB 256 9.3%
>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>>> UN 10.238.137.250 533.77 MB 256 7.8%
>>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>>> UN 10.93.91.139 478.45 KB 256 8.1%
>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>>> UN 10.138.2.20 1.12 MB 256 7.9%
>>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>>> UN 10.93.31.44 282.65 MB 256 7.8%
>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>>> UN 10.236.138.169 223.66 MB 256 9.1%
>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>>> UN 10.137.7.90 11.36 MB 256 7.4%
>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>>> UN 10.93.77.166 837.64 MB 256 8.8%
>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>>> UN 10.120.249.140 838.59 MB 256 9.4%
>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>>> UN 10.90.246.128 216.75 MB 256 8.4%
>>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>>> UN 10.123.95.248 147.1 MB 256 7.2%
>>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>>> UN 10.136.11.40 4.24 MB 256 8.5%
>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>>> UN 10.87.90.42 11.56 MB 256 8.0%
>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>>> UN 10.87.75.147 549 MB 256 8.3%
>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>>> UN 10.151.49.88 119.86 MB 256 8.9%
>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>>> UN 10.87.83.107 484.3 MB 256 8.3%
>>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>>> UN 10.137.20.183 137.67 MB 256 8.4%
>>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>>> UN 10.238.170.159 49.17 MB 256 9.4%
>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>>>
>>>> Is there something else that I should be doing differently?
>>>>
>>>> Thanks for your help!
>>>>
>>>> Suruchi
>>>>
>>>>
>>>>
>>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com>wrote:
>>>>
>>>>> The only thing you need to guarantee is that Cassandra doesn't start
>>>>> with num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all
>>>>> the data before starting it with higher num_tokens.
>>>>>
>>>>>
>>>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com> wrote:
>>>>>
>>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>
>>>>>>> Do you suggest I should try with some other installation mechanism?
>>>>>>> Are there any known problems with the tar installation of cassandra 1.2.9
>>>>>>> that I should be aware of?
>>>>>>>
>>>>>>
>>>>>> I was asking in the context of this JIRA :
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>>>
>>>>>> Which does not seem to apply in your case!
>>>>>>
>>>>>> =Rob
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Mohit Anchlia <mo...@gmail.com>.
Can you run nodetool repair on all the nodes first and look at the keys?
On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> Yes, the key distribution does vary across the nodes. For example, on the
> node with the highest data, Number of Keys (estimate) is 6527744 for a
> particular column family, whereas for the same column family on the node
> with least data, Number of Keys (estimate) = 3840.
>
> Is there a way to control this distribution by setting some parameter of
> cassandra.
>
> I am using the Murmur3 partitioner with NetworkTopologyStrategy.
>
> Thanks,
> Suruchi
>
>
>
> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>
>> Can you check cfstats to see number of keys per node?
>>
>>
>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
>> suruchi.deodhar@generalsentiment.com> wrote:
>>
>>> Thanks for your replies. I wiped out my data from the cluster and also
>>> cleared the commitlog before restarting it with num_tokens=256. I then
>>> uploaded data using sstableloader.
>>>
>>> However, I am still not able to see a uniform distribution of data
>>> across nodes of the clusters.
>>>
>>> The output of the bin/nodetool -h localhost status commands looks like
>>> follows. Some nodes have data as low as 1.12MB while some have as high as
>>> 912.57 MB.
>>>
>>> Datacenter: us-east
>>> ===================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns (effective) Host
>>> ID Rack
>>> UN 10.238.133.174 856.66 MB 256 8.4%
>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>>> UN 10.238.133.97 439.02 MB 256 7.7%
>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>>> UN 10.151.86.146 1.05 GB 256 8.0%
>>> 8952645d-4a27-4670-afb2-65061c205734 1a
>>> UN 10.138.10.9 912.57 MB 256 8.6%
>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>>> UN 10.87.87.240 70.85 MB 256 8.6%
>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>>> UN 10.93.5.157 60.56 MB 256 7.6%
>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>>> UN 10.92.231.170 866.73 MB 256 9.3%
>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>>> UN 10.238.137.250 533.77 MB 256 7.8%
>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>>> UN 10.93.91.139 478.45 KB 256 8.1%
>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>>> UN 10.138.2.20 1.12 MB 256 7.9%
>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>>> UN 10.93.31.44 282.65 MB 256 7.8%
>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>>> UN 10.236.138.169 223.66 MB 256 9.1%
>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>>> UN 10.137.7.90 11.36 MB 256 7.4%
>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>>> UN 10.93.77.166 837.64 MB 256 8.8%
>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>>> UN 10.120.249.140 838.59 MB 256 9.4%
>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>>> UN 10.90.246.128 216.75 MB 256 8.4%
>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>>> UN 10.123.95.248 147.1 MB 256 7.2%
>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>>> UN 10.136.11.40 4.24 MB 256 8.5%
>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>>> UN 10.87.90.42 11.56 MB 256 8.0%
>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>>> UN 10.87.75.147 549 MB 256 8.3%
>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>>> UN 10.151.49.88 119.86 MB 256 8.9%
>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>>> UN 10.87.83.107 484.3 MB 256 8.3%
>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>>> UN 10.137.20.183 137.67 MB 256 8.4%
>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>>> UN 10.238.170.159 49.17 MB 256 9.4%
>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>>
>>> Is there something else that I should be doing differently?
>>>
>>> Thanks for your help!
>>>
>>> Suruchi
>>>
>>>
>>>
>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com>wrote:
>>>
>>>> The only thing you need to guarantee is that Cassandra doesn't start
>>>> with num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all
>>>> the data before starting it with higher num_tokens.
>>>>
>>>>
>>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com> wrote:
>>>>
>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>
>>>>>> Do you suggest I should try with some other installation mechanism?
>>>>>> Are there any known problems with the tar installation of cassandra 1.2.9
>>>>>> that I should be aware of?
>>>>>>
>>>>>
>>>>> I was asking in the context of this JIRA :
>>>>>
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>>
>>>>> Which does not seem to apply in your case!
>>>>>
>>>>> =Rob
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Suruchi Deodhar <su...@generalsentiment.com>.
Yes, the key distribution does vary across the nodes. For example, on the
node with the highest data, Number of Keys (estimate) is 6527744 for a
particular column family, whereas for the same column family on the node
with least data, Number of Keys (estimate) = 3840.
Is there a way to control this distribution by setting some parameter of
cassandra.
I am using the Murmur3 partitioner with NetworkTopologyStrategy.
Thanks,
Suruchi
On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> Can you check cfstats to see number of keys per node?
>
>
> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
> suruchi.deodhar@generalsentiment.com> wrote:
>
>> Thanks for your replies. I wiped out my data from the cluster and also
>> cleared the commitlog before restarting it with num_tokens=256. I then
>> uploaded data using sstableloader.
>>
>> However, I am still not able to see a uniform distribution of data across
>> nodes of the clusters.
>>
>> The output of the bin/nodetool -h localhost status commands looks like
>> follows. Some nodes have data as low as 1.12MB while some have as high as
>> 912.57 MB.
>>
>> Datacenter: us-east
>> ===================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN 10.238.133.174 856.66 MB 256 8.4%
>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
>> UN 10.238.133.97 439.02 MB 256 7.7%
>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
>> UN 10.151.86.146 1.05 GB 256 8.0%
>> 8952645d-4a27-4670-afb2-65061c205734 1a
>> UN 10.138.10.9 912.57 MB 256 8.6%
>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
>> UN 10.87.87.240 70.85 MB 256 8.6%
>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
>> UN 10.93.5.157 60.56 MB 256 7.6%
>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
>> UN 10.92.231.170 866.73 MB 256 9.3%
>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
>> UN 10.238.137.250 533.77 MB 256 7.8%
>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
>> UN 10.93.91.139 478.45 KB 256 8.1%
>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
>> UN 10.138.2.20 1.12 MB 256 7.9%
>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
>> UN 10.93.31.44 282.65 MB 256 7.8%
>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
>> UN 10.236.138.169 223.66 MB 256 9.1%
>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
>> UN 10.137.7.90 11.36 MB 256 7.4%
>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
>> UN 10.93.77.166 837.64 MB 256 8.8%
>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
>> UN 10.120.249.140 838.59 MB 256 9.4%
>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
>> UN 10.90.246.128 216.75 MB 256 8.4%
>> 054911ec-969d-43d9-aea1-db445706e4d2 1b
>> UN 10.123.95.248 147.1 MB 256 7.2%
>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
>> UN 10.136.11.40 4.24 MB 256 8.5%
>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
>> UN 10.87.90.42 11.56 MB 256 8.0%
>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
>> UN 10.87.75.147 549 MB 256 8.3%
>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
>> UN 10.151.49.88 119.86 MB 256 8.9%
>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
>> UN 10.87.83.107 484.3 MB 256 8.3%
>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
>> UN 10.137.20.183 137.67 MB 256 8.4%
>> 15951592-8ab2-473d-920a-da6e9d99507d 1a
>> UN 10.238.170.159 49.17 MB 256 9.4%
>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>>
>> Is there something else that I should be doing differently?
>>
>> Thanks for your help!
>>
>> Suruchi
>>
>>
>>
>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com> wrote:
>>
>>> The only thing you need to guarantee is that Cassandra doesn't start
>>> with num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all
>>> the data before starting it with higher num_tokens.
>>>
>>>
>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com> wrote:
>>>
>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>
>>>>> Do you suggest I should try with some other installation mechanism?
>>>>> Are there any known problems with the tar installation of cassandra 1.2.9
>>>>> that I should be aware of?
>>>>>
>>>>
>>>> I was asking in the context of this JIRA :
>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>
>>>> Which does not seem to apply in your case!
>>>>
>>>> =Rob
>>>>
>>>
>>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Mohit Anchlia <mo...@gmail.com>.
Can you check cfstats to see number of keys per node?
On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> Thanks for your replies. I wiped out my data from the cluster and also
> cleared the commitlog before restarting it with num_tokens=256. I then
> uploaded data using sstableloader.
>
> However, I am still not able to see a uniform distribution of data across
> nodes of the clusters.
>
> The output of the bin/nodetool -h localhost status commands looks like
> follows. Some nodes have data as low as 1.12MB while some have as high as
> 912.57 MB.
>
> Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 10.238.133.174 856.66 MB 256 8.4%
> e41d8863-ce37-4d5c-a428-bfacea432a35 1a
> UN 10.238.133.97 439.02 MB 256 7.7%
> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
> UN 10.151.86.146 1.05 GB 256 8.0%
> 8952645d-4a27-4670-afb2-65061c205734 1a
> UN 10.138.10.9 912.57 MB 256 8.6%
> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
> UN 10.87.87.240 70.85 MB 256 8.6%
> ea066827-83bc-458c-83e8-bd15b7fc783c 1b
> UN 10.93.5.157 60.56 MB 256 7.6%
> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b
> UN 10.92.231.170 866.73 MB 256 9.3%
> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
> UN 10.238.137.250 533.77 MB 256 7.8%
> 84301648-afff-4f06-aa0b-4be421e0d08f 1a
> UN 10.93.91.139 478.45 KB 256 8.1%
> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
> UN 10.138.2.20 1.12 MB 256 7.9%
> a6d4672a-0915-4c64-ba47-9f190abbf951 1a
> UN 10.93.31.44 282.65 MB 256 7.8%
> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
> UN 10.236.138.169 223.66 MB 256 9.1%
> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
> UN 10.137.7.90 11.36 MB 256 7.4%
> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
> UN 10.93.77.166 837.64 MB 256 8.8%
> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
> UN 10.120.249.140 838.59 MB 256 9.4%
> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
> UN 10.90.246.128 216.75 MB 256 8.4%
> 054911ec-969d-43d9-aea1-db445706e4d2 1b
> UN 10.123.95.248 147.1 MB 256 7.2%
> a17deca1-9644-4520-9e62-ac66fc6fef60 1b
> UN 10.136.11.40 4.24 MB 256 8.5%
> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a
> UN 10.87.90.42 11.56 MB 256 8.0%
> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
> UN 10.87.75.147 549 MB 256 8.3%
> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
> UN 10.151.49.88 119.86 MB 256 8.9%
> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a
> UN 10.87.83.107 484.3 MB 256 8.3%
> 0019439b-9f8a-4965-91b8-7108bbb55593 1b
> UN 10.137.20.183 137.67 MB 256 8.4%
> 15951592-8ab2-473d-920a-da6e9d99507d 1a
> UN 10.238.170.159 49.17 MB 256 9.4%
> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
>
> Is there something else that I should be doing differently?
>
> Thanks for your help!
>
> Suruchi
>
>
>
> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com> wrote:
>
>> The only thing you need to guarantee is that Cassandra doesn't start with
>> num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all the
>> data before starting it with higher num_tokens.
>>
>>
>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com> wrote:
>>
>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>
>>>> Do you suggest I should try with some other installation mechanism? Are
>>>> there any known problems with the tar installation of cassandra 1.2.9 that
>>>> I should be aware of?
>>>>
>>>
>>> I was asking in the context of this JIRA :
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>
>>> Which does not seem to apply in your case!
>>>
>>> =Rob
>>>
>>
>>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Suruchi Deodhar <su...@generalsentiment.com>.
Thanks for your replies. I wiped out my data from the cluster and also
cleared the commitlog before restarting it with num_tokens=256. I then
uploaded data using sstableloader.
However, I am still not able to see a uniform distribution of data across
nodes of the clusters.
The output of the bin/nodetool -h localhost status commands looks like
follows. Some nodes have data as low as 1.12MB while some have as high as
912.57 MB.
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.238.133.174 856.66 MB 256 8.4%
e41d8863-ce37-4d5c-a428-bfacea432a35 1a
UN 10.238.133.97 439.02 MB 256 7.7%
1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a
UN 10.151.86.146 1.05 GB 256 8.0%
8952645d-4a27-4670-afb2-65061c205734 1a
UN 10.138.10.9 912.57 MB 256 8.6%
25ccea82-49d2-43d9-830c-b9c9cee026ec 1a
UN 10.87.87.240 70.85 MB 256 8.6%
ea066827-83bc-458c-83e8-bd15b7fc783c 1b
UN 10.93.5.157 60.56 MB 256 7.6%
4ab9111c-39b4-4d15-9401-359d9d853c16 1b
UN 10.92.231.170 866.73 MB 256 9.3%
a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b
UN 10.238.137.250 533.77 MB 256 7.8%
84301648-afff-4f06-aa0b-4be421e0d08f 1a
UN 10.93.91.139 478.45 KB 256 8.1%
682dd848-7c7f-4ddb-a960-119cf6491aa1 1b
UN 10.138.2.20 1.12 MB 256 7.9%
a6d4672a-0915-4c64-ba47-9f190abbf951 1a
UN 10.93.31.44 282.65 MB 256 7.8%
67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b
UN 10.236.138.169 223.66 MB 256 9.1%
cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a
UN 10.137.7.90 11.36 MB 256 7.4%
17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a
UN 10.93.77.166 837.64 MB 256 8.8%
9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b
UN 10.120.249.140 838.59 MB 256 9.4%
e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b
UN 10.90.246.128 216.75 MB 256 8.4%
054911ec-969d-43d9-aea1-db445706e4d2 1b
UN 10.123.95.248 147.1 MB 256 7.2%
a17deca1-9644-4520-9e62-ac66fc6fef60 1b
UN 10.136.11.40 4.24 MB 256 8.5%
66be1173-b822-40b5-b650-cb38ae3c7a51 1a
UN 10.87.90.42 11.56 MB 256 8.0%
dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b
UN 10.87.75.147 549 MB 256 8.3%
ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b
UN 10.151.49.88 119.86 MB 256 8.9%
57043573-ab1b-4e3c-8044-58376f7ce08f 1a
UN 10.87.83.107 484.3 MB 256 8.3%
0019439b-9f8a-4965-91b8-7108bbb55593 1b
UN 10.137.20.183 137.67 MB 256 8.4%
15951592-8ab2-473d-920a-da6e9d99507d 1a
UN 10.238.170.159 49.17 MB 256 9.4%
32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a
Is there something else that I should be doing differently?
Thanks for your help!
Suruchi
On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <ri...@wentnet.com> wrote:
> The only thing you need to guarantee is that Cassandra doesn't start with
> num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all the
> data before starting it with higher num_tokens.
>
>
> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
>> suruchi.deodhar@generalsentiment.com> wrote:
>>
>>> Do you suggest I should try with some other installation mechanism? Are
>>> there any known problems with the tar installation of cassandra 1.2.9 that
>>> I should be aware of?
>>>
>>
>> I was asking in the context of this JIRA :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>
>> Which does not seem to apply in your case!
>>
>> =Rob
>>
>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Richard Low <ri...@wentnet.com>.
The only thing you need to guarantee is that Cassandra doesn't start with
num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all the
data before starting it with higher num_tokens.
On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
> suruchi.deodhar@generalsentiment.com> wrote:
>
>> Do you suggest I should try with some other installation mechanism? Are
>> there any known problems with the tar installation of cassandra 1.2.9 that
>> I should be aware of?
>>
>
> I was asking in the context of this JIRA :
>
> https://issues.apache.org/jira/browse/CASSANDRA-2356
>
> Which does not seem to apply in your case!
>
> =Rob
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> Do you suggest I should try with some other installation mechanism? Are
> there any known problems with the tar installation of cassandra 1.2.9 that
> I should be aware of?
>
I was asking in the context of this JIRA :
https://issues.apache.org/jira/browse/CASSANDRA-2356
Which does not seem to apply in your case!
=Rob
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Suruchi Deodhar <su...@generalsentiment.com>.
Hi Rob,
Do you suggest I should try with some other installation mechanism? Are
there any known problems with the tar installation of cassandra 1.2.9 that
I should be aware of? Please do let me know.
Thanks,
Suruchi
On Thu, Sep 19, 2013 at 1:04 PM, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> Hi Robert,
> I downloaded apache-cassandra-1.2.9.tar.gz from
> http://cassandra.apache.org/download/ (
> http://apache.mirrors.tds.net/cassandra/1.2.9/apache-cassandra-1.2.9-bin.tar.gz) and installed it on the individual nodes of the cassandra cluster.
> Thanks,
> Suruchi
>
>
> On Thu, Sep 19, 2013 at 12:35 PM, Robert Coli <rc...@eventbrite.com>wrote:
>
>> On Thu, Sep 19, 2013 at 7:03 AM, Richard Low <ri...@wentnet.com> wrote:
>>
>>> I think what has happened is that Cassandra was started with num_tokens
>>> = 1, then shutdown and num_tokens set to 256. When this happens, the first
>>> time Cassandra chooses a single random token. Then when restarted it
>>> splits the token into 256 adjacent ranges.
>>>
>>
>> Suruchi,
>>
>> By which mechanism did you install Cassandra? I ask out of concern that
>> there may be an issue in the some packaging leading to the above sequence
>> of events.
>>
>> =Rob
>>
>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Suruchi Deodhar <su...@generalsentiment.com>.
Hi Robert,
I downloaded apache-cassandra-1.2.9.tar.gz from
http://cassandra.apache.org/download/ (
http://apache.mirrors.tds.net/cassandra/1.2.9/apache-cassandra-1.2.9-bin.tar.gz)
and installed it on the individual nodes of the cassandra cluster.
Thanks,
Suruchi
On Thu, Sep 19, 2013 at 12:35 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Sep 19, 2013 at 7:03 AM, Richard Low <ri...@wentnet.com> wrote:
>
>> I think what has happened is that Cassandra was started with num_tokens =
>> 1, then shutdown and num_tokens set to 256. When this happens, the first
>> time Cassandra chooses a single random token. Then when restarted it
>> splits the token into 256 adjacent ranges.
>>
>
> Suruchi,
>
> By which mechanism did you install Cassandra? I ask out of concern that
> there may be an issue in the some packaging leading to the above sequence
> of events.
>
> =Rob
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Sep 19, 2013 at 7:03 AM, Richard Low <ri...@wentnet.com> wrote:
> I think what has happened is that Cassandra was started with num_tokens =
> 1, then shutdown and num_tokens set to 256. When this happens, the first
> time Cassandra chooses a single random token. Then when restarted it
> splits the token into 256 adjacent ranges.
>
Suruchi,
By which mechanism did you install Cassandra? I ask out of concern that
there may be an issue in the some packaging leading to the above sequence
of events.
=Rob
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Richard Low <ri...@wentnet.com>.
I think what has happened is that Cassandra was started with num_tokens =
1, then shutdown and num_tokens set to 256. When this happens, the first
time Cassandra chooses a single random token. Then when restarted it
splits the token into 256 adjacent ranges.
You can see something like this has happened because the tokens for each
node are sequential.
The way to fix it is to, assuming you don't want the data, shutdown your
cluster, wipe the whole data and commitlog directories, then start
Cassandra again.
Richard.
On 19 September 2013 13:16, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:
> Hi Richard,
> This is a brand new cluster which started with num_tokens =256 on first
> boot and chose random tokens. The attached ring status is after data is
> loaded into the cluster for the first time using sdtableloader and remains
> that way even after Cassandra is restarted.
>
> Thanks,
> Suruchi
>
> On Sep 19, 2013, at 3:46, Richard Low <ri...@wentnet.com> wrote:
>
> On 19 September 2013 02:06, Jayadev Jayaraman <jd...@gmail.com> wrote:
>
> We use vnodes with num_tokens = 256 ( 256 tokens per node ) . After
>> loading some data with sstableloader , we find that the cluster is heavily
>> imbalanced :
>>
>
> How did you select the tokens? Is this a brand new cluster which started
> on first boot with num_tokens = 256 and chose random tokens? Or did you
> start with num_tokens = 1 and then increase it?
>
> Richard.
>
>
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Suruchi Deodhar <su...@generalsentiment.com>.
Hi Richard,
This is a brand new cluster which started with num_tokens =256 on first boot and chose random tokens. The attached ring status is after data is loaded into the cluster for the first time using sdtableloader and remains that way even after Cassandra is restarted.
Thanks,
Suruchi
On Sep 19, 2013, at 3:46, Richard Low <ri...@wentnet.com> wrote:
> On 19 September 2013 02:06, Jayadev Jayaraman <jd...@gmail.com> wrote:
>
>> We use vnodes with num_tokens = 256 ( 256 tokens per node ) . After loading some data with sstableloader , we find that the cluster is heavily imbalanced :
>
> How did you select the tokens? Is this a brand new cluster which started on first boot with num_tokens = 256 and chose random tokens? Or did you start with num_tokens = 1 and then increase it?
>
> Richard.
Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Posted by Richard Low <ri...@wentnet.com>.
On 19 September 2013 02:06, Jayadev Jayaraman <jd...@gmail.com> wrote:
We use vnodes with num_tokens = 256 ( 256 tokens per node ) . After loading
> some data with sstableloader , we find that the cluster is heavily
> imbalanced :
>
How did you select the tokens? Is this a brand new cluster which started
on first boot with num_tokens = 256 and chose random tokens? Or did you
start with num_tokens = 1 and then increase it?
Richard.