You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Katriel Traum <ka...@google.com> on 2013/11/13 12:46:43 UTC
Uneven effective ownership
Hello list,
I have a problem with my cluster ownership not being as expected.
I have 2 DC cluster using NetworkTopologyStrategy on and
EC2MultiRegionSnitch with cassandra 1.1.5. My placement strategy for all
keyspaces is: {eu-west: 3, us-east:3 }, and I have 6 nodes in eu-west and 3
in us-east.
I expected the resulting effective ownership for nodes in us-east to be
100% (Which it is), and %50 in eu-west (Which it is not).
The cluster started out with 3 nodes in eu-west, and after adding 3 more
nodes, I got the following effective ownership:
1.1.1.1 eu-west 1a Up Normal 89.33 GB 83.33%
0
2.2.2.1 us-east 1b Up Normal 88.84 GB 100.00%
1
1.1.1.4 eu-west 1a Up Normal 75.61 GB 16.67%
28356863910078205288614550619314017621
1.1.1.2 eu-west 1b Up Normal 89 GB 83.33%
56713727820156410577229101238628035242
2.2.2.2 us-east 1d Up Normal 88.34 GB 100.00%
56713727820156410577229101238628035243
1.1.1.5 eu-west 1b Up Normal 75.43 GB 16.67%
85070591730234615865843651857942052863
1.1.1.3 eu-west 1c Up Normal 81.1 GB 83.33%
113427455640312821154458202477256070484
2.2.2.3 us-east 1e Up Normal 93.22 GB 100.00%
113427455640312821154458202477256070485
1.1.1.6 eu-west 1c Up Normal 75.39 GB 16.67%
141784319550391026443072753096570088105
What am I missing here?
TIA,
Katriel
Re: Uneven effective ownership
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Nov 13, 2013 at 5:34 AM, Katriel Traum <ka...@google.com> wrote:
> Thats it! I knew NetworkTopologyStrategy goes through racks when writing,
> but didn't consider that the first one will most likely be used.
>
> I'll re-arrange my nodes to be a,b,c,a,b,c
>
https://issues.apache.org/jira/browse/CASSANDRA-3810
=Rob
Re: Uneven effective ownership
Posted by Katriel Traum <ka...@google.com>.
Thats it! I knew NetworkTopologyStrategy goes through racks when writing,
but didn't consider that the first one will most likely be used.
I'll re-arrange my nodes to be a,b,c,a,b,c
Thanks,
Katriel
On Wed, Nov 13, 2013 at 3:15 PM, Derek Williams <de...@fyrie.net> wrote:
> The problem is that NetworkTopologyStrategy will try to pick nodes that
> have a different rack when going around the ring, so the second node in
> each rack always gets skipped unless it was the first node picked. Your
> nodes is eu-west go a,a,b,b,c,c but they should be a,b,c,a,b,c.
>
>
> On Wed, Nov 13, 2013 at 11:46 AM, Katriel Traum <ka...@google.com>wrote:
>
>> Hello list,
>>
>> I have a problem with my cluster ownership not being as expected.
>>
>> I have 2 DC cluster using NetworkTopologyStrategy on and
>> EC2MultiRegionSnitch with cassandra 1.1.5. My placement strategy for all
>> keyspaces is: {eu-west: 3, us-east:3 }, and I have 6 nodes in eu-west and 3
>> in us-east.
>> I expected the resulting effective ownership for nodes in us-east to be
>> 100% (Which it is), and %50 in eu-west (Which it is not).
>>
>> The cluster started out with 3 nodes in eu-west, and after adding 3 more
>> nodes, I got the following effective ownership:
>>
>> 1.1.1.1 eu-west 1a Up Normal 89.33 GB 83.33%
>> 0
>> 2.2.2.1 us-east 1b Up Normal 88.84 GB 100.00%
>> 1
>> 1.1.1.4 eu-west 1a Up Normal 75.61 GB 16.67%
>> 28356863910078205288614550619314017621
>> 1.1.1.2 eu-west 1b Up Normal 89 GB 83.33%
>> 56713727820156410577229101238628035242
>> 2.2.2.2 us-east 1d Up Normal 88.34 GB 100.00%
>> 56713727820156410577229101238628035243
>> 1.1.1.5 eu-west 1b Up Normal 75.43 GB 16.67%
>> 85070591730234615865843651857942052863
>> 1.1.1.3 eu-west 1c Up Normal 81.1 GB 83.33%
>> 113427455640312821154458202477256070484
>> 2.2.2.3 us-east 1e Up Normal 93.22 GB 100.00%
>> 113427455640312821154458202477256070485
>> 1.1.1.6 eu-west 1c Up Normal 75.39 GB 16.67%
>> 141784319550391026443072753096570088105
>>
>> What am I missing here?
>>
>> TIA,
>> Katriel
>>
>
>
>
> --
> Derek Williams
>
Re: Uneven effective ownership
Posted by Derek Williams <de...@fyrie.net>.
The problem is that NetworkTopologyStrategy will try to pick nodes that
have a different rack when going around the ring, so the second node in
each rack always gets skipped unless it was the first node picked. Your
nodes is eu-west go a,a,b,b,c,c but they should be a,b,c,a,b,c.
On Wed, Nov 13, 2013 at 11:46 AM, Katriel Traum <ka...@google.com> wrote:
> Hello list,
>
> I have a problem with my cluster ownership not being as expected.
>
> I have 2 DC cluster using NetworkTopologyStrategy on and
> EC2MultiRegionSnitch with cassandra 1.1.5. My placement strategy for all
> keyspaces is: {eu-west: 3, us-east:3 }, and I have 6 nodes in eu-west and 3
> in us-east.
> I expected the resulting effective ownership for nodes in us-east to be
> 100% (Which it is), and %50 in eu-west (Which it is not).
>
> The cluster started out with 3 nodes in eu-west, and after adding 3 more
> nodes, I got the following effective ownership:
>
> 1.1.1.1 eu-west 1a Up Normal 89.33 GB 83.33%
> 0
> 2.2.2.1 us-east 1b Up Normal 88.84 GB 100.00%
> 1
> 1.1.1.4 eu-west 1a Up Normal 75.61 GB 16.67%
> 28356863910078205288614550619314017621
> 1.1.1.2 eu-west 1b Up Normal 89 GB 83.33%
> 56713727820156410577229101238628035242
> 2.2.2.2 us-east 1d Up Normal 88.34 GB 100.00%
> 56713727820156410577229101238628035243
> 1.1.1.5 eu-west 1b Up Normal 75.43 GB 16.67%
> 85070591730234615865843651857942052863
> 1.1.1.3 eu-west 1c Up Normal 81.1 GB 83.33%
> 113427455640312821154458202477256070484
> 2.2.2.3 us-east 1e Up Normal 93.22 GB 100.00%
> 113427455640312821154458202477256070485
> 1.1.1.6 eu-west 1c Up Normal 75.39 GB 16.67%
> 141784319550391026443072753096570088105
>
> What am I missing here?
>
> TIA,
> Katriel
>
--
Derek Williams
Re: Uneven effective ownership
Posted by Hannu Kröger <hk...@gmail.com>.
Hello Katriel,
Are you using replication factor 3? So it seems that in us-east it's all
fine with each machine having 100% of the data. The problem is in eu-west
where you have unbalanced nodes. You should rearrange the tokens in that
eu-west DC.
Check e.g. this token calculator to calculate good tokens:
http://www.geroba.com/cassandra/cassandra-token-calculator/
Info about rebalancing the cluster:
http://www.datastax.com/dev/blog/balancing-your-cassandra-cluster
BR,
Hannu
2013/11/13 Katriel Traum <ka...@google.com>
> Hello list,
>
> I have a problem with my cluster ownership not being as expected.
>
> I have 2 DC cluster using NetworkTopologyStrategy on and
> EC2MultiRegionSnitch with cassandra 1.1.5. My placement strategy for all
> keyspaces is: {eu-west: 3, us-east:3 }, and I have 6 nodes in eu-west and 3
> in us-east.
> I expected the resulting effective ownership for nodes in us-east to be
> 100% (Which it is), and %50 in eu-west (Which it is not).
>
> The cluster started out with 3 nodes in eu-west, and after adding 3 more
> nodes, I got the following effective ownership:
>
> 1.1.1.1 eu-west 1a Up Normal 89.33 GB 83.33%
> 0
> 2.2.2.1 us-east 1b Up Normal 88.84 GB 100.00%
> 1
> 1.1.1.4 eu-west 1a Up Normal 75.61 GB 16.67%
> 28356863910078205288614550619314017621
> 1.1.1.2 eu-west 1b Up Normal 89 GB 83.33%
> 56713727820156410577229101238628035242
> 2.2.2.2 us-east 1d Up Normal 88.34 GB 100.00%
> 56713727820156410577229101238628035243
> 1.1.1.5 eu-west 1b Up Normal 75.43 GB 16.67%
> 85070591730234615865843651857942052863
> 1.1.1.3 eu-west 1c Up Normal 81.1 GB 83.33%
> 113427455640312821154458202477256070484
> 2.2.2.3 us-east 1e Up Normal 93.22 GB 100.00%
> 113427455640312821154458202477256070485
> 1.1.1.6 eu-west 1c Up Normal 75.39 GB 16.67%
> 141784319550391026443072753096570088105
>
> What am I missing here?
>
> TIA,
> Katriel
>