You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Shuler (JIRA)" <ji...@apache.org> on 2014/09/24 19:52:34 UTC
[jira] [Resolved] (CASSANDRA-5264) Nodetool ring is reporting
incorrect Effective ownership after upgrading from 1.1.2 -> 1.1.9
[ https://issues.apache.org/jira/browse/CASSANDRA-5264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Shuler resolved CASSANDRA-5264.
---------------------------------------
Resolution: Not a Problem
The 1.1 and 1.2 branches are no longer under development. I'm going to close this ticket, as it appears Jonathan mentioned a simple explanation and this appears to be expected under the circumstances.
> Nodetool ring is reporting incorrect Effective ownership after upgrading from 1.1.2 -> 1.1.9
> --------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-5264
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5264
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.9
> Environment: Amazon AWS Linux, Large instance (8gig of RAM, ephemeral storage). 12 Node cluster. Replication Factor 3, all queries performed with LOCAL_QUORUM
> Reporter: Michael Theroux
> Priority: Minor
>
> We upgraded from Cassandra 1.1.2 to 1.1.9 yesterday. All indications are the upgrade went well. Repair works as expected, and all our data is available. Performance is as good, if not better, than it was previously.
> However, nodetool ring is reporting inconsistent and incorrect results. This was my ring information before the upgrade:
> Address DC Rack Status State Load Effective-Ownership Token
> Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> 10.0.4.22 us-east 1a Up Normal 77.75 GB 25.00% Token(bytes[00000000000000000000000000000001])
> 10.0.10.23 us-east 1d Up Normal 82.68 GB 25.00% Token(bytes[15555555555555555555555555555555])
> 10.0.8.20 us-east 1c Up Normal 81.72 GB 25.00% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
> 10.0.4.23 us-east 1a Up Normal 82.65 GB 25.00% Token(bytes[40000000000000000000000000000000])
> 10.0.10.20 us-east 1d Up Normal 80.2 GB 25.00% Token(bytes[55555555555555555555555555555554])
> 10.0.8.23 us-east 1c Up Normal 77.06 GB 25.00% Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac])
> 10.0.4.21 us-east 1a Up Normal 81.37 GB 25.00% Token(bytes[80000000000000000000000000000000])
> 10.0.10.24 us-east 1d Up Normal 83.37 GB 25.00% Token(bytes[95555555555555555555555555555558])
> 10.0.8.21 us-east 1c Up Normal 84.33 GB 25.00% Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> 10.0.4.25 us-east 1a Up Normal 79.91 GB 25.00% Token(bytes[c0000000000000000000000000000000])
> 10.0.10.21 us-east 1d Up Normal 83.46 GB 25.00% Token(bytes[d5555555555555555555555555555558])
> 10.0.8.24 us-east 1c Up Normal 90.66 GB 25.00% Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> This is my ring information after the upgrade:
> 10.0.4.22 us-east 1a Up Normal 77.74 GB 99.89% Token(bytes[00000000000000000000000000000001])
> 10.0.10.23 us-east 1d Up Normal 82.82 GB 64.14% Token(bytes[15555555555555555555555555555555])
> 10.0.8.20 us-east 1c Up Normal 81.89 GB 30.55% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
> 10.0.4.23 us-east 1a Up Normal 82.77 GB 0.04% Token(bytes[40000000000000000000000000000000])
> 10.0.10.20 us-east 1d Up Normal 80.32 GB 0.04% Token(bytes[55555555555555555555555555555554])
> 10.0.8.23 us-east 1c Up Normal 77.07 GB 0.04% Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac])
> 10.0.4.21 us-east 1a Up Normal 81.35 GB 0.04% Token(bytes[80000000000000000000000000000000])
> 10.0.10.24 us-east 1d Up Normal 83.49 GB 0.04% Token(bytes[95555555555555555555555555555558])
> 10.0.8.21 us-east 1c Up Normal 84.47 GB 0.04% Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> 10.0.4.25 us-east 1a Up Normal 80.11 GB 0.04% Token(bytes[c0000000000000000000000000000000])
> 10.0.10.21 us-east 1d Up Normal 83.5 GB 35.79% Token(bytes[d5555555555555555555555555555558])
> 10.0.8.24 us-east 1c Up Normal 90.72 GB 69.38% Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> We use ByteOrderedPartitioning (we hash our own keys), and as you can see from above, were achieving a somewhat equal distribution of data amongst the nodes.
> The node that always seems to own 99% of the keys is the node that I run "nodetool ring" on. Running "nodetool ring" on two of these nodes at the same time resulted in:
> From 10.0.4.22:
> Address DC Rack Status State Load Effective-Ownership Token
> Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> 10.0.4.22 us-east 1a Up Normal 77.72 GB 99.89% Token(bytes[00000000000000000000000000000001])
> 10.0.10.23 us-east 1d Up Normal 82.74 GB 64.13% Token(bytes[15555555555555555555555555555555])
> 10.0.8.20 us-east 1c Up Normal 81.79 GB 30.55% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
> 10.0.4.23 us-east 1a Up Normal 82.66 GB 0.04% Token(bytes[40000000000000000000000000000000])
> 10.0.10.20 us-east 1d Up Normal 80.21 GB 0.04% Token(bytes[55555555555555555555555555555554])
> 10.0.8.23 us-east 1c Up Normal 77.07 GB 0.04% Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac])
> 10.0.4.21 us-east 1a Up Normal 81.38 GB 0.04% Token(bytes[80000000000000000000000000000000])
> 10.0.10.24 us-east 1d Up Normal 83.43 GB 0.04% Token(bytes[95555555555555555555555555555558])
> 10.0.8.21 us-east 1c Up Normal 84.42 GB 0.04% Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> 10.0.4.25 us-east 1a Up Normal 80.06 GB 0.04% Token(bytes[c0000000000000000000000000000000])
> 10.0.10.21 us-east 1d Up Normal 83.49 GB 35.80% Token(bytes[d5555555555555555555555555555558])
> 10.0.8.24 us-east 1c Up Normal 90.72 GB 69.37% Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> From 10.0.8.23:
> Address DC Rack Status State Load Effective-Ownership Token
> Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> 10.0.4.22 us-east 1a Up Normal 77.72 GB 0.04% Token(bytes[00000000000000000000000000000001])
> 10.0.10.23 us-east 1d Up Normal 82.78 GB 0.04% Token(bytes[15555555555555555555555555555555])
> 10.0.8.20 us-east 1c Up Normal 81.79 GB 0.04% Token(bytes[2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa])
> 10.0.4.23 us-east 1a Up Normal 82.66 GB 33.84% Token(bytes[40000000000000000000000000000000])
> 10.0.10.20 us-east 1d Up Normal 80.21 GB 67.51% Token(bytes[55555555555555555555555555555554])
> 10.0.8.23 us-east 1c Up Normal 77.07 GB 99.89% Token(bytes[6aaaaaaaaaaaaaaaaaaaaaaaaaaaaaac])
> 10.0.4.21 us-east 1a Up Normal 81.38 GB 66.09% Token(bytes[80000000000000000000000000000000])
> 10.0.10.24 us-east 1d Up Normal 83.43 GB 32.41% Token(bytes[95555555555555555555555555555558])
> 10.0.8.21 us-east 1c Up Normal 84.42 GB 0.04% Token(bytes[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> 10.0.4.25 us-east 1a Up Normal 80.06 GB 0.04% Token(bytes[c0000000000000000000000000000000])
> 10.0.10.21 us-east 1d Up Normal 83.49 GB 0.04% Token(bytes[d5555555555555555555555555555558])
> 10.0.8.24 us-east 1c Up Normal 90.72 GB 0.04% Token(bytes[eaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8])
> I did a little digging, and do see some changes to how cassandra calculates the splits in 1.1.8 as part of CASSANDRA-4803 (StorageService.getSplits()), although I'm not familiar enough to tell if its a cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)