You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dikang Gu <di...@gmail.com> on 2011/04/12 11:00:34 UTC
Questions about the nodetool ring.
I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
[root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p 8090
ring
Address Status State Load Owns Token
109028275973926493413574716008500203721
192.168.1.25 Up Normal 157.25 MB 69.92%
57856537434773737201679995572503935972
192.168.1.27 Up Normal 201.71 MB 24.28%
99165710459060760249270263771474737125
192.168.1.28 Up Normal 68.12 MB 5.80%
109028275973926493413574716008500203721
The load and owns vary on each node, is this normal? And is there a way to
balance the three nodes?
Thanks.
--
Dikang Gu
0086 - 18611140205
Re: Questions about the nodetool ring.
Posted by aaron morton <aa...@thelastpickle.com>.
If you are seeing a different views of the ring from different nodes you may have some sickness
http://www.datastax.com/docs/0.7/troubleshooting/index#view-of-ring-differs-between-some-nodes
The "?" in the ring output happens when one node does not know if the other is alice or dead. This could be due to the corrupt gossip state described in the link above.
During a move the node will decommission and stop taking requests for the rang it was responsible for. But other nodes int he cluster will take it's place. Once it starts bootstrapping it will start accepting writes but not reads. The cluster stays online for all token ranges.
Dikang, did you allow the first move to complete before starting the second?
Aaron
On 12 Apr 2011, at 22:55, Jonathan Colby wrote:
> when you do a move, the node is decommissioned and bootstrapped. During the autobootstrap process the node will not receive reads until bootstrapping is complete. I assume during the decommission phase the node will also be unavailable, someone correct me if I'm wrong.
>
> the ring distribution looks better now.
>
> The "?" I get all the time too. And if you run "ring" against different hosts, the question marks probably appear in different places. I'm not sure if it means there is a problem. I haven't taken those question marks too seriously.
>
>
>
> On Apr 12, 2011, at 11:57 AM, Dikang Gu wrote:
>
>> After the nodetool move, I got this:
>>
>> [root@server3 apache-cassandra-0.7.4]# bin/nodetool -h 10.18.101.213 ring
>> Address Status State Load Owns Token
>> 113427455640312821154458202477256070485
>> 10.18.101.211 ? Normal 82.31 MB 33.33% 0
>> 10.18.101.212 ? Normal 84.24 MB 33.33% 56713727820156410577229101238628035242
>> 10.18.101.213 Up Normal 54.44 MB 33.33% 113427455640312821154458202477256070485
>>
>> Is this correct? Why is the status "?" ?
>>
>> Thanks.
>>
>> On Tue, Apr 12, 2011 at 5:43 PM, Dikang Gu <di...@gmail.com> wrote:
>> The 3 nodes were added to the cluster at the same time, so I'm not sure whey the data vary.
>>
>> I calculate the tokens and get:
>> node 0: 0
>> node 1: 56713727820156410577229101238628035242
>> node 2: 113427455640312821154458202477256070485
>>
>> So I should set these tokens to the three nodes?
>>
>> And during the time I execute the nodetool move commands, can the cassandra servers serve the front end requests at the same time? Is the data safe?
>>
>> Thanks.
>>
>> On Tue, Apr 12, 2011 at 5:15 PM, Jonathan Colby <jo...@gmail.com> wrote:
>> This is "normal" when you just add single nodes. When no token is assigned, the new node takes a portion of the ring from the most heavily loaded node. As a consequence of this, the nodes will be out of balance.
>>
>> In other words, when you double the amount nodes you would not have this problem.
>>
>> The best way to rebalance the cluster is to generate new tokens and use the nodetool move <new-token> command to rebalance the nodes, one at a time.
>>
>> After rebalancing you can run "cleanup" so the nodes get rid of data they no longer are responsible for.
>>
>> links:
>>
>> http://wiki.apache.org/cassandra/Operations#Range_changes
>>
>> http://wiki.apache.org/cassandra/Operations#Moving_or_Removing_nodes
>>
>> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>>
>>
>>
>> On Apr 12, 2011, at 11:00 AM, Dikang Gu wrote:
>>
>> > I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
>> >
>> > [root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p 8090 ring
>> > Address Status State Load Owns Token
>> > 109028275973926493413574716008500203721
>> > 192.168.1.25 Up Normal 157.25 MB 69.92% 57856537434773737201679995572503935972
>> > 192.168.1.27 Up Normal 201.71 MB 24.28% 99165710459060760249270263771474737125
>> > 192.168.1.28 Up Normal 68.12 MB 5.80% 109028275973926493413574716008500203721
>> >
>> > The load and owns vary on each node, is this normal? And is there a way to balance the three nodes?
>> >
>> > Thanks.
>> >
>> > --
>> > Dikang Gu
>> >
>> > 0086 - 18611140205
>> >
>>
>>
>>
>>
>> --
>> Dikang Gu
>>
>> 0086 - 18611140205
>>
>>
>>
>>
>> --
>> Dikang Gu
>>
>> 0086 - 18611140205
>>
>
Re: Questions about the nodetool ring.
Posted by Jonathan Colby <jo...@gmail.com>.
when you do a move, the node is decommissioned and bootstrapped. During the autobootstrap process the node will not receive reads until bootstrapping is complete. I assume during the decommission phase the node will also be unavailable, someone correct me if I'm wrong.
the ring distribution looks better now.
The "?" I get all the time too. And if you run "ring" against different hosts, the question marks probably appear in different places. I'm not sure if it means there is a problem. I haven't taken those question marks too seriously.
On Apr 12, 2011, at 11:57 AM, Dikang Gu wrote:
> After the nodetool move, I got this:
>
> [root@server3 apache-cassandra-0.7.4]# bin/nodetool -h 10.18.101.213 ring
> Address Status State Load Owns Token
> 113427455640312821154458202477256070485
> 10.18.101.211 ? Normal 82.31 MB 33.33% 0
> 10.18.101.212 ? Normal 84.24 MB 33.33% 56713727820156410577229101238628035242
> 10.18.101.213 Up Normal 54.44 MB 33.33% 113427455640312821154458202477256070485
>
> Is this correct? Why is the status "?" ?
>
> Thanks.
>
> On Tue, Apr 12, 2011 at 5:43 PM, Dikang Gu <di...@gmail.com> wrote:
> The 3 nodes were added to the cluster at the same time, so I'm not sure whey the data vary.
>
> I calculate the tokens and get:
> node 0: 0
> node 1: 56713727820156410577229101238628035242
> node 2: 113427455640312821154458202477256070485
>
> So I should set these tokens to the three nodes?
>
> And during the time I execute the nodetool move commands, can the cassandra servers serve the front end requests at the same time? Is the data safe?
>
> Thanks.
>
> On Tue, Apr 12, 2011 at 5:15 PM, Jonathan Colby <jo...@gmail.com> wrote:
> This is "normal" when you just add single nodes. When no token is assigned, the new node takes a portion of the ring from the most heavily loaded node. As a consequence of this, the nodes will be out of balance.
>
> In other words, when you double the amount nodes you would not have this problem.
>
> The best way to rebalance the cluster is to generate new tokens and use the nodetool move <new-token> command to rebalance the nodes, one at a time.
>
> After rebalancing you can run "cleanup" so the nodes get rid of data they no longer are responsible for.
>
> links:
>
> http://wiki.apache.org/cassandra/Operations#Range_changes
>
> http://wiki.apache.org/cassandra/Operations#Moving_or_Removing_nodes
>
> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>
>
>
> On Apr 12, 2011, at 11:00 AM, Dikang Gu wrote:
>
> > I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
> >
> > [root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p 8090 ring
> > Address Status State Load Owns Token
> > 109028275973926493413574716008500203721
> > 192.168.1.25 Up Normal 157.25 MB 69.92% 57856537434773737201679995572503935972
> > 192.168.1.27 Up Normal 201.71 MB 24.28% 99165710459060760249270263771474737125
> > 192.168.1.28 Up Normal 68.12 MB 5.80% 109028275973926493413574716008500203721
> >
> > The load and owns vary on each node, is this normal? And is there a way to balance the three nodes?
> >
> > Thanks.
> >
> > --
> > Dikang Gu
> >
> > 0086 - 18611140205
> >
>
>
>
>
> --
> Dikang Gu
>
> 0086 - 18611140205
>
>
>
>
> --
> Dikang Gu
>
> 0086 - 18611140205
>
Re: Questions about the nodetool ring.
Posted by Dikang Gu <di...@gmail.com>.
After the nodetool move, I got this:
[root@server3 apache-cassandra-0.7.4]# bin/nodetool -h 10.18.101.213 ring
Address Status State Load Owns Token
113427455640312821154458202477256070485
10.18.101.211 ? Normal 82.31 MB 33.33% 0
10.18.101.212 ? Normal 84.24 MB 33.33%
56713727820156410577229101238628035242
10.18.101.213 Up Normal 54.44 MB 33.33%
113427455640312821154458202477256070485
Is this correct? Why is the status "?" ?
Thanks.
On Tue, Apr 12, 2011 at 5:43 PM, Dikang Gu <di...@gmail.com> wrote:
> The 3 nodes were added to the cluster at the same time, so I'm not sure
> whey the data vary.
>
> I calculate the tokens and get:
> node 0: 0
> node 1: 56713727820156410577229101238628035242
> node 2: 113427455640312821154458202477256070485
>
> So I should set these tokens to the three nodes?
>
> And during the time I execute the nodetool move commands, can the cassandra
> servers serve the front end requests at the same time? Is the data safe?
>
> Thanks.
>
> On Tue, Apr 12, 2011 at 5:15 PM, Jonathan Colby <jo...@gmail.com>wrote:
>
>> This is "normal" when you just add single nodes. When no token is
>> assigned, the new node takes a portion of the ring from the most heavily
>> loaded node. As a consequence of this, the nodes will be out of balance.
>>
>> In other words, when you double the amount nodes you would not have this
>> problem.
>>
>> The best way to rebalance the cluster is to generate new tokens and use
>> the nodetool move <new-token> command to rebalance the nodes, one at a time.
>>
>> After rebalancing you can run "cleanup" so the nodes get rid of data they
>> no longer are responsible for.
>>
>> links:
>>
>> http://wiki.apache.org/cassandra/Operations#Range_changes
>>
>> http://wiki.apache.org/cassandra/Operations#Moving_or_Removing_nodes
>>
>> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>>
>>
>>
>> On Apr 12, 2011, at 11:00 AM, Dikang Gu wrote:
>>
>> > I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
>> >
>> > [root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p
>> 8090 ring
>> > Address Status State Load Owns Token
>> >
>> 109028275973926493413574716008500203721
>> > 192.168.1.25 Up Normal 157.25 MB 69.92%
>> 57856537434773737201679995572503935972
>> > 192.168.1.27 Up Normal 201.71 MB 24.28%
>> 99165710459060760249270263771474737125
>> > 192.168.1.28 Up Normal 68.12 MB 5.80%
>> 109028275973926493413574716008500203721
>> >
>> > The load and owns vary on each node, is this normal? And is there a way
>> to balance the three nodes?
>> >
>> > Thanks.
>> >
>> > --
>> > Dikang Gu
>> >
>> > 0086 - 18611140205
>> >
>>
>>
>
>
> --
> Dikang Gu
>
> 0086 - 18611140205
>
>
--
Dikang Gu
0086 - 18611140205
Re: Questions about the nodetool ring.
Posted by Dikang Gu <di...@gmail.com>.
The 3 nodes were added to the cluster at the same time, so I'm not sure whey
the data vary.
I calculate the tokens and get:
node 0: 0
node 1: 56713727820156410577229101238628035242
node 2: 113427455640312821154458202477256070485
So I should set these tokens to the three nodes?
And during the time I execute the nodetool move commands, can the cassandra
servers serve the front end requests at the same time? Is the data safe?
Thanks.
On Tue, Apr 12, 2011 at 5:15 PM, Jonathan Colby <jo...@gmail.com>wrote:
> This is "normal" when you just add single nodes. When no token is
> assigned, the new node takes a portion of the ring from the most heavily
> loaded node. As a consequence of this, the nodes will be out of balance.
>
> In other words, when you double the amount nodes you would not have this
> problem.
>
> The best way to rebalance the cluster is to generate new tokens and use the
> nodetool move <new-token> command to rebalance the nodes, one at a time.
>
> After rebalancing you can run "cleanup" so the nodes get rid of data they
> no longer are responsible for.
>
> links:
>
> http://wiki.apache.org/cassandra/Operations#Range_changes
>
> http://wiki.apache.org/cassandra/Operations#Moving_or_Removing_nodes
>
> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>
>
>
> On Apr 12, 2011, at 11:00 AM, Dikang Gu wrote:
>
> > I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
> >
> > [root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p
> 8090 ring
> > Address Status State Load Owns Token
> >
> 109028275973926493413574716008500203721
> > 192.168.1.25 Up Normal 157.25 MB 69.92%
> 57856537434773737201679995572503935972
> > 192.168.1.27 Up Normal 201.71 MB 24.28%
> 99165710459060760249270263771474737125
> > 192.168.1.28 Up Normal 68.12 MB 5.80%
> 109028275973926493413574716008500203721
> >
> > The load and owns vary on each node, is this normal? And is there a way
> to balance the three nodes?
> >
> > Thanks.
> >
> > --
> > Dikang Gu
> >
> > 0086 - 18611140205
> >
>
>
--
Dikang Gu
0086 - 18611140205
Re: Questions about the nodetool ring.
Posted by Jonathan Colby <jo...@gmail.com>.
This is "normal" when you just add single nodes. When no token is assigned, the new node takes a portion of the ring from the most heavily loaded node. As a consequence of this, the nodes will be out of balance.
In other words, when you double the amount nodes you would not have this problem.
The best way to rebalance the cluster is to generate new tokens and use the nodetool move <new-token> command to rebalance the nodes, one at a time.
After rebalancing you can run "cleanup" so the nodes get rid of data they no longer are responsible for.
links:
http://wiki.apache.org/cassandra/Operations#Range_changes
http://wiki.apache.org/cassandra/Operations#Moving_or_Removing_nodes
http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
On Apr 12, 2011, at 11:00 AM, Dikang Gu wrote:
> I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
>
> [root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p 8090 ring
> Address Status State Load Owns Token
> 109028275973926493413574716008500203721
> 192.168.1.25 Up Normal 157.25 MB 69.92% 57856537434773737201679995572503935972
> 192.168.1.27 Up Normal 201.71 MB 24.28% 99165710459060760249270263771474737125
> 192.168.1.28 Up Normal 68.12 MB 5.80% 109028275973926493413574716008500203721
>
> The load and owns vary on each node, is this normal? And is there a way to balance the three nodes?
>
> Thanks.
>
> --
> Dikang Gu
>
> 0086 - 18611140205
>