You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Tiwari, Tarun" <Ta...@Kronos.com> on 2015/04/22 07:14:18 UTC
Cluster imbalance caused due to #Num_Tokens
Hi,
While setting up a cluster for our POC, when we installed Cassandra on the 1st node we gave num_tokens: 256 , while on next 2 nodes which were added later we left it blank in Cassandra.yaml.
This made our cluster an unbalanced one with nodetool status showing 99% load on one server. Now even if I am setting up num tokens in the other 2 nodes as 256, its not seeming to effect. The wiki article http://wiki.apache.org/cassandra/VirtualNodes/Balance doesn't seem to provide steps to correct from this situation.
I read that there was nodetool balance kind of command in Cassandra 0.7 but not anymore.
UN Node3 23.72 MB 1 0.4% 41a71df-7e6c-40ab-902f-237697eaaf3e rack1
UN Node2 79.35 MB 1 0.5% 98c493b-f661-491e-9d1f-1803f859528b rack1
UN Node1 86.93 MB 256 99.1% a35ccca-556c-4f77-aa6d-7e3dad41ecf8 rack1
Is there something that we can do now balance the cluster?
Regards,
Tarun
Re: Cluster imbalance caused due to #Num_Tokens
Posted by Kiran mk <co...@gmail.com>.
Bring down the second node using nodetool removenode or decommission
Add the node with num_tokens and run the nodetool repair.
At last run the nodetool cleanup on both the nodes (one after the other)
Observe after some time using nodetool status.
On Apr 23, 2015 12:39 AM, "Robert Coli" <rc...@eventbrite.com> wrote:
> On Tue, Apr 21, 2015 at 10:14 PM, Tiwari, Tarun <Ta...@kronos.com>
> wrote:
>
>> I read that there was nodetool balance kind of command in Cassandra 0.7
>> but not anymore.
>>
>
> It never worked, really.
>
>
>> UN Node3 23.72 MB 1 0.4%
>> 41a71df-7e6c-40ab-902f-237697eaaf3e rack1
>>
>> UN Node2 79.35 MB 1 0.5% 98c493b-f661-491e-9d1f-1803f859528b
>> rack1
>>
>> UN Node1 86.93 MB 256 99.1% a35ccca-556c-4f77-aa6d-7e3dad41ecf8
>> rack1
>>
>>
>>
>> Is there something that we can do now balance the cluster?
>>
>
> Just run removenode on the two nodes with almost no data, one at a time.
>
> Then add them back with vnodes enabled, one at a time.
>
> As an aside, if the size of this cluster will never be greater than 9 or
> so nodes, you Mostly Lose by using vnodes.
>
> =Rob
>
>
Re: Cluster imbalance caused due to #Num_Tokens
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Apr 21, 2015 at 10:14 PM, Tiwari, Tarun <Ta...@kronos.com>
wrote:
> I read that there was nodetool balance kind of command in Cassandra 0.7
> but not anymore.
>
It never worked, really.
> UN Node3 23.72 MB 1 0.4%
> 41a71df-7e6c-40ab-902f-237697eaaf3e rack1
>
> UN Node2 79.35 MB 1 0.5% 98c493b-f661-491e-9d1f-1803f859528b
> rack1
>
> UN Node1 86.93 MB 256 99.1% a35ccca-556c-4f77-aa6d-7e3dad41ecf8
> rack1
>
>
>
> Is there something that we can do now balance the cluster?
>
Just run removenode on the two nodes with almost no data, one at a time.
Then add them back with vnodes enabled, one at a time.
As an aside, if the size of this cluster will never be greater than 9 or so
nodes, you Mostly Lose by using vnodes.
=Rob