You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Sasha Dolgy <sd...@gmail.com> on 2011/03/14 23:27:21 UTC
Re: nodetool loadbalance
With my six node cluster ... nodetool loadbalance should be run on one
node or all six? I run it on one and the ownership percentage gets
even more unbalanced. So... in the spirit of the evening, I run it on
another node ..... as you see, the ownership % keeps increasing and
the token numbers keep increasing. is this expected? I would have
hoped that the behaviour would be a little bit different. Maybe I've
missed something big....
Address Status State Load Owns Token
124804735337540159479107746638263794797
10.0.0.1 Up Normal 225 KB 40.78%
24053088190195663439419935163232881936
10.0.0.2 Up Normal 201.21 KB 19.17%
56667357399723182105247119364967854254
10.0.0.3 Up Normal 213.15 KB 17.61%
86624712919272143003828971968762407027
10.0.0.4 Up Normal 214.54 KB 11.22%
105714724128406151241468359303513100912
10.0.0.5 Up Normal 206.39 KB 5.61%
115259729732973155360288052970888447854
10.0.0.6 Up Normal 242.93 KB 5.61%
124804735337540159479107746638263794797
Address Status State Load Owns Token
124804735337540159479107746638263794797
10.0.0.2 Up Normal 206.15 KB 59.95%
56667357399723182105247119364967854254
10.0.0.3 Up Normal 218.1 KB 17.61%
86624712919272143003828971968762407027
10.0.0.4 Up Normal 219.48 KB 11.22%
105714724128406151241468359303513100912
10.0.0.5 Up Normal 216.31 KB 5.61%
115259729732973155360288052970888447854
10.0.0.1 Up Normal 224.32 KB 2.81%
120032232535256657419697899804576121325
10.0.0.6 Up Normal 242.93 KB 2.81%
124804735337540159479107746638263794797
Address Status State Load Owns Token
124804735337540159479107746638263794797
10.0.0.3 Up Normal 218.1 KB 77.56%
86624712919272143003828971968762407027
10.0.0.4 Up Normal 225.71 KB 11.22%
105714724128406151241468359303513100912
10.0.0.5 Up Normal 216.31 KB 5.61%
115259729732973155360288052970888447854
10.0.0.1 Up Normal 224.32 KB 2.81%
120032232535256657419697899804576121325
10.0.0.2 Up Normal 201.21 KB 1.40%
122418483936398408449402823221419958061
10.0.0.6 Up Normal 242.93 KB 1.40%
124804735337540159479107746638263794797
Address Status State Load Owns Token
124804735337540159479107746638263794797
10.0.0.4 Up Normal 225.71 KB 88.78%
105714724128406151241468359303513100912
10.0.0.5 Up Normal 225.61 KB 5.61%
115259729732973155360288052970888447854
10.0.0.1 Up Normal 224.32 KB 2.81%
120032232535256657419697899804576121325
10.0.0.2 Up Normal 201.21 KB 1.40%
122418483936398408449402823221419958061
10.0.0.3 Up Normal 218.13 KB 0.70%
123611609636969283964255284929841876429
10.0.0.6 Up Normal 242.93 KB 0.70%
124804735337540159479107746638263794797
Address Status State Load Owns Token
124804735337540159479107746638263794797
10.0.0.5 Up Leaving 225.61 KB 94.39%
115259729732973155360288052970888447854
10.0.0.1 Up Normal 214.39 KB 2.81%
120032232535256657419697899804576121325
10.0.0.2 Up Normal 201.21 KB 1.40%
122418483936398408449402823221419958061
10.0.0.3 Up Normal 218.13 KB 0.70%
123611609636969283964255284929841876429
10.0.0.4 Up Normal 220.77 KB 0.35%
124208172487254721721681515784052835613
10.0.0.6 Up Normal 248.08 KB 0.35%
124804735337540159479107746638263794797
124804735337540159479107746638263794797
10.0.0.1 Up Normal 214.39 KB 97.19%
120032232535256657419697899804576121325
10.0.0.2 Up Normal 205.47 KB 1.40%
122418483936398408449402823221419958061
10.0.0.3 Up Normal 218.13 KB 0.70%
123611609636969283964255284929841876429
10.0.0.4 Up Normal 220.77 KB 0.35%
124208172487254721721681515784052835613
10.0.0.5 Up Normal 220.63 KB 0.18%
124506453912397440600394631211158315205
10.0.0.6 Up Normal 248.08 KB 0.18%
124804735337540159479107746638263794797
97.19% ... awesome.
On Mon, Mar 14, 2011 at 10:47 PM, Narendra Sharma
<na...@gmail.com> wrote:
> On the same page there is a section on Load Balance that talks about python
> script to compute tokens. I believe your question is more about assigning
> new tokens and not compute tokens.
>
> 1. "nodetool loadbalance" will result in recomputation of tokens. It will
> pick tokens based on the load and not the once assigned by you.
> 2. You can either use decommission and bootstrap with new tokens OR Use
> "nodetool move"
>
> Thanks,
> Naren
Re: nodetool loadbalance
Posted by Sasha Dolgy <sd...@gmail.com>.
Yes, a lot of what is on the wiki makes perfect sense when read the right
way. suppose there arent enough pictures or before/after info online to
help the knowledge flow.
On Mar 14, 2011 11:52 PM, "Jonathan Ellis" <jb...@gmail.com> wrote:
> You should read http://wiki.apache.org/cassandra/Operations before
> running loadbalance.
>
> On Mon, Mar 14, 2011 at 5:27 PM, Sasha Dolgy <sd...@gmail.com> wrote:
>> With my six node cluster ... nodetool loadbalance should be run on one
>> node or all six? I run it on one and the ownership percentage gets
>> even more unbalanced. So... in the spirit of the evening, I run it on
>> another node ..... as you see, the ownership % keeps increasing and
>> the token numbers keep increasing. is this expected? I would have
>> hoped that the behaviour would be a little bit different. Maybe I've
>> missed something big....
>>
>>
>>
>> Address Status State Load Owns Token
>>
>> 124804735337540159479107746638263794797
>> 10.0.0.1 Up Normal 225 KB 40.78%
>> 24053088190195663439419935163232881936
>> 10.0.0.2 Up Normal 201.21 KB 19.17%
>> 56667357399723182105247119364967854254
>> 10.0.0.3 Up Normal 213.15 KB 17.61%
>> 86624712919272143003828971968762407027
>> 10.0.0.4 Up Normal 214.54 KB 11.22%
>> 105714724128406151241468359303513100912
>> 10.0.0.5 Up Normal 206.39 KB 5.61%
>> 115259729732973155360288052970888447854
>> 10.0.0.6 Up Normal 242.93 KB 5.61%
>> 124804735337540159479107746638263794797
>>
>>
>> Address Status State Load Owns Token
>>
>> 124804735337540159479107746638263794797
>> 10.0.0.2 Up Normal 206.15 KB 59.95%
>> 56667357399723182105247119364967854254
>> 10.0.0.3 Up Normal 218.1 KB 17.61%
>> 86624712919272143003828971968762407027
>> 10.0.0.4 Up Normal 219.48 KB 11.22%
>> 105714724128406151241468359303513100912
>> 10.0.0.5 Up Normal 216.31 KB 5.61%
>> 115259729732973155360288052970888447854
>> 10.0.0.1 Up Normal 224.32 KB 2.81%
>> 120032232535256657419697899804576121325
>> 10.0.0.6 Up Normal 242.93 KB 2.81%
>> 124804735337540159479107746638263794797
>>
>>
>> Address Status State Load Owns Token
>>
>> 124804735337540159479107746638263794797
>> 10.0.0.3 Up Normal 218.1 KB 77.56%
>> 86624712919272143003828971968762407027
>> 10.0.0.4 Up Normal 225.71 KB 11.22%
>> 105714724128406151241468359303513100912
>> 10.0.0.5 Up Normal 216.31 KB 5.61%
>> 115259729732973155360288052970888447854
>> 10.0.0.1 Up Normal 224.32 KB 2.81%
>> 120032232535256657419697899804576121325
>> 10.0.0.2 Up Normal 201.21 KB 1.40%
>> 122418483936398408449402823221419958061
>> 10.0.0.6 Up Normal 242.93 KB 1.40%
>> 124804735337540159479107746638263794797
>>
>> Address Status State Load Owns Token
>>
>> 124804735337540159479107746638263794797
>> 10.0.0.4 Up Normal 225.71 KB 88.78%
>> 105714724128406151241468359303513100912
>> 10.0.0.5 Up Normal 225.61 KB 5.61%
>> 115259729732973155360288052970888447854
>> 10.0.0.1 Up Normal 224.32 KB 2.81%
>> 120032232535256657419697899804576121325
>> 10.0.0.2 Up Normal 201.21 KB 1.40%
>> 122418483936398408449402823221419958061
>> 10.0.0.3 Up Normal 218.13 KB 0.70%
>> 123611609636969283964255284929841876429
>> 10.0.0.6 Up Normal 242.93 KB 0.70%
>> 124804735337540159479107746638263794797
>>
>>
>> Address Status State Load Owns Token
>>
>> 124804735337540159479107746638263794797
>> 10.0.0.5 Up Leaving 225.61 KB 94.39%
>> 115259729732973155360288052970888447854
>> 10.0.0.1 Up Normal 214.39 KB 2.81%
>> 120032232535256657419697899804576121325
>> 10.0.0.2 Up Normal 201.21 KB 1.40%
>> 122418483936398408449402823221419958061
>> 10.0.0.3 Up Normal 218.13 KB 0.70%
>> 123611609636969283964255284929841876429
>> 10.0.0.4 Up Normal 220.77 KB 0.35%
>> 124208172487254721721681515784052835613
>> 10.0.0.6 Up Normal 248.08 KB 0.35%
>> 124804735337540159479107746638263794797
>>
>>
>>
>> 124804735337540159479107746638263794797
>> 10.0.0.1 Up Normal 214.39 KB 97.19%
>> 120032232535256657419697899804576121325
>> 10.0.0.2 Up Normal 205.47 KB 1.40%
>> 122418483936398408449402823221419958061
>> 10.0.0.3 Up Normal 218.13 KB 0.70%
>> 123611609636969283964255284929841876429
>> 10.0.0.4 Up Normal 220.77 KB 0.35%
>> 124208172487254721721681515784052835613
>> 10.0.0.5 Up Normal 220.63 KB 0.18%
>> 124506453912397440600394631211158315205
>> 10.0.0.6 Up Normal 248.08 KB 0.18%
>> 124804735337540159479107746638263794797
>>
>>
>> 97.19% ... awesome.
>>
>>
>> On Mon, Mar 14, 2011 at 10:47 PM, Narendra Sharma
>> <na...@gmail.com> wrote:
>>> On the same page there is a section on Load Balance that talks about
python
>>> script to compute tokens. I believe your question is more about
assigning
>>> new tokens and not compute tokens.
>>>
>>> 1. "nodetool loadbalance" will result in recomputation of tokens. It
will
>>> pick tokens based on the load and not the once assigned by you.
>>> 2. You can either use decommission and bootstrap with new tokens OR Use
>>> "nodetool move"
>>>
>>> Thanks,
>>> Naren
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: nodetool loadbalance
Posted by Jonathan Ellis <jb...@gmail.com>.
You should read http://wiki.apache.org/cassandra/Operations before
running loadbalance.
On Mon, Mar 14, 2011 at 5:27 PM, Sasha Dolgy <sd...@gmail.com> wrote:
> With my six node cluster ... nodetool loadbalance should be run on one
> node or all six? I run it on one and the ownership percentage gets
> even more unbalanced. So... in the spirit of the evening, I run it on
> another node ..... as you see, the ownership % keeps increasing and
> the token numbers keep increasing. is this expected? I would have
> hoped that the behaviour would be a little bit different. Maybe I've
> missed something big....
>
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.1 Up Normal 225 KB 40.78%
> 24053088190195663439419935163232881936
> 10.0.0.2 Up Normal 201.21 KB 19.17%
> 56667357399723182105247119364967854254
> 10.0.0.3 Up Normal 213.15 KB 17.61%
> 86624712919272143003828971968762407027
> 10.0.0.4 Up Normal 214.54 KB 11.22%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 206.39 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.6 Up Normal 242.93 KB 5.61%
> 124804735337540159479107746638263794797
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.2 Up Normal 206.15 KB 59.95%
> 56667357399723182105247119364967854254
> 10.0.0.3 Up Normal 218.1 KB 17.61%
> 86624712919272143003828971968762407027
> 10.0.0.4 Up Normal 219.48 KB 11.22%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 216.31 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 224.32 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.6 Up Normal 242.93 KB 2.81%
> 124804735337540159479107746638263794797
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.3 Up Normal 218.1 KB 77.56%
> 86624712919272143003828971968762407027
> 10.0.0.4 Up Normal 225.71 KB 11.22%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 216.31 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 224.32 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 201.21 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.6 Up Normal 242.93 KB 1.40%
> 124804735337540159479107746638263794797
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.4 Up Normal 225.71 KB 88.78%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 225.61 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 224.32 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 201.21 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.3 Up Normal 218.13 KB 0.70%
> 123611609636969283964255284929841876429
> 10.0.0.6 Up Normal 242.93 KB 0.70%
> 124804735337540159479107746638263794797
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.5 Up Leaving 225.61 KB 94.39%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 214.39 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 201.21 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.3 Up Normal 218.13 KB 0.70%
> 123611609636969283964255284929841876429
> 10.0.0.4 Up Normal 220.77 KB 0.35%
> 124208172487254721721681515784052835613
> 10.0.0.6 Up Normal 248.08 KB 0.35%
> 124804735337540159479107746638263794797
>
>
>
> 124804735337540159479107746638263794797
> 10.0.0.1 Up Normal 214.39 KB 97.19%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 205.47 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.3 Up Normal 218.13 KB 0.70%
> 123611609636969283964255284929841876429
> 10.0.0.4 Up Normal 220.77 KB 0.35%
> 124208172487254721721681515784052835613
> 10.0.0.5 Up Normal 220.63 KB 0.18%
> 124506453912397440600394631211158315205
> 10.0.0.6 Up Normal 248.08 KB 0.18%
> 124804735337540159479107746638263794797
>
>
> 97.19% ... awesome.
>
>
> On Mon, Mar 14, 2011 at 10:47 PM, Narendra Sharma
> <na...@gmail.com> wrote:
>> On the same page there is a section on Load Balance that talks about python
>> script to compute tokens. I believe your question is more about assigning
>> new tokens and not compute tokens.
>>
>> 1. "nodetool loadbalance" will result in recomputation of tokens. It will
>> pick tokens based on the load and not the once assigned by you.
>> 2. You can either use decommission and bootstrap with new tokens OR Use
>> "nodetool move"
>>
>> Thanks,
>> Naren
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: nodetool loadbalance
Posted by Sasha Dolgy <sd...@gmail.com>.
Using the tokens I generated earlier, i ran nodetool move <new token>
on each node and things look much better for the Owns % ...
Address Status State Load Owns Token
170141183460469231731687303715884105725
10.0.0.1 Up Normal 234.51 KB 0.00% 0
10.0.0.2 Up Normal 224.13 KB 20.00%
34028236692093846346337460743176821145
10.0.0.3 Up Normal 238.3 KB 40.00%
102084710076281539039012382229530463435
10.0.0.4 Up Normal 222.93 KB 12.65%
123611609636969283964255284929841876429
10.0.0.5 Up Normal 238.04 KB 7.35%
136112946768375385385349842972707284580
10.0.0.6 Up Normal 235.92 KB 20.00%
170141183460469231731687303715884105725
All of these nodes had been brought online using auto_bootstrap, with
exception for the very first one, which was a seed for the rest. Is
this a necessary task that has to be performed on all nodes as new
nodes are added ... to maintain a good balance ... in that the tokens
don't re-balance themselves?
Or, as this is a test environment with low load, there isn't enough
good data for the bootstrap to determine where best to put the new
node token?
-sd
On Mon, Mar 14, 2011 at 11:27 PM, Sasha Dolgy <sd...@gmail.com> wrote:
> With my six node cluster ... nodetool loadbalance should be run on one
> node or all six? I run it on one and the ownership percentage gets
> even more unbalanced. So... in the spirit of the evening, I run it on
> another node ..... as you see, the ownership % keeps increasing and
> the token numbers keep increasing. is this expected? I would have
> hoped that the behaviour would be a little bit different. Maybe I've
> missed something big....
>
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.1 Up Normal 225 KB 40.78%
> 24053088190195663439419935163232881936
> 10.0.0.2 Up Normal 201.21 KB 19.17%
> 56667357399723182105247119364967854254
> 10.0.0.3 Up Normal 213.15 KB 17.61%
> 86624712919272143003828971968762407027
> 10.0.0.4 Up Normal 214.54 KB 11.22%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 206.39 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.6 Up Normal 242.93 KB 5.61%
> 124804735337540159479107746638263794797
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.2 Up Normal 206.15 KB 59.95%
> 56667357399723182105247119364967854254
> 10.0.0.3 Up Normal 218.1 KB 17.61%
> 86624712919272143003828971968762407027
> 10.0.0.4 Up Normal 219.48 KB 11.22%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 216.31 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 224.32 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.6 Up Normal 242.93 KB 2.81%
> 124804735337540159479107746638263794797
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.3 Up Normal 218.1 KB 77.56%
> 86624712919272143003828971968762407027
> 10.0.0.4 Up Normal 225.71 KB 11.22%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 216.31 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 224.32 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 201.21 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.6 Up Normal 242.93 KB 1.40%
> 124804735337540159479107746638263794797
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.4 Up Normal 225.71 KB 88.78%
> 105714724128406151241468359303513100912
> 10.0.0.5 Up Normal 225.61 KB 5.61%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 224.32 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 201.21 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.3 Up Normal 218.13 KB 0.70%
> 123611609636969283964255284929841876429
> 10.0.0.6 Up Normal 242.93 KB 0.70%
> 124804735337540159479107746638263794797
>
>
> Address Status State Load Owns Token
>
> 124804735337540159479107746638263794797
> 10.0.0.5 Up Leaving 225.61 KB 94.39%
> 115259729732973155360288052970888447854
> 10.0.0.1 Up Normal 214.39 KB 2.81%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 201.21 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.3 Up Normal 218.13 KB 0.70%
> 123611609636969283964255284929841876429
> 10.0.0.4 Up Normal 220.77 KB 0.35%
> 124208172487254721721681515784052835613
> 10.0.0.6 Up Normal 248.08 KB 0.35%
> 124804735337540159479107746638263794797
>
>
>
> 124804735337540159479107746638263794797
> 10.0.0.1 Up Normal 214.39 KB 97.19%
> 120032232535256657419697899804576121325
> 10.0.0.2 Up Normal 205.47 KB 1.40%
> 122418483936398408449402823221419958061
> 10.0.0.3 Up Normal 218.13 KB 0.70%
> 123611609636969283964255284929841876429
> 10.0.0.4 Up Normal 220.77 KB 0.35%
> 124208172487254721721681515784052835613
> 10.0.0.5 Up Normal 220.63 KB 0.18%
> 124506453912397440600394631211158315205
> 10.0.0.6 Up Normal 248.08 KB 0.18%
> 124804735337540159479107746638263794797
>
>
> 97.19% ... awesome.