You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by buddhasystem <po...@bnl.gov> on 2011/04/01 00:06:49 UTC
Node added, no performance boost -- are the tokens correct?
I just configured a cluster of two nodes -- do these token values make sense?
The reason I'm asking that so far I don't see load balancing to be
happening, judging from performance.
Address Status State Load Owns Token
170141183460469231731687303715884105728
130.199.185.194 Up Normal 153.52 GB 50.00%
85070591730234615865843651857942052864
130.199.185.193 Up Normal 199.82 GB 50.00%
170141183460469231731687303715884105728
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Node added, no performance boost -- are the tokens correct?
Posted by Eric Gilmore <er...@datastax.com>.
The DS docs go with "should" regarding setting the initial token to zero.
It's not a "must," but you get enough convenience out of never having to
move tokens on that node that I'm not sure why you wouldn't do it.
If anyone has a compelling reason not to do so, I'm happy to hear it :)
On Fri, Apr 1, 2011 at 10:58 AM, Edward Capriolo <ed...@gmail.com>wrote:
> On Fri, Apr 1, 2011 at 1:15 PM, Peter Schuller
> <pe...@infidyne.com> wrote:
> >> Now, I moved the tokens. I still observe that read latency deteriorated
> with
> >> 3 machines vs original one. Replication factor is 1, Cassandra version
> 0.7.2
> >> (didn't have time to upgrade as I need results by this weekend).
> >
> > Read *latency* is fully expected to increase if you just add a node.
> > *Throughput* should increase, unless you have a workload that manages
> > to be more expensive on RPC than actual reads/writes.
> >
> > Latency would only be improved by additional nodes under some significant
> load.
> >
> > How are you benchmarking? Are you concurrently submitting requests to
> > all nodes at the same time? Try using stress.py from the Cassandra
> > tree as a comparison.
> >
> > If you're sending one request at a time, there is no expectation at
> > all of a performance improvement - just a decrease in performance.
> >
> > --
> > / Peter Schuller
> >
>
> To be clear on this issue. It does not matter where the tokens start
> it only matters that they are equally spaced around the token space.
>
> So for a 4 node clusters your tokens should either be
> 1 * ((2^127) / 4) = 42535295865117307932921825928971026432
> 2 * ((2^127) / 4) = 85070591730234615865843651857942052864
> 3 * ((2^127) / 4) = 127605887595351923798765477786913079296
> 4 * ((2^127) / 4) = 170141183460469231731687303715884105728
>
> or
> 0 * ((2^127) / 4) = 0
> 1 * ((2^127) / 4) = 42535295865117307932921825928971026432
> 2 * ((2^127) / 4) = 85070591730234615865843651857942052864
> 3 * ((2^127) / 4) = 127605887595351923798765477786913079296
>
> If you move one you have to move the rest because the distance between
> 170141183460469231731687303715884105728 and 0 is 1
>
Re: Node added, no performance boost -- are the tokens correct?
Posted by Edward Capriolo <ed...@gmail.com>.
On Fri, Apr 1, 2011 at 1:15 PM, Peter Schuller
<pe...@infidyne.com> wrote:
>> Now, I moved the tokens. I still observe that read latency deteriorated with
>> 3 machines vs original one. Replication factor is 1, Cassandra version 0.7.2
>> (didn't have time to upgrade as I need results by this weekend).
>
> Read *latency* is fully expected to increase if you just add a node.
> *Throughput* should increase, unless you have a workload that manages
> to be more expensive on RPC than actual reads/writes.
>
> Latency would only be improved by additional nodes under some significant load.
>
> How are you benchmarking? Are you concurrently submitting requests to
> all nodes at the same time? Try using stress.py from the Cassandra
> tree as a comparison.
>
> If you're sending one request at a time, there is no expectation at
> all of a performance improvement - just a decrease in performance.
>
> --
> / Peter Schuller
>
To be clear on this issue. It does not matter where the tokens start
it only matters that they are equally spaced around the token space.
So for a 4 node clusters your tokens should either be
1 * ((2^127) / 4) = 42535295865117307932921825928971026432
2 * ((2^127) / 4) = 85070591730234615865843651857942052864
3 * ((2^127) / 4) = 127605887595351923798765477786913079296
4 * ((2^127) / 4) = 170141183460469231731687303715884105728
or
0 * ((2^127) / 4) = 0
1 * ((2^127) / 4) = 42535295865117307932921825928971026432
2 * ((2^127) / 4) = 85070591730234615865843651857942052864
3 * ((2^127) / 4) = 127605887595351923798765477786913079296
If you move one you have to move the rest because the distance between
170141183460469231731687303715884105728 and 0 is 1
Re: Node added, no performance boost -- are the tokens correct?
Posted by Peter Schuller <pe...@infidyne.com>.
> Now, I moved the tokens. I still observe that read latency deteriorated with
> 3 machines vs original one. Replication factor is 1, Cassandra version 0.7.2
> (didn't have time to upgrade as I need results by this weekend).
Read *latency* is fully expected to increase if you just add a node.
*Throughput* should increase, unless you have a workload that manages
to be more expensive on RPC than actual reads/writes.
Latency would only be improved by additional nodes under some significant load.
How are you benchmarking? Are you concurrently submitting requests to
all nodes at the same time? Try using stress.py from the Cassandra
tree as a comparison.
If you're sending one request at a time, there is no expectation at
all of a performance improvement - just a decrease in performance.
--
/ Peter Schuller
Re: Node added, no performance boost -- are the tokens correct?
Posted by buddhasystem <po...@bnl.gov>.
Yup, I screwed up the token setting, my bad.
Now, I moved the tokens. I still observe that read latency deteriorated with
3 machines vs original one. Replication factor is 1, Cassandra version 0.7.2
(didn't have time to upgrade as I need results by this weekend).
Key and row caching was disabled to get the worse case scenario test
results.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6229564.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Node added, no performance boost -- are the tokens correct?
Posted by Edward Capriolo <ed...@gmail.com>.
On Thu, Mar 31, 2011 at 6:15 PM, Eric Gilmore <er...@datastax.com> wrote:
> A script that I have says the following:
>
> $ python ctokens.py
> How many nodes are in your cluster? 2
> node 0: 0
> node 1: 85070591730234615865843651857942052864
>
> The first token should be zero, for the reasons discussed here:
> http://www.datastax.com/dev/tutorials/getting_started_0_7/configuring#initial-token-values
>
> More details are available in
> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>
> The DS docs have some weak areas, but these two pages have been pretty well
> vetted over the past months :)
>
>
>
> On Thu, Mar 31, 2011 at 3:06 PM, buddhasystem <po...@bnl.gov> wrote:
>>
>> I just configured a cluster of two nodes -- do these token values make
>> sense?
>> The reason I'm asking that so far I don't see load balancing to be
>> happening, judging from performance.
>>
>> Address Status State Load Owns Token
>>
>> 170141183460469231731687303715884105728
>> 130.199.185.194 Up Normal 153.52 GB 50.00%
>> 85070591730234615865843651857942052864
>> 130.199.185.193 Up Normal 199.82 GB 50.00%
>> 170141183460469231731687303715884105728
>>
>>
>> --
>> View this message in context:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at
>> Nabble.com.
>
>
The first token does not "really" have to be zero. They just have been
spread evenly across the token space.
Re: Node added, no performance boost -- are the tokens correct?
Posted by buddhasystem <po...@bnl.gov>.
On two different clusters, if I set the token to zero, on a node, its
ownership drops to zero after migration.
After I added the third one and moved tokens, I now have this:
33.33% 56713727820156410577229101238628035242
33.33% 113427455640312821154458202477256070484
33.33% 170141183460469231731687303715884105727
No zeroes.
Eric Gilmore-3 wrote:
>
> A script that I have says the following:
>
> $ python ctokens.py
> How many nodes are in your cluster? 2
> node 0: 0
> node 1: 85070591730234615865843651857942052864
>
> The first token should be zero, for the reasons discussed here:
> http://www.datastax.com/dev/tutorials/getting_started_0_7/configuring#initial-token-values
>
> More details are available in
> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>
> The DS docs have some weak areas, but these two pages have been pretty
> well
> vetted over the past months :)
>
>
>
> On Thu, Mar 31, 2011 at 3:06 PM, buddhasystem <potekhin@bnl.gov>
> wrote:
>
>> I just configured a cluster of two nodes -- do these token values make
>> sense?
>> The reason I'm asking that so far I don't see load balancing to be
>> happening, judging from performance.
>>
>> Address Status State Load Owns Token
>>
>> 170141183460469231731687303715884105728
>> 130.199.185.194 Up Normal 153.52 GB 50.00%
>> 85070591730234615865843651857942052864
>> 130.199.185.193 Up Normal 199.82 GB 50.00%
>> 170141183460469231731687303715884105728
>>
>>
>> --
>> View this message in context:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at
>> Nabble.com.
>>
>
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6231845.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Node added, no performance boost -- are the tokens correct?
Posted by Eric Gilmore <er...@datastax.com>.
A script that I have says the following:
$ python ctokens.py
How many nodes are in your cluster? 2
node 0: 0
node 1: 85070591730234615865843651857942052864
The first token should be zero, for the reasons discussed here:
http://www.datastax.com/dev/tutorials/getting_started_0_7/configuring#initial-token-values
More details are available in
http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
The DS docs have some weak areas, but these two pages have been pretty well
vetted over the past months :)
On Thu, Mar 31, 2011 at 3:06 PM, buddhasystem <po...@bnl.gov> wrote:
> I just configured a cluster of two nodes -- do these token values make
> sense?
> The reason I'm asking that so far I don't see load balancing to be
> happening, judging from performance.
>
> Address Status State Load Owns Token
>
> 170141183460469231731687303715884105728
> 130.199.185.194 Up Normal 153.52 GB 50.00%
> 85070591730234615865843651857942052864
> 130.199.185.193 Up Normal 199.82 GB 50.00%
> 170141183460469231731687303715884105728
>
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>