You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by buddhasystem <po...@bnl.gov> on 2011/04/01 00:06:49 UTC

Node added, no performance boost -- are the tokens correct?

I just configured a cluster of two nodes -- do these token values make sense?
The reason I'm asking that so far I don't see load balancing to be
happening, judging from performance.

Address         Status State   Load            Owns    Token
                                                      
170141183460469231731687303715884105728
130.199.185.194 Up     Normal  153.52 GB       50.00% 
85070591730234615865843651857942052864
130.199.185.193 Up     Normal  199.82 GB       50.00% 
170141183460469231731687303715884105728


--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Node added, no performance boost -- are the tokens correct?

Posted by Eric Gilmore <er...@datastax.com>.
The DS docs go with "should" regarding setting the initial token to zero.
It's not a "must," but you get enough convenience out of never having to
move tokens on that node that I'm not sure why you wouldn't do it.

If anyone has a compelling reason not to do so, I'm happy to hear it :)

On Fri, Apr 1, 2011 at 10:58 AM, Edward Capriolo <ed...@gmail.com>wrote:

> On Fri, Apr 1, 2011 at 1:15 PM, Peter Schuller
> <pe...@infidyne.com> wrote:
> >> Now, I moved the tokens. I still observe that read latency deteriorated
> with
> >> 3 machines vs original one. Replication factor is 1, Cassandra version
> 0.7.2
> >> (didn't have time to upgrade as I need results by this weekend).
> >
> > Read *latency* is fully expected to increase if you just add a node.
> > *Throughput* should increase, unless you have a workload that manages
> > to be more expensive on RPC than actual reads/writes.
> >
> > Latency would only be improved by additional nodes under some significant
> load.
> >
> > How are you benchmarking? Are you concurrently submitting requests to
> > all nodes at the same time? Try using stress.py from the Cassandra
> > tree as a comparison.
> >
> > If you're sending one request at a time, there is no expectation at
> > all of a performance improvement - just a decrease in performance.
> >
> > --
> > / Peter Schuller
> >
>
> To be clear on this issue. It does not matter where the tokens start
> it only matters that they are equally spaced around the token space.
>
> So for a 4 node clusters your tokens should either be
> 1 * ((2^127) / 4) = 42535295865117307932921825928971026432
> 2 * ((2^127) / 4) = 85070591730234615865843651857942052864
> 3 * ((2^127) / 4) = 127605887595351923798765477786913079296
> 4 * ((2^127) / 4) = 170141183460469231731687303715884105728
>
> or
> 0 * ((2^127) / 4) = 0
> 1 * ((2^127) / 4) = 42535295865117307932921825928971026432
> 2 * ((2^127) / 4) = 85070591730234615865843651857942052864
> 3 * ((2^127) / 4) = 127605887595351923798765477786913079296
>
> If you move one you have to move the rest because the distance between
> 170141183460469231731687303715884105728 and 0 is 1
>

Re: Node added, no performance boost -- are the tokens correct?

Posted by Edward Capriolo <ed...@gmail.com>.
On Fri, Apr 1, 2011 at 1:15 PM, Peter Schuller
<pe...@infidyne.com> wrote:
>> Now, I moved the tokens. I still observe that read latency deteriorated with
>> 3 machines vs original one. Replication factor is 1, Cassandra version 0.7.2
>> (didn't have time to upgrade as I need results by this weekend).
>
> Read *latency* is fully expected to increase if you just add a node.
> *Throughput* should increase, unless you have a workload that manages
> to be more expensive on RPC than actual reads/writes.
>
> Latency would only be improved by additional nodes under some significant load.
>
> How are you benchmarking? Are you concurrently submitting requests to
> all nodes at the same time? Try using stress.py from the Cassandra
> tree as a comparison.
>
> If you're sending one request at a time, there is no expectation at
> all of a performance improvement - just a decrease in performance.
>
> --
> / Peter Schuller
>

To be clear on this issue. It does not matter where the tokens start
it only matters that they are equally spaced around the token space.

So for a 4 node clusters your tokens should either be
1 * ((2^127) / 4) = 42535295865117307932921825928971026432
2 * ((2^127) / 4) = 85070591730234615865843651857942052864
3 * ((2^127) / 4) = 127605887595351923798765477786913079296
4 * ((2^127) / 4) = 170141183460469231731687303715884105728

or
0 * ((2^127) / 4) = 0
1 * ((2^127) / 4) = 42535295865117307932921825928971026432
2 * ((2^127) / 4) = 85070591730234615865843651857942052864
3 * ((2^127) / 4) = 127605887595351923798765477786913079296

If you move one you have to move the rest because the distance between
170141183460469231731687303715884105728 and 0 is 1

Re: Node added, no performance boost -- are the tokens correct?

Posted by Peter Schuller <pe...@infidyne.com>.
> Now, I moved the tokens. I still observe that read latency deteriorated with
> 3 machines vs original one. Replication factor is 1, Cassandra version 0.7.2
> (didn't have time to upgrade as I need results by this weekend).

Read *latency* is fully expected to increase if you just add a node.
*Throughput* should increase, unless you have a workload that manages
to be more expensive on RPC than actual reads/writes.

Latency would only be improved by additional nodes under some significant load.

How are you benchmarking? Are you concurrently submitting requests to
all nodes at the same time? Try using stress.py from the Cassandra
tree as a comparison.

If you're sending one request at a time, there is no expectation at
all of a performance improvement - just a decrease in performance.

-- 
/ Peter Schuller

Re: Node added, no performance boost -- are the tokens correct?

Posted by buddhasystem <po...@bnl.gov>.
Yup, I screwed up the token setting, my bad.

Now, I moved the tokens. I still observe that read latency deteriorated with
3 machines vs original one. Replication factor is 1, Cassandra version 0.7.2
(didn't have time to upgrade as I need results by this weekend).

Key and row caching was disabled to get the worse case scenario test
results.


--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6229564.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Node added, no performance boost -- are the tokens correct?

Posted by Edward Capriolo <ed...@gmail.com>.
On Thu, Mar 31, 2011 at 6:15 PM, Eric Gilmore <er...@datastax.com> wrote:
> A script that I have says the following:
>
> $ python ctokens.py
> How many nodes are in your cluster? 2
> node 0: 0
> node 1: 85070591730234615865843651857942052864
>
> The first token should be zero, for the reasons discussed here:
> http://www.datastax.com/dev/tutorials/getting_started_0_7/configuring#initial-token-values
>
> More details are available in
> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>
> The DS docs have some weak areas, but these two pages have been pretty well
> vetted over the past months :)
>
>
>
> On Thu, Mar 31, 2011 at 3:06 PM, buddhasystem <po...@bnl.gov> wrote:
>>
>> I just configured a cluster of two nodes -- do these token values make
>> sense?
>> The reason I'm asking that so far I don't see load balancing to be
>> happening, judging from performance.
>>
>> Address         Status State   Load            Owns    Token
>>
>> 170141183460469231731687303715884105728
>> 130.199.185.194 Up     Normal  153.52 GB       50.00%
>> 85070591730234615865843651857942052864
>> 130.199.185.193 Up     Normal  199.82 GB       50.00%
>> 170141183460469231731687303715884105728
>>
>>
>> --
>> View this message in context:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at
>> Nabble.com.
>
>

The first token does not "really" have to be zero. They just have been
spread evenly across the token space.

Re: Node added, no performance boost -- are the tokens correct?

Posted by buddhasystem <po...@bnl.gov>.
On two different clusters, if I set the token to zero, on a node, its
ownership drops to zero after migration.



After I added the third one and moved tokens, I now have this:

33.33%  56713727820156410577229101238628035242
33.33%  113427455640312821154458202477256070484
33.33%  170141183460469231731687303715884105727

No zeroes.





Eric Gilmore-3 wrote:
> 
> A script that I have says the following:
> 
> $ python ctokens.py
> How many nodes are in your cluster? 2
> node 0: 0
> node 1: 85070591730234615865843651857942052864
> 
> The first token should be zero, for the reasons discussed here:
> http://www.datastax.com/dev/tutorials/getting_started_0_7/configuring#initial-token-values
> 
> More details are available in
> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
> 
> The DS docs have some weak areas, but these two pages have been pretty
> well
> vetted over the past months :)
> 
> 
> 
> On Thu, Mar 31, 2011 at 3:06 PM, buddhasystem &lt;potekhin@bnl.gov&gt;
> wrote:
> 
>> I just configured a cluster of two nodes -- do these token values make
>> sense?
>> The reason I'm asking that so far I don't see load balancing to be
>> happening, judging from performance.
>>
>> Address         Status State   Load            Owns    Token
>>
>> 170141183460469231731687303715884105728
>> 130.199.185.194 Up     Normal  153.52 GB       50.00%
>> 85070591730234615865843651857942052864
>> 130.199.185.193 Up     Normal  199.82 GB       50.00%
>> 170141183460469231731687303715884105728
>>
>>
>> --
>> View this message in context:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
>> Sent from the cassandra-user@incubator.apache.org mailing list archive at
>> Nabble.com.
>>
> 


--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6231845.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Node added, no performance boost -- are the tokens correct?

Posted by Eric Gilmore <er...@datastax.com>.
A script that I have says the following:

$ python ctokens.py
How many nodes are in your cluster? 2
node 0: 0
node 1: 85070591730234615865843651857942052864

The first token should be zero, for the reasons discussed here:
http://www.datastax.com/dev/tutorials/getting_started_0_7/configuring#initial-token-values

More details are available in
http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity

The DS docs have some weak areas, but these two pages have been pretty well
vetted over the past months :)



On Thu, Mar 31, 2011 at 3:06 PM, buddhasystem <po...@bnl.gov> wrote:

> I just configured a cluster of two nodes -- do these token values make
> sense?
> The reason I'm asking that so far I don't see load balancing to be
> happening, judging from performance.
>
> Address         Status State   Load            Owns    Token
>
> 170141183460469231731687303715884105728
> 130.199.185.194 Up     Normal  153.52 GB       50.00%
> 85070591730234615865843651857942052864
> 130.199.185.193 Up     Normal  199.82 GB       50.00%
> 170141183460469231731687303715884105728
>
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Node-added-no-performance-boost-are-the-tokens-correct-tp6228872p6228872.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>