Re: Question about num_tokens

[Jai Bheemsen Rao Dhanwada <ja...@gmail.com>]

Thanks for the response and details. I am just curious about the below
statement mentioned in the doc. I am pretty confident that my clusters are
going to grow to 100+ nodes (same DC or combining all DCs). I am just
concerned that the doc says it is *not recommended for clusters over 50
nodes*.

16

Best for heavily elastic clusters which expand and shrink regularly, but
may have issues availability with larger clusters. Not recommended for
clusters over 50 nodes.

On Sun, Mar 13, 2022 at 11:34 PM Elliott Sims <el...@backblaze.com> wrote:

> More tokens:  better data distribution, more expensive repairs, higher
> probability of a multi-host outage taking some data offline and affecting
> availability.
>
> I think with >100 nodes the repair times and availability improvements
> make a strong case for 16 tokens even though it means you'll need more
> total raw space.
>
> Switching from 256 to 16 vnodes definitely will make data distribution
> worse.  I'm not sure "hot spot" is the right description so much as a wider
> curve.  I've got one cluster that hasn't been migrated from 256 to 16, and
> it has about a 6% delta between the smallest and largest nodes instead of
> more like 20% on the 16-vnode clusters.  The newer
> allocate_tokens_for_keyspace and (better)
> allocate_tokens_for_replication_factor options help limit the data
> distribution issues, but don't totally eliminate them.
>
> On the other hand, the 16-vnode cluster takes less than half as long to
> complete repairs via Reaper.  It also spends more time on GC, though I
> can't tell whether that's due to vnodes or other differences.
>
> On Sun, Mar 13, 2022 at 5:59 PM Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> wrote:
>
>> Hello Team,
>>
>> I am currently using num_tokens: 256 (default in 3.11.X version) for my
>> clusters and trying to understand the advantages vs disadvantages of
>> changing it to 16 (I believe 16 is the new recommended value).  As per the cassandra
>> documentation
>> <https://cassandra.apache.org/doc/latest/cassandra/getting_started/production.html#tokens> 16
>> is not recommended for the cluster over 50 nodes.
>>
>> Best for heavily elastic clusters which expand and shrink regularly, but
>>> may have issues availability with larger clusters. Not recommended for
>>> clusters over 50 nodes.
>>
>>
>> I have a few questions.
>>
>>
>>    1. What are the general recommendations for a production cluster
>>    which is > 100 nodes and are heavily elastic in terms of adding and
>>    removing nodes.
>>    2. If I am switching from 256 -> 16 tokens, does this cause any
>>    hotspots by having the data concentrated to only a few nodes and not
>>    distributing equally across all the nodes?
>>
>>
> This email, including its contents and any attachment(s), may contain
> confidential and/or proprietary information and is solely for the review
> and use of the intended recipient(s). If you have received this email in
> error, please notify the sender and permanently delete this email, its
> content, and any attachment(s). Any disclosure, copying, or taking of any
> action in reliance on an email received in error is strictly prohibited.
>

Re: Question about num_tokens

[Elliott Sims <el...@backblaze.com>]

I'm not sure I entirely agree with the docs there, as they don't quite
match my experiences, but it's going to depend a lot on your specific needs
and other parts of the configuration.

I think data distribution with low num_tokens is generally considered to be
less of a problem with larger clusters, but I'm not entirely sure that's
true in practice.  You get more of an even-looking distribution of data,
but still a bigger gap between the most and least utilized host and
therefore a need for a larger cluster.

The docs link to a PDF of a study for num_tokens values vs availability
with multiple-node failures.  The gist is that if multiple hosts fail you
may get lucky and those hosts won't contain overlapping token ranges
(therefore avoiding loss of availability).  A lower num_tokens increases
your odds of getting lucky there.  Running NetworkTopology also improves
your odds as well as makes it easier to determine whether a given set of
nodes going offline might affect availability.  It also gives you some
control in terms of reducing the odds of correlated failures on multiple
replicas from things like power or network outages.

I also think current Reaper's token range repairs and intelligence around
consolidating token ranges and safe concurrency limit the downside of
higher num_tokens values.  I've seen pretty good repair performance with
num_tokens 16 and no significant penalty adding in hosts with num_tokens 32
even, though 256 is still significantly slower.  If you're moving to 4.0+
and using incremental repairs, a lower num_tokens value may become
important again.

Anecdotally, running num_tokens 16 and even a mix of num_tokens 16 and 32
has been just fine with multiple clusters over 100 nodes.

On Tue, Aug 16, 2022 at 12:15 PM Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:

> Thanks for the response and details. I am just curious about the below
> statement mentioned in the doc. I am pretty confident that my clusters are
> going to grow to 100+ nodes (same DC or combining all DCs). I am just
> concerned that the doc says it is *not recommended for clusters over 50
> nodes*.
>
> 16
>
> Best for heavily elastic clusters which expand and shrink regularly, but
> may have issues availability with larger clusters. Not recommended for
> clusters over 50 nodes.
>
> On Sun, Mar 13, 2022 at 11:34 PM Elliott Sims <el...@backblaze.com>
> wrote:
>
>> More tokens:  better data distribution, more expensive repairs, higher
>> probability of a multi-host outage taking some data offline and affecting
>> availability.
>>
>> I think with >100 nodes the repair times and availability improvements
>> make a strong case for 16 tokens even though it means you'll need more
>> total raw space.
>>
>> Switching from 256 to 16 vnodes definitely will make data distribution
>> worse.  I'm not sure "hot spot" is the right description so much as a wider
>> curve.  I've got one cluster that hasn't been migrated from 256 to 16, and
>> it has about a 6% delta between the smallest and largest nodes instead of
>> more like 20% on the 16-vnode clusters.  The newer
>> allocate_tokens_for_keyspace and (better)
>> allocate_tokens_for_replication_factor options help limit the data
>> distribution issues, but don't totally eliminate them.
>>
>> On the other hand, the 16-vnode cluster takes less than half as long to
>> complete repairs via Reaper.  It also spends more time on GC, though I
>> can't tell whether that's due to vnodes or other differences.
>>
>> On Sun, Mar 13, 2022 at 5:59 PM Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>>
>>> Hello Team,
>>>
>>> I am currently using num_tokens: 256 (default in 3.11.X version) for my
>>> clusters and trying to understand the advantages vs disadvantages of
>>> changing it to 16 (I believe 16 is the new recommended value).  As per the cassandra
>>> documentation
>>> <https://cassandra.apache.org/doc/latest/cassandra/getting_started/production.html#tokens> 16
>>> is not recommended for the cluster over 50 nodes.
>>>
>>> Best for heavily elastic clusters which expand and shrink regularly, but
>>>> may have issues availability with larger clusters. Not recommended for
>>>> clusters over 50 nodes.
>>>
>>>
>>> I have a few questions.
>>>
>>>
>>>    1. What are the general recommendations for a production cluster
>>>    which is > 100 nodes and are heavily elastic in terms of adding and
>>>    removing nodes.
>>>    2. If I am switching from 256 -> 16 tokens, does this cause any
>>>    hotspots by having the data concentrated to only a few nodes and not
>>>    distributing equally across all the nodes?
>>>
>>>
>> This email, including its contents and any attachment(s), may contain
>> confidential and/or proprietary information and is solely for the review
>> and use of the intended recipient(s). If you have received this email in
>> error, please notify the sender and permanently delete this email, its
>> content, and any attachment(s). Any disclosure, copying, or taking of any
>> action in reliance on an email received in error is strictly prohibited.
>>
>

-- 
This email, including its contents and any attachment(s), may contain 
confidential and/or proprietary information and is solely for the review 
and use of the intended recipient(s). If you have received this email in 
error, please notify the sender and permanently delete this email, its 
content, and any attachment(s).  Any disclosure, copying, or taking of any 
action in reliance on an email received in error is strictly prohibited.