You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Max C." <mc...@core43.com> on 2018/08/29 18:18:16 UTC
Recommended num_tokens setting for small cluster
Hello Everyone,
Datastax recommends num_tokens = 8 as a sensible default, rather than num_tokens = 256:
https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/config/configVnodes.html <https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/config/configVnodes.html>
… but then I see stories like this (unbalanced cluster when using num_tokens=12), which are very concerning:
https://danielparker.me/cassandra/vnodes/tokens/increasing-vnodes-cassandra/ <https://danielparker.me/cassandra/vnodes/tokens/increasing-vnodes-cassandra/>
We’re currently running 3.0.x, 3 nodes, RF=3, num_tokens=256, spinning disks, soon to be 2 DCs. My guess is that our cluster will probably not grow beyond 10 nodes (10 TB?)
I’d like to minimize the chance of hitting a roadblock down the road due to having num_tokens set inappropriately. We can change this right now pretty easily (our dataset is small but growing). Should we switch from 256 to 8? 32?
Has anyone had num_tokens = 8 (or similarly small number) and experienced growing pains? What do you think the recommended setting should be?
Thanks for the advice. :-)
- Max
Re: Recommended num_tokens setting for small cluster
Posted by Mohamadreza Rostami <mo...@gmail.com>.
Hi
for best performance you can start form node that has lower disk capacity and set it to 256 and for other nods use proportion to this node to find num of tokens.
> On Shahrivar 9, 1397 AP, at 10:40, Max C. <mc...@core43.com> wrote:
>
> Jeff/Kurt/Alex — thanks so much for your feedback on this issue, and thanks for all of the help you guys have lent to people on this list over the years. :-)
>
> - Max
>
>> On Aug 29, 2018, at 11:38 pm, Oleksandr Shulgin <oleksandr.shulgin@zalando.de <ma...@zalando.de>> wrote:
>>
>> On Thu, Aug 30, 2018 at 12:05 AM kurt greaves <kurt@instaclustr.com <ma...@instaclustr.com>> wrote:
>> For 10 nodes you probably want to use between 32 and 64. Make sure you use the token allocation algorithm by specifying allocate_tokens_for_keyspace
>>
>> We are using 16 tokens with 30 nodes on Cassandra 3.0. And yes, we have used allocate_tokens_for_keyspace option to achieve better load distribution than with the random allocation (which is the default). Currently we see the disk usage between 1.5 and 1.7TB, which is acceptable variance for us.
>>
>> If you're using DSE, you're more lucky because it's easier to bootstrap new DC with the smart token allocation algorithm. Simply because the parameter you need to specify does not depend on any keyspaces being replicated to the new nodes, you just specify the target replication factor to optimize for.
>>
>> Cheers,
>> --
>> Alex
>>
>
Re: Recommended num_tokens setting for small cluster
Posted by "Max C." <mc...@core43.com>.
Jeff/Kurt/Alex — thanks so much for your feedback on this issue, and thanks for all of the help you guys have lent to people on this list over the years. :-)
- Max
> On Aug 29, 2018, at 11:38 pm, Oleksandr Shulgin <oleksandr.shulgin@zalando.de <ma...@zalando.de>> wrote:
>
> On Thu, Aug 30, 2018 at 12:05 AM kurt greaves <kurt@instaclustr.com <ma...@instaclustr.com>> wrote:
> For 10 nodes you probably want to use between 32 and 64. Make sure you use the token allocation algorithm by specifying allocate_tokens_for_keyspace
>
> We are using 16 tokens with 30 nodes on Cassandra 3.0. And yes, we have used allocate_tokens_for_keyspace option to achieve better load distribution than with the random allocation (which is the default). Currently we see the disk usage between 1.5 and 1.7TB, which is acceptable variance for us.
>
> If you're using DSE, you're more lucky because it's easier to bootstrap new DC with the smart token allocation algorithm. Simply because the parameter you need to specify does not depend on any keyspaces being replicated to the new nodes, you just specify the target replication factor to optimize for.
>
> Cheers,
> --
> Alex
>
Re: Recommended num_tokens setting for small cluster
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Thu, Aug 30, 2018 at 12:05 AM kurt greaves <ku...@instaclustr.com> wrote:
> For 10 nodes you probably want to use between 32 and 64. Make sure you use
> the token allocation algorithm by specifying allocate_tokens_for_keyspace
>
We are using 16 tokens with 30 nodes on Cassandra 3.0. And yes, we have
used allocate_tokens_for_keyspace option to achieve better load
distribution than with the random allocation (which is the default).
Currently we see the disk usage between 1.5 and 1.7TB, which is acceptable
variance for us.
If you're using DSE, you're more lucky because it's easier to bootstrap new
DC with the smart token allocation algorithm. Simply because the parameter
you need to specify does not depend on any keyspaces being replicated to
the new nodes, you just specify the target replication factor to optimize
for.
Cheers,
--
Alex
Re: Recommended num_tokens setting for small cluster
Posted by kurt greaves <ku...@instaclustr.com>.
For 10 nodes you probably want to use between 32 and 64. Make sure you use
the token allocation algorithm by specifying allocate_tokens_for_keyspace
On Thu., 30 Aug. 2018, 04:40 Jeff Jirsa, <jj...@gmail.com> wrote:
> 3.0 has a (optional?) feature to guarantee better distribution, and the
> blog focuses on 2.2.
>
> Using fewer will minimize your risk of unavailability if any two hosts
> fail.
>
> --
> Jeff Jirsa
>
>
> On Aug 29, 2018, at 11:18 AM, Max C. <mc...@core43.com> wrote:
>
> Hello Everyone,
>
> Datastax recommends num_tokens = 8 as a sensible default, rather than
> num_tokens = 256:
>
>
> https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/config/configVnodes.html
>
> … but then I see stories like this (unbalanced cluster when using
> num_tokens=12), which are very concerning:
>
>
> https://danielparker.me/cassandra/vnodes/tokens/increasing-vnodes-cassandra/
>
> We’re currently running 3.0.x, 3 nodes, RF=3, num_tokens=256, spinning
> disks, soon to be 2 DCs. My guess is that our cluster will probably not
> grow beyond 10 nodes (10 TB?)
>
> I’d like to minimize the chance of hitting a roadblock down the road due
> to having num_tokens set inappropriately. We can change this right now
> pretty easily (our dataset is small but growing). Should we switch from
> 256 to 8? 32?
>
> Has anyone had num_tokens = 8 (or similarly small number) and experienced
> growing pains? What do you think the recommended setting should be?
>
> Thanks for the advice. :-)
>
> - Max
>
>
Re: Recommended num_tokens setting for small cluster
Posted by Jeff Jirsa <jj...@gmail.com>.
3.0 has a (optional?) feature to guarantee better distribution, and the blog focuses on 2.2.
Using fewer will minimize your risk of unavailability if any two hosts fail.
--
Jeff Jirsa
> On Aug 29, 2018, at 11:18 AM, Max C. <mc...@core43.com> wrote:
>
> Hello Everyone,
>
> Datastax recommends num_tokens = 8 as a sensible default, rather than num_tokens = 256:
>
> https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/config/configVnodes.html
>
> … but then I see stories like this (unbalanced cluster when using num_tokens=12), which are very concerning:
>
> https://danielparker.me/cassandra/vnodes/tokens/increasing-vnodes-cassandra/
>
> We’re currently running 3.0.x, 3 nodes, RF=3, num_tokens=256, spinning disks, soon to be 2 DCs. My guess is that our cluster will probably not grow beyond 10 nodes (10 TB?)
>
> I’d like to minimize the chance of hitting a roadblock down the road due to having num_tokens set inappropriately. We can change this right now pretty easily (our dataset is small but growing). Should we switch from 256 to 8? 32?
>
> Has anyone had num_tokens = 8 (or similarly small number) and experienced growing pains? What do you think the recommended setting should be?
>
> Thanks for the advice. :-)
>
> - Max
>