Change number of vnodes on an existing cluster

[Phil Luckhurst <ph...@powerassure.com>]

We have a 3 node 2.0.7 cluster with RF=3. At the moment these are configured
to have the default 256 vnodes we'd like to try reducing that to see what
effect it has on some of our CQL query times.

It seems from  CASSANDRA-7057
<https://issues.apache.org/jira/browse/CASSANDRA-7057>   that there is no
automatic method for this but is it possible to do this manually and if so
what are the steps required? Do we need to add new nodes with the number of
vnodes we require and then decommission the existing ones or is it possible
to do it just with our existing 3 nodes?

Thanks
Phil



--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Change-number-of-vnodes-on-an-existing-cluster-tp7594646.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Change number of vnodes on an existing cluster

[Phil Luckhurst <ph...@powerassure.com>]

Thank you Rob, that's all really useful information. As our production
cluster is going to grow over time it looks like we may need to stick with
vnodes (but maybe not 256) and as you say hope the work to improve their
inefficiencies progresses quite quickly.

> In real reality, vnodes were almost certainly set to default in 2.0 so
> that they could be hardened by both early adopters (cassandra experts) and
> noobs (cassandra anti-experts) encountering problems with this default. As
> Eric Evans mentioned at the time in a related post, this is to some extent
> the reality of open source software development. Where he and I appear to
> disagree is on whether it is reasonable to set new features as defaults
> and thereby use noobs as part of your QA process for same.

I totally agree with you here. As a 'noob' reading through the Cassandra
documentation before installing for the first time everything pushes you
toward using vnodes. It's not until you start reading in forums such as
this, blog posts, and JIRA issues that it becomes clear that they do come
with caveats.

I'll look forward to the 'always-specify-initial_token best practice' post.

Thanks,
Phil



--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Change-number-of-vnodes-on-an-existing-cluster-tp7594646p7594661.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Change number of vnodes on an existing cluster

[Robert Coli <rc...@eventbrite.com>]

On Thu, May 22, 2014 at 2:16 PM, Phil Luckhurst <
phil.luckhurst@powerassure.com> wrote:

> Thanks Rob, I didn't realize that you could use the initial_token when
> using
> vnodes.
>

I am planning to write a blog post on the always-specify-initial_token best
practice. It protects users in a variety of cases, at the cost of
comparatively little management overhead.

For reference, the format of this when using vnodes is a comma delimited
list.


> I see what you mean now that with RF=N having multiple vnodes is not
> actually achieving anything unless we add further nodes, we hadn't really
> considered that when we initially installed with the default yaml file.
>

Cassandra "design" and its implicit perspective on defaults often seem to
not be able to make up its mind : are they for the developer running a
single node on his laptop, or the operator deploying a reasonably sized
production cluster? The answer seems to be "it depends on who is asking for
what and who is merging the patch."

It is my view that vnodes as a default is unreasonable because most
clusters are unreasonably small and will stay that way, and vnodes only
lose in that case. Vnodes only help you if RF is some multiple of N, but
100% of vnode users pay fixed overhead costs. As it is clear that vnodes
lose in many common cases, I strongly doubt there was any formal-esque
argument made for it as a reasonable default for all users.

The paper trail suggests that virtually no public justification was made
for making it the default, perhaps there was some discussion not
immediately visible in google :

https://issues.apache.org/jira/browse/CASSANDRA-5869 - "Make vnodes default
in 2.0"
"
Description

_Click To Add Description_
"

http://grokbase.com/t/cassandra/dev/1383gva1fm/time-for-a-release-candidate
"
We hadn't actually set vnodes to be on by default, which we should do
after having 1.2 default to old single token per node. Fixed in
CASSANDRA-5869 so let's roll an rc2.
"

The stated rationale here appears to be that a major version has passed,
and therefore it is appropriate to change the default. This decision does
not seem related in any way to the volume of still quite serious problems
vnodes had as of the 2.0 rc1 era.

In real reality, vnodes were almost certainly set to default in 2.0 so that
they could be hardened by both early adopters (cassandra experts) and noobs
(cassandra anti-experts) encountering problems with this default. As Eric
Evans mentioned at the time in a related post, this is to some extent the
reality of open source software development. Where he and I appear to
disagree is on whether it is reasonable to set new features as defaults and
thereby use noobs as part of your QA process for same.

If a Cassandra Operator notices something new and shiny has been made
default or is being pushed heavily in a brand new version (vnodes, level
compaction at 5mb, serial repair, etc., etc.) they should do their own
evaluation and determine whether they want to participate in the QA process
for this new feature, in production. My view is that it would be a lot more
reasonable to continue to promote such new things heavily but with VERY
LARGE REPEATED CAVEATS so that only the former class of willful early
adopter experts will catch these bullets.

tl;dr - Prematurely and optimistically making immature features like vnodes
the default seems to mostly ensure that noobs such as yourself get a very
bad initial experience with Cassandra.

For a small cluster, e.g. 9 nodes with RF=3 would you actually recommend
> using vnodes at all and if so how many?


The key consideration when deciding vnodes or not is not really how many
nodes you initially deploy, it's how many nodes you are likely to ever
have. If you have a reasonable expectation of ever having over RF*3 nodes
where RF>=3, you probably want to use vnodes, so that you have a faster and
less degrading rebuild experience when 1/9 nodes fails. If you don't, it's
probably not worth the trouble.

The caveat here is that it is non-trivial to get from non-vnodes to vnodes
if RF<N, which is why I phrase the choice whether to use vnodes or not as
relating to the expectation of actually winning by using them.

There is one final caveat, which is that significant work is being done to
reduce the fixed costs and inefficiencies associated with vnodes. Each of
these improvements lowers the threshold at which vnodes will become a
reasonable default.

=Rob

Re: Change number of vnodes on an existing cluster

[Phil Luckhurst <ph...@powerassure.com>]

Thanks Rob, I didn't realize that you could use the initial_token when using
vnodes.

I see what you mean now that with RF=N having multiple vnodes is not
actually achieving anything unless we add further nodes, we hadn't really
considered that when we initially installed with the default yaml file.

For a small cluster, e.g. 9 nodes with RF=3 would you actually recommend
using vnodes at all and if so how many?

Phil



Robert Coli-3 wrote
> On Thu, May 22, 2014 at 4:31 AM, Phil Luckhurst <

> phil.luckhurst@

>> wrote:
> 
>> We have a 3 node 2.0.7 cluster with RF=3. At the moment these are
>> configured
>> to have the default 256 vnodes we'd like to try reducing that to see what
>> effect it has on some of our CQL query times.
>>
> 
> Because you have RF=N, all nodes have all data. This means that it is
> actually irrelevant how many vnodes (or nodes) you have, you just get lose
> from using them at all.
> 
> However to reduce the number of vnodes to a number that is ok in your case
> but also might be reasonable when you have RF != N, you can just :
> 
> 1) get a list of tokens per node via a one-liner like this :
> 
> nodetool info -T | grep Token | awk '{print $3}' | paste -s -d,
> 
> 2) modify this list by removing however many tokens you want to get to the
> new number of vnodes
> 
> 3) insert this list into the initial_token line of cassandra.yaml on each
> node [1]
> 
> 4) rolling re-start nodes with auto_bootstrap:false [2]
> 
> My *belief* is that you do not need a step 3.5 "nuke the system keyspace
> and reload schema, potentially with the entire cluster down", but it's
> possible that other nodes may remember your old vnodes unless you do. Test
> in a non-production environment, obviously.
> 
> If the above is too complicated and you have the spare hosts, adding 3 new
> nodes and then decommissioning the old ones is a safe and simple way to
> achieve the same goal.
> 
> =Rob
> [1] Note that I recommend this as a best practice for the use of vnodes,
> always populate initial_token.
> [2]
> https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/





--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Change-number-of-vnodes-on-an-existing-cluster-tp7594646p7594657.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Change number of vnodes on an existing cluster

[Robert Coli <rc...@eventbrite.com>]

On Thu, May 22, 2014 at 4:31 AM, Phil Luckhurst <
phil.luckhurst@powerassure.com> wrote:

> We have a 3 node 2.0.7 cluster with RF=3. At the moment these are
> configured
> to have the default 256 vnodes we'd like to try reducing that to see what
> effect it has on some of our CQL query times.
>

Because you have RF=N, all nodes have all data. This means that it is
actually irrelevant how many vnodes (or nodes) you have, you just get lose
from using them at all.

However to reduce the number of vnodes to a number that is ok in your case
but also might be reasonable when you have RF != N, you can just :

1) get a list of tokens per node via a one-liner like this :

nodetool info -T | grep Token | awk '{print $3}' | paste -s -d,

2) modify this list by removing however many tokens you want to get to the
new number of vnodes

3) insert this list into the initial_token line of cassandra.yaml on each
node [1]

4) rolling re-start nodes with auto_bootstrap:false [2]

My *belief* is that you do not need a step 3.5 "nuke the system keyspace
and reload schema, potentially with the entire cluster down", but it's
possible that other nodes may remember your old vnodes unless you do. Test
in a non-production environment, obviously.

If the above is too complicated and you have the spare hosts, adding 3 new
nodes and then decommissioning the old ones is a safe and simple way to
achieve the same goal.

=Rob
[1] Note that I recommend this as a best practice for the use of vnodes,
always populate initial_token.
[2]
https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/