You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Lapo Luchini <la...@lapo.it> on 2021/03/20 15:08:46 UTC
Changing num_tokens and migrating to 4.0
I have a 6 nodes production cluster running 3.11.9 with the default
num_tokens=256… which is fine but I later discovered is a bit of a
hassle to do repairs and is probably better to lower that to 16.
I'm adding two new nodes with much higher space storage and I was
wondering which migration strategy is better.
If I got it correct I was thinking about this:
1. add the 2 new nodes as a new "temporary DC", with num_token=16 RF=3
2. repair it all, then test it a bit
3. switch production applications to "DC-temp"
4. drop the old 6-node DC
5. re-create it from scratch with num_token=16 RF=3
6. switch production applications to "main DC" again
7. drop "DC-temp", eventually integrate nodes into "main DC"
I'd also like to migrate from 3.11.9 to 4.0-beta2 (I'm running on
FreeBSD so those are the options), does it make sense to do it during
the mentioned "num_tokens migration" (at step 1, or 5) or does it make
more sense to do it at step 8, as a in-place rolling upgrade of each of
the 6 (or 8) nodes?
Did I get it correctly?
Can it be done "better"?
Thanks in advance for any suggestion or correction!
--
Lapo Luchini
lapo@lapo.it
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
RE: Changing num_tokens and migrating to 4.0
Posted by "Durity, Sean R" <SE...@homedepot.com>.
I have a cluster (almost 200 nodes) with a variety of disk sizes and use different numbers of tokens so that the machines can use the disk they have. It is a very handy feature! While I agree that a node with larger disk may handle more requests, that may not be enough to impact CPU or memory. I rarely see Cassandra CPU-bound for my use cases. These are primarily write use cases with a low number of clients with far fewer reads. There is just a lot of data to keep.
Sean Durity
From: Alex Ott <al...@gmail.com>
Sent: Saturday, March 20, 2021 1:01 PM
To: user <us...@cassandra.apache.org>
Subject: [EXTERNAL] Re: Changing num_tokens and migrating to 4.0
if the nodes are almost the same, except the disk space, then giving them more may make siltation worse - they will get more requests than other nodes, and won't have resources to process them.
In Cassandra the disk size isn't the main "success" factor - it's a memory, CPU, disk type (SSD), etc.
On Sat, Mar 20, 2021 at 5:26 PM Lapo Luchini <la...@lapo.it>> wrote:
Hi, thanks for suggestions!
I'll definitely migrate to 4.0 after all this is done, then.
Old prod DC I fear can't suffer losing a node right now (a few nodes
have the disk 70% full), but I can maybe find a third node for the new
DC right away.
BTW the new nodes have got 3× the disk space, but are not so much
different regarding CPU and RAM: does it make any sense giving them a
bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old
DC hosts or "asymmetrical" clusters lead to problems?
No real need to do that anyways, moving from 6 nodes to (eventually) 8
should be enough lessen the load on the disks, and before more space is
needed I will probably have more nodes.
Lapo
On 2021-03-20 16:23, Alex Ott wrote:
> I personally maybe would go following way (need to calculate how many
> joins/decommissions will be at the end):
>
> * Decommission one node from prod DC
> * Form new DC from two new machines and decommissioned one.
> * Rebuild DC from existing one, make sure that repair finished, etc.
> * Switch traffic
> * Remove old DC
> * Add nodes from old DC one by one into new DC
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<ma...@cassandra.apache.org>
For additional commands, e-mail: user-help@cassandra.apache.org<ma...@cassandra.apache.org>
--
With best wishes, Alex Ott
http://alexott.net/ [alexott.net]<https://urldefense.com/v3/__http:/alexott.net/__;!!M-nmYVHPHQ!Z88elRlG30NKZR9PseTGVv5t4zPzVdEmbaqJdUVKjfSbuqPg3fU1FnbH1o6KTwF8XWTzS8A$>
Twitter: alexott_en (English), alexott (Russian)
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Changing num_tokens and migrating to 4.0
Posted by Alex Ott <al...@gmail.com>.
if the nodes are almost the same, except the disk space, then giving them
more may make siltation worse - they will get more requests than other
nodes, and won't have resources to process them.
In Cassandra the disk size isn't the main "success" factor - it's a memory,
CPU, disk type (SSD), etc.
On Sat, Mar 20, 2021 at 5:26 PM Lapo Luchini <la...@lapo.it> wrote:
> Hi, thanks for suggestions!
> I'll definitely migrate to 4.0 after all this is done, then.
>
> Old prod DC I fear can't suffer losing a node right now (a few nodes
> have the disk 70% full), but I can maybe find a third node for the new
> DC right away.
>
> BTW the new nodes have got 3× the disk space, but are not so much
> different regarding CPU and RAM: does it make any sense giving them a
> bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old
> DC hosts or "asymmetrical" clusters lead to problems?
>
> No real need to do that anyways, moving from 6 nodes to (eventually) 8
> should be enough lessen the load on the disks, and before more space is
> needed I will probably have more nodes.
>
> Lapo
>
> On 2021-03-20 16:23, Alex Ott wrote:
> > I personally maybe would go following way (need to calculate how many
> > joins/decommissions will be at the end):
> >
> > * Decommission one node from prod DC
> > * Form new DC from two new machines and decommissioned one.
> > * Rebuild DC from existing one, make sure that repair finished, etc.
> > * Switch traffic
> > * Remove old DC
> > * Add nodes from old DC one by one into new DC
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
--
With best wishes, Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Re: Changing num_tokens and migrating to 4.0
Posted by Lapo Luchini <la...@lapo.it>.
Hi, thanks for suggestions!
I'll definitely migrate to 4.0 after all this is done, then.
Old prod DC I fear can't suffer losing a node right now (a few nodes
have the disk 70% full), but I can maybe find a third node for the new
DC right away.
BTW the new nodes have got 3× the disk space, but are not so much
different regarding CPU and RAM: does it make any sense giving them a
bit more num_tokens (maybe 20-30 instead of 16) than the rest of the old
DC hosts or "asymmetrical" clusters lead to problems?
No real need to do that anyways, moving from 6 nodes to (eventually) 8
should be enough lessen the load on the disks, and before more space is
needed I will probably have more nodes.
Lapo
On 2021-03-20 16:23, Alex Ott wrote:
> I personally maybe would go following way (need to calculate how many
> joins/decommissions will be at the end):
>
> * Decommission one node from prod DC
> * Form new DC from two new machines and decommissioned one.
> * Rebuild DC from existing one, make sure that repair finished, etc.
> * Switch traffic
> * Remove old DC
> * Add nodes from old DC one by one into new DC
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Changing num_tokens and migrating to 4.0
Posted by Alex Ott <al...@gmail.com>.
The are several things to consider here:
- You can't have DC of two nodes with RF=3...
- Are you sure that new DC will handle all production traffic?
- if new nodes much more powerful than other (memory/CPU/disk type) that
could also cause unpredictable spikes when request will hit the "smaller"
node.
I personally maybe would go following way (need to calculate how many
joins/decommissions will be at the end):
- Decommission one node from prod DC
- Form new DC from two new machines and decommissioned one.
- Rebuild DC from existing one, make sure that repair finished, etc.
- Switch traffic
- Remove old DC
- Add nodes from old DC one by one into new DC
Upgrade to Cassandra 4.0 should be done either prior to that, or after -
you shouldn't do it when doing bootstrapping/decomissioning...
On Sat, Mar 20, 2021 at 4:09 PM Lapo Luchini <la...@lapo.it> wrote:
> I have a 6 nodes production cluster running 3.11.9 with the default
> num_tokens=256… which is fine but I later discovered is a bit of a
> hassle to do repairs and is probably better to lower that to 16.
>
> I'm adding two new nodes with much higher space storage and I was
> wondering which migration strategy is better.
>
> If I got it correct I was thinking about this:
> 1. add the 2 new nodes as a new "temporary DC", with num_token=16 RF=3
> 2. repair it all, then test it a bit
> 3. switch production applications to "DC-temp"
> 4. drop the old 6-node DC
> 5. re-create it from scratch with num_token=16 RF=3
> 6. switch production applications to "main DC" again
> 7. drop "DC-temp", eventually integrate nodes into "main DC"
>
> I'd also like to migrate from 3.11.9 to 4.0-beta2 (I'm running on
> FreeBSD so those are the options), does it make sense to do it during
> the mentioned "num_tokens migration" (at step 1, or 5) or does it make
> more sense to do it at step 8, as a in-place rolling upgrade of each of
> the 6 (or 8) nodes?
>
> Did I get it correctly?
> Can it be done "better"?
>
> Thanks in advance for any suggestion or correction!
>
> --
> Lapo Luchini
> lapo@lapo.it
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
--
With best wishes, Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Re: Changing num_tokens and migrating to 4.0
Posted by Lapo Luchini <la...@lapo.it>.
On 2021-03-22 01:27, Kane Wilson wrote:
> You should be able to get repairs working fine if you use a tool such as
> cassandra-reaper to manage it for you for such a small cluster. I would
> look into that before doing major cluster topology changes, as these can
> be complex and risky.
I was looking into a migration as I'm already using Cassanrea Reaper and
on the biggest keyspace is often taking more than 7 days to complete (I
set the segment timeout up to 2h, and most of 138 segments take more
than 1h, sometimes even failing the 2h limits due to, AFAICT, lengthy
compactions).
--
Lapo Luchini
lapo@lapo.it
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Changing num_tokens and migrating to 4.0
Posted by Kane Wilson <k...@raft.so>.
You should be able to get repairs working fine if you use a tool such as
cassandra-reaper to manage it for you for such a small cluster. I would
look into that before doing major cluster topology changes, as these can be
complex and risky. I definitely wouldn't go about it in the way you've
described above, as you're likely to run into load issues.
I also would advise against upgrading to the 4.0 beta releases in
production. Leave running betas in production to the experts. There has not
been enough real world testing to really have any kind of guarantee about
the stability of 4.0 beta2.
If cassandra-reaper doesn't work for you you'll want to migrate in one of
two ways, both of which have their own caveats:
1. A complete DC migration, ideally with the same number of nodes to handle
load. All new nodes would have the lower token count and should use the
token allocation algorithm. Downside here is expensive but this is the
least risk option as it can be managed independently of the existing DC.
2. Decommission and re-add one node at a time, utilising the token
allocation algorithm when you re-add nodes. This is more risky as both
decommission and addition will incur considerable load on the existing
nodes, and you also may not have the spare storage capacity for it to work.
You could solve this however by adding a new node first.
Note I've left a lot of details out here, as these processes can be quite
involved. You should test each method before you do it.
raft.so - Cassandra consulting, support, and managed services
On Sun, Mar 21, 2021 at 2:09 AM Lapo Luchini <la...@lapo.it> wrote:
> I have a 6 nodes production cluster running 3.11.9 with the default
> num_tokens=256… which is fine but I later discovered is a bit of a
> hassle to do repairs and is probably better to lower that to 16.
>
> I'm adding two new nodes with much higher space storage and I was
> wondering which migration strategy is better.
>
> If I got it correct I was thinking about this:
> 1. add the 2 new nodes as a new "temporary DC", with num_token=16 RF=3
> 2. repair it all, then test it a bit
> 3. switch production applications to "DC-temp"
> 4. drop the old 6-node DC
> 5. re-create it from scratch with num_token=16 RF=3
> 6. switch production applications to "main DC" again
> 7. drop "DC-temp", eventually integrate nodes into "main DC"
>
> I'd also like to migrate from 3.11.9 to 4.0-beta2 (I'm running on
> FreeBSD so those are the options), does it make sense to do it during
> the mentioned "num_tokens migration" (at step 1, or 5) or does it make
> more sense to do it at step 8, as a in-place rolling upgrade of each of
> the 6 (or 8) nodes?
>
> Did I get it correctly?
> Can it be done "better"?
>
> Thanks in advance for any suggestion or correction!
>
> --
> Lapo Luchini
> lapo@lapo.it
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>