You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Timon Wong <ti...@gmail.com> on 2016/08/06 09:03:36 UTC
Doing rolling upgrades on two DCs
Hello, newbie here.
Currently we are using Cassandra v2.2.1 (two DCs, naming them DC1, DC2,
respectively), and recently we have two issues:
1. Cannot adding new node due to CASSANDRA-10961
2. Cannot repair node, due to CASSANDRA-10501
We learned that we can do rolling upgrade, since we are only using one DC1
for now (DC2 is just for replica), we have following steps in mind:
1. Backup;
2. Doing upgrade on DC2. (Follows this guide:
https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandraDetails.html
);
3. Do some checks (client compatibility, etc);
4. Adding nodes to DC2 (using new Cassandra);
5. Traffic our loads to DC2;
6. Upgrading DC1.
Is that possible, or do we miss something?
Thanks in advance!
Timon
Re: Doing rolling upgrades on two DCs
Posted by Erik Forkalsud <ef...@cj.com>.
The low-risk approach would be to change one thing at a time, but I
don't think there's
a technical reason you couldn't change the commit log and data directory
locations
while updating to a new version.
- EF
On 08/08/2016 07:58 AM, Timon Wong wrote:
> Thank you!
> We would do a patch level upgrade, from 2.2.1 to 2.2.7 (the latest
> stable). Besides, we want to adjust both `commitlog_directory` and
> `data_file_directories` settings (separate them into two better IOPS
> volumes), and we assumed they should be done before or after upgrade?
>
> On Sun, Aug 7, 2016 at 5:38 AM, Forkalsrud, Erik <eforkalsrud@cj.com
> <ma...@cj.com>> wrote:
>
>
> You didn't mention which version you're upgrading to, but
> generally you shouldn't add or
> remove nodes while the nodes your cluster is not all on on the
> same version. So if you're
> going to add nodes, you should do that before you upgrade the
> first node, or after you have
> upgraded all of them.
>
> Shifting traffic away from DC1 may not be necessary, but shouldn't
> hurt.
>
>
> - EF
>
> _____________________________________
> From: Timon Wong [timon86.wang@gmail.com
> <ma...@gmail.com>]
> Sent: Saturday, August 6, 2016 02:03
> To: user@cassandra.apache.org <ma...@cassandra.apache.org>
> Subject: Doing rolling upgrades on two DCs
>
> Hello, newbie here.
>
> Currently we are using Cassandra v2.2.1 (two DCs, naming them DC1,
> DC2, respectively), and recently we have two issues:
>
> 1. Cannot adding new node due to CASSANDRA-10961
> 2. Cannot repair node, due to CASSANDRA-10501
>
> We learned that we can do rolling upgrade, since we are only using
> one DC1 for now (DC2 is just for replica), we have following steps
> in mind:
>
> 1. Backup;
> 2. Doing upgrade on DC2. (Follows this guide:
> https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandraDetails.html
> <https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandraDetails.html>);
> 3. Do some checks (client compatibility, etc);
> 4. Adding nodes to DC2 (using new Cassandra);
> 5. Traffic our loads to DC2;
> 6. Upgrading DC1.
>
> Is that possible, or do we miss something?
>
> Thanks in advance!
>
> Timon
>
>
Re: Doing rolling upgrades on two DCs
Posted by Timon Wong <ti...@gmail.com>.
Thank you!
We would do a patch level upgrade, from 2.2.1 to 2.2.7 (the latest stable).
Besides, we want to adjust both `commitlog_directory` and
`data_file_directories` settings (separate them into two better IOPS
volumes), and we assumed they should be done before or after upgrade?
On Sun, Aug 7, 2016 at 5:38 AM, Forkalsrud, Erik <ef...@cj.com> wrote:
>
> You didn't mention which version you're upgrading to, but generally you
> shouldn't add or
> remove nodes while the nodes your cluster is not all on on the same
> version. So if you're
> going to add nodes, you should do that before you upgrade the first node,
> or after you have
> upgraded all of them.
>
> Shifting traffic away from DC1 may not be necessary, but shouldn't hurt.
>
>
> - EF
>
> _____________________________________
> From: Timon Wong [timon86.wang@gmail.com]
> Sent: Saturday, August 6, 2016 02:03
> To: user@cassandra.apache.org
> Subject: Doing rolling upgrades on two DCs
>
> Hello, newbie here.
>
> Currently we are using Cassandra v2.2.1 (two DCs, naming them DC1, DC2,
> respectively), and recently we have two issues:
>
> 1. Cannot adding new node due to CASSANDRA-10961
> 2. Cannot repair node, due to CASSANDRA-10501
>
> We learned that we can do rolling upgrade, since we are only using one DC1
> for now (DC2 is just for replica), we have following steps in mind:
>
> 1. Backup;
> 2. Doing upgrade on DC2. (Follows this guide:
> https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/
> upgrdCassandraDetails.html);
> 3. Do some checks (client compatibility, etc);
> 4. Adding nodes to DC2 (using new Cassandra);
> 5. Traffic our loads to DC2;
> 6. Upgrading DC1.
>
> Is that possible, or do we miss something?
>
> Thanks in advance!
>
> Timon
>
RE: Doing rolling upgrades on two DCs
Posted by "Forkalsrud, Erik" <ef...@cj.com>.
You didn't mention which version you're upgrading to, but generally you shouldn't add or
remove nodes while the nodes your cluster is not all on on the same version. So if you're
going to add nodes, you should do that before you upgrade the first node, or after you have
upgraded all of them.
Shifting traffic away from DC1 may not be necessary, but shouldn't hurt.
- EF
_____________________________________
From: Timon Wong [timon86.wang@gmail.com]
Sent: Saturday, August 6, 2016 02:03
To: user@cassandra.apache.org
Subject: Doing rolling upgrades on two DCs
Hello, newbie here.
Currently we are using Cassandra v2.2.1 (two DCs, naming them DC1, DC2, respectively), and recently we have two issues:
1. Cannot adding new node due to CASSANDRA-10961
2. Cannot repair node, due to CASSANDRA-10501
We learned that we can do rolling upgrade, since we are only using one DC1 for now (DC2 is just for replica), we have following steps in mind:
1. Backup;
2. Doing upgrade on DC2. (Follows this guide: https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandraDetails.html);
3. Do some checks (client compatibility, etc);
4. Adding nodes to DC2 (using new Cassandra);
5. Traffic our loads to DC2;
6. Upgrading DC1.
Is that possible, or do we miss something?
Thanks in advance!
Timon