You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Marc Selwan <ma...@datastax.com> on 2018/12/01 16:54:55 UTC
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Hi Shravan,
Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before
doing the major upgrade? It's generally favorable to go to the latest patch
release as often times they include fixes that smooth over the upgrade
process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current
version)
Best,
Marc
On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
> Hello,
>
> I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9.
> I came up with the version based on [1]. I followed upgrade steps as in
> [2]. I was testing the same in the lab and encountered issues (streaming
> just fails and hangs for ever) with bootstrapping a 3.0.9 node on a
> partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The
> production cluster that I am supporting is pretty large and I am
> anticipating to end up in a situation like this (Hope not) and would like
> to be prepared.
>
> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
> cluster?
> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
> 3) Is there an alternative approach to the upgrade large clusters. i.e
> instead of going through nodetool upgradesstables on each node in rolling
> fashion
>
>
> As per [1] the general restriction is to avoid decommissioning or adding
> nodes but in reality there can be failures or maintenance that warrants us
> to do so.
>
>
> Please point me in the right direction.
>
>
> Thanks,
> Shravan
>
>
> [1]
> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>
> [2]
> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
>
> --
Marc Selwan | DataStax | Product Management | (925) 413-7079
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Jeff Jirsa <jj...@gmail.com>.
Schema won’t be transferred cross-majors
--
Jeff Jirsa
> On Dec 4, 2018, at 10:51 PM, Shravan R <sk...@gmail.com> wrote:
>
> Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded cluster (2.1.9 + 3.x) and I was not able to do so. The schema never settled.
>
> How does the below approach sound like?
> Update the software binary on all nodes to use cassandra-3.x upon a restart.
> Restart all nodes in a rolling fashion
> Run nodetool upgradesstables in a rolling fashion
>
> Is there a risk on pending nodetool upgradesstables?
>
>> On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>
>>> On Dec 2, 2018, at 12:40 PM, Shravan R <sk...@gmail.com> wrote:
>>>
>>> Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the version part that you suggested. The reason to go direct to 3.x is to take a bi leap and reduce overall effort to upgrade a large cluster (development included).
>>>
>>> I have these questions from my original post. Appreciate if you could shed some light and point me in the right direction.
>>>
>>> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
>>
>> If any of the replicas have already upgraded, which is almost guaranteed if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade everything else and then deal with it. If a host fails mid upgrade you’ll likely have some period of unavailables while you bounce the replicas to finish, then you can decom
>>
>>
>>
>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>>
>> This may work fine, but test it because I’m not certain. It should be able to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
>>
>>> 3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
>>
>> Bounce them all as quickly as is practical, do the upgradesstables after the bounces complete
>>
>>
>>
>>>
>>>
>>>> On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
>>>> Dmitry is right. Generally speaking always go with the latest bug fix release.
>>>>
>>>>> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <sa...@gmail.com> wrote:
>>>>> See more here
>>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
>>>>>
>>>>>> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com> wrote:
>>>>>> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug corrupting sstables on alter.
>>>>>>
>>>>>>> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com> wrote:
>>>>>>> Hi Shravan,
>>>>>>>
>>>>>>> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before doing the major upgrade? It's generally favorable to go to the latest patch release as often times they include fixes that smooth over the upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current version)
>>>>>>>
>>>>>>> Best,
>>>>>>> Marc
>>>>>>>
>>>>>>>> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. I came up with the version based on [1]. I followed upgrade steps as in [2]. I was testing the same in the lab and encountered issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9 node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The production cluster that I am supporting is pretty large and I am anticipating to end up in a situation like this (Hope not) and would like to be prepared.
>>>>>>>>
>>>>>>>> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
>>>>>>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>>>>>>>> 3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
>>>>>>>>
>>>>>>>>
>>>>>>>> As per [1] the general restriction is to avoid decommissioning or adding nodes but in reality there can be failures or maintenance that warrants us to do so.
>>>>>>>>
>>>>>>>>
>>>>>>>> Please point me in the right direction.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Shravan
>>>>>>>>
>>>>>>>>
>>>>>>>> [1] https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>>>>>>>
>>>>>>>> [2] https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>>>>>>>>
>>>>>>> --
>>>>>>> Marc Selwan | DataStax | Product Management | (925) 413-7079
>>>>>>>
>>>>>>>
>>>> --
>>>> Jon Haddad
>>>> http://www.rustyrazorblade.com
>>>> twitter: rustyrazorblade
RE: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by "Durity, Sean R" <SE...@homedepot.com>.
In my understanding, there is a balance of getting upgradesstables done vs normal activity. I think the cluster can function fine with old and new sstables, but there can be a performance hit to reading the older version (perhaps). Personally, I don’t restart repairs until upgradesstables is completed. So, I push to get upgradesstables completed as soon as possible.
Sean Durity
From: Shravan R <sk...@gmail.com>
Sent: Tuesday, December 04, 2018 3:39 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Thanks Sean. I have automation in place that can put the new binary and restart the node to a newer version as quickly as possible. upgradesstables is I/O intensive and it takes time and is proportional to the data on the node. Given these constraints, is there a risk due to prolonged upgradesstables?
On Tue, Dec 4, 2018 at 12:20 PM Durity, Sean R <SE...@homedepot.com>> wrote:
We have had great success with Cassandra upgrades with applications staying on-line. It is one of the strongest benefits of Cassandra. A couple things I incorporate into upgrades:
- The main task is getting the new binaries loaded, then restarting the node – in a rolling fashion. Get this done as quickly as possible
- Streaming between versions is usually problematic. So, I never do any node additions or decommissions during an upgrade
- With applications running, there is not an acceptable back-out plan (either lose data or take a long outage or both), so we are always going forward. So, lower life cycle testing is important before hitting production
- Upgrading is a more frequent activity, so get the process/automation in place. The upgrade process should not be a reason to delay, especially for minor version upgrades that might be quickly necessary (security issue or bug fix).
Sean Durity
From: Shravan R <sk...@gmail.com>>
Sent: Tuesday, December 04, 2018 12:22 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded cluster (2.1.9 + 3.x) and I was not able to do so. The schema never settled.
How does the below approach sound like?
1. Update the software binary on all nodes to use cassandra-3.x upon a restart.
2. Restart all nodes in a rolling fashion
3. Run nodetool upgradesstables in a rolling fashion
Is there a risk on pending nodetool upgradesstables?
On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa <jj...@gmail.com>> wrote:
On Dec 2, 2018, at 12:40 PM, Shravan R <sk...@gmail.com>> wrote:
Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the version part that you suggested. The reason to go direct to 3.x is to take a bi leap and reduce overall effort to upgrade a large cluster (development included).
I have these questions from my original post. Appreciate if you could shed some light and point me in the right direction.
1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
If any of the replicas have already upgraded, which is almost guaranteed if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade everything else and then deal with it. If a host fails mid upgrade you’ll likely have some period of unavailables while you bounce the replicas to finish, then you can decom
2) How to bootstrap a 3.x node to a partially upgraded cluster?
This may work fine, but test it because I’m not certain. It should be able to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
Bounce them all as quickly as is practical, do the upgradesstables after the bounces complete
On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jo...@jonhaddad.com>> wrote:
Dmitry is right. Generally speaking always go with the latest bug fix release.
On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <sa...@gmail.com>> wrote:
See more here
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_plugins_servlet_mobile-23issue_CASSANDRA-2D13004&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=8csmPWgUEWao6E4wthrG_-BX5a2OQJKXpkKtFLjSPlI&e=>
On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com>> wrote:
Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug corrupting sstables on alter.
On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>> wrote:
Hi Shravan,
Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before doing the major upgrade? It's generally favorable to go to the latest patch release as often times they include fixes that smooth over the upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current version)
Best,
Marc
On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com>> wrote:
Hello,
I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. I came up with the version based on [1]. I followed upgrade steps as in [2]. I was testing the same in the lab and encountered issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9 node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The production cluster that I am supporting is pretty large and I am anticipating to end up in a situation like this (Hope not) and would like to be prepared.
1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
2) How to bootstrap a 3.x node to a partially upgraded cluster?
3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
As per [1] the general restriction is to avoid decommissioning or adding nodes but in reality there can be failures or maintenance that warrants us to do so.
Please point me in the right direction.
Thanks,
Shravan
[1] https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_upgrade_doc_upgrade_datastax-5Fenterprise_upgdDSE50.html-23upgdDSE50-5F-5Fcstar-2Dversion-2Dchange&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=IV10uiJPS2fyCHGYw7yR90A8cxcpjy9Is40YUWgPvF0&e=>
[2] https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/<https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
--
Marc Selwan | DataStax | Product Management | (925) 413-7079
--
Jon Haddad
http://www.rustyrazorblade.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=HPfRIl5knAn9tpUyT06P6YaktRTDwyFSMO7r4GHXBzY&e=>
twitter: rustyrazorblade
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Shravan R <sk...@gmail.com>.
Thanks Sean. I have automation in place that can put the new binary and
restart the node to a newer version as quickly as possible. upgradesstables
is I/O intensive and it takes time and is proportional to the data on the
node. Given these constraints, is there a risk due to prolonged
upgradesstables?
On Tue, Dec 4, 2018 at 12:20 PM Durity, Sean R <SE...@homedepot.com>
wrote:
> We have had great success with Cassandra upgrades with applications
> staying on-line. It is one of the strongest benefits of Cassandra. A couple
> things I incorporate into upgrades:
>
> - The main task is getting the new binaries loaded, then
> restarting the node – in a rolling fashion. Get this done as quickly as
> possible
>
> - Streaming between versions is usually problematic. So, I never
> do any node additions or decommissions during an upgrade
>
> - With applications running, there is not an acceptable back-out
> plan (either lose data or take a long outage or both), so we are always
> going forward. So, lower life cycle testing is important before hitting
> production
>
> - Upgrading is a more frequent activity, so get the
> process/automation in place. The upgrade process should not be a reason to
> delay, especially for minor version upgrades that might be quickly
> necessary (security issue or bug fix).
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* Shravan R <sk...@gmail.com>
> *Sent:* Tuesday, December 04, 2018 12:22 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
>
>
>
> Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded
> cluster (2.1.9 + 3.x) and I was *not* able to do so. The schema never
> settled.
>
>
>
> How does the below approach sound like?
>
> 1. Update the software binary on all nodes to use cassandra-3.x upon a
> restart.
> 2. Restart all nodes in a rolling fashion
> 3. Run nodetool upgradesstables in a rolling fashion
>
>
>
> Is there a risk on pending nodetool upgradesstables?
>
>
>
> On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
>
>
>
> On Dec 2, 2018, at 12:40 PM, Shravan R <sk...@gmail.com> wrote:
>
> Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the
> version part that you suggested. The reason to go direct to 3.x is to take
> a bi leap and reduce overall effort to upgrade a large cluster (development
> included).
>
>
>
> I have these questions from my original post. Appreciate if you could shed
> some light and point me in the right direction.
>
>
>
> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
> cluster?
>
>
>
> If any of the replicas have already upgraded, which is almost guaranteed
> if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade
> everything else and then deal with it. If a host fails mid upgrade you’ll
> likely have some period of unavailables while you bounce the replicas to
> finish, then you can decom
>
>
>
>
>
>
>
> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>
>
>
> This may work fine, but test it because I’m not certain. It should be able
> to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
>
>
>
> 3) Is there an alternative approach to the upgrade large clusters. i.e
> instead of going through nodetool upgradesstables on each node in rolling
> fashion
>
>
>
> Bounce them all as quickly as is practical, do the upgradesstables after
> the bounces complete
>
>
>
>
>
>
>
>
>
>
>
> On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
> Dmitry is right. Generally speaking always go with the latest bug fix
> release.
>
>
>
> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <sa...@gmail.com>
> wrote:
>
> See more here
>
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_plugins_servlet_mobile-23issue_CASSANDRA-2D13004&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=8csmPWgUEWao6E4wthrG_-BX5a2OQJKXpkKtFLjSPlI&e=>
>
>
>
> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com>
> wrote:
>
> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug
> corrupting sstables on alter.
>
>
>
> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>
> wrote:
>
> Hi Shravan,
>
>
>
> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before
> doing the major upgrade? It's generally favorable to go to the latest patch
> release as often times they include fixes that smooth over the upgrade
> process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current
> version)
>
>
>
> Best,
>
> Marc
>
>
>
> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>
> Hello,
>
>
>
> I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9.
> I came up with the version based on [1]. I followed upgrade steps as in
> [2]. I was testing the same in the lab and encountered issues (streaming
> just fails and hangs for ever) with bootstrapping a 3.0.9 node on a
> partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The
> production cluster that I am supporting is pretty large and I am
> anticipating to end up in a situation like this (Hope not) and would like
> to be prepared.
>
>
>
> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
> cluster?
>
> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>
> 3) Is there an alternative approach to the upgrade large clusters. i.e
> instead of going through nodetool upgradesstables on each node in rolling
> fashion
>
>
>
>
>
> As per [1] the general restriction is to avoid decommissioning or adding
> nodes but in reality there can be failures or maintenance that warrants us
> to do so.
>
>
>
>
>
> Please point me in the right direction.
>
>
>
>
>
> Thanks,
>
> Shravan
>
>
>
>
>
> [1]
> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_upgrade_doc_upgrade_datastax-5Fenterprise_upgdDSE50.html-23upgdDSE50-5F-5Fcstar-2Dversion-2Dchange&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=IV10uiJPS2fyCHGYw7yR90A8cxcpjy9Is40YUWgPvF0&e=>
>
>
>
> [2]
> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
>
>
>
> --
>
> Marc Selwan | DataStax | Product Management | (925) 413-7079
>
>
>
>
>
> --
>
> Jon Haddad
> http://www.rustyrazorblade.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=HPfRIl5knAn9tpUyT06P6YaktRTDwyFSMO7r4GHXBzY&e=>
> twitter: rustyrazorblade
>
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
RE: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by "Durity, Sean R" <SE...@homedepot.com>.
We have had great success with Cassandra upgrades with applications staying on-line. It is one of the strongest benefits of Cassandra. A couple things I incorporate into upgrades:
- The main task is getting the new binaries loaded, then restarting the node – in a rolling fashion. Get this done as quickly as possible
- Streaming between versions is usually problematic. So, I never do any node additions or decommissions during an upgrade
- With applications running, there is not an acceptable back-out plan (either lose data or take a long outage or both), so we are always going forward. So, lower life cycle testing is important before hitting production
- Upgrading is a more frequent activity, so get the process/automation in place. The upgrade process should not be a reason to delay, especially for minor version upgrades that might be quickly necessary (security issue or bug fix).
Sean Durity
From: Shravan R <sk...@gmail.com>
Sent: Tuesday, December 04, 2018 12:22 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded cluster (2.1.9 + 3.x) and I was not able to do so. The schema never settled.
How does the below approach sound like?
1. Update the software binary on all nodes to use cassandra-3.x upon a restart.
2. Restart all nodes in a rolling fashion
3. Run nodetool upgradesstables in a rolling fashion
Is there a risk on pending nodetool upgradesstables?
On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa <jj...@gmail.com>> wrote:
On Dec 2, 2018, at 12:40 PM, Shravan R <sk...@gmail.com>> wrote:
Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the version part that you suggested. The reason to go direct to 3.x is to take a bi leap and reduce overall effort to upgrade a large cluster (development included).
I have these questions from my original post. Appreciate if you could shed some light and point me in the right direction.
1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
If any of the replicas have already upgraded, which is almost guaranteed if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade everything else and then deal with it. If a host fails mid upgrade you’ll likely have some period of unavailables while you bounce the replicas to finish, then you can decom
2) How to bootstrap a 3.x node to a partially upgraded cluster?
This may work fine, but test it because I’m not certain. It should be able to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
Bounce them all as quickly as is practical, do the upgradesstables after the bounces complete
On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jo...@jonhaddad.com>> wrote:
Dmitry is right. Generally speaking always go with the latest bug fix release.
On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <sa...@gmail.com>> wrote:
See more here
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_plugins_servlet_mobile-23issue_CASSANDRA-2D13004&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=8csmPWgUEWao6E4wthrG_-BX5a2OQJKXpkKtFLjSPlI&e=>
On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com>> wrote:
Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug corrupting sstables on alter.
On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>> wrote:
Hi Shravan,
Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before doing the major upgrade? It's generally favorable to go to the latest patch release as often times they include fixes that smooth over the upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current version)
Best,
Marc
On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com>> wrote:
Hello,
I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. I came up with the version based on [1]. I followed upgrade steps as in [2]. I was testing the same in the lab and encountered issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9 node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The production cluster that I am supporting is pretty large and I am anticipating to end up in a situation like this (Hope not) and would like to be prepared.
1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
2) How to bootstrap a 3.x node to a partially upgraded cluster?
3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
As per [1] the general restriction is to avoid decommissioning or adding nodes but in reality there can be failures or maintenance that warrants us to do so.
Please point me in the right direction.
Thanks,
Shravan
[1] https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_upgrade_doc_upgrade_datastax-5Fenterprise_upgdDSE50.html-23upgdDSE50-5F-5Fcstar-2Dversion-2Dchange&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=IV10uiJPS2fyCHGYw7yR90A8cxcpjy9Is40YUWgPvF0&e=>
[2] https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/<https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
--
Marc Selwan | DataStax | Product Management | (925) 413-7079
--
Jon Haddad
http://www.rustyrazorblade.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=HPfRIl5knAn9tpUyT06P6YaktRTDwyFSMO7r4GHXBzY&e=>
twitter: rustyrazorblade
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Shravan R <sk...@gmail.com>.
Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded
cluster (2.1.9 + 3.x) and I was *not* able to do so. The schema never
settled.
How does the below approach sound like?
1. Update the software binary on all nodes to use cassandra-3.x upon a
restart.
2. Restart all nodes in a rolling fashion
3. Run nodetool upgradesstables in a rolling fashion
Is there a risk on pending nodetool upgradesstables?
On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
>
> On Dec 2, 2018, at 12:40 PM, Shravan R <sk...@gmail.com> wrote:
>
> Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the
> version part that you suggested. The reason to go direct to 3.x is to take
> a bi leap and reduce overall effort to upgrade a large cluster (development
> included).
>
> I have these questions from my original post. Appreciate if you could shed
> some light and point me in the right direction.
>
> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
> cluster?
>
>
> If any of the replicas have already upgraded, which is almost guaranteed
> if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade
> everything else and then deal with it. If a host fails mid upgrade you’ll
> likely have some period of unavailables while you bounce the replicas to
> finish, then you can decom
>
>
>
> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>
>
> This may work fine, but test it because I’m not certain. It should be able
> to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
>
> 3) Is there an alternative approach to the upgrade large clusters. i.e
> instead of going through nodetool upgradesstables on each node in rolling
> fashion
>
>
> Bounce them all as quickly as is practical, do the upgradesstables after
> the bounces complete
>
>
>
>
>
> On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
>> Dmitry is right. Generally speaking always go with the latest bug fix
>> release.
>>
>> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <
>> saprykin.dmitry@gmail.com> wrote:
>>
>>> See more here
>>>
>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
>>>
>>> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <
>>> saprykin.dmitry@gmail.com> wrote:
>>>
>>>> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty
>>>> bug corrupting sstables on alter.
>>>>
>>>> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>
>>>> wrote:
>>>>
>>>>> Hi Shravan,
>>>>>
>>>>> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release
>>>>> before doing the major upgrade? It's generally favorable to go to the
>>>>> latest patch release as often times they include fixes that smooth over the
>>>>> upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20
>>>>> (current version)
>>>>>
>>>>> Best,
>>>>> Marc
>>>>>
>>>>> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache
>>>>>> Cassandra-3.0.9. I came up with the version based on [1]. I followed
>>>>>> upgrade steps as in [2]. I was testing the same in the lab and encountered
>>>>>> issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9
>>>>>> node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on
>>>>>> 3.0.9]. The production cluster that I am supporting is pretty large and I
>>>>>> am anticipating to end up in a situation like this (Hope not) and would
>>>>>> like to be prepared.
>>>>>>
>>>>>> 1) How do deal with decommissioning a 2.1.9 node in a partially
>>>>>> upgraded cluster?
>>>>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>>>>>> 3) Is there an alternative approach to the upgrade large clusters.
>>>>>> i.e instead of going through nodetool upgradesstables on each node in
>>>>>> rolling fashion
>>>>>>
>>>>>>
>>>>>> As per [1] the general restriction is to avoid decommissioning or
>>>>>> adding nodes but in reality there can be failures or maintenance that
>>>>>> warrants us to do so.
>>>>>>
>>>>>>
>>>>>> Please point me in the right direction.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Shravan
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>>>>>
>>>>>> [2]
>>>>>> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
>>>>>>
>>>>>> --
>>>>> Marc Selwan | DataStax | Product Management | (925) 413-7079
>>>>>
>>>>>
>>>>> --
>> Jon Haddad
>> http://www.rustyrazorblade.com
>> twitter: rustyrazorblade
>>
>
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Jeff Jirsa <jj...@gmail.com>.
> On Dec 2, 2018, at 12:40 PM, Shravan R <sk...@gmail.com> wrote:
>
> Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the version part that you suggested. The reason to go direct to 3.x is to take a bi leap and reduce overall effort to upgrade a large cluster (development included).
>
> I have these questions from my original post. Appreciate if you could shed some light and point me in the right direction.
>
> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
If any of the replicas have already upgraded, which is almost guaranteed if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade everything else and then deal with it. If a host fails mid upgrade you’ll likely have some period of unavailables while you bounce the replicas to finish, then you can decom
> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
This may work fine, but test it because I’m not certain. It should be able to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
> 3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
Bounce them all as quickly as is practical, do the upgradesstables after the bounces complete
>
>
>> On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
>> Dmitry is right. Generally speaking always go with the latest bug fix release.
>>
>>> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <sa...@gmail.com> wrote:
>>> See more here
>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
>>>
>>>> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com> wrote:
>>>> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug corrupting sstables on alter.
>>>>
>>>>> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com> wrote:
>>>>> Hi Shravan,
>>>>>
>>>>> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before doing the major upgrade? It's generally favorable to go to the latest patch release as often times they include fixes that smooth over the upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current version)
>>>>>
>>>>> Best,
>>>>> Marc
>>>>>
>>>>>> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. I came up with the version based on [1]. I followed upgrade steps as in [2]. I was testing the same in the lab and encountered issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9 node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The production cluster that I am supporting is pretty large and I am anticipating to end up in a situation like this (Hope not) and would like to be prepared.
>>>>>>
>>>>>> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
>>>>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>>>>>> 3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion
>>>>>>
>>>>>>
>>>>>> As per [1] the general restriction is to avoid decommissioning or adding nodes but in reality there can be failures or maintenance that warrants us to do so.
>>>>>>
>>>>>>
>>>>>> Please point me in the right direction.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Shravan
>>>>>>
>>>>>>
>>>>>> [1] https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>>>>>
>>>>>> [2] https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>>>>>>
>>>>> --
>>>>> Marc Selwan | DataStax | Product Management | (925) 413-7079
>>>>>
>>>>>
>> --
>> Jon Haddad
>> http://www.rustyrazorblade.com
>> twitter: rustyrazorblade
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Shravan R <sk...@gmail.com>.
Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the
version part that you suggested. The reason to go direct to 3.x is to take
a bi leap and reduce overall effort to upgrade a large cluster (development
included).
I have these questions from my original post. Appreciate if you could shed
some light and point me in the right direction.
1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
cluster?
2) How to bootstrap a 3.x node to a partially upgraded cluster?
3) Is there an alternative approach to the upgrade large clusters. i.e
instead of going through nodetool upgradesstables on each node in rolling
fashion
On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Dmitry is right. Generally speaking always go with the latest bug fix
> release.
>
> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <sa...@gmail.com>
> wrote:
>
>> See more here
>>
>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
>>
>> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com>
>> wrote:
>>
>>> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty
>>> bug corrupting sstables on alter.
>>>
>>> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>
>>> wrote:
>>>
>>>> Hi Shravan,
>>>>
>>>> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release
>>>> before doing the major upgrade? It's generally favorable to go to the
>>>> latest patch release as often times they include fixes that smooth over the
>>>> upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20
>>>> (current version)
>>>>
>>>> Best,
>>>> Marc
>>>>
>>>> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache
>>>>> Cassandra-3.0.9. I came up with the version based on [1]. I followed
>>>>> upgrade steps as in [2]. I was testing the same in the lab and encountered
>>>>> issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9
>>>>> node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on
>>>>> 3.0.9]. The production cluster that I am supporting is pretty large and I
>>>>> am anticipating to end up in a situation like this (Hope not) and would
>>>>> like to be prepared.
>>>>>
>>>>> 1) How do deal with decommissioning a 2.1.9 node in a partially
>>>>> upgraded cluster?
>>>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>>>>> 3) Is there an alternative approach to the upgrade large clusters. i.e
>>>>> instead of going through nodetool upgradesstables on each node in rolling
>>>>> fashion
>>>>>
>>>>>
>>>>> As per [1] the general restriction is to avoid decommissioning or
>>>>> adding nodes but in reality there can be failures or maintenance that
>>>>> warrants us to do so.
>>>>>
>>>>>
>>>>> Please point me in the right direction.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Shravan
>>>>>
>>>>>
>>>>> [1]
>>>>> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>>>>
>>>>> [2]
>>>>> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
>>>>>
>>>>> --
>>>> Marc Selwan | DataStax | Product Management | (925) 413-7079
>>>>
>>>>
>>>> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Dmitry is right. Generally speaking always go with the latest bug fix
release.
On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <sa...@gmail.com>
wrote:
> See more here
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
>
> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com>
> wrote:
>
>> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty
>> bug corrupting sstables on alter.
>>
>> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>
>> wrote:
>>
>>> Hi Shravan,
>>>
>>> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release
>>> before doing the major upgrade? It's generally favorable to go to the
>>> latest patch release as often times they include fixes that smooth over the
>>> upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20
>>> (current version)
>>>
>>> Best,
>>> Marc
>>>
>>> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache
>>>> Cassandra-3.0.9. I came up with the version based on [1]. I followed
>>>> upgrade steps as in [2]. I was testing the same in the lab and encountered
>>>> issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9
>>>> node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on
>>>> 3.0.9]. The production cluster that I am supporting is pretty large and I
>>>> am anticipating to end up in a situation like this (Hope not) and would
>>>> like to be prepared.
>>>>
>>>> 1) How do deal with decommissioning a 2.1.9 node in a partially
>>>> upgraded cluster?
>>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>>>> 3) Is there an alternative approach to the upgrade large clusters. i.e
>>>> instead of going through nodetool upgradesstables on each node in rolling
>>>> fashion
>>>>
>>>>
>>>> As per [1] the general restriction is to avoid decommissioning or
>>>> adding nodes but in reality there can be failures or maintenance that
>>>> warrants us to do so.
>>>>
>>>>
>>>> Please point me in the right direction.
>>>>
>>>>
>>>> Thanks,
>>>> Shravan
>>>>
>>>>
>>>> [1]
>>>> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>>>
>>>> [2]
>>>> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
>>>>
>>>> --
>>> Marc Selwan | DataStax | Product Management | (925) 413-7079
>>>
>>>
>>> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Dmitry Saprykin <sa...@gmail.com>.
See more here
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <sa...@gmail.com>
wrote:
> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug
> corrupting sstables on alter.
>
> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>
> wrote:
>
>> Hi Shravan,
>>
>> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before
>> doing the major upgrade? It's generally favorable to go to the latest patch
>> release as often times they include fixes that smooth over the upgrade
>> process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current
>> version)
>>
>> Best,
>> Marc
>>
>> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache
>>> Cassandra-3.0.9. I came up with the version based on [1]. I followed
>>> upgrade steps as in [2]. I was testing the same in the lab and encountered
>>> issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9
>>> node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on
>>> 3.0.9]. The production cluster that I am supporting is pretty large and I
>>> am anticipating to end up in a situation like this (Hope not) and would
>>> like to be prepared.
>>>
>>> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
>>> cluster?
>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>>> 3) Is there an alternative approach to the upgrade large clusters. i.e
>>> instead of going through nodetool upgradesstables on each node in rolling
>>> fashion
>>>
>>>
>>> As per [1] the general restriction is to avoid decommissioning or adding
>>> nodes but in reality there can be failures or maintenance that warrants us
>>> to do so.
>>>
>>>
>>> Please point me in the right direction.
>>>
>>>
>>> Thanks,
>>> Shravan
>>>
>>>
>>> [1]
>>> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>>
>>> [2]
>>> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
>>>
>>> --
>> Marc Selwan | DataStax | Product Management | (925) 413-7079
>>
>>
>>
Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Posted by Dmitry Saprykin <sa...@gmail.com>.
Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug
corrupting sstables on alter.
On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <ma...@datastax.com>
wrote:
> Hi Shravan,
>
> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before
> doing the major upgrade? It's generally favorable to go to the latest patch
> release as often times they include fixes that smooth over the upgrade
> process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current
> version)
>
> Best,
> Marc
>
> On Fri, Nov 30, 2018 at 3:13 PM Shravan R <sk...@gmail.com> wrote:
>
>> Hello,
>>
>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache
>> Cassandra-3.0.9. I came up with the version based on [1]. I followed
>> upgrade steps as in [2]. I was testing the same in the lab and encountered
>> issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9
>> node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on
>> 3.0.9]. The production cluster that I am supporting is pretty large and I
>> am anticipating to end up in a situation like this (Hope not) and would
>> like to be prepared.
>>
>> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
>> cluster?
>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>> 3) Is there an alternative approach to the upgrade large clusters. i.e
>> instead of going through nodetool upgradesstables on each node in rolling
>> fashion
>>
>>
>> As per [1] the general restriction is to avoid decommissioning or adding
>> nodes but in reality there can be failures or maintenance that warrants us
>> to do so.
>>
>>
>> Please point me in the right direction.
>>
>>
>> Thanks,
>> Shravan
>>
>>
>> [1]
>> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>
>> [2]
>> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
>>
>> --
> Marc Selwan | DataStax | Product Management | (925) 413-7079
>
>
>