You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nate McCall <zz...@gmail.com> on 2018/10/01 00:29:42 UTC
Re: Rolling back Cassandra upgrades (tarball)
> I have a cluster on v3.0.11 I am planning to upgrade this to 3.10.
> Is rolling back the binaries a viable solution?
What's the goal with moving form 3.0 to 3.x?
Also, our latest release in 3.x is 3.11.3 and has a couple of
important bug fixes over 3.10 (which is a bit dated at this point).
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: [EXTERNAL] Re: Rolling back Cassandra upgrades (tarball)
Posted by Jeff Jirsa <jj...@gmail.com>.
sstable version alone isn’t sufficient - there can be other surprises that will break the lower version (commitlog format change, new types or concepts like UDTs that may appear in the schema, etc)
I think 3.11 to 3.0 still works but I’m not certain of it personally
--
Jeff Jirsa
> On Oct 1, 2018, at 6:13 PM, Christophe Schmitz <ch...@instaclustr.com> wrote:
>
> Adding to the thread:
> SSTable format is identical between 3.0.x and 3..11.x, so your SSTable files are compatible, in this case. BTW an easy way to check that is to look at the SSTables filename convention; first letters ('mc' in this case) indicate the SSTable storage format version.
> In the future, if you really really want rollback when doing a major upgrade with a change of SSTable format, your only option will be to create a secondary data center (same number of nodes, same Cassandra version, please check your keyspaces are using NetworkTopologyStrategy). You will be able to upgrade the Cassandra version of one DC, while keeping the other DC to the current version. You will need to consider carefully the consistency level of your application (probably LOCAL_QUORUM) so that your application is writing to one DC, with automatic replication on the secondary DC. Once you are happy, you can decommission the old version DC (check carefully your application endpoint configuration, local_dc configuration)
> Hope this helps.
>
>
> Christophe Schmitz - Instaclustr - Cassandra | Kafka | Spark Consulting
>
>
>
>> On Mon, 1 Oct 2018 at 23:18 Durity, Sean R <SE...@homedepot.com> wrote:
>> Version choices aside, I am an advocate for forward-only (in most cases). Here is my reasoning, so that you can evaluate for your situation:
>> - upgrades are done while the application is up and live and writing data (no app downtime)
>> - the upgrade usually includes a change to the sstable version (which is unreadable in the older version)
>> - any data written to upgraded nodes will be written in the new sstable format
>> + this includes any compaction that takes place on upgraded nodes, so even an app outage doesn't protect you
>> - so, there is no going back, unless you are willing to lose new (or compacted) data written to any upgraded nodes
>>
>> As you can tell, if the assumptions don't hold true, a roll back may be possible. For example, if the sstable version is the same (e.g., for a minor upgrade), then the risk of lost data is gone. Or, if you are able to stop your application during the upgrade process and stop compaction. Etc.
>>
>> You could upgrade a single node to see how it behaves. If there is some problem, you could wipe out the data, go back to the old version, and bootstrap it again. Once I get to the 2nd node, though, I am only going forward.
>>
>> Sean Durity
>>
>>
>> -----Original Message-----
>> From: Jeff Jirsa <jj...@gmail.com>
>> Sent: Sunday, September 30, 2018 8:38 PM
>> To: user@cassandra.apache.org
>> Subject: [EXTERNAL] Re: Rolling back Cassandra upgrades (tarball)
>>
>> Definitely don’t go to 3.10, go to 3.11.3 or newest 3.0 instead
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Sep 30, 2018, at 5:29 PM, Nate McCall <zz...@gmail.com> wrote:
>>
>> >> I have a cluster on v3.0.11 I am planning to upgrade this to 3.10.
>> >> Is rolling back the binaries a viable solution?
>> >
>> > What's the goal with moving form 3.0 to 3.x?
>> >
>> > Also, our latest release in 3.x is 3.11.3 and has a couple of
>> > important bug fixes over 3.10 (which is a bit dated at this point).
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> > For additional commands, e-mail: user-help@cassandra.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>> ________________________________
>>
>> The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
Re: [EXTERNAL] Re: Rolling back Cassandra upgrades (tarball)
Posted by Christophe Schmitz <ch...@instaclustr.com>.
Adding to the thread:
- SSTable format is identical between 3.0.x and 3..11.x, so your SSTable
files are compatible, in this case. BTW an easy way to check that is to
look at the SSTables filename convention; first letters ('mc' in this case)
indicate the SSTable storage format version.
- In the future, if you really really want rollback when doing a major
upgrade with a change of SSTable format, your only option will be to create
a secondary data center (same number of nodes, same Cassandra version,
please check your keyspaces are using NetworkTopologyStrategy). You will be
able to upgrade the Cassandra version of one DC, while keeping the other DC
to the current version. You will need to consider carefully the consistency
level of your application (probably LOCAL_QUORUM) so that your application
is writing to one DC, with automatic replication on the secondary DC. Once
you are happy, you can decommission the old version DC (check carefully
your application endpoint configuration, local_dc configuration)
Hope this helps.
Christophe Schmitz - Instaclustr <https://www.instaclustr.com/> - Cassandra
| Kafka | Spark Consulting
On Mon, 1 Oct 2018 at 23:18 Durity, Sean R <SE...@homedepot.com>
wrote:
> Version choices aside, I am an advocate for forward-only (in most cases).
> Here is my reasoning, so that you can evaluate for your situation:
> - upgrades are done while the application is up and live and writing data
> (no app downtime)
> - the upgrade usually includes a change to the sstable version (which is
> unreadable in the older version)
> - any data written to upgraded nodes will be written in the new sstable
> format
> + this includes any compaction that takes place on upgraded nodes, so even
> an app outage doesn't protect you
> - so, there is no going back, unless you are willing to lose new (or
> compacted) data written to any upgraded nodes
>
> As you can tell, if the assumptions don't hold true, a roll back may be
> possible. For example, if the sstable version is the same (e.g., for a
> minor upgrade), then the risk of lost data is gone. Or, if you are able to
> stop your application during the upgrade process and stop compaction. Etc.
>
> You could upgrade a single node to see how it behaves. If there is some
> problem, you could wipe out the data, go back to the old version, and
> bootstrap it again. Once I get to the 2nd node, though, I am only going
> forward.
>
> Sean Durity
>
>
> -----Original Message-----
> From: Jeff Jirsa <jj...@gmail.com>
> Sent: Sunday, September 30, 2018 8:38 PM
> To: user@cassandra.apache.org
> Subject: [EXTERNAL] Re: Rolling back Cassandra upgrades (tarball)
>
> Definitely don’t go to 3.10, go to 3.11.3 or newest 3.0 instead
>
>
> --
> Jeff Jirsa
>
>
> On Sep 30, 2018, at 5:29 PM, Nate McCall <zz...@gmail.com> wrote:
>
> >> I have a cluster on v3.0.11 I am planning to upgrade this to 3.10.
> >> Is rolling back the binaries a viable solution?
> >
> > What's the goal with moving form 3.0 to 3.x?
> >
> > Also, our latest release in 3.x is 3.11.3 and has a couple of
> > important bug fixes over 3.10 (which is a bit dated at this point).
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: user-help@cassandra.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
> ________________________________
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
RE: [EXTERNAL] Re: Rolling back Cassandra upgrades (tarball)
Posted by "Durity, Sean R" <SE...@homedepot.com>.
Version choices aside, I am an advocate for forward-only (in most cases). Here is my reasoning, so that you can evaluate for your situation:
- upgrades are done while the application is up and live and writing data (no app downtime)
- the upgrade usually includes a change to the sstable version (which is unreadable in the older version)
- any data written to upgraded nodes will be written in the new sstable format
+ this includes any compaction that takes place on upgraded nodes, so even an app outage doesn't protect you
- so, there is no going back, unless you are willing to lose new (or compacted) data written to any upgraded nodes
As you can tell, if the assumptions don't hold true, a roll back may be possible. For example, if the sstable version is the same (e.g., for a minor upgrade), then the risk of lost data is gone. Or, if you are able to stop your application during the upgrade process and stop compaction. Etc.
You could upgrade a single node to see how it behaves. If there is some problem, you could wipe out the data, go back to the old version, and bootstrap it again. Once I get to the 2nd node, though, I am only going forward.
Sean Durity
-----Original Message-----
From: Jeff Jirsa <jj...@gmail.com>
Sent: Sunday, September 30, 2018 8:38 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Rolling back Cassandra upgrades (tarball)
Definitely don’t go to 3.10, go to 3.11.3 or newest 3.0 instead
--
Jeff Jirsa
On Sep 30, 2018, at 5:29 PM, Nate McCall <zz...@gmail.com> wrote:
>> I have a cluster on v3.0.11 I am planning to upgrade this to 3.10.
>> Is rolling back the binaries a viable solution?
>
> What's the goal with moving form 3.0 to 3.x?
>
> Also, our latest release in 3.x is 3.11.3 and has a couple of
> important bug fixes over 3.10 (which is a bit dated at this point).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Rolling back Cassandra upgrades (tarball)
Posted by Jeff Jirsa <jj...@gmail.com>.
Definitely don’t go to 3.10, go to 3.11.3 or newest 3.0 instead
--
Jeff Jirsa
On Sep 30, 2018, at 5:29 PM, Nate McCall <zz...@gmail.com> wrote:
>> I have a cluster on v3.0.11 I am planning to upgrade this to 3.10.
>> Is rolling back the binaries a viable solution?
>
> What's the goal with moving form 3.0 to 3.x?
>
> Also, our latest release in 3.x is 3.11.3 and has a couple of
> important bug fixes over 3.10 (which is a bit dated at this point).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org