You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Branimir Lambov <bl...@apache.org> on 2023/02/20 15:03:01 UTC

Downgradability

Hi everyone,

There has been a discussion lately about changes to the sstable format in
the context of being able to abort a cluster upgrade, and the fact that
changes to sstables can prevent downgraded nodes from reading any data
written during their temporary operation with the new version.

Most of the discussion is in CASSANDRA-18134
<https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is spreading
into CASSANDRA-14277 <https://issues.apache.org/jira/browse/CASSANDRA-14227>
and CASSANDRA-17698 <https://issues.apache.org/jira/browse/CASSANDRA-17698>,
none of which is a good place to discuss the topic seriously.

Downgradability is a worthy goal and is listed in the current roadmap. I
would like to open a discussion here on how it would be achieved.

My understanding of what has been suggested so far translates to:
- avoid changes to sstable formats;
- if there are changes, implement them in a way that is
backwards-compatible, e.g. by duplicating data, so that a new version is
presented in a component or portion of a component that legacy nodes will
not try to read;
- if the latter is not feasible, make sure the changes are only applied if
a feature flag has been enabled.

To me this approach introduces several risks:
- it bloats file and parsing complexity;
- it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
ticket once this requirement is in place);
- it needs care to avoid risky solutions to address technical issues with
the format versioning (e.g. staying on n-versions for 5.0 and needing a
bump for a 4.1 bugfix might require porting over support for new features);
- it requires separate and uncoordinated solutions to the problem and
switching mechanisms for each individual change.

An alternative solution is to implement/complete CASSANDRA-8110
<https://issues.apache.org/jira/browse/CASSANDRA-8110>, which provides a
method of writing sstables for a target version. During upgrades, a node
could be set to produce sstables corresponding to the older version, and
there is a very straightforward way to implement modifications to formats
like the tickets above to conform to its requirements.

What do people think should be the way forward?

Regards,
Branimir

Re: Downgradability

Posted by Benedict <be...@apache.org>.
This isn’t a feature in any normal sense? It’s a commitment, and it requires
every contributor to consider it as part of work they produce.

  

I agree we should put in some work to make this easier, and to provide some
test guarantees that we don’t break it. But are we overstating the difficulty?
We already use the descriptor version for serialising the contents, it’s only
the metadata that ignores this today - and downgrade tests should be as simple
as making upgrade tests go backwards.

  

But in the meantime none of that seems like an excuse to fail to address this
in work that may break it, where it is not particularly costly to do so.

  

> On 21 Feb 2023, at 08:57, Branimir Lambov <br...@datastax.com>
> wrote:  
>  
>

> 
>
> It appears to me that the first thing we need to start this feature off is a
> definition of a suite of tests together with a set of rules to keep the
> suite up to date with new features as they are introduced. The moment that
> suite is in place, we can start having some confidence that we can enforce
> downgradability.
>
>  
>
>
> Something like this will definitely catch incompatibilities in SSTable
> formats (such as the one in CASSANDRA-17698 that I managed to miss during
> review), but will also be able to identify incompatible system schema
> changes among others, and at the same time rightfully ignore non-breaking
> changes such as modifications to the key cache serialization formats.
>
>  
>
>
> Is downgradability in scope for 5.0? It is a feature like any other, and I
> don't see any difficulty adding it (with support for downgrade to 4.x) a
> little later in the 5.x timeline.
>
>  
>
>
> Regards,
>
> Branimir
>
>  
>
>
> On Tue, Feb 21, 2023 at 9:40 AM Jacek Lewandowski
> <[lewandowski.jacek@gmail.com](mailto:lewandowski.jacek@gmail.com)> wrote:  
>
>

>> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
multiple sstable formats support. It allows for providing an implementation of
SSTableFormat along with SSTableReader and SSTableWriter. That could be
extended easily to support different implementations for certain version
ranges, like one impl for ma-nz, other for oa+, etc. without having a
confusing implementation with a lot of conditional blocks. Old formats in such
case could be maintained separately from the main code and easily switched any
time.

>>

>>  
>
>>

>> thanks  
> \- - -- --- ----- -------- -------------  
> Jacek Lewandowski
>>

>>  
>
>>

>>  
>
>>

>> wt., 21 lut 2023 o 01:46 Yuki Morishita
<[yukim@apache.org](mailto:yukim@apache.org)> napisał(a):  
>
>>

>>> Hi,

>>>

>>>  
>
>>>

>>> What I wanted to address in my comment in
CASSANDRA-8110([https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-
tabpanel#comment-17641705](https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-
tabpanel#comment-17641705)) is to focus on better upgrade experience.

>>>

>>>  
>
>>>

>>> Upgrading the cluster can be painful for some orgs with mission critical
Cassandra cluster, where they cannot tolerate less availability because of the
inability to replace the downed node.

>>>

>>> They also need to plan rolling back to the previous state when something
happens along the way.

>>>

>>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at least
enabling SSTable streaming during the upgrade by not upgrading the SSTable
version. This can make the cluster to easily rollback to the previous version.  
>
>>>

>>> Downgrading SSTable is not the primary focus (though Cassandra needs to
implement the way to write SSTable in older versions, so it is somewhat
related.)

>>>

>>>  
>
>>>

>>> I'm preparing the design doc for the change.

>>>

>>> Also, if I should create a separate ticket from CASSANDRA-8110 for the
clarity of the goal of the change, please let me know.

>>>

>>>  
>
>>>

>>>  
>
>>>

>>> On Tue, Feb 21, 2023 at 5:31 AM Benedict
<[benedict@apache.org](mailto:benedict@apache.org)> wrote:  
>
>>>

>>>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
will have to eventually also tackle system schema changes (probably not hard),
and may have to think a little carefully about other things, eg with TTLs the
format change is only the contract about what values can be present, so we
have to make sure the data validity checks are consistent with the format we
write. It isn’t as simple as writing an earlier version in this case (unless
we permit truncating the TTL, perhaps)

>>>>

>>>>  
>
>>>>

>>>> On 20 Feb 2023, at 20:24, Benedict
<[benedict@apache.org](mailto:benedict@apache.org)> wrote:  
>
>>>>

>>>>>  
>
>>>>

>>>>> 

>>>>>

>>>>> In a self-organising community, things that aren’t self-policed
naturally end up policed in an adhoc manner, and with difficulty. I’m not sure
that’s the same as arbitrary enforcement. It seems to me the real issue is
nobody noticed this was agreed and/or forgot and didn’t think about it much.

>>>>>

>>>>>  
>
>>>>>

>>>>> But, even without any prior agreement, it’s perfectly reasonable to
request that things do not break compatibility if they do not need to, as part
of the normal patch integration process.

>>>>>

>>>>>  
>
>>>>>

>>>>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
agreement to do this. But we can and should address the problem of new columns
in schema tables, as this happens often between versions. I’m not sure it has
in 4.1 though?

>>>>>

>>>>>  
>
>>>>>

>>>>> Regarding downgrade versions, surely this should simply be the same as
upgrade versions we support?

>>>>>

>>>>>  
>
>>>>>

>>>>>  
>
>>>>>

>>>>>> On 20 Feb 2023, at 20:02, Jeff Jirsa
<[jjirsa@gmail.com](mailto:jjirsa@gmail.com)> wrote:  
>  
>
>>>>>

>>>>>> 

>>>>>>

>>>>>> I'm not even convinced even 8110 addresses this - just writing sstables
in old versions won't help if we ever add things like new types or new types
of collections without other control abilities. Claude's other email in
another thread a few hours ago talks about some of these surprises -
"Specifically during the 3.1 -> 4.0 changes a column broadcast_port was added
to system/local.  This means that 3.1 system can not read the table as it has
no definition for it.  I tried marking the column for deletion in the metadata
and in the serialization header.  The later got past the column not found
problem, but I suspect that it just means that data columns after
broadcast_port shifted and so incorrectly read." - this is a harder problem to
solve than just versioning sstables and network protocols.  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> Stepping back a bit, we have downgrade ability listed as a goal, but
it's not (as far as I can tell) universally enforced, nor is it clear at which
point we will be able to concretely say "this release can be downgraded to X".
Until we actually define and agree that this is a real goal with a concrete
version where downgrade-ability becomes real, it feels like things are
somewhat arbitrarily enforced, which is probably very frustrating for people
trying to commit work/tickets.

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> \- Jeff

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi
<[djoshi@apache.org](mailto:djoshi@apache.org)> wrote:  
>
>>>>>>

>>>>>>> I’m a big fan of maintaining backward compatibility. Downgradability
implies that we could potentially roll back an upgrade at any time. While I
don’t think we need to retain the ability to downgrade in perpetuity it would
be a good objective to maintain strict backward compatibility and therefore
downgradability until a certain point. This would imply versioning metadata
and extending it in such a way that prior version(s) could continue
functioning. This can certainly be expensive to implement and might bloat on-
disk storage. However, we could always offer an option for the operator to
optimize the on-disk structures for the current version then we can rewrite
them in the latest version. This optimizes the storage and opens up new
functionality. This means new features that can work with old on-disk
structures will be available while others that strictly require new versions
of the data structures will be unavailable until the operator migrates to the
new version. This migration IMO should be irreversible. Beyond this point the
operator will lose the ability to downgrade which is ok.  
>  
>
>>>>>>>

>>>>>>> Dinesh

>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani
<[jakers@gmail.com](mailto:jakers@gmail.com)> wrote:  
>  
>
>>>>>>>

>>>>>>>> 

>>>>>>>>

>>>>>>>> There has been progress on

>>>>>>>>

>>>>>>>>
<https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928>

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> Which is similar to what datastax does for DSE. Would this be an
acceptable solution?

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> Jake

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell
<[cclive1601@gmail.com](mailto:cclive1601@gmail.com)> wrote:  
>
>>>>>>>>

>>>>>>>>> It seems “An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110)” can
give us more options if it is finished😉

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Branimir Lambov
<[blambov@apache.org](mailto:blambov@apache.org)>于2023年2月20日 周一下午11:03写道:  
>
>>>>>>>>>

>>>>>>>>>> Hi everyone,  
>
>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> There has been a discussion lately about changes to the sstable
format in the context of being able to abort a cluster upgrade, and the fact
that changes to sstables can prevent downgraded nodes from reading any data
written during their temporary operation with the new version.

>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> Most of the discussion is in
[CASSANDRA-18134](https://issues.apache.org/jira/browse/CASSANDRA-18134), and
is spreading into
[CASSANDRA-14277](https://issues.apache.org/jira/browse/CASSANDRA-14227) and
[CASSANDRA-17698](https://issues.apache.org/jira/browse/CASSANDRA-17698), none
of which is a good place to discuss the topic seriously.

>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> Downgradability is a worthy goal and is listed in the current
roadmap. I would like to open a discussion here on how it would be achieved.  
>
>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> My understanding of what has been suggested so far translates to:

>>>>>>>>>>

>>>>>>>>>> \- avoid changes to sstable formats;

>>>>>>>>>>

>>>>>>>>>> \- if there are changes, implement them in a way that is backwards-
compatible, e.g. by duplicating data, so that a new version is presented in a
component or portion of a component that legacy nodes will not try to read;

>>>>>>>>>>

>>>>>>>>>> \- if the latter is not feasible, make sure the changes are only
applied if a feature flag has been enabled.

>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> To me this approach introduces several risks:

>>>>>>>>>>

>>>>>>>>>> \- it bloats file and parsing complexity;

>>>>>>>>>>

>>>>>>>>>> \- it discourages improvement (e.g. CASSANDRA-17698 is no longer a
LHF ticket once this requirement is in place);

>>>>>>>>>>

>>>>>>>>>> \- it needs care to avoid risky solutions to address technical
issues with the format versioning (e.g. staying on n-versions for 5.0 and
needing a bump for a 4.1 bugfix might require porting over support for new
features);  
>
>>>>>>>>>>

>>>>>>>>>> \- it requires separate and uncoordinated solutions to the problem
and switching mechanisms for each individual change.

>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110), which
provides a method of writing sstables for a target version. During upgrades, a
node could be set to produce sstables corresponding to the older version, and
there is a very straightforward way to implement modifications to formats like
the tickets above to conform to its requirements.

>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> What do people think should be the way forward?

>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> Regards,

>>>>>>>>>>

>>>>>>>>>> Branimir  
>
>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> \--  
>
>>>>>>>>>

>>>>>>>>> you are the apple of my eye !

>>>>>>>>

>>>>>>>> \--  
>
>>>>>>>>

>>>>>>>> <http://twitter.com/tjake>

>
>  
>
>
>  
>
>
> \--  
>
>
> Branimir Lambov
>
> e. [branimir.lambov@datastax.com](mailto:branimir.lambov@datastax.com)  
>  w. [www.datastax.com](http://www.datastax.com)
>
>  
>


Re: Downgradability

Posted by Benedict <be...@apache.org>.
Those tickets mostly do not need to break compatibility, and it is pretty easy
for them to avoid doing so without any additional facilities.

  

Only the TTL ticket has an excuse, as it debatably needs to support a higher
version under certain non-default config settings. However there are no
serialisation changes, so even here this is only a matter of selecting the
version we write to the descriptor and no other changes at all.

  

Can somebody explain to me what is so burdensome, that we seem to be spending
longer debating it than it would take to implement the necessary changes?

  

  

  

> On 22 Feb 2023, at 21:23, Jeremiah D Jordan <je...@gmail.com>
> wrote:  
>  
>

> We have multiple tickets about to merge that introduce new on disk format
> changes.  I see no reason to block those indefinitely while we figure out
> how to do the on disk format downgrade stuff.
>
>  
>
>
> -Jeremiah  
>
>
>  
>
>

>> On Feb 22, 2023, at 3:12 PM, Benedict <be...@apache.org> wrote:

>>

>>  
>
>>

>> Ok I will be honest, I was fairly sure we hadn’t yet broken downgrade - but
I was wrong. CASSANDRA-18061 introduced a new column to a system table, which
is a breaking change.

>>

>>  
>
>>

>> But that’s it, as far as I can tell. I have run a downgrade test
successfully after reverting that ticket, using the one line patch below. This
makes every in-jvm upgrade test also a downgrade test. I’m sure somebody more
familiar with dtests can readily do the same there.

>>

>>  
>
>>

>> While we look to fix 18061 and enable downgrade tests (and get a clean run
of the full suite), can we all agree not to introduce new breaking changes?

>>

>>  
>
>>

>>  
>
>>

>> index e41444fe52..085b25f8af 100644  
> \---
> a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java  
> +++
> b/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java  
> @@ -104,6 +104,7 @@ public class UpgradeTestBase extends DistributedTestBase  
>
> .addEdge(v40, v41)  
>
> .addEdge(v40, v42)  
>
> .addEdge(v41, v42)  
> \+
> .addEdge(v42, v41)  
>
> .build();
>>

>>  
>
>>

>>  
>
>>

>>> On 22 Feb 2023, at 15:08, Jeff Jirsa <jj...@gmail.com> wrote:  
>  
>
>>

>>> When people are serious about this requirement, they’ll build the
downgrade equivalents of the upgrade tests and run them automatically, often,
so people understand what the real gap is and when something new makes it
break  
>  
>
>>>

>>> Until those tests exist, I think collectively we should all stop
pretending like this is dogma. Best effort is best effort.

>>>

>>>  
>
>>>

>>>  
>
>>>

>>>  
>
>>>

>>>> On Feb 22, 2023, at 6:57 AM, Branimir Lambov
<br...@datastax.com> wrote:  
>  
>
>>>

>>>> 

>>>>

>>>> > 1\. Major SSTable changes should begin with forward-compatibility in a
prior release.  
>
>>>>

>>>>  
>
>>>>

>>>> This requires "feature" changes, i.e. new non-trivial code for previous
patch releases. It also entails porting over any further format modification.

>>>>

>>>>  
>
>>>>

>>>> Instead of this, in combination with your second point, why not implement
backwards write compatibility? The opt-in is then clearer to define (i.e.
upgrades start with e.g. a "4.1-compatible" settings set that includes file
format compatibility and disabling of new features, new nodes start with
"current" settings set). When the upgrade completes and the user is happy with
the result, the settings set can be replaced.

>>>>

>>>>  
>
>>>>

>>>> Doesn't this achieve what you want (and we all agree is a worthy goal)
with much less effort for everyone? Supporting backwards-compatible writing is
trivial, and we even have a proof-of-concept in the stats metadata serializer.
It also simplifies by a serious margin the amount of work and thinking one has
to do when a format improvement is implemented -- e.g. the TTL patch can just
address this in exactly the way the problem was addressed in earlier versions
of the format, by capping to 2038, without any need to specify, obey or test
any configuration flags.

>>>>

>>>>  
>
>>>>

>>>> >> It’s a commitment, and it requires every contributor to consider it as
part of work they produce.  
>  
> > But it shouldn't be a burden. Ability to downgrade is a testable problem,
> so I see this work as a function of the suite of tests the project is
> willing to agree on supporting.  
>
>>>>

>>>>  
>
>>>>

>>>> I fully agree with this sentiment, and I feel that the current "try to
not introduce breaking changes" approach is adding the burden, but not the
benefits -- because the latter cannot be proven, and are most likely already
broken.

>>>>

>>>>  
>
>>>>

>>>> Regards,

>>>>

>>>> Branimir

>>>>

>>>>  
>
>>>>

>>>> On Wed, Feb 22, 2023 at 1:01 AM Abe Ratnofsky
<[abe@aber.io](mailto:abe@aber.io)> wrote:  
>
>>>>

>>>>> Some interesting existing work on this subject is "Understanding and
Detecting Software Upgrade Failures in Distributed Systems" -
[https://dl.acm.org/doi/10.1145/3477132.3483577](https://urldefense.com/v3/__https://dl.acm.org/doi/10.1145/3477132.3483577__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NWm0FLO7w$),
also summarized by Andrey Satarin here:
[https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-
systems/](https://urldefense.com/v3/__https://asatarin.github.io/talks/2022-09-upgrade-
failures-in-distributed-
systems/__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NUfWWwFsA$)

>>>>>

>>>>>  
>
>>>>>

>>>>> They specifically tested Cassandra upgrades, and have a solid list of
defects that they found. They also describe their testing mechanism DUPTester,
which includes a component that confirms that the leftover state from one
version can start up on the next version. There is a wider scope of upgrade
defects highlighted in the paper, beyond SSTable version support.

>>>>>

>>>>>  
>
>>>>>

>>>>> I believe the project would benefit from expanding our test suite
similarly, by parametrizing more tests on upgrade version pairs.

>>>>>

>>>>>  
>
>>>>>

>>>>> Also, per Benedict's comment:

>>>>>

>>>>>  
>
>>>>>

>>>>> > It’s a commitment, and it requires every contributor to consider it as
part of work they produce.

>>>>>

>>>>>  
>
>>>>>

>>>>> But it shouldn't be a burden. Ability to downgrade is a testable
problem, so I see this work as a function of the suite of tests the project is
willing to agree on supporting.

>>>>>

>>>>>  
>
>>>>>

>>>>> Specifically - I agree with Scott's proposal to emulate the HDFS
upgrade-then-finalize approach. I would also support automatic finalization
based on a time threshold or similar, to balance the priorities of safe and
straightforward upgrades. Users need to be aware of the range of SSTable
formats supported by a given version, and how to handle when their SSTables
wouldn't be supported by an upcoming upgrade.

>>>>>

>>>>>  
>
>>>>>

>>>>> \--

>>>>>

>>>>> Abe

>>>>

>>>>  
>
>>>>

>>>>  
>
>>>>

>>>> \--  
>
>>>>

>>>> Branimir Lambov

>>>>

>>>> e. [branimir.lambov@datastax.com](mailto:branimir.lambov@datastax.com)  
>  w. [www.datastax.com](http://www.datastax.com/)
>>>>

>>>>  
>
>
>  
>


Re: Downgradability

Posted by Jeremiah D Jordan <je...@gmail.com>.
We have multiple tickets about to merge that introduce new on disk format changes.  I see no reason to block those indefinitely while we figure out how to do the on disk format downgrade stuff.

-Jeremiah

> On Feb 22, 2023, at 3:12 PM, Benedict <be...@apache.org> wrote:
> 
> Ok I will be honest, I was fairly sure we hadn’t yet broken downgrade - but I was wrong. CASSANDRA-18061 introduced a new column to a system table, which is a breaking change. 
> 
> But that’s it, as far as I can tell. I have run a downgrade test successfully after reverting that ticket, using the one line patch below. This makes every in-jvm upgrade test also a downgrade test. I’m sure somebody more familiar with dtests can readily do the same there.
> 
> While we look to fix 18061 and enable downgrade tests (and get a clean run of the full suite), can we all agree not to introduce new breaking changes?
> 
> 
> index e41444fe52..085b25f8af 100644
> --- a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java
> +++ b/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java
> @@ -104,6 +104,7 @@ public class UpgradeTestBase extends DistributedTestBase
>                                                                          .addEdge(v40, v41)
>                                                                          .addEdge(v40, v42)
>                                                                          .addEdge(v41, v42)
> +                                                                         .addEdge(v42, v41)
>                                                                          .build();
> 
> 
>> On 22 Feb 2023, at 15:08, Jeff Jirsa <jj...@gmail.com> wrote:
>> 
>> When people are serious about this requirement, they’ll build the downgrade equivalents of the upgrade tests and run them automatically, often, so people understand what the real gap is and when something new makes it break 
>> 
>> Until those tests exist, I think collectively we should all stop pretending like this is dogma. Best effort is best effort. 
>> 
>> 
>> 
>>> On Feb 22, 2023, at 6:57 AM, Branimir Lambov <br...@datastax.com> wrote:
>>> 
>>> 
>>> > 1. Major SSTable changes should begin with forward-compatibility in a prior release.
>>> 
>>> This requires "feature" changes, i.e. new non-trivial code for previous patch releases. It also entails porting over any further format modification.
>>> 
>>> Instead of this, in combination with your second point, why not implement backwards write compatibility? The opt-in is then clearer to define (i.e. upgrades start with e.g. a "4.1-compatible" settings set that includes file format compatibility and disabling of new features, new nodes start with "current" settings set). When the upgrade completes and the user is happy with the result, the settings set can be replaced.
>>> 
>>> Doesn't this achieve what you want (and we all agree is a worthy goal) with much less effort for everyone? Supporting backwards-compatible writing is trivial, and we even have a proof-of-concept in the stats metadata serializer. It also simplifies by a serious margin the amount of work and thinking one has to do when a format improvement is implemented -- e.g. the TTL patch can just address this in exactly the way the problem was addressed in earlier versions of the format, by capping to 2038, without any need to specify, obey or test any configuration flags.
>>> 
>>> >> It’s a commitment, and it requires every contributor to consider it as part of work they produce.
>>> 
>>> > But it shouldn't be a burden. Ability to downgrade is a testable problem, so I see this work as a function of the suite of tests the project is willing to agree on supporting.
>>> 
>>> I fully agree with this sentiment, and I feel that the current "try to not introduce breaking changes" approach is adding the burden, but not the benefits -- because the latter cannot be proven, and are most likely already broken.
>>> 
>>> Regards,
>>> Branimir
>>> 
>>> On Wed, Feb 22, 2023 at 1:01 AM Abe Ratnofsky <abe@aber.io <ma...@aber.io>> wrote:
>>>> Some interesting existing work on this subject is "Understanding and Detecting Software Upgrade Failures in Distributed Systems" - https://dl.acm.org/doi/10.1145/3477132.3483577 <https://urldefense.com/v3/__https://dl.acm.org/doi/10.1145/3477132.3483577__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NWm0FLO7w$>, also summarized by Andrey Satarin here: https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-systems/ <https://urldefense.com/v3/__https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-systems/__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NUfWWwFsA$>
>>>> 
>>>> They specifically tested Cassandra upgrades, and have a solid list of defects that they found. They also describe their testing mechanism DUPTester, which includes a component that confirms that the leftover state from one version can start up on the next version. There is a wider scope of upgrade defects highlighted in the paper, beyond SSTable version support.
>>>> 
>>>> I believe the project would benefit from expanding our test suite similarly, by parametrizing more tests on upgrade version pairs.
>>>> 
>>>> Also, per Benedict's comment:
>>>> 
>>>> > It’s a commitment, and it requires every contributor to consider it as part of work they produce.
>>>> 
>>>> But it shouldn't be a burden. Ability to downgrade is a testable problem, so I see this work as a function of the suite of tests the project is willing to agree on supporting.
>>>> 
>>>> Specifically - I agree with Scott's proposal to emulate the HDFS upgrade-then-finalize approach. I would also support automatic finalization based on a time threshold or similar, to balance the priorities of safe and straightforward upgrades. Users need to be aware of the range of SSTable formats supported by a given version, and how to handle when their SSTables wouldn't be supported by an upcoming upgrade.
>>>> 
>>>> --
>>>> Abe
>>> 
>>> 
>>> -- 
>>> Branimir Lambov
>>> e. branimir.lambov@datastax.com <ma...@datastax.com>
>>> w. www.datastax.com <http://www.datastax.com/>
>>> 


Re: Downgradability

Posted by Benedict <be...@apache.org>.
Ok I will be honest, I was fairly sure we hadn’t yet broken downgrade - but I
was wrong. CASSANDRA-18061 introduced a new column to a system table, which is
a breaking change.

  

But that’s it, as far as I can tell. I have run a downgrade test successfully
after reverting that ticket, using the one line patch below. This makes every
in-jvm upgrade test also a downgrade test. I’m sure somebody more familiar
with dtests can readily do the same there.

  

While we look to fix 18061 and enable downgrade tests (and get a clean run of
the full suite), can we all agree not to introduce new breaking changes?

  

  

index e41444fe52..085b25f8af 100644  
\---
a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java  
+++
b/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java  
@@ -104,6 +104,7 @@ public class UpgradeTestBase extends DistributedTestBase  
                                                                         .addEdge(v40, v41)  
                                                                         .addEdge(v40, v42)  
                                                                         .addEdge(v41, v42)  
\+
.addEdge(v42, v41)  
                                                                         .build();

  

  

> On 22 Feb 2023, at 15:08, Jeff Jirsa <jj...@gmail.com> wrote:  
>  
>

> When people are serious about this requirement, they’ll build the downgrade
> equivalents of the upgrade tests and run them automatically, often, so
> people understand what the real gap is and when something new makes it break  
>  
>
>
> Until those tests exist, I think collectively we should all stop pretending
> like this is dogma. Best effort is best effort.
>
>  
>
>
>  
>
>
>  
>
>

>> On Feb 22, 2023, at 6:57 AM, Branimir Lambov <br...@datastax.com>
wrote:  
>  
>
>

>> 

>>

>> > 1\. Major SSTable changes should begin with forward-compatibility in a
prior release.  
>
>>

>>  
>
>>

>> This requires "feature" changes, i.e. new non-trivial code for previous
patch releases. It also entails porting over any further format modification.

>>

>>  
>
>>

>> Instead of this, in combination with your second point, why not implement
backwards write compatibility? The opt-in is then clearer to define (i.e.
upgrades start with e.g. a "4.1-compatible" settings set that includes file
format compatibility and disabling of new features, new nodes start with
"current" settings set). When the upgrade completes and the user is happy with
the result, the settings set can be replaced.

>>

>>  
>
>>

>> Doesn't this achieve what you want (and we all agree is a worthy goal) with
much less effort for everyone? Supporting backwards-compatible writing is
trivial, and we even have a proof-of-concept in the stats metadata serializer.
It also simplifies by a serious margin the amount of work and thinking one has
to do when a format improvement is implemented -- e.g. the TTL patch can just
address this in exactly the way the problem was addressed in earlier versions
of the format, by capping to 2038, without any need to specify, obey or test
any configuration flags.

>>

>>  
>
>>

>> >> It’s a commitment, and it requires every contributor to consider it as
part of work they produce.  
>  
> > But it shouldn't be a burden. Ability to downgrade is a testable problem,
> so I see this work as a function of the suite of tests the project is
> willing to agree on supporting.  
>
>>

>>  
>
>>

>> I fully agree with this sentiment, and I feel that the current "try to not
introduce breaking changes" approach is adding the burden, but not the
benefits -- because the latter cannot be proven, and are most likely already
broken.

>>

>>  
>
>>

>> Regards,

>>

>> Branimir

>>

>>  
>
>>

>> On Wed, Feb 22, 2023 at 1:01 AM Abe Ratnofsky
<[abe@aber.io](mailto:abe@aber.io)> wrote:  
>
>>

>>> Some interesting existing work on this subject is "Understanding and
Detecting Software Upgrade Failures in Distributed Systems" -
[https://dl.acm.org/doi/10.1145/3477132.3483577](https://urldefense.com/v3/__https://dl.acm.org/doi/10.1145/3477132.3483577__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NWm0FLO7w$),
also summarized by Andrey Satarin here:
[https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-
systems/](https://urldefense.com/v3/__https://asatarin.github.io/talks/2022-09-upgrade-
failures-in-distributed-
systems/__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NUfWWwFsA$)

>>>

>>>  
>
>>>

>>> They specifically tested Cassandra upgrades, and have a solid list of
defects that they found. They also describe their testing mechanism DUPTester,
which includes a component that confirms that the leftover state from one
version can start up on the next version. There is a wider scope of upgrade
defects highlighted in the paper, beyond SSTable version support.

>>>

>>>  
>
>>>

>>> I believe the project would benefit from expanding our test suite
similarly, by parametrizing more tests on upgrade version pairs.

>>>

>>>  
>
>>>

>>> Also, per Benedict's comment:

>>>

>>>  
>
>>>

>>> > It’s a commitment, and it requires every contributor to consider it as
part of work they produce.

>>>

>>>  
>
>>>

>>> But it shouldn't be a burden. Ability to downgrade is a testable problem,
so I see this work as a function of the suite of tests the project is willing
to agree on supporting.

>>>

>>>  
>
>>>

>>> Specifically - I agree with Scott's proposal to emulate the HDFS upgrade-
then-finalize approach. I would also support automatic finalization based on a
time threshold or similar, to balance the priorities of safe and
straightforward upgrades. Users need to be aware of the range of SSTable
formats supported by a given version, and how to handle when their SSTables
wouldn't be supported by an upcoming upgrade.

>>>

>>>  
>
>>>

>>> \--

>>>

>>> Abe

>>

>>  
>
>>

>>  
>
>>

>> \--  
>
>>

>> Branimir Lambov

>>

>> e. [branimir.lambov@datastax.com](mailto:branimir.lambov@datastax.com)  
>  w. [www.datastax.com](http://www.datastax.com)
>>

>>  
>


Re: Downgradability

Posted by Josh McKenzie <jm...@apache.org>.
> why not implement backwards write compatibility?
+1 to this from a philosophical perspective. Keeping prior releases completely in the dark about new release sstable formats is a clean approach, and we should already have the code around to ser/deser the prior version's data on the next version.

On Wed, Feb 22, 2023, at 10:07 AM, Jeff Jirsa wrote:
> When people are serious about this requirement, they’ll build the downgrade equivalents of the upgrade tests and run them automatically, often, so people understand what the real gap is and when something new makes it break 
> 
> Until those tests exist, I think collectively we should all stop pretending like this is dogma. Best effort is best effort. 
> 
> 
> 
>> On Feb 22, 2023, at 6:57 AM, Branimir Lambov <br...@datastax.com> wrote:
>> 
>> > 1. Major SSTable changes should begin with forward-compatibility in a prior release.
>> 
>> This requires "feature" changes, i.e. new non-trivial code for previous patch releases. It also entails porting over any further format modification.
>> 
>> Instead of this, in combination with your second point, why not implement backwards write compatibility? The opt-in is then clearer to define (i.e. upgrades start with e.g. a "4.1-compatible" settings set that includes file format compatibility and disabling of new features, new nodes start with "current" settings set). When the upgrade completes and the user is happy with the result, the settings set can be replaced.
>> 
>> Doesn't this achieve what you want (and we all agree is a worthy goal) with much less effort for everyone? Supporting backwards-compatible writing is trivial, and we even have a proof-of-concept in the stats metadata serializer. It also simplifies by a serious margin the amount of work and thinking one has to do when a format improvement is implemented -- e.g. the TTL patch can just address this in exactly the way the problem was addressed in earlier versions of the format, by capping to 2038, without any need to specify, obey or test any configuration flags.
>> 
>> >> It’s a commitment, and it requires every contributor to consider it as part of work they produce.
>> 
>> > But it shouldn't be a burden. Ability to downgrade is a testable problem, so I see this work as a function of the suite of tests the project is willing to agree on supporting.
>> 
>> I fully agree with this sentiment, and I feel that the current "try to not introduce breaking changes" approach is adding the burden, but not the benefits -- because the latter cannot be proven, and are most likely already broken.
>> 
>> Regards,
>> Branimir
>> 
>> On Wed, Feb 22, 2023 at 1:01 AM Abe Ratnofsky <ab...@aber.io> wrote:
>>> Some interesting existing work on this subject is "Understanding and Detecting Software Upgrade Failures in Distributed Systems" - https://dl.acm.org/doi/10.1145/3477132.3483577 <https://urldefense.com/v3/__https://dl.acm.org/doi/10.1145/3477132.3483577__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NWm0FLO7w$>, also summarized by Andrey Satarin here: https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-systems/ <https://urldefense.com/v3/__https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-systems/__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NUfWWwFsA$>
>>> 
>>> They specifically tested Cassandra upgrades, and have a solid list of defects that they found. They also describe their testing mechanism DUPTester, which includes a component that confirms that the leftover state from one version can start up on the next version. There is a wider scope of upgrade defects highlighted in the paper, beyond SSTable version support.
>>> 
>>> I believe the project would benefit from expanding our test suite similarly, by parametrizing more tests on upgrade version pairs.
>>> 
>>> Also, per Benedict's comment:
>>> 
>>> > It’s a commitment, and it requires every contributor to consider it as part of work they produce.
>>> 
>>> But it shouldn't be a burden. Ability to downgrade is a testable problem, so I see this work as a function of the suite of tests the project is willing to agree on supporting.
>>> 
>>> Specifically - I agree with Scott's proposal to emulate the HDFS upgrade-then-finalize approach. I would also support automatic finalization based on a time threshold or similar, to balance the priorities of safe and straightforward upgrades. Users need to be aware of the range of SSTable formats supported by a given version, and how to handle when their SSTables wouldn't be supported by an upcoming upgrade.
>>> 
>>> --
>>> Abe
>> 
>> 
>> --
>> Branimir Lambov
>> e. branimir.lambov@datastax.com
>> w. www.datastax.com
>> 

Re: Downgradability

Posted by Jeff Jirsa <jj...@gmail.com>.
When people are serious about this requirement, they’ll build the downgrade
equivalents of the upgrade tests and run them automatically, often, so people
understand what the real gap is and when something new makes it break  
  

Until those tests exist, I think collectively we should all stop pretending
like this is dogma. Best effort is best effort.

  

  

  

> On Feb 22, 2023, at 6:57 AM, Branimir Lambov <br...@datastax.com>
> wrote:  
>  
>

> 
>
> > 1\. Major SSTable changes should begin with forward-compatibility in a
> prior release.  
>
>
>  
>
>
> This requires "feature" changes, i.e. new non-trivial code for previous
> patch releases. It also entails porting over any further format
> modification.
>
>  
>
>
> Instead of this, in combination with your second point, why not implement
> backwards write compatibility? The opt-in is then clearer to define (i.e.
> upgrades start with e.g. a "4.1-compatible" settings set that includes file
> format compatibility and disabling of new features, new nodes start with
> "current" settings set). When the upgrade completes and the user is happy
> with the result, the settings set can be replaced.
>
>  
>
>
> Doesn't this achieve what you want (and we all agree is a worthy goal) with
> much less effort for everyone? Supporting backwards-compatible writing is
> trivial, and we even have a proof-of-concept in the stats metadata
> serializer. It also simplifies by a serious margin the amount of work and
> thinking one has to do when a format improvement is implemented -- e.g. the
> TTL patch can just address this in exactly the way the problem was addressed
> in earlier versions of the format, by capping to 2038, without any need to
> specify, obey or test any configuration flags.
>
>  
>
>
> >> It’s a commitment, and it requires every contributor to consider it as
> part of work they produce.  
>  
> > But it shouldn't be a burden. Ability to downgrade is a testable problem,
> so I see this work as a function of the suite of tests the project is
> willing to agree on supporting.  
>
>
>  
>
>
> I fully agree with this sentiment, and I feel that the current "try to not
> introduce breaking changes" approach is adding the burden, but not the
> benefits -- because the latter cannot be proven, and are most likely already
> broken.
>
>  
>
>
> Regards,
>
> Branimir
>
>  
>
>
> On Wed, Feb 22, 2023 at 1:01 AM Abe Ratnofsky
> <[abe@aber.io](mailto:abe@aber.io)> wrote:  
>
>

>> Some interesting existing work on this subject is "Understanding and
Detecting Software Upgrade Failures in Distributed Systems" -
[https://dl.acm.org/doi/10.1145/3477132.3483577](https://urldefense.com/v3/__https://dl.acm.org/doi/10.1145/3477132.3483577__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NWm0FLO7w$),
also summarized by Andrey Satarin here:
[https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-
systems/](https://urldefense.com/v3/__https://asatarin.github.io/talks/2022-09-upgrade-
failures-in-distributed-
systems/__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NUfWWwFsA$)

>>

>>  
>
>>

>> They specifically tested Cassandra upgrades, and have a solid list of
defects that they found. They also describe their testing mechanism DUPTester,
which includes a component that confirms that the leftover state from one
version can start up on the next version. There is a wider scope of upgrade
defects highlighted in the paper, beyond SSTable version support.

>>

>>  
>
>>

>> I believe the project would benefit from expanding our test suite
similarly, by parametrizing more tests on upgrade version pairs.

>>

>>  
>
>>

>> Also, per Benedict's comment:

>>

>>  
>
>>

>> > It’s a commitment, and it requires every contributor to consider it as
part of work they produce.

>>

>>  
>
>>

>> But it shouldn't be a burden. Ability to downgrade is a testable problem,
so I see this work as a function of the suite of tests the project is willing
to agree on supporting.

>>

>>  
>
>>

>> Specifically - I agree with Scott's proposal to emulate the HDFS upgrade-
then-finalize approach. I would also support automatic finalization based on a
time threshold or similar, to balance the priorities of safe and
straightforward upgrades. Users need to be aware of the range of SSTable
formats supported by a given version, and how to handle when their SSTables
wouldn't be supported by an upcoming upgrade.

>>

>>  
>
>>

>> \--

>>

>> Abe

>
>  
>
>
>  
>
>
> \--  
>
>
> Branimir Lambov
>
> e. [branimir.lambov@datastax.com](mailto:branimir.lambov@datastax.com)  
>  w. [www.datastax.com](http://www.datastax.com)
>
>  
>


Re: Downgradability

Posted by Branimir Lambov <br...@datastax.com>.
> 1. Major SSTable changes should begin with forward-compatibility in a
prior release.

This requires "feature" changes, i.e. new non-trivial code for previous
patch releases. It also entails porting over any further format
modification.

Instead of this, in combination with your second point, why not implement
backwards write compatibility? The opt-in is then clearer to define (i.e.
upgrades start with e.g. a "4.1-compatible" settings set that includes file
format compatibility and disabling of new features, new nodes start with
"current" settings set). When the upgrade completes and the user is happy
with the result, the settings set can be replaced.

Doesn't this achieve what you want (and we all agree is a worthy goal) with
much less effort for everyone? Supporting backwards-compatible writing is
trivial, and we even have a proof-of-concept in the stats metadata
serializer. It also simplifies by a serious margin the amount of work and
thinking one has to do when a format improvement is implemented -- e.g. the
TTL patch can just address this in exactly the way the problem was
addressed in earlier versions of the format, by capping to 2038, without
any need to specify, obey or test any configuration flags.

>> It’s a commitment, and it requires every contributor to consider it as
part of work they produce.

> But it shouldn't be a burden. Ability to downgrade is a testable problem,
so I see this work as a function of the suite of tests the project is
willing to agree on supporting.

I fully agree with this sentiment, and I feel that the current "try to not
introduce breaking changes" approach is adding the burden, but not the
benefits -- because the latter cannot be proven, and are most likely
already broken.

Regards,
Branimir

On Wed, Feb 22, 2023 at 1:01 AM Abe Ratnofsky <ab...@aber.io> wrote:

> Some interesting existing work on this subject is "Understanding and
> Detecting Software Upgrade Failures in Distributed Systems" -
> https://dl.acm.org/doi/10.1145/3477132.3483577
> <https://urldefense.com/v3/__https://dl.acm.org/doi/10.1145/3477132.3483577__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NWm0FLO7w$>,
> also summarized by Andrey Satarin here:
> https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-systems/
> <https://urldefense.com/v3/__https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-systems/__;!!PbtH5S7Ebw!ZUMhWOKjMaK62HKCGLYN0rAhZbbX8fOJkgCsfMgjYO5EgJQulefcb5pwH4q5oU5ylLl6W56W-NUfWWwFsA$>
>
> They specifically tested Cassandra upgrades, and have a solid list of
> defects that they found. They also describe their testing mechanism
> DUPTester, which includes a component that confirms that the leftover state
> from one version can start up on the next version. There is a wider scope
> of upgrade defects highlighted in the paper, beyond SSTable version support.
>
> I believe the project would benefit from expanding our test suite
> similarly, by parametrizing more tests on upgrade version pairs.
>
> Also, per Benedict's comment:
>
> > It’s a commitment, and it requires every contributor to consider it as
> part of work they produce.
>
> But it shouldn't be a burden. Ability to downgrade is a testable problem,
> so I see this work as a function of the suite of tests the project is
> willing to agree on supporting.
>
> Specifically - I agree with Scott's proposal to emulate the HDFS
> upgrade-then-finalize approach. I would also support automatic finalization
> based on a time threshold or similar, to balance the priorities of safe and
> straightforward upgrades. Users need to be aware of the range of SSTable
> formats supported by a given version, and how to handle when their SSTables
> wouldn't be supported by an upcoming upgrade.
>
> --
> Abe
>


-- 
Branimir Lambov
e. branimir.lambov@datastax.com
w. www.datastax.com

Re: Downgradability

Posted by Abe Ratnofsky <ab...@aber.io>.
Some interesting existing work on this subject is "Understanding and Detecting Software Upgrade Failures in Distributed Systems" - https://dl.acm.org/doi/10.1145/3477132.3483577, also summarized by Andrey Satarin here: https://asatarin.github.io/talks/2022-09-upgrade-failures-in-distributed-systems/

They specifically tested Cassandra upgrades, and have a solid list of defects that they found. They also describe their testing mechanism DUPTester, which includes a component that confirms that the leftover state from one version can start up on the next version. There is a wider scope of upgrade defects highlighted in the paper, beyond SSTable version support.

I believe the project would benefit from expanding our test suite similarly, by parametrizing more tests on upgrade version pairs.

Also, per Benedict's comment:

> It’s a commitment, and it requires every contributor to consider it as part of work they produce.

But it shouldn't be a burden. Ability to downgrade is a testable problem, so I see this work as a function of the suite of tests the project is willing to agree on supporting.

Specifically - I agree with Scott's proposal to emulate the HDFS upgrade-then-finalize approach. I would also support automatic finalization based on a time threshold or similar, to balance the priorities of safe and straightforward upgrades. Users need to be aware of the range of SSTable formats supported by a given version, and how to handle when their SSTables wouldn't be supported by an upcoming upgrade.

--
Abe

Re: Downgradability

Posted by Jeremiah D Jordan <je...@gmail.com>.
If we can get opt-in major format upgrades, as well as an offline sstabledowngrade tool, I think we have a good first step that would make downgrades possible.

Given Jacek’s work on the sstable format API, and the work from Yuki and Claude on old formats, I think we are pretty close to having both of those be viable?

I think with the opt-in major format upgrades, the main thing will be to ensure that all new features that were built around the new format either fail gracefully, or for a change in behavior opt to the old behavior until the new format is available?  If a new feature is using a feature flag this could be a simple check to throw a configuration exception if the feature is enabled, but the new sstable format is not available.

No features have yet merged that bump the sstable major version, but a few are finishing up that will.  Do we want to block merging those changes until discussions here finish?  I don’t think that we need to?  The ticket which brings in the ability to opt-in to the sstable format change can also fix up the existing code to check the flag?

-Jeremiah


> On Feb 21, 2023, at 10:29 AM, Benedict <be...@apache.org> wrote:
> 
> As always, Scott puts it much more eloquently than I can. 
> 
> The only thing I’d quibble with is that I think it is better to make changes backwards compatible, rather than make earlier releases forwards compatible - and where this is prohibitively costly to simply make a feature that depends on it unavailable until the switch to the new major format.
> 
> This provides the greatest flexibility for users, as they can upgrade from and downgrade to the same versions. There’s no scrambling for a different downgrade target you haven’t qualified when finding out there’s an unacceptable bug. 
> There’s also less delta between pre-upgrade and post-downgrade behaviour.
> 
> We have plenty of practice doing this kind of thing. It’s not that hard.
> 
> But, if we want to go the forward compatibility route that’s still far better than nothing.
> 
> 
>> On 21 Feb 2023, at 16:17, C. Scott Andreas <sc...@paradoxica.net> wrote:
>> 
>> 
>> I realize my feedback on this has been spread across tickets and older mailing list / wiki discussions, so I'll offer a proposal here.
>> 
>> Starting with goals -
>> 
>> 1. Cassandra users must be able to abort and revert an upgrade to a new version of the database that introduces a new major SSTable format.
>> 
>> This reduces risk of upgrading to a build that also introduces a non-data-format-related bug that is intolerable. This goal does not specify a mechanism or downgrade target - just the "downgradability" goal.
>> 
>> 2. Where possible, Cassandra users should be able to opt into writing of a new major SSTable format.
>> 
>> This reduces that risk further by allowing users to decouple data format changes from the upgrade itself. There may be cases where new features or bug fixes prevent this from being possible, but I'll offer it as a goal.
>> 
>> 3. It should be possible for users to perform the downgrade in-place by launching the database using a previous version's binary.
>> 
>> This avoids the need for complex orchestration of offline commands like a hypothetical `downgradesstables`.
>> 
>> 
>> The following approach would allow us to accomplish these goals:
>> 
>> 1. Major SSTable changes should begin with forward-compatibility in a prior release.
>> 
>> In a release prior to one that revs major SSTable versions, we should implement the ability to read the SSTables that we intend to write in the next major version. This would allow someone to (eg.,) revert from 5.0 to 4.2 if they encountered a regression that caused an outage without data loss. This downgrade path should be well-specified and called out in NEWS.txt.
>> 
>> 2. Where possible, major SSTable format changes should be opt-in (if the features / bugfixes introduced allow).
>> 
>> This would be via a flag to enable writing the new format once an operator has determined that post-upgrade their clusters are sufficiently stable. This is an approach that HDFS has adopted. Following a rolling upgrade of HDFS, downgrade remains possible until an operator executes a "finalize" operation to migrate NameNode metadata to the new version's. An approach like this would allow users to perform a staged upgrade in which they first rev the version of the database, followed by opting into its new format to derisk (eg.,) adoption of BTI-indexed SSTables.
>> 
>> These approaches aren't meant to discourage SSTable format evolution - but to make it safer, and ideally faster. They don't specify duplicative serialization or a game of Twister to hide fields in locations where old versions don't think to look. Forward compatibility in a prior release could be landed at the same time as the major format revision itself, so long as we cut releases from both branches.
>> 
>> Ability to back out an upgrade until finalized would dramatically lower the risk of adopting new releases of Apache Cassandra. For many users, the qualification cycle for a new release is more than a year - and a *lot* of work.
>> 
>> Reducing the risk of upgrading to new releases repositions Cassandra as a database that can be treated with greater trust -- especially for multi-petabyte, mission critical systems. Our user community will advance to newer releases more quickly and we'll be able to shorten the maintenance cycles for older releases. In the same way that CI stability enables us to move faster and more confidently in the project, safety features like this will enable our users (and indeed ourselves) to move more confidently to adopt them.
>> 
>> – Scott
>> 
>> 
>>> On Feb 21, 2023, at 4:51 AM, "Claude Warren, Jr via dev" <de...@cassandra.apache.org> wrote:
>>> 
>>> 
>>> My goal in implementing CASSANDRA-8928 <https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928> was to be able to take the current version 4.x and write it as the earliest 3.x version possible.  The reasoning being that if that was possible then whatever 3.x version was executed would be able to automatically read the early 3.x version.  My thought was that each release version would have the ability to downgrade to the earliest previous version.  In this way if users need to they could string together a number of downgrader versions to move from 5.x to 3.x. 
>>> 
>>> My testing has been pretty straightforward, I created 4 docker containers using the standard published Cassandra docker images for 3.1 and 4.0 with data mounted on an external drive.  two of the containers (one of each version) did not automatically start Cassandra.  My process was then:
>>> start and stop Cassandra 4.0 to create the default data files
>>> start the Cassandra 4.0 container that does not automatically run Cassandra and execute the new downgrade functionality.
>>> start the Cassandra 3.1 container and dump the logs.  If the system started then I knew that I at least had a proof of concept.  So far no-go.
>>> 
>>> 
>>> On Tue, Feb 21, 2023 at 8:57 AM Branimir Lambov <branimir.lambov@datastax.com <ma...@datastax.com>> wrote:
>>>> It appears to me that the first thing we need to start this feature off is a definition of a suite of tests together with a set of rules to keep the suite up to date with new features as they are introduced. The moment that suite is in place, we can start having some confidence that we can enforce downgradability.
>>>> 
>>>> Something like this will definitely catch incompatibilities in SSTable formats (such as the one in CASSANDRA-17698 that I managed to miss during review), but will also be able to identify incompatible system schema changes among others, and at the same time rightfully ignore non-breaking changes such as modifications to the key cache serialization formats.
>>>> 
>>>> Is downgradability in scope for 5.0? It is a feature like any other, and I don't see any difficulty adding it (with support for downgrade to 4.x) a little later in the 5.x timeline.
>>>> 
>>>> Regards,
>>>> Branimir
>>>> 
>>>> 
>>>> On Tue, Feb 21, 2023 at 9:40 AM Jacek Lewandowski <lewandowski.jacek@gmail.com <ma...@gmail.com>> wrote:
>>>>> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce multiple sstable formats support. It allows for providing an implementation of SSTableFormat along with SSTableReader and SSTableWriter. That could be extended easily to support different implementations for certain version ranges, like one impl for ma-nz, other for oa+, etc. without having a confusing implementation with a lot of conditional blocks. Old formats in such case could be maintained separately from the main code and easily switched any time. 
>>>>> 
>>>>> thanks
>>>>> - - -- --- ----- -------- -------------
>>>>> Jacek Lewandowski
>>>>> 
>>>>> 
>>>>> wt., 21 lut 2023 o 01:46 Yuki Morishita <yukim@apache.org <ma...@apache.org>> napisał(a):
>>>>>> Hi,
>>>>>> 
>>>>>> What I wanted to address in my comment in CASSANDRA-8110(https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705) is to focus on better upgrade experience.
>>>>>> 
>>>>>> Upgrading the cluster can be painful for some orgs with mission critical Cassandra cluster, where they cannot tolerate less availability because of the inability to replace the downed node.
>>>>>> They also need to plan rolling back to the previous state when something happens along the way.
>>>>>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at least enabling SSTable streaming during the upgrade by not upgrading the SSTable version. This can make the cluster to easily rollback to the previous version.
>>>>>> Downgrading SSTable is not the primary focus (though Cassandra needs to implement the way to write SSTable in older versions, so it is somewhat related.)
>>>>>> 
>>>>>> I'm preparing the design doc for the change.
>>>>>> Also, if I should create a separate ticket from CASSANDRA-8110 for the clarity of the goal of the change, please let me know.
>>>>>> 
>>>>>> 
>>>>>> On Tue, Feb 21, 2023 at 5:31 AM Benedict <benedict@apache.org <ma...@apache.org>> wrote:
>>>>>>> 
>>>>>>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We will have to eventually also tackle system schema changes (probably not hard), and may have to think a little carefully about other things, eg with TTLs the format change is only the contract about what values can be present, so we have to make sure the data validity checks are consistent with the format we write. It isn’t as simple as writing an earlier version in this case (unless we permit truncating the TTL, perhaps) 
>>>>>>> 
>>>>>>> On 20 Feb 2023, at 20:24, Benedict <benedict@apache.org <ma...@apache.org>> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> In a self-organising community, things that aren’t self-policed naturally end up policed in an adhoc manner, and with difficulty. I’m not sure that’s the same as arbitrary enforcement. It seems to me the real issue is nobody noticed this was agreed and/or forgot and didn’t think about it much. 
>>>>>>>> 
>>>>>>>> But, even without any prior agreement, it’s perfectly reasonable to request that things do not break compatibility if they do not need to, as part of the normal patch integration process.
>>>>>>>> 
>>>>>>>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any agreement to do this. But we can and should address the problem of new columns in schema tables, as this happens often between versions. I’m not sure it has in 4.1 though?
>>>>>>>> 
>>>>>>>> Regarding downgrade versions, surely this should simply be the same as upgrade versions we support?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On 20 Feb 2023, at 20:02, Jeff Jirsa <jjirsa@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>> 
>>>>>>>>> I'm not even convinced even 8110 addresses this - just writing sstables in old versions won't help if we ever add things like new types or new types of collections without other control abilities. Claude's other email in another thread a few hours ago talks about some of these surprises - "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was added to system/local.  This means that 3.1 system can not read the table as it has no definition for it.  I tried marking the column for deletion in the metadata and in the serialization header.  The later got past the column not found problem, but I suspect that it just means that data columns after broadcast_port shifted and so incorrectly read." - this is a harder problem to solve than just versioning sstables and network protocols. 
>>>>>>>>> 
>>>>>>>>> Stepping back a bit, we have downgrade ability listed as a goal, but it's not (as far as I can tell) universally enforced, nor is it clear at which point we will be able to concretely say "this release can be downgraded to X".   Until we actually define and agree that this is a real goal with a concrete version where downgrade-ability becomes real, it feels like things are somewhat arbitrarily enforced, which is probably very frustrating for people trying to commit work/tickets.
>>>>>>>>> 
>>>>>>>>> - Jeff
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <djoshi@apache.org <ma...@apache.org>> wrote:
>>>>>>>>>> I’m a big fan of maintaining backward compatibility. Downgradability implies that we could potentially roll back an upgrade at any time. While I don’t think we need to retain the ability to downgrade in perpetuity it would be a good objective to maintain strict backward compatibility and therefore downgradability until a certain point. This would imply versioning metadata and extending it in such a way that prior version(s) could continue functioning. This can certainly be expensive to implement and might bloat on-disk storage. However, we could always offer an option for the operator to optimize the on-disk structures for the current version then we can rewrite them in the latest version. This optimizes the storage and opens up new functionality. This means new features that can work with old on-disk structures will be available while others that strictly require new versions of the data structures will be unavailable until the operator migrates to the new version. This migration IMO should be irreversible. Beyond this point the operator will lose the ability to downgrade which is ok.
>>>>>>>>>> 
>>>>>>>>>> Dinesh
>>>>>>>>>> 
>>>>>>>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <jakers@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> There has been progress on 
>>>>>>>>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>>>>>>>> 
>>>>>>>>>>> Which is similar to what datastax does for DSE. Would this be an acceptable solution?
>>>>>>>>>>> 
>>>>>>>>>>> Jake 
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cclive1601@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>>>>> It seems “An alternative solution is to implement/complete CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>” can give us more options if it is finished😉
>>>>>>>>>>>> 
>>>>>>>>>>>> Branimir Lambov <blambov@apache.org <ma...@apache.org>>于2023年2月20日 周一下午11:03写道:
>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There has been a discussion lately about changes to the sstable format in the context of being able to abort a cluster upgrade, and the fact that changes to sstables can prevent downgraded nodes from reading any data written during their temporary operation with the new version.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Most of the discussion is in CASSANDRA-18134 <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is spreading into CASSANDRA-14277 <https://issues.apache.org/jira/browse/CASSANDRA-14227> and CASSANDRA-17698 <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of which is a good place to discuss the topic seriously.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Downgradability is a worthy goal and is listed in the current roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> My understanding of what has been suggested so far translates to:
>>>>>>>>>>>>> - avoid changes to sstable formats;
>>>>>>>>>>>>> - if there are changes, implement them in a way that is backwards-compatible, e.g. by duplicating data, so that a new version is presented in a component or portion of a component that legacy nodes will not try to read;
>>>>>>>>>>>>> - if the latter is not feasible, make sure the changes are only applied if a feature flag has been enabled.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> To me this approach introduces several risks:
>>>>>>>>>>>>> - it bloats file and parsing complexity;
>>>>>>>>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF ticket once this requirement is in place);
>>>>>>>>>>>>> - it needs care to avoid risky solutions to address technical issues with the format versioning (e.g. staying on n-versions for 5.0 and needing a bump for a 4.1 bugfix might require porting over support for new features);
>>>>>>>>>>>>> - it requires separate and uncoordinated solutions to the problem and switching mechanisms for each individual change.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> An alternative solution is to implement/complete CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which provides a method of writing sstables for a target version. During upgrades, a node could be set to produce sstables corresponding to the older version, and there is a very straightforward way to implement modifications to formats like the tickets above to conform to its requirements. 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> What do people think should be the way forward?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Branimir
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> you are the apple of my eye !
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> http://twitter.com/tjake
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Branimir Lambov
>>>> e. branimir.lambov@datastax.com <ma...@datastax.com>
>>>> w. www.datastax.com <http://www.datastax.com/>
>>>> 
>> 
>> 


Re: Downgradability

Posted by Benedict <be...@apache.org>.
As always, Scott puts it much more eloquently than I can.

  

The only thing I’d quibble with is that I think it is better to make changes
backwards compatible, rather than make earlier releases forwards compatible -
and where this is prohibitively costly to simply make a feature that depends
on it unavailable until the switch to the new major format.

  

This provides the greatest flexibility for users, as they can upgrade from and
downgrade to the same versions. There’s no scrambling for a different
downgrade target you haven’t qualified when finding out there’s an
unacceptable bug.

There’s also less delta between pre-upgrade and post-downgrade behaviour.

  

We have plenty of practice doing this kind of thing. It’s not that hard.

  

But, if we want to go the forward compatibility route that’s still far better
than nothing.

  

  

> On 21 Feb 2023, at 16:17, C. Scott Andreas <sc...@paradoxica.net> wrote:  
>  
>

> 
>
> I realize my feedback on this has been spread across tickets and older
> mailing list / wiki discussions, so I'll offer a proposal here.  
>
>
>  
>
>
> Starting with goals -  
>
>
>  
>
>
> 1\. Cassandra users must be able to abort and revert an upgrade to a new
> version of the database that introduces a new major SSTable format.  
>
>
>  
>
>
> This reduces risk of upgrading to a build that also introduces a non-data-
> format-related bug that is intolerable. This goal does not specify a
> mechanism or downgrade target - just the "downgradability" goal.  
>
>
>  
>
>
> 2\. Where possible, Cassandra users should be able to opt into writing of a
> new major SSTable format.  
>
>
>  
>
>
> This reduces that risk further by allowing users to decouple data format
> changes from the upgrade itself. There may be cases where new features or
> bug fixes prevent this from being possible, but I'll offer it as a goal.  
>
>
>  
>
>
> 3\. It should be possible for users to perform the downgrade in-place by
> launching the database using a previous version's binary.  
>
>
>  
>
>
> This avoids the need for complex orchestration of offline commands like a
> hypothetical `downgradesstables`.  
>
>
>  
>
>
>  
>
>
> The following approach would allow us to accomplish these goals:  
>
>
>  
>
>
> 1\. Major SSTable changes should begin with forward-compatibility in a prior
> release.  
>
>
>  
>
>
> In a release prior to one that revs major SSTable versions, we should
> implement the ability to read the SSTables that we intend to write in the
> next major version. This would allow someone to (eg.,) revert from 5.0 to
> 4.2 if they encountered a regression that caused an outage without data
> loss. This downgrade path should be well-specified and called out in
> NEWS.txt.  
>
>
>  
>
>
> 2\. Where possible, major SSTable format changes should be opt-in (if the
> features / bugfixes introduced allow).  
>
>
>  
>
>
> This would be via a flag to enable writing the new format once an operator
> has determined that post-upgrade their clusters are sufficiently stable.
> This is an approach that HDFS has adopted. Following a rolling upgrade of
> HDFS, downgrade remains possible until an operator executes a "finalize"
> operation to migrate NameNode metadata to the new version's. An approach
> like this would allow users to perform a staged upgrade in which they first
> rev the version of the database, followed by opting into its new format to
> derisk (eg.,) adoption of BTI-indexed SSTables.  
>
>
>  
>
>
> These approaches aren't meant to discourage SSTable format evolution - but
> to make it safer, and ideally faster. They don't specify duplicative
> serialization or a game of Twister to hide fields in locations where old
> versions don't think to look. Forward compatibility in a prior release could
> be landed at the same time as the major format revision itself, so long as
> we cut releases from both branches.  
>
>
>  
>
>
> Ability to back out an upgrade until finalized would dramatically lower the
> risk of adopting new releases of Apache Cassandra. For many users, the
> qualification cycle for a new release is more than a year - and a *lot* of
> work.  
>
>
>  
>
>
> Reducing the risk of upgrading to new releases repositions Cassandra as a
> database that can be treated with greater trust -- especially for multi-
> petabyte, mission critical systems. Our user community will advance to newer
> releases more quickly and we'll be able to shorten the maintenance cycles
> for older releases. In the same way that CI stability enables us to move
> faster and more confidently in the project, safety features like this will
> enable our users (and indeed ourselves) to move more confidently to adopt
> them.  
>
>
>  
>
>
> – Scott  
>
>
>  
>
>
>  
>
>

>> On Feb 21, 2023, at 4:51 AM, "Claude Warren, Jr via dev"
<de...@cassandra.apache.org> wrote:  
>
>>

>>  
>
>>

>>  
>
>>

>> My goal in implementing
[CASSANDRA-8928](https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928)
was to be able to take the current version 4.x and write it as the earliest
3.x version possible.  The reasoning being that if that was possible then
whatever 3.x version was executed would be able to automatically read the
early 3.x version.  My thought was that each release version would have the
ability to downgrade to the earliest previous version.  In this way if users
need to they could string together a number of downgrader versions to move
from 5.x to 3.x.  
>
>>

>>  
>
>>

>> My testing has been pretty straightforward, I created 4 docker containers
using the standard published Cassandra docker images for 3.1 and 4.0 with data
mounted on an external drive.  two of the containers (one of each version) did
not automatically start Cassandra.  My process was then:  
>
>>

>>   1. start and stop Cassandra 4.0 to create the default data files  
>
>>   2. start the Cassandra 4.0 container that does not automatically run
Cassandra and execute the new downgrade functionality.  
>
>>   3. start the Cassandra 3.1 container and dump the logs.  If the system
started then I knew that I at least had a proof of concept.  So far no-go.  
>
>>

>>

>>  
>
>>

>>  
>
>>

>> On Tue, Feb 21, 2023 at 8:57 AM Branimir Lambov
<[branimir.lambov@datastax.com](mailto:branimir.lambov@datastax.com)> wrote:  
>
>>

>>> It appears to me that the first thing we need to start this feature off is
a definition of a suite of tests together with a set of rules to keep the
suite up to date with new features as they are introduced. The moment that
suite is in place, we can start having some confidence that we can enforce
downgradability.  
>
>>>

>>>  
>
>>>

>>> Something like this will definitely catch incompatibilities in SSTable
formats (such as the one in CASSANDRA-17698 that I managed to miss during
review), but will also be able to identify incompatible system schema changes
among others, and at the same time rightfully ignore non-breaking changes such
as modifications to the key cache serialization formats.  
>
>>>

>>>  
>
>>>

>>> Is downgradability in scope for 5.0? It is a feature like any other, and I
don't see any difficulty adding it (with support for downgrade to 4.x) a
little later in the 5.x timeline.  
>
>>>

>>>  
>
>>>

>>> Regards,  
>
>>>

>>> Branimir  
>
>>>

>>>  
>
>>>

>>>  
>
>>>

>>> On Tue, Feb 21, 2023 at 9:40 AM Jacek Lewandowski
<[lewandowski.jacek@gmail.com](mailto:lewandowski.jacek@gmail.com)> wrote:  
>
>>>

>>>> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
multiple sstable formats support. It allows for providing an implementation of
SSTableFormat along with SSTableReader and SSTableWriter. That could be
extended easily to support different implementations for certain version
ranges, like one impl for ma-nz, other for oa+, etc. without having a
confusing implementation with a lot of conditional blocks. Old formats in such
case could be maintained separately from the main code and easily switched any
time.  
>
>>>>

>>>>  
>
>>>>

>>>> thanks  
> \- - -- --- ----- -------- -------------  
> Jacek Lewandowski
>>>>

>>>>  
>
>>>>

>>>>  
>
>>>>

>>>> wt., 21 lut 2023 o 01:46 Yuki Morishita
<[yukim@apache.org](mailto:yukim@apache.org)> napisał(a):  
>
>>>>

>>>>> Hi,  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> What I wanted to address in my comment in
CASSANDRA-8110([https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-
tabpanel#comment-17641705](https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-
tabpanel#comment-17641705)) is to focus on better upgrade experience.  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> Upgrading the cluster can be painful for some orgs with mission critical
Cassandra cluster, where they cannot tolerate less availability because of the
inability to replace the downed node.  
>
>>>>>

>>>>> They also need to plan rolling back to the previous state when something
happens along the way.  
>
>>>>>

>>>>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at
least enabling SSTable streaming during the upgrade by not upgrading the
SSTable version. This can make the cluster to easily rollback to the previous
version.  
>
>>>>>

>>>>> Downgrading SSTable is not the primary focus (though Cassandra needs to
implement the way to write SSTable in older versions, so it is somewhat
related.)  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> I'm preparing the design doc for the change.  
>
>>>>>

>>>>> Also, if I should create a separate ticket from CASSANDRA-8110 for the
clarity of the goal of the change, please let me know.  
>
>>>>>

>>>>>  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> On Tue, Feb 21, 2023 at 5:31 AM Benedict
<[benedict@apache.org](mailto:benedict@apache.org)> wrote:  
>
>>>>>

>>>>>>  
>
>>>>>>

>>>>>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
will have to eventually also tackle system schema changes (probably not hard),
and may have to think a little carefully about other things, eg with TTLs the
format change is only the contract about what values can be present, so we
have to make sure the data validity checks are consistent with the format we
write. It isn’t as simple as writing an earlier version in this case (unless
we permit truncating the TTL, perhaps)  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> On 20 Feb 2023, at 20:24, Benedict
<[benedict@apache.org](mailto:benedict@apache.org)> wrote:  
>
>>>>>>

>>>>>>>  
>
>>>>>>

>>>>>>>   
>
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> In a self-organising community, things that aren’t self-policed
naturally end up policed in an adhoc manner, and with difficulty. I’m not sure
that’s the same as arbitrary enforcement. It seems to me the real issue is
nobody noticed this was agreed and/or forgot and didn’t think about it much.  
>
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> But, even without any prior agreement, it’s perfectly reasonable to
request that things do not break compatibility if they do not need to, as part
of the normal patch integration process.  
>
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
agreement to do this. But we can and should address the problem of new columns
in schema tables, as this happens often between versions. I’m not sure it has
in 4.1 though?  
>
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> Regarding downgrade versions, surely this should simply be the same as
upgrade versions we support?  
>
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>>> On 20 Feb 2023, at 20:02, Jeff Jirsa
<[jjirsa@gmail.com](mailto:jjirsa@gmail.com)> wrote:  
>
>>>>>>>

>>>>>>>>   
>
>>>>>>>>

>>>>>>>> I'm not even convinced even 8110 addresses this - just writing
sstables in old versions won't help if we ever add things like new types or
new types of collections without other control abilities. Claude's other email
in another thread a few hours ago talks about some of these surprises -
"Specifically during the 3.1 -> 4.0 changes a column broadcast_port was added
to system/local.  This means that 3.1 system can not read the table as it has
no definition for it.  I tried marking the column for deletion in the metadata
and in the serialization header.  The later got past the column not found
problem, but I suspect that it just means that data columns after
broadcast_port shifted and so incorrectly read." - this is a harder problem to
solve than just versioning sstables and network protocols.  
>
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> Stepping back a bit, we have downgrade ability listed as a goal, but
it's not (as far as I can tell) universally enforced, nor is it clear at which
point we will be able to concretely say "this release can be downgraded to X".
Until we actually define and agree that this is a real goal with a concrete
version where downgrade-ability becomes real, it feels like things are
somewhat arbitrarily enforced, which is probably very frustrating for people
trying to commit work/tickets.  
>
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> \- Jeff  
>
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi
<[djoshi@apache.org](mailto:djoshi@apache.org)> wrote:  
>
>>>>>>>>

>>>>>>>>> I’m a big fan of maintaining backward compatibility. Downgradability
implies that we could potentially roll back an upgrade at any time. While I
don’t think we need to retain the ability to downgrade in perpetuity it would
be a good objective to maintain strict backward compatibility and therefore
downgradability until a certain point. This would imply versioning metadata
and extending it in such a way that prior version(s) could continue
functioning. This can certainly be expensive to implement and might bloat on-
disk storage. However, we could always offer an option for the operator to
optimize the on-disk structures for the current version then we can rewrite
them in the latest version. This optimizes the storage and opens up new
functionality. This means new features that can work with old on-disk
structures will be available while others that strictly require new versions
of the data structures will be unavailable until the operator migrates to the
new version. This migration IMO should be irreversible. Beyond this point the
operator will lose the ability to downgrade which is ok.  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Dinesh  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani
<[jakers@gmail.com](mailto:jakers@gmail.com)> wrote:  
>
>>>>>>>>>

>>>>>>>>>>   
>
>>>>>>>>>>

>>>>>>>>>> There has been progress on  
>
>>>>>>>>>>

>>>>>>>>>>
<https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928>  
>
>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> Which is similar to what datastax does for DSE. Would this be an
acceptable solution?  
>
>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> Jake  
>
>>>>>>>>>>

>>>>>>>>>>  
>
>>>>>>>>>>

>>>>>>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell
<[cclive1601@gmail.com](mailto:cclive1601@gmail.com)> wrote:  
>
>>>>>>>>>>

>>>>>>>>>>> It seems “An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110)” can
give us more options if it is finished😉  
>
>>>>>>>>>>>

>>>>>>>>>>>  
>
>>>>>>>>>>>

>>>>>>>>>>> Branimir Lambov
<[blambov@apache.org](mailto:blambov@apache.org)>于2023年2月20日 周一下午11:03写道:  
>
>>>>>>>>>>>

>>>>>>>>>>>> Hi everyone,  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> There has been a discussion lately about changes to the sstable
format in the context of being able to abort a cluster upgrade, and the fact
that changes to sstables can prevent downgraded nodes from reading any data
written during their temporary operation with the new version.  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> Most of the discussion is in
[CASSANDRA-18134](https://issues.apache.org/jira/browse/CASSANDRA-18134), and
is spreading into
[CASSANDRA-14277](https://issues.apache.org/jira/browse/CASSANDRA-14227) and
[CASSANDRA-17698](https://issues.apache.org/jira/browse/CASSANDRA-17698), none
of which is a good place to discuss the topic seriously.  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> Downgradability is a worthy goal and is listed in the current
roadmap. I would like to open a discussion here on how it would be achieved.  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> My understanding of what has been suggested so far translates to:  
>
>>>>>>>>>>>>

>>>>>>>>>>>> \- avoid changes to sstable formats;  
>
>>>>>>>>>>>>

>>>>>>>>>>>> \- if there are changes, implement them in a way that is
backwards-compatible, e.g. by duplicating data, so that a new version is
presented in a component or portion of a component that legacy nodes will not
try to read;  
>
>>>>>>>>>>>>

>>>>>>>>>>>> \- if the latter is not feasible, make sure the changes are only
applied if a feature flag has been enabled.  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> To me this approach introduces several risks:  
>
>>>>>>>>>>>>

>>>>>>>>>>>> \- it bloats file and parsing complexity;  
>
>>>>>>>>>>>>

>>>>>>>>>>>> \- it discourages improvement (e.g. CASSANDRA-17698 is no longer
a LHF ticket once this requirement is in place);  
>
>>>>>>>>>>>>

>>>>>>>>>>>> \- it needs care to avoid risky solutions to address technical
issues with the format versioning (e.g. staying on n-versions for 5.0 and
needing a bump for a 4.1 bugfix might require porting over support for new
features);  
>
>>>>>>>>>>>>

>>>>>>>>>>>> \- it requires separate and uncoordinated solutions to the
problem and switching mechanisms for each individual change.  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110), which
provides a method of writing sstables for a target version. During upgrades, a
node could be set to produce sstables corresponding to the older version, and
there is a very straightforward way to implement modifications to formats like
the tickets above to conform to its requirements.  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> What do people think should be the way forward?  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>> Regards,  
>
>>>>>>>>>>>>

>>>>>>>>>>>> Branimir  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>>

>>>>>>>>>>>>  
>
>>>>>>>>>>>

>>>>>>>>>>> \--  
>
>>>>>>>>>>>

>>>>>>>>>>> you are the apple of my eye !  
>
>>>>>>>>>>

>>>>>>>>>> \--  
>
>>>>>>>>>>

>>>>>>>>>> <http://twitter.com/tjake>  
>
>>>

>>>  
>
>>>

>>>  
>
>>>

>>> \--  
>
>>>

>>> Branimir Lambov  
>
>>>

>>> e. [branimir.lambov@datastax.com](mailto:branimir.lambov@datastax.com)  
>
>>>

>>> w. [www.datastax.com](http://www.datastax.com)  
>
>>>

>>>  
>
>
>  
>


Re: Downgradability

Posted by "C. Scott Andreas" <sc...@paradoxica.net>.
I realize my feedback on this has been spread across tickets and older mailing list / wiki discussions, so I'll offer a proposal here.Starting with goals -1. Cassandra users must be able to abort and revert an upgrade to a new version of the database that introduces a new major SSTable format.This reduces risk of upgrading to a build that also introduces a non-data-format-related bug that is intolerable. This goal does not specify a mechanism or downgrade target - just the "downgradability" goal.2. Where possible, Cassandra users should be able to opt into writing of a new major SSTable format.This reduces that risk further by allowing users to decouple data format changes from the upgrade itself. There may be cases where new features or bug fixes prevent this from being possible, but I'll offer it as a goal.3. It should be possible for users to perform the downgrade in-place by launching the database using a previous version's binary.This avoids the need for complex orchestration of offline commands like a hypothetical `downgradesstables`.The following approach would allow us to accomplish these goals:1. Major SSTable changes should begin with forward-compatibility in a prior release.In a release prior to one that revs major SSTable versions, we should implement the ability to read the SSTables that we intend to write in the next major version. This would allow someone to (eg.,) revert from 5.0 to 4.2 if they encountered a regression that caused an outage without data loss. This downgrade path should be well-specified and called out in NEWS.txt.2. Where possible, major SSTable format changes should be opt-in (if the features / bugfixes introduced allow).This would be via a flag to enable writing the new format once an operator has determined that post-upgrade their clusters are sufficiently stable. This is an approach that HDFS has adopted. Following a rolling upgrade of HDFS, downgrade remains possible until an operator executes a "finalize" operation to migrate NameNode metadata to the new version's. An approach like this would allow users to perform a staged upgrade in which they first rev the version of the database, followed by opting into its new format to derisk (eg.,) adoption of BTI-indexed SSTables.These approaches aren't meant to discourage SSTable format evolution - but to make it safer, and ideally faster. They don't specify duplicative serialization or a game of Twister to hide fields in locations where old versions don't think to look. Forward compatibility in a prior release could be landed at the same time as the major format revision itself, so long as we cut releases from both branches.Ability to back out an upgrade until finalized would dramatically lower the risk of adopting new releases of Apache Cassandra. For many users, the qualification cycle for a new release is more than a year - and a *lot* of work.Reducing the risk of upgrading to new releases repositions Cassandra as a database that can be treated with greater trust -- especially for multi-petabyte, mission critical systems. Our user community will advance to newer releases more quickly and we'll be able to shorten the maintenance cycles for older releases. In the same way that CI stability enables us to move faster and more confidently in the project, safety features like this will enable our users (and indeed ourselves) to move more confidently to adopt them.– ScottOn Feb 21, 2023, at 4:51 AM, "Claude Warren, Jr via dev" <de...@cassandra.apache.org> wrote:My goal in implementing CASSANDRA-8928 was to be able to take the current version 4.x and write it as the earliest 3.x version possible.  The reasoning being that if that was possible then whatever 3.x version was executed would be able to automatically read the early 3.x version.  My thought was that each release version would have the ability to downgrade to the earliest previous version.  In this way if users need to they could string together a number of downgrader versions to move from 5.x to 3.x. My testing has been pretty straightforward, I created 4 docker containers using the standard published Cassandra docker images for 3.1 and 4.0 with data mounted on an external drive.  two of the containers (one of each version) did not automatically start Cassandra.  My process was then:start and stop Cassandra 4.0 to create the default data filesstart the Cassandra 4.0 container that does not automatically run Cassandra and execute the new downgrade functionality.start the Cassandra 3.1 container and dump the logs.  If the system started then I knew that I at least had a proof of concept.  So far no-go.On Tue, Feb 21, 2023 at 8:57 AM Branimir Lambov <br...@datastax.com> wrote:It appears to me that the first thing we need to start this feature off is a definition of a suite of tests together with a set of rules to keep the suite up to date with new features as they are introduced. The moment that suite is in place, we can start having some confidence that we can enforce downgradability.Something like this will definitely catch incompatibilities in SSTable formats (such as the one in CASSANDRA-17698 that I managed to miss during review), but will also be able to identify incompatible system schema changes among others, and at the same time rightfully ignore non-breaking changes such as modifications to the key cache serialization formats.Is downgradability in scope for 5.0? It is a feature like any other, and I don't see any difficulty adding it (with support for downgrade to 4.x) a little later in the 5.x timeline.Regards,BranimirOn Tue, Feb 21, 2023 at 9:40 AM Jacek Lewandowski <le...@gmail.com> wrote:I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce multiple sstable formats support. It allows for providing an implementation of SSTableFormat along with SSTableReader and SSTableWriter. That could be extended easily to support different implementations for certain version ranges, like one impl for ma-nz, other for oa+, etc. without having a confusing implementation with a lot of conditional blocks. Old formats in such case could be maintained separately from the main code and easily switched any time. thanks- - -- --- ----- -------- -------------Jacek Lewandowskiwt., 21 lut 2023 o 01:46 Yuki Morishita <yu...@apache.org> napisał(a):Hi,What I wanted to address in my comment in CASSANDRA-8110(https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705) is to focus on better upgrade experience.Upgrading the cluster can be painful for some orgs with mission critical Cassandra cluster, where they cannot tolerate less availability because of the inability to replace the downed node.They also need to plan rolling back to the previous state when something happens along the way.The change I proposed in CASSANDRA-8110 is to achieve the goal of at least enabling SSTable streaming during the upgrade by not upgrading the SSTable version. This can make the cluster to easily rollback to the previous version.Downgrading SSTable is not the primary focus (though Cassandra needs to implement the way to write SSTable in older versions, so it is somewhat related.)I'm preparing the design doc for the change.Also, if I should create a separate ticket from CASSANDRA-8110 for the clarity of the goal of the change, please let me know.On Tue, Feb 21, 2023 at 5:31 AM Benedict <be...@apache.org> wrote:FWIW I think 8110 is the right approach, even if it isn’t a panacea. We will have to eventually also tackle system schema changes (probably not hard), and may have to think a little carefully about other things, eg with TTLs the format change is only the contract about what values can be present, so we have to make sure the data validity checks are consistent with the format we write. It isn’t as simple as writing an earlier version in this case (unless we permit truncating the TTL, perhaps) On 20 Feb 2023, at 20:24, Benedict <be...@apache.org> wrote:In a self-organising community, things that aren’t self-policed naturally end up policed in an adhoc manner, and with difficulty. I’m not sure that’s the same as arbitrary enforcement. It seems to me the real issue is nobody noticed this was agreed and/or forgot and didn’t think about it much. But, even without any prior agreement, it’s perfectly reasonable to request that things do not break compatibility if they do not need to, as part of the normal patch integration process.Issues with 3.1->4.0 aren’t particularly relevant as they predate any agreement to do this. But we can and should address the problem of new columns in schema tables, as this happens often between versions. I’m not sure it has in 4.1 though?Regarding downgrade versions, surely this should simply be the same as upgrade versions we support?On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:I'm not even convinced even 8110 addresses this - just writing sstables in old versions won't help if we ever add things like new types or new types of collections without other control abilities. Claude's other email in another thread a few hours ago talks about some of these surprises - "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was added to system/local.  This means that 3.1 system can not read the table as it has no definition for it.  I tried marking the column for deletion in the metadata and in the serialization header.  The later got past the column not found problem, but I suspect that it just means that data columns after broadcast_port shifted and so incorrectly read." - this is a harder problem to solve than just versioning sstables and network protocols. Stepping back a bit, we have downgrade ability listed as a goal, but it's not (as far as I can tell) universally enforced, nor is it clear at which point we will be able to concretely say "this release can be downgraded to X".   Until we actually define and agree that this is a real goal with a concrete version where downgrade-ability becomes real, it feels like things are somewhat arbitrarily enforced, which is probably very frustrating for people trying to commit work/tickets.- JeffOn Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:I’m a big fan of maintaining backward compatibility. Downgradability implies that we could potentially roll back an upgrade at any time. While I don’t think we need to retain the ability to downgrade in perpetuity it would be a good objective to maintain strict backward compatibility and therefore downgradability until a certain point. This would imply versioning metadata and extending it in such a way that prior version(s) could continue functioning. This can certainly be expensive to implement and might bloat on-disk storage. However, we could always offer an option for the operator to optimize the on-disk structures for the current version then we can rewrite them in the latest version. This optimizes the storage and opens up new functionality. This means new features that can work with old on-disk structures will be available while others that strictly require new versions of the data structures will be unavailable until the operator migrates to the new version. This migration IMO should be irreversible. Beyond this point the operator will lose the ability to downgrade which is ok.DineshOn Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:There has been progress on https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928Which is similar to what datastax does for DSE. Would this be an acceptable solution?Jake On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com> wrote:It seems “An alternative solution is to implement/complete CASSANDRA-8110” can give us more options if it is finished😉Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:Hi everyone,There has been a discussion lately about changes to the sstable format in the context of being able to abort a cluster upgrade, and the fact that changes to sstables can prevent downgraded nodes from reading any data written during their temporary operation with the new version.Most of the discussion is in CASSANDRA-18134, and is spreading into CASSANDRA-14277 and CASSANDRA-17698, none of which is a good place to discuss the topic seriously.Downgradability is a worthy goal and is listed in the current roadmap. I would like to open a discussion here on how it would be achieved.My understanding of what has been suggested so far translates to:- avoid changes to sstable formats;- if there are changes, implement them in a way that is backwards-compatible, e.g. by duplicating data, so that a new version is presented in a component or portion of a component that legacy nodes will not try to read;- if the latter is not feasible, make sure the changes are only applied if a feature flag has been enabled.To me this approach introduces several risks:- it bloats file and parsing complexity;- it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF ticket once this requirement is in place);- it needs care to avoid risky solutions to address technical issues with the format versioning (e.g. staying on n-versions for 5.0 and needing a bump for a 4.1 bugfix might require porting over support for new features);- it requires separate and uncoordinated solutions to the problem and switching mechanisms for each individual change.An alternative solution is to implement/complete CASSANDRA-8110, which provides a method of writing sstables for a target version. During upgrades, a node could be set to produce sstables corresponding to the older version, and there is a very straightforward way to implement modifications to formats like the tickets above to conform to its requirements. What do people think should be the way forward?Regards,Branimir-- you are the apple of my eye !-- http://twitter.com/tjake-- Branimir Lambove. branimir.lambov@datastax.com w. www.datastax.com

Re: Downgradability

Posted by "Claude Warren, Jr via dev" <de...@cassandra.apache.org>.
My goal in implementing CASSANDRA-8928
<https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928>
was
to be able to take the current version 4.x and write it as the earliest 3.x
version possible.  The reasoning being that if that was possible then
whatever 3.x version was executed would be able to automatically read the
early 3.x version.  My thought was that each release version would have the
ability to downgrade to the earliest previous version.  In this way if
users need to they could string together a number of downgrader versions to
move from 5.x to 3.x.

My testing has been pretty straightforward, I created 4 docker containers
using the standard published Cassandra docker images for 3.1 and 4.0 with
data mounted on an external drive.  two of the containers (one of each
version) did not automatically start Cassandra.  My process was then:

   1. start and stop Cassandra 4.0 to create the default data files
   2. start the Cassandra 4.0 container that does not automatically run
   Cassandra and execute the new downgrade functionality.
   3. start the Cassandra 3.1 container and dump the logs.  If the system
   started then I knew that I at least had a proof of concept.  So far no-go.



On Tue, Feb 21, 2023 at 8:57 AM Branimir Lambov <
branimir.lambov@datastax.com> wrote:

> It appears to me that the first thing we need to start this feature off is
> a definition of a suite of tests together with a set of rules to keep the
> suite up to date with new features as they are introduced. The moment that
> suite is in place, we can start having some confidence that we can enforce
> downgradability.
>
> Something like this will definitely catch incompatibilities in SSTable
> formats (such as the one in CASSANDRA-17698 that I managed to miss during
> review), but will also be able to identify incompatible system schema
> changes among others, and at the same time rightfully ignore non-breaking
> changes such as modifications to the key cache serialization formats.
>
> Is downgradability in scope for 5.0? It is a feature like any other, and I
> don't see any difficulty adding it (with support for downgrade to 4.x) a
> little later in the 5.x timeline.
>
> Regards,
> Branimir
>
> On Tue, Feb 21, 2023 at 9:40 AM Jacek Lewandowski <
> lewandowski.jacek@gmail.com> wrote:
>
>> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
>> multiple sstable formats support. It allows for providing an implementation
>> of SSTableFormat along with SSTableReader and SSTableWriter. That could be
>> extended easily to support different implementations for certain version
>> ranges, like one impl for ma-nz, other for oa+, etc. without having a
>> confusing implementation with a lot of conditional blocks. Old formats in
>> such case could be maintained separately from the main code and easily
>> switched any time.
>>
>> thanks
>> - - -- --- ----- -------- -------------
>> Jacek Lewandowski
>>
>>
>> wt., 21 lut 2023 o 01:46 Yuki Morishita <yu...@apache.org> napisał(a):
>>
>>> Hi,
>>>
>>> What I wanted to address in my comment in CASSANDRA-8110(
>>> https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705)
>>> is to focus on better upgrade experience.
>>>
>>> Upgrading the cluster can be painful for some orgs with mission critical
>>> Cassandra cluster, where they cannot tolerate less availability because of
>>> the inability to replace the downed node.
>>> They also need to plan rolling back to the previous state when something
>>> happens along the way.
>>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at
>>> least enabling SSTable streaming during the upgrade by not upgrading the
>>> SSTable version. This can make the cluster to easily rollback to the
>>> previous version.
>>> Downgrading SSTable is not the primary focus (though Cassandra needs to
>>> implement the way to write SSTable in older versions, so it is somewhat
>>> related.)
>>>
>>> I'm preparing the design doc for the change.
>>> Also, if I should create a separate ticket from CASSANDRA-8110 for the
>>> clarity of the goal of the change, please let me know.
>>>
>>>
>>> On Tue, Feb 21, 2023 at 5:31 AM Benedict <be...@apache.org> wrote:
>>>
>>>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
>>>> will have to eventually also tackle system schema changes (probably not
>>>> hard), and may have to think a little carefully about other things, eg with
>>>> TTLs the format change is only the contract about what values can be
>>>> present, so we have to make sure the data validity checks are consistent
>>>> with the format we write. It isn’t as simple as writing an earlier version
>>>> in this case (unless we permit truncating the TTL, perhaps)
>>>>
>>>> On 20 Feb 2023, at 20:24, Benedict <be...@apache.org> wrote:
>>>>
>>>>
>>>> 
>>>> In a self-organising community, things that aren’t self-policed
>>>> naturally end up policed in an adhoc manner, and with difficulty. I’m not
>>>> sure that’s the same as arbitrary enforcement. It seems to me the real
>>>> issue is nobody noticed this was agreed and/or forgot and didn’t think
>>>> about it much.
>>>>
>>>> But, even without any prior agreement, it’s perfectly reasonable to
>>>> request that things do not break compatibility if they do not need to, as
>>>> part of the normal patch integration process.
>>>>
>>>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
>>>> agreement to do this. But we can and should address the problem of new
>>>> columns in schema tables, as this happens often between versions. I’m not
>>>> sure it has in 4.1 though?
>>>>
>>>> Regarding downgrade versions, surely this should simply be the same as
>>>> upgrade versions we support?
>>>>
>>>>
>>>> On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:
>>>>
>>>> 
>>>> I'm not even convinced even 8110 addresses this - just writing sstables
>>>> in old versions won't help if we ever add things like new types or new
>>>> types of collections without other control abilities. Claude's other email
>>>> in another thread a few hours ago talks about some of these surprises -
>>>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>>> added to system/local.  This means that 3.1 system can not read the table
>>>> as it has no definition for it.  I tried marking the column for deletion in
>>>> the metadata and in the serialization header.  The later got past the
>>>> column not found problem, but I suspect that it just means that data
>>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>>> harder problem to solve than just versioning sstables and network
>>>> protocols.
>>>>
>>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>>> which point we will be able to concretely say "this release can be
>>>> downgraded to X".   Until we actually define and agree that this is a
>>>> real goal with a concrete version where downgrade-ability becomes real, it
>>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>>> frustrating for people trying to commit work/tickets.
>>>>
>>>> - Jeff
>>>>
>>>>
>>>>
>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org>
>>>> wrote:
>>>>
>>>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>>>> implies that we could potentially roll back an upgrade at any time. While I
>>>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>>>> would be a good objective to maintain strict backward compatibility and
>>>>> therefore downgradability until a certain point. This would imply
>>>>> versioning metadata and extending it in such a way that prior version(s)
>>>>> could continue functioning. This can certainly be expensive to implement
>>>>> and might bloat on-disk storage. However, we could always offer an option
>>>>> for the operator to optimize the on-disk structures for the current version
>>>>> then we can rewrite them in the latest version. This optimizes the storage
>>>>> and opens up new functionality. This means new features that can work with
>>>>> old on-disk structures will be available while others that strictly require
>>>>> new versions of the data structures will be unavailable until the operator
>>>>> migrates to the new version. This migration IMO should be irreversible.
>>>>> Beyond this point the operator will lose the ability to downgrade which is
>>>>> ok.
>>>>>
>>>>> Dinesh
>>>>>
>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>>>
>>>>> 
>>>>> There has been progress on
>>>>>
>>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>>
>>>>> Which is similar to what datastax does for DSE. Would this be an
>>>>> acceptable solution?
>>>>>
>>>>> Jake
>>>>>
>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> It seems “An alternative solution is to implement/complete
>>>>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>>>>> can give us more options if it is finished😉
>>>>>>
>>>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> There has been a discussion lately about changes to the sstable
>>>>>>> format in the context of being able to abort a cluster upgrade, and the
>>>>>>> fact that changes to sstables can prevent downgraded nodes from reading any
>>>>>>> data written during their temporary operation with the new version.
>>>>>>>
>>>>>>> Most of the discussion is in CASSANDRA-18134
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>>>> spreading into CASSANDRA-14277
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>>>> CASSANDRA-17698
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>>>> which is a good place to discuss the topic seriously.
>>>>>>>
>>>>>>> Downgradability is a worthy goal and is listed in the current
>>>>>>> roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>>
>>>>>>> My understanding of what has been suggested so far translates to:
>>>>>>> - avoid changes to sstable formats;
>>>>>>> - if there are changes, implement them in a way that is
>>>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>>>> presented in a component or portion of a component that legacy nodes will
>>>>>>> not try to read;
>>>>>>> - if the latter is not feasible, make sure the changes are only
>>>>>>> applied if a feature flag has been enabled.
>>>>>>>
>>>>>>> To me this approach introduces several risks:
>>>>>>> - it bloats file and parsing complexity;
>>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a
>>>>>>> LHF ticket once this requirement is in place);
>>>>>>> - it needs care to avoid risky solutions to address technical issues
>>>>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>>>>> features);
>>>>>>> - it requires separate and uncoordinated solutions to the problem
>>>>>>> and switching mechanisms for each individual change.
>>>>>>>
>>>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>>>> provides a method of writing sstables for a target version. During
>>>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>>>> older version, and there is a very straightforward way to implement
>>>>>>> modifications to formats like the tickets above to conform to its
>>>>>>> requirements.
>>>>>>>
>>>>>>> What do people think should be the way forward?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Branimir
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>> you are the apple of my eye !
>>>>>>
>>>>> --
>>>>> http://twitter.com/tjake
>>>>>
>>>>>
>
> --
> Branimir Lambov
> e. branimir.lambov@datastax.com
> w. www.datastax.com
>
>

Re: Downgradability

Posted by Branimir Lambov <br...@datastax.com>.
It appears to me that the first thing we need to start this feature off is
a definition of a suite of tests together with a set of rules to keep the
suite up to date with new features as they are introduced. The moment that
suite is in place, we can start having some confidence that we can enforce
downgradability.

Something like this will definitely catch incompatibilities in SSTable
formats (such as the one in CASSANDRA-17698 that I managed to miss during
review), but will also be able to identify incompatible system schema
changes among others, and at the same time rightfully ignore non-breaking
changes such as modifications to the key cache serialization formats.

Is downgradability in scope for 5.0? It is a feature like any other, and I
don't see any difficulty adding it (with support for downgrade to 4.x) a
little later in the 5.x timeline.

Regards,
Branimir

On Tue, Feb 21, 2023 at 9:40 AM Jacek Lewandowski <
lewandowski.jacek@gmail.com> wrote:

> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
> multiple sstable formats support. It allows for providing an implementation
> of SSTableFormat along with SSTableReader and SSTableWriter. That could be
> extended easily to support different implementations for certain version
> ranges, like one impl for ma-nz, other for oa+, etc. without having a
> confusing implementation with a lot of conditional blocks. Old formats in
> such case could be maintained separately from the main code and easily
> switched any time.
>
> thanks
> - - -- --- ----- -------- -------------
> Jacek Lewandowski
>
>
> wt., 21 lut 2023 o 01:46 Yuki Morishita <yu...@apache.org> napisał(a):
>
>> Hi,
>>
>> What I wanted to address in my comment in CASSANDRA-8110(
>> https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705)
>> is to focus on better upgrade experience.
>>
>> Upgrading the cluster can be painful for some orgs with mission critical
>> Cassandra cluster, where they cannot tolerate less availability because of
>> the inability to replace the downed node.
>> They also need to plan rolling back to the previous state when something
>> happens along the way.
>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at
>> least enabling SSTable streaming during the upgrade by not upgrading the
>> SSTable version. This can make the cluster to easily rollback to the
>> previous version.
>> Downgrading SSTable is not the primary focus (though Cassandra needs to
>> implement the way to write SSTable in older versions, so it is somewhat
>> related.)
>>
>> I'm preparing the design doc for the change.
>> Also, if I should create a separate ticket from CASSANDRA-8110 for the
>> clarity of the goal of the change, please let me know.
>>
>>
>> On Tue, Feb 21, 2023 at 5:31 AM Benedict <be...@apache.org> wrote:
>>
>>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
>>> will have to eventually also tackle system schema changes (probably not
>>> hard), and may have to think a little carefully about other things, eg with
>>> TTLs the format change is only the contract about what values can be
>>> present, so we have to make sure the data validity checks are consistent
>>> with the format we write. It isn’t as simple as writing an earlier version
>>> in this case (unless we permit truncating the TTL, perhaps)
>>>
>>> On 20 Feb 2023, at 20:24, Benedict <be...@apache.org> wrote:
>>>
>>>
>>> 
>>> In a self-organising community, things that aren’t self-policed
>>> naturally end up policed in an adhoc manner, and with difficulty. I’m not
>>> sure that’s the same as arbitrary enforcement. It seems to me the real
>>> issue is nobody noticed this was agreed and/or forgot and didn’t think
>>> about it much.
>>>
>>> But, even without any prior agreement, it’s perfectly reasonable to
>>> request that things do not break compatibility if they do not need to, as
>>> part of the normal patch integration process.
>>>
>>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
>>> agreement to do this. But we can and should address the problem of new
>>> columns in schema tables, as this happens often between versions. I’m not
>>> sure it has in 4.1 though?
>>>
>>> Regarding downgrade versions, surely this should simply be the same as
>>> upgrade versions we support?
>>>
>>>
>>> On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>> 
>>> I'm not even convinced even 8110 addresses this - just writing sstables
>>> in old versions won't help if we ever add things like new types or new
>>> types of collections without other control abilities. Claude's other email
>>> in another thread a few hours ago talks about some of these surprises -
>>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>> added to system/local.  This means that 3.1 system can not read the table
>>> as it has no definition for it.  I tried marking the column for deletion in
>>> the metadata and in the serialization header.  The later got past the
>>> column not found problem, but I suspect that it just means that data
>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>> harder problem to solve than just versioning sstables and network
>>> protocols.
>>>
>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>> which point we will be able to concretely say "this release can be
>>> downgraded to X".   Until we actually define and agree that this is a
>>> real goal with a concrete version where downgrade-ability becomes real, it
>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>> frustrating for people trying to commit work/tickets.
>>>
>>> - Jeff
>>>
>>>
>>>
>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:
>>>
>>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>>> implies that we could potentially roll back an upgrade at any time. While I
>>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>>> would be a good objective to maintain strict backward compatibility and
>>>> therefore downgradability until a certain point. This would imply
>>>> versioning metadata and extending it in such a way that prior version(s)
>>>> could continue functioning. This can certainly be expensive to implement
>>>> and might bloat on-disk storage. However, we could always offer an option
>>>> for the operator to optimize the on-disk structures for the current version
>>>> then we can rewrite them in the latest version. This optimizes the storage
>>>> and opens up new functionality. This means new features that can work with
>>>> old on-disk structures will be available while others that strictly require
>>>> new versions of the data structures will be unavailable until the operator
>>>> migrates to the new version. This migration IMO should be irreversible.
>>>> Beyond this point the operator will lose the ability to downgrade which is
>>>> ok.
>>>>
>>>> Dinesh
>>>>
>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>>
>>>> 
>>>> There has been progress on
>>>>
>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>
>>>> Which is similar to what datastax does for DSE. Would this be an
>>>> acceptable solution?
>>>>
>>>> Jake
>>>>
>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>>> wrote:
>>>>
>>>>> It seems “An alternative solution is to implement/complete
>>>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>>>> can give us more options if it is finished😉
>>>>>
>>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> There has been a discussion lately about changes to the sstable
>>>>>> format in the context of being able to abort a cluster upgrade, and the
>>>>>> fact that changes to sstables can prevent downgraded nodes from reading any
>>>>>> data written during their temporary operation with the new version.
>>>>>>
>>>>>> Most of the discussion is in CASSANDRA-18134
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>>> spreading into CASSANDRA-14277
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>>> CASSANDRA-17698
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>>> which is a good place to discuss the topic seriously.
>>>>>>
>>>>>> Downgradability is a worthy goal and is listed in the current
>>>>>> roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>
>>>>>> My understanding of what has been suggested so far translates to:
>>>>>> - avoid changes to sstable formats;
>>>>>> - if there are changes, implement them in a way that is
>>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>>> presented in a component or portion of a component that legacy nodes will
>>>>>> not try to read;
>>>>>> - if the latter is not feasible, make sure the changes are only
>>>>>> applied if a feature flag has been enabled.
>>>>>>
>>>>>> To me this approach introduces several risks:
>>>>>> - it bloats file and parsing complexity;
>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>>>>> ticket once this requirement is in place);
>>>>>> - it needs care to avoid risky solutions to address technical issues
>>>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>>>> features);
>>>>>> - it requires separate and uncoordinated solutions to the problem and
>>>>>> switching mechanisms for each individual change.
>>>>>>
>>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>>> provides a method of writing sstables for a target version. During
>>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>>> older version, and there is a very straightforward way to implement
>>>>>> modifications to formats like the tickets above to conform to its
>>>>>> requirements.
>>>>>>
>>>>>> What do people think should be the way forward?
>>>>>>
>>>>>> Regards,
>>>>>> Branimir
>>>>>>
>>>>>>
>>>>>> --
>>>>> you are the apple of my eye !
>>>>>
>>>> --
>>>> http://twitter.com/tjake
>>>>
>>>>

-- 
Branimir Lambov
e. branimir.lambov@datastax.com
w. www.datastax.com

Re: Downgradability

Posted by Benjamin Lerer <bl...@apache.org>.
>
> It seems to me the real issue is nobody noticed this was agreed and/or
> forgot and didn’t think about it much.


I have some trouble keeping up with all the discussions those days so I
might have missed the discussion. The only place I remembered where this
subject was mentioned was during the roadmap discussion over a year ago. My
understanding of that discussion was only that there was a wish to build a
solution for that problem. Are you referring to another discussion I
missed?

Le mar. 21 févr. 2023 à 08:41, Jacek Lewandowski <
lewandowski.jacek@gmail.com> a écrit :

> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
> multiple sstable formats support. It allows for providing an implementation
> of SSTableFormat along with SSTableReader and SSTableWriter. That could be
> extended easily to support different implementations for certain version
> ranges, like one impl for ma-nz, other for oa+, etc. without having a
> confusing implementation with a lot of conditional blocks. Old formats in
> such case could be maintained separately from the main code and easily
> switched any time.
>
> thanks
> - - -- --- ----- -------- -------------
> Jacek Lewandowski
>
>
> wt., 21 lut 2023 o 01:46 Yuki Morishita <yu...@apache.org> napisał(a):
>
>> Hi,
>>
>> What I wanted to address in my comment in CASSANDRA-8110(
>> https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705)
>> is to focus on better upgrade experience.
>>
>> Upgrading the cluster can be painful for some orgs with mission critical
>> Cassandra cluster, where they cannot tolerate less availability because of
>> the inability to replace the downed node.
>> They also need to plan rolling back to the previous state when something
>> happens along the way.
>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at
>> least enabling SSTable streaming during the upgrade by not upgrading the
>> SSTable version. This can make the cluster to easily rollback to the
>> previous version.
>> Downgrading SSTable is not the primary focus (though Cassandra needs to
>> implement the way to write SSTable in older versions, so it is somewhat
>> related.)
>>
>> I'm preparing the design doc for the change.
>> Also, if I should create a separate ticket from CASSANDRA-8110 for the
>> clarity of the goal of the change, please let me know.
>>
>>
>> On Tue, Feb 21, 2023 at 5:31 AM Benedict <be...@apache.org> wrote:
>>
>>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
>>> will have to eventually also tackle system schema changes (probably not
>>> hard), and may have to think a little carefully about other things, eg with
>>> TTLs the format change is only the contract about what values can be
>>> present, so we have to make sure the data validity checks are consistent
>>> with the format we write. It isn’t as simple as writing an earlier version
>>> in this case (unless we permit truncating the TTL, perhaps)
>>>
>>> On 20 Feb 2023, at 20:24, Benedict <be...@apache.org> wrote:
>>>
>>>
>>> 
>>> In a self-organising community, things that aren’t self-policed
>>> naturally end up policed in an adhoc manner, and with difficulty. I’m not
>>> sure that’s the same as arbitrary enforcement. It seems to me the real
>>> issue is nobody noticed this was agreed and/or forgot and didn’t think
>>> about it much.
>>>
>>> But, even without any prior agreement, it’s perfectly reasonable to
>>> request that things do not break compatibility if they do not need to, as
>>> part of the normal patch integration process.
>>>
>>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
>>> agreement to do this. But we can and should address the problem of new
>>> columns in schema tables, as this happens often between versions. I’m not
>>> sure it has in 4.1 though?
>>>
>>> Regarding downgrade versions, surely this should simply be the same as
>>> upgrade versions we support?
>>>
>>>
>>> On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>> 
>>> I'm not even convinced even 8110 addresses this - just writing sstables
>>> in old versions won't help if we ever add things like new types or new
>>> types of collections without other control abilities. Claude's other email
>>> in another thread a few hours ago talks about some of these surprises -
>>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>> added to system/local.  This means that 3.1 system can not read the table
>>> as it has no definition for it.  I tried marking the column for deletion in
>>> the metadata and in the serialization header.  The later got past the
>>> column not found problem, but I suspect that it just means that data
>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>> harder problem to solve than just versioning sstables and network
>>> protocols.
>>>
>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>> which point we will be able to concretely say "this release can be
>>> downgraded to X".   Until we actually define and agree that this is a
>>> real goal with a concrete version where downgrade-ability becomes real, it
>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>> frustrating for people trying to commit work/tickets.
>>>
>>> - Jeff
>>>
>>>
>>>
>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:
>>>
>>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>>> implies that we could potentially roll back an upgrade at any time. While I
>>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>>> would be a good objective to maintain strict backward compatibility and
>>>> therefore downgradability until a certain point. This would imply
>>>> versioning metadata and extending it in such a way that prior version(s)
>>>> could continue functioning. This can certainly be expensive to implement
>>>> and might bloat on-disk storage. However, we could always offer an option
>>>> for the operator to optimize the on-disk structures for the current version
>>>> then we can rewrite them in the latest version. This optimizes the storage
>>>> and opens up new functionality. This means new features that can work with
>>>> old on-disk structures will be available while others that strictly require
>>>> new versions of the data structures will be unavailable until the operator
>>>> migrates to the new version. This migration IMO should be irreversible.
>>>> Beyond this point the operator will lose the ability to downgrade which is
>>>> ok.
>>>>
>>>> Dinesh
>>>>
>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>>
>>>> 
>>>> There has been progress on
>>>>
>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>
>>>> Which is similar to what datastax does for DSE. Would this be an
>>>> acceptable solution?
>>>>
>>>> Jake
>>>>
>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>>> wrote:
>>>>
>>>>> It seems “An alternative solution is to implement/complete
>>>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>>>> can give us more options if it is finished😉
>>>>>
>>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> There has been a discussion lately about changes to the sstable
>>>>>> format in the context of being able to abort a cluster upgrade, and the
>>>>>> fact that changes to sstables can prevent downgraded nodes from reading any
>>>>>> data written during their temporary operation with the new version.
>>>>>>
>>>>>> Most of the discussion is in CASSANDRA-18134
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>>> spreading into CASSANDRA-14277
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>>> CASSANDRA-17698
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>>> which is a good place to discuss the topic seriously.
>>>>>>
>>>>>> Downgradability is a worthy goal and is listed in the current
>>>>>> roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>
>>>>>> My understanding of what has been suggested so far translates to:
>>>>>> - avoid changes to sstable formats;
>>>>>> - if there are changes, implement them in a way that is
>>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>>> presented in a component or portion of a component that legacy nodes will
>>>>>> not try to read;
>>>>>> - if the latter is not feasible, make sure the changes are only
>>>>>> applied if a feature flag has been enabled.
>>>>>>
>>>>>> To me this approach introduces several risks:
>>>>>> - it bloats file and parsing complexity;
>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>>>>> ticket once this requirement is in place);
>>>>>> - it needs care to avoid risky solutions to address technical issues
>>>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>>>> features);
>>>>>> - it requires separate and uncoordinated solutions to the problem and
>>>>>> switching mechanisms for each individual change.
>>>>>>
>>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>>> provides a method of writing sstables for a target version. During
>>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>>> older version, and there is a very straightforward way to implement
>>>>>> modifications to formats like the tickets above to conform to its
>>>>>> requirements.
>>>>>>
>>>>>> What do people think should be the way forward?
>>>>>>
>>>>>> Regards,
>>>>>> Branimir
>>>>>>
>>>>>>
>>>>>> --
>>>>> you are the apple of my eye !
>>>>>
>>>> --
>>>> http://twitter.com/tjake
>>>>
>>>>

Re: Downgradability

Posted by Jacek Lewandowski <le...@gmail.com>.
I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
multiple sstable formats support. It allows for providing an implementation
of SSTableFormat along with SSTableReader and SSTableWriter. That could be
extended easily to support different implementations for certain version
ranges, like one impl for ma-nz, other for oa+, etc. without having a
confusing implementation with a lot of conditional blocks. Old formats in
such case could be maintained separately from the main code and easily
switched any time.

thanks
- - -- --- ----- -------- -------------
Jacek Lewandowski


wt., 21 lut 2023 o 01:46 Yuki Morishita <yu...@apache.org> napisał(a):

> Hi,
>
> What I wanted to address in my comment in CASSANDRA-8110(
> https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705)
> is to focus on better upgrade experience.
>
> Upgrading the cluster can be painful for some orgs with mission critical
> Cassandra cluster, where they cannot tolerate less availability because of
> the inability to replace the downed node.
> They also need to plan rolling back to the previous state when something
> happens along the way.
> The change I proposed in CASSANDRA-8110 is to achieve the goal of at least
> enabling SSTable streaming during the upgrade by not upgrading the SSTable
> version. This can make the cluster to easily rollback to the previous
> version.
> Downgrading SSTable is not the primary focus (though Cassandra needs to
> implement the way to write SSTable in older versions, so it is somewhat
> related.)
>
> I'm preparing the design doc for the change.
> Also, if I should create a separate ticket from CASSANDRA-8110 for the
> clarity of the goal of the change, please let me know.
>
>
> On Tue, Feb 21, 2023 at 5:31 AM Benedict <be...@apache.org> wrote:
>
>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
>> will have to eventually also tackle system schema changes (probably not
>> hard), and may have to think a little carefully about other things, eg with
>> TTLs the format change is only the contract about what values can be
>> present, so we have to make sure the data validity checks are consistent
>> with the format we write. It isn’t as simple as writing an earlier version
>> in this case (unless we permit truncating the TTL, perhaps)
>>
>> On 20 Feb 2023, at 20:24, Benedict <be...@apache.org> wrote:
>>
>>
>> 
>> In a self-organising community, things that aren’t self-policed naturally
>> end up policed in an adhoc manner, and with difficulty. I’m not sure that’s
>> the same as arbitrary enforcement. It seems to me the real issue is nobody
>> noticed this was agreed and/or forgot and didn’t think about it much.
>>
>> But, even without any prior agreement, it’s perfectly reasonable to
>> request that things do not break compatibility if they do not need to, as
>> part of the normal patch integration process.
>>
>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
>> agreement to do this. But we can and should address the problem of new
>> columns in schema tables, as this happens often between versions. I’m not
>> sure it has in 4.1 though?
>>
>> Regarding downgrade versions, surely this should simply be the same as
>> upgrade versions we support?
>>
>>
>> On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:
>>
>> 
>> I'm not even convinced even 8110 addresses this - just writing sstables
>> in old versions won't help if we ever add things like new types or new
>> types of collections without other control abilities. Claude's other email
>> in another thread a few hours ago talks about some of these surprises -
>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>> added to system/local.  This means that 3.1 system can not read the table
>> as it has no definition for it.  I tried marking the column for deletion in
>> the metadata and in the serialization header.  The later got past the
>> column not found problem, but I suspect that it just means that data
>> columns after broadcast_port shifted and so incorrectly read." - this is a
>> harder problem to solve than just versioning sstables and network
>> protocols.
>>
>> Stepping back a bit, we have downgrade ability listed as a goal, but it's
>> not (as far as I can tell) universally enforced, nor is it clear at which
>> point we will be able to concretely say "this release can be downgraded to
>> X".   Until we actually define and agree that this is a real goal with a
>> concrete version where downgrade-ability becomes real, it feels like things
>> are somewhat arbitrarily enforced, which is probably very frustrating for
>> people trying to commit work/tickets.
>>
>> - Jeff
>>
>>
>>
>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:
>>
>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>> implies that we could potentially roll back an upgrade at any time. While I
>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>> would be a good objective to maintain strict backward compatibility and
>>> therefore downgradability until a certain point. This would imply
>>> versioning metadata and extending it in such a way that prior version(s)
>>> could continue functioning. This can certainly be expensive to implement
>>> and might bloat on-disk storage. However, we could always offer an option
>>> for the operator to optimize the on-disk structures for the current version
>>> then we can rewrite them in the latest version. This optimizes the storage
>>> and opens up new functionality. This means new features that can work with
>>> old on-disk structures will be available while others that strictly require
>>> new versions of the data structures will be unavailable until the operator
>>> migrates to the new version. This migration IMO should be irreversible.
>>> Beyond this point the operator will lose the ability to downgrade which is
>>> ok.
>>>
>>> Dinesh
>>>
>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>
>>> 
>>> There has been progress on
>>>
>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>
>>> Which is similar to what datastax does for DSE. Would this be an
>>> acceptable solution?
>>>
>>> Jake
>>>
>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>> wrote:
>>>
>>>> It seems “An alternative solution is to implement/complete
>>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>>> can give us more options if it is finished😉
>>>>
>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> There has been a discussion lately about changes to the sstable format
>>>>> in the context of being able to abort a cluster upgrade, and the fact that
>>>>> changes to sstables can prevent downgraded nodes from reading any data
>>>>> written during their temporary operation with the new version.
>>>>>
>>>>> Most of the discussion is in CASSANDRA-18134
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>> spreading into CASSANDRA-14277
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>> CASSANDRA-17698
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>> which is a good place to discuss the topic seriously.
>>>>>
>>>>> Downgradability is a worthy goal and is listed in the current roadmap.
>>>>> I would like to open a discussion here on how it would be achieved.
>>>>>
>>>>> My understanding of what has been suggested so far translates to:
>>>>> - avoid changes to sstable formats;
>>>>> - if there are changes, implement them in a way that is
>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>> presented in a component or portion of a component that legacy nodes will
>>>>> not try to read;
>>>>> - if the latter is not feasible, make sure the changes are only
>>>>> applied if a feature flag has been enabled.
>>>>>
>>>>> To me this approach introduces several risks:
>>>>> - it bloats file and parsing complexity;
>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>>>> ticket once this requirement is in place);
>>>>> - it needs care to avoid risky solutions to address technical issues
>>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>>> features);
>>>>> - it requires separate and uncoordinated solutions to the problem and
>>>>> switching mechanisms for each individual change.
>>>>>
>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>> provides a method of writing sstables for a target version. During
>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>> older version, and there is a very straightforward way to implement
>>>>> modifications to formats like the tickets above to conform to its
>>>>> requirements.
>>>>>
>>>>> What do people think should be the way forward?
>>>>>
>>>>> Regards,
>>>>> Branimir
>>>>>
>>>>>
>>>>> --
>>>> you are the apple of my eye !
>>>>
>>> --
>>> http://twitter.com/tjake
>>>
>>>

Re: Downgradability

Posted by Yuki Morishita <yu...@apache.org>.
Hi,

What I wanted to address in my comment in CASSANDRA-8110(
https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705)
is to focus on better upgrade experience.

Upgrading the cluster can be painful for some orgs with mission critical
Cassandra cluster, where they cannot tolerate less availability because of
the inability to replace the downed node.
They also need to plan rolling back to the previous state when something
happens along the way.
The change I proposed in CASSANDRA-8110 is to achieve the goal of at least
enabling SSTable streaming during the upgrade by not upgrading the SSTable
version. This can make the cluster to easily rollback to the previous
version.
Downgrading SSTable is not the primary focus (though Cassandra needs to
implement the way to write SSTable in older versions, so it is somewhat
related.)

I'm preparing the design doc for the change.
Also, if I should create a separate ticket from CASSANDRA-8110 for the
clarity of the goal of the change, please let me know.


On Tue, Feb 21, 2023 at 5:31 AM Benedict <be...@apache.org> wrote:

> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
> will have to eventually also tackle system schema changes (probably not
> hard), and may have to think a little carefully about other things, eg with
> TTLs the format change is only the contract about what values can be
> present, so we have to make sure the data validity checks are consistent
> with the format we write. It isn’t as simple as writing an earlier version
> in this case (unless we permit truncating the TTL, perhaps)
>
> On 20 Feb 2023, at 20:24, Benedict <be...@apache.org> wrote:
>
>
> 
> In a self-organising community, things that aren’t self-policed naturally
> end up policed in an adhoc manner, and with difficulty. I’m not sure that’s
> the same as arbitrary enforcement. It seems to me the real issue is nobody
> noticed this was agreed and/or forgot and didn’t think about it much.
>
> But, even without any prior agreement, it’s perfectly reasonable to
> request that things do not break compatibility if they do not need to, as
> part of the normal patch integration process.
>
> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
> agreement to do this. But we can and should address the problem of new
> columns in schema tables, as this happens often between versions. I’m not
> sure it has in 4.1 though?
>
> Regarding downgrade versions, surely this should simply be the same as
> upgrade versions we support?
>
>
> On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:
>
> 
> I'm not even convinced even 8110 addresses this - just writing sstables in
> old versions won't help if we ever add things like new types or new types
> of collections without other control abilities. Claude's other email in
> another thread a few hours ago talks about some of these surprises -
> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
> added to system/local.  This means that 3.1 system can not read the table
> as it has no definition for it.  I tried marking the column for deletion in
> the metadata and in the serialization header.  The later got past the
> column not found problem, but I suspect that it just means that data
> columns after broadcast_port shifted and so incorrectly read." - this is a
> harder problem to solve than just versioning sstables and network
> protocols.
>
> Stepping back a bit, we have downgrade ability listed as a goal, but it's
> not (as far as I can tell) universally enforced, nor is it clear at which
> point we will be able to concretely say "this release can be downgraded to
> X".   Until we actually define and agree that this is a real goal with a
> concrete version where downgrade-ability becomes real, it feels like things
> are somewhat arbitrarily enforced, which is probably very frustrating for
> people trying to commit work/tickets.
>
> - Jeff
>
>
>
> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:
>
>> I’m a big fan of maintaining backward compatibility. Downgradability
>> implies that we could potentially roll back an upgrade at any time. While I
>> don’t think we need to retain the ability to downgrade in perpetuity it
>> would be a good objective to maintain strict backward compatibility and
>> therefore downgradability until a certain point. This would imply
>> versioning metadata and extending it in such a way that prior version(s)
>> could continue functioning. This can certainly be expensive to implement
>> and might bloat on-disk storage. However, we could always offer an option
>> for the operator to optimize the on-disk structures for the current version
>> then we can rewrite them in the latest version. This optimizes the storage
>> and opens up new functionality. This means new features that can work with
>> old on-disk structures will be available while others that strictly require
>> new versions of the data structures will be unavailable until the operator
>> migrates to the new version. This migration IMO should be irreversible.
>> Beyond this point the operator will lose the ability to downgrade which is
>> ok.
>>
>> Dinesh
>>
>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>
>> 
>> There has been progress on
>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>
>> Which is similar to what datastax does for DSE. Would this be an
>> acceptable solution?
>>
>> Jake
>>
>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>> wrote:
>>
>>> It seems “An alternative solution is to implement/complete
>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>> can give us more options if it is finished😉
>>>
>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>
>>>> Hi everyone,
>>>>
>>>> There has been a discussion lately about changes to the sstable format
>>>> in the context of being able to abort a cluster upgrade, and the fact that
>>>> changes to sstables can prevent downgraded nodes from reading any data
>>>> written during their temporary operation with the new version.
>>>>
>>>> Most of the discussion is in CASSANDRA-18134
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>> spreading into CASSANDRA-14277
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>> CASSANDRA-17698 <https://issues.apache.org/jira/browse/CASSANDRA-17698>,
>>>> none of which is a good place to discuss the topic seriously.
>>>>
>>>> Downgradability is a worthy goal and is listed in the current roadmap.
>>>> I would like to open a discussion here on how it would be achieved.
>>>>
>>>> My understanding of what has been suggested so far translates to:
>>>> - avoid changes to sstable formats;
>>>> - if there are changes, implement them in a way that is
>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>> presented in a component or portion of a component that legacy nodes will
>>>> not try to read;
>>>> - if the latter is not feasible, make sure the changes are only applied
>>>> if a feature flag has been enabled.
>>>>
>>>> To me this approach introduces several risks:
>>>> - it bloats file and parsing complexity;
>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>>> ticket once this requirement is in place);
>>>> - it needs care to avoid risky solutions to address technical issues
>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>> features);
>>>> - it requires separate and uncoordinated solutions to the problem and
>>>> switching mechanisms for each individual change.
>>>>
>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which provides
>>>> a method of writing sstables for a target version. During upgrades, a node
>>>> could be set to produce sstables corresponding to the older version, and
>>>> there is a very straightforward way to implement modifications to formats
>>>> like the tickets above to conform to its requirements.
>>>>
>>>> What do people think should be the way forward?
>>>>
>>>> Regards,
>>>> Branimir
>>>>
>>>>
>>>> --
>>> you are the apple of my eye !
>>>
>> --
>> http://twitter.com/tjake
>>
>>

Re: Downgradability

Posted by Benedict <be...@apache.org>.
FWIW I think 8110 is the right approach, even if it isn’t a panacea. We will
have to eventually also tackle system schema changes (probably not hard), and
may have to think a little carefully about other things, eg with TTLs the
format change is only the contract about what values can be present, so we
have to make sure the data validity checks are consistent with the format we
write. It isn’t as simple as writing an earlier version in this case (unless
we permit truncating the TTL, perhaps)

  

On 20 Feb 2023, at 20:24, Benedict <be...@apache.org> wrote:  

>  
>

> 
>
> In a self-organising community, things that aren’t self-policed naturally
> end up policed in an adhoc manner, and with difficulty. I’m not sure that’s
> the same as arbitrary enforcement. It seems to me the real issue is nobody
> noticed this was agreed and/or forgot and didn’t think about it much.
>
>  
>
>
> But, even without any prior agreement, it’s perfectly reasonable to request
> that things do not break compatibility if they do not need to, as part of
> the normal patch integration process.
>
>  
>
>
> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
> agreement to do this. But we can and should address the problem of new
> columns in schema tables, as this happens often between versions. I’m not
> sure it has in 4.1 though?
>
>  
>
>
> Regarding downgrade versions, surely this should simply be the same as
> upgrade versions we support?
>
>  
>
>
>  
>
>

>> On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:  
>  
>
>

>> 

>>

>> I'm not even convinced even 8110 addresses this - just writing sstables in
old versions won't help if we ever add things like new types or new types of
collections without other control abilities. Claude's other email in another
thread a few hours ago talks about some of these surprises - "Specifically
during the 3.1 -> 4.0 changes a column broadcast_port was added to
system/local.  This means that 3.1 system can not read the table as it has no
definition for it.  I tried marking the column for deletion in the metadata
and in the serialization header.  The later got past the column not found
problem, but I suspect that it just means that data columns after
broadcast_port shifted and so incorrectly read." - this is a harder problem to
solve than just versioning sstables and network protocols.  
>
>>

>>  
>
>>

>> Stepping back a bit, we have downgrade ability listed as a goal, but it's
not (as far as I can tell) universally enforced, nor is it clear at which
point we will be able to concretely say "this release can be downgraded to X".
Until we actually define and agree that this is a real goal with a concrete
version where downgrade-ability becomes real, it feels like things are
somewhat arbitrarily enforced, which is probably very frustrating for people
trying to commit work/tickets.

>>

>>  
>
>>

>> \- Jeff

>>

>>  
>
>>

>>  
>
>>

>>  
>
>>

>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi
<[djoshi@apache.org](mailto:djoshi@apache.org)> wrote:  
>
>>

>>> I’m a big fan of maintaining backward compatibility. Downgradability
implies that we could potentially roll back an upgrade at any time. While I
don’t think we need to retain the ability to downgrade in perpetuity it would
be a good objective to maintain strict backward compatibility and therefore
downgradability until a certain point. This would imply versioning metadata
and extending it in such a way that prior version(s) could continue
functioning. This can certainly be expensive to implement and might bloat on-
disk storage. However, we could always offer an option for the operator to
optimize the on-disk structures for the current version then we can rewrite
them in the latest version. This optimizes the storage and opens up new
functionality. This means new features that can work with old on-disk
structures will be available while others that strictly require new versions
of the data structures will be unavailable until the operator migrates to the
new version. This migration IMO should be irreversible. Beyond this point the
operator will lose the ability to downgrade which is ok.  
>  
>
>>>

>>> Dinesh

>>>

>>>  
>
>>>

>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani
<[jakers@gmail.com](mailto:jakers@gmail.com)> wrote:  
>  
>
>>>

>>>> 

>>>>

>>>> There has been progress on

>>>>

>>>>
<https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928>

>>>>

>>>>  
>
>>>>

>>>> Which is similar to what datastax does for DSE. Would this be an
acceptable solution?

>>>>

>>>>  
>
>>>>

>>>> Jake

>>>>

>>>>  
>
>>>>

>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell
<[cclive1601@gmail.com](mailto:cclive1601@gmail.com)> wrote:  
>
>>>>

>>>>> It seems “An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110)” can
give us more options if it is finished😉

>>>>>

>>>>>  
>
>>>>>

>>>>> Branimir Lambov
<[blambov@apache.org](mailto:blambov@apache.org)>于2023年2月20日 周一下午11:03写道:  
>
>>>>>

>>>>>> Hi everyone,  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> There has been a discussion lately about changes to the sstable format
in the context of being able to abort a cluster upgrade, and the fact that
changes to sstables can prevent downgraded nodes from reading any data written
during their temporary operation with the new version.

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> Most of the discussion is in
[CASSANDRA-18134](https://issues.apache.org/jira/browse/CASSANDRA-18134), and
is spreading into
[CASSANDRA-14277](https://issues.apache.org/jira/browse/CASSANDRA-14227) and
[CASSANDRA-17698](https://issues.apache.org/jira/browse/CASSANDRA-17698), none
of which is a good place to discuss the topic seriously.

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> Downgradability is a worthy goal and is listed in the current roadmap.
I would like to open a discussion here on how it would be achieved.  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> My understanding of what has been suggested so far translates to:

>>>>>>

>>>>>> \- avoid changes to sstable formats;

>>>>>>

>>>>>> \- if there are changes, implement them in a way that is backwards-
compatible, e.g. by duplicating data, so that a new version is presented in a
component or portion of a component that legacy nodes will not try to read;

>>>>>>

>>>>>> \- if the latter is not feasible, make sure the changes are only
applied if a feature flag has been enabled.

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> To me this approach introduces several risks:

>>>>>>

>>>>>> \- it bloats file and parsing complexity;

>>>>>>

>>>>>> \- it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
ticket once this requirement is in place);

>>>>>>

>>>>>> \- it needs care to avoid risky solutions to address technical issues
with the format versioning (e.g. staying on n-versions for 5.0 and needing a
bump for a 4.1 bugfix might require porting over support for new features);  
>
>>>>>>

>>>>>> \- it requires separate and uncoordinated solutions to the problem and
switching mechanisms for each individual change.

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110), which
provides a method of writing sstables for a target version. During upgrades, a
node could be set to produce sstables corresponding to the older version, and
there is a very straightforward way to implement modifications to formats like
the tickets above to conform to its requirements.

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> What do people think should be the way forward?

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>> Regards,

>>>>>>

>>>>>> Branimir  
>
>>>>>>

>>>>>>  
>
>>>>>>

>>>>>>  
>
>>>>>

>>>>> \--  
>
>>>>>

>>>>> you are the apple of my eye !

>>>>

>>>> \--  
>
>>>>

>>>> <http://twitter.com/tjake>


Re: Downgradability

Posted by Benedict <be...@apache.org>.
In a self-organising community, things that aren’t self-policed naturally end
up policed in an adhoc manner, and with difficulty. I’m not sure that’s the
same as arbitrary enforcement. It seems to me the real issue is nobody noticed
this was agreed and/or forgot and didn’t think about it much.

  

But, even without any prior agreement, it’s perfectly reasonable to request
that things do not break compatibility if they do not need to, as part of the
normal patch integration process.

  

Issues with 3.1->4.0 aren’t particularly relevant as they predate any
agreement to do this. But we can and should address the problem of new columns
in schema tables, as this happens often between versions. I’m not sure it has
in 4.1 though?

  

Regarding downgrade versions, surely this should simply be the same as upgrade
versions we support?

  

  

> On 20 Feb 2023, at 20:02, Jeff Jirsa <jj...@gmail.com> wrote:  
>  
>

> 
>
> I'm not even convinced even 8110 addresses this - just writing sstables in
> old versions won't help if we ever add things like new types or new types of
> collections without other control abilities. Claude's other email in another
> thread a few hours ago talks about some of these surprises - "Specifically
> during the 3.1 -> 4.0 changes a column broadcast_port was added to
> system/local.  This means that 3.1 system can not read the table as it has
> no definition for it.  I tried marking the column for deletion in the
> metadata and in the serialization header.  The later got past the column not
> found problem, but I suspect that it just means that data columns after
> broadcast_port shifted and so incorrectly read." - this is a harder problem
> to solve than just versioning sstables and network protocols.  
>
>
>  
>
>
> Stepping back a bit, we have downgrade ability listed as a goal, but it's
> not (as far as I can tell) universally enforced, nor is it clear at which
> point we will be able to concretely say "this release can be downgraded to
> X".   Until we actually define and agree that this is a real goal with a
> concrete version where downgrade-ability becomes real, it feels like things
> are somewhat arbitrarily enforced, which is probably very frustrating for
> people trying to commit work/tickets.
>
>  
>
>
> \- Jeff
>
>  
>
>
>  
>
>
>  
>
>
> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi
> <[djoshi@apache.org](mailto:djoshi@apache.org)> wrote:  
>
>

>> I’m a big fan of maintaining backward compatibility. Downgradability
implies that we could potentially roll back an upgrade at any time. While I
don’t think we need to retain the ability to downgrade in perpetuity it would
be a good objective to maintain strict backward compatibility and therefore
downgradability until a certain point. This would imply versioning metadata
and extending it in such a way that prior version(s) could continue
functioning. This can certainly be expensive to implement and might bloat on-
disk storage. However, we could always offer an option for the operator to
optimize the on-disk structures for the current version then we can rewrite
them in the latest version. This optimizes the storage and opens up new
functionality. This means new features that can work with old on-disk
structures will be available while others that strictly require new versions
of the data structures will be unavailable until the operator migrates to the
new version. This migration IMO should be irreversible. Beyond this point the
operator will lose the ability to downgrade which is ok.  
>  
>
>>

>> Dinesh

>>

>>  
>
>>

>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani
<[jakers@gmail.com](mailto:jakers@gmail.com)> wrote:  
>  
>
>>

>>> 

>>>

>>> There has been progress on

>>>

>>>
<https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928>

>>>

>>>  
>
>>>

>>> Which is similar to what datastax does for DSE. Would this be an
acceptable solution?

>>>

>>>  
>
>>>

>>> Jake

>>>

>>>  
>
>>>

>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell
<[cclive1601@gmail.com](mailto:cclive1601@gmail.com)> wrote:  
>
>>>

>>>> It seems “An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110)” can
give us more options if it is finished😉

>>>>

>>>>  
>
>>>>

>>>> Branimir Lambov
<[blambov@apache.org](mailto:blambov@apache.org)>于2023年2月20日 周一下午11:03写道:  
>
>>>>

>>>>> Hi everyone,  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> There has been a discussion lately about changes to the sstable format
in the context of being able to abort a cluster upgrade, and the fact that
changes to sstables can prevent downgraded nodes from reading any data written
during their temporary operation with the new version.

>>>>>

>>>>>  
>
>>>>>

>>>>> Most of the discussion is in
[CASSANDRA-18134](https://issues.apache.org/jira/browse/CASSANDRA-18134), and
is spreading into
[CASSANDRA-14277](https://issues.apache.org/jira/browse/CASSANDRA-14227) and
[CASSANDRA-17698](https://issues.apache.org/jira/browse/CASSANDRA-17698), none
of which is a good place to discuss the topic seriously.

>>>>>

>>>>>  
>
>>>>>

>>>>> Downgradability is a worthy goal and is listed in the current roadmap. I
would like to open a discussion here on how it would be achieved.  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> My understanding of what has been suggested so far translates to:

>>>>>

>>>>> \- avoid changes to sstable formats;

>>>>>

>>>>> \- if there are changes, implement them in a way that is backwards-
compatible, e.g. by duplicating data, so that a new version is presented in a
component or portion of a component that legacy nodes will not try to read;

>>>>>

>>>>> \- if the latter is not feasible, make sure the changes are only applied
if a feature flag has been enabled.

>>>>>

>>>>>  
>
>>>>>

>>>>> To me this approach introduces several risks:

>>>>>

>>>>> \- it bloats file and parsing complexity;

>>>>>

>>>>> \- it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
ticket once this requirement is in place);

>>>>>

>>>>> \- it needs care to avoid risky solutions to address technical issues
with the format versioning (e.g. staying on n-versions for 5.0 and needing a
bump for a 4.1 bugfix might require porting over support for new features);  
>
>>>>>

>>>>> \- it requires separate and uncoordinated solutions to the problem and
switching mechanisms for each individual change.

>>>>>

>>>>>  
>
>>>>>

>>>>> An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110), which
provides a method of writing sstables for a target version. During upgrades, a
node could be set to produce sstables corresponding to the older version, and
there is a very straightforward way to implement modifications to formats like
the tickets above to conform to its requirements.

>>>>>

>>>>>  
>
>>>>>

>>>>> What do people think should be the way forward?

>>>>>

>>>>>  
>
>>>>>

>>>>> Regards,

>>>>>

>>>>> Branimir  
>
>>>>>

>>>>>  
>
>>>>>

>>>>>  
>
>>>>

>>>> \--  
>
>>>>

>>>> you are the apple of my eye !

>>>

>>> \--  
>
>>>

>>> <http://twitter.com/tjake>


Re: Downgradability

Posted by "Claude Warren, Jr via dev" <de...@cassandra.apache.org>.
You also need to remove the system.local.broadcast_port column as that does
not exist in the earlier version and when the earlier version attempts to
read it the code throws an UnknownColumnException.

On Thu, Feb 23, 2023 at 11:27 AM Jacek Lewandowski <
lewandowski.jacek@gmail.com> wrote:

> Running upgrade tests backwards is great idea which does not require extra
> work.
>
> For stats metadata it already supports writing in previous serialization
> version
>
> We need a small fix in compression metadata and that's it.
>
> A flag with the write format version is probably LHF.
>
> Maybe let's try,  we still have time to fix it before 5.0
>
>
> czw., 23 lut 2023, 10:57 użytkownik Benedict <be...@apache.org>
> napisał:
>
>> Forget downgradeability for a moment: we should not be breaking format
>> compatibility without good reason. Bumping a major version isn’t enough of
>> a reason.
>>
>> Can somebody explain to me why this is being fought tooth and nail, when
>> the work involved is absolutely minimal?
>>
>> Regarding tests: what more do you want, than running our upgrade suite
>> backwards?
>>
>>
>> On 23 Feb 2023, at 09:45, Benjamin Lerer <bl...@apache.org> wrote:
>>
>> 
>>
>>> Can somebody explain to me what is so burdensome, that we seem to be
>>> spending longer debating it than it would take to implement the necessary
>>> changes?
>>
>>
>> I believe that we all agree on the outcome. Everybody wants
>> downgradability. The issue is on the path to get there.
>>
>> As far as I am concerned, I would like to see a proper solution and as
>> Jeff suggested the equivalent of the upgrade tests as gatekeepers. Having
>> everybody trying to enforce it on his own way will only lead to a poor
>> result in my opinion with some addoc code that might not really guarantee
>> real downgradability in the end.
>> We have rushed in the past to get feature outs and pay the price for it.
>> I simply prefer that we take the time to do things right.
>>
>> Thanks to Scott and you, downgradability got a much better visibility so
>> no matter what approach we pick, I am convinced that we will get there.
>>
>> Le jeu. 23 févr. 2023 à 09:49, Claude Warren, Jr via dev <
>> dev@cassandra.apache.org> a écrit :
>>
>>> Broken downgrading can be fixed (I think) by modifying the
>>> SearializationHeader.toHeader() method where it currently throws an
>>> UnknownColumnException.  If we can, instead of throwing the exception,
>>> create a dropped column for the unexpected column then I think the code
>>> will work.
>>>
>>> I realise that to do this in the wild is not possible as it is a
>>> change to released code, but we could handle it going forward.
>>>
>>> On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo <he...@datastax.com>
>>> wrote:
>>>
>>>> ... ok apparently shift+enter  sends messages now?
>>>>
>>>> I was just saying if at least the file format AND system/tables -
>>>> anything written to disk - can be protected with a switch, then it allows
>>>> for quick downgrade by shutting down the entire cluster and restarting with
>>>> the downgraded binary. It's a start.
>>>>
>>>> To be able to do that live in a distributed system needs to consider
>>>> much more: gossip, streaming, drivers, and ultimately all features, because
>>>> we don't' want an application developer to use a shiny new thing that a)
>>>> may not be available on all nodes, or b) may disappear if the cluster has
>>>> to be downgraded later.
>>>>
>>>> henrik
>>>>
>>>> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo <he...@datastax.com>
>>>> wrote:
>>>>
>>>>> Just this once I'm going to be really brief :-)
>>>>>
>>>>> Just wanted to share for reference how Mongodb implemented
>>>>> downgradeability around their 4.4 version:
>>>>> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>>>>>
>>>>> Jeff you're right. Ultimately this is about more than file formats.
>>>>> However, ideally if at least the
>>>>>
>>>>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>>
>>>>>> I'm not even convinced even 8110 addresses this - just writing
>>>>>> sstables in old versions won't help if we ever add things like new types or
>>>>>> new types of collections without other control abilities. Claude's other
>>>>>> email in another thread a few hours ago talks about some of these surprises
>>>>>> - "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>>>>> added to system/local.  This means that 3.1 system can not read the table
>>>>>> as it has no definition for it.  I tried marking the column for deletion in
>>>>>> the metadata and in the serialization header.  The later got past the
>>>>>> column not found problem, but I suspect that it just means that data
>>>>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>>>>> harder problem to solve than just versioning sstables and network
>>>>>> protocols.
>>>>>>
>>>>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>>>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>>>>> which point we will be able to concretely say "this release can be
>>>>>> downgraded to X".   Until we actually define and agree that this is a
>>>>>> real goal with a concrete version where downgrade-ability becomes real, it
>>>>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>>>>> frustrating for people trying to commit work/tickets.
>>>>>>
>>>>>> - Jeff
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>>>>>> implies that we could potentially roll back an upgrade at any time. While I
>>>>>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>>>>>> would be a good objective to maintain strict backward compatibility and
>>>>>>> therefore downgradability until a certain point. This would imply
>>>>>>> versioning metadata and extending it in such a way that prior version(s)
>>>>>>> could continue functioning. This can certainly be expensive to implement
>>>>>>> and might bloat on-disk storage. However, we could always offer an option
>>>>>>> for the operator to optimize the on-disk structures for the current version
>>>>>>> then we can rewrite them in the latest version. This optimizes the storage
>>>>>>> and opens up new functionality. This means new features that can work with
>>>>>>> old on-disk structures will be available while others that strictly require
>>>>>>> new versions of the data structures will be unavailable until the operator
>>>>>>> migrates to the new version. This migration IMO should be irreversible.
>>>>>>> Beyond this point the operator will lose the ability to downgrade which is
>>>>>>> ok.
>>>>>>>
>>>>>>> Dinesh
>>>>>>>
>>>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>>>>>
>>>>>>> 
>>>>>>> There has been progress on
>>>>>>>
>>>>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>>>>
>>>>>>> Which is similar to what datastax does for DSE. Would this be an
>>>>>>> acceptable solution?
>>>>>>>
>>>>>>> Jake
>>>>>>>
>>>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> It seems “An alternative solution is to implement/complete
>>>>>>>> CASSANDRA-8110
>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>” can give
>>>>>>>> us more options if it is finished😉
>>>>>>>>
>>>>>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> There has been a discussion lately about changes to the sstable
>>>>>>>>> format in the context of being able to abort a cluster upgrade, and the
>>>>>>>>> fact that changes to sstables can prevent downgraded nodes from reading any
>>>>>>>>> data written during their temporary operation with the new version.
>>>>>>>>>
>>>>>>>>> Most of the discussion is in CASSANDRA-18134
>>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>>>>>> spreading into CASSANDRA-14277
>>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>>>>>> CASSANDRA-17698
>>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>>>>>> which is a good place to discuss the topic seriously.
>>>>>>>>>
>>>>>>>>> Downgradability is a worthy goal and is listed in the current
>>>>>>>>> roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>>>>
>>>>>>>>> My understanding of what has been suggested so far translates to:
>>>>>>>>> - avoid changes to sstable formats;
>>>>>>>>> - if there are changes, implement them in a way that is
>>>>>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>>>>>> presented in a component or portion of a component that legacy nodes will
>>>>>>>>> not try to read;
>>>>>>>>> - if the latter is not feasible, make sure the changes are only
>>>>>>>>> applied if a feature flag has been enabled.
>>>>>>>>>
>>>>>>>>> To me this approach introduces several risks:
>>>>>>>>> - it bloats file and parsing complexity;
>>>>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a
>>>>>>>>> LHF ticket once this requirement is in place);
>>>>>>>>> - it needs care to avoid risky solutions to address
>>>>>>>>> technical issues with the format versioning (e.g. staying on n-versions for
>>>>>>>>> 5.0 and needing a bump for a 4.1 bugfix might require porting over support
>>>>>>>>> for new features);
>>>>>>>>> - it requires separate and uncoordinated solutions to the problem
>>>>>>>>> and switching mechanisms for each individual change.
>>>>>>>>>
>>>>>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>>>>>> provides a method of writing sstables for a target version. During
>>>>>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>>>>>> older version, and there is a very straightforward way to implement
>>>>>>>>> modifications to formats like the tickets above to conform to its
>>>>>>>>> requirements.
>>>>>>>>>
>>>>>>>>> What do people think should be the way forward?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Branimir
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> you are the apple of my eye !
>>>>>>>>
>>>>>>> --
>>>>>>> http://twitter.com/tjake
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Henrik Ingo
>>>>>
>>>>> c. +358 40 569 7354
>>>>>
>>>>> w. www.datastax.com
>>>>>
>>>>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>>>>> <https://www.linkedin.com/company/datastax/>
>>>>> <https://github.com/datastax/>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Henrik Ingo
>>>>
>>>> c. +358 40 569 7354
>>>>
>>>> w. www.datastax.com
>>>>
>>>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>>>> <https://www.linkedin.com/company/datastax/>
>>>> <https://github.com/datastax/>
>>>>
>>>>

Re: Downgradability

Posted by Jacek Lewandowski <le...@gmail.com>.
Running upgrade tests backwards is great idea which does not require extra
work.

For stats metadata it already supports writing in previous serialization
version

We need a small fix in compression metadata and that's it.

A flag with the write format version is probably LHF.

Maybe let's try,  we still have time to fix it before 5.0


czw., 23 lut 2023, 10:57 użytkownik Benedict <be...@apache.org> napisał:

> Forget downgradeability for a moment: we should not be breaking format
> compatibility without good reason. Bumping a major version isn’t enough of
> a reason.
>
> Can somebody explain to me why this is being fought tooth and nail, when
> the work involved is absolutely minimal?
>
> Regarding tests: what more do you want, than running our upgrade suite
> backwards?
>
>
> On 23 Feb 2023, at 09:45, Benjamin Lerer <bl...@apache.org> wrote:
>
> 
>
>> Can somebody explain to me what is so burdensome, that we seem to be
>> spending longer debating it than it would take to implement the necessary
>> changes?
>
>
> I believe that we all agree on the outcome. Everybody wants
> downgradability. The issue is on the path to get there.
>
> As far as I am concerned, I would like to see a proper solution and as
> Jeff suggested the equivalent of the upgrade tests as gatekeepers. Having
> everybody trying to enforce it on his own way will only lead to a poor
> result in my opinion with some addoc code that might not really guarantee
> real downgradability in the end.
> We have rushed in the past to get feature outs and pay the price for it. I
> simply prefer that we take the time to do things right.
>
> Thanks to Scott and you, downgradability got a much better visibility so
> no matter what approach we pick, I am convinced that we will get there.
>
> Le jeu. 23 févr. 2023 à 09:49, Claude Warren, Jr via dev <
> dev@cassandra.apache.org> a écrit :
>
>> Broken downgrading can be fixed (I think) by modifying the
>> SearializationHeader.toHeader() method where it currently throws an
>> UnknownColumnException.  If we can, instead of throwing the exception,
>> create a dropped column for the unexpected column then I think the code
>> will work.
>>
>> I realise that to do this in the wild is not possible as it is a
>> change to released code, but we could handle it going forward.
>>
>> On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo <he...@datastax.com>
>> wrote:
>>
>>> ... ok apparently shift+enter  sends messages now?
>>>
>>> I was just saying if at least the file format AND system/tables -
>>> anything written to disk - can be protected with a switch, then it allows
>>> for quick downgrade by shutting down the entire cluster and restarting with
>>> the downgraded binary. It's a start.
>>>
>>> To be able to do that live in a distributed system needs to consider
>>> much more: gossip, streaming, drivers, and ultimately all features, because
>>> we don't' want an application developer to use a shiny new thing that a)
>>> may not be available on all nodes, or b) may disappear if the cluster has
>>> to be downgraded later.
>>>
>>> henrik
>>>
>>> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo <he...@datastax.com>
>>> wrote:
>>>
>>>> Just this once I'm going to be really brief :-)
>>>>
>>>> Just wanted to share for reference how Mongodb implemented
>>>> downgradeability around their 4.4 version:
>>>> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>>>>
>>>> Jeff you're right. Ultimately this is about more than file formats.
>>>> However, ideally if at least the
>>>>
>>>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>>>
>>>>> I'm not even convinced even 8110 addresses this - just writing
>>>>> sstables in old versions won't help if we ever add things like new types or
>>>>> new types of collections without other control abilities. Claude's other
>>>>> email in another thread a few hours ago talks about some of these surprises
>>>>> - "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>>>> added to system/local.  This means that 3.1 system can not read the table
>>>>> as it has no definition for it.  I tried marking the column for deletion in
>>>>> the metadata and in the serialization header.  The later got past the
>>>>> column not found problem, but I suspect that it just means that data
>>>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>>>> harder problem to solve than just versioning sstables and network
>>>>> protocols.
>>>>>
>>>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>>>> which point we will be able to concretely say "this release can be
>>>>> downgraded to X".   Until we actually define and agree that this is a
>>>>> real goal with a concrete version where downgrade-ability becomes real, it
>>>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>>>> frustrating for people trying to commit work/tickets.
>>>>>
>>>>> - Jeff
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>>>>> implies that we could potentially roll back an upgrade at any time. While I
>>>>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>>>>> would be a good objective to maintain strict backward compatibility and
>>>>>> therefore downgradability until a certain point. This would imply
>>>>>> versioning metadata and extending it in such a way that prior version(s)
>>>>>> could continue functioning. This can certainly be expensive to implement
>>>>>> and might bloat on-disk storage. However, we could always offer an option
>>>>>> for the operator to optimize the on-disk structures for the current version
>>>>>> then we can rewrite them in the latest version. This optimizes the storage
>>>>>> and opens up new functionality. This means new features that can work with
>>>>>> old on-disk structures will be available while others that strictly require
>>>>>> new versions of the data structures will be unavailable until the operator
>>>>>> migrates to the new version. This migration IMO should be irreversible.
>>>>>> Beyond this point the operator will lose the ability to downgrade which is
>>>>>> ok.
>>>>>>
>>>>>> Dinesh
>>>>>>
>>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>>>>
>>>>>> 
>>>>>> There has been progress on
>>>>>>
>>>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>>>
>>>>>> Which is similar to what datastax does for DSE. Would this be an
>>>>>> acceptable solution?
>>>>>>
>>>>>> Jake
>>>>>>
>>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> It seems “An alternative solution is to implement/complete
>>>>>>> CASSANDRA-8110
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>” can give us
>>>>>>> more options if it is finished😉
>>>>>>>
>>>>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> There has been a discussion lately about changes to the sstable
>>>>>>>> format in the context of being able to abort a cluster upgrade, and the
>>>>>>>> fact that changes to sstables can prevent downgraded nodes from reading any
>>>>>>>> data written during their temporary operation with the new version.
>>>>>>>>
>>>>>>>> Most of the discussion is in CASSANDRA-18134
>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>>>>> spreading into CASSANDRA-14277
>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>>>>> CASSANDRA-17698
>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>>>>> which is a good place to discuss the topic seriously.
>>>>>>>>
>>>>>>>> Downgradability is a worthy goal and is listed in the current
>>>>>>>> roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>>>
>>>>>>>> My understanding of what has been suggested so far translates to:
>>>>>>>> - avoid changes to sstable formats;
>>>>>>>> - if there are changes, implement them in a way that is
>>>>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>>>>> presented in a component or portion of a component that legacy nodes will
>>>>>>>> not try to read;
>>>>>>>> - if the latter is not feasible, make sure the changes are only
>>>>>>>> applied if a feature flag has been enabled.
>>>>>>>>
>>>>>>>> To me this approach introduces several risks:
>>>>>>>> - it bloats file and parsing complexity;
>>>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a
>>>>>>>> LHF ticket once this requirement is in place);
>>>>>>>> - it needs care to avoid risky solutions to address
>>>>>>>> technical issues with the format versioning (e.g. staying on n-versions for
>>>>>>>> 5.0 and needing a bump for a 4.1 bugfix might require porting over support
>>>>>>>> for new features);
>>>>>>>> - it requires separate and uncoordinated solutions to the problem
>>>>>>>> and switching mechanisms for each individual change.
>>>>>>>>
>>>>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>>>>> provides a method of writing sstables for a target version. During
>>>>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>>>>> older version, and there is a very straightforward way to implement
>>>>>>>> modifications to formats like the tickets above to conform to its
>>>>>>>> requirements.
>>>>>>>>
>>>>>>>> What do people think should be the way forward?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Branimir
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>> you are the apple of my eye !
>>>>>>>
>>>>>> --
>>>>>> http://twitter.com/tjake
>>>>>>
>>>>>>
>>>>
>>>> --
>>>>
>>>> Henrik Ingo
>>>>
>>>> c. +358 40 569 7354
>>>>
>>>> w. www.datastax.com
>>>>
>>>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>>>> <https://www.linkedin.com/company/datastax/>
>>>> <https://github.com/datastax/>
>>>>
>>>>
>>>
>>> --
>>>
>>> Henrik Ingo
>>>
>>> c. +358 40 569 7354
>>>
>>> w. www.datastax.com
>>>
>>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>>> <https://www.linkedin.com/company/datastax/>
>>> <https://github.com/datastax/>
>>>
>>>

Re: Downgradability

Posted by guo Maxwell <cc...@gmail.com>.
Hello everyone :
What is the final conclusion of this discuss ?

As https://issues.apache.org/jira/browse/CASSANDRA-18934 has been created ,
and we know that for system table of different c* version , the schema info
may be different as we may
add or delete column or modify the table properties in different versions.
For 18934 , because I have introduced a new column " compaction_properties"
for compaction_history table in 5.0. So the downgrade test to 4.1 failed at
the 4.1 node startup stage  due to  4.1’s code encountering an incompatible
persisted column "compaction_properties".

So now for me, I might want to know, which solution did everyone finally
decide to use for the sstable ?

1. Are our current downgrades intended to only support specific version
downgrades, like 5.0 -> 4.1 ?
2.Can we modify the low version code now ? that is say , if we add some
feature  for example in 4.1.0 and that means we may only support downgrade
to 4.1.1
3.If we choose to rewrite sstable, please help consider the differences in
system tables between different versions.

Jacek Lewandowski <le...@gmail.com> 于2023年3月6日周一 16:58写道:

> A bit of organization - I've created
> https://issues.apache.org/jira/browse/CASSANDRA-18300 epic to track
> tickets related to the downgradability. Please add the tickets you are
> aware of.
>
> thanks
> - - -- --- ----- -------- -------------
> Jacek Lewandowski
>
>
> czw., 23 lut 2023 o 17:47 Benedict <be...@apache.org> napisał(a):
>
>> Either way, it feels like this has become much more of a big deal than it
>> needed to.
>>
>> I would prefer the pending patches to avoid breaking compatibility, as I
>> think they can do it easily. But, if we agree to block release until we can
>> double back to fix it with versioned writing (which I agree with Jacek are
>> LHF - I think we literally just need a method that chooses the descriptor
>> version) then let’s not further agonise over this.
>>
>> Alternatively I’d be happy to work with the authors to get this done
>> alongside their work, as I don’t think it would hold anything up. We just
>> need something to pick a descriptor besides latest on write, everything
>> else is basically there for these patches.
>>
>>
>> On 23 Feb 2023, at 16:37, Henrik Ingo <he...@datastax.com> wrote:
>>
>> 
>> Right. So I'm speculating everyone else who worked on a patch that breaks
>> compatibility has been working under the mindset "I'll just put this behind
>> the same switch". Or something more vague / even less correct, such as
>> assuming that tries would become the default immediately.
>>
>> At least in my mind when we talk about the "switch to enable tries" I do
>> also consider things like "don't break streaming". So I guess whether one
>> considers "a switch" to exist already or not, might be subjective in this
>> case, because people have different assumptions on the definition of done
>> of such a switch.
>>
>> henrik
>>
>> On Thu, Feb 23, 2023 at 2:53 PM Benedict <be...@apache.org> wrote:
>>
>>> I don’t think there’s anything about a new format that requires a
>>> version bump, but I could be missing something.
>>>
>>> We have to have a switch to enable tries already don’t we? I’m pretty
>>> sure we haven’t talked about switching the default format?
>>>
>>> On 23 Feb 2023, at 12:12, Henrik Ingo <he...@datastax.com> wrote:
>>>
>>> 
>>> On Thu, Feb 23, 2023 at 11:57 AM Benedict <be...@apache.org> wrote:
>>>
>>>> Can somebody explain to me why this is being fought tooth and nail,
>>>> when the work involved is absolutely minimal?
>>>>
>>>>
>>> I don't know how each individual has been thinking about this, but it
>>> seems to me just looking at all the tasks that at least the introduction of
>>> tries is a major format change anyway - since it's the whole point - and
>>> therefore people working on other tasks may have assumed the format is
>>> changing anyway and therefore something like a switch (is this what is
>>> referred to as the C-8110 solution?) will take care of it for everyone.
>>>
>>> I'm not sure there's consensus that such a switch is a sufficient
>>> resolution to this discussion, but if there were such a consensus, the next
>>> question would be whether the patches that are otherwise ready now can
>>> merge, or whether they will all be blocked waiting for the compatibility
>>> solution first. And possibly better testing, etc. Letting them merge would
>>> be justified by the desire to have more frequent and smaller increments of
>>> work merged into trunk... well, I'm not going to repeat everything from
>>> that discussion but the same pro's and con's apply.
>>>
>>> henrik
>>> --
>>>
>>> Henrik Ingo
>>>
>>> c. +358 40 569 7354
>>>
>>> w. www.datastax.com
>>>
>>>
>>> <https://urldefense.com/v3/__https://www.facebook.com/datastax__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n0n_sPsT$>
>>> <https://twitter.com/datastax>
>>> <https://urldefense.com/v3/__https://www.linkedin.com/company/datastax/__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n2bcAuFd$>
>>> <https://github.com/datastax/>
>>>
>>>
>>
>> --
>>
>> Henrik Ingo
>>
>> c. +358 40 569 7354
>>
>> w. www.datastax.com
>>
>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>> <https://www.linkedin.com/company/datastax/>
>> <https://github.com/datastax/>
>>
>>

Re: Downgradability

Posted by Jacek Lewandowski <le...@gmail.com>.
A bit of organization - I've created
https://issues.apache.org/jira/browse/CASSANDRA-18300 epic to track tickets
related to the downgradability. Please add the tickets you are aware of.

thanks
- - -- --- ----- -------- -------------
Jacek Lewandowski


czw., 23 lut 2023 o 17:47 Benedict <be...@apache.org> napisał(a):

> Either way, it feels like this has become much more of a big deal than it
> needed to.
>
> I would prefer the pending patches to avoid breaking compatibility, as I
> think they can do it easily. But, if we agree to block release until we can
> double back to fix it with versioned writing (which I agree with Jacek are
> LHF - I think we literally just need a method that chooses the descriptor
> version) then let’s not further agonise over this.
>
> Alternatively I’d be happy to work with the authors to get this done
> alongside their work, as I don’t think it would hold anything up. We just
> need something to pick a descriptor besides latest on write, everything
> else is basically there for these patches.
>
>
> On 23 Feb 2023, at 16:37, Henrik Ingo <he...@datastax.com> wrote:
>
> 
> Right. So I'm speculating everyone else who worked on a patch that breaks
> compatibility has been working under the mindset "I'll just put this behind
> the same switch". Or something more vague / even less correct, such as
> assuming that tries would become the default immediately.
>
> At least in my mind when we talk about the "switch to enable tries" I do
> also consider things like "don't break streaming". So I guess whether one
> considers "a switch" to exist already or not, might be subjective in this
> case, because people have different assumptions on the definition of done
> of such a switch.
>
> henrik
>
> On Thu, Feb 23, 2023 at 2:53 PM Benedict <be...@apache.org> wrote:
>
>> I don’t think there’s anything about a new format that requires a version
>> bump, but I could be missing something.
>>
>> We have to have a switch to enable tries already don’t we? I’m pretty
>> sure we haven’t talked about switching the default format?
>>
>> On 23 Feb 2023, at 12:12, Henrik Ingo <he...@datastax.com> wrote:
>>
>> 
>> On Thu, Feb 23, 2023 at 11:57 AM Benedict <be...@apache.org> wrote:
>>
>>> Can somebody explain to me why this is being fought tooth and nail, when
>>> the work involved is absolutely minimal?
>>>
>>>
>> I don't know how each individual has been thinking about this, but it
>> seems to me just looking at all the tasks that at least the introduction of
>> tries is a major format change anyway - since it's the whole point - and
>> therefore people working on other tasks may have assumed the format is
>> changing anyway and therefore something like a switch (is this what is
>> referred to as the C-8110 solution?) will take care of it for everyone.
>>
>> I'm not sure there's consensus that such a switch is a sufficient
>> resolution to this discussion, but if there were such a consensus, the next
>> question would be whether the patches that are otherwise ready now can
>> merge, or whether they will all be blocked waiting for the compatibility
>> solution first. And possibly better testing, etc. Letting them merge would
>> be justified by the desire to have more frequent and smaller increments of
>> work merged into trunk... well, I'm not going to repeat everything from
>> that discussion but the same pro's and con's apply.
>>
>> henrik
>> --
>>
>> Henrik Ingo
>>
>> c. +358 40 569 7354
>>
>> w. www.datastax.com
>>
>>
>> <https://urldefense.com/v3/__https://www.facebook.com/datastax__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n0n_sPsT$>
>> <https://twitter.com/datastax>
>> <https://urldefense.com/v3/__https://www.linkedin.com/company/datastax/__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n2bcAuFd$>
>> <https://github.com/datastax/>
>>
>>
>
> --
>
> Henrik Ingo
>
> c. +358 40 569 7354
>
> w. www.datastax.com
>
> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
> <https://www.linkedin.com/company/datastax/>
> <https://github.com/datastax/>
>
>

Re: Downgradability

Posted by Benedict <be...@apache.org>.
Either way, it feels like this has become much more of a big deal than it
needed to.

  

I would prefer the pending patches to avoid breaking compatibility, as I think
they can do it easily. But, if we agree to block release until we can double
back to fix it with versioned writing (which I agree with Jacek are LHF - I
think we literally just need a method that chooses the descriptor version)
then let’s not further agonise over this.

  

Alternatively I’d be happy to work with the authors to get this done alongside
their work, as I don’t think it would hold anything up. We just need something
to pick a descriptor besides latest on write, everything else is basically
there for these patches.

  

  

> On 23 Feb 2023, at 16:37, Henrik Ingo <he...@datastax.com> wrote:  
>  
>

> 
>
> Right. So I'm speculating everyone else who worked on a patch that breaks
> compatibility has been working under the mindset "I'll just put this behind
> the same switch". Or something more vague / even less correct, such as
> assuming that tries would become the default immediately.
>
>  
>
>
> At least in my mind when we talk about the "switch to enable tries" I do
> also consider things like "don't break streaming". So I guess whether one
> considers "a switch" to exist already or not, might be subjective in this
> case, because people have different assumptions on the definition of done of
> such a switch.  
>
>
>  
>
>
> henrik
>
>  
>
>
> On Thu, Feb 23, 2023 at 2:53 PM Benedict
> <[benedict@apache.org](mailto:benedict@apache.org)> wrote:  
>
>

>> I don’t think there’s anything about a new format that requires a version
bump, but I could be missing something.

>>

>>  
>
>>

>> We have to have a switch to enable tries already don’t we? I’m pretty sure
we haven’t talked about switching the default format?

>>

>>  
>
>>

>>> On 23 Feb 2023, at 12:12, Henrik Ingo
<[henrik.ingo@datastax.com](mailto:henrik.ingo@datastax.com)> wrote:  
>  
>
>>

>>> 

>>>

>>> On Thu, Feb 23, 2023 at 11:57 AM Benedict
<[benedict@apache.org](mailto:benedict@apache.org)> wrote:  
>
>>>

>>>> Can somebody explain to me why this is being fought tooth and nail, when
the work involved is absolutely minimal?  
>
>>>>

>>>>  
>
>>>

>>>  
>
>>>

>>> I don't know how each individual has been thinking about this, but it
seems to me just looking at all the tasks that at least the introduction of
tries is a major format change anyway - since it's the whole point - and
therefore people working on other tasks may have assumed the format is
changing anyway and therefore something like a switch (is this what is
referred to as the C-8110 solution?) will take care of it for everyone.

>>>

>>>  
>
>>>

>>> I'm not sure there's consensus that such a switch is a sufficient
resolution to this discussion, but if there were such a consensus, the next
question would be whether the patches that are otherwise ready now can merge,
or whether they will all be blocked waiting for the compatibility solution
first. And possibly better testing, etc. Letting them merge would be justified
by the desire to have more frequent and smaller increments of work merged into
trunk... well, I'm not going to repeat everything from that discussion but the
same pro's and con's apply.

>>>

>>>  
>
>>>

>>> henrik

>>>

>>> \--  
>
>>>

>>> ![](https://lh5.googleusercontent.com/UwlCp-
Ixn21QzYv9oNnaGy0cKfFk1ukEBVKSv4V3-nQShsR-
cib_VeSuNm4M_xZxyAzTTr0Et7MsQuTDhUGcmWQyfVP801Flif-
SGT2x38lFRGkgoMUB4cot1DB9xd7Y0x2P0wJWA-
gQ5k4rzytFSoLCP4wJntmJzhlqTuQQsOanCBHeejtSBcBry5v6kw)

>>>

>>> |

>>>

>>> Henrik Ingo

>>>

>>> c. +358 40 569 7354

>>>

>>> w. [www.datastax.com](http://www.datastax.com)

>>>

>>> [![](https://lh3.googleusercontent.com/T6MEp9neZySKd-eg-tkz96Yf4qG_Xsgu-
IznDkdHfsHCjAnnHQP6OsPCdj8rsDvgKs-GJS6TA7Yx5HlK-
zfRlE64j0zDpDG9cI29VaG948x5xLgUU4KKctaHNAhbpJ_pDwzRag9K7yCibGblB5Ix5z6Xj99Vc92V9nYSmR4HIj5F9T_TVI7ayW2n2_lp5Q)](https://urldefense.com/v3/__https://www.facebook.com/datastax__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n0n_sPsT$)
[![](https://lh3.googleusercontent.com/Xrju2UthJiMtMS5jFknV8AhVO45tfhXSR6U0F8Qam1Mu2taE2SeVcl5ExaxU5l6pG0fHjv2b6vvUOe12WQldMqsOHknC7wQtBVYiX9ff3fLMtFAbjVRM0MGTKvPsjAcMI_FNvcIcuWIBP_zwRuh3b3g6hjHOW0ik9bDPuuYMvdLWIF8C8YgKDYQ-
nV9dlQ)](https://twitter.com/datastax)
[![](https://lh5.googleusercontent.com/OS41kMrzmJhmkvdmkHU-
pq69Nzy1tOz36NIwGs61oz9cGj42TTggsXk58MY1Lqn5FyIK77jedKh3UN-1RMCgCqduMQeUNU5fVKjCBNvSOpp6NjBLZp-2NMypQnw7JoyPoeI_iXfygfzquE89GLoel7Tiq1Jtz6ueaaVA9goEhUn2rWIJMQ28DPrEj4xqfg)](https://urldefense.com/v3/__https://www.linkedin.com/company/datastax/__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n2bcAuFd$)
[![](https://lh3.googleusercontent.com/AVBOsupbzcVirw6fNWEIerGj-
CT9XuzDmGpAa5KimxCLGAICw7_TV040RngH0I_0Z9ZEWERsQOiCSqD4ORslxJdqFiuFc1qgIoA9QMXW_yRklRJrrrCO0rQ47Hvt9QtfAz7swR_Vn6N8wZPYE2APUJAo-
oB_X71omearuZFBjL9VKbhbrZXn9HQ7aGSxIA)](https://github.com/datastax/)

>>>

>>>  
>  
>>> ---|---  
>  
>  
>
>
>  
>
>
> \--  
>
>
> ![](https://lh5.googleusercontent.com/UwlCp-
> Ixn21QzYv9oNnaGy0cKfFk1ukEBVKSv4V3-nQShsR-
> cib_VeSuNm4M_xZxyAzTTr0Et7MsQuTDhUGcmWQyfVP801Flif-
> SGT2x38lFRGkgoMUB4cot1DB9xd7Y0x2P0wJWA-
> gQ5k4rzytFSoLCP4wJntmJzhlqTuQQsOanCBHeejtSBcBry5v6kw)
>
> |
>
> Henrik Ingo
>
> c. +358 40 569 7354
>
> w. [www.datastax.com](http://www.datastax.com)
>
> [![](https://lh3.googleusercontent.com/T6MEp9neZySKd-eg-tkz96Yf4qG_Xsgu-
> IznDkdHfsHCjAnnHQP6OsPCdj8rsDvgKs-GJS6TA7Yx5HlK-
> zfRlE64j0zDpDG9cI29VaG948x5xLgUU4KKctaHNAhbpJ_pDwzRag9K7yCibGblB5Ix5z6Xj99Vc92V9nYSmR4HIj5F9T_TVI7ayW2n2_lp5Q)](https://www.facebook.com/datastax)
> [![](https://lh3.googleusercontent.com/Xrju2UthJiMtMS5jFknV8AhVO45tfhXSR6U0F8Qam1Mu2taE2SeVcl5ExaxU5l6pG0fHjv2b6vvUOe12WQldMqsOHknC7wQtBVYiX9ff3fLMtFAbjVRM0MGTKvPsjAcMI_FNvcIcuWIBP_zwRuh3b3g6hjHOW0ik9bDPuuYMvdLWIF8C8YgKDYQ-
> nV9dlQ)](https://twitter.com/datastax)
> [![](https://lh5.googleusercontent.com/OS41kMrzmJhmkvdmkHU-
> pq69Nzy1tOz36NIwGs61oz9cGj42TTggsXk58MY1Lqn5FyIK77jedKh3UN-1RMCgCqduMQeUNU5fVKjCBNvSOpp6NjBLZp-2NMypQnw7JoyPoeI_iXfygfzquE89GLoel7Tiq1Jtz6ueaaVA9goEhUn2rWIJMQ28DPrEj4xqfg)](https://www.linkedin.com/company/datastax/)
> [![](https://lh3.googleusercontent.com/AVBOsupbzcVirw6fNWEIerGj-
> CT9XuzDmGpAa5KimxCLGAICw7_TV040RngH0I_0Z9ZEWERsQOiCSqD4ORslxJdqFiuFc1qgIoA9QMXW_yRklRJrrrCO0rQ47Hvt9QtfAz7swR_Vn6N8wZPYE2APUJAo-
> oB_X71omearuZFBjL9VKbhbrZXn9HQ7aGSxIA)](https://github.com/datastax/)
>
>  
>  
> ---|---


Re: Downgradability

Posted by Henrik Ingo <he...@datastax.com>.
Right. So I'm speculating everyone else who worked on a patch that breaks
compatibility has been working under the mindset "I'll just put this behind
the same switch". Or something more vague / even less correct, such as
assuming that tries would become the default immediately.

At least in my mind when we talk about the "switch to enable tries" I do
also consider things like "don't break streaming". So I guess whether one
considers "a switch" to exist already or not, might be subjective in this
case, because people have different assumptions on the definition of done
of such a switch.

henrik

On Thu, Feb 23, 2023 at 2:53 PM Benedict <be...@apache.org> wrote:

> I don’t think there’s anything about a new format that requires a version
> bump, but I could be missing something.
>
> We have to have a switch to enable tries already don’t we? I’m pretty sure
> we haven’t talked about switching the default format?
>
> On 23 Feb 2023, at 12:12, Henrik Ingo <he...@datastax.com> wrote:
>
> 
> On Thu, Feb 23, 2023 at 11:57 AM Benedict <be...@apache.org> wrote:
>
>> Can somebody explain to me why this is being fought tooth and nail, when
>> the work involved is absolutely minimal?
>>
>>
> I don't know how each individual has been thinking about this, but it
> seems to me just looking at all the tasks that at least the introduction of
> tries is a major format change anyway - since it's the whole point - and
> therefore people working on other tasks may have assumed the format is
> changing anyway and therefore something like a switch (is this what is
> referred to as the C-8110 solution?) will take care of it for everyone.
>
> I'm not sure there's consensus that such a switch is a sufficient
> resolution to this discussion, but if there were such a consensus, the next
> question would be whether the patches that are otherwise ready now can
> merge, or whether they will all be blocked waiting for the compatibility
> solution first. And possibly better testing, etc. Letting them merge would
> be justified by the desire to have more frequent and smaller increments of
> work merged into trunk... well, I'm not going to repeat everything from
> that discussion but the same pro's and con's apply.
>
> henrik
> --
>
> Henrik Ingo
>
> c. +358 40 569 7354
>
> w. www.datastax.com
>
>
> <https://urldefense.com/v3/__https://www.facebook.com/datastax__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n0n_sPsT$>
> <https://twitter.com/datastax>
> <https://urldefense.com/v3/__https://www.linkedin.com/company/datastax/__;!!PbtH5S7Ebw!dOQqeDGZHgRdaV7zT4J-u7QGa4b2HCSNBgF8KrDldGjvy_guOGUws3L2sV2X5y_vzNYF7iZ85aa0n2bcAuFd$>
> <https://github.com/datastax/>
>
>

-- 

Henrik Ingo

c. +358 40 569 7354

w. www.datastax.com

<https://www.facebook.com/datastax>  <https://twitter.com/datastax>
<https://www.linkedin.com/company/datastax/>  <https://github.com/datastax/>

Re: Downgradability

Posted by Benedict <be...@apache.org>.
I don’t think there’s anything about a new format that requires a version
bump, but I could be missing something.

  

We have to have a switch to enable tries already don’t we? I’m pretty sure we
haven’t talked about switching the default format?

  

> On 23 Feb 2023, at 12:12, Henrik Ingo <he...@datastax.com> wrote:  
>  
>

> 
>
> On Thu, Feb 23, 2023 at 11:57 AM Benedict
> <[benedict@apache.org](mailto:benedict@apache.org)> wrote:  
>
>

>> Can somebody explain to me why this is being fought tooth and nail, when
the work involved is absolutely minimal?  
>
>>

>>  
>
>
>  
>
>
> I don't know how each individual has been thinking about this, but it seems
> to me just looking at all the tasks that at least the introduction of tries
> is a major format change anyway - since it's the whole point - and therefore
> people working on other tasks may have assumed the format is changing anyway
> and therefore something like a switch (is this what is referred to as the
> C-8110 solution?) will take care of it for everyone.
>
>  
>
>
> I'm not sure there's consensus that such a switch is a sufficient resolution
> to this discussion, but if there were such a consensus, the next question
> would be whether the patches that are otherwise ready now can merge, or
> whether they will all be blocked waiting for the compatibility solution
> first. And possibly better testing, etc. Letting them merge would be
> justified by the desire to have more frequent and smaller increments of work
> merged into trunk... well, I'm not going to repeat everything from that
> discussion but the same pro's and con's apply.
>
>  
>
>
> henrik
>
> \--  
>
>
> ![](https://lh5.googleusercontent.com/UwlCp-
> Ixn21QzYv9oNnaGy0cKfFk1ukEBVKSv4V3-nQShsR-
> cib_VeSuNm4M_xZxyAzTTr0Et7MsQuTDhUGcmWQyfVP801Flif-
> SGT2x38lFRGkgoMUB4cot1DB9xd7Y0x2P0wJWA-
> gQ5k4rzytFSoLCP4wJntmJzhlqTuQQsOanCBHeejtSBcBry5v6kw)
>
> |
>
> Henrik Ingo
>
> c. +358 40 569 7354
>
> w. [www.datastax.com](http://www.datastax.com)
>
> [![](https://lh3.googleusercontent.com/T6MEp9neZySKd-eg-tkz96Yf4qG_Xsgu-
> IznDkdHfsHCjAnnHQP6OsPCdj8rsDvgKs-GJS6TA7Yx5HlK-
> zfRlE64j0zDpDG9cI29VaG948x5xLgUU4KKctaHNAhbpJ_pDwzRag9K7yCibGblB5Ix5z6Xj99Vc92V9nYSmR4HIj5F9T_TVI7ayW2n2_lp5Q)](https://www.facebook.com/datastax)
> [![](https://lh3.googleusercontent.com/Xrju2UthJiMtMS5jFknV8AhVO45tfhXSR6U0F8Qam1Mu2taE2SeVcl5ExaxU5l6pG0fHjv2b6vvUOe12WQldMqsOHknC7wQtBVYiX9ff3fLMtFAbjVRM0MGTKvPsjAcMI_FNvcIcuWIBP_zwRuh3b3g6hjHOW0ik9bDPuuYMvdLWIF8C8YgKDYQ-
> nV9dlQ)](https://twitter.com/datastax)
> [![](https://lh5.googleusercontent.com/OS41kMrzmJhmkvdmkHU-
> pq69Nzy1tOz36NIwGs61oz9cGj42TTggsXk58MY1Lqn5FyIK77jedKh3UN-1RMCgCqduMQeUNU5fVKjCBNvSOpp6NjBLZp-2NMypQnw7JoyPoeI_iXfygfzquE89GLoel7Tiq1Jtz6ueaaVA9goEhUn2rWIJMQ28DPrEj4xqfg)](https://www.linkedin.com/company/datastax/)
> [![](https://lh3.googleusercontent.com/AVBOsupbzcVirw6fNWEIerGj-
> CT9XuzDmGpAa5KimxCLGAICw7_TV040RngH0I_0Z9ZEWERsQOiCSqD4ORslxJdqFiuFc1qgIoA9QMXW_yRklRJrrrCO0rQ47Hvt9QtfAz7swR_Vn6N8wZPYE2APUJAo-
> oB_X71omearuZFBjL9VKbhbrZXn9HQ7aGSxIA)](https://github.com/datastax/)
>
>  
>  
> ---|---


Re: Downgradability

Posted by Henrik Ingo <he...@datastax.com>.
On Thu, Feb 23, 2023 at 11:57 AM Benedict <be...@apache.org> wrote:

> Can somebody explain to me why this is being fought tooth and nail, when
> the work involved is absolutely minimal?
>
>
I don't know how each individual has been thinking about this, but it seems
to me just looking at all the tasks that at least the introduction of tries
is a major format change anyway - since it's the whole point - and
therefore people working on other tasks may have assumed the format is
changing anyway and therefore something like a switch (is this what is
referred to as the C-8110 solution?) will take care of it for everyone.

I'm not sure there's consensus that such a switch is a sufficient
resolution to this discussion, but if there were such a consensus, the next
question would be whether the patches that are otherwise ready now can
merge, or whether they will all be blocked waiting for the compatibility
solution first. And possibly better testing, etc. Letting them merge would
be justified by the desire to have more frequent and smaller increments of
work merged into trunk... well, I'm not going to repeat everything from
that discussion but the same pro's and con's apply.

henrik
-- 

Henrik Ingo

c. +358 40 569 7354

w. www.datastax.com

<https://www.facebook.com/datastax>  <https://twitter.com/datastax>
<https://www.linkedin.com/company/datastax/>  <https://github.com/datastax/>

Re: Downgradability

Posted by Benedict <be...@apache.org>.
Forget downgradeability for a moment: we should not be breaking format
compatibility without good reason. Bumping a major version isn’t enough of a
reason.

  

Can somebody explain to me why this is being fought tooth and nail, when the
work involved is absolutely minimal?

  

Regarding tests: what more do you want, than running our upgrade suite
backwards?

  

  

> On 23 Feb 2023, at 09:45, Benjamin Lerer <bl...@apache.org> wrote:  
>  
>

> 
>

>> Can somebody explain to me what is so burdensome, that we seem to be
spending longer debating it than it would take to implement the necessary
changes?

>
>  
>
>
> I believe that we all agree on the outcome. Everybody wants downgradability.
> The issue is on the path to get there.
>
>  
>
>
> As far as I am concerned, I would like to see a proper solution and as Jeff
> suggested the equivalent of the upgrade tests as gatekeepers. Having
> everybody trying to enforce it on his own way will only lead to a poor
> result in my opinion with some addoc code that might not really guarantee
> real downgradability in the end.  
>
>
> We have rushed in the past to get feature outs and pay the price for it. I
> simply prefer that we take the time to do things right.
>
>  
>
>
> Thanks to Scott and you, downgradability got a much better visibility so no
> matter what approach we pick, I am convinced that we will get there.  
>
>
>  
>
>
> Le jeu. 23 févr. 2023 à 09:49, Claude Warren, Jr via dev
> <[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)> a écrit :  
>
>

>> Broken downgrading can be fixed (I think) by modifying the
SearializationHeader.toHeader() method where it currently throws an
UnknownColumnException.  If we can, instead of throwing the exception, create
a dropped column for the unexpected column then I think the code will work.

>>

>>  
>
>>

>> I realise that to do this in the wild is not possible as it is a change to
released code, but we could handle it going forward.

>>

>>  
>
>>

>> On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo
<[henrik.ingo@datastax.com](mailto:henrik.ingo@datastax.com)> wrote:  
>
>>

>>> ... ok apparently shift+enter  sends messages now?

>>>

>>>  
>
>>>

>>> I was just saying if at least the file format AND system/tables - anything
written to disk - can be protected with a switch, then it allows for quick
downgrade by shutting down the entire cluster and restarting with the
downgraded binary. It's a start.

>>>

>>>  
>
>>>

>>> To be able to do that live in a distributed system needs to consider much
more: gossip, streaming, drivers, and ultimately all features, because we
don't' want an application developer to use a shiny new thing that a) may not
be available on all nodes, or b) may disappear if the cluster has to be
downgraded later.

>>>

>>>  
>
>>>

>>> henrik

>>>

>>>  
>
>>>

>>> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo
<[henrik.ingo@datastax.com](mailto:henrik.ingo@datastax.com)> wrote:  
>
>>>

>>>> Just this once I'm going to be really brief :-)

>>>>

>>>>  
>
>>>>

>>>> Just wanted to share for reference how Mongodb implemented
downgradeability around their 4.4 version:
<https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-
cluster/>

>>>>

>>>>  
>
>>>>

>>>> Jeff you're right. Ultimately this is about more than file formats.
However, ideally if at least the

>>>>

>>>>  
>
>>>>

>>>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa
<[jjirsa@gmail.com](mailto:jjirsa@gmail.com)> wrote:  
>
>>>>

>>>>> I'm not even convinced even 8110 addresses this - just writing sstables
in old versions won't help if we ever add things like new types or new types
of collections without other control abilities. Claude's other email in
another thread a few hours ago talks about some of these surprises -
"Specifically during the 3.1 -> 4.0 changes a column broadcast_port was added
to system/local.  This means that 3.1 system can not read the table as it has
no definition for it.  I tried marking the column for deletion in the metadata
and in the serialization header.  The later got past the column not found
problem, but I suspect that it just means that data columns after
broadcast_port shifted and so incorrectly read." - this is a harder problem to
solve than just versioning sstables and network protocols.  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> Stepping back a bit, we have downgrade ability listed as a goal, but
it's not (as far as I can tell) universally enforced, nor is it clear at which
point we will be able to concretely say "this release can be downgraded to X".
Until we actually define and agree that this is a real goal with a concrete
version where downgrade-ability becomes real, it feels like things are
somewhat arbitrarily enforced, which is probably very frustrating for people
trying to commit work/tickets.

>>>>>

>>>>>  
>
>>>>>

>>>>> \- Jeff

>>>>>

>>>>>  
>
>>>>>

>>>>>  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi
<[djoshi@apache.org](mailto:djoshi@apache.org)> wrote:  
>
>>>>>

>>>>>> I’m a big fan of maintaining backward compatibility. Downgradability
implies that we could potentially roll back an upgrade at any time. While I
don’t think we need to retain the ability to downgrade in perpetuity it would
be a good objective to maintain strict backward compatibility and therefore
downgradability until a certain point. This would imply versioning metadata
and extending it in such a way that prior version(s) could continue
functioning. This can certainly be expensive to implement and might bloat on-
disk storage. However, we could always offer an option for the operator to
optimize the on-disk structures for the current version then we can rewrite
them in the latest version. This optimizes the storage and opens up new
functionality. This means new features that can work with old on-disk
structures will be available while others that strictly require new versions
of the data structures will be unavailable until the operator migrates to the
new version. This migration IMO should be irreversible. Beyond this point the
operator will lose the ability to downgrade which is ok.  
>  
>
>>>>>>

>>>>>> Dinesh

>>>>>>

>>>>>>  
>
>>>>>>

>>>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani
<[jakers@gmail.com](mailto:jakers@gmail.com)> wrote:  
>  
>
>>>>>>

>>>>>>> 

>>>>>>>

>>>>>>> There has been progress on

>>>>>>>

>>>>>>>
<https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928>

>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> Which is similar to what datastax does for DSE. Would this be an
acceptable solution?

>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> Jake

>>>>>>>

>>>>>>>  
>
>>>>>>>

>>>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell
<[cclive1601@gmail.com](mailto:cclive1601@gmail.com)> wrote:  
>
>>>>>>>

>>>>>>>> It seems “An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110)” can
give us more options if it is finished😉

>>>>>>>>

>>>>>>>>  
>
>>>>>>>>

>>>>>>>> Branimir Lambov
<[blambov@apache.org](mailto:blambov@apache.org)>于2023年2月20日 周一下午11:03写道:  
>
>>>>>>>>

>>>>>>>>> Hi everyone,  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> There has been a discussion lately about changes to the sstable
format in the context of being able to abort a cluster upgrade, and the fact
that changes to sstables can prevent downgraded nodes from reading any data
written during their temporary operation with the new version.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Most of the discussion is in
[CASSANDRA-18134](https://issues.apache.org/jira/browse/CASSANDRA-18134), and
is spreading into
[CASSANDRA-14277](https://issues.apache.org/jira/browse/CASSANDRA-14227) and
[CASSANDRA-17698](https://issues.apache.org/jira/browse/CASSANDRA-17698), none
of which is a good place to discuss the topic seriously.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Downgradability is a worthy goal and is listed in the current
roadmap. I would like to open a discussion here on how it would be achieved.  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> My understanding of what has been suggested so far translates to:

>>>>>>>>>

>>>>>>>>> \- avoid changes to sstable formats;

>>>>>>>>>

>>>>>>>>> \- if there are changes, implement them in a way that is backwards-
compatible, e.g. by duplicating data, so that a new version is presented in a
component or portion of a component that legacy nodes will not try to read;

>>>>>>>>>

>>>>>>>>> \- if the latter is not feasible, make sure the changes are only
applied if a feature flag has been enabled.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> To me this approach introduces several risks:

>>>>>>>>>

>>>>>>>>> \- it bloats file and parsing complexity;

>>>>>>>>>

>>>>>>>>> \- it discourages improvement (e.g. CASSANDRA-17698 is no longer a
LHF ticket once this requirement is in place);

>>>>>>>>>

>>>>>>>>> \- it needs care to avoid risky solutions to address technical
issues with the format versioning (e.g. staying on n-versions for 5.0 and
needing a bump for a 4.1 bugfix might require porting over support for new
features);  
>
>>>>>>>>>

>>>>>>>>> \- it requires separate and uncoordinated solutions to the problem
and switching mechanisms for each individual change.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110), which
provides a method of writing sstables for a target version. During upgrades, a
node could be set to produce sstables corresponding to the older version, and
there is a very straightforward way to implement modifications to formats like
the tickets above to conform to its requirements.

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> What do people think should be the way forward?

>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>> Regards,

>>>>>>>>>

>>>>>>>>> Branimir  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>>

>>>>>>>>>  
>
>>>>>>>>

>>>>>>>> \--  
>
>>>>>>>>

>>>>>>>> you are the apple of my eye !

>>>>>>>

>>>>>>> \--  
>
>>>>>>>

>>>>>>> <http://twitter.com/tjake>

>>>>

>>>>  
>
>>>>

>>>>  
>
>>>>

>>>> \--  
>
>>>>

>>>> ![](https://lh5.googleusercontent.com/UwlCp-
Ixn21QzYv9oNnaGy0cKfFk1ukEBVKSv4V3-nQShsR-
cib_VeSuNm4M_xZxyAzTTr0Et7MsQuTDhUGcmWQyfVP801Flif-
SGT2x38lFRGkgoMUB4cot1DB9xd7Y0x2P0wJWA-
gQ5k4rzytFSoLCP4wJntmJzhlqTuQQsOanCBHeejtSBcBry5v6kw)

>>>>

>>>> |

>>>>

>>>> Henrik Ingo

>>>>

>>>> c. +358 40 569 7354

>>>>

>>>> w. [www.datastax.com](http://www.datastax.com)

>>>>

>>>> [![](https://lh3.googleusercontent.com/T6MEp9neZySKd-eg-tkz96Yf4qG_Xsgu-
IznDkdHfsHCjAnnHQP6OsPCdj8rsDvgKs-GJS6TA7Yx5HlK-
zfRlE64j0zDpDG9cI29VaG948x5xLgUU4KKctaHNAhbpJ_pDwzRag9K7yCibGblB5Ix5z6Xj99Vc92V9nYSmR4HIj5F9T_TVI7ayW2n2_lp5Q)](https://www.facebook.com/datastax)
[![](https://lh3.googleusercontent.com/Xrju2UthJiMtMS5jFknV8AhVO45tfhXSR6U0F8Qam1Mu2taE2SeVcl5ExaxU5l6pG0fHjv2b6vvUOe12WQldMqsOHknC7wQtBVYiX9ff3fLMtFAbjVRM0MGTKvPsjAcMI_FNvcIcuWIBP_zwRuh3b3g6hjHOW0ik9bDPuuYMvdLWIF8C8YgKDYQ-
nV9dlQ)](https://twitter.com/datastax)
[![](https://lh5.googleusercontent.com/OS41kMrzmJhmkvdmkHU-
pq69Nzy1tOz36NIwGs61oz9cGj42TTggsXk58MY1Lqn5FyIK77jedKh3UN-1RMCgCqduMQeUNU5fVKjCBNvSOpp6NjBLZp-2NMypQnw7JoyPoeI_iXfygfzquE89GLoel7Tiq1Jtz6ueaaVA9goEhUn2rWIJMQ28DPrEj4xqfg)](https://www.linkedin.com/company/datastax/)
[![](https://lh3.googleusercontent.com/AVBOsupbzcVirw6fNWEIerGj-
CT9XuzDmGpAa5KimxCLGAICw7_TV040RngH0I_0Z9ZEWERsQOiCSqD4ORslxJdqFiuFc1qgIoA9QMXW_yRklRJrrrCO0rQ47Hvt9QtfAz7swR_Vn6N8wZPYE2APUJAo-
oB_X71omearuZFBjL9VKbhbrZXn9HQ7aGSxIA)](https://github.com/datastax/)

>>>>

>>>>  
>  
>>>> ---|---  
>>>  
>>>  
>
>>>

>>>  
>
>>>

>>> \--  
>
>>>

>>> ![](https://lh5.googleusercontent.com/UwlCp-
Ixn21QzYv9oNnaGy0cKfFk1ukEBVKSv4V3-nQShsR-
cib_VeSuNm4M_xZxyAzTTr0Et7MsQuTDhUGcmWQyfVP801Flif-
SGT2x38lFRGkgoMUB4cot1DB9xd7Y0x2P0wJWA-
gQ5k4rzytFSoLCP4wJntmJzhlqTuQQsOanCBHeejtSBcBry5v6kw)

>>>

>>> |

>>>

>>> Henrik Ingo

>>>

>>> c. +358 40 569 7354

>>>

>>> w. [www.datastax.com](http://www.datastax.com)

>>>

>>> [![](https://lh3.googleusercontent.com/T6MEp9neZySKd-eg-tkz96Yf4qG_Xsgu-
IznDkdHfsHCjAnnHQP6OsPCdj8rsDvgKs-GJS6TA7Yx5HlK-
zfRlE64j0zDpDG9cI29VaG948x5xLgUU4KKctaHNAhbpJ_pDwzRag9K7yCibGblB5Ix5z6Xj99Vc92V9nYSmR4HIj5F9T_TVI7ayW2n2_lp5Q)](https://www.facebook.com/datastax)
[![](https://lh3.googleusercontent.com/Xrju2UthJiMtMS5jFknV8AhVO45tfhXSR6U0F8Qam1Mu2taE2SeVcl5ExaxU5l6pG0fHjv2b6vvUOe12WQldMqsOHknC7wQtBVYiX9ff3fLMtFAbjVRM0MGTKvPsjAcMI_FNvcIcuWIBP_zwRuh3b3g6hjHOW0ik9bDPuuYMvdLWIF8C8YgKDYQ-
nV9dlQ)](https://twitter.com/datastax)
[![](https://lh5.googleusercontent.com/OS41kMrzmJhmkvdmkHU-
pq69Nzy1tOz36NIwGs61oz9cGj42TTggsXk58MY1Lqn5FyIK77jedKh3UN-1RMCgCqduMQeUNU5fVKjCBNvSOpp6NjBLZp-2NMypQnw7JoyPoeI_iXfygfzquE89GLoel7Tiq1Jtz6ueaaVA9goEhUn2rWIJMQ28DPrEj4xqfg)](https://www.linkedin.com/company/datastax/)
[![](https://lh3.googleusercontent.com/AVBOsupbzcVirw6fNWEIerGj-
CT9XuzDmGpAa5KimxCLGAICw7_TV040RngH0I_0Z9ZEWERsQOiCSqD4ORslxJdqFiuFc1qgIoA9QMXW_yRklRJrrrCO0rQ47Hvt9QtfAz7swR_Vn6N8wZPYE2APUJAo-
oB_X71omearuZFBjL9VKbhbrZXn9HQ7aGSxIA)](https://github.com/datastax/)

>>>

>>>  
>  
>>> ---|---


Re: Downgradability

Posted by Benjamin Lerer <bl...@apache.org>.
>
> Can somebody explain to me what is so burdensome, that we seem to be
> spending longer debating it than it would take to implement the necessary
> changes?


I believe that we all agree on the outcome. Everybody wants
downgradability. The issue is on the path to get there.

As far as I am concerned, I would like to see a proper solution and as Jeff
suggested the equivalent of the upgrade tests as gatekeepers. Having
everybody trying to enforce it on his own way will only lead to a poor
result in my opinion with some addoc code that might not really guarantee
real downgradability in the end.
We have rushed in the past to get feature outs and pay the price for it. I
simply prefer that we take the time to do things right.

Thanks to Scott and you, downgradability got a much better visibility so no
matter what approach we pick, I am convinced that we will get there.

Le jeu. 23 févr. 2023 à 09:49, Claude Warren, Jr via dev <
dev@cassandra.apache.org> a écrit :

> Broken downgrading can be fixed (I think) by modifying the
> SearializationHeader.toHeader() method where it currently throws an
> UnknownColumnException.  If we can, instead of throwing the exception,
> create a dropped column for the unexpected column then I think the code
> will work.
>
> I realise that to do this in the wild is not possible as it is a change to
> released code, but we could handle it going forward.
>
> On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo <he...@datastax.com>
> wrote:
>
>> ... ok apparently shift+enter  sends messages now?
>>
>> I was just saying if at least the file format AND system/tables -
>> anything written to disk - can be protected with a switch, then it allows
>> for quick downgrade by shutting down the entire cluster and restarting with
>> the downgraded binary. It's a start.
>>
>> To be able to do that live in a distributed system needs to consider much
>> more: gossip, streaming, drivers, and ultimately all features, because we
>> don't' want an application developer to use a shiny new thing that a) may
>> not be available on all nodes, or b) may disappear if the cluster has to be
>> downgraded later.
>>
>> henrik
>>
>> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo <he...@datastax.com>
>> wrote:
>>
>>> Just this once I'm going to be really brief :-)
>>>
>>> Just wanted to share for reference how Mongodb implemented
>>> downgradeability around their 4.4 version:
>>> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>>>
>>> Jeff you're right. Ultimately this is about more than file formats.
>>> However, ideally if at least the
>>>
>>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>>> I'm not even convinced even 8110 addresses this - just writing sstables
>>>> in old versions won't help if we ever add things like new types or new
>>>> types of collections without other control abilities. Claude's other email
>>>> in another thread a few hours ago talks about some of these surprises -
>>>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>>> added to system/local.  This means that 3.1 system can not read the table
>>>> as it has no definition for it.  I tried marking the column for deletion in
>>>> the metadata and in the serialization header.  The later got past the
>>>> column not found problem, but I suspect that it just means that data
>>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>>> harder problem to solve than just versioning sstables and network
>>>> protocols.
>>>>
>>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>>> which point we will be able to concretely say "this release can be
>>>> downgraded to X".   Until we actually define and agree that this is a
>>>> real goal with a concrete version where downgrade-ability becomes real, it
>>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>>> frustrating for people trying to commit work/tickets.
>>>>
>>>> - Jeff
>>>>
>>>>
>>>>
>>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org>
>>>> wrote:
>>>>
>>>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>>>> implies that we could potentially roll back an upgrade at any time. While I
>>>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>>>> would be a good objective to maintain strict backward compatibility and
>>>>> therefore downgradability until a certain point. This would imply
>>>>> versioning metadata and extending it in such a way that prior version(s)
>>>>> could continue functioning. This can certainly be expensive to implement
>>>>> and might bloat on-disk storage. However, we could always offer an option
>>>>> for the operator to optimize the on-disk structures for the current version
>>>>> then we can rewrite them in the latest version. This optimizes the storage
>>>>> and opens up new functionality. This means new features that can work with
>>>>> old on-disk structures will be available while others that strictly require
>>>>> new versions of the data structures will be unavailable until the operator
>>>>> migrates to the new version. This migration IMO should be irreversible.
>>>>> Beyond this point the operator will lose the ability to downgrade which is
>>>>> ok.
>>>>>
>>>>> Dinesh
>>>>>
>>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>>>
>>>>> 
>>>>> There has been progress on
>>>>>
>>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>>
>>>>> Which is similar to what datastax does for DSE. Would this be an
>>>>> acceptable solution?
>>>>>
>>>>> Jake
>>>>>
>>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> It seems “An alternative solution is to implement/complete
>>>>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>>>>> can give us more options if it is finished😉
>>>>>>
>>>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> There has been a discussion lately about changes to the sstable
>>>>>>> format in the context of being able to abort a cluster upgrade, and the
>>>>>>> fact that changes to sstables can prevent downgraded nodes from reading any
>>>>>>> data written during their temporary operation with the new version.
>>>>>>>
>>>>>>> Most of the discussion is in CASSANDRA-18134
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>>>> spreading into CASSANDRA-14277
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>>>> CASSANDRA-17698
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>>>> which is a good place to discuss the topic seriously.
>>>>>>>
>>>>>>> Downgradability is a worthy goal and is listed in the current
>>>>>>> roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>>
>>>>>>> My understanding of what has been suggested so far translates to:
>>>>>>> - avoid changes to sstable formats;
>>>>>>> - if there are changes, implement them in a way that is
>>>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>>>> presented in a component or portion of a component that legacy nodes will
>>>>>>> not try to read;
>>>>>>> - if the latter is not feasible, make sure the changes are only
>>>>>>> applied if a feature flag has been enabled.
>>>>>>>
>>>>>>> To me this approach introduces several risks:
>>>>>>> - it bloats file and parsing complexity;
>>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a
>>>>>>> LHF ticket once this requirement is in place);
>>>>>>> - it needs care to avoid risky solutions to address technical issues
>>>>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>>>>> features);
>>>>>>> - it requires separate and uncoordinated solutions to the problem
>>>>>>> and switching mechanisms for each individual change.
>>>>>>>
>>>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>>>> provides a method of writing sstables for a target version. During
>>>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>>>> older version, and there is a very straightforward way to implement
>>>>>>> modifications to formats like the tickets above to conform to its
>>>>>>> requirements.
>>>>>>>
>>>>>>> What do people think should be the way forward?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Branimir
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>> you are the apple of my eye !
>>>>>>
>>>>> --
>>>>> http://twitter.com/tjake
>>>>>
>>>>>
>>>
>>> --
>>>
>>> Henrik Ingo
>>>
>>> c. +358 40 569 7354
>>>
>>> w. www.datastax.com
>>>
>>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>>> <https://www.linkedin.com/company/datastax/>
>>> <https://github.com/datastax/>
>>>
>>>
>>
>> --
>>
>> Henrik Ingo
>>
>> c. +358 40 569 7354
>>
>> w. www.datastax.com
>>
>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>> <https://www.linkedin.com/company/datastax/>
>> <https://github.com/datastax/>
>>
>>

Re: Downgradability

Posted by "Claude Warren, Jr via dev" <de...@cassandra.apache.org>.
Broken downgrading can be fixed (I think) by modifying the
SearializationHeader.toHeader() method where it currently throws an
UnknownColumnException.  If we can, instead of throwing the exception,
create a dropped column for the unexpected column then I think the code
will work.

I realise that to do this in the wild is not possible as it is a change to
released code, but we could handle it going forward.

On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo <he...@datastax.com>
wrote:

> ... ok apparently shift+enter  sends messages now?
>
> I was just saying if at least the file format AND system/tables - anything
> written to disk - can be protected with a switch, then it allows for quick
> downgrade by shutting down the entire cluster and restarting with the
> downgraded binary. It's a start.
>
> To be able to do that live in a distributed system needs to consider much
> more: gossip, streaming, drivers, and ultimately all features, because we
> don't' want an application developer to use a shiny new thing that a) may
> not be available on all nodes, or b) may disappear if the cluster has to be
> downgraded later.
>
> henrik
>
> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo <he...@datastax.com>
> wrote:
>
>> Just this once I'm going to be really brief :-)
>>
>> Just wanted to share for reference how Mongodb implemented
>> downgradeability around their 4.4 version:
>> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>>
>> Jeff you're right. Ultimately this is about more than file formats.
>> However, ideally if at least the
>>
>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> I'm not even convinced even 8110 addresses this - just writing sstables
>>> in old versions won't help if we ever add things like new types or new
>>> types of collections without other control abilities. Claude's other email
>>> in another thread a few hours ago talks about some of these surprises -
>>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>> added to system/local.  This means that 3.1 system can not read the table
>>> as it has no definition for it.  I tried marking the column for deletion in
>>> the metadata and in the serialization header.  The later got past the
>>> column not found problem, but I suspect that it just means that data
>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>> harder problem to solve than just versioning sstables and network
>>> protocols.
>>>
>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>> it's not (as far as I can tell) universally enforced, nor is it clear at
>>> which point we will be able to concretely say "this release can be
>>> downgraded to X".   Until we actually define and agree that this is a
>>> real goal with a concrete version where downgrade-ability becomes real, it
>>> feels like things are somewhat arbitrarily enforced, which is probably very
>>> frustrating for people trying to commit work/tickets.
>>>
>>> - Jeff
>>>
>>>
>>>
>>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:
>>>
>>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>>> implies that we could potentially roll back an upgrade at any time. While I
>>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>>> would be a good objective to maintain strict backward compatibility and
>>>> therefore downgradability until a certain point. This would imply
>>>> versioning metadata and extending it in such a way that prior version(s)
>>>> could continue functioning. This can certainly be expensive to implement
>>>> and might bloat on-disk storage. However, we could always offer an option
>>>> for the operator to optimize the on-disk structures for the current version
>>>> then we can rewrite them in the latest version. This optimizes the storage
>>>> and opens up new functionality. This means new features that can work with
>>>> old on-disk structures will be available while others that strictly require
>>>> new versions of the data structures will be unavailable until the operator
>>>> migrates to the new version. This migration IMO should be irreversible.
>>>> Beyond this point the operator will lose the ability to downgrade which is
>>>> ok.
>>>>
>>>> Dinesh
>>>>
>>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>>
>>>> 
>>>> There has been progress on
>>>>
>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>>
>>>> Which is similar to what datastax does for DSE. Would this be an
>>>> acceptable solution?
>>>>
>>>> Jake
>>>>
>>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>>> wrote:
>>>>
>>>>> It seems “An alternative solution is to implement/complete
>>>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>>>> can give us more options if it is finished😉
>>>>>
>>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> There has been a discussion lately about changes to the sstable
>>>>>> format in the context of being able to abort a cluster upgrade, and the
>>>>>> fact that changes to sstables can prevent downgraded nodes from reading any
>>>>>> data written during their temporary operation with the new version.
>>>>>>
>>>>>> Most of the discussion is in CASSANDRA-18134
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>>> spreading into CASSANDRA-14277
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>>> CASSANDRA-17698
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>>> which is a good place to discuss the topic seriously.
>>>>>>
>>>>>> Downgradability is a worthy goal and is listed in the current
>>>>>> roadmap. I would like to open a discussion here on how it would be achieved.
>>>>>>
>>>>>> My understanding of what has been suggested so far translates to:
>>>>>> - avoid changes to sstable formats;
>>>>>> - if there are changes, implement them in a way that is
>>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>>> presented in a component or portion of a component that legacy nodes will
>>>>>> not try to read;
>>>>>> - if the latter is not feasible, make sure the changes are only
>>>>>> applied if a feature flag has been enabled.
>>>>>>
>>>>>> To me this approach introduces several risks:
>>>>>> - it bloats file and parsing complexity;
>>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>>>>> ticket once this requirement is in place);
>>>>>> - it needs care to avoid risky solutions to address technical issues
>>>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>>>> features);
>>>>>> - it requires separate and uncoordinated solutions to the problem and
>>>>>> switching mechanisms for each individual change.
>>>>>>
>>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>>> provides a method of writing sstables for a target version. During
>>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>>> older version, and there is a very straightforward way to implement
>>>>>> modifications to formats like the tickets above to conform to its
>>>>>> requirements.
>>>>>>
>>>>>> What do people think should be the way forward?
>>>>>>
>>>>>> Regards,
>>>>>> Branimir
>>>>>>
>>>>>>
>>>>>> --
>>>>> you are the apple of my eye !
>>>>>
>>>> --
>>>> http://twitter.com/tjake
>>>>
>>>>
>>
>> --
>>
>> Henrik Ingo
>>
>> c. +358 40 569 7354
>>
>> w. www.datastax.com
>>
>> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
>> <https://www.linkedin.com/company/datastax/>
>> <https://github.com/datastax/>
>>
>>
>
> --
>
> Henrik Ingo
>
> c. +358 40 569 7354
>
> w. www.datastax.com
>
> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
> <https://www.linkedin.com/company/datastax/>
> <https://github.com/datastax/>
>
>

Re: Downgradability

Posted by Henrik Ingo <he...@datastax.com>.
... ok apparently shift+enter  sends messages now?

I was just saying if at least the file format AND system/tables - anything
written to disk - can be protected with a switch, then it allows for quick
downgrade by shutting down the entire cluster and restarting with the
downgraded binary. It's a start.

To be able to do that live in a distributed system needs to consider much
more: gossip, streaming, drivers, and ultimately all features, because we
don't' want an application developer to use a shiny new thing that a) may
not be available on all nodes, or b) may disappear if the cluster has to be
downgraded later.

henrik

On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo <he...@datastax.com>
wrote:

> Just this once I'm going to be really brief :-)
>
> Just wanted to share for reference how Mongodb implemented
> downgradeability around their 4.4 version:
> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>
> Jeff you're right. Ultimately this is about more than file formats.
> However, ideally if at least the
>
> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> I'm not even convinced even 8110 addresses this - just writing sstables
>> in old versions won't help if we ever add things like new types or new
>> types of collections without other control abilities. Claude's other email
>> in another thread a few hours ago talks about some of these surprises -
>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>> added to system/local.  This means that 3.1 system can not read the table
>> as it has no definition for it.  I tried marking the column for deletion in
>> the metadata and in the serialization header.  The later got past the
>> column not found problem, but I suspect that it just means that data
>> columns after broadcast_port shifted and so incorrectly read." - this is a
>> harder problem to solve than just versioning sstables and network
>> protocols.
>>
>> Stepping back a bit, we have downgrade ability listed as a goal, but it's
>> not (as far as I can tell) universally enforced, nor is it clear at which
>> point we will be able to concretely say "this release can be downgraded to
>> X".   Until we actually define and agree that this is a real goal with a
>> concrete version where downgrade-ability becomes real, it feels like things
>> are somewhat arbitrarily enforced, which is probably very frustrating for
>> people trying to commit work/tickets.
>>
>> - Jeff
>>
>>
>>
>> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:
>>
>>> I’m a big fan of maintaining backward compatibility. Downgradability
>>> implies that we could potentially roll back an upgrade at any time. While I
>>> don’t think we need to retain the ability to downgrade in perpetuity it
>>> would be a good objective to maintain strict backward compatibility and
>>> therefore downgradability until a certain point. This would imply
>>> versioning metadata and extending it in such a way that prior version(s)
>>> could continue functioning. This can certainly be expensive to implement
>>> and might bloat on-disk storage. However, we could always offer an option
>>> for the operator to optimize the on-disk structures for the current version
>>> then we can rewrite them in the latest version. This optimizes the storage
>>> and opens up new functionality. This means new features that can work with
>>> old on-disk structures will be available while others that strictly require
>>> new versions of the data structures will be unavailable until the operator
>>> migrates to the new version. This migration IMO should be irreversible.
>>> Beyond this point the operator will lose the ability to downgrade which is
>>> ok.
>>>
>>> Dinesh
>>>
>>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>>
>>> 
>>> There has been progress on
>>>
>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>>
>>> Which is similar to what datastax does for DSE. Would this be an
>>> acceptable solution?
>>>
>>> Jake
>>>
>>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>>> wrote:
>>>
>>>> It seems “An alternative solution is to implement/complete
>>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>>> can give us more options if it is finished😉
>>>>
>>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> There has been a discussion lately about changes to the sstable format
>>>>> in the context of being able to abort a cluster upgrade, and the fact that
>>>>> changes to sstables can prevent downgraded nodes from reading any data
>>>>> written during their temporary operation with the new version.
>>>>>
>>>>> Most of the discussion is in CASSANDRA-18134
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>>> spreading into CASSANDRA-14277
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>>> CASSANDRA-17698
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17698>, none of
>>>>> which is a good place to discuss the topic seriously.
>>>>>
>>>>> Downgradability is a worthy goal and is listed in the current roadmap.
>>>>> I would like to open a discussion here on how it would be achieved.
>>>>>
>>>>> My understanding of what has been suggested so far translates to:
>>>>> - avoid changes to sstable formats;
>>>>> - if there are changes, implement them in a way that is
>>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>>> presented in a component or portion of a component that legacy nodes will
>>>>> not try to read;
>>>>> - if the latter is not feasible, make sure the changes are only
>>>>> applied if a feature flag has been enabled.
>>>>>
>>>>> To me this approach introduces several risks:
>>>>> - it bloats file and parsing complexity;
>>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>>>> ticket once this requirement is in place);
>>>>> - it needs care to avoid risky solutions to address technical issues
>>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>>> features);
>>>>> - it requires separate and uncoordinated solutions to the problem and
>>>>> switching mechanisms for each individual change.
>>>>>
>>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which
>>>>> provides a method of writing sstables for a target version. During
>>>>> upgrades, a node could be set to produce sstables corresponding to the
>>>>> older version, and there is a very straightforward way to implement
>>>>> modifications to formats like the tickets above to conform to its
>>>>> requirements.
>>>>>
>>>>> What do people think should be the way forward?
>>>>>
>>>>> Regards,
>>>>> Branimir
>>>>>
>>>>>
>>>>> --
>>>> you are the apple of my eye !
>>>>
>>> --
>>> http://twitter.com/tjake
>>>
>>>
>
> --
>
> Henrik Ingo
>
> c. +358 40 569 7354
>
> w. www.datastax.com
>
> <https://www.facebook.com/datastax>  <https://twitter.com/datastax>
> <https://www.linkedin.com/company/datastax/>
> <https://github.com/datastax/>
>
>

-- 

Henrik Ingo

c. +358 40 569 7354

w. www.datastax.com

<https://www.facebook.com/datastax>  <https://twitter.com/datastax>
<https://www.linkedin.com/company/datastax/>  <https://github.com/datastax/>

Re: Downgradability

Posted by Henrik Ingo <he...@datastax.com>.
Just this once I'm going to be really brief :-)

Just wanted to share for reference how Mongodb implemented downgradeability
around their 4.4 version:
https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/

Jeff you're right. Ultimately this is about more than file formats.
However, ideally if at least the

On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa <jj...@gmail.com> wrote:

> I'm not even convinced even 8110 addresses this - just writing sstables in
> old versions won't help if we ever add things like new types or new types
> of collections without other control abilities. Claude's other email in
> another thread a few hours ago talks about some of these surprises -
> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
> added to system/local.  This means that 3.1 system can not read the table
> as it has no definition for it.  I tried marking the column for deletion in
> the metadata and in the serialization header.  The later got past the
> column not found problem, but I suspect that it just means that data
> columns after broadcast_port shifted and so incorrectly read." - this is a
> harder problem to solve than just versioning sstables and network
> protocols.
>
> Stepping back a bit, we have downgrade ability listed as a goal, but it's
> not (as far as I can tell) universally enforced, nor is it clear at which
> point we will be able to concretely say "this release can be downgraded to
> X".   Until we actually define and agree that this is a real goal with a
> concrete version where downgrade-ability becomes real, it feels like things
> are somewhat arbitrarily enforced, which is probably very frustrating for
> people trying to commit work/tickets.
>
> - Jeff
>
>
>
> On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:
>
>> I’m a big fan of maintaining backward compatibility. Downgradability
>> implies that we could potentially roll back an upgrade at any time. While I
>> don’t think we need to retain the ability to downgrade in perpetuity it
>> would be a good objective to maintain strict backward compatibility and
>> therefore downgradability until a certain point. This would imply
>> versioning metadata and extending it in such a way that prior version(s)
>> could continue functioning. This can certainly be expensive to implement
>> and might bloat on-disk storage. However, we could always offer an option
>> for the operator to optimize the on-disk structures for the current version
>> then we can rewrite them in the latest version. This optimizes the storage
>> and opens up new functionality. This means new features that can work with
>> old on-disk structures will be available while others that strictly require
>> new versions of the data structures will be unavailable until the operator
>> migrates to the new version. This migration IMO should be irreversible.
>> Beyond this point the operator will lose the ability to downgrade which is
>> ok.
>>
>> Dinesh
>>
>> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>>
>> 
>> There has been progress on
>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>>
>> Which is similar to what datastax does for DSE. Would this be an
>> acceptable solution?
>>
>> Jake
>>
>> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com>
>> wrote:
>>
>>> It seems “An alternative solution is to implement/complete
>>> CASSANDRA-8110 <https://issues.apache.org/jira/browse/CASSANDRA-8110>”
>>> can give us more options if it is finished😉
>>>
>>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>>
>>>> Hi everyone,
>>>>
>>>> There has been a discussion lately about changes to the sstable format
>>>> in the context of being able to abort a cluster upgrade, and the fact that
>>>> changes to sstables can prevent downgraded nodes from reading any data
>>>> written during their temporary operation with the new version.
>>>>
>>>> Most of the discussion is in CASSANDRA-18134
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>>> spreading into CASSANDRA-14277
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>>> CASSANDRA-17698 <https://issues.apache.org/jira/browse/CASSANDRA-17698>,
>>>> none of which is a good place to discuss the topic seriously.
>>>>
>>>> Downgradability is a worthy goal and is listed in the current roadmap.
>>>> I would like to open a discussion here on how it would be achieved.
>>>>
>>>> My understanding of what has been suggested so far translates to:
>>>> - avoid changes to sstable formats;
>>>> - if there are changes, implement them in a way that is
>>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>>> presented in a component or portion of a component that legacy nodes will
>>>> not try to read;
>>>> - if the latter is not feasible, make sure the changes are only applied
>>>> if a feature flag has been enabled.
>>>>
>>>> To me this approach introduces several risks:
>>>> - it bloats file and parsing complexity;
>>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>>> ticket once this requirement is in place);
>>>> - it needs care to avoid risky solutions to address technical issues
>>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>>> a bump for a 4.1 bugfix might require porting over support for new
>>>> features);
>>>> - it requires separate and uncoordinated solutions to the problem and
>>>> switching mechanisms for each individual change.
>>>>
>>>> An alternative solution is to implement/complete CASSANDRA-8110
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which provides
>>>> a method of writing sstables for a target version. During upgrades, a node
>>>> could be set to produce sstables corresponding to the older version, and
>>>> there is a very straightforward way to implement modifications to formats
>>>> like the tickets above to conform to its requirements.
>>>>
>>>> What do people think should be the way forward?
>>>>
>>>> Regards,
>>>> Branimir
>>>>
>>>>
>>>> --
>>> you are the apple of my eye !
>>>
>> --
>> http://twitter.com/tjake
>>
>>

-- 

Henrik Ingo

c. +358 40 569 7354

w. www.datastax.com

<https://www.facebook.com/datastax>  <https://twitter.com/datastax>
<https://www.linkedin.com/company/datastax/>  <https://github.com/datastax/>

Re: Downgradability

Posted by Jeff Jirsa <jj...@gmail.com>.
I'm not even convinced even 8110 addresses this - just writing sstables in
old versions won't help if we ever add things like new types or new types
of collections without other control abilities. Claude's other email in
another thread a few hours ago talks about some of these surprises -
"Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
added to system/local.  This means that 3.1 system can not read the table
as it has no definition for it.  I tried marking the column for deletion in
the metadata and in the serialization header.  The later got past the
column not found problem, but I suspect that it just means that data
columns after broadcast_port shifted and so incorrectly read." - this is a
harder problem to solve than just versioning sstables and network
protocols.

Stepping back a bit, we have downgrade ability listed as a goal, but it's
not (as far as I can tell) universally enforced, nor is it clear at which
point we will be able to concretely say "this release can be downgraded to
X".   Until we actually define and agree that this is a real goal with a
concrete version where downgrade-ability becomes real, it feels like things
are somewhat arbitrarily enforced, which is probably very frustrating for
people trying to commit work/tickets.

- Jeff



On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi <dj...@apache.org> wrote:

> I’m a big fan of maintaining backward compatibility. Downgradability
> implies that we could potentially roll back an upgrade at any time. While I
> don’t think we need to retain the ability to downgrade in perpetuity it
> would be a good objective to maintain strict backward compatibility and
> therefore downgradability until a certain point. This would imply
> versioning metadata and extending it in such a way that prior version(s)
> could continue functioning. This can certainly be expensive to implement
> and might bloat on-disk storage. However, we could always offer an option
> for the operator to optimize the on-disk structures for the current version
> then we can rewrite them in the latest version. This optimizes the storage
> and opens up new functionality. This means new features that can work with
> old on-disk structures will be available while others that strictly require
> new versions of the data structures will be unavailable until the operator
> migrates to the new version. This migration IMO should be irreversible.
> Beyond this point the operator will lose the ability to downgrade which is
> ok.
>
> Dinesh
>
> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:
>
> 
> There has been progress on
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928
>
> Which is similar to what datastax does for DSE. Would this be an
> acceptable solution?
>
> Jake
>
> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com> wrote:
>
>> It seems “An alternative solution is to implement/complete CASSANDRA-8110
>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>” can give us more
>> options if it is finished😉
>>
>> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>>
>>> Hi everyone,
>>>
>>> There has been a discussion lately about changes to the sstable format
>>> in the context of being able to abort a cluster upgrade, and the fact that
>>> changes to sstables can prevent downgraded nodes from reading any data
>>> written during their temporary operation with the new version.
>>>
>>> Most of the discussion is in CASSANDRA-18134
>>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>>> spreading into CASSANDRA-14277
>>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>>> CASSANDRA-17698 <https://issues.apache.org/jira/browse/CASSANDRA-17698>,
>>> none of which is a good place to discuss the topic seriously.
>>>
>>> Downgradability is a worthy goal and is listed in the current roadmap. I
>>> would like to open a discussion here on how it would be achieved.
>>>
>>> My understanding of what has been suggested so far translates to:
>>> - avoid changes to sstable formats;
>>> - if there are changes, implement them in a way that is
>>> backwards-compatible, e.g. by duplicating data, so that a new version is
>>> presented in a component or portion of a component that legacy nodes will
>>> not try to read;
>>> - if the latter is not feasible, make sure the changes are only applied
>>> if a feature flag has been enabled.
>>>
>>> To me this approach introduces several risks:
>>> - it bloats file and parsing complexity;
>>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>>> ticket once this requirement is in place);
>>> - it needs care to avoid risky solutions to address technical issues
>>> with the format versioning (e.g. staying on n-versions for 5.0 and needing
>>> a bump for a 4.1 bugfix might require porting over support for new
>>> features);
>>> - it requires separate and uncoordinated solutions to the problem and
>>> switching mechanisms for each individual change.
>>>
>>> An alternative solution is to implement/complete CASSANDRA-8110
>>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which provides
>>> a method of writing sstables for a target version. During upgrades, a node
>>> could be set to produce sstables corresponding to the older version, and
>>> there is a very straightforward way to implement modifications to formats
>>> like the tickets above to conform to its requirements.
>>>
>>> What do people think should be the way forward?
>>>
>>> Regards,
>>> Branimir
>>>
>>>
>>> --
>> you are the apple of my eye !
>>
> --
> http://twitter.com/tjake
>
>

Re: Downgradability

Posted by Dinesh Joshi <dj...@apache.org>.
I’m a big fan of maintaining backward compatibility. Downgradability implies
that we could potentially roll back an upgrade at any time. While I don’t
think we need to retain the ability to downgrade in perpetuity it would be a
good objective to maintain strict backward compatibility and therefore
downgradability until a certain point. This would imply versioning metadata
and extending it in such a way that prior version(s) could continue
functioning. This can certainly be expensive to implement and might bloat on-
disk storage. However, we could always offer an option for the operator to
optimize the on-disk structures for the current version then we can rewrite
them in the latest version. This optimizes the storage and opens up new
functionality. This means new features that can work with old on-disk
structures will be available while others that strictly require new versions
of the data structures will be unavailable until the operator migrates to the
new version. This migration IMO should be irreversible. Beyond this point the
operator will lose the ability to downgrade which is ok.  
  

Dinesh

  

> On Feb 20, 2023, at 10:40 AM, Jake Luciani <ja...@gmail.com> wrote:  
>  
>

> 
>
> There has been progress on
>
> <https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928>
>
>  
>
>
> Which is similar to what datastax does for DSE. Would this be an acceptable
> solution?
>
>  
>
>
> Jake
>
>  
>
>
> On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell
> <[cclive1601@gmail.com](mailto:cclive1601@gmail.com)> wrote:  
>
>

>> It seems “An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110)” can
give us more options if it is finished😉

>>

>>  
>
>>

>> Branimir Lambov
<[blambov@apache.org](mailto:blambov@apache.org)>于2023年2月20日 周一下午11:03写道:  
>
>>

>>> Hi everyone,  
>
>>>

>>>  
>
>>>

>>> There has been a discussion lately about changes to the sstable format in
the context of being able to abort a cluster upgrade, and the fact that
changes to sstables can prevent downgraded nodes from reading any data written
during their temporary operation with the new version.

>>>

>>>  
>
>>>

>>> Most of the discussion is in
[CASSANDRA-18134](https://issues.apache.org/jira/browse/CASSANDRA-18134), and
is spreading into
[CASSANDRA-14277](https://issues.apache.org/jira/browse/CASSANDRA-14227) and
[CASSANDRA-17698](https://issues.apache.org/jira/browse/CASSANDRA-17698), none
of which is a good place to discuss the topic seriously.

>>>

>>>  
>
>>>

>>> Downgradability is a worthy goal and is listed in the current roadmap. I
would like to open a discussion here on how it would be achieved.  
>
>>>

>>>  
>
>>>

>>> My understanding of what has been suggested so far translates to:

>>>

>>> \- avoid changes to sstable formats;

>>>

>>> \- if there are changes, implement them in a way that is backwards-
compatible, e.g. by duplicating data, so that a new version is presented in a
component or portion of a component that legacy nodes will not try to read;

>>>

>>> \- if the latter is not feasible, make sure the changes are only applied
if a feature flag has been enabled.

>>>

>>>  
>
>>>

>>> To me this approach introduces several risks:

>>>

>>> \- it bloats file and parsing complexity;

>>>

>>> \- it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
ticket once this requirement is in place);

>>>

>>> \- it needs care to avoid risky solutions to address technical issues with
the format versioning (e.g. staying on n-versions for 5.0 and needing a bump
for a 4.1 bugfix might require porting over support for new features);  
>
>>>

>>> \- it requires separate and uncoordinated solutions to the problem and
switching mechanisms for each individual change.

>>>

>>>  
>
>>>

>>> An alternative solution is to implement/complete
[CASSANDRA-8110](https://issues.apache.org/jira/browse/CASSANDRA-8110), which
provides a method of writing sstables for a target version. During upgrades, a
node could be set to produce sstables corresponding to the older version, and
there is a very straightforward way to implement modifications to formats like
the tickets above to conform to its requirements.

>>>

>>>  
>
>>>

>>> What do people think should be the way forward?

>>>

>>>  
>
>>>

>>> Regards,

>>>

>>> Branimir  
>
>>>

>>>  
>
>>>

>>>  
>
>>

>> \--  
>
>>

>> you are the apple of my eye !

>
> \--  
>
>
> <http://twitter.com/tjake>


Re: Downgradability

Posted by Jake Luciani <ja...@gmail.com>.
There has been progress on
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-8928

Which is similar to what datastax does for DSE. Would this be an acceptable
solution?

Jake

On Mon, Feb 20, 2023 at 11:17 AM guo Maxwell <cc...@gmail.com> wrote:

> It seems “An alternative solution is to implement/complete CASSANDRA-8110
> <https://issues.apache.org/jira/browse/CASSANDRA-8110>” can give us more
> options if it is finished😉
>
> Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:
>
>> Hi everyone,
>>
>> There has been a discussion lately about changes to the sstable format in
>> the context of being able to abort a cluster upgrade, and the fact that
>> changes to sstables can prevent downgraded nodes from reading any data
>> written during their temporary operation with the new version.
>>
>> Most of the discussion is in CASSANDRA-18134
>> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is
>> spreading into CASSANDRA-14277
>> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
>> CASSANDRA-17698 <https://issues.apache.org/jira/browse/CASSANDRA-17698>,
>> none of which is a good place to discuss the topic seriously.
>>
>> Downgradability is a worthy goal and is listed in the current roadmap. I
>> would like to open a discussion here on how it would be achieved.
>>
>> My understanding of what has been suggested so far translates to:
>> - avoid changes to sstable formats;
>> - if there are changes, implement them in a way that is
>> backwards-compatible, e.g. by duplicating data, so that a new version is
>> presented in a component or portion of a component that legacy nodes will
>> not try to read;
>> - if the latter is not feasible, make sure the changes are only applied
>> if a feature flag has been enabled.
>>
>> To me this approach introduces several risks:
>> - it bloats file and parsing complexity;
>> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
>> ticket once this requirement is in place);
>> - it needs care to avoid risky solutions to address technical issues with
>> the format versioning (e.g. staying on n-versions for 5.0 and needing a
>> bump for a 4.1 bugfix might require porting over support for new features);
>> - it requires separate and uncoordinated solutions to the problem and
>> switching mechanisms for each individual change.
>>
>> An alternative solution is to implement/complete CASSANDRA-8110
>> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which provides a
>> method of writing sstables for a target version. During upgrades, a node
>> could be set to produce sstables corresponding to the older version, and
>> there is a very straightforward way to implement modifications to formats
>> like the tickets above to conform to its requirements.
>>
>> What do people think should be the way forward?
>>
>> Regards,
>> Branimir
>>
>>
>> --
> you are the apple of my eye !
>
-- 
http://twitter.com/tjake

Re: Downgradability

Posted by guo Maxwell <cc...@gmail.com>.
It seems “An alternative solution is to implement/complete CASSANDRA-8110
<https://issues.apache.org/jira/browse/CASSANDRA-8110>” can give us more
options if it is finished😉

Branimir Lambov <bl...@apache.org>于2023年2月20日 周一下午11:03写道:

> Hi everyone,
>
> There has been a discussion lately about changes to the sstable format in
> the context of being able to abort a cluster upgrade, and the fact that
> changes to sstables can prevent downgraded nodes from reading any data
> written during their temporary operation with the new version.
>
> Most of the discussion is in CASSANDRA-18134
> <https://issues.apache.org/jira/browse/CASSANDRA-18134>, and is spreading
> into CASSANDRA-14277
> <https://issues.apache.org/jira/browse/CASSANDRA-14227> and
> CASSANDRA-17698 <https://issues.apache.org/jira/browse/CASSANDRA-17698>,
> none of which is a good place to discuss the topic seriously.
>
> Downgradability is a worthy goal and is listed in the current roadmap. I
> would like to open a discussion here on how it would be achieved.
>
> My understanding of what has been suggested so far translates to:
> - avoid changes to sstable formats;
> - if there are changes, implement them in a way that is
> backwards-compatible, e.g. by duplicating data, so that a new version is
> presented in a component or portion of a component that legacy nodes will
> not try to read;
> - if the latter is not feasible, make sure the changes are only applied if
> a feature flag has been enabled.
>
> To me this approach introduces several risks:
> - it bloats file and parsing complexity;
> - it discourages improvement (e.g. CASSANDRA-17698 is no longer a LHF
> ticket once this requirement is in place);
> - it needs care to avoid risky solutions to address technical issues with
> the format versioning (e.g. staying on n-versions for 5.0 and needing a
> bump for a 4.1 bugfix might require porting over support for new features);
> - it requires separate and uncoordinated solutions to the problem and
> switching mechanisms for each individual change.
>
> An alternative solution is to implement/complete CASSANDRA-8110
> <https://issues.apache.org/jira/browse/CASSANDRA-8110>, which provides a
> method of writing sstables for a target version. During upgrades, a node
> could be set to produce sstables corresponding to the older version, and
> there is a very straightforward way to implement modifications to formats
> like the tickets above to conform to its requirements.
>
> What do people think should be the way forward?
>
> Regards,
> Branimir
>
>
> --
you are the apple of my eye !