Supporting 2.2 -> 5.0 upgrades

["Miklosovic, Stefan via dev" <de...@cassandra.apache.org>]

Hey,

I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.

I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.

Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?

(1) https://issues.apache.org/jira/browse/CASSANDRA-19002

Regards

Re: Supporting 2.2 -> 5.0 upgrades

[Štefan Miklošovič <sm...@apache.org>]

btw if you think about that ... if we ever felt some strong urge to move
from Ant we could not overcome and we had code split to more modules /
subprojects already, then moving such an already-splitted project from Ant
to whatever else would be just an exercise. It would be like "OK we split
Cassandra's codebase to 10 modules" so moving that to Maven or whatever
would be just about what? About writing poms and wiring that together.

I suggest we try to make a simple module by extracting some functionality
to see how it would look like, gain the experience to do that and iterate.
I can imagine untying all the guts so we can ship that as separate jar will
be the work in itself (and most probably the majority of that).

My main concern was to not complicate it further and rather to start from
scratch but if we take an iterative process like that, that might also make
it easier to move away eventually if we think it is the right move as we
would have split it already anyway.

On Thu, Dec 12, 2024 at 6:01 PM Benedict <be...@apache.org> wrote:

> Why would ant get in the way? We already build multiple jars, and accord
> will be a submodule. We have far more organisational issues to overcome
> than ant.
>
> I have for a while advocated for a shared lib to also share between Harry,
> accord, dtests etc
>
> I am however not 100% sure about splitting read/write path, at least not
> as first posited. The idea of maintaining it as an API for dropping in
> different jars is a whole other world of potential pain I don’t want to
> countenance. Supporting eg bulk readers or writers or other integrations
> seems pretty feasible though.
>
> On 12 Dec 2024, at 16:53, Paulo Motta <pa...@apache.org> wrote:
>
> 
> >  I think that will not happen until we are out of Ant as doing this
> multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>
> This is a contentious/controversial topic, but the more I work with gradle
> the more I lean towards ant's simplicity. That said, I'd support moving
> away if it becomes a technical blocker to break up cassandra-all - and if
> this happen I would vote for maven as replacement. :-D
>
> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <
> dev@cassandra.apache.org> wrote:
>
>> These are all good ideas but in practical terms I think that will not
>> happen until we are out of Ant as doing this multi jar / subproject mumbo
>> jumbo is not too much appealing to ... anybody?
>>
>> ________________________________________
>> From: Paulo Motta <pa...@apache.org>
>> Sent: Thursday, December 12, 2024 17:35
>> To: dev@cassandra.apache.org
>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>
>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>
>>
>>
>> >  +1 on moving the read/write logic into its own jar.
>>
>> +1, not only read-write logic but anything used by both the server and
>> subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other
>> interfaces.
>>
>> I think one way to do that would be to split cassandra-all into
>> cassandra-server and cassandra-common (anything used by both subprojects
>> and server), but not sure if this would be feasible or what it would take.
>>
>> If there's loose agreement this would be a feasible path I'd be happy to
>> create a JIRA to investigate what this would take.
>>
>> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <drohrer@apple.com<mailto:
>> drohrer@apple.com>> wrote:
>> +1 on moving the read/write logic into its own jar.
>>
>> Doug
>>
>> > On Dec 11, 2024, at 7:21 PM, David Capwell <dcapwell@apple.com<mailto:
>> dcapwell@apple.com>> wrote:
>> >
>> > From a disk format point of view the only thing I remember was the disk
>> type bug with UDTs.  Bringing that logic back was hard as the type system
>> (in 5.0) tries to avoid allowing construction of invalid states, and we
>> would need to weaken that in order to enable the migration. Assuming the
>> user migrated from 3.x to 4.x then the sstable metadata should have been
>> rewritten to fix this bug.
>> >
>> > One thought (though know its a ton of effort).. we have talked about
>> for a long time about moving the reading/writing logic into its jar (so
>> tools don’t need cassandra-all and can limit the dependencies)… if we did
>> that we could try to solve this as an out of process migration… have the
>> 2.2 reader then write using 6.0 writer (ignoring compact storage… )…
>> >
>> >> On Dec 11, 2024, at 4:59 AM, Benedict <benedict@apache.org<mailto:
>> benedict@apache.org>> wrote:
>> >>
>> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am
>> fairly sure 4.x supported upgrade from 3.0.x also.
>> >>
>> >>
>> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <
>> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
>> >>>
>> >>> I see. That makes sense. I think that by 3.x you meant basically the
>> latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to
>> support 2.2 -> 3.11 straight away. I need to check where we are at in that
>> area.
>> >>>
>> >>> ________________________________________
>> >>> From: Benedict <be...@apache.org>>
>> >>> Sent: Wednesday, December 11, 2024 13:09
>> >>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
>> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<mailto:
>> dev@cassandra.apache.org>; Miklosovic, Stefan
>> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>> >>>
>> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> 2.2 is particularly hard because of the major storage format changes
>> that took place.
>> >>>
>> >>> I think if we want to retain (restore) upgrade support from 3.x I
>> would support that, but 2.x is probably too burdensome and likely to have
>> too many hard edges.
>> >>>
>> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that
>> would be a pretty friendly upgrade path all things considered.
>> >>>
>> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <
>> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
>> >>>>
>> >>>> Hey,
>> >>>>
>> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0
>> would be cool to support.
>> >>>>
>> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0
>> work and fixed few issues along the way (1), hence I can imagine that
>> supporting 2.2 -> 5.0 would be basically the same thing just on steroids
>> and more involved? Anyway, having a stab into this is not useless at all, I
>> will at least go deep into the upgrade stuff I have never given a lot of
>> thought to which is good learning experience.
>> >>>>
>> >>>> Any tips where to start? Was any progress done by anybody already in
>> this matter to not start from zero?
>> >>>>
>> >>>> (1)
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>> >>>>
>> >>>> Regards
>> >>>
>> >>
>> >
>>
>>

Re: Supporting 2.2 -> 5.0 upgrades

[Mick Semb Wever <mc...@apache.org>]

On Fri, 13 Dec 2024 at 06:06, Jeremiah Jordan <je...@gmail.com>
wrote:

> TL;DR - in progress migration off 2.2 to 5.0 is annoying as there were
>>> different bugs in the past we have to support again.  Out of process
>>> migration to me feels far more plausible, but feels annoying without
>>> splitting off our reader/writer… doable… just more annoying…
>>
>>
> This is your main blocker for the 5.0/trunk code converting 2.2 sstables
> correctly.  You would have to bring back a bunch of code paths special
> casing those versions and dealing with bugs in them.  If you are actually
> interested in figuring out an offline 2.2 to 5.0 path I would recommend you
> do it in two steps.  Offline 2.2 to 4.1? or what ever the latest thing is
> that can sstableupgrade a 2.2 sstable, and then offline the result of that
> with 5.0.  But at that point maybe you just do it online in those two steps.
>


Stepping through safe upgrade edges is currently a requirement, and I
suspect will often have value.

Something like `nodetool sstableupgrade --step 3.0,4.1,5.0`
 would be useful.

While it would take longer to do offline upgrades, it allows us to maintain
simpler code, and draw the lineations when we must.

Re: Supporting 2.2 -> 5.0 upgrades

[Jeremiah Jordan <je...@gmail.com>]

>
> TL;DR - in progress migration off 2.2 to 5.0 is annoying as there were
>> different bugs in the past we have to support again.  Out of process
>> migration to me feels far more plausible, but feels annoying without
>> splitting off our reader/writer… doable… just more annoying…
>
>
This is your main blocker for the 5.0/trunk code converting 2.2 sstables
correctly.  You would have to bring back a bunch of code paths special
casing those versions and dealing with bugs in them.  If you are actually
interested in figuring out an offline 2.2 to 5.0 path I would recommend you
do it in two steps.  Offline 2.2 to 4.1? or what ever the latest thing is
that can sstableupgrade a 2.2 sstable, and then offline the result of that
with 5.0.  But at that point maybe you just do it online in those two steps.


On Dec 12, 2024 at 11:34:31 AM, Štefan Miklošovič <sm...@apache.org>
wrote:

> I think it does not make a lot of sense to get away from Ant unless we
> split it into more jars. Splitting it into more jars while moving away from
> Ant at the same time is just too much work. So, what is the point of having
> monolithic cassandra-all in Gradle / Maven? Smoother release? We mastered
> that already. We are releasing, aren't we? Dependencies? That is working
> too. Sure cache and all the "hacks" would go away overnight but otherwise
> ... I think modularising it first so it is easier to reuse and so on is
> more important.
>
> On Thu, Dec 12, 2024 at 6:21 PM David Capwell <dc...@apple.com> wrote:
>
>> but I still find myself very rarely interacting with ant
>>
>>
>> I think that is where most people are as not many actually maintain or
>> modify ant… there are so many things that bug me (lack of cache, making
>> sure new people use the right version (was totally fun to learn 4.1 didn’t
>> build with the default ant that got installed when I was helping out a new
>> higher… we had to downgrade… woohoo….), hand rolled IDE integration, etc…),
>> but I would disagree that ant is a blocker for different jars… we could
>> switch off ant for Makefile, or bash, and we would still be able to produce
>> new jars… Its more of… how many people actually feel comfortable enough to
>> alter our build system to make such a change?  I don’t…
>>
>> Our build system does not impact our ability to offer migration from 2.2
>> to 5.0… so don’t want to keep distracting this thread…
>>
>> TL;DR - in progress migration off 2.2 to 5.0 is annoying as there were
>> different bugs in the past we have to support again.  Out of process
>> migration to me feels far more plausible, but feels annoying without
>> splitting off our reader/writer… doable… just more annoying...
>>
>> On Dec 12, 2024, at 9:04 AM, Alex Petrov <al...@coffeenco.de> wrote:
>>
>> > I have for a while advocated for a shared lib to also share between
>> Harry, accord, dtests etc
>>
>> Big +1 for a shared lib for our concurrency and test utils. Been
>> intending to start working on this for a while now, but never got to do
>> this so far.
>>
>> On Thu, Dec 12, 2024, at 5:58 PM, Benedict wrote:
>>
>>
>> Why would ant get in the way? We already build multiple jars, and accord
>> will be a submodule. We have far more organisational issues to overcome
>> than ant.
>>
>> I have for a while advocated for a shared lib to also share between
>> Harry, accord, dtests etc
>>
>> I am however not 100% sure about splitting read/write path, at least not
>> as first posited. The idea of maintaining it as an API for dropping in
>> different jars is a whole other world of potential pain I don’t want to
>> countenance. Supporting eg bulk readers or writers or other integrations
>> seems pretty feasible though.
>>
>>
>> On 12 Dec 2024, at 16:53, Paulo Motta <pa...@apache.org> wrote:
>>
>> 
>> >  I think that will not happen until we are out of Ant as doing this
>> multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>>
>> This is a contentious/controversial topic, but the more I work with
>> gradle the more I lean towards ant's simplicity. That said, I'd support
>> moving away if it becomes a technical blocker to break up cassandra-all -
>> and if this happen I would vote for maven as replacement. :-D
>>
>> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <
>> dev@cassandra.apache.org> wrote:
>>
>> These are all good ideas but in practical terms I think that will not
>> happen until we are out of Ant as doing this multi jar / subproject mumbo
>> jumbo is not too much appealing to ... anybody?
>>
>> ________________________________________
>> From: Paulo Motta <pa...@apache.org>
>> Sent: Thursday, December 12, 2024 17:35
>> To: dev@cassandra.apache.org
>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>
>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>
>>
>>
>> >  +1 on moving the read/write logic into its own jar.
>>
>> +1, not only read-write logic but anything used by both the server and
>> subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other
>> interfaces.
>>
>> I think one way to do that would be to split cassandra-all into
>> cassandra-server and cassandra-common (anything used by both subprojects
>> and server), but not sure if this would be feasible or what it would take.
>>
>> If there's loose agreement this would be a feasible path I'd be happy to
>> create a JIRA to investigate what this would take.
>>
>> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <drohrer@apple.com<mailto:
>> drohrer@apple.com>> wrote:
>> +1 on moving the read/write logic into its own jar.
>>
>> Doug
>>
>> > On Dec 11, 2024, at 7:21 PM, David Capwell <dcapwell@apple.com<mailto:
>> dcapwell@apple.com>> wrote:
>> >
>> > From a disk format point of view the only thing I remember was the disk
>> type bug with UDTs.  Bringing that logic back was hard as the type system
>> (in 5.0) tries to avoid allowing construction of invalid states, and we
>> would need to weaken that in order to enable the migration. Assuming the
>> user migrated from 3.x to 4.x then the sstable metadata should have been
>> rewritten to fix this bug.
>> >
>> > One thought (though know its a ton of effort).. we have talked about
>> for a long time about moving the reading/writing logic into its jar (so
>> tools don’t need cassandra-all and can limit the dependencies)… if we did
>> that we could try to solve this as an out of process migration… have the
>> 2.2 reader then write using 6.0 writer (ignoring compact storage… )…
>> >
>> >> On Dec 11, 2024, at 4:59 AM, Benedict <benedict@apache.org<mailto:
>> benedict@apache.org>> wrote:
>> >>
>> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am
>> fairly sure 4.x supported upgrade from 3.0.x also.
>> >>
>> >>
>> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <
>> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
>> >>>
>> >>> I see. That makes sense. I think that by 3.x you meant basically the
>> latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to
>> support 2.2 -> 3.11 straight away. I need to check where we are at in that
>> area.
>> >>>
>> >>> ________________________________________
>> >>> From: Benedict <be...@apache.org>>
>> >>> Sent: Wednesday, December 11, 2024 13:09
>> >>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
>> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<mailto:
>> dev@cassandra.apache.org>; Miklosovic, Stefan
>> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>> >>>
>> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> 2.2 is particularly hard because of the major storage format changes
>> that took place.
>> >>>
>> >>> I think if we want to retain (restore) upgrade support from 3.x I
>> would support that, but 2.x is probably too burdensome and likely to have
>> too many hard edges.
>> >>>
>> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that
>> would be a pretty friendly upgrade path all things considered.
>> >>>
>> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <
>> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
>> >>>>
>> >>>> Hey,
>> >>>>
>> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0
>> would be cool to support.
>> >>>>
>> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0
>> work and fixed few issues along the way (1), hence I can imagine that
>> supporting 2.2 -> 5.0 would be basically the same thing just on steroids
>> and more involved? Anyway, having a stab into this is not useless at all, I
>> will at least go deep into the upgrade stuff I have never given a lot of
>> thought to which is good learning experience.
>> >>>>
>> >>>> Any tips where to start? Was any progress done by anybody already in
>> this matter to not start from zero?
>> >>>>
>> >>>> (1)
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>> >>>>
>> >>>> Regards
>> >>>
>> >>
>> >
>>
>>
>>

Re: Supporting 2.2 -> 5.0 upgrades

[Štefan Miklošovič <sm...@apache.org>]

I think it does not make a lot of sense to get away from Ant unless we
split it into more jars. Splitting it into more jars while moving away from
Ant at the same time is just too much work. So, what is the point of having
monolithic cassandra-all in Gradle / Maven? Smoother release? We mastered
that already. We are releasing, aren't we? Dependencies? That is working
too. Sure cache and all the "hacks" would go away overnight but otherwise
... I think modularising it first so it is easier to reuse and so on is
more important.

On Thu, Dec 12, 2024 at 6:21 PM David Capwell <dc...@apple.com> wrote:

> but I still find myself very rarely interacting with ant
>
>
> I think that is where most people are as not many actually maintain or
> modify ant… there are so many things that bug me (lack of cache, making
> sure new people use the right version (was totally fun to learn 4.1 didn’t
> build with the default ant that got installed when I was helping out a new
> higher… we had to downgrade… woohoo….), hand rolled IDE integration, etc…),
> but I would disagree that ant is a blocker for different jars… we could
> switch off ant for Makefile, or bash, and we would still be able to produce
> new jars… Its more of… how many people actually feel comfortable enough to
> alter our build system to make such a change?  I don’t…
>
> Our build system does not impact our ability to offer migration from 2.2
> to 5.0… so don’t want to keep distracting this thread…
>
> TL;DR - in progress migration off 2.2 to 5.0 is annoying as there were
> different bugs in the past we have to support again.  Out of process
> migration to me feels far more plausible, but feels annoying without
> splitting off our reader/writer… doable… just more annoying...
>
> On Dec 12, 2024, at 9:04 AM, Alex Petrov <al...@coffeenco.de> wrote:
>
> > I have for a while advocated for a shared lib to also share between
> Harry, accord, dtests etc
>
> Big +1 for a shared lib for our concurrency and test utils. Been intending
> to start working on this for a while now, but never got to do this so far.
>
> On Thu, Dec 12, 2024, at 5:58 PM, Benedict wrote:
>
>
> Why would ant get in the way? We already build multiple jars, and accord
> will be a submodule. We have far more organisational issues to overcome
> than ant.
>
> I have for a while advocated for a shared lib to also share between Harry,
> accord, dtests etc
>
> I am however not 100% sure about splitting read/write path, at least not
> as first posited. The idea of maintaining it as an API for dropping in
> different jars is a whole other world of potential pain I don’t want to
> countenance. Supporting eg bulk readers or writers or other integrations
> seems pretty feasible though.
>
>
> On 12 Dec 2024, at 16:53, Paulo Motta <pa...@apache.org> wrote:
>
> 
> >  I think that will not happen until we are out of Ant as doing this
> multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>
> This is a contentious/controversial topic, but the more I work with gradle
> the more I lean towards ant's simplicity. That said, I'd support moving
> away if it becomes a technical blocker to break up cassandra-all - and if
> this happen I would vote for maven as replacement. :-D
>
> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <
> dev@cassandra.apache.org> wrote:
>
> These are all good ideas but in practical terms I think that will not
> happen until we are out of Ant as doing this multi jar / subproject mumbo
> jumbo is not too much appealing to ... anybody?
>
> ________________________________________
> From: Paulo Motta <pa...@apache.org>
> Sent: Thursday, December 12, 2024 17:35
> To: dev@cassandra.apache.org
> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>
> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>
>
>
> >  +1 on moving the read/write logic into its own jar.
>
> +1, not only read-write logic but anything used by both the server and
> subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other
> interfaces.
>
> I think one way to do that would be to split cassandra-all into
> cassandra-server and cassandra-common (anything used by both subprojects
> and server), but not sure if this would be feasible or what it would take.
>
> If there's loose agreement this would be a feasible path I'd be happy to
> create a JIRA to investigate what this would take.
>
> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <drohrer@apple.com<mailto:
> drohrer@apple.com>> wrote:
> +1 on moving the read/write logic into its own jar.
>
> Doug
>
> > On Dec 11, 2024, at 7:21 PM, David Capwell <dcapwell@apple.com<mailto:
> dcapwell@apple.com>> wrote:
> >
> > From a disk format point of view the only thing I remember was the disk
> type bug with UDTs.  Bringing that logic back was hard as the type system
> (in 5.0) tries to avoid allowing construction of invalid states, and we
> would need to weaken that in order to enable the migration. Assuming the
> user migrated from 3.x to 4.x then the sstable metadata should have been
> rewritten to fix this bug.
> >
> > One thought (though know its a ton of effort).. we have talked about for
> a long time about moving the reading/writing logic into its jar (so tools
> don’t need cassandra-all and can limit the dependencies)… if we did that we
> could try to solve this as an out of process migration… have the 2.2 reader
> then write using 6.0 writer (ignoring compact storage… )…
> >
> >> On Dec 11, 2024, at 4:59 AM, Benedict <benedict@apache.org<mailto:
> benedict@apache.org>> wrote:
> >>
> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am
> fairly sure 4.x supported upgrade from 3.0.x also.
> >>
> >>
> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
> >>>
> >>> I see. That makes sense. I think that by 3.x you meant basically the
> latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to
> support 2.2 -> 3.11 straight away. I need to check where we are at in that
> area.
> >>>
> >>> ________________________________________
> >>> From: Benedict <be...@apache.org>>
> >>> Sent: Wednesday, December 11, 2024 13:09
> >>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<mailto:
> dev@cassandra.apache.org>; Miklosovic, Stefan
> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
> >>>
> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
> >>>
> >>>
> >>>
> >>>
> >>> 2.2 is particularly hard because of the major storage format changes
> that took place.
> >>>
> >>> I think if we want to retain (restore) upgrade support from 3.x I
> would support that, but 2.x is probably too burdensome and likely to have
> too many hard edges.
> >>>
> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that
> would be a pretty friendly upgrade path all things considered.
> >>>
> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
> >>>>
> >>>> Hey,
> >>>>
> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0
> would be cool to support.
> >>>>
> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work
> and fixed few issues along the way (1), hence I can imagine that supporting
> 2.2 -> 5.0 would be basically the same thing just on steroids and more
> involved? Anyway, having a stab into this is not useless at all, I will at
> least go deep into the upgrade stuff I have never given a lot of thought to
> which is good learning experience.
> >>>>
> >>>> Any tips where to start? Was any progress done by anybody already in
> this matter to not start from zero?
> >>>>
> >>>> (1)
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
> >>>>
> >>>> Regards
> >>>
> >>
> >
>
>
>

Re: Supporting 2.2 -> 5.0 upgrades

[David Capwell <dc...@apple.com>]

> but I still find myself very rarely interacting with ant

I think that is where most people are as not many actually maintain or modify ant… there are so many things that bug me (lack of cache, making sure new people use the right version (was totally fun to learn 4.1 didn’t build with the default ant that got installed when I was helping out a new higher… we had to downgrade… woohoo….), hand rolled IDE integration, etc…), but I would disagree that ant is a blocker for different jars… we could switch off ant for Makefile, or bash, and we would still be able to produce new jars… Its more of… how many people actually feel comfortable enough to alter our build system to make such a change?  I don’t…

Our build system does not impact our ability to offer migration from 2.2 to 5.0… so don’t want to keep distracting this thread…

TL;DR - in progress migration off 2.2 to 5.0 is annoying as there were different bugs in the past we have to support again.  Out of process migration to me feels far more plausible, but feels annoying without splitting off our reader/writer… doable… just more annoying...

> On Dec 12, 2024, at 9:04 AM, Alex Petrov <al...@coffeenco.de> wrote:
> 
> > I have for a while advocated for a shared lib to also share between Harry, accord, dtests etc
> 
> Big +1 for a shared lib for our concurrency and test utils. Been intending to start working on this for a while now, but never got to do this so far.
> 
> On Thu, Dec 12, 2024, at 5:58 PM, Benedict wrote:
>> 
>> Why would ant get in the way? We already build multiple jars, and accord will be a submodule. We have far more organisational issues to overcome than ant.
>> 
>> I have for a while advocated for a shared lib to also share between Harry, accord, dtests etc
>> 
>> I am however not 100% sure about splitting read/write path, at least not as first posited. The idea of maintaining it as an API for dropping in different jars is a whole other world of potential pain I don’t want to countenance. Supporting eg bulk readers or writers or other integrations seems pretty feasible though.
>> 
>> 
>>> On 12 Dec 2024, at 16:53, Paulo Motta <pa...@apache.org> wrote:
>>> 
>>> >  I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>>> 
>>> This is a contentious/controversial topic, but the more I work with gradle the more I lean towards ant's simplicity. That said, I'd support moving away if it becomes a technical blocker to break up cassandra-all - and if this happen I would vote for maven as replacement. :-D
>>> 
>>> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <dev@cassandra.apache.org <ma...@cassandra.apache.org>> wrote:
>>> These are all good ideas but in practical terms I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>>> 
>>> ________________________________________
>>> From: Paulo Motta <paulo@apache.org <ma...@apache.org>>
>>> Sent: Thursday, December 12, 2024 17:35
>>> To: dev@cassandra.apache.org <ma...@cassandra.apache.org>
>>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>> 
>>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>> 
>>> 
>>> 
>>> >  +1 on moving the read/write logic into its own jar.
>>> 
>>> +1, not only read-write logic but anything used by both the server and subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other interfaces.
>>> 
>>> I think one way to do that would be to split cassandra-all into cassandra-server and cassandra-common (anything used by both subprojects and server), but not sure if this would be feasible or what it would take.
>>> 
>>> If there's loose agreement this would be a feasible path I'd be happy to create a JIRA to investigate what this would take.
>>> 
>>> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <drohrer@apple.com <ma...@apple.com><mailto:drohrer@apple.com <ma...@apple.com>>> wrote:
>>> +1 on moving the read/write logic into its own jar.
>>> 
>>> Doug
>>> 
>>> > On Dec 11, 2024, at 7:21 PM, David Capwell <dcapwell@apple.com <ma...@apple.com><mailto:dcapwell@apple.com <ma...@apple.com>>> wrote:
>>> >
>>> > From a disk format point of view the only thing I remember was the disk type bug with UDTs.  Bringing that logic back was hard as the type system (in 5.0) tries to avoid allowing construction of invalid states, and we would need to weaken that in order to enable the migration. Assuming the user migrated from 3.x to 4.x then the sstable metadata should have been rewritten to fix this bug.
>>> >
>>> > One thought (though know its a ton of effort).. we have talked about for a long time about moving the reading/writing logic into its jar (so tools don’t need cassandra-all and can limit the dependencies)… if we did that we could try to solve this as an out of process migration… have the 2.2 reader then write using 6.0 writer (ignoring compact storage… )…
>>> >
>>> >> On Dec 11, 2024, at 4:59 AM, Benedict <benedict@apache.org <ma...@apache.org><mailto:benedict@apache.org <ma...@apache.org>>> wrote:
>>> >>
>>> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.
>>> >>
>>> >>
>>> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <dev@cassandra.apache.org <ma...@cassandra.apache.org><mailto:dev@cassandra.apache.org <ma...@cassandra.apache.org>>> wrote:
>>> >>>
>>> >>> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
>>> >>>
>>> >>> ________________________________________
>>> >>> From: Benedict <benedict@apache.org <ma...@apache.org><mailto:benedict@apache.org <ma...@apache.org>>>
>>> >>> Sent: Wednesday, December 11, 2024 13:09
>>> >>> To: dev@cassandra.apache.org <ma...@cassandra.apache.org><mailto:dev@cassandra.apache.org <ma...@cassandra.apache.org>>
>>> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org <ma...@cassandra.apache.org><mailto:dev@cassandra.apache.org <ma...@cassandra.apache.org>>; Miklosovic, Stefan
>>> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>> >>>
>>> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> 2.2 is particularly hard because of the major storage format changes that took place.
>>> >>>
>>> >>> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
>>> >>>
>>> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
>>> >>>
>>> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <dev@cassandra.apache.org <ma...@cassandra.apache.org><mailto:dev@cassandra.apache.org <ma...@cassandra.apache.org>>> wrote:
>>> >>>>
>>> >>>> Hey,
>>> >>>>
>>> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>>> >>>>
>>> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>>> >>>>
>>> >>>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>>> >>>>
>>> >>>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>>> >>>>
>>> >>>> Regards
>>> >>>
>>> >>
>>> >

Re: Supporting 2.2 -> 5.0 upgrades

[Alex Petrov <al...@coffeenco.de>]

> I have for a while advocated for a shared lib to also share between Harry, accord, dtests etc

Big +1 for a shared lib for our concurrency and test utils. Been intending to start working on this for a while now, but never got to do this so far.

On Thu, Dec 12, 2024, at 5:58 PM, Benedict wrote:
> 
> Why would ant get in the way? We already build multiple jars, and accord will be a submodule. We have far more organisational issues to overcome than ant.
> 
> I have for a while advocated for a shared lib to also share between Harry, accord, dtests etc
> 
> I am however not 100% sure about splitting read/write path, at least not as first posited. The idea of maintaining it as an API for dropping in different jars is a whole other world of potential pain I don’t want to countenance. Supporting eg bulk readers or writers or other integrations seems pretty feasible though.
> 
> 
>> On 12 Dec 2024, at 16:53, Paulo Motta <pa...@apache.org> wrote:
>> 
>> >  I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>> 
>> This is a contentious/controversial topic, but the more I work with gradle the more I lean towards ant's simplicity. That said, I'd support moving away if it becomes a technical blocker to break up cassandra-all - and if this happen I would vote for maven as replacement. :-D
>> 
>> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>>> These are all good ideas but in practical terms I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>>> 
>>> ________________________________________
>>> From: Paulo Motta <pa...@apache.org>
>>> Sent: Thursday, December 12, 2024 17:35
>>> To: dev@cassandra.apache.org
>>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>> 
>>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>> 
>>> 
>>> 
>>> >  +1 on moving the read/write logic into its own jar.
>>> 
>>> +1, not only read-write logic but anything used by both the server and subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other interfaces.
>>> 
>>> I think one way to do that would be to split cassandra-all into cassandra-server and cassandra-common (anything used by both subprojects and server), but not sure if this would be feasible or what it would take.
>>> 
>>> If there's loose agreement this would be a feasible path I'd be happy to create a JIRA to investigate what this would take.
>>> 
>>> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <dr...@apple.com>> wrote:
>>> +1 on moving the read/write logic into its own jar.
>>> 
>>> Doug
>>> 
>>> > On Dec 11, 2024, at 7:21 PM, David Capwell <dc...@apple.com>> wrote:
>>> >
>>> > From a disk format point of view the only thing I remember was the disk type bug with UDTs.  Bringing that logic back was hard as the type system (in 5.0) tries to avoid allowing construction of invalid states, and we would need to weaken that in order to enable the migration. Assuming the user migrated from 3.x to 4.x then the sstable metadata should have been rewritten to fix this bug.
>>> >
>>> > One thought (though know its a ton of effort).. we have talked about for a long time about moving the reading/writing logic into its jar (so tools don’t need cassandra-all and can limit the dependencies)… if we did that we could try to solve this as an out of process migration… have the 2.2 reader then write using 6.0 writer (ignoring compact storage… )…
>>> >
>>> >> On Dec 11, 2024, at 4:59 AM, Benedict <be...@apache.org>> wrote:
>>> >>
>>> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.
>>> >>
>>> >>
>>> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>>> >>>
>>> >>> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
>>> >>>
>>> >>> ________________________________________
>>> >>> From: Benedict <be...@apache.org>>
>>> >>> Sent: Wednesday, December 11, 2024 13:09
>>> >>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
>>> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<ma...@cassandra.apache.org>; Miklosovic, Stefan
>>> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>> >>>
>>> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> 2.2 is particularly hard because of the major storage format changes that took place.
>>> >>>
>>> >>> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
>>> >>>
>>> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
>>> >>>
>>> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>>> >>>>
>>> >>>> Hey,
>>> >>>>
>>> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>>> >>>>
>>> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>>> >>>>
>>> >>>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>>> >>>>
>>> >>>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>>> >>>>
>>> >>>> Regards
>>> >>>
>>> >>
>>> >

Re: Supporting 2.2 -> 5.0 upgrades

[Benedict <be...@apache.org>]

Why would ant get in the way? We already build multiple jars, and accord will
be a submodule. We have far more organisational issues to overcome than ant.

  

I have for a while advocated for a shared lib to also share between Harry,
accord, dtests etc

  

I am however not 100% sure about splitting read/write path, at least not as
first posited. The idea of maintaining it as an API for dropping in different
jars is a whole other world of potential pain I don’t want to countenance.
Supporting eg bulk readers or writers or other integrations seems pretty
feasible though.

  

> On 12 Dec 2024, at 16:53, Paulo Motta <pa...@apache.org> wrote:  
>  
>

> 
>
> >  I think that will not happen until we are out of Ant as doing this multi
> jar / subproject mumbo jumbo is not too much appealing to ... anybody?  
>  
> This is a contentious/controversial topic, but the more I work with gradle
> the more I lean towards ant's simplicity. That said, I'd support moving away
> if it becomes a technical blocker to break up cassandra-all - and if this
> happen I would vote for maven as replacement. :-D  
>
>
>  
>
>
> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev
> <[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)> wrote:  
>
>

>> These are all good ideas but in practical terms I think that will not
happen until we are out of Ant as doing this multi jar / subproject mumbo
jumbo is not too much appealing to ... anybody?  
>  
>  ________________________________________  
>  From: Paulo Motta <[paulo@apache.org](mailto:paulo@apache.org)>  
>  Sent: Thursday, December 12, 2024 17:35  
>  To: [dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)  
>  Subject: Re: Supporting 2.2 -> 5.0 upgrades  
>  
>  EXTERNAL EMAIL - USE CAUTION when clicking links or attachments  
>  
>  
>  
>  >  +1 on moving the read/write logic into its own jar.  
>  
>  +1, not only read-write logic but anything used by both the server and
> subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other
> interfaces.  
>  
>  I think one way to do that would be to split cassandra-all into cassandra-
> server and cassandra-common (anything used by both subprojects and server),
> but not sure if this would be feasible or what it would take.  
>  
>  If there's loose agreement this would be a feasible path I'd be happy to
> create a JIRA to investigate what this would take.  
>  
>  On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer
> <[drohrer@apple.com](mailto:drohrer@apple.com)<mailto:[drohrer@apple.com](mailto:drohrer@apple.com)>>
> wrote:  
>  +1 on moving the read/write logic into its own jar.  
>  
>  Doug  
>  
>  > On Dec 11, 2024, at 7:21 PM, David Capwell
> <[dcapwell@apple.com](mailto:dcapwell@apple.com)<mailto:[dcapwell@apple.com](mailto:dcapwell@apple.com)>>
> wrote:  
>  >  
>  > From a disk format point of view the only thing I remember was the disk
> type bug with UDTs.  Bringing that logic back was hard as the type system
> (in 5.0) tries to avoid allowing construction of invalid states, and we
> would need to weaken that in order to enable the migration. Assuming the
> user migrated from 3.x to 4.x then the sstable metadata should have been
> rewritten to fix this bug.  
>  >  
>  > One thought (though know its a ton of effort).. we have talked about for
> a long time about moving the reading/writing logic into its jar (so tools
> don’t need cassandra-all and can limit the dependencies)… if we did that we
> could try to solve this as an out of process migration… have the 2.2 reader
> then write using 6.0 writer (ignoring compact storage… )…  
>  >  
>  >> On Dec 11, 2024, at 4:59 AM, Benedict
> <[benedict@apache.org](mailto:benedict@apache.org)<mailto:[benedict@apache.org](mailto:benedict@apache.org)>>
> wrote:  
>  >>  
>  >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am
> fairly sure 4.x supported upgrade from 3.0.x also.  
>  >>  
>  >>  
>  >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev
> <[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)<mailto:[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)>>
> wrote:  
>  >>>  
>  >>> I see. That makes sense. I think that by 3.x you meant basically the
> latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to
> support 2.2 -> 3.11 straight away. I need to check where we are at in that
> area.  
>  >>>  
>  >>> ________________________________________  
>  >>> From: Benedict
> <[benedict@apache.org](mailto:benedict@apache.org)<mailto:[benedict@apache.org](mailto:benedict@apache.org)>>  
>  >>> Sent: Wednesday, December 11, 2024 13:09  
>  >>> To:
> [dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)<mailto:[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)>  
>  >>> Cc: Miklosovic, Stefan;
> [dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)<mailto:[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)>;
> Miklosovic, Stefan  
>  >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades  
>  >>>  
>  >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments  
>  >>>  
>  >>>  
>  >>>  
>  >>>  
>  >>> 2.2 is particularly hard because of the major storage format changes
> that took place.  
>  >>>  
>  >>> I think if we want to retain (restore) upgrade support from 3.x I would
> support that, but 2.x is probably too burdensome and likely to have too many
> hard edges.  
>  >>>  
>  >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that
> would be a pretty friendly upgrade path all things considered.  
>  >>>  
>  >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev
> <[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)<mailto:[dev@cassandra.apache.org](mailto:dev@cassandra.apache.org)>>
> wrote:  
>  >>>>  
>  >>>> Hey,  
>  >>>>  
>  >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0
> would be cool to support.  
>  >>>>  
>  >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work
> and fixed few issues along the way (1), hence I can imagine that supporting
> 2.2 -> 5.0 would be basically the same thing just on steroids and more
> involved? Anyway, having a stab into this is not useless at all, I will at
> least go deep into the upgrade stuff I have never given a lot of thought to
> which is good learning experience.  
>  >>>>  
>  >>>> Any tips where to start? Was any progress done by anybody already in
> this matter to not start from zero?  
>  >>>>  
>  >>>> (1)
> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$>  
>  >>>>  
>  >>>> Regards  
>  >>>  
>  >>  
>  >  
>  
>

Re: Supporting 2.2 -> 5.0 upgrades

[Josh McKenzie <jm...@apache.org>]

> will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
There's some folks working on a CEP around our build system and supporting a shared library (came up in a thread on #cassandra-dev; that's the extent of my knowledge on it excepting that it's still ongoing). Let's sit tight and not derail this topic on that point.

Assuming a build system that facilitated clean multi-project dependency support and the ability to work on everything in one unified place, I think there's still value in enumerating what sections off the DB we might want to extract in order to make the upgrade support more robust and extensible (as per this thread's original point). 

On Thu, Dec 12, 2024, at 11:53 AM, Paulo Motta wrote:
> >  I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
> 
> This is a contentious/controversial topic, but the more I work with gradle the more I lean towards ant's simplicity. That said, I'd support moving away if it becomes a technical blocker to break up cassandra-all - and if this happen I would vote for maven as replacement. :-D
> 
> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>> These are all good ideas but in practical terms I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>> 
>> ________________________________________
>> From: Paulo Motta <pa...@apache.org>
>> Sent: Thursday, December 12, 2024 17:35
>> To: dev@cassandra.apache.org
>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>> 
>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>> 
>> 
>> 
>> >  +1 on moving the read/write logic into its own jar.
>> 
>> +1, not only read-write logic but anything used by both the server and subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other interfaces.
>> 
>> I think one way to do that would be to split cassandra-all into cassandra-server and cassandra-common (anything used by both subprojects and server), but not sure if this would be feasible or what it would take.
>> 
>> If there's loose agreement this would be a feasible path I'd be happy to create a JIRA to investigate what this would take.
>> 
>> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <dr...@apple.com>> wrote:
>> +1 on moving the read/write logic into its own jar.
>> 
>> Doug
>> 
>> > On Dec 11, 2024, at 7:21 PM, David Capwell <dc...@apple.com>> wrote:
>> >
>> > From a disk format point of view the only thing I remember was the disk type bug with UDTs.  Bringing that logic back was hard as the type system (in 5.0) tries to avoid allowing construction of invalid states, and we would need to weaken that in order to enable the migration. Assuming the user migrated from 3.x to 4.x then the sstable metadata should have been rewritten to fix this bug.
>> >
>> > One thought (though know its a ton of effort).. we have talked about for a long time about moving the reading/writing logic into its jar (so tools don’t need cassandra-all and can limit the dependencies)… if we did that we could try to solve this as an out of process migration… have the 2.2 reader then write using 6.0 writer (ignoring compact storage… )…
>> >
>> >> On Dec 11, 2024, at 4:59 AM, Benedict <be...@apache.org>> wrote:
>> >>
>> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.
>> >>
>> >>
>> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>> >>>
>> >>> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
>> >>>
>> >>> ________________________________________
>> >>> From: Benedict <be...@apache.org>>
>> >>> Sent: Wednesday, December 11, 2024 13:09
>> >>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
>> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<ma...@cassandra.apache.org>; Miklosovic, Stefan
>> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>> >>>
>> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> 2.2 is particularly hard because of the major storage format changes that took place.
>> >>>
>> >>> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
>> >>>
>> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
>> >>>
>> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>> >>>>
>> >>>> Hey,
>> >>>>
>> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>> >>>>
>> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>> >>>>
>> >>>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>> >>>>
>> >>>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>> >>>>
>> >>>> Regards
>> >>>
>> >>
>> >

Re: Supporting 2.2 -> 5.0 upgrades

[Alex Petrov <al...@coffeenco.de>]

>  I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?

I find it more or less equally painful to make a change in a large Gradle, Maven, or Ant project. I consider myself a pretty active contributor, but I still find myself very rarely interacting with ant, and when I did, it never caused any trouble. That said, there are many other important things that we can improve, so I'd focus on them. Same goes for submodules. 

On Thu, Dec 12, 2024, at 5:53 PM, Paulo Motta wrote:
> >  I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
> 
> This is a contentious/controversial topic, but the more I work with gradle the more I lean towards ant's simplicity. That said, I'd support moving away if it becomes a technical blocker to break up cassandra-all - and if this happen I would vote for maven as replacement. :-D
> 
> On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>> These are all good ideas but in practical terms I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?
>> 
>> ________________________________________
>> From: Paulo Motta <pa...@apache.org>
>> Sent: Thursday, December 12, 2024 17:35
>> To: dev@cassandra.apache.org
>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>> 
>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>> 
>> 
>> 
>> >  +1 on moving the read/write logic into its own jar.
>> 
>> +1, not only read-write logic but anything used by both the server and subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other interfaces.
>> 
>> I think one way to do that would be to split cassandra-all into cassandra-server and cassandra-common (anything used by both subprojects and server), but not sure if this would be feasible or what it would take.
>> 
>> If there's loose agreement this would be a feasible path I'd be happy to create a JIRA to investigate what this would take.
>> 
>> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <dr...@apple.com>> wrote:
>> +1 on moving the read/write logic into its own jar.
>> 
>> Doug
>> 
>> > On Dec 11, 2024, at 7:21 PM, David Capwell <dc...@apple.com>> wrote:
>> >
>> > From a disk format point of view the only thing I remember was the disk type bug with UDTs.  Bringing that logic back was hard as the type system (in 5.0) tries to avoid allowing construction of invalid states, and we would need to weaken that in order to enable the migration. Assuming the user migrated from 3.x to 4.x then the sstable metadata should have been rewritten to fix this bug.
>> >
>> > One thought (though know its a ton of effort).. we have talked about for a long time about moving the reading/writing logic into its jar (so tools don’t need cassandra-all and can limit the dependencies)… if we did that we could try to solve this as an out of process migration… have the 2.2 reader then write using 6.0 writer (ignoring compact storage… )…
>> >
>> >> On Dec 11, 2024, at 4:59 AM, Benedict <be...@apache.org>> wrote:
>> >>
>> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.
>> >>
>> >>
>> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>> >>>
>> >>> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
>> >>>
>> >>> ________________________________________
>> >>> From: Benedict <be...@apache.org>>
>> >>> Sent: Wednesday, December 11, 2024 13:09
>> >>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
>> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<ma...@cassandra.apache.org>; Miklosovic, Stefan
>> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>> >>>
>> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> 2.2 is particularly hard because of the major storage format changes that took place.
>> >>>
>> >>> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
>> >>>
>> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
>> >>>
>> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>> >>>>
>> >>>> Hey,
>> >>>>
>> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>> >>>>
>> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>> >>>>
>> >>>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>> >>>>
>> >>>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>> >>>>
>> >>>> Regards
>> >>>
>> >>
>> >

Re: Supporting 2.2 -> 5.0 upgrades

[Paulo Motta <pa...@apache.org>]

>  I think that will not happen until we are out of Ant as doing this multi
jar / subproject mumbo jumbo is not too much appealing to ... anybody?

This is a contentious/controversial topic, but the more I work with gradle
the more I lean towards ant's simplicity. That said, I'd support moving
away if it becomes a technical blocker to break up cassandra-all - and if
this happen I would vote for maven as replacement. :-D

On Thu, Dec 12, 2024 at 11:42 AM Miklosovic, Stefan via dev <
dev@cassandra.apache.org> wrote:

> These are all good ideas but in practical terms I think that will not
> happen until we are out of Ant as doing this multi jar / subproject mumbo
> jumbo is not too much appealing to ... anybody?
>
> ________________________________________
> From: Paulo Motta <pa...@apache.org>
> Sent: Thursday, December 12, 2024 17:35
> To: dev@cassandra.apache.org
> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>
> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>
>
>
> >  +1 on moving the read/write logic into its own jar.
>
> +1, not only read-write logic but anything used by both the server and
> subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other
> interfaces.
>
> I think one way to do that would be to split cassandra-all into
> cassandra-server and cassandra-common (anything used by both subprojects
> and server), but not sure if this would be feasible or what it would take.
>
> If there's loose agreement this would be a feasible path I'd be happy to
> create a JIRA to investigate what this would take.
>
> On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <drohrer@apple.com<mailto:
> drohrer@apple.com>> wrote:
> +1 on moving the read/write logic into its own jar.
>
> Doug
>
> > On Dec 11, 2024, at 7:21 PM, David Capwell <dcapwell@apple.com<mailto:
> dcapwell@apple.com>> wrote:
> >
> > From a disk format point of view the only thing I remember was the disk
> type bug with UDTs.  Bringing that logic back was hard as the type system
> (in 5.0) tries to avoid allowing construction of invalid states, and we
> would need to weaken that in order to enable the migration. Assuming the
> user migrated from 3.x to 4.x then the sstable metadata should have been
> rewritten to fix this bug.
> >
> > One thought (though know its a ton of effort).. we have talked about for
> a long time about moving the reading/writing logic into its jar (so tools
> don’t need cassandra-all and can limit the dependencies)… if we did that we
> could try to solve this as an out of process migration… have the 2.2 reader
> then write using 6.0 writer (ignoring compact storage… )…
> >
> >> On Dec 11, 2024, at 4:59 AM, Benedict <benedict@apache.org<mailto:
> benedict@apache.org>> wrote:
> >>
> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am
> fairly sure 4.x supported upgrade from 3.0.x also.
> >>
> >>
> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
> >>>
> >>> I see. That makes sense. I think that by 3.x you meant basically the
> latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to
> support 2.2 -> 3.11 straight away. I need to check where we are at in that
> area.
> >>>
> >>> ________________________________________
> >>> From: Benedict <be...@apache.org>>
> >>> Sent: Wednesday, December 11, 2024 13:09
> >>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<mailto:
> dev@cassandra.apache.org>; Miklosovic, Stefan
> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
> >>>
> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
> >>>
> >>>
> >>>
> >>>
> >>> 2.2 is particularly hard because of the major storage format changes
> that took place.
> >>>
> >>> I think if we want to retain (restore) upgrade support from 3.x I
> would support that, but 2.x is probably too burdensome and likely to have
> too many hard edges.
> >>>
> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that
> would be a pretty friendly upgrade path all things considered.
> >>>
> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org<ma...@cassandra.apache.org>> wrote:
> >>>>
> >>>> Hey,
> >>>>
> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0
> would be cool to support.
> >>>>
> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work
> and fixed few issues along the way (1), hence I can imagine that supporting
> 2.2 -> 5.0 would be basically the same thing just on steroids and more
> involved? Anyway, having a stab into this is not useless at all, I will at
> least go deep into the upgrade stuff I have never given a lot of thought to
> which is good learning experience.
> >>>>
> >>>> Any tips where to start? Was any progress done by anybody already in
> this matter to not start from zero?
> >>>>
> >>>> (1)
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
> >>>>
> >>>> Regards
> >>>
> >>
> >
>
>

Re: Supporting 2.2 -> 5.0 upgrades

["Miklosovic, Stefan via dev" <de...@cassandra.apache.org>]

These are all good ideas but in practical terms I think that will not happen until we are out of Ant as doing this multi jar / subproject mumbo jumbo is not too much appealing to ... anybody?

________________________________________
From: Paulo Motta <pa...@apache.org>
Sent: Thursday, December 12, 2024 17:35
To: dev@cassandra.apache.org
Subject: Re: Supporting 2.2 -> 5.0 upgrades

EXTERNAL EMAIL - USE CAUTION when clicking links or attachments



>  +1 on moving the read/write logic into its own jar.

+1, not only read-write logic but anything used by both the server and subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other interfaces.

I think one way to do that would be to split cassandra-all into cassandra-server and cassandra-common (anything used by both subprojects and server), but not sure if this would be feasible or what it would take.

If there's loose agreement this would be a feasible path I'd be happy to create a JIRA to investigate what this would take.

On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <dr...@apple.com>> wrote:
+1 on moving the read/write logic into its own jar.

Doug

> On Dec 11, 2024, at 7:21 PM, David Capwell <dc...@apple.com>> wrote:
>
> From a disk format point of view the only thing I remember was the disk type bug with UDTs.  Bringing that logic back was hard as the type system (in 5.0) tries to avoid allowing construction of invalid states, and we would need to weaken that in order to enable the migration. Assuming the user migrated from 3.x to 4.x then the sstable metadata should have been rewritten to fix this bug.
>
> One thought (though know its a ton of effort).. we have talked about for a long time about moving the reading/writing logic into its jar (so tools don’t need cassandra-all and can limit the dependencies)… if we did that we could try to solve this as an out of process migration… have the 2.2 reader then write using 6.0 writer (ignoring compact storage… )…
>
>> On Dec 11, 2024, at 4:59 AM, Benedict <be...@apache.org>> wrote:
>>
>> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.
>>
>>
>>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>>>
>>> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
>>>
>>> ________________________________________
>>> From: Benedict <be...@apache.org>>
>>> Sent: Wednesday, December 11, 2024 13:09
>>> To: dev@cassandra.apache.org<ma...@cassandra.apache.org>
>>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org<ma...@cassandra.apache.org>; Miklosovic, Stefan
>>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>>
>>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>>
>>>
>>>
>>>
>>> 2.2 is particularly hard because of the major storage format changes that took place.
>>>
>>> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
>>>
>>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
>>>
>>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org>> wrote:
>>>>
>>>> Hey,
>>>>
>>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>>>>
>>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>>>>
>>>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>>>>
>>>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>>>>
>>>> Regards
>>>
>>
>

Re: Supporting 2.2 -> 5.0 upgrades

[Paulo Motta <pa...@apache.org>]

>  +1 on moving the read/write logic into its own jar.

+1, not only read-write logic but anything used by both the server and
subprojects (ie. cassandra-sidecar), for example JMX Mbeans and other
interfaces.

I think one way to do that would be to split cassandra-all into
cassandra-server and cassandra-common (anything used by both subprojects
and server), but not sure if this would be feasible or what it would take.

If there's loose agreement this would be a feasible path I'd be happy to
create a JIRA to investigate what this would take.

On Thu, Dec 12, 2024 at 11:26 AM Doug Rohrer <dr...@apple.com> wrote:

> +1 on moving the read/write logic into its own jar.
>
> Doug
>
> > On Dec 11, 2024, at 7:21 PM, David Capwell <dc...@apple.com> wrote:
> >
> > From a disk format point of view the only thing I remember was the disk
> type bug with UDTs.  Bringing that logic back was hard as the type system
> (in 5.0) tries to avoid allowing construction of invalid states, and we
> would need to weaken that in order to enable the migration. Assuming the
> user migrated from 3.x to 4.x then the sstable metadata should have been
> rewritten to fix this bug.
> >
> > One thought (though know its a ton of effort).. we have talked about for
> a long time about moving the reading/writing logic into its jar (so tools
> don’t need cassandra-all and can limit the dependencies)… if we did that we
> could try to solve this as an out of process migration… have the 2.2 reader
> then write using 6.0 writer (ignoring compact storage… )…
> >
> >> On Dec 11, 2024, at 4:59 AM, Benedict <be...@apache.org> wrote:
> >>
> >> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am
> fairly sure 4.x supported upgrade from 3.0.x also.
> >>
> >>
> >>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org> wrote:
> >>>
> >>> I see. That makes sense. I think that by 3.x you meant basically the
> latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to
> support 2.2 -> 3.11 straight away. I need to check where we are at in that
> area.
> >>>
> >>> ________________________________________
> >>> From: Benedict <be...@apache.org>
> >>> Sent: Wednesday, December 11, 2024 13:09
> >>> To: dev@cassandra.apache.org
> >>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org; Miklosovic, Stefan
> >>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
> >>>
> >>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
> >>>
> >>>
> >>>
> >>>
> >>> 2.2 is particularly hard because of the major storage format changes
> that took place.
> >>>
> >>> I think if we want to retain (restore) upgrade support from 3.x I
> would support that, but 2.x is probably too burdensome and likely to have
> too many hard edges.
> >>>
> >>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that
> would be a pretty friendly upgrade path all things considered.
> >>>
> >>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org> wrote:
> >>>>
> >>>> Hey,
> >>>>
> >>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0
> would be cool to support.
> >>>>
> >>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work
> and fixed few issues along the way (1), hence I can imagine that supporting
> 2.2 -> 5.0 would be basically the same thing just on steroids and more
> involved? Anyway, having a stab into this is not useless at all, I will at
> least go deep into the upgrade stuff I have never given a lot of thought to
> which is good learning experience.
> >>>>
> >>>> Any tips where to start? Was any progress done by anybody already in
> this matter to not start from zero?
> >>>>
> >>>> (1)
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
> >>>>
> >>>> Regards
> >>>
> >>
> >
>
>

Re: Supporting 2.2 -> 5.0 upgrades

[Doug Rohrer <dr...@apple.com>]

+1 on moving the read/write logic into its own jar.

Doug

> On Dec 11, 2024, at 7:21 PM, David Capwell <dc...@apple.com> wrote:
> 
> From a disk format point of view the only thing I remember was the disk type bug with UDTs.  Bringing that logic back was hard as the type system (in 5.0) tries to avoid allowing construction of invalid states, and we would need to weaken that in order to enable the migration. Assuming the user migrated from 3.x to 4.x then the sstable metadata should have been rewritten to fix this bug.
> 
> One thought (though know its a ton of effort).. we have talked about for a long time about moving the reading/writing logic into its jar (so tools don’t need cassandra-all and can limit the dependencies)… if we did that we could try to solve this as an out of process migration… have the 2.2 reader then write using 6.0 writer (ignoring compact storage… )… 
> 
>> On Dec 11, 2024, at 4:59 AM, Benedict <be...@apache.org> wrote:
>> 
>> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.
>> 
>> 
>>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>>> 
>>> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
>>> 
>>> ________________________________________
>>> From: Benedict <be...@apache.org>
>>> Sent: Wednesday, December 11, 2024 13:09
>>> To: dev@cassandra.apache.org
>>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org; Miklosovic, Stefan
>>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>>> 
>>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>>> 
>>> 
>>> 
>>> 
>>> 2.2 is particularly hard because of the major storage format changes that took place.
>>> 
>>> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
>>> 
>>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
>>> 
>>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>>>> 
>>>> Hey,
>>>> 
>>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>>>> 
>>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>>>> 
>>>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>>>> 
>>>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>>>> 
>>>> Regards
>>> 
>> 
> 

Re: Supporting 2.2 -> 5.0 upgrades

[David Capwell <dc...@apple.com>]

From a disk format point of view the only thing I remember was the disk type bug with UDTs.  Bringing that logic back was hard as the type system (in 5.0) tries to avoid allowing construction of invalid states, and we would need to weaken that in order to enable the migration. Assuming the user migrated from 3.x to 4.x then the sstable metadata should have been rewritten to fix this bug.

One thought (though know its a ton of effort).. we have talked about for a long time about moving the reading/writing logic into its jar (so tools don’t need cassandra-all and can limit the dependencies)… if we did that we could try to solve this as an out of process migration… have the 2.2 reader then write using 6.0 writer (ignoring compact storage… )… 

> On Dec 11, 2024, at 4:59 AM, Benedict <be...@apache.org> wrote:
> 
> I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.
> 
> 
>> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>> 
>> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
>> 
>> ________________________________________
>> From: Benedict <be...@apache.org>
>> Sent: Wednesday, December 11, 2024 13:09
>> To: dev@cassandra.apache.org
>> Cc: Miklosovic, Stefan; dev@cassandra.apache.org; Miklosovic, Stefan
>> Subject: Re: Supporting 2.2 -> 5.0 upgrades
>> 
>> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>> 
>> 
>> 
>> 
>> 2.2 is particularly hard because of the major storage format changes that took place.
>> 
>> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
>> 
>> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
>> 
>>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>>> 
>>> Hey,
>>> 
>>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>>> 
>>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>>> 
>>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>>> 
>>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>>> 
>>> Regards
>> 
> 

Re: Supporting 2.2 -> 5.0 upgrades

[Benedict <be...@apache.org>]

I think 3.11 supported upgrade from 2.2, but I haven’t checked. I am fairly sure 4.x supported upgrade from 3.0.x also.


> On 11 Dec 2024, at 12:53, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
> 
> I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.
> 
> ________________________________________
> From: Benedict <be...@apache.org>
> Sent: Wednesday, December 11, 2024 13:09
> To: dev@cassandra.apache.org
> Cc: Miklosovic, Stefan; dev@cassandra.apache.org; Miklosovic, Stefan
> Subject: Re: Supporting 2.2 -> 5.0 upgrades
> 
> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
> 
> 
> 
> 
> 2.2 is particularly hard because of the major storage format changes that took place.
> 
> I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.
> 
> I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.
> 
>> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>> 
>> Hey,
>> 
>> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>> 
>> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>> 
>> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>> 
>> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>> 
>> Regards
> 

Re: Supporting 2.2 -> 5.0 upgrades

["Miklosovic, Stefan via dev" <de...@cassandra.apache.org>]

I see. That makes sense. I think that by 3.x you meant basically the latest 3.11, right? I guess 2.2 -> 3.0 already works, we would just try to support 2.2 -> 3.11 straight away. I need to check where we are at in that area.

________________________________________
From: Benedict <be...@apache.org>
Sent: Wednesday, December 11, 2024 13:09
To: dev@cassandra.apache.org
Cc: Miklosovic, Stefan; dev@cassandra.apache.org; Miklosovic, Stefan
Subject: Re: Supporting 2.2 -> 5.0 upgrades

EXTERNAL EMAIL - USE CAUTION when clicking links or attachments




2.2 is particularly hard because of the major storage format changes that took place.

I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.

I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.

> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
>
> Hey,
>
> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
>
> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
>
> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
>
> (1) https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-19002__;!!Nhn8V6BzJA!RFZoz6sQSrP_qLd0K_eNWO3UAc1s8mTT5SkFalUMwM7_l9gWfb4cnfTFvdY68zsh5-REW7T8ALTPQwqMM_gWWSyp$
>
> Regards

Re: Supporting 2.2 -> 5.0 upgrades

[Benedict <be...@apache.org>]

2.2 is particularly hard because of the major storage format changes that took place.

I think if we want to retain (restore) upgrade support from 3.x I would support that, but 2.x is probably too burdensome and likely to have too many hard edges.

I think if users only had to upgrade 2.2->3.x then eg 3.x->6.0 that would be a pretty friendly upgrade path all things considered.

> On 11 Dec 2024, at 12:03, Miklosovic, Stefan via dev <de...@cassandra.apache.org> wrote:
> 
> Hey,
> 
> I want to fork the thread where we are mentioning that 2.2 -> 5.0 would be cool to support.
> 
> I was involved in checking that offline upgrades from 3.0 to 5.0 work and fixed few issues along the way (1), hence I can imagine that supporting 2.2 -> 5.0 would be basically the same thing just on steroids and more involved? Anyway, having a stab into this is not useless at all, I will at least go deep into the upgrade stuff I have never given a lot of thought to which is good learning experience.
> 
> Any tips where to start? Was any progress done by anybody already in this matter to not start from zero?
> 
> (1) https://issues.apache.org/jira/browse/CASSANDRA-19002
> 
> Regards