[DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

This came up in the release versioning thread and we punted to its own thread.

*Topic: How do we want to handle JDK version support in C* releases?*

Oracle LTS policy here: https://www.oracle.com/java/technologies/java-se-support-roadmap.html

My first rough thoughts:
 1. Any given C* release will only support 2 JDK / JRE at any given time
 2. We only change JDK support on MAJOR releases (i.e. not patch)
 3. When we add support for a new JDK, we drop support for the older in that release
   1. Every consecutive release *must share support for a runtime JDK version with one version older* (this allows for live upgrades, CI, etc). So if we take a long time releasing C* and multiple JDK's rev, we don't jump from [11,17] supported to [21,23] and leave users in a lurch on the straddle
 4. We confirm C* builds on both supported JDK's
 5. We run all test suites against all supported JDK's for a release (yet more incentive to clean up slow tests..)
 6. Language level support will be constrained to oldest JDK supported
 7. We make an effort to support the latest LTS JDK on any given C* release (i.e. 21 now, 25 next major when it goes LTS Sep of this year)
The major downside I can think of with the above is operators will need to verify a new JDK maximally once every 2-3 years as older LTS support is dropped.

What am I not thinking of? What are other downsides with the above proposal?

Re: [DISCUSS] How we handle JDK support

[Brandon Williams <dr...@gmail.com>]

This makes sense to me. The release scripts confirm the JDK being used
now, and this was the impetus to add that.

Kind Regards,
Brandon

On Tue, Jun 10, 2025 at 8:09 AM Josh McKenzie <jm...@apache.org> wrote:
>
> Was drafting up a VOTE thread and have one thing I want to add - anyone think of any major issues w/this I'm missing?
>
> • All supported branches are built on the oldest JDK they support. Our support model is: build on single JDK, run on oldest up to current LTS
>
> On Sat, Jun 7, 2025, at 10:53 AM, Josh McKenzie wrote:
>
> Any other thoughts on this? We think we're ready for a [VOTE] thread?
>
> On Tue, Jun 3, 2025, at 2:07 PM, Josh McKenzie wrote:
>
> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
>
> We already have plenty of cases where we have to write a patch for one branch, merge -s ours, then --amend in a different patch for the subsequent branch. I would expect this process (late-discovered multi-branch applying patches) would be lighter weight than that and far less frequent.
>
> My intuition is that the number of patches that:
>
> Are bugfixes or non-invasive improvements we think are trunk only so leverage latest language level features, that
> are non-trivial in size or scope, and
> we later determine we need to backport to older branches
>
> Is probably going to be vanishingly small. In most cases when working on a bug or improvement it's pretty clear where it needs to land day 1.
>
>
> On Tue, Jun 3, 2025, at 11:36 AM, Štefan Miklošovič wrote:
>
> Hi Josh,
>
> One observation, you wrote:
>
> "Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches."
>
> This is interesting. There are two possibilities
>
> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
>
> 2) The patch would be written in such a way in trunk that it would use language features which would be present in older branches already so a backport would be easier in this regard as the code as such would not need to be reworked if it does not differ functionally a lot otherwise.
>
> It seems to me you have already identified what might happen in your older emails here:
>
> "Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?"
>
> Totally agree.
>
> I do not know what approach I would use by default but thinking about it more I would probably do 2) just for the sake of not rewriting it in older branches. A patch might be complicated already enough and keeping in mind to use newest features in trunk and not using them in older branches is just too much of a mental jugglery to handle for me.
>
> I guess that the discussion when the usage of newest language features would be recommended and justified would follow. I do not think that blindly using the latest greatest _everywhere_ is always good when maintainers need to take care of it for years in the light of backports and bug fixes etc.
>
> It probably also depends on how complex the patch is and if using the newest language level would yield some considerable performance gains etc.
>
> Regards
>
> On Tue, Jun 3, 2025 at 4:38 PM Josh McKenzie <jm...@apache.org> wrote:
>
>
> I was attempting to convey the language level support for a given branch at any snapshot in time; my reading of Doug's email was for something that shouldn't be a problem with this new paradigm (since we would be doing bugfixes targeting oldest then merge up, and new improvements and features should be going trunk only).
>
> Flowing through time, here's what a new release looks like at time of release. I'll format it as:
> C* Version / JDK Build Version / JDK Run Version(s) / Language Level
>
> For our first release:
>
> C* 1.0 / 11 / 11 / 11
>
> Then we release the next C* version; for sake of illustration let's assume a new JDK LTS 12:
>
> C* 2.0 / 12 / 12 / 12
> C* 1.0 / 11 / 11+12 / 11
>
> Notice: we added the ability to run C* 1.0 on JDK12 and otherwise didn't modify the properties of the branch re: JDK's.
>
> Then our 3rd release of C*, same assumption w/JDK LTS 13 support added:
>
> C* 3.0 / 13 / 13 / 13
> C* 2.0 / 12 / 12+13 / 12
> C* 1.0 / 11 / 11+12+13 / 11
>
> The ability to run on the new JDK13 is backported to all supported branches. Otherwise: no JDK-related changes.
>
> And upon our 4th release, we drop support for C*1.0:
>
> C* 4.0 / 14 / 14 / 14
> C* 3.0 / 13 / 13+14 / 13
> C* 2.0 / 12 / 12+13+14 / 12
>
> The properties this gives us:
>
> At any given time as a project we have a JDK version that shares support between 2 branches.
> Any supported version of C* has a JDK that it can be run on that allows upgrade to any other supported version of C* on that same JDK.
> Operators retain the ability to verify a new JDK independently of the new C* version (i.e. bump C* 2.0 from JDK12 to JDK13, then move from C* 2.0 on JDK13 to C* 3.0 on JDK13).
> We are able to aggressively adopt new language level features for improvements and new features on trunk
> Bugfixes that apply to older GA branches will need to target maximally the language level of the oldest branch to which they apply
> Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches
>
> Does this exercise help clarify?
>
> On Tue, Jun 3, 2025, at 8:22 AM, Mick Semb Wever wrote:
>
> What's difficult for me to grok here is you're treating each branch as static, while the discussion has already evolved to branches evolving over time (with jdks being back-ported, etc).  Might be more helpful to explain states and events at different points of branch lifecycles…
>
> On Tue, 3 Jun 2025 at 00:16, Josh McKenzie <jm...@apache.org> wrote:
>
>
> I originally had "everyone supports highest language level whee" which of course would fail to build on older branches.
>
> So this new paradigm would give us the following branch:language level support (assuming JDK bump on each release which also won't always happen):
> - trunk: latest
> - trunk-1: latest lang - 1
> - trunk-2: latest lang - 2
>
> So while trunk-1 and trunk-2 would both support the newest JDK (wherever possible) for runtime, they wouldn't be switched to the new language level. That'd leave us able to use the newest language features on trunk much more rapidly while effectively snapshotting the supported language on older branches to the lowest JDK they support (which, when they're last in line and about to fall off, is the JDK that was newest at the time they came about).
>
> Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?
>
> Unless I'm completely missing something. I was the one who originally just wanted to "latest JDK All The Things" for a hot minute there. =/
>
> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>
> Only thing I’d suggest changing here is “Trunk targets the language level of that JDK” shouldn’t happen until after we’ve confirmed the back port of the new JDK LTS changes to previous versions - otherwise, you have folks starting to use new language features and then have to rip them all out when you find that some previous supported Cassandra release can’t use that JDK.
>
> Doug
>
> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>
> revised snapshot of the state of conversation here:
>
> [New LTS JDK Adoption]
>
> Trunk supports 1 JDK at a time
> After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
> Trunk targets the language level of that JDK
> CI on trunk is that single JDK only
> We merge new JDK LTS support to all supported branches at the same time as trunk
>
> In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
>
> Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
>
> [Branch JDK Support]
>
> N-2: JDK, JDK-1, JDK-2
> N-1: JDK, JDK-1
> N: JDK
>
> [CI, JDK's, Upgrades]
>
> CI:
>
> For each branch we run per-commit CI for the latest JDK they support
> TODO: Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
> TODO: We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads?)
>
> Upgrades
>
> N-2 -> N-1: tested on JDK and JDK-1
> N-2 -> N: tested on JDK
> N-1 -> N: tested on JDK
>
>
> -----------------------
> The above has 2 non-trivial CI orchestration investments:
>
> Running all CI across all supported JDK on a cadence
> Adding some basic perf smoke tests
>
> Both seem reasonable to me.
>
> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>
>
>    .
>
>
>
> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>
>
>
> This.
> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
>
>
>
> I *like* the idea … :)
>
>
>
>
>
>
>

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

Was drafting up a VOTE thread and have one thing I want to add - anyone think of any major issues w/this I'm missing?

• All supported branches are *built on the oldest JDK they support*. Our support model is: build on single JDK, run on oldest up to current LTS

On Sat, Jun 7, 2025, at 10:53 AM, Josh McKenzie wrote:
> Any other thoughts on this? We think we're ready for a [VOTE] thread?
> 
> On Tue, Jun 3, 2025, at 2:07 PM, Josh McKenzie wrote:
>>> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
>> We already have plenty of cases where we have to write a patch for one branch, merge -s ours, then --amend in a different patch for the subsequent branch. I would expect this process (late-discovered multi-branch applying patches) would be lighter weight than that and far less frequent.
>> 
>> My intuition is that the number of patches that:
>>  1. Are bugfixes or non-invasive improvements we think are trunk only so leverage latest language level features, that
>>  2. are non-trivial in size or scope, and
>>  3. we later determine we need to backport to older branches
>> Is probably going to be vanishingly small. In most cases when working on a bug or improvement it's pretty clear where it needs to land day 1.
>> 
>> 
>> On Tue, Jun 3, 2025, at 11:36 AM, Štefan Miklošovič wrote:
>>> Hi Josh,
>>> 
>>> One observation, you wrote:
>>> 
>>> "Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches."
>>> 
>>> This is interesting. There are two possibilities
>>> 
>>> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
>>> 
>>> 2) The patch would be written in such a way in trunk that it would use language features which would be present in older branches already so a backport would be easier in this regard as the code as such would not need to be reworked if it does not differ functionally a lot otherwise.
>>> 
>>> It seems to me you have already identified what might happen in your older emails here:
>>> 
>>> "Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?"
>>> 
>>> Totally agree.
>>> 
>>> I do not know what approach I would use by default but thinking about it more I would probably do 2) just for the sake of not rewriting it in older branches. A patch might be complicated already enough and keeping in mind to use newest features in trunk and not using them in older branches is just too much of a mental jugglery to handle for me.
>>> 
>>> I guess that the discussion when the usage of newest language features would be recommended and justified would follow. I do not think that blindly using the latest greatest _everywhere_ is always good when maintainers need to take care of it for years in the light of backports and bug fixes etc.
>>> 
>>> It probably also depends on how complex the patch is and if using the newest language level would yield some considerable performance gains etc.
>>> 
>>> Regards
>>> 
>>> On Tue, Jun 3, 2025 at 4:38 PM Josh McKenzie <jm...@apache.org> wrote:
>>>> __
>>>> I was attempting to convey the language level support for a given branch at any snapshot in time; my reading of Doug's email was for something that shouldn't be a problem with this new paradigm (since we would be doing bugfixes targeting oldest then merge up, and new improvements and features should be going trunk only).
>>>> 
>>>> Flowing through time, here's what a new release looks like at time of release. I'll format it as:
>>>> *C* Version / JDK Build Version / JDK Run Version(s) / Language Level*
>>>> 
>>>> For our first release:
>>>>  • C* 1.0 / 11 / 11 / 11
>>>> Then we release the next C* version; for sake of illustration let's assume a new JDK LTS 12:
>>>>  • C* 2.0 / 12 / 12 / 12
>>>>  • C* 1.0 / 11 / 11+12 / 11
>>>> Notice: we added the ability to *run* C* 1.0 on JDK12 and otherwise didn't modify the properties of the branch re: JDK's.
>>>> 
>>>> Then our 3rd release of C*, same assumption w/JDK LTS 13 support added:
>>>>  • C* 3.0 / 13 / 13 / 13
>>>>  • C* 2.0 / 12 / 12+13 / 12
>>>>  • C* 1.0 / 11 / 11+12+13 / 11
>>>> The ability to run on the new JDK13 is backported to all supported branches. Otherwise: no JDK-related changes.
>>>> 
>>>> And upon our 4th release, we drop support for C*1.0:
>>>>  • C* 4.0 / 14 / 14 / 14
>>>>  • C* 3.0 / 13 / 13+14 / 13
>>>>  • C* 2.0 / 12 / 12+13+14 / 12
>>>> The properties this gives us:
>>>>  1. At any given time as a project we have a JDK version that shares support between 2 branches.
>>>>  2. Any supported version of C* has a JDK that it can be run on that allows upgrade to any other supported version of C* on that same JDK.
>>>>  3. Operators retain the ability to verify a new JDK independently of the new C* version (i.e. bump C* 2.0 from JDK12 to JDK13, then move from C* 2.0 on JDK13 to C* 3.0 on JDK13).
>>>>  4. We are able to aggressively adopt new language level features for improvements and new features on trunk
>>>>  5. Bugfixes that apply to older GA branches will need to target maximally the language level of the oldest branch to which they apply
>>>>  6. Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches
>>>> Does this exercise help clarify?
>>>> 
>>>> On Tue, Jun 3, 2025, at 8:22 AM, Mick Semb Wever wrote:
>>>>> What's difficult for me to grok here is you're treating each branch as static, while the discussion has already evolved to branches evolving over time (with jdks being back-ported, etc).  Might be more helpful to explain states and events at different points of branch lifecycles… 
>>>>> 
>>>>> On Tue, 3 Jun 2025 at 00:16, Josh McKenzie <jm...@apache.org> wrote:
>>>>>> __
>>>>>> I originally had "everyone supports highest language level whee" which of course would fail to build on older branches.
>>>>>> 
>>>>>> So this new paradigm would give us the following branch:language level support (assuming JDK bump on each release which also won't always happen):
>>>>>> - trunk: latest
>>>>>> - trunk-1: latest lang - 1
>>>>>> - trunk-2: latest lang - 2
>>>>>> 
>>>>>> So while trunk-1 and trunk-2 would both *support* the newest JDK (wherever possible) for runtime, they wouldn't be switched to the new language level. That'd leave us able to use the newest language features on trunk much more rapidly while *effectively snapshotting the supported language on older branches to the lowest JDK they support* (which, when they're last in line and about to fall off, is the JDK that was newest at the time they came about).
>>>>>> 
>>>>>> Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?
>>>>>> 
>>>>>> Unless I'm completely missing something. I was the one who originally just wanted to "latest JDK All The Things" for a hot minute there. =/
>>>>>> 
>>>>>> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>>>>>>> Only thing I’d suggest changing here is “Trunk targets the language level of that JDK” shouldn’t happen until after we’ve confirmed the back port of the new JDK LTS changes to previous versions - otherwise, you have folks starting to use new language features and then have to rip them all out when you find that some previous supported Cassandra release can’t use that JDK.
>>>>>>> 
>>>>>>> Doug
>>>>>>> 
>>>>>>>> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>>>>>>>> 
>>>>>>>> revised snapshot of the state of conversation here:
>>>>>>>> 
>>>>>>>> *[New LTS JDK Adoption]*
>>>>>>>>  • Trunk supports 1 JDK at a time
>>>>>>>>  • After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
>>>>>>>>  • Trunk targets the language level of that JDK
>>>>>>>>  • CI on trunk is that single JDK only
>>>>>>>>  • We merge new JDK LTS support to all supported branches at the same time as trunk
>>>>>>>>    • In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
>>>>>>>>  • Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
>>>>>>>> *[Branch JDK Support]*
>>>>>>>>  • N-2: JDK, JDK-1, JDK-2
>>>>>>>>  • N-1: JDK, JDK-1
>>>>>>>>  • N: JDK
>>>>>>>> *[CI, JDK's, Upgrades]*
>>>>>>>>  • CI:
>>>>>>>>    • For each branch we run per-commit CI for the latest JDK they support
>>>>>>>>    • *TODO: *Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
>>>>>>>>    • *TODO: *We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads? <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>)
>>>>>>>>  • Upgrades
>>>>>>>>    • N-2 -> N-1: tested on JDK and JDK-1
>>>>>>>>    • N-2 -> N: tested on JDK
>>>>>>>>    • N-1 -> N: tested on JDK
>>>>>>>> 
>>>>>>>> -----------------------
>>>>>>>> The above has 2 non-trivial CI orchestration investments:
>>>>>>>>  1. Running all CI across all supported JDK on a cadence
>>>>>>>>  2. Adding some basic perf smoke tests
>>>>>>>> Both seem reasonable to me.
>>>>>>>> 
>>>>>>>> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>>>>>>>>>  
>>>>>>>>>    .
>>>>>>>>>   
>>>>>>>>>   
>>>>>>>>>>> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> This.
>>>>>>>>>> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I *like* the idea … :) 
>>>>>>>> 
>>>>>> 
>>>> 
>> 
> 

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

Any other thoughts on this? We think we're ready for a [VOTE] thread?

On Tue, Jun 3, 2025, at 2:07 PM, Josh McKenzie wrote:
>> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
> We already have plenty of cases where we have to write a patch for one branch, merge -s ours, then --amend in a different patch for the subsequent branch. I would expect this process (late-discovered multi-branch applying patches) would be lighter weight than that and far less frequent.
> 
> My intuition is that the number of patches that:
>  1. Are bugfixes or non-invasive improvements we think are trunk only so leverage latest language level features, that
>  2. are non-trivial in size or scope, and
>  3. we later determine we need to backport to older branches
> Is probably going to be vanishingly small. In most cases when working on a bug or improvement it's pretty clear where it needs to land day 1.
> 
> 
> On Tue, Jun 3, 2025, at 11:36 AM, Štefan Miklošovič wrote:
>> Hi Josh,
>> 
>> One observation, you wrote:
>> 
>> "Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches."
>> 
>> This is interesting. There are two possibilities
>> 
>> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
>> 
>> 2) The patch would be written in such a way in trunk that it would use language features which would be present in older branches already so a backport would be easier in this regard as the code as such would not need to be reworked if it does not differ functionally a lot otherwise.
>> 
>> It seems to me you have already identified what might happen in your older emails here:
>> 
>> "Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?"
>> 
>> Totally agree.
>> 
>> I do not know what approach I would use by default but thinking about it more I would probably do 2) just for the sake of not rewriting it in older branches. A patch might be complicated already enough and keeping in mind to use newest features in trunk and not using them in older branches is just too much of a mental jugglery to handle for me.
>> 
>> I guess that the discussion when the usage of newest language features would be recommended and justified would follow. I do not think that blindly using the latest greatest _everywhere_ is always good when maintainers need to take care of it for years in the light of backports and bug fixes etc.
>> 
>> It probably also depends on how complex the patch is and if using the newest language level would yield some considerable performance gains etc.
>> 
>> Regards
>> 
>> On Tue, Jun 3, 2025 at 4:38 PM Josh McKenzie <jm...@apache.org> wrote:
>>> __
>>> I was attempting to convey the language level support for a given branch at any snapshot in time; my reading of Doug's email was for something that shouldn't be a problem with this new paradigm (since we would be doing bugfixes targeting oldest then merge up, and new improvements and features should be going trunk only).
>>> 
>>> Flowing through time, here's what a new release looks like at time of release. I'll format it as:
>>> *C* Version / JDK Build Version / JDK Run Version(s) / Language Level*
>>> 
>>> For our first release:
>>>  • C* 1.0 / 11 / 11 / 11
>>> Then we release the next C* version; for sake of illustration let's assume a new JDK LTS 12:
>>>  • C* 2.0 / 12 / 12 / 12
>>>  • C* 1.0 / 11 / 11+12 / 11
>>> Notice: we added the ability to *run* C* 1.0 on JDK12 and otherwise didn't modify the properties of the branch re: JDK's.
>>> 
>>> Then our 3rd release of C*, same assumption w/JDK LTS 13 support added:
>>>  • C* 3.0 / 13 / 13 / 13
>>>  • C* 2.0 / 12 / 12+13 / 12
>>>  • C* 1.0 / 11 / 11+12+13 / 11
>>> The ability to run on the new JDK13 is backported to all supported branches. Otherwise: no JDK-related changes.
>>> 
>>> And upon our 4th release, we drop support for C*1.0:
>>>  • C* 4.0 / 14 / 14 / 14
>>>  • C* 3.0 / 13 / 13+14 / 13
>>>  • C* 2.0 / 12 / 12+13+14 / 12
>>> The properties this gives us:
>>>  1. At any given time as a project we have a JDK version that shares support between 2 branches.
>>>  2. Any supported version of C* has a JDK that it can be run on that allows upgrade to any other supported version of C* on that same JDK.
>>>  3. Operators retain the ability to verify a new JDK independently of the new C* version (i.e. bump C* 2.0 from JDK12 to JDK13, then move from C* 2.0 on JDK13 to C* 3.0 on JDK13).
>>>  4. We are able to aggressively adopt new language level features for improvements and new features on trunk
>>>  5. Bugfixes that apply to older GA branches will need to target maximally the language level of the oldest branch to which they apply
>>>  6. Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches
>>> Does this exercise help clarify?
>>> 
>>> On Tue, Jun 3, 2025, at 8:22 AM, Mick Semb Wever wrote:
>>>> What's difficult for me to grok here is you're treating each branch as static, while the discussion has already evolved to branches evolving over time (with jdks being back-ported, etc).  Might be more helpful to explain states and events at different points of branch lifecycles… 
>>>> 
>>>> On Tue, 3 Jun 2025 at 00:16, Josh McKenzie <jm...@apache.org> wrote:
>>>>> __
>>>>> I originally had "everyone supports highest language level whee" which of course would fail to build on older branches.
>>>>> 
>>>>> So this new paradigm would give us the following branch:language level support (assuming JDK bump on each release which also won't always happen):
>>>>> - trunk: latest
>>>>> - trunk-1: latest lang - 1
>>>>> - trunk-2: latest lang - 2
>>>>> 
>>>>> So while trunk-1 and trunk-2 would both *support* the newest JDK (wherever possible) for runtime, they wouldn't be switched to the new language level. That'd leave us able to use the newest language features on trunk much more rapidly while *effectively snapshotting the supported language on older branches to the lowest JDK they support* (which, when they're last in line and about to fall off, is the JDK that was newest at the time they came about).
>>>>> 
>>>>> Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?
>>>>> 
>>>>> Unless I'm completely missing something. I was the one who originally just wanted to "latest JDK All The Things" for a hot minute there. =/
>>>>> 
>>>>> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>>>>>> Only thing I’d suggest changing here is “Trunk targets the language level of that JDK” shouldn’t happen until after we’ve confirmed the back port of the new JDK LTS changes to previous versions - otherwise, you have folks starting to use new language features and then have to rip them all out when you find that some previous supported Cassandra release can’t use that JDK.
>>>>>> 
>>>>>> Doug
>>>>>> 
>>>>>>> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>>>>>>> 
>>>>>>> revised snapshot of the state of conversation here:
>>>>>>> 
>>>>>>> *[New LTS JDK Adoption]*
>>>>>>>  • Trunk supports 1 JDK at a time
>>>>>>>  • After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
>>>>>>>  • Trunk targets the language level of that JDK
>>>>>>>  • CI on trunk is that single JDK only
>>>>>>>  • We merge new JDK LTS support to all supported branches at the same time as trunk
>>>>>>>    • In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
>>>>>>>  • Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
>>>>>>> *[Branch JDK Support]*
>>>>>>>  • N-2: JDK, JDK-1, JDK-2
>>>>>>>  • N-1: JDK, JDK-1
>>>>>>>  • N: JDK
>>>>>>> *[CI, JDK's, Upgrades]*
>>>>>>>  • CI:
>>>>>>>    • For each branch we run per-commit CI for the latest JDK they support
>>>>>>>    • *TODO: *Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
>>>>>>>    • *TODO: *We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads? <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>)
>>>>>>>  • Upgrades
>>>>>>>    • N-2 -> N-1: tested on JDK and JDK-1
>>>>>>>    • N-2 -> N: tested on JDK
>>>>>>>    • N-1 -> N: tested on JDK
>>>>>>> 
>>>>>>> -----------------------
>>>>>>> The above has 2 non-trivial CI orchestration investments:
>>>>>>>  1. Running all CI across all supported JDK on a cadence
>>>>>>>  2. Adding some basic perf smoke tests
>>>>>>> Both seem reasonable to me.
>>>>>>> 
>>>>>>> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>>>>>>>>  
>>>>>>>>    .
>>>>>>>>   
>>>>>>>>   
>>>>>>>>>> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> This.
>>>>>>>>> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I *like* the idea … :) 
>>>>>>> 
>>>>> 
>>> 
> 

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
We already have plenty of cases where we have to write a patch for one branch, merge -s ours, then --amend in a different patch for the subsequent branch. I would expect this process (late-discovered multi-branch applying patches) would be lighter weight than that and far less frequent.

My intuition is that the number of patches that:
 1. Are bugfixes or non-invasive improvements we think are trunk only so leverage latest language level features, that
 2. are non-trivial in size or scope, and
 3. we later determine we need to backport to older branches
Is probably going to be vanishingly small. In most cases when working on a bug or improvement it's pretty clear where it needs to land day 1.


On Tue, Jun 3, 2025, at 11:36 AM, Štefan Miklošovič wrote:
> Hi Josh,
> 
> One observation, you wrote:
> 
> "Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches."
> 
> This is interesting. There are two possibilities
> 
> 1) As you said, in trunk, we would go with the latest language level features and if we backported and these language features are not present there because of older JDK, the patch would need to be rewritten to comply.
> 
> 2) The patch would be written in such a way in trunk that it would use language features which would be present in older branches already so a backport would be easier in this regard as the code as such would not need to be reworked if it does not differ functionally a lot otherwise.
> 
> It seems to me you have already identified what might happen in your older emails here:
> 
> "Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?"
> 
> Totally agree.
> 
> I do not know what approach I would use by default but thinking about it more I would probably do 2) just for the sake of not rewriting it in older branches. A patch might be complicated already enough and keeping in mind to use newest features in trunk and not using them in older branches is just too much of a mental jugglery to handle for me.
> 
> I guess that the discussion when the usage of newest language features would be recommended and justified would follow. I do not think that blindly using the latest greatest _everywhere_ is always good when maintainers need to take care of it for years in the light of backports and bug fixes etc.
> 
> It probably also depends on how complex the patch is and if using the newest language level would yield some considerable performance gains etc.
> 
> Regards
> 
> On Tue, Jun 3, 2025 at 4:38 PM Josh McKenzie <jm...@apache.org> wrote:
>> __
>> I was attempting to convey the language level support for a given branch at any snapshot in time; my reading of Doug's email was for something that shouldn't be a problem with this new paradigm (since we would be doing bugfixes targeting oldest then merge up, and new improvements and features should be going trunk only).
>> 
>> Flowing through time, here's what a new release looks like at time of release. I'll format it as:
>> *C* Version / JDK Build Version / JDK Run Version(s) / Language Level*
>> 
>> For our first release:
>>  • C* 1.0 / 11 / 11 / 11
>> Then we release the next C* version; for sake of illustration let's assume a new JDK LTS 12:
>>  • C* 2.0 / 12 / 12 / 12
>>  • C* 1.0 / 11 / 11+12 / 11
>> Notice: we added the ability to *run* C* 1.0 on JDK12 and otherwise didn't modify the properties of the branch re: JDK's.
>> 
>> Then our 3rd release of C*, same assumption w/JDK LTS 13 support added:
>>  • C* 3.0 / 13 / 13 / 13
>>  • C* 2.0 / 12 / 12+13 / 12
>>  • C* 1.0 / 11 / 11+12+13 / 11
>> The ability to run on the new JDK13 is backported to all supported branches. Otherwise: no JDK-related changes.
>> 
>> And upon our 4th release, we drop support for C*1.0:
>>  • C* 4.0 / 14 / 14 / 14
>>  • C* 3.0 / 13 / 13+14 / 13
>>  • C* 2.0 / 12 / 12+13+14 / 12
>> The properties this gives us:
>>  1. At any given time as a project we have a JDK version that shares support between 2 branches.
>>  2. Any supported version of C* has a JDK that it can be run on that allows upgrade to any other supported version of C* on that same JDK.
>>  3. Operators retain the ability to verify a new JDK independently of the new C* version (i.e. bump C* 2.0 from JDK12 to JDK13, then move from C* 2.0 on JDK13 to C* 3.0 on JDK13).
>>  4. We are able to aggressively adopt new language level features for improvements and new features on trunk
>>  5. Bugfixes that apply to older GA branches will need to target maximally the language level of the oldest branch to which they apply
>>  6. Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches
>> Does this exercise help clarify?
>> 
>> On Tue, Jun 3, 2025, at 8:22 AM, Mick Semb Wever wrote:
>>> What's difficult for me to grok here is you're treating each branch as static, while the discussion has already evolved to branches evolving over time (with jdks being back-ported, etc).  Might be more helpful to explain states and events at different points of branch lifecycles… 
>>> 
>>> On Tue, 3 Jun 2025 at 00:16, Josh McKenzie <jm...@apache.org> wrote:
>>>> __
>>>> I originally had "everyone supports highest language level whee" which of course would fail to build on older branches.
>>>> 
>>>> So this new paradigm would give us the following branch:language level support (assuming JDK bump on each release which also won't always happen):
>>>> - trunk: latest
>>>> - trunk-1: latest lang - 1
>>>> - trunk-2: latest lang - 2
>>>> 
>>>> So while trunk-1 and trunk-2 would both *support* the newest JDK (wherever possible) for runtime, they wouldn't be switched to the new language level. That'd leave us able to use the newest language features on trunk much more rapidly while *effectively snapshotting the supported language on older branches to the lowest JDK they support* (which, when they're last in line and about to fall off, is the JDK that was newest at the time they came about).
>>>> 
>>>> Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?
>>>> 
>>>> Unless I'm completely missing something. I was the one who originally just wanted to "latest JDK All The Things" for a hot minute there. =/
>>>> 
>>>> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>>>>> Only thing I’d suggest changing here is “Trunk targets the language level of that JDK” shouldn’t happen until after we’ve confirmed the back port of the new JDK LTS changes to previous versions - otherwise, you have folks starting to use new language features and then have to rip them all out when you find that some previous supported Cassandra release can’t use that JDK.
>>>>> 
>>>>> Doug
>>>>> 
>>>>>> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>>>>>> 
>>>>>> revised snapshot of the state of conversation here:
>>>>>> 
>>>>>> *[New LTS JDK Adoption]*
>>>>>>  • Trunk supports 1 JDK at a time
>>>>>>  • After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
>>>>>>  • Trunk targets the language level of that JDK
>>>>>>  • CI on trunk is that single JDK only
>>>>>>  • We merge new JDK LTS support to all supported branches at the same time as trunk
>>>>>>    • In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
>>>>>>  • Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
>>>>>> *[Branch JDK Support]*
>>>>>>  • N-2: JDK, JDK-1, JDK-2
>>>>>>  • N-1: JDK, JDK-1
>>>>>>  • N: JDK
>>>>>> *[CI, JDK's, Upgrades]*
>>>>>>  • CI:
>>>>>>    • For each branch we run per-commit CI for the latest JDK they support
>>>>>>    • *TODO: *Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
>>>>>>    • *TODO: *We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads? <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>)
>>>>>>  • Upgrades
>>>>>>    • N-2 -> N-1: tested on JDK and JDK-1
>>>>>>    • N-2 -> N: tested on JDK
>>>>>>    • N-1 -> N: tested on JDK
>>>>>> 
>>>>>> -----------------------
>>>>>> The above has 2 non-trivial CI orchestration investments:
>>>>>>  1. Running all CI across all supported JDK on a cadence
>>>>>>  2. Adding some basic perf smoke tests
>>>>>> Both seem reasonable to me.
>>>>>> 
>>>>>> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>>>>>>>  
>>>>>>>    .
>>>>>>>   
>>>>>>>   
>>>>>>>>> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> This.
>>>>>>>> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
>>>>>>> 
>>>>>>> 
>>>>>>> I *like* the idea … :) 
>>>>>> 
>>>> 
>> 

Re: [DISCUSS] How we handle JDK support

[Štefan Miklošovič <sm...@apache.org>]

Hi Josh,

One observation, you wrote:

"Backports from trunk to older branches that utilize the latest language
level would need to be refactored to work on older branches."

This is interesting. There are two possibilities

1) As you said, in trunk, we would go with the latest language level
features and if we backported and these language features are not present
there because of older JDK, the patch would need to be rewritten to comply.

2) The patch would be written in such a way in trunk that it would use
language features which would be present in older branches already so a
backport would be easier in this regard as the code as such would not need
to be reworked if it does not differ functionally a lot otherwise.

It seems to me you have already identified what might happen in your older
emails here:

"Our risk would be patches going to trunk targeting new language features
we then found out we needed to back-port would require some massaging to be
compatible with older branches. I suspect that'll be a rare edge-case so
seems ok?"

Totally agree.

I do not know what approach I would use by default but thinking about it
more I would probably do 2) just for the sake of not rewriting it in older
branches. A patch might be complicated already enough and keeping in mind
to use newest features in trunk and not using them in older branches is
just too much of a mental jugglery to handle for me.

I guess that the discussion when the usage of newest language features
would be recommended and justified would follow. I do not think that
blindly using the latest greatest _everywhere_ is always good when
maintainers need to take care of it for years in the light of backports and
bug fixes etc.

It probably also depends on how complex the patch is and if using the
newest language level would yield some considerable performance gains etc.

Regards

On Tue, Jun 3, 2025 at 4:38 PM Josh McKenzie <jm...@apache.org> wrote:

> I was attempting to convey the language level support for a given branch
> at any snapshot in time; my reading of Doug's email was for something that
> shouldn't be a problem with this new paradigm (since we would be doing
> bugfixes targeting oldest then merge up, and new improvements and features
> should be going trunk only).
>
> Flowing through time, here's what a new release looks like at time of
> release. I'll format it as:
> *C* Version / JDK Build Version / JDK Run Version(s) / Language Level*
>
> For our first release:
>
>    - C* 1.0 / 11 / 11 / 11
>
> Then we release the next C* version; for sake of illustration let's assume
> a new JDK LTS 12:
>
>    - C* 2.0 / 12 / 12 / 12
>    - C* 1.0 / 11 / 11+12 / 11
>
> Notice: we added the ability to *run* C* 1.0 on JDK12 and otherwise
> didn't modify the properties of the branch re: JDK's.
>
> Then our 3rd release of C*, same assumption w/JDK LTS 13 support added:
>
>    - C* 3.0 / 13 / 13 / 13
>    - C* 2.0 / 12 / 12+13 / 12
>    - C* 1.0 / 11 / 11+12+13 / 11
>
> The ability to run on the new JDK13 is backported to all supported
> branches. Otherwise: no JDK-related changes.
>
> And upon our 4th release, we drop support for C*1.0:
>
>    - C* 4.0 / 14 / 14 / 14
>    - C* 3.0 / 13 / 13+14 / 13
>    - C* 2.0 / 12 / 12+13+14 / 12
>
> The properties this gives us:
>
>    1. At any given time as a project we have a JDK version that shares
>    support between 2 branches.
>    2. Any supported version of C* has a JDK that it can be run on that
>    allows upgrade to any other supported version of C* on that same JDK.
>    3. Operators retain the ability to verify a new JDK independently of
>    the new C* version (i.e. bump C* 2.0 from JDK12 to JDK13, then move from C*
>    2.0 on JDK13 to C* 3.0 on JDK13).
>    4. We are able to aggressively adopt new language level features for
>    improvements and new features on trunk
>    5. Bugfixes that apply to older GA branches will need to target
>    maximally the language level of the oldest branch to which they apply
>    6. Backports from trunk to older branches that utilize the latest
>    language level would need to be refactored to work on older branches
>
> Does this exercise help clarify?
>
> On Tue, Jun 3, 2025, at 8:22 AM, Mick Semb Wever wrote:
>
> What's difficult for me to grok here is you're treating each branch as
> static, while the discussion has already evolved to branches evolving over
> time (with jdks being back-ported, etc).  Might be more helpful to explain
> states and events at different points of branch lifecycles…
>
> On Tue, 3 Jun 2025 at 00:16, Josh McKenzie <jm...@apache.org> wrote:
>
>
> I originally had "everyone supports highest language level whee" which of
> course would fail to build on older branches.
>
> So this new paradigm would give us the following branch:language level
> support (assuming JDK bump on each release which also won't always happen):
> - trunk: latest
> - trunk-1: latest lang - 1
> - trunk-2: latest lang - 2
>
> So while trunk-1 and trunk-2 would both *support* the newest JDK
> (wherever possible) for runtime, they wouldn't be switched to the new
> language level. That'd leave us able to use the newest language features on
> trunk much more rapidly while *effectively snapshotting the supported
> language on older branches to the lowest JDK they support* (which, when
> they're last in line and about to fall off, is the JDK that was newest at
> the time they came about).
>
> Our risk would be patches going to trunk targeting new language features
> we then found out we needed to back-port would require some massaging to be
> compatible with older branches. I suspect that'll be a rare edge-case so
> seems ok?
>
> Unless I'm completely missing something. I was the one who originally just
> wanted to "latest JDK All The Things" for a hot minute there. =/
>
> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>
> Only thing I’d suggest changing here is “Trunk targets the language level
> of that JDK” shouldn’t happen until after we’ve confirmed the back port of
> the new JDK LTS changes to previous versions - otherwise, you have folks
> starting to use new language features and then have to rip them all out
> when you find that some previous supported Cassandra release can’t use that
> JDK.
>
> Doug
>
> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>
> revised snapshot of the state of conversation here:
>
> *[New LTS JDK Adoption]*
>
>    - Trunk supports 1 JDK at a time
>    - After a branch is cut for a release, we push to get trunk to support
>    latest LTS JDK version available at that time
>    - Trunk targets the language level of that JDK
>    - CI on trunk is that single JDK only
>    - We merge new JDK LTS support to all supported branches at the same
>    time as trunk
>       - In the very rare case a feature would have to be removed due to
>       JDK change (think UDF's scripting engine), we instead keep the maximum
>       allowable JDK for that feature supported on trunk and subsequent releases.
>       We then drop that JDK across all branches once the oldest C* w/that feature
>       ages out of support.
>    - Otherwise, we don't need to worry about dropping JDK support as that
>    will happen naturally w/the dropping of support for a branch. Branches will
>    slowly gain JDK support w/each subsequent trunk-based LTS integration.
>
> *[Branch JDK Support]*
>
>    - N-2: JDK, JDK-1, JDK-2
>    - N-1: JDK, JDK-1
>    - N: JDK
>
> *[CI, JDK's, Upgrades]*
>
>    - CI:
>       - For each branch we run per-commit CI for the latest JDK they
>       support
>       - *TODO: *Periodically we run all CI pipelines for older JDK's
>       per-branch (cadence TBD)
>       - *TODO: *We add basic perf testing across all GA branches with
>       reference workloads (easy-cass-stress workloads?
>       <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>
>       )
>    - Upgrades
>       - N-2 -> N-1: tested on JDK and JDK-1
>       - N-2 -> N: tested on JDK
>       - N-1 -> N: tested on JDK
>
>
> -----------------------
> The above has 2 non-trivial CI orchestration investments:
>
>    1. Running all CI across all supported JDK on a cadence
>    2. Adding some basic perf smoke tests
>
> Both seem reasonable to me.
>
> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>
>
>    .
>
>
>
> For the rare edge case where we have to stop supporting something entirely
> because it's incompatible with a JDK release (has this happened more than
> the 1 time?) - I think a reasonable fallback is to just not backport new
> JDK support and consider carrying forward the older JDK support until the
> release w/the feature in it is EoL'ed. That'd allow us to continue to run
> in-jvm upgrade dtests between the versions on the older JDK.
>
>
>
> This.
> I think the idea of adding new major JDKs to release branches for a number
> of reasons, in theory at least.  …
>
>
>
> I *like* the idea … :)
>
>
>
>
>

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

Ok. Final Form before I take it to a VOTE:
-----------------------------------------------
*[New LTS JDK Adoption]*
 • When a new JDK goes LTS, we prioritize:
   • Moving trunk to build, run, pass CI, and support language level of that JDK, dropping others
   • Adding support to *run* on that JDK to all supported GA releases, passing all CI using that JDK
 • These 2 operations must merge atomically. This allows us to preserve the contract of allowing like-to-like JDK's for a live C* upgrade
*[Build, run, language level, Pre Commit CI, EOL]*
 • trunk builds, runs, has CI on, and supports the language level of 1 JDK at any given time (ideally latest LTS JDK)
 • Supported non-trunk GA branches:
   • build, run, pass CI, and support the language level of *the oldest JDK they are certified for*
   • Are supported to *run* on all LTS JDK's between their oldest supported and newest LTS supported by trunk
 • In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. The feature is flagged for deprecate-then-remove or re-implementation based on dev ML discussion. If removed, we drop the required older JDK across all branches when the feature is removed. Supporting new LTS JDK's is considered higher priority than supporting features that JDK's are no longer compatible with, pending debate on the dev ML.
 • Dropping JDK support happens naturally as old releases go EOL.
*[Post Commit CI]*
 • Periodically we will run all CI pipelines for all *runtime* supported JDK's for that branch
 • We will add basic perf testing across all GA branches with reference workloads from easy-cass-stress for a simple performance-based smoke test

On Wed, Jun 11, 2025, at 4:53 PM, Josh McKenzie wrote:
>> it might not always be smooth sailing – we might be losing some agility.  But let's give it a go and find out.
> Yeah, good point. When I bring it to VOTE I'll see if I can't massage the wording a tiny bit around our intent to get to latest LTS support on trunk but retain flexibility to wait if during a burn down to a release, if problems are discovered, if certain things are deprecated and necessitate change or waiting on lifecycle, etc.
> 
> The directional statement of intent - we want to be on the latest JDK on trunk when reasonable so the next release supports latest LTS - is where I think the real value is. Leaves a lot of flexibility though on timing.
> 
> 
> On Wed, Jun 11, 2025, at 4:09 PM, Mick Semb Wever wrote:
>> You replied accurately to what i said, we're aligned.  One point to continue is below…
>> 
>>  
>>> 
>>>> And then, will the confidence of jdk12 in trunk be complete before its merge, and at what point can jdk11 safely be dropped ?
>>>> 
>>>> The action of dropping jdk11 in trunk is just a one-liner in build.xml (actually code removal can happen later), but it's a commitment from us that jdk12 will be production ready.  If attaining that confidence happens post-merge then there's a window where there is a chance we end up having to cut 2.0 with 11+12 ? 
>>> This is an interesting question. Do we have sufficient confidence that when an OpenJDK LTS release hits it's stable? Assuming we can run our entire suite of CI (and add in a future harry stress, a harry soak, and simulator into the mix), what other bar for readiness do we have? 
>>> 
>> 
>> 
>> CI pre-commit doesn't catch everything.  We can assume new LTS jdks come out reasonably safe now, but we've also seen the time it takes to stabilise something post-merge.  Whether it's flakies, test variants that weren't run, especially if we start moving tests to only post-commit weekly runs, or just access to people available to be running harry.  Or bugs reported by users running the new jdk on the older branches that come in late creating a hold up (and forcing a commitment: because there's no alternative jdk now; on) trunk.  
>> 
>> I don't have any objections here, just pointing out it might not always be smooth sailing – we might be losing some agility.  But let's give it a go and find out.
> 

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

> it might not always be smooth sailing – we might be losing some agility.  But let's give it a go and find out.
Yeah, good point. When I bring it to VOTE I'll see if I can't massage the wording a tiny bit around our intent to get to latest LTS support on trunk but retain flexibility to wait if during a burn down to a release, if problems are discovered, if certain things are deprecated and necessitate change or waiting on lifecycle, etc.

The directional statement of intent - we want to be on the latest JDK on trunk when reasonable so the next release supports latest LTS - is where I think the real value is. Leaves a lot of flexibility though on timing.


On Wed, Jun 11, 2025, at 4:09 PM, Mick Semb Wever wrote:
> You replied accurately to what i said, we're aligned.  One point to continue is below…
> 
>  
>> 
>>> And then, will the confidence of jdk12 in trunk be complete before its merge, and at what point can jdk11 safely be dropped ?
>>> 
>>> The action of dropping jdk11 in trunk is just a one-liner in build.xml (actually code removal can happen later), but it's a commitment from us that jdk12 will be production ready.  If attaining that confidence happens post-merge then there's a window where there is a chance we end up having to cut 2.0 with 11+12 ? 
>> This is an interesting question. Do we have sufficient confidence that when an OpenJDK LTS release hits it's stable? Assuming we can run our entire suite of CI (and add in a future harry stress, a harry soak, and simulator into the mix), what other bar for readiness do we have? 
>> 
> 
> 
> CI pre-commit doesn't catch everything.  We can assume new LTS jdks come out reasonably safe now, but we've also seen the time it takes to stabilise something post-merge.  Whether it's flakies, test variants that weren't run, especially if we start moving tests to only post-commit weekly runs, or just access to people available to be running harry.  Or bugs reported by users running the new jdk on the older branches that come in late creating a hold up (and forcing a commitment: because there's no alternative jdk now; on) trunk.  
> 
> I don't have any objections here, just pointing out it might not always be smooth sailing – we might be losing some agility.  But let's give it a go and find out.

Re: [DISCUSS] How we handle JDK support

[Mick Semb Wever <mc...@apache.org>]

You replied accurately to what i said, we're aligned.  One point to
continue is below…



> And then, will the confidence of jdk12 in trunk be complete before its
> merge, and at what point can jdk11 safely be dropped ?
>
> The action of dropping jdk11 in trunk is just a one-liner in build.xml
> (actually code removal can happen later), but it's a commitment from us
> that jdk12 will be production ready.  If attaining that confidence happens
> post-merge then there's a window where there is a chance we end up having
> to cut 2.0 with 11+12 ?
>
> This is an interesting question. Do we have sufficient confidence that
> when an OpenJDK LTS release hits it's stable? Assuming we can run our
> entire suite of CI (and add in a future harry stress, a harry soak, and
> simulator into the mix), what other bar for readiness do we have?
>


CI pre-commit doesn't catch everything.  We can assume new LTS jdks come
out reasonably safe now, but we've also seen the time it takes to stabilise
something post-merge.  Whether it's flakies, test variants that weren't
run, especially if we start moving tests to only post-commit weekly runs,
or just access to people available to be running harry.  Or bugs reported
by users running the new jdk on the older branches that come in late
creating a hold up (and forcing a commitment: because there's no
alternative jdk now; on) trunk.

I don't have any objections here, just pointing out it might not always be
smooth sailing – we might be losing some agility.  But let's give it a go
and find out.

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

> How did jdk12 suddenly appear ?  How did that happen in trunk ? 
> While 2.0 was trunk, it must have started off as jdk11 only. During trunk dev jdk12 was worked on, and the runtime aspect of jdk12 backported to 1.0.
I'm having a hard time tracking what you're saying here. =/

Another way of phrasing what's listed above is:
-------------------------------------------------
When a new JDK goes LTS, we prioritize:
 1. Moving trunk to build and run on that JDK, dropping others
 2. Upping language level on trunk to that JDK
 3. Adding support to run on that JDK to all supported GA releases
We'd need to merge 1 and 3 atomically to preserve the "you can upgrade from a GA version to newest GA on the same JDK" property. The above would also mean we'd add the ability to run all GA versions of C* on the most recent LTS JDK as soon as it's supported on trunk during a patch release cycle. Which I think is probably fine?

A requirement for the above from a CI / confidence perspective would be:
 1. Trunk passes all CI on the new JDK (including potentially new harry stress and simulator test pipelines)
 2. Other GA branches pass all CI running on the new JDK
> This means that jdk12 runtime support was added to 1.0 before 2.0 was branched or released.  
I think this is addressed w/the above clarification. wdyt?

> And then, will the confidence of jdk12 in trunk be complete before its merge, and at what point can jdk11 safely be dropped ?
> 
> The action of dropping jdk11 in trunk is just a one-liner in build.xml (actually code removal can happen later), but it's a commitment from us that jdk12 will be production ready.  If attaining that confidence happens post-merge then there's a window where there is a chance we end up having to cut 2.0 with 11+12 ? 
This is an interesting question. Do we have sufficient confidence that when an OpenJDK LTS release hits it's stable? Assuming we can run our entire suite of CI (and add in a future harry stress, a harry soak, and simulator into the mix), what other bar for readiness do we have? 

On Wed, Jun 11, 2025, at 4:05 AM, Mick Semb Wever wrote:
> Pushing a little for added clarity, see inline, while totally fine with this approach.
> 
> 
> 
>> Then we release the next C* version; for sake of illustration let's assume a new JDK LTS 12:
>>  • C* 2.0 / 12 / 12 / 12
>>  • C* 1.0 / 11 / 11+12 / 11
>> 
> 
> How did jdk12 suddenly appear ?  How did that happen in trunk ? 
> While 2.0 was trunk, it must have started off as jdk11 only. During trunk dev jdk12 was worked on, and the runtime aspect of jdk12 backported to 1.0.
> 
> This means that jdk12 runtime support was added to 1.0 before 2.0 was branched or released.  
> 
> And then, will the confidence of jdk12 in trunk be complete before its merge, and at what point can jdk11 safely be dropped ?
> 
> The action of dropping jdk11 in trunk is just a one-liner in build.xml (actually code removal can happen later), but it's a commitment from us that jdk12 will be production ready.  If attaining that confidence happens post-merge then there's a window where there is a chance we end up having to cut 2.0 with 11+12 ? 
> 
> It's a minor detail thick in the weeds, and maybe easily addressed if/when we hit it, but we should be realistic to our dev input being not always constant and reliable.
> 
> 
>  

Re: [DISCUSS] How we handle JDK support

[Mick Semb Wever <mc...@apache.org>]

Pushing a little for added clarity, see inline, while totally fine with
this approach.



Then we release the next C* version; for sake of illustration let's assume
> a new JDK LTS 12:
>
>    - C* 2.0 / 12 / 12 / 12
>    - C* 1.0 / 11 / 11+12 / 11
>
>
How did jdk12 suddenly appear ?  How did that happen in trunk ?
While 2.0 was trunk, it must have started off as jdk11 only. During trunk
dev jdk12 was worked on, and the runtime aspect of jdk12 backported to 1.0.

This means that jdk12 runtime support was added to 1.0 before 2.0 was
branched or released.

And then, will the confidence of jdk12 in trunk be complete before its
merge, and at what point can jdk11 safely be dropped ?

The action of dropping jdk11 in trunk is just a one-liner in build.xml
(actually code removal can happen later), but it's a commitment from us
that jdk12 will be production ready.  If attaining that confidence happens
post-merge then there's a window where there is a chance we end up having
to cut 2.0 with 11+12 ?

It's a minor detail thick in the weeds, and maybe easily addressed if/when
we hit it, but we should be realistic to our dev input being not always
constant and reliable.

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

I was attempting to convey the language level support for a given branch at any snapshot in time; my reading of Doug's email was for something that shouldn't be a problem with this new paradigm (since we would be doing bugfixes targeting oldest then merge up, and new improvements and features should be going trunk only).

Flowing through time, here's what a new release looks like at time of release. I'll format it as:
*C* Version / JDK Build Version / JDK Run Version(s) / Language Level*

For our first release:
 • C* 1.0 / 11 / 11 / 11
Then we release the next C* version; for sake of illustration let's assume a new JDK LTS 12:
 • C* 2.0 / 12 / 12 / 12
 • C* 1.0 / 11 / 11+12 / 11
Notice: we added the ability to *run* C* 1.0 on JDK12 and otherwise didn't modify the properties of the branch re: JDK's.

Then our 3rd release of C*, same assumption w/JDK LTS 13 support added:
 • C* 3.0 / 13 / 13 / 13
 • C* 2.0 / 12 / 12+13 / 12
 • C* 1.0 / 11 / 11+12+13 / 11
The ability to run on the new JDK13 is backported to all supported branches. Otherwise: no JDK-related changes.

And upon our 4th release, we drop support for C*1.0:
 • C* 4.0 / 14 / 14 / 14
 • C* 3.0 / 13 / 13+14 / 13
 • C* 2.0 / 12 / 12+13+14 / 12
The properties this gives us:
 1. At any given time as a project we have a JDK version that shares support between 2 branches.
 2. Any supported version of C* has a JDK that it can be run on that allows upgrade to any other supported version of C* on that same JDK.
 3. Operators retain the ability to verify a new JDK independently of the new C* version (i.e. bump C* 2.0 from JDK12 to JDK13, then move from C* 2.0 on JDK13 to C* 3.0 on JDK13).
 4. We are able to aggressively adopt new language level features for improvements and new features on trunk
 5. Bugfixes that apply to older GA branches will need to target maximally the language level of the oldest branch to which they apply
 6. Backports from trunk to older branches that utilize the latest language level would need to be refactored to work on older branches
Does this exercise help clarify?

On Tue, Jun 3, 2025, at 8:22 AM, Mick Semb Wever wrote:
> What's difficult for me to grok here is you're treating each branch as static, while the discussion has already evolved to branches evolving over time (with jdks being back-ported, etc).  Might be more helpful to explain states and events at different points of branch lifecycles… 
> 
> On Tue, 3 Jun 2025 at 00:16, Josh McKenzie <jm...@apache.org> wrote:
>> __
>> I originally had "everyone supports highest language level whee" which of course would fail to build on older branches.
>> 
>> So this new paradigm would give us the following branch:language level support (assuming JDK bump on each release which also won't always happen):
>> - trunk: latest
>> - trunk-1: latest lang - 1
>> - trunk-2: latest lang - 2
>> 
>> So while trunk-1 and trunk-2 would both *support* the newest JDK (wherever possible) for runtime, they wouldn't be switched to the new language level. That'd leave us able to use the newest language features on trunk much more rapidly while *effectively snapshotting the supported language on older branches to the lowest JDK they support* (which, when they're last in line and about to fall off, is the JDK that was newest at the time they came about).
>> 
>> Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?
>> 
>> Unless I'm completely missing something. I was the one who originally just wanted to "latest JDK All The Things" for a hot minute there. =/
>> 
>> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>>> Only thing I’d suggest changing here is “Trunk targets the language level of that JDK” shouldn’t happen until after we’ve confirmed the back port of the new JDK LTS changes to previous versions - otherwise, you have folks starting to use new language features and then have to rip them all out when you find that some previous supported Cassandra release can’t use that JDK.
>>> 
>>> Doug
>>> 
>>>> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>>>> 
>>>> revised snapshot of the state of conversation here:
>>>> 
>>>> *[New LTS JDK Adoption]*
>>>>  • Trunk supports 1 JDK at a time
>>>>  • After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
>>>>  • Trunk targets the language level of that JDK
>>>>  • CI on trunk is that single JDK only
>>>>  • We merge new JDK LTS support to all supported branches at the same time as trunk
>>>>    • In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
>>>>  • Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
>>>> *[Branch JDK Support]*
>>>>  • N-2: JDK, JDK-1, JDK-2
>>>>  • N-1: JDK, JDK-1
>>>>  • N: JDK
>>>> *[CI, JDK's, Upgrades]*
>>>>  • CI:
>>>>    • For each branch we run per-commit CI for the latest JDK they support
>>>>    • *TODO: *Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
>>>>    • *TODO: *We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads? <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>)
>>>>  • Upgrades
>>>>    • N-2 -> N-1: tested on JDK and JDK-1
>>>>    • N-2 -> N: tested on JDK
>>>>    • N-1 -> N: tested on JDK
>>>> 
>>>> -----------------------
>>>> The above has 2 non-trivial CI orchestration investments:
>>>>  1. Running all CI across all supported JDK on a cadence
>>>>  2. Adding some basic perf smoke tests
>>>> Both seem reasonable to me.
>>>> 
>>>> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>>>>>  
>>>>>    .
>>>>>   
>>>>>   
>>>>>>> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> This.
>>>>>> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
>>>>> 
>>>>> 
>>>>> I *like* the idea … :) 
>>>> 
>> 

Re: [DISCUSS] How we handle JDK support

[Mick Semb Wever <mc...@apache.org>]

What's difficult for me to grok here is you're treating each branch as
static, while the discussion has already evolved to branches evolving over
time (with jdks being back-ported, etc).  Might be more helpful to explain
states and events at different points of branch lifecycles…

On Tue, 3 Jun 2025 at 00:16, Josh McKenzie <jm...@apache.org> wrote:

> I originally had "everyone supports highest language level whee" which of
> course would fail to build on older branches.
>
> So this new paradigm would give us the following branch:language level
> support (assuming JDK bump on each release which also won't always happen):
> - trunk: latest
> - trunk-1: latest lang - 1
> - trunk-2: latest lang - 2
>
> So while trunk-1 and trunk-2 would both *support* the newest JDK
> (wherever possible) for runtime, they wouldn't be switched to the new
> language level. That'd leave us able to use the newest language features on
> trunk much more rapidly while *effectively snapshotting the supported
> language on older branches to the lowest JDK they support* (which, when
> they're last in line and about to fall off, is the JDK that was newest at
> the time they came about).
>
> Our risk would be patches going to trunk targeting new language features
> we then found out we needed to back-port would require some massaging to be
> compatible with older branches. I suspect that'll be a rare edge-case so
> seems ok?
>
> Unless I'm completely missing something. I was the one who originally just
> wanted to "latest JDK All The Things" for a hot minute there. =/
>
> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>
> Only thing I’d suggest changing here is “Trunk targets the language level
> of that JDK” shouldn’t happen until after we’ve confirmed the back port of
> the new JDK LTS changes to previous versions - otherwise, you have folks
> starting to use new language features and then have to rip them all out
> when you find that some previous supported Cassandra release can’t use that
> JDK.
>
> Doug
>
> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>
> revised snapshot of the state of conversation here:
>
> *[New LTS JDK Adoption]*
>
>    - Trunk supports 1 JDK at a time
>    - After a branch is cut for a release, we push to get trunk to support
>    latest LTS JDK version available at that time
>    - Trunk targets the language level of that JDK
>    - CI on trunk is that single JDK only
>    - We merge new JDK LTS support to all supported branches at the same
>    time as trunk
>       - In the very rare case a feature would have to be removed due to
>       JDK change (think UDF's scripting engine), we instead keep the maximum
>       allowable JDK for that feature supported on trunk and subsequent releases.
>       We then drop that JDK across all branches once the oldest C* w/that feature
>       ages out of support.
>    - Otherwise, we don't need to worry about dropping JDK support as that
>    will happen naturally w/the dropping of support for a branch. Branches will
>    slowly gain JDK support w/each subsequent trunk-based LTS integration.
>
> *[Branch JDK Support]*
>
>    - N-2: JDK, JDK-1, JDK-2
>    - N-1: JDK, JDK-1
>    - N: JDK
>
> *[CI, JDK's, Upgrades]*
>
>    - CI:
>       - For each branch we run per-commit CI for the latest JDK they
>       support
>       - *TODO: *Periodically we run all CI pipelines for older JDK's
>       per-branch (cadence TBD)
>       - *TODO: *We add basic perf testing across all GA branches with
>       reference workloads (easy-cass-stress workloads?
>       <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>
>       )
>    - Upgrades
>       - N-2 -> N-1: tested on JDK and JDK-1
>       - N-2 -> N: tested on JDK
>       - N-1 -> N: tested on JDK
>
>
> -----------------------
> The above has 2 non-trivial CI orchestration investments:
>
>    1. Running all CI across all supported JDK on a cadence
>    2. Adding some basic perf smoke tests
>
> Both seem reasonable to me.
>
> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>
>
>    .
>
>
>
> For the rare edge case where we have to stop supporting something entirely
> because it's incompatible with a JDK release (has this happened more than
> the 1 time?) - I think a reasonable fallback is to just not backport new
> JDK support and consider carrying forward the older JDK support until the
> release w/the feature in it is EoL'ed. That'd allow us to continue to run
> in-jvm upgrade dtests between the versions on the older JDK.
>
>
>
> This.
> I think the idea of adding new major JDKs to release branches for a number
> of reasons, in theory at least.  …
>
>
>
> I *like* the idea … :)
>
>
>
>

Re: [DISCUSS] How we handle JDK support

[Ekaterina Dimitrova <e....@gmail.com>]

I think the risk outlined here is real only if we don’t run the upgrade
tests to trunk, no?

That should be probably the ultimate safeguard. And while pre-commit we do
not run all tests for every patch - I think that does not apply to JDK
addition/removal where we would run all suites normally pre-commit due to
the nature of the change and the blast it has.

On Mon, 2 Jun 2025 at 18:17, Josh McKenzie <jm...@apache.org> wrote:

> I originally had "everyone supports highest language level whee" which of
> course would fail to build on older branches.
>
> So this new paradigm would give us the following branch:language level
> support (assuming JDK bump on each release which also won't always happen):
> - trunk: latest
> - trunk-1: latest lang - 1
> - trunk-2: latest lang - 2
>
> So while trunk-1 and trunk-2 would both *support* the newest JDK
> (wherever possible) for runtime, they wouldn't be switched to the new
> language level. That'd leave us able to use the newest language features on
> trunk much more rapidly while *effectively snapshotting the supported
> language on older branches to the lowest JDK they support* (which, when
> they're last in line and about to fall off, is the JDK that was newest at
> the time they came about).
>
> Our risk would be patches going to trunk targeting new language features
> we then found out we needed to back-port would require some massaging to be
> compatible with older branches. I suspect that'll be a rare edge-case so
> seems ok?
>
> Unless I'm completely missing something. I was the one who originally just
> wanted to "latest JDK All The Things" for a hot minute there. =/
>
> On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
>
> Only thing I’d suggest changing here is “Trunk targets the language level
> of that JDK” shouldn’t happen until after we’ve confirmed the back port of
> the new JDK LTS changes to previous versions - otherwise, you have folks
> starting to use new language features and then have to rip them all out
> when you find that some previous supported Cassandra release can’t use that
> JDK.
>
> Doug
>
> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>
> revised snapshot of the state of conversation here:
>
> *[New LTS JDK Adoption]*
>
>    - Trunk supports 1 JDK at a time
>    - After a branch is cut for a release, we push to get trunk to support
>    latest LTS JDK version available at that time
>    - Trunk targets the language level of that JDK
>    - CI on trunk is that single JDK only
>    - We merge new JDK LTS support to all supported branches at the same
>    time as trunk
>       - In the very rare case a feature would have to be removed due to
>       JDK change (think UDF's scripting engine), we instead keep the maximum
>       allowable JDK for that feature supported on trunk and subsequent releases.
>       We then drop that JDK across all branches once the oldest C* w/that feature
>       ages out of support.
>    - Otherwise, we don't need to worry about dropping JDK support as that
>    will happen naturally w/the dropping of support for a branch. Branches will
>    slowly gain JDK support w/each subsequent trunk-based LTS integration.
>
> *[Branch JDK Support]*
>
>    - N-2: JDK, JDK-1, JDK-2
>    - N-1: JDK, JDK-1
>    - N: JDK
>
> *[CI, JDK's, Upgrades]*
>
>    - CI:
>       - For each branch we run per-commit CI for the latest JDK they
>       support
>       - *TODO: *Periodically we run all CI pipelines for older JDK's
>       per-branch (cadence TBD)
>       - *TODO: *We add basic perf testing across all GA branches with
>       reference workloads (easy-cass-stress workloads?
>       <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>
>       )
>    - Upgrades
>       - N-2 -> N-1: tested on JDK and JDK-1
>       - N-2 -> N: tested on JDK
>       - N-1 -> N: tested on JDK
>
>
> -----------------------
> The above has 2 non-trivial CI orchestration investments:
>
>    1. Running all CI across all supported JDK on a cadence
>    2. Adding some basic perf smoke tests
>
> Both seem reasonable to me.
>
> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>
>
>    .
>
>
>
> For the rare edge case where we have to stop supporting something entirely
> because it's incompatible with a JDK release (has this happened more than
> the 1 time?) - I think a reasonable fallback is to just not backport new
> JDK support and consider carrying forward the older JDK support until the
> release w/the feature in it is EoL'ed. That'd allow us to continue to run
> in-jvm upgrade dtests between the versions on the older JDK.
>
>
>
> This.
> I think the idea of adding new major JDKs to release branches for a number
> of reasons, in theory at least.  …
>
>
>
> I *like* the idea … :)
>
>
>
>

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

I originally had "everyone supports highest language level whee" which of course would fail to build on older branches.

So this new paradigm would give us the following branch:language level support (assuming JDK bump on each release which also won't always happen):
- trunk: latest
- trunk-1: latest lang - 1
- trunk-2: latest lang - 2

So while trunk-1 and trunk-2 would both *support* the newest JDK (wherever possible) for runtime, they wouldn't be switched to the new language level. That'd leave us able to use the newest language features on trunk much more rapidly while *effectively snapshotting the supported language on older branches to the lowest JDK they support* (which, when they're last in line and about to fall off, is the JDK that was newest at the time they came about).

Our risk would be patches going to trunk targeting new language features we then found out we needed to back-port would require some massaging to be compatible with older branches. I suspect that'll be a rare edge-case so seems ok?

Unless I'm completely missing something. I was the one who originally just wanted to "latest JDK All The Things" for a hot minute there. =/

On Mon, Jun 2, 2025, at 9:40 AM, Doug Rohrer wrote:
> Only thing I’d suggest changing here is “Trunk targets the language level of that JDK” shouldn’t happen until after we’ve confirmed the back port of the new JDK LTS changes to previous versions - otherwise, you have folks starting to use new language features and then have to rip them all out when you find that some previous supported Cassandra release can’t use that JDK.
> 
> Doug
> 
>> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
>> 
>> revised snapshot of the state of conversation here:
>> 
>> *[New LTS JDK Adoption]*
>>  • Trunk supports 1 JDK at a time
>>  • After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
>>  • Trunk targets the language level of that JDK
>>  • CI on trunk is that single JDK only
>>  • We merge new JDK LTS support to all supported branches at the same time as trunk
>>    • In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
>>  • Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
>> *[Branch JDK Support]*
>>  • N-2: JDK, JDK-1, JDK-2
>>  • N-1: JDK, JDK-1
>>  • N: JDK
>> *[CI, JDK's, Upgrades]*
>>  • CI:
>>    • For each branch we run per-commit CI for the latest JDK they support
>>    • *TODO: *Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
>>    • *TODO: *We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads? <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>)
>>  • Upgrades
>>    • N-2 -> N-1: tested on JDK and JDK-1
>>    • N-2 -> N: tested on JDK
>>    • N-1 -> N: tested on JDK
>> 
>> -----------------------
>> The above has 2 non-trivial CI orchestration investments:
>>  1. Running all CI across all supported JDK on a cadence
>>  2. Adding some basic perf smoke tests
>> Both seem reasonable to me.
>> 
>> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>>>  
>>>    .
>>>   
>>>   
>>>>> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>>>>> 
>>>> 
>>>> 
>>>> This.
>>>> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
>>> 
>>> 
>>> I *like* the idea … :) 
>> 

Re: [DISCUSS] How we handle JDK support

[Doug Rohrer <dr...@apple.com>]

Only thing I’d suggest changing here is “Trunk targets the language level of that JDK” shouldn’t happen until after we’ve confirmed the back port of the new JDK LTS changes to previous versions - otherwise, you have folks starting to use new language features and then have to rip them all out when you find that some previous supported Cassandra release can’t use that JDK.

Doug

> On May 27, 2025, at 10:37 AM, Josh McKenzie <jm...@apache.org> wrote:
> 
> revised snapshot of the state of conversation here:
> 
> [New LTS JDK Adoption]
> Trunk supports 1 JDK at a time
> After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
> Trunk targets the language level of that JDK
> CI on trunk is that single JDK only
> We merge new JDK LTS support to all supported branches at the same time as trunk
> In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
> Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
> [Branch JDK Support]
> N-2: JDK, JDK-1, JDK-2
> N-1: JDK, JDK-1
> N: JDK
> [CI, JDK's, Upgrades]
> CI:
> For each branch we run per-commit CI for the latest JDK they support
> TODO: Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
> TODO: We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads? <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>)
> Upgrades
> N-2 -> N-1: tested on JDK and JDK-1
> N-2 -> N: tested on JDK
> N-1 -> N: tested on JDK
> 
> -----------------------
> The above has 2 non-trivial CI orchestration investments:
> Running all CI across all supported JDK on a cadence
> Adding some basic perf smoke tests
> Both seem reasonable to me.
> 
> On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>>  
>>    .
>>   
>>   
>> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>> 
>> 
>> 
>> This.
>> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
>> 
>> 
>> I *like* the idea … :) 
> 

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

revised snapshot of the state of conversation here:

*[New LTS JDK Adoption]*
 • Trunk supports 1 JDK at a time
 • After a branch is cut for a release, we push to get trunk to support latest LTS JDK version available at that time
 • Trunk targets the language level of that JDK
 • CI on trunk is that single JDK only
 • We merge new JDK LTS support to all supported branches at the same time as trunk
   • In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine), we instead keep the maximum allowable JDK for that feature supported on trunk and subsequent releases. We then drop that JDK across all branches once the oldest C* w/that feature ages out of support.
 • Otherwise, we don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
*[Branch JDK Support]*
 • N-2: JDK, JDK-1, JDK-2
 • N-1: JDK, JDK-1
 • N: JDK
*[CI, JDK's, Upgrades]*
 • CI:
   • For each branch we run per-commit CI for the latest JDK they support
   • *TODO: *Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
   • *TODO: *We add basic perf testing across all GA branches with reference workloads (easy-cass-stress workloads? <https://github.com/apache/cassandra-easy-stress/tree/main/src/main/kotlin/com/rustyrazorblade/easycassstress/workloads>)
 • Upgrades
   • N-2 -> N-1: tested on JDK and JDK-1
   • N-2 -> N: tested on JDK
   • N-1 -> N: tested on JDK

-----------------------
The above has 2 non-trivial CI orchestration investments:
 1. Running all CI across all supported JDK on a cadence
 2. Adding some basic perf smoke tests
Both seem reasonable to me.

On Fri, May 23, 2025, at 7:39 AM, Mick Semb Wever wrote:
>  
>    .
>   
>   
>>> For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.
>>> 
>> 
>> 
>> This.
>> I think the idea of adding new major JDKs to release branches for a number of reasons, in theory at least.  …
> 
> 
> I *like* the idea … :) 

Re: [DISCUSS] How we handle JDK support

[Mick Semb Wever <mc...@apache.org>]

   .



> For the rare edge case where we have to stop supporting something entirely
>> because it's incompatible with a JDK release (has this happened more than
>> the 1 time?) - I think a reasonable fallback is to just not backport new
>> JDK support and consider carrying forward the older JDK support until the
>> release w/the feature in it is EoL'ed. That'd allow us to continue to run
>> in-jvm upgrade dtests between the versions on the older JDK.
>>
>
>
> This.
> I think the idea of adding new major JDKs to release branches for a number
> of reasons, in theory at least.  …
>


I *like* the idea … :)

Re: [DISCUSS] How we handle JDK support

[Mick Semb Wever <mc...@apache.org>]

   .

>
> For the rare edge case where we have to stop supporting something entirely
> because it's incompatible with a JDK release (has this happened more than
> the 1 time?) - I think a reasonable fallback is to just not backport new
> JDK support and consider carrying forward the older JDK support until the
> release w/the feature in it is EoL'ed. That'd allow us to continue to run
> in-jvm upgrade dtests between the versions on the older JDK.
>


This.
I think the idea of adding new major JDKs to release branches for a number
of reasons, in theory at least.  In practice I think this would have to be
discussed and evaluated for each new JDK and release branch.

And this would mean we can drop a JDK in trunk once it's no longer the
latest in all latest maintained patch versions.

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

Great context - thanks for that insight.

Operators running the older supported versions of C* will retain the *option* to run the older JDK, however if they want to upgrade their JDK version and C* version *separately* under the above paradigm, they'd need to rev their JDK separately on their clusters before running the C* version upgrade.

The need to bump deps for JDK support is very real and that does concern me - really great point. Bumping dependencies in an older C* version because a newer JDK you're not interested in using needed to be supported would not be a positive experience for a user; you're effectively taking on risk for no new functionality. *In theory*, we could have conditional dependency inclusion based on what version of the JDK you're building a cassandra build for. A cursory inspection of this topic in gradle and ant both shows it's possible, if a bit cleaner and simpler in the former than the latter.

My recollection of JDK17 to JDK21 was under 5 differences we needed to bump, so maintaining a per-jdk list of conditional dependency versions wouldn't be an overwhelming burden - at least from that 1 example. Do you recall how many dependencies needed to be bumped on the JDK11 to JDK17 transition Ekaterina?

And your point about our lack of performance testing and JDK changes translating into perf changes is also one that resonates strongly with me as well. There's a straightforward fix there too. :D

For the rare edge case where we have to stop supporting something entirely because it's incompatible with a JDK release (has this happened more than the 1 time?) - I think a reasonable fallback is to just not backport new JDK support and consider carrying forward the older JDK support until the release w/the feature in it is EoL'ed. That'd allow us to continue to run in-jvm upgrade dtests between the versions on the older JDK.

Also - we can't up language level on older branches w/newer JDK support which I hand-waved at in my wishlist. They'd obviously not build on older JDKs if we did that, and that'd force the ecosystem that relies on cassandra-all update at that time as well which wouldn't be pretty.

On Wed, May 21, 2025, at 3:27 PM, Ekaterina Dimitrova wrote:
> Benedict, I am not sure what do you mean by optional feature. FWIW we cannot compile cassandra-4.1 until we removed the feature in cassandra-5.0. I, as a user would be very disappointed a feature to be removed in a patch release. 
> 
> Yes, replacing nashorn was the unpleasant part. I did not raise the nashorn part as if removing the scripted UDFs was a hard technical task, but more to flag we wouldn’t want to make such breaking changes in patch releases.
> 
> “We may well hit similar issues in future, some perhaps even harder to surmount, but I’m sure we can address them as they come on a case by case basis. Worst case we have to postpone the migration by one major for any deprecation to take effect.”
> 
> Agreed, though the lack of performance testing still stands for me. 
> 
> I just got reminded - there was also some time format issue with JDK11 that Scott mentioned before, if I remember correctly.
> 
> So yeah, these are the type of things we may have in front of us. Also, I can’t wait to find a replacement for jamm so we don’t have to think of it anymore. 
> 
> On Wed, 21 May 2025 at 15:17, Benedict <be...@apache.org> wrote:
>> 
>> Yes the issue of Nashorn did spring to mind, but as I recall this was an optional feature. I don’t remember how hard it would have been to simply declare the feature unavailable if you use the newer JDK, but my vague recollection is the hard part was primarily finding a suitable replacement.
>> 
>> We may well hit similar issues in future, some perhaps even harder to surmount, but I’m sure we can address them as they come on a case by case basis. Worst case we have to postpone the migration by one major for any deprecation to take effect.
>> 
>> 
>>> On 21 May 2025, at 19:57, Ekaterina Dimitrova <e....@gmail.com> wrote:
>>> 
>>> “I'm curious what this raises for you. “
>>> 
>>> A few points that come to mind:
>>> 
>>> - every time we switch/add JDKs we also need to do a bunch of changes in CI systems, ccm, etc, not only C* - so more work to call out. Also, if we make older versions support newer JDK, I guess we need to ensure drivers, etc will support it too probably? Are we discussing JDK support here only for Cassandra repo?
>>> - very often we need to bump library versions to support newer JDK versions but at the same time we try not to upgrade dependencies in patch release; only if it is bug related, in most cases
>>> - whether it is a lot of work or not to backport, I’d say it depends. My assumption is that if we keep our maintenance regularly going (which we missed with the long development cycle of 4.0) - it is more feasible. Though we know that we removed a whole feature to move to JDK17 quicker - the scripted UDFs. If we have similar needs at any time - we can’t do such breaking changes in a patch release.
>>> - Benedict made a great point on performance changes with JDK upgrades - we do not have regular performance testing so probably introducing a new JDK in a patch version will come with a huge warning - test thoroughly and move to prod at your own judgement or something like that. 
>>> 
>>> I guess there are more things to consider but these are immediate things that come to my mind now.
>>> 
>>> Best regards,
>>> Ekaterina
>>> 
>>> On Wed, 21 May 2025 at 10:31, Josh McKenzie <jm...@apache.org> wrote:
>>>> __
>>>> Lessons learned from advancing JDK support on trunk *should* translate into older branches making that effort much smaller; Ekaterina you have a lot of experience here so I'm curious what this raises for you. I like the productivity implications of us being able to adopt new language features faster on trunk; I think this is a solid evolution of the idea, definitely.
>>>> 
>>>> Distilling to bulleted lists to try and snapshot the state of the thread w/the above proposal:
>>>> 
>>>> *[New LTS JDK Adoption]*
>>>>  • Trunk supports 1 JDK at a time
>>>>  • That JDK will be the GA LTS the day we cut a frozen branch for a new major (i.e. from moment of previous release bifurcation, trunk snapshots the JDK at that moment). Obviously there will be some flexibility here in terms of when the work lands on trunk and supporting on other branches, but the general pattern / intent hold - push to snapshot latest GA LTS JDK on trunk ASAP after branching for a major.
>>>>  • Trunk targets the language level of that JDK
>>>>  • CI on trunk is that single JDK only
>>>>  • We merge new JDK LTS support to all supported branches at the same time as trunk
>>>>  • We up the supported language level for all supported branches to the latest supported JDK at this time
>>>>  • We don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
>>>> *[Branch JDK Support]*
>>>>  • N-2: JDK, JDK-1, JDK-2
>>>>  • N-1: JDK, JDK-1
>>>>  • N: JDK
>>>> *[CI, JDK's, Upgrades]*
>>>>  • CI:
>>>>    • For each branch we run per-commit CI for the latest JDK they support
>>>>    • Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
>>>>  • Upgrades
>>>>    • N-2 -> N-1: tested on JDK and JDK-1
>>>>    • N-2 -> N: tested on JDK
>>>>    • N-1 -> N: tested on JDK
>>>> That'd give us 4 upgrade paths we'd need to support and test which feels like it's in the territory of "doable on each commit" if we limit the upgrade tests to the in-jvm variety and let the periodic run capture the python upgrade tests space.
>>>> 
>>>> On Wed, May 21, 2025, at 9:30 AM, Benedict wrote:
>>>>> 
>>>>> Perhaps we should consider back porting support for newer Java LTS releases to older C* versions, and suggesting users upgrade JDK first. This way we can have trunk always on the latest LTS, advancing language feature support more quickly. 
>>>>> 
>>>>> That is, we would have something like 
>>>>> 
>>>>> N-2: JDK, JDK-1, JDK-2
>>>>> N-1: JDK, JDK-1
>>>>> N: JDK
>>>>> 
>>>>> I think to assist those deploying trunk and reduce churn for development, we might only want to advance the LTS version for trunk after we release a new major, fixing the next release’s Java version at that point.
>>>>> 
>>>>>>  • On 21 May 2025, at 13:57, Josh McKenzie <jm...@apache.org> wrote:
>>>>>> 
>>>>>>> You don’t have to run every suite on every commit since as folks have pointed out for the most part the JVM isn’t culprit. Need to run it enough times to catch when it is for some assumption of “enough”. 
>>>>>> So riffing on this. We could move to something like:
>>>>>>  • For each given supported C* branch, confirm it **builds **on all supported JDKs (pre-commit verification, post-commit reactive runs)
>>>>>>  • Constrain language level on any given C* branch to **lowest supported JDK**
>>>>>>  • Run all reactive post-commit CI pipelines against *the *highest supported JDK only**
>>>>>>  • Once a N (day, week, month?), run all pipelines against all supported JDKs on all branches
>>>>>>    • Augment notification mechanisms so it squawks to dev list and slack on failure of non-highest JDK pipelines
>>>>>> That approach would tweak our balance towards our perception of the infrequency of per-JDK failures while allowing us to "scale up" the matrix of tests that we perform.
>>>>>> 
>>>>>> i.e. once a week we could have a heavy 9x run (3 branches, 3 JDK's) which we could then plan around and space out in terms of resource allocation, but otherwise we run a single set of pipelines per branch post-commit.
>>>>>> 
>>>>>> That'd give us the confidence to say "we tested the upgrade path we're recommending for you" without having to pay the tax of doing it on every commit or allowing potential defects to pile up to a once-a-year JDK-specific bug-bash.
>>>>>> 
>>>>>> In terms of JDK support when bumping (mapping of relative C* version and relative JDK version):
>>>>>>  • N-2: JDK-2, JDK-3, JDK-4
>>>>>>  • N-1: JDK-1, JDK-2, JDK-3 
>>>>>>  • N: JDK, JDK-1, JDK-2
>>>>>> So we'd have 3 supported LTS per branch, be able to adhere to "you can upgrade from N-2 to N using the same JDK", and allow us to balance our CI coverage to our expected surfacing of defects.
>>>>>> 
>>>>>> Then if we rev JDK we support on any given N+1, we end up with (keeping with N above as reference):
>>>>>>  • N-1: JDK-1, JDK-2, JDK-3
>>>>>>  • N: JDK, JDK-1, JDK-2
>>>>>>  • N+1: JDK+1, JDK, JDK-1
>>>>>> So shared JDK across all 3 on that rev is JDK-1.
>>>>>> 
>>>>>> I think 3 LTS per branch gives us the ability to both add / drop a JDK per major and test / provide for upgrades from N-2 to N w/out requiring a new JDK cert too.
>>>>>> 
>>>>>> On Wed, May 21, 2025, at 3:27 AM, Mick Semb Wever wrote:
>>>>>>>    .
>>>>>>>   
>>>>>>>>> So yeah. I think we'll need to figure out how much coverage is reasonable to call something "tested". I don't think it's sustainable for us to have, at any given time, 3 branches we test across 3 JDK's each with all our in-jvm test suites is it?
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Correct.
>>>>>>> For non-upgrade tests, where testing against more than one jdk exists, we should start the conversation of the value of running more than one JDK for all tests per-commit CI, before we go adding a third.
>>>>>>> 
>>>>>>> I'm not against weekly/fortnightly CI runs, just that it deserves the discussion of cost (it's not necessarily cheaper due to saturation, nor are we a team that has assigned build barons).  The actual change is relatively easy, just adding a profile and a jdk element here: https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135 
>>>>>> 
>>>> 

Re: [DISCUSS] How we handle JDK support

[Ekaterina Dimitrova <e....@gmail.com>]

Benedict, I am not sure what do you mean by optional feature. FWIW we
cannot compile cassandra-4.1 until we removed the feature in cassandra-5.0.
I, as a user would be very disappointed a feature to be removed in a patch
release.

Yes, replacing nashorn was the unpleasant part. I did not raise the nashorn
part as if removing the scripted UDFs was a hard technical task, but more
to flag we wouldn’t want to make such breaking changes in patch releases.

“We may well hit similar issues in future, some perhaps even harder to
surmount, but I’m sure we can address them as they come on a case by case
basis. Worst case we have to postpone the migration by one major for any
deprecation to take effect.”

Agreed, though the lack of performance testing still stands for me.

I just got reminded - there was also some time format issue with JDK11 that
Scott mentioned before, if I remember correctly.

So yeah, these are the type of things we may have in front of us. Also, I
can’t wait to find a replacement for jamm so we don’t have to think of it
anymore.

On Wed, 21 May 2025 at 15:17, Benedict <be...@apache.org> wrote:

> Yes the issue of Nashorn did spring to mind, but as I recall this was an
> optional feature. I don’t remember how hard it would have been to simply
> declare the feature unavailable if you use the newer JDK, but my vague
> recollection is the hard part was primarily finding a suitable replacement.
>
> We may well hit similar issues in future, some perhaps even harder to
> surmount, but I’m sure we can address them as they come on a case by case
> basis. Worst case we have to postpone the migration by one major for any
> deprecation to take effect.
>
> On 21 May 2025, at 19:57, Ekaterina Dimitrova <e....@gmail.com>
> wrote:
>
> 
>
> “I'm curious what this raises for you. “
>
> A few points that come to mind:
>
> - every time we switch/add JDKs we also need to do a bunch of changes in
> CI systems, ccm, etc, not only C* - so more work to call out. Also, if we
> make older versions support newer JDK, I guess we need to ensure drivers,
> etc will support it too probably? Are we discussing JDK support here only
> for Cassandra repo?
> - very often we need to bump library versions to support newer JDK
> versions but at the same time we try not to upgrade dependencies in patch
> release; only if it is bug related, in most cases
> - whether it is a lot of work or not to backport, I’d say it depends. My
> assumption is that if we keep our maintenance regularly going (which we
> missed with the long development cycle of 4.0) - it is more feasible.
> Though we know that we removed a whole feature to move to JDK17 quicker -
> the scripted UDFs. If we have similar needs at any time - we can’t do such
> breaking changes in a patch release.
> - Benedict made a great point on performance changes with JDK upgrades -
> we do not have regular performance testing so probably introducing a new
> JDK in a patch version will come with a huge warning - test thoroughly and
> move to prod at your own judgement or something like that.
>
> I guess there are more things to consider but these are immediate things
> that come to my mind now.
>
> Best regards,
> Ekaterina
>
> On Wed, 21 May 2025 at 10:31, Josh McKenzie <jm...@apache.org> wrote:
>
>> Lessons learned from advancing JDK support on trunk *should* translate
>> into older branches making that effort much smaller; Ekaterina you have a
>> lot of experience here so I'm curious what this raises for you. I like the
>> productivity implications of us being able to adopt new language features
>> faster on trunk; I think this is a solid evolution of the idea, definitely.
>>
>> Distilling to bulleted lists to try and snapshot the state of the thread
>> w/the above proposal:
>>
>> *[New LTS JDK Adoption]*
>>
>>    - Trunk supports 1 JDK at a time
>>    - That JDK will be the GA LTS the day we cut a frozen branch for a
>>    new major (i.e. from moment of previous release bifurcation, trunk
>>    snapshots the JDK at that moment). Obviously there will be some flexibility
>>    here in terms of when the work lands on trunk and supporting on other
>>    branches, but the general pattern / intent hold - push to snapshot latest
>>    GA LTS JDK on trunk ASAP after branching for a major.
>>    - Trunk targets the language level of that JDK
>>    - CI on trunk is that single JDK only
>>    - We merge new JDK LTS support to all supported branches at the same
>>    time as trunk
>>    - We up the supported language level for all supported branches to
>>    the latest supported JDK at this time
>>    - We don't need to worry about dropping JDK support as that will
>>    happen naturally w/the dropping of support for a branch. Branches will
>>    slowly gain JDK support w/each subsequent trunk-based LTS integration.
>>
>> *[Branch JDK Support]*
>>
>>    - N-2: JDK, JDK-1, JDK-2
>>    - N-1: JDK, JDK-1
>>    - N: JDK
>>
>> *[CI, JDK's, Upgrades]*
>>
>>    - CI:
>>       - For each branch we run per-commit CI for the latest JDK they
>>       support
>>       - Periodically we run all CI pipelines for older JDK's per-branch
>>       (cadence TBD)
>>    - Upgrades
>>       - N-2 -> N-1: tested on JDK and JDK-1
>>       - N-2 -> N: tested on JDK
>>       - N-1 -> N: tested on JDK
>>
>> That'd give us 4 upgrade paths we'd need to support and test which feels
>> like it's in the territory of "doable on each commit" if we limit the
>> upgrade tests to the in-jvm variety and let the periodic run capture the
>> python upgrade tests space.
>>
>> On Wed, May 21, 2025, at 9:30 AM, Benedict wrote:
>>
>>
>> Perhaps we should consider back porting support for newer Java LTS
>> releases to older C* versions, and suggesting users upgrade JDK first. This
>> way we can have trunk always on the latest LTS, advancing language feature
>> support more quickly.
>>
>> That is, we would have something like
>>
>> N-2: JDK, JDK-1, JDK-2
>> N-1: JDK, JDK-1
>> N: JDK
>>
>> I think to assist those deploying trunk and reduce churn for development,
>> we might only want to advance the LTS version for trunk after we release a
>> new major, fixing the next release’s Java version at that point.
>>
>>
>>    - On 21 May 2025, at 13:57, Josh McKenzie <jm...@apache.org>
>>    wrote:
>>
>> 
>>
>> You don’t have to run every suite on every commit since as folks have
>> pointed out for the most part the JVM isn’t culprit. Need to run it enough
>> times to catch when it is for some assumption of “enough”.
>>
>> So riffing on this. We could move to something like:
>>
>>    - For each given supported C* branch, confirm it *builds *on all
>>    supported JDKs (pre-commit verification, post-commit reactive runs)
>>    - Constrain language level on any given C* branch to *lowest
>>    supported JDK*
>>    - Run all reactive post-commit CI pipelines against *the highest
>>    supported JDK only*
>>    - Once a N (day, week, month?), run all pipelines against all
>>    supported JDKs on all branches
>>       - Augment notification mechanisms so it squawks to dev list and
>>       slack on failure of non-highest JDK pipelines
>>
>> That approach would tweak our balance towards our perception of the
>> infrequency of per-JDK failures while allowing us to "scale up" the matrix
>> of tests that we perform.
>>
>> i.e. once a week we could have a heavy 9x run (3 branches, 3 JDK's) which
>> we could then plan around and space out in terms of resource allocation,
>> but otherwise we run a single set of pipelines per branch post-commit.
>>
>> That'd give us the confidence to say "we tested the upgrade path we're
>> recommending for you" without having to pay the tax of doing it on every
>> commit or allowing potential defects to pile up to a once-a-year
>> JDK-specific bug-bash.
>>
>> In terms of JDK support when bumping (mapping of relative C* version and
>> relative JDK version):
>>
>>    - N-2: JDK-2, JDK-3, JDK-4
>>    - N-1: JDK-1, JDK-2, JDK-3
>>    - N: JDK, JDK-1, JDK-2
>>
>> So we'd have 3 supported LTS per branch, be able to adhere to "you can
>> upgrade from N-2 to N using the same JDK", and allow us to balance our CI
>> coverage to our expected surfacing of defects.
>>
>> Then if we rev JDK we support on any given N+1, we end up with (keeping
>> with N above as reference):
>>
>>    - N-1: JDK-1, JDK-2, JDK-3
>>    - N: JDK, JDK-1, JDK-2
>>    - N+1: JDK+1, JDK, JDK-1
>>
>> So shared JDK across all 3 on that rev is JDK-1.
>>
>> I think 3 LTS per branch gives us the ability to both add / drop a JDK
>> per major and test / provide for upgrades from N-2 to N w/out requiring a
>> new JDK cert too.
>>
>> On Wed, May 21, 2025, at 3:27 AM, Mick Semb Wever wrote:
>>
>>    .
>>
>>
>> So yeah. I think we'll need to figure out how much coverage is reasonable
>> to call something "tested". I don't think it's sustainable for us to have,
>> at any given time, 3 branches we test across 3 JDK's each with all our
>> in-jvm test suites is it?
>>
>>
>>
>> Correct.
>> For non-upgrade tests, where testing against more than one jdk exists, we
>> should start the conversation of the value of running more than one JDK for
>> all tests per-commit CI, before we go adding a third.
>>
>> I'm not against weekly/fortnightly CI runs, just that it deserves the
>> discussion of cost (it's not necessarily cheaper due to saturation, nor are
>> we a team that has assigned build barons).  The actual change is relatively
>> easy, just adding a profile and a jdk element here:
>> https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135
>>
>>
>>
>>
>>

Re: [DISCUSS] How we handle JDK support

[Benedict <be...@apache.org>]

Yes the issue of Nashorn did spring to mind, but as I recall this was an
optional feature. I don’t remember how hard it would have been to simply
declare the feature unavailable if you use the newer JDK, but my vague
recollection is the hard part was primarily finding a suitable replacement.

  

We may well hit similar issues in future, some perhaps even harder to
surmount, but I’m sure we can address them as they come on a case by case
basis. Worst case we have to postpone the migration by one major for any
deprecation to take effect.

  

> On 21 May 2025, at 19:57, Ekaterina Dimitrova <e....@gmail.com> wrote:  
>  
>

> 
>
> “I'm curious what this raises for you. “  
>
>
>  
>
>
> A few points that come to mind:
>
>  
>
>
> \- every time we switch/add JDKs we also need to do a bunch of changes in CI
> systems, ccm, etc, not only C* - so more work to call out. Also, if we make
> older versions support newer JDK, I guess we need to ensure drivers, etc
> will support it too probably? Are we discussing JDK support here only for
> Cassandra repo?
>
> \- very often we need to bump library versions to support newer JDK versions
> but at the same time we try not to upgrade dependencies in patch release;
> only if it is bug related, in most cases
>
> \- whether it is a lot of work or not to backport, I’d say it depends. My
> assumption is that if we keep our maintenance regularly going (which we
> missed with the long development cycle of 4.0) - it is more feasible. Though
> we know that we removed a whole feature to move to JDK17 quicker - the
> scripted UDFs. If we have similar needs at any time - we can’t do such
> breaking changes in a patch release.
>
> \- Benedict made a great point on performance changes with JDK upgrades - we
> do not have regular performance testing so probably introducing a new JDK in
> a patch version will come with a huge warning - test thoroughly and move to
> prod at your own judgement or something like that.
>
>  
>
>
> I guess there are more things to consider but these are immediate things
> that come to my mind now.
>
>  
>
>
> Best regards,
>
> Ekaterina
>
>  
>
>
> On Wed, 21 May 2025 at 10:31, Josh McKenzie
> <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:  
>
>

>>  __

>>

>> Lessons learned from advancing JDK support on trunk _should_ translate into
older branches making that effort much smaller; Ekaterina you have a lot of
experience here so I'm curious what this raises for you. I like the
productivity implications of us being able to adopt new language features
faster on trunk; I think this is a solid evolution of the idea, definitely.

>>

>>  
>
>>

>> Distilling to bulleted lists to try and snapshot the state of the thread
w/the above proposal:

>>

>>  
>
>>

>>  **[New LTS JDK Adoption]**

>>

>>   * Trunk supports 1 JDK at a time

>>   * That JDK will be the GA LTS the day we cut a frozen branch for a new
major (i.e. from moment of previous release bifurcation, trunk snapshots the
JDK at that moment). Obviously there will be some flexibility here in terms of
when the work lands on trunk and supporting on other branches, but the general
pattern / intent hold - push to snapshot latest GA LTS JDK on trunk ASAP after
branching for a major.

>>   * Trunk targets the language level of that JDK

>>

>>   * CI on trunk is that single JDK only  
>
>>

>>   * We merge new JDK LTS support to all supported branches at the same time
as trunk

>>   * We up the supported language level for all supported branches to the
latest supported JDK at this time

>>   * We don't need to worry about dropping JDK support as that will happen
naturally w/the dropping of support for a branch. Branches will slowly gain
JDK support w/each subsequent trunk-based LTS integration.

>>

>>

>>  **[Branch JDK Support]**

>>

>>   * N-2: JDK, JDK-1, JDK-2

>>   * N-1: JDK, JDK-1

>>   * N: JDK

>>

>>

>>  **[CI, JDK's, Upgrades]**

>>

>>   * CI:

>>     * For each branch we run per-commit CI for the latest JDK they support

>>     * Periodically we run all CI pipelines for older JDK's per-branch
(cadence TBD)

>>   * Upgrades

>>     * N-2 -> N-1: tested on JDK and JDK-1

>>     * N-2 -> N: tested on JDK

>>     * N-1 -> N: tested on JDK

>>

>>

>> That'd give us 4 upgrade paths we'd need to support and test which feels
like it's in the territory of "doable on each commit" if we limit the upgrade
tests to the in-jvm variety and let the periodic run capture the python
upgrade tests space.

>>

>>  
>
>>

>> On Wed, May 21, 2025, at 9:30 AM, Benedict wrote:

>>

>>>  
>
>>>

>>> Perhaps we should consider back porting support for newer Java LTS
releases to older C* versions, and suggesting users upgrade JDK first. This
way we can have trunk always on the latest LTS, advancing language feature
support more quickly.

>>>

>>>  
>
>>>

>>> That is, we would have something like

>>>

>>>  
>
>>>

>>> N-2: JDK, JDK-1, JDK-2

>>>

>>> N-1: JDK, JDK-1

>>>

>>> N: JDK

>>>

>>>  
>
>>>

>>> I think to assist those deploying trunk and reduce churn for development,
we might only want to advance the LTS version for trunk after we release a new
major, fixing the next release’s Java version at that point.

>>>

>>>  
>
>>>

>>>>   * On 21 May 2025, at 13:57, Josh McKenzie
<[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:

>>>>

>>>

>>>> 

>>>>

>>>>> You don’t have to run every suite on every commit since as folks have
pointed out for the most part the JVM isn’t culprit. Need to run it enough
times to catch when it is for some assumption of “enough”.

>>>>

>>>> So riffing on this. We could move to something like:

>>>>

>>>>   * For each given supported C* branch, confirm it **_builds_** on all
supported JDKs (pre-commit verification, post-commit reactive runs)

>>>>   * Constrain language level on any given C* branch to  _ **lowest
supported JDK**_

>>>>   * Run all reactive post-commit CI pipelines against  _the **highest
supported JDK only**_

>>>>   * Once a N (day, week, month?), run all pipelines against all supported
JDKs on all branches

>>>>     * Augment notification mechanisms so it squawks to dev list and slack
on failure of non-highest JDK pipelines

>>>>

>>>>

>>>> That approach would tweak our balance towards our perception of the
infrequency of per-JDK failures while allowing us to "scale up" the matrix of
tests that we perform.

>>>>

>>>>  
>
>>>>

>>>> i.e. once a week we could have a heavy 9x run (3 branches, 3 JDK's) which
we could then plan around and space out in terms of resource allocation, but
otherwise we run a single set of pipelines per branch post-commit.

>>>>

>>>>  
>
>>>>

>>>> That'd give us the confidence to say "we tested the upgrade path we're
recommending for you" without having to pay the tax of doing it on every
commit or allowing potential defects to pile up to a once-a-year JDK-specific
bug-bash.

>>>>

>>>>  
>
>>>>

>>>> In terms of JDK support when bumping (mapping of relative C* version and
relative JDK version):

>>>>

>>>>   * N-2: JDK-2, JDK-3, JDK-4

>>>>   * N-1: JDK-1, JDK-2, JDK-3

>>>>   * N: JDK, JDK-1, JDK-2

>>>>

>>>>

>>>> So we'd have 3 supported LTS per branch, be able to adhere to "you can
upgrade from N-2 to N using the same JDK", and allow us to balance our CI
coverage to our expected surfacing of defects.

>>>>

>>>>  
>
>>>>

>>>> Then if we rev JDK we support on any given N+1, we end up with (keeping
with N above as reference):

>>>>

>>>>   * N-1: JDK-1, JDK-2, JDK-3

>>>>   * N: JDK, JDK-1, JDK-2

>>>>   * N+1: JDK+1, JDK, JDK-1

>>>>

>>>>

>>>> So shared JDK across all 3 on that rev is JDK-1.

>>>>

>>>>  
>
>>>>

>>>> I think 3 LTS per branch gives us the ability to both add / drop a JDK
per major and test / provide for upgrades from N-2 to N w/out requiring a new
JDK cert too.

>>>>

>>>>  
>
>>>>

>>>> On Wed, May 21, 2025, at 3:27 AM, Mick Semb Wever wrote:

>>>>

>>>>>    .

>>>>>

>>>>>  
>>>>>

>>>>>> > So yeah. I think we'll need to figure out how much coverage is
reasonable to call something "tested". I don't think it's sustainable for us
to have, at any given time, 3 branches we test across 3 JDK's each with all
our in-jvm test suites is it?

>>>>>>>

>>>>>>>  
>
>>>>>

>>>>>  
>
>>>>>

>>>>>  
>
>>>>>

>>>>> Correct.

>>>>>

>>>>> For non-upgrade tests, where testing against more than one jdk exists,
we should start the conversation of the value of running more than one JDK for
all tests per-commit CI, before we go adding a third.

>>>>>

>>>>>  
>
>>>>>

>>>>> I'm not against weekly/fortnightly CI runs, just that it deserves the
discussion of cost (it's not necessarily cheaper due to saturation, nor are we
a team that has assigned build barons).  The actual change is relatively easy,
just adding a profile and a jdk element here:
<https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135>

>>>>

>>>>  
>
>>

>>  
>

Re: [DISCUSS] How we handle JDK support

[Ekaterina Dimitrova <e....@gmail.com>]

“I'm curious what this raises for you. “

A few points that come to mind:

- every time we switch/add JDKs we also need to do a bunch of changes in CI
systems, ccm, etc, not only C* - so more work to call out. Also, if we make
older versions support newer JDK, I guess we need to ensure drivers, etc
will support it too probably? Are we discussing JDK support here only for
Cassandra repo?
- very often we need to bump library versions to support newer JDK versions
but at the same time we try not to upgrade dependencies in patch release;
only if it is bug related, in most cases
- whether it is a lot of work or not to backport, I’d say it depends. My
assumption is that if we keep our maintenance regularly going (which we
missed with the long development cycle of 4.0) - it is more feasible.
Though we know that we removed a whole feature to move to JDK17 quicker -
the scripted UDFs. If we have similar needs at any time - we can’t do such
breaking changes in a patch release.
- Benedict made a great point on performance changes with JDK upgrades - we
do not have regular performance testing so probably introducing a new JDK
in a patch version will come with a huge warning - test thoroughly and move
to prod at your own judgement or something like that.

I guess there are more things to consider but these are immediate things
that come to my mind now.

Best regards,
Ekaterina

On Wed, 21 May 2025 at 10:31, Josh McKenzie <jm...@apache.org> wrote:

> Lessons learned from advancing JDK support on trunk *should* translate
> into older branches making that effort much smaller; Ekaterina you have a
> lot of experience here so I'm curious what this raises for you. I like the
> productivity implications of us being able to adopt new language features
> faster on trunk; I think this is a solid evolution of the idea, definitely.
>
> Distilling to bulleted lists to try and snapshot the state of the thread
> w/the above proposal:
>
> *[New LTS JDK Adoption]*
>
>    - Trunk supports 1 JDK at a time
>    - That JDK will be the GA LTS the day we cut a frozen branch for a new
>    major (i.e. from moment of previous release bifurcation, trunk snapshots
>    the JDK at that moment). Obviously there will be some flexibility here in
>    terms of when the work lands on trunk and supporting on other branches, but
>    the general pattern / intent hold - push to snapshot latest GA LTS JDK on
>    trunk ASAP after branching for a major.
>    - Trunk targets the language level of that JDK
>    - CI on trunk is that single JDK only
>    - We merge new JDK LTS support to all supported branches at the same
>    time as trunk
>    - We up the supported language level for all supported branches to the
>    latest supported JDK at this time
>    - We don't need to worry about dropping JDK support as that will
>    happen naturally w/the dropping of support for a branch. Branches will
>    slowly gain JDK support w/each subsequent trunk-based LTS integration.
>
> *[Branch JDK Support]*
>
>    - N-2: JDK, JDK-1, JDK-2
>    - N-1: JDK, JDK-1
>    - N: JDK
>
> *[CI, JDK's, Upgrades]*
>
>    - CI:
>       - For each branch we run per-commit CI for the latest JDK they
>       support
>       - Periodically we run all CI pipelines for older JDK's per-branch
>       (cadence TBD)
>    - Upgrades
>       - N-2 -> N-1: tested on JDK and JDK-1
>       - N-2 -> N: tested on JDK
>       - N-1 -> N: tested on JDK
>
> That'd give us 4 upgrade paths we'd need to support and test which feels
> like it's in the territory of "doable on each commit" if we limit the
> upgrade tests to the in-jvm variety and let the periodic run capture the
> python upgrade tests space.
>
> On Wed, May 21, 2025, at 9:30 AM, Benedict wrote:
>
>
> Perhaps we should consider back porting support for newer Java LTS
> releases to older C* versions, and suggesting users upgrade JDK first. This
> way we can have trunk always on the latest LTS, advancing language feature
> support more quickly.
>
> That is, we would have something like
>
> N-2: JDK, JDK-1, JDK-2
> N-1: JDK, JDK-1
> N: JDK
>
> I think to assist those deploying trunk and reduce churn for development,
> we might only want to advance the LTS version for trunk after we release a
> new major, fixing the next release’s Java version at that point.
>
>
>    - On 21 May 2025, at 13:57, Josh McKenzie <jm...@apache.org> wrote:
>
> 
>
> You don’t have to run every suite on every commit since as folks have
> pointed out for the most part the JVM isn’t culprit. Need to run it enough
> times to catch when it is for some assumption of “enough”.
>
> So riffing on this. We could move to something like:
>
>    - For each given supported C* branch, confirm it *builds *on all
>    supported JDKs (pre-commit verification, post-commit reactive runs)
>    - Constrain language level on any given C* branch to *lowest supported
>    JDK*
>    - Run all reactive post-commit CI pipelines against *the highest
>    supported JDK only*
>    - Once a N (day, week, month?), run all pipelines against all
>    supported JDKs on all branches
>       - Augment notification mechanisms so it squawks to dev list and
>       slack on failure of non-highest JDK pipelines
>
> That approach would tweak our balance towards our perception of the
> infrequency of per-JDK failures while allowing us to "scale up" the matrix
> of tests that we perform.
>
> i.e. once a week we could have a heavy 9x run (3 branches, 3 JDK's) which
> we could then plan around and space out in terms of resource allocation,
> but otherwise we run a single set of pipelines per branch post-commit.
>
> That'd give us the confidence to say "we tested the upgrade path we're
> recommending for you" without having to pay the tax of doing it on every
> commit or allowing potential defects to pile up to a once-a-year
> JDK-specific bug-bash.
>
> In terms of JDK support when bumping (mapping of relative C* version and
> relative JDK version):
>
>    - N-2: JDK-2, JDK-3, JDK-4
>    - N-1: JDK-1, JDK-2, JDK-3
>    - N: JDK, JDK-1, JDK-2
>
> So we'd have 3 supported LTS per branch, be able to adhere to "you can
> upgrade from N-2 to N using the same JDK", and allow us to balance our CI
> coverage to our expected surfacing of defects.
>
> Then if we rev JDK we support on any given N+1, we end up with (keeping
> with N above as reference):
>
>    - N-1: JDK-1, JDK-2, JDK-3
>    - N: JDK, JDK-1, JDK-2
>    - N+1: JDK+1, JDK, JDK-1
>
> So shared JDK across all 3 on that rev is JDK-1.
>
> I think 3 LTS per branch gives us the ability to both add / drop a JDK per
> major and test / provide for upgrades from N-2 to N w/out requiring a new
> JDK cert too.
>
> On Wed, May 21, 2025, at 3:27 AM, Mick Semb Wever wrote:
>
>    .
>
>
> So yeah. I think we'll need to figure out how much coverage is reasonable
> to call something "tested". I don't think it's sustainable for us to have,
> at any given time, 3 branches we test across 3 JDK's each with all our
> in-jvm test suites is it?
>
>
>
> Correct.
> For non-upgrade tests, where testing against more than one jdk exists, we
> should start the conversation of the value of running more than one JDK for
> all tests per-commit CI, before we go adding a third.
>
> I'm not against weekly/fortnightly CI runs, just that it deserves the
> discussion of cost (it's not necessarily cheaper due to saturation, nor are
> we a team that has assigned build barons).  The actual change is relatively
> easy, just adding a profile and a jdk element here:
> https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135
>
>
>
>
>

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

Lessons learned from advancing JDK support on trunk *should* translate into older branches making that effort much smaller; Ekaterina you have a lot of experience here so I'm curious what this raises for you. I like the productivity implications of us being able to adopt new language features faster on trunk; I think this is a solid evolution of the idea, definitely.

Distilling to bulleted lists to try and snapshot the state of the thread w/the above proposal:

*[New LTS JDK Adoption]*
 • Trunk supports 1 JDK at a time
 • That JDK will be the GA LTS the day we cut a frozen branch for a new major (i.e. from moment of previous release bifurcation, trunk snapshots the JDK at that moment). Obviously there will be some flexibility here in terms of when the work lands on trunk and supporting on other branches, but the general pattern / intent hold - push to snapshot latest GA LTS JDK on trunk ASAP after branching for a major.
 • Trunk targets the language level of that JDK
 • CI on trunk is that single JDK only
 • We merge new JDK LTS support to all supported branches at the same time as trunk
 • We up the supported language level for all supported branches to the latest supported JDK at this time
 • We don't need to worry about dropping JDK support as that will happen naturally w/the dropping of support for a branch. Branches will slowly gain JDK support w/each subsequent trunk-based LTS integration.
*[Branch JDK Support]*
 • N-2: JDK, JDK-1, JDK-2
 • N-1: JDK, JDK-1
 • N: JDK
*[CI, JDK's, Upgrades]*
 • CI:
   • For each branch we run per-commit CI for the latest JDK they support
   • Periodically we run all CI pipelines for older JDK's per-branch (cadence TBD)
 • Upgrades
   • N-2 -> N-1: tested on JDK and JDK-1
   • N-2 -> N: tested on JDK
   • N-1 -> N: tested on JDK
That'd give us 4 upgrade paths we'd need to support and test which feels like it's in the territory of "doable on each commit" if we limit the upgrade tests to the in-jvm variety and let the periodic run capture the python upgrade tests space.

On Wed, May 21, 2025, at 9:30 AM, Benedict wrote:
> 
> Perhaps we should consider back porting support for newer Java LTS releases to older C* versions, and suggesting users upgrade JDK first. This way we can have trunk always on the latest LTS, advancing language feature support more quickly. 
> 
> That is, we would have something like 
> 
> N-2: JDK, JDK-1, JDK-2
> N-1: JDK, JDK-1
> N: JDK
> 
> I think to assist those deploying trunk and reduce churn for development, we might only want to advance the LTS version for trunk after we release a new major, fixing the next release’s Java version at that point.
> 
>>  • On 21 May 2025, at 13:57, Josh McKenzie <jm...@apache.org> wrote:
>> 
>>> You don’t have to run every suite on every commit since as folks have pointed out for the most part the JVM isn’t culprit. Need to run it enough times to catch when it is for some assumption of “enough”. 
>> So riffing on this. We could move to something like:
>>  • For each given supported C* branch, confirm it **builds **on all supported JDKs (pre-commit verification, post-commit reactive runs)
>>  • Constrain language level on any given C* branch to **lowest supported JDK**
>>  • Run all reactive post-commit CI pipelines against *the *highest supported JDK only**
>>  • Once a N (day, week, month?), run all pipelines against all supported JDKs on all branches
>>    • Augment notification mechanisms so it squawks to dev list and slack on failure of non-highest JDK pipelines
>> That approach would tweak our balance towards our perception of the infrequency of per-JDK failures while allowing us to "scale up" the matrix of tests that we perform.
>> 
>> i.e. once a week we could have a heavy 9x run (3 branches, 3 JDK's) which we could then plan around and space out in terms of resource allocation, but otherwise we run a single set of pipelines per branch post-commit.
>> 
>> That'd give us the confidence to say "we tested the upgrade path we're recommending for you" without having to pay the tax of doing it on every commit or allowing potential defects to pile up to a once-a-year JDK-specific bug-bash.
>> 
>> In terms of JDK support when bumping (mapping of relative C* version and relative JDK version):
>>  • N-2: JDK-2, JDK-3, JDK-4
>>  • N-1: JDK-1, JDK-2, JDK-3 
>>  • N: JDK, JDK-1, JDK-2
>> So we'd have 3 supported LTS per branch, be able to adhere to "you can upgrade from N-2 to N using the same JDK", and allow us to balance our CI coverage to our expected surfacing of defects.
>> 
>> Then if we rev JDK we support on any given N+1, we end up with (keeping with N above as reference):
>>  • N-1: JDK-1, JDK-2, JDK-3
>>  • N: JDK, JDK-1, JDK-2
>>  • N+1: JDK+1, JDK, JDK-1
>> So shared JDK across all 3 on that rev is JDK-1.
>> 
>> I think 3 LTS per branch gives us the ability to both add / drop a JDK per major and test / provide for upgrades from N-2 to N w/out requiring a new JDK cert too.
>> 
>> On Wed, May 21, 2025, at 3:27 AM, Mick Semb Wever wrote:
>>>    .
>>>   
>>>>> So yeah. I think we'll need to figure out how much coverage is reasonable to call something "tested". I don't think it's sustainable for us to have, at any given time, 3 branches we test across 3 JDK's each with all our in-jvm test suites is it?
>>>>> 
>>> 
>>> 
>>> Correct.
>>> For non-upgrade tests, where testing against more than one jdk exists, we should start the conversation of the value of running more than one JDK for all tests per-commit CI, before we go adding a third.
>>> 
>>> I'm not against weekly/fortnightly CI runs, just that it deserves the discussion of cost (it's not necessarily cheaper due to saturation, nor are we a team that has assigned build barons).  The actual change is relatively easy, just adding a profile and a jdk element here: https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135 
>> 

Re: [DISCUSS] How we handle JDK support

[Benedict <be...@apache.org>]

Perhaps we should consider back porting support for newer Java LTS releases to older C* versions, and suggesting users upgrade JDK first. This way we can have trunk always on the latest LTS, advancing language feature support more quickly. 

That is, we would have something like 

N-2: JDK, JDK-1, JDK-2
N-1: JDK, JDK-1
N: JDK

I think to assist those deploying trunk and reduce churn for development, we might only want to advance the LTS version for trunk after we release a new major, fixing the next release’s Java version at that point.

> On 21 May 2025, at 13:57, Josh McKenzie <jm...@apache.org> wrote:
> 
>> 
>> You don’t have to run every suite on every commit since as folks have pointed out for the most part the JVM isn’t culprit. Need to run it enough times to catch when it is for some assumption of “enough”. 
> So riffing on this. We could move to something like:
> For each given supported C* branch, confirm it builds on all supported JDKs (pre-commit verification, post-commit reactive runs)
> Constrain language level on any given C* branch to lowest supported JDK
> Run all reactive post-commit CI pipelines against the highest supported JDK only
> Once a N (day, week, month?), run all pipelines against all supported JDKs on all branches
> Augment notification mechanisms so it squawks to dev list and slack on failure of non-highest JDK pipelines
> That approach would tweak our balance towards our perception of the infrequency of per-JDK failures while allowing us to "scale up" the matrix of tests that we perform.
> 
> i.e. once a week we could have a heavy 9x run (3 branches, 3 JDK's) which we could then plan around and space out in terms of resource allocation, but otherwise we run a single set of pipelines per branch post-commit.
> 
> That'd give us the confidence to say "we tested the upgrade path we're recommending for you" without having to pay the tax of doing it on every commit or allowing potential defects to pile up to a once-a-year JDK-specific bug-bash.
> 
> In terms of JDK support when bumping (mapping of relative C* version and relative JDK version):
> N-2: JDK-2, JDK-3, JDK-4
> N-1: JDK-1, JDK-2, JDK-3 
> N: JDK, JDK-1, JDK-2
> So we'd have 3 supported LTS per branch, be able to adhere to "you can upgrade from N-2 to N using the same JDK", and allow us to balance our CI coverage to our expected surfacing of defects.
> 
> Then if we rev JDK we support on any given N+1, we end up with (keeping with N above as reference):
> N-1: JDK-1, JDK-2, JDK-3
> N: JDK, JDK-1, JDK-2
> N+1: JDK+1, JDK, JDK-1
> So shared JDK across all 3 on that rev is JDK-1.
> 
> I think 3 LTS per branch gives us the ability to both add / drop a JDK per major and test / provide for upgrades from N-2 to N w/out requiring a new JDK cert too.
> 
>> On Wed, May 21, 2025, at 3:27 AM, Mick Semb Wever wrote:
>>    .
>>   
>> So yeah. I think we'll need to figure out how much coverage is reasonable to call something "tested". I don't think it's sustainable for us to have, at any given time, 3 branches we test across 3 JDK's each with all our in-jvm test suites is it?
>> 
>> 
>> 
>> Correct.
>> For non-upgrade tests, where testing against more than one jdk exists, we should start the conversation of the value of running more than one JDK for all tests per-commit CI, before we go adding a third.
>> 
>> I'm not against weekly/fortnightly CI runs, just that it deserves the discussion of cost (it's not necessarily cheaper due to saturation, nor are we a team that has assigned build barons).  The actual change is relatively easy, just adding a profile and a jdk element here: https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135 
> 

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

> You don’t have to run every suite on every commit since as folks have pointed out for the most part the JVM isn’t culprit. Need to run it enough times to catch when it is for some assumption of “enough”. 
So riffing on this. We could move to something like:
 • For each given supported C* branch, confirm it **builds **on all supported JDKs (pre-commit verification, post-commit reactive runs)
 • Constrain language level on any given C* branch to **lowest supported JDK**
 • Run all reactive post-commit CI pipelines against *the *highest supported JDK only**
 • Once a N (day, week, month?), run all pipelines against all supported JDKs on all branches
   • Augment notification mechanisms so it squawks to dev list and slack on failure of non-highest JDK pipelines
That approach would tweak our balance towards our perception of the infrequency of per-JDK failures while allowing us to "scale up" the matrix of tests that we perform.

i.e. once a week we could have a heavy 9x run (3 branches, 3 JDK's) which we could then plan around and space out in terms of resource allocation, but otherwise we run a single set of pipelines per branch post-commit.

That'd give us the confidence to say "we tested the upgrade path we're recommending for you" without having to pay the tax of doing it on every commit or allowing potential defects to pile up to a once-a-year JDK-specific bug-bash.

In terms of JDK support when bumping (mapping of relative C* version and relative JDK version):
 • N-2: JDK-2, JDK-3, JDK-4
 • N-1: JDK-1, JDK-2, JDK-3 
 • N: JDK, JDK-1, JDK-2
So we'd have 3 supported LTS per branch, be able to adhere to "you can upgrade from N-2 to N using the same JDK", and allow us to balance our CI coverage to our expected surfacing of defects.

Then if we rev JDK we support on any given N+1, we end up with (keeping with N above as reference):
 • N-1: JDK-1, JDK-2, JDK-3
 • N: JDK, JDK-1, JDK-2
 • N+1: JDK+1, JDK, JDK-1
So shared JDK across all 3 on that rev is JDK-1.

I think 3 LTS per branch gives us the ability to both add / drop a JDK per major and test / provide for upgrades from N-2 to N w/out requiring a new JDK cert too.

On Wed, May 21, 2025, at 3:27 AM, Mick Semb Wever wrote:
>    .
>   
>>> So yeah. I think we'll need to figure out how much coverage is reasonable to call something "tested". I don't think it's sustainable for us to have, at any given time, 3 branches we test across 3 JDK's each with all our in-jvm test suites is it?
>>> 
> 
> 
> Correct.
> For non-upgrade tests, where testing against more than one jdk exists, we should start the conversation of the value of running more than one JDK for all tests per-commit CI, before we go adding a third.
> 
> I'm not against weekly/fortnightly CI runs, just that it deserves the discussion of cost (it's not necessarily cheaper due to saturation, nor are we a team that has assigned build barons).  The actual change is relatively easy, just adding a profile and a jdk element here: https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135 

Re: [DISCUSS] How we handle JDK support

[Mick Semb Wever <mc...@apache.org>]

   .


> So yeah. I think we'll need to figure out how much coverage is reasonable
>> to call something "tested". I don't think it's sustainable for us to have,
>> at any given time, 3 branches we test across 3 JDK's each with all our
>> in-jvm test suites is it?
>>
>

Correct.
For non-upgrade tests, where testing against more than one jdk exists, we
should start the conversation of the value of running more than one JDK for
all tests per-commit CI, before we go adding a third.

I'm not against weekly/fortnightly CI runs, just that it deserves the
discussion of cost (it's not necessarily cheaper due to saturation, nor are
we a team that has assigned build barons).  The actual change is relatively
easy, just adding a profile and a jdk element here:
https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile#L126-L135

Re: [DISCUSS] How we handle JDK support

[Jordan West <jw...@apache.org>]

When I worked on Riak we had a much more complex matrix due to supporting
even more backwards compatibility. It’s not unfeasible. You don’t have to
run every suite on every commit since as folks have pointed out for the
most part the JVM isn’t culprit. Need to run it enough times to catch when
it is for some assumption of “enough”.

We used to run them closer to releases but that would often lead to a
scramble to fix. Periodically would be my preference. And on demand as
engineers see need when they think they may have changed something where
the JVM could have an impact.

However, one challenge is we may be more limited on test resources as I
believe Mck pointed out on another thread.

Jordan

On Tue, May 20, 2025 at 19:52 Josh McKenzie <jm...@apache.org> wrote:

> The problem with (2) being only "overlapping JDK version support on
> consecutive releases" instead of an overlapping JDK over all `N-2` releases
> is that we say we support upgrade paths that we never test (w/
> jvm-dtest-upgrade).  Here, I would rather add a third LTS JDK to a release
> to maintain that `N-2` testing, than to push something untested onto users.
>
> Good point. With 4.0 in the mix right now (i.e. 3 supported branches) we
> have a pretty nasty matrix we'd have to test:
>
>    - 4.0 (N-3): 8, 11
>    - 4.1 (N-2): 8, 11
>    - 5.0 (N-1): 11, 17
>    - 6.0: 11, 17, 21
>
> Upgrade tests:
>
>    - JDK8: 4.0 -> 4.1
>    - JDK11: 4.0 -> 4.1
>    - JDK11: 4.0 -> 5.0
>    - JDK11: 4.1 -> 6.0
>    - JDK11: 5.0 -> 6.0
>    - JDK17: 5.0 -> 6.0
>
> If 4.0 weren't in the mix, we'd be looking at still having 4 jvm-upgrade
> tests we'd need to run w/the "shared JDK for N-2" paradigm (as opposed to
> "break JVM and C* version together"):
>
>    - N-2 to N-1 on shared JDK-2
>    - N-2 to N on shared JDK-2
>    - N-1 to N on shared JDK-2
>    - N-1 to N on shared JDK-1 (if we bump to a new JDK we support here).
>
> So yeah. I think we'll need to figure out how much coverage is reasonable
> to call something "tested". I don't think it's sustainable for us to have,
> at any given time, 3 branches we test across 3 JDK's each with all our
> in-jvm test suites is it?
>
> On Tue, May 20, 2025, at 2:24 PM, Ekaterina Dimitrova wrote:
>
> Another thing to consider is the usage of JDK internals. The JDK
> developers do not promise backward compatibility for internals. We still
> have things like jamm that need updates, and not only jamm. Sometimes they
> can fail us silently despite fully green CI.
>
> Performance is a good point - we don’t even have regular performance
> testing.
>
> With that said - I see pros and cons in both suggestions here. Just wanted
> to bring visibility to yet another wrinkle (the jdk internals usage) as I
> am sure there will be a lot of people who haven’t even heard of jamm
> probably.
>
> On Tue, 20 May 2025 at 14:09, Benedict <be...@apache.org> wrote:
>
>
> There are performance differences between JVMs. I agree that bug testing
> of JVM versions for clients is not very important, but isolating JVM
> characteristic changes from database characteristic changes is important,
> for me at least.
>
>
> On 20 May 2025, at 17:47, Jon Haddad <jo...@rustyrazorblade.com> wrote:
>
> 
>
> If you're upgrading an environment without doing any additional testing -
> sure, it can be helpful to isolate the issue.
>
> However, outside of this scenario, where you actually test your upgrade
> process and vet the functionality, I don't see it as a big gain - certainly
> not enough of one to hold the project from moving forward.  If there are
> bugs in the JVM itself, they should have been found already.  We're almost
> 2 years behind in the LTS release, we have plenty of testing, this stuff
> should be caught long before it's time to upgrade.
>
> The way I see it, we should do one of the following:
>
> * support multiple versions and only limited overlap.  For example, we
> support 11, 17, 21, 24 in 6.0,  then 21, 24, 27 in 7.0.
> * or do the two version thing and not bother with overlap.
>
> The main reason I can think of to continue to support older versions is
> dependency compatibility.  If we use C*-all within the bulk reader and that
> requires supporting older JVMs for Spark itself.
>
> The other reasons (A/B JVM testing & debugging upgrade bugs) are pretty
> weak in comparison to the gains to be had from moving forward.   For
> example, dropping older versions means we can:
>
> * Use generational ZGC (jdk 21+) as standard.  No more long pauses.
> * Move all allocation to per-thread arenas (leveraging memory layouts &
> structured memory) to avoid allocation on the heap (21+)
> * Ability to use virtual threads (I think 24+)
>
> With our current policy, this is our timetable:
>
> * 6.0 : 17 + 21. 2025
> * 70: 21 + 24. 2026
> * 8.0 24 + 27: 2027
>
> We're a year away from having a release that can use any of the "newer"
> JVM features and 2 years away from having the ability to do some
> intelligent memory management.  That's a long time for an entire community
> to wait because of the users who want to do A/B tests against JVM versions,
> or want the ability to debug potential java release issues in production
> without a staging environment.
>
> Jon
>
>
>
> On Tue, May 20, 2025 at 9:07 AM Brandon Williams <dr...@gmail.com> wrote:
>
> On Tue, May 20, 2025 at 10:59 AM Jon Haddad <jo...@rustyrazorblade.com>
> wrote:
> > > There is also that recommendation that I keep on hearing - don’t do C*
> major upgrade and JDK upgrade simultaneously. I believe that was one of the
> reasons for overlap too
> >
> > There's no practical reason for this today.  Maybe in the Java 6 or 8
> days, sure.  But now, it's a useless requirement.
>
> If I'm going to encounter a strange bug after upgrading, I'd like the
> surface area to be limited to one of C* or the JVM if possible.
>
> Kind Regards,
> Brandon
>
>
>

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

> The problem with (2) being only "overlapping JDK version support on consecutive releases" instead of an overlapping JDK over all `N-2` releases is that we say we support upgrade paths that we never test (w/ jvm-dtest-upgrade).  Here, I would rather add a third LTS JDK to a release to maintain that `N-2` testing, than to push something untested onto users.
Good point. With 4.0 in the mix right now (i.e. 3 supported branches) we have a pretty nasty matrix we'd have to test:
 • 4.0 (N-3): 8, 11
 • 4.1 (N-2): 8, 11
 • 5.0 (N-1): 11, 17
 • 6.0: 11, 17, 21
Upgrade tests:
 • JDK8: 4.0 -> 4.1
 • JDK11: 4.0 -> 4.1
 • JDK11: 4.0 -> 5.0
 • JDK11: 4.1 -> 6.0
 • JDK11: 5.0 -> 6.0
 • JDK17: 5.0 -> 6.0
If 4.0 weren't in the mix, we'd be looking at still having 4 jvm-upgrade tests we'd need to run w/the "shared JDK for N-2" paradigm (as opposed to "break JVM and C* version together"):
 • N-2 to N-1 on shared JDK-2
 • N-2 to N on shared JDK-2
 • N-1 to N on shared JDK-2
 • N-1 to N on shared JDK-1 (if we bump to a new JDK we support here).
So yeah. I think we'll need to figure out how much coverage is reasonable to call something "tested". I don't think it's sustainable for us to have, at any given time, 3 branches we test across 3 JDK's each with all our in-jvm test suites is it?

On Tue, May 20, 2025, at 2:24 PM, Ekaterina Dimitrova wrote:
> Another thing to consider is the usage of JDK internals. The JDK developers do not promise backward compatibility for internals. We still have things like jamm that need updates, and not only jamm. Sometimes they can fail us silently despite fully green CI. 
> 
> Performance is a good point - we don’t even have regular performance testing.
> 
> With that said - I see pros and cons in both suggestions here. Just wanted to bring visibility to yet another wrinkle (the jdk internals usage) as I am sure there will be a lot of people who haven’t even heard of jamm probably.
> 
> On Tue, 20 May 2025 at 14:09, Benedict <be...@apache.org> wrote:
>> 
>> There are performance differences between JVMs. I agree that bug testing of JVM versions for clients is not very important, but isolating JVM characteristic changes from database characteristic changes is important, for me at least.
>> 
>> 
>>> On 20 May 2025, at 17:47, Jon Haddad <jo...@rustyrazorblade.com> wrote:
>>> 
>>> If you're upgrading an environment without doing any additional testing - sure, it can be helpful to isolate the issue.
>>> 
>>> However, outside of this scenario, where you actually test your upgrade process and vet the functionality, I don't see it as a big gain - certainly not enough of one to hold the project from moving forward.  If there are bugs in the JVM itself, they should have been found already.  We're almost 2 years behind in the LTS release, we have plenty of testing, this stuff should be caught long before it's time to upgrade.
>>> 
>>> The way I see it, we should do one of the following:
>>> 
>>> * support multiple versions and only limited overlap.  For example, we support 11, 17, 21, 24 in 6.0,  then 21, 24, 27 in 7.0.
>>> * or do the two version thing and not bother with overlap. 
>>> 
>>> The main reason I can think of to continue to support older versions is dependency compatibility.  If we use C*-all within the bulk reader and that requires supporting older JVMs for Spark itself.
>>> 
>>> The other reasons (A/B JVM testing & debugging upgrade bugs) are pretty weak in comparison to the gains to be had from moving forward.   For example, dropping older versions means we can:
>>> 
>>> * Use generational ZGC (jdk 21+) as standard.  No more long pauses.
>>> * Move all allocation to per-thread arenas (leveraging memory layouts & structured memory) to avoid allocation on the heap (21+)
>>> * Ability to use virtual threads (I think 24+)
>>> 
>>> With our current policy, this is our timetable:
>>> 
>>> * 6.0 : 17 + 21. 2025
>>> * 70: 21 + 24. 2026
>>> * 8.0 24 + 27: 2027
>>> 
>>> We're a year away from having a release that can use any of the "newer" JVM features and 2 years away from having the ability to do some intelligent memory management.  That's a long time for an entire community to wait because of the users who want to do A/B tests against JVM versions, or want the ability to debug potential java release issues in production without a staging environment.
>>> 
>>> Jon
>>> 
>>> 
>>> 
>>> On Tue, May 20, 2025 at 9:07 AM Brandon Williams <dr...@gmail.com> wrote:
>>>> On Tue, May 20, 2025 at 10:59 AM Jon Haddad <jo...@rustyrazorblade.com> wrote:
>>>> > > There is also that recommendation that I keep on hearing - don’t do C* major upgrade and JDK upgrade simultaneously. I believe that was one of the reasons for overlap too
>>>> >
>>>> > There's no practical reason for this today.  Maybe in the Java 6 or 8 days, sure.  But now, it's a useless requirement.
>>>> 
>>>> If I'm going to encounter a strange bug after upgrading, I'd like the
>>>> surface area to be limited to one of C* or the JVM if possible.
>>>> 
>>>> Kind Regards,
>>>> Brandon

Re: [DISCUSS] How we handle JDK support

[Ekaterina Dimitrova <e....@gmail.com>]

Another thing to consider is the usage of JDK internals. The JDK developers
do not promise backward compatibility for internals. We still have things
like jamm that need updates, and not only jamm. Sometimes they can fail us
silently despite fully green CI.

Performance is a good point - we don’t even have regular performance
testing.

With that said - I see pros and cons in both suggestions here. Just wanted
to bring visibility to yet another wrinkle (the jdk internals usage) as I
am sure there will be a lot of people who haven’t even heard of jamm
probably.

On Tue, 20 May 2025 at 14:09, Benedict <be...@apache.org> wrote:

> There are performance differences between JVMs. I agree that bug testing
> of JVM versions for clients is not very important, but isolating JVM
> characteristic changes from database characteristic changes is important,
> for me at least.
>
> On 20 May 2025, at 17:47, Jon Haddad <jo...@rustyrazorblade.com> wrote:
>
> 
>
> If you're upgrading an environment without doing any additional testing -
> sure, it can be helpful to isolate the issue.
>
> However, outside of this scenario, where you actually test your upgrade
> process and vet the functionality, I don't see it as a big gain - certainly
> not enough of one to hold the project from moving forward.  If there are
> bugs in the JVM itself, they should have been found already.  We're almost
> 2 years behind in the LTS release, we have plenty of testing, this stuff
> should be caught long before it's time to upgrade.
>
> The way I see it, we should do one of the following:
>
> * support multiple versions and only limited overlap.  For example, we
> support 11, 17, 21, 24 in 6.0,  then 21, 24, 27 in 7.0.
> * or do the two version thing and not bother with overlap.
>
> The main reason I can think of to continue to support older versions is
> dependency compatibility.  If we use C*-all within the bulk reader and that
> requires supporting older JVMs for Spark itself.
>
> The other reasons (A/B JVM testing & debugging upgrade bugs) are pretty
> weak in comparison to the gains to be had from moving forward.   For
> example, dropping older versions means we can:
>
> * Use generational ZGC (jdk 21+) as standard.  No more long pauses.
> * Move all allocation to per-thread arenas (leveraging memory layouts &
> structured memory) to avoid allocation on the heap (21+)
> * Ability to use virtual threads (I think 24+)
>
> With our current policy, this is our timetable:
>
> * 6.0 : 17 + 21. 2025
> * 70: 21 + 24. 2026
> * 8.0 24 + 27: 2027
>
> We're a year away from having a release that can use any of the "newer"
> JVM features and 2 years away from having the ability to do some
> intelligent memory management.  That's a long time for an entire community
> to wait because of the users who want to do A/B tests against JVM versions,
> or want the ability to debug potential java release issues in production
> without a staging environment.
>
> Jon
>
>
>
> On Tue, May 20, 2025 at 9:07 AM Brandon Williams <dr...@gmail.com> wrote:
>
>> On Tue, May 20, 2025 at 10:59 AM Jon Haddad <jo...@rustyrazorblade.com>
>> wrote:
>> > > There is also that recommendation that I keep on hearing - don’t do
>> C* major upgrade and JDK upgrade simultaneously. I believe that was one of
>> the reasons for overlap too
>> >
>> > There's no practical reason for this today.  Maybe in the Java 6 or 8
>> days, sure.  But now, it's a useless requirement.
>>
>> If I'm going to encounter a strange bug after upgrading, I'd like the
>> surface area to be limited to one of C* or the JVM if possible.
>>
>> Kind Regards,
>> Brandon
>>
>

Re: [DISCUSS] How we handle JDK support

[Benedict <be...@apache.org>]

There are performance differences between JVMs. I agree that bug testing of
JVM versions for clients is not very important, but isolating JVM
characteristic changes from database characteristic changes is important, for
me at least.

  

> On 20 May 2025, at 17:47, Jon Haddad <jo...@rustyrazorblade.com> wrote:  
>  
>

> 
>
> If you're upgrading an environment without doing any additional testing -
> sure, it can be helpful to isolate the issue.
>
>  
>
>
> However, outside of this scenario, where you actually test your upgrade
> process and vet the functionality, I don't see it as a big gain - certainly
> not enough of one to hold the project from moving forward.  If there are
> bugs in the JVM itself, they should have been found already.  We're almost 2
> years behind in the LTS release, we have plenty of testing, this stuff
> should be caught long before it's time to upgrade.
>
>  
>
>
> The way I see it, we should do one of the following:
>
>  
>
>
> * support multiple versions and only limited overlap.  For example, we
> support 11, 17, 21, 24 in 6.0,  then 21, 24, 27 in 7.0.
>
> * or do the two version thing and not bother with overlap.  
>
>
>  
>
>
> The main reason I can think of to continue to support older versions is
> dependency compatibility.  If we use C*-all within the bulk reader and that
> requires supporting older JVMs for Spark itself.  
>
>
>  
>
>
> The other reasons (A/B JVM testing & debugging upgrade bugs) are pretty weak
> in comparison to the gains to be had from moving forward.   For example,
> dropping older versions means we can:
>
>  
>
>
> * Use generational ZGC (jdk 21+) as standard.  No more long pauses.
>
> * Move all allocation to per-thread arenas (leveraging memory layouts &
> structured memory) to avoid allocation on the heap (21+)
>
> * Ability to use virtual threads (I think 24+)
>
>  
>
>
> With our current policy, this is our timetable:
>
>  
>
>
> * 6.0 : 17 + 21. 2025
>
> * 70: 21 \+ 24. 2026
>
> * 8.0 24 \+ 27: 2027
>
>  
>
>
> We're a year away from having a release that can use any of the "newer" JVM
> features and 2 years away from having the ability to do some intelligent
> memory management.  That's a long time for an entire community to wait
> because of the users who want to do A/B tests against JVM versions, or want
> the ability to debug potential java release issues in production without a
> staging environment.
>
>  
>
>
> Jon
>
>  
>
>
>  
>
>
>  
>
>
> On Tue, May 20, 2025 at 9:07 AM Brandon Williams
> <[driftx@gmail.com](mailto:driftx@gmail.com)> wrote:  
>
>

>> On Tue, May 20, 2025 at 10:59 AM Jon Haddad
<[jon@rustyrazorblade.com](mailto:jon@rustyrazorblade.com)> wrote:  
>  > > There is also that recommendation that I keep on hearing - don’t do C*
> major upgrade and JDK upgrade simultaneously. I believe that was one of the
> reasons for overlap too  
>  >  
>  > There's no practical reason for this today.  Maybe in the Java 6 or 8
> days, sure.  But now, it's a useless requirement.  
>  
>  If I'm going to encounter a strange bug after upgrading, I'd like the  
>  surface area to be limited to one of C* or the JVM if possible.  
>  
>  Kind Regards,  
>  Brandon  
>

Re: [DISCUSS] How we handle JDK support

[Jon Haddad <jo...@rustyrazorblade.com>]

If you're upgrading an environment without doing any additional testing -
sure, it can be helpful to isolate the issue.

However, outside of this scenario, where you actually test your upgrade
process and vet the functionality, I don't see it as a big gain - certainly
not enough of one to hold the project from moving forward.  If there are
bugs in the JVM itself, they should have been found already.  We're almost
2 years behind in the LTS release, we have plenty of testing, this stuff
should be caught long before it's time to upgrade.

The way I see it, we should do one of the following:

* support multiple versions and only limited overlap.  For example, we
support 11, 17, 21, 24 in 6.0,  then 21, 24, 27 in 7.0.
* or do the two version thing and not bother with overlap.

The main reason I can think of to continue to support older versions is
dependency compatibility.  If we use C*-all within the bulk reader and that
requires supporting older JVMs for Spark itself.

The other reasons (A/B JVM testing & debugging upgrade bugs) are pretty
weak in comparison to the gains to be had from moving forward.   For
example, dropping older versions means we can:

* Use generational ZGC (jdk 21+) as standard.  No more long pauses.
* Move all allocation to per-thread arenas (leveraging memory layouts &
structured memory) to avoid allocation on the heap (21+)
* Ability to use virtual threads (I think 24+)

With our current policy, this is our timetable:

* 6.0 : 17 + 21. 2025
* 70: 21 + 24. 2026
* 8.0 24 + 27: 2027

We're a year away from having a release that can use any of the "newer" JVM
features and 2 years away from having the ability to do some intelligent
memory management.  That's a long time for an entire community to wait
because of the users who want to do A/B tests against JVM versions, or want
the ability to debug potential java release issues in production without a
staging environment.

Jon



On Tue, May 20, 2025 at 9:07 AM Brandon Williams <dr...@gmail.com> wrote:

> On Tue, May 20, 2025 at 10:59 AM Jon Haddad <jo...@rustyrazorblade.com>
> wrote:
> > > There is also that recommendation that I keep on hearing - don’t do C*
> major upgrade and JDK upgrade simultaneously. I believe that was one of the
> reasons for overlap too
> >
> > There's no practical reason for this today.  Maybe in the Java 6 or 8
> days, sure.  But now, it's a useless requirement.
>
> If I'm going to encounter a strange bug after upgrading, I'd like the
> surface area to be limited to one of C* or the JVM if possible.
>
> Kind Regards,
> Brandon
>

Re: [DISCUSS] How we handle JDK support

[Mick Semb Wever <mc...@apache.org>]

    .



> If it came down to either:
>
>    1. Support latest 2 LTS JDK on any given C* release, no overlap.
>    Operators deal with it.
>    2. Require 1 overlapping JDK version support on consecutive releases.
>    Users may have to wait a year for new JDK features
>
>



My opinions…
 - we shouldn't be getting users to do upgrades that we aren't testing with
jvm-dtest-upgrades,
 - we should recommend operators to upgrade one thing (C* or jdk) at a
time, and
 - make it possible for operators to only upgrade one thing (C* or jdk) at
a time.

That rules out (1).  This isn't saying operators have to upgrade only one
thing at a time, just that we recognise it as good operator hygiene, and as
a valuable tactic for operators when they do hit bugs.

The problem with (2) being only "overlapping JDK version support on
consecutive releases" instead of an overlapping JDK over all `N-2` releases
is that we say we support upgrade paths that we never test (w/
jvm-dtest-upgrade).  Here, I would rather add a third LTS JDK to a release
to maintain that `N-2` testing, than to push something untested onto users.

Re: [DISCUSS] How we handle JDK support

[Josh McKenzie <jm...@apache.org>]

> This gets stated on perhaps an annual basis, so perhaps we should start having these conversations on wiki to avoid the repetition.
I didn't state this in my original email: my goal is to come to a consensus and codify it in the wiki going forward.

> * or do the two version thing and not bother with overlap. 
To state it back to make sure I understand:
In this scenario, you're advocating for us supporting 2 LTS per major but not bothering with overlap. That would put non-containerized bare metal operators in a scenario where they needed to have 2 JDK's in their environment and start the new version of C* w/the new JDK vs. old w/old. This would require the "install all the JDK's you need access to" paradigm which almost all linux distros support quite seamlessly.

And for containerized environments, this is a no-op since JDK is bundled.

vs. if we require overlap, we'll basically have at worst a year delay between an LTS JDK release and a C* release that supports it, assuming we release C* the day before the new JDK goes LTS. That accurate? Assuming we're up to date on JDK support and we release C* yearly, that doesn't seem *too* bad a tax to pay to me to get the flexibility of having JDK and C* version upgrades decoupled for operators.

And as with many other topics, if we had the habit to cut an alpha from trunk once a quarter and were quick about integrating new JDK support, we'd cut that wait down from 12 months to 3.

After talking that through, I have a slight preference for the "couple them, consecutive releases must share 1 JDK they support, and we cut alphas". The "cut alphas" thing we've discussed before but didn't hit a consensus and document so that kind of pollutes this thread.

If it came down to either:
 1. Support latest 2 LTS JDK on any given C* release, no overlap. Operators deal with it.
 2. Require 1 overlapping JDK version support on consecutive releases. Users may have to wait a year for new JDK features
... I'm neutral. Both have pros and cons.


On Tue, May 20, 2025, at 12:06 PM, Brandon Williams wrote:
> On Tue, May 20, 2025 at 10:59 AM Jon Haddad <jo...@rustyrazorblade.com> wrote:
> > > There is also that recommendation that I keep on hearing - don’t do C* major upgrade and JDK upgrade simultaneously. I believe that was one of the reasons for overlap too
> >
> > There's no practical reason for this today.  Maybe in the Java 6 or 8 days, sure.  But now, it's a useless requirement.
> 
> If I'm going to encounter a strange bug after upgrading, I'd like the
> surface area to be limited to one of C* or the JVM if possible.
> 
> Kind Regards,
> Brandon
> 

Re: [DISCUSS] How we handle JDK support

[Brandon Williams <dr...@gmail.com>]

On Tue, May 20, 2025 at 10:59 AM Jon Haddad <jo...@rustyrazorblade.com> wrote:
> > There is also that recommendation that I keep on hearing - don’t do C* major upgrade and JDK upgrade simultaneously. I believe that was one of the reasons for overlap too
>
> There's no practical reason for this today.  Maybe in the Java 6 or 8 days, sure.  But now, it's a useless requirement.

If I'm going to encounter a strange bug after upgrading, I'd like the
surface area to be limited to one of C* or the JVM if possible.

Kind Regards,
Brandon

Re: [DISCUSS] How we handle JDK support

[Jon Haddad <jo...@rustyrazorblade.com>]

While I understand *some* users care, we shouldn't take the preferences of
a very small minority of users as project policy.  I've worked with
hundreds of teams, and I can't recall a single one caring about doing A/B
testing in JVM versions.

> There is also that recommendation that I keep on hearing - don’t do C*
major upgrade and JDK upgrade simultaneously. I believe that was one of the
reasons for overlap too

There's no practical reason for this today.  Maybe in the Java 6 or 8 days,
sure.  But now, it's a useless requirement.

Jon



On Tue, May 20, 2025 at 8:45 AM Benedict <be...@apache.org> wrote:

> In-jvm dtests need to execute an upgrade path on a single jvm, but this is
> close to always possible on the latest jvm. We haven’t hit any issues that
> I know of in this respect during any version change, so I don’t think this
> is a real concern.
>
> Some users do care about overlapping JVM compatibility. Containers may be
> exceptionally common but it is far from 100% coverage, and some operators
> (myself included) prefer to decouple the effects of two different version
> upgrades so as to decipher what change causes what effect.
>
> This gets stated on perhaps an annual basis, so perhaps we should start
> having these conversations on wiki to avoid the repetition.
>
>
>
> On 20 May 2025, at 16:37, Jon Haddad <jo...@rustyrazorblade.com> wrote:
>
> 
> There’s no “leaving users in the lurch” by requiring JDK upgrades.
>
> If users are using containers (pretty much everyone i talk to) then the
> JDK is included, versions don’t matter.
>
> If not, every modern Linux distro supports multiple installed JDKS. Again,
> not a problem.
>
> So far the only convincing argument I’ve heard is that we need overlapping
> versions because of in-JVM dtests. Which is a shame, but sounds unavoidable
> without a massive change that I’m sure nobody is interested in.
>
> Let’s not pretend like we’re doing this for the users though, they aren’t
> getting any benefit from it.
>
> Jon
>
>
> On Tue, May 20, 2025 at 8:12 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>> This came up in the release versioning thread and we punted to its own
>> thread.
>>
>> *Topic: How do we want to handle JDK version support in C* releases?*
>>
>> Oracle LTS policy here:
>> https://www.oracle.com/java/technologies/java-se-support-roadmap.html
>>
>> My first rough thoughts:
>>
>>    1. Any given C* release will only support 2 JDK / JRE at any given
>>    time
>>    2. We only change JDK support on MAJOR releases (i.e. not patch)
>>    3. When we add support for a new JDK, we drop support for the older
>>    in that release
>>       1. Every consecutive release *must share support for a runtime JDK
>>       version with one version older* (this allows for live upgrades,
>>       CI, etc). So if we take a long time releasing C* and multiple JDK's rev, we
>>       don't jump from [11,17] supported to [21,23] and leave users in a lurch on
>>       the straddle
>>       4. We confirm C* builds on both supported JDK's
>>    5. We run all test suites against all supported JDK's for a release
>>    (yet more incentive to clean up slow tests..)
>>    6. Language level support will be constrained to oldest JDK supported
>>    7. We make an effort to support the latest LTS JDK on any given C*
>>    release (i.e. 21 now, 25 next major when it goes LTS Sep of this year)
>>
>> The major downside I can think of with the above is operators will need
>> to verify a new JDK maximally once every 2-3 years as older LTS support is
>> dropped.
>>
>> What am I not thinking of? What are other downsides with the above
>> proposal?
>>
>>

Re: [DISCUSS] How we handle JDK support

[Benedict <be...@apache.org>]

In-jvm dtests need to execute an upgrade path on a single jvm, but this is
close to always possible on the latest jvm. We haven’t hit any issues that I
know of in this respect during any version change, so I don’t think this is a
real concern.

  

Some users do care about overlapping JVM compatibility. Containers may be
exceptionally common but it is far from 100% coverage, and some operators
(myself included) prefer to decouple the effects of two different version
upgrades so as to decipher what change causes what effect.

  

This gets stated on perhaps an annual basis, so perhaps we should start having
these conversations on wiki to avoid the repetition.

  

  

  

> On 20 May 2025, at 16:37, Jon Haddad <jo...@rustyrazorblade.com> wrote:  
>  
>

> 
>
> There’s no “leaving users in the lurch” by requiring JDK upgrades.  
>
>
>  
>
>
> If users are using containers (pretty much everyone i talk to) then the JDK
> is included, versions don’t matter.
>
>  
>
>
> If not, every modern Linux distro supports multiple installed JDKS. Again,
> not a problem.
>
>  
>
>
> So far the only convincing argument I’ve heard is that we need overlapping
> versions because of in-JVM dtests. Which is a shame, but sounds unavoidable
> without a massive change that I’m sure nobody is interested in.
>
>  
>
>
> Let’s not pretend like we’re doing this for the users though, they aren’t
> getting any benefit from it.  
>
>  
>
>
> Jon
>
>  
>
>
>  
>
>
> On Tue, May 20, 2025 at 8:12 AM Josh McKenzie
> <[jmckenzie@apache.org](mailto:jmckenzie@apache.org)> wrote:  
>
>

>>  __

>>

>> This came up in the release versioning thread and we punted to its own
thread.

>>

>>  
>
>>

>>  **Topic: How do we want to handle JDK version support in C* releases?**

>>

>>  
>
>>

>> Oracle LTS policy here: <https://www.oracle.com/java/technologies/java-se-
support-roadmap.html>  
>
>>

>>  
>
>>

>> My first rough thoughts:

>>

>>   1. Any given C* release will only support 2 JDK / JRE at any given time

>>   2. We only change JDK support on MAJOR releases (i.e. not patch)

>>   3. When we add support for a new JDK, we drop support for the older in
that release

>>     1. Every consecutive release  _must share support for a runtime JDK
version with one version older_ (this allows for live upgrades, CI, etc). So
if we take a long time releasing C* and multiple JDK's rev, we don't jump from
[11,17] supported to [21,23] and leave users in a lurch on the straddle  
>
>>   4. We confirm C* builds on both supported JDK's

>>   5. We run all test suites against all supported JDK's for a release (yet
more incentive to clean up slow tests..)

>>   6. Language level support will be constrained to oldest JDK supported

>>   7. We make an effort to support the latest LTS JDK on any given C*
release (i.e. 21 now, 25 next major when it goes LTS Sep of this year)

>>

>>

>> The major downside I can think of with the above is operators will need to
verify a new JDK maximally once every 2-3 years as older LTS support is
dropped.

>>

>>  
>
>>

>> What am I not thinking of? What are other downsides with the above
proposal?

>>

>>  
>

Re: [DISCUSS] How we handle JDK support

[Ekaterina Dimitrova <e....@gmail.com>]

There is also that recommendation that I keep on hearing - don’t do C*
major upgrade and JDK upgrade simultaneously. I believe that was one of the
reasons for overlap too

On Tue, 20 May 2025 at 11:36, Jon Haddad <jo...@rustyrazorblade.com> wrote:

> There’s no “leaving users in the lurch” by requiring JDK upgrades.
>
> If users are using containers (pretty much everyone i talk to) then the
> JDK is included, versions don’t matter.
>
> If not, every modern Linux distro supports multiple installed JDKS. Again,
> not a problem.
>
> So far the only convincing argument I’ve heard is that we need overlapping
> versions because of in-JVM dtests. Which is a shame, but sounds unavoidable
> without a massive change that I’m sure nobody is interested in.
>
> Let’s not pretend like we’re doing this for the users though, they aren’t
> getting any benefit from it.
>
> Jon
>
>
> On Tue, May 20, 2025 at 8:12 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>> This came up in the release versioning thread and we punted to its own
>> thread.
>>
>> *Topic: How do we want to handle JDK version support in C* releases?*
>>
>> Oracle LTS policy here:
>> https://www.oracle.com/java/technologies/java-se-support-roadmap.html
>>
>> My first rough thoughts:
>>
>>    1. Any given C* release will only support 2 JDK / JRE at any given
>>    time
>>    2. We only change JDK support on MAJOR releases (i.e. not patch)
>>    3. When we add support for a new JDK, we drop support for the older
>>    in that release
>>       1. Every consecutive release *must share support for a runtime JDK
>>       version with one version older* (this allows for live upgrades,
>>       CI, etc). So if we take a long time releasing C* and multiple JDK's rev, we
>>       don't jump from [11,17] supported to [21,23] and leave users in a lurch on
>>       the straddle
>>       4. We confirm C* builds on both supported JDK's
>>    5. We run all test suites against all supported JDK's for a release
>>    (yet more incentive to clean up slow tests..)
>>    6. Language level support will be constrained to oldest JDK supported
>>    7. We make an effort to support the latest LTS JDK on any given C*
>>    release (i.e. 21 now, 25 next major when it goes LTS Sep of this year)
>>
>> The major downside I can think of with the above is operators will need
>> to verify a new JDK maximally once every 2-3 years as older LTS support is
>> dropped.
>>
>> What am I not thinking of? What are other downsides with the above
>> proposal?
>>
>>

Re: [DISCUSS] How we handle JDK support

[Jon Haddad <jo...@rustyrazorblade.com>]

There’s no “leaving users in the lurch” by requiring JDK upgrades.

If users are using containers (pretty much everyone i talk to) then the JDK
is included, versions don’t matter.

If not, every modern Linux distro supports multiple installed JDKS. Again,
not a problem.

So far the only convincing argument I’ve heard is that we need overlapping
versions because of in-JVM dtests. Which is a shame, but sounds unavoidable
without a massive change that I’m sure nobody is interested in.

Let’s not pretend like we’re doing this for the users though, they aren’t
getting any benefit from it.

Jon


On Tue, May 20, 2025 at 8:12 AM Josh McKenzie <jm...@apache.org> wrote:

> This came up in the release versioning thread and we punted to its own
> thread.
>
> *Topic: How do we want to handle JDK version support in C* releases?*
>
> Oracle LTS policy here:
> https://www.oracle.com/java/technologies/java-se-support-roadmap.html
>
> My first rough thoughts:
>
>    1. Any given C* release will only support 2 JDK / JRE at any given time
>    2. We only change JDK support on MAJOR releases (i.e. not patch)
>    3. When we add support for a new JDK, we drop support for the older in
>    that release
>       1. Every consecutive release *must share support for a runtime JDK
>       version with one version older* (this allows for live upgrades, CI,
>       etc). So if we take a long time releasing C* and multiple JDK's rev, we
>       don't jump from [11,17] supported to [21,23] and leave users in a lurch on
>       the straddle
>       4. We confirm C* builds on both supported JDK's
>    5. We run all test suites against all supported JDK's for a release
>    (yet more incentive to clean up slow tests..)
>    6. Language level support will be constrained to oldest JDK supported
>    7. We make an effort to support the latest LTS JDK on any given C*
>    release (i.e. 21 now, 25 next major when it goes LTS Sep of this year)
>
> The major downside I can think of with the above is operators will need to
> verify a new JDK maximally once every 2-3 years as older LTS support is
> dropped.
>
> What am I not thinking of? What are other downsides with the above
> proposal?
>
>

Re: [DISCUSS] How we handle JDK support

[Jordan West <jw...@apache.org>]

On Tue, May 20, 2025 at 08:12 Josh McKenzie <jm...@apache.org> wrote:

> This came up in the release versioning thread and we punted to its own
> thread.
>
> *Topic: How do we want to handle JDK version support in C* releases?*
>
> Oracle LTS policy here:
> https://www.oracle.com/java/technologies/java-se-support-roadmap.html
>
> My first rough thoughts:
>
>    1. Any given C* release will only support 2 JDK / JRE at any given time
>    2. We only change JDK support on MAJOR releases (i.e. not patch)
>    3. When we add support for a new JDK, we drop support for the older in
>    that release
>       1. Every consecutive release *must share support for a runtime JDK
>       version with one version older* (this allows for live upgrades, CI,
>       etc). So if we take a long time releasing C* and multiple JDK's rev, we
>       don't jump from [11,17] supported to [21,23] and leave users in a lurch on
>       the straddle
>
> I agree having overlap is a good thing but we don’t need to be
consecutive. Using this as an example, we could go [17,23] and skip 21 if
23 is already out and mature.


>    1.
>       1. We confirm C* builds on both supported JDK's
>    2. We run all test suites against all supported JDK's for a release
>    (yet more incentive to clean up slow tests..)
>    3. Language level support will be constrained to oldest JDK supported
>    4. We make an effort to support the latest LTS JDK on any given C*
>    release (i.e. 21 now, 25 next major when it goes LTS Sep of this year)
>
> The major downside I can think of with the above is operators will need to
> verify a new JDK maximally once every 2-3 years as older LTS support is
> dropped.
>
> What am I not thinking of? What are other downsides with the above
> proposal?
>
>