Re: [DISCUSS] Understanding fork motivations (backport branch follow-up)

[Josh McKenzie <jm...@apache.org>]

>  • [Must have] Minimum JDK21 support on 4.1 and higher
Is this purely due to genZGC? While it's playing very nicely so far, I would expect the gap between tuned G1 and genZGC to be less than tuned CMS and genZGC.

@Jon Haddad - got any insight on the above (CMS vs. G1 vs. shenandoah vs. genZGC)?

fwiw - I think backporting the ability to run on JDK21 (or any newer JDK tbh) should be pretty simple and a favorable cost/benefit. Just want to explore if it's just the GC in the new JDK or something else at runtime people are looking for.

On Fri, Oct 31, 2025, at 2:14 PM, Jaydeep Chovatia wrote:
>  • [Must have] Minimum JDK21 support on 4.1 and higher
>  • 

Re: [DISCUSS] Understanding fork motivations (backport branch follow-up)

[Jon Haddad <jo...@rustyrazorblade.com>]

I've been running a few hundred nodes with Shenandoah on Java 17 for the
last year, and while I can say it's been mostly nice it still has some
issues.  While pause times are definitely better than G1 (typically < 3ms),
Shenandoah isn't generational, so there's quite a bit of overhead from GC
when the allocation rate is high.  Shenandoah achieves low pause time by
throttling the allocator (pacing).  Since we allocate a *ton* during
compaction, we need to use huge heaps to compensate.  I've seen pacing take
up 35% of CPU time when profiling a cluster with perf issues.

<mildly off topic rant>
Generational ZGC should be a significant improvement, but it's not a silver
bullet.  I'd classify it as more of a crutch to mitigate the excessive
allocation found in our read and compaction paths.  Addressing a small
handful of unnecessary allocations would allow us to run much smaller heaps
while also reducing pause time (both median and at the tail) and
significantly overall throughput.  Every allocation results in a CPU stall,
inflates the side of the live set and adds overhead to every collection.  I
love the improvements in new collectors but I've found that people
sometimes massively overestimate what they can do for us and use it as an
excuse to abdicate from responsible memory management.

See CASSANDRA-20428 [1] for an example of an allocation in the hot path
that if we addressed would *massively* improve pause times and throughput.

To the future, newer JDK's have off heap Arenas [2], Memory Segments,
Memory Layouts and Structured access [3].  Leaning into that would allow us
to almost completely eliminate every single on-heap allocation, giving us
the ability to run C* on tiny heaps and have practically zero GC.

If we did that the collector an operator picks wouldn't matter at all,
because the collector would almost never run.

Feel free to branch this into a new discussion if you want to go down this
rabbit hole with me :)
</mildly off topic rant>

Jon

[1] https://issues.apache.org/jira/browse/CASSANDRA-20428
[2]
https://docs.oracle.com/en/java/javase/21/core/memory-segments-and-arenas.html
[3]
https://docs.oracle.com/en/java/javase/21/core/memory-layouts-and-structured-access.html



On Tue, Nov 11, 2025 at 7:29 AM Josh McKenzie <jm...@apache.org> wrote:

>
>    - [Must have] Minimum JDK21 support on 4.1 and higher
>
> Is this purely due to genZGC? While it's playing very nicely so far, I
> would expect the gap between tuned G1 and genZGC to be less than tuned CMS
> and genZGC.
>
> @Jon Haddad - got any insight on the above (CMS vs. G1 vs. shenandoah vs.
> genZGC)?
>
> fwiw - I think backporting the ability to run on JDK21 (or any newer JDK
> tbh) should be pretty simple and a favorable cost/benefit. Just want to
> explore if it's just the GC in the new JDK or something else at runtime
> people are looking for.
>
> On Fri, Oct 31, 2025, at 2:14 PM, Jaydeep Chovatia wrote:
>
>
>    - [Must have] Minimum JDK21 support on 4.1 and higher
>    -
>
>
>

Re: [DISCUSS] Understanding fork motivations (backport branch follow-up)

[Josh McKenzie <jm...@apache.org>]

We agreed on not dropping a JDK if it'd kill a feature (or at least called out the shape of this challenge): https://cwiki.apache.org/confluence/display/CASSANDRA/JDK+Version+Support+Policy

> In the very rare case a feature would have to be removed due to JDK change (think UDF's scripting engine)...
So we at least spoke to the implications on *part* of the intersection of JDK support and features. Good points Ekaterina and Scott - this would throw a wrinkle in my "backporting runtime support to older branches should be easy" claim. Barring those kind of exceptions I don't think it'll be a huge *technical* hurdle going forward. I'm not sure how we want to tackle this and if there'd be a path for conditional run of newer JDK w/Nashorn disabled at runtime that'd work; don't know enough about that impl to know if it's even a sane line of questioning. :)

> - jamm update will be needed, to the version we have in 5.0 or we may need new release of jamm for versions post JDK 17. It may lead to flushing change of behavior etc in a patch release
I reached out to Jonathan Ellis about this; he's receptive to us merging the jamm codebase into the C* repo which would make it far simpler to modify, test, and maintain. I don't love the idea of having to carry jamm support into the future but on reflection I think that primarily stems from the hurdle of maintaining and validating it in a separate repo. There aren't really any obvious license-compatible alternatives so at the very least, getting it to low friction to validate and modify jamm should help.

On Tue, Nov 11, 2025, at 11:03 AM, C. Scott Andreas wrote:
> The most pressing motivation for folks to get off JDK11 is that many organizations consider it end-of-life. It was released 7 years ago and Oracle Premier support ended two years ago, so it's in its final "extended support" lifecycle which runs through 2032 at what I'm sure is substantial expense.
> 
> Agree with Ekaterina's concerns about Nashorn removal in JDK15+ and the impact to API surface.
> 
> Good motivation to stay current in all respects – for us to continue advancing support for new JDK LTS releases; and for users to upgrade to current major versions of the database.
> 
>> On Nov 11, 2025, at 7:48 AM, Ekaterina Dimitrova <e....@gmail.com> wrote:
>> 
>> 
>> “fwiw - I think backporting the ability to run on JDK21 (or any newer JDK tbh) should be pretty simple and a favorable cost/benefit. Just want to explore if it's just the GC in the new JDK or something else at runtime people are looking for.”
>> 
>> Some caveats which are not about complexity but which need to be considered:
>> - we removed scripted UDFs to move forward and are not going to remove them in 4.1 (CASSANDRA-18252)
>> 
>> - jamm update will be needed, to the version we have in 5.0 or we may need new release of jamm for versions post JDK 17. It may lead to flushing change of behavior etc in a patch release
>> 
>> 
>> 
>> On Tue, 11 Nov 2025 at 10:29, Josh McKenzie <jm...@apache.org> wrote:
>>> __
>>>>  • [Must have] Minimum JDK21 support on 4.1 and higher
>>> Is this purely due to genZGC? While it's playing very nicely so far, I would expect the gap between tuned G1 and genZGC to be less than tuned CMS and genZGC.
>>> 
>>> @Jon Haddad - got any insight on the above (CMS vs. G1 vs. shenandoah vs. genZGC)?
>>> 
>>> fwiw - I think backporting the ability to run on JDK21 (or any newer JDK tbh) should be pretty simple and a favorable cost/benefit. Just want to explore if it's just the GC in the new JDK or something else at runtime people are looking for.
>>> 
>>> On Fri, Oct 31, 2025, at 2:14 PM, Jaydeep Chovatia wrote:
>>>>  • [Must have] Minimum JDK21 support on 4.1 and higher
>>>>  • 
>>> 
> 
> 

Re: [DISCUSS] Understanding fork motivations (backport branch follow-up)

["C. Scott Andreas" <sc...@paradoxica.net>]

The most pressing motivation for folks to get off JDK11 is that many organizations consider it end-of-life. It was released 7 years ago and Oracle Premier support ended two years ago, so it's in its final "extended support" lifecycle which runs through 2032 at what I'm sure is substantial expense. Agree with Ekaterina's concerns about Nashorn removal in JDK15+ and the impact to API surface. Good motivation to stay current in all respects – for us to continue advancing support for new JDK LTS releases; and for users to upgrade to current major versions of the database. On Nov 11, 2025, at 7:48 AM, Ekaterina Dimitrova <e....@gmail.com> wrote: “fwiw - I think backporting the ability to run on JDK21 (or any newer JDK tbh) should be pretty simple and a favorable cost/benefit. Just want to explore if it's just the GC in the new JDK or something else at runtime people are looking for.” Some caveats which are not about complexity but which need to be considered: - we removed scripted UDFs to move forward and are not going to remove them in 4.1 (CASSANDRA-18252) - jamm update will be needed, to the version we have in 5.0 or we may need new release of jamm for versions post JDK 17. It may lead to flushing change of behavior etc in a patch release On Tue, 11 Nov 2025 at 10:29, Josh McKenzie < jmckenzie@apache.org > wrote: [Must have] Minimum JDK21 support on 4.1 and higher Is this purely due to genZGC? While it's playing very nicely so far, I would expect the gap between tuned G1 and genZGC to be less than tuned CMS and genZGC. @Jon Haddad - got any insight on the above (CMS vs. G1 vs. shenandoah vs. genZGC)? fwiw - I think backporting the ability to run on JDK21 (or any newer JDK tbh) should be pretty simple and a favorable cost/benefit. Just want to explore if it's just the GC in the new JDK or something else at runtime people are looking for. On Fri, Oct 31, 2025, at 2:14 PM, Jaydeep Chovatia wrote: [Must have] Minimum JDK21 support on 4.1 and higher

Re: [DISCUSS] Understanding fork motivations (backport branch follow-up)

[Ekaterina Dimitrova <e....@gmail.com>]

“fwiw - I think backporting the ability to run on JDK21 (or any newer JDK
tbh) should be pretty simple and a favorable cost/benefit. Just want to
explore if it's just the GC in the new JDK or something else at runtime
people are looking for.”

Some caveats which are not about complexity but which need to be considered:
- we removed scripted UDFs to move forward and are not going to remove them
in 4.1 (CASSANDRA-18252)

- jamm update will be needed, to the version we have in 5.0 or we may need
new release of jamm for versions post JDK 17. It may lead to flushing
change of behavior etc in a patch release



On Tue, 11 Nov 2025 at 10:29, Josh McKenzie <jm...@apache.org> wrote:

>
>    - [Must have] Minimum JDK21 support on 4.1 and higher
>
> Is this purely due to genZGC? While it's playing very nicely so far, I
> would expect the gap between tuned G1 and genZGC to be less than tuned CMS
> and genZGC.
>
> @Jon Haddad - got any insight on the above (CMS vs. G1 vs. shenandoah vs.
> genZGC)?
>
> fwiw - I think backporting the ability to run on JDK21 (or any newer JDK
> tbh) should be pretty simple and a favorable cost/benefit. Just want to
> explore if it's just the GC in the new JDK or something else at runtime
> people are looking for.
>
> On Fri, Oct 31, 2025, at 2:14 PM, Jaydeep Chovatia wrote:
>
>
>    - [Must have] Minimum JDK21 support on 4.1 and higher
>    -
>
>
>