Cassandra project status update 2022-09-26

[Josh McKenzie <jm...@apache.org>]

/wave

ApacheCon is next week! Looking forward to seeing many of you there. We should add some coverage of that to the next biweekly maybe; that could be fun.

The 4.1 beta and ga releases continue to be held back by roughly 10 failing tests - often different failures on each run that can be seen here: https://butler.cassandra.apache.org/#/ci/upstream/compare/Cassandra-4.1/cassandra-4.1

Quite a few of us have been chatting back and forth about this for a few weeks now as we keep an eye on efforts, the environment, and the calendar. I opened a thread earlier today about us considering moving from an "ASF CI is green" gatekeeper to "circle-ci or ASF CI are green" here: https://lists.apache.org/thread/o166v7nr9lxnzdy5511tv40rr9t6zbrw as a temporary measure to unblock our release process and open up our ability to iterate on the ASF CI environment. Please chime in if you have a point of view you'd like to share - one of the great points on the thread already raised is that we'd need to add a suite or two to circle as well as still run a specific subset (build, packaging, etc) on ASF CI, so there's some nuance there but certainly something we can work out.


[New Contributors Getting Started]
While the above context and discussion may shift things, we currently have 16 tickets that are test failures blocking 4.1 in some form or fashion that can be found here: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252&quickFilter=2456. Test failures are a great way to get exposed to some of the broader classes in the codebase, getting your environment setup, and getting to know the domain a little bit.

We also have 11 tickets currently unassigned that we consider "Starter Tickets", or "lhf" (low-hanging-fruit): https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2162&quickFilter=2160. i.e. challenging enough to be interesting but probably not bad enough to melt your brain. No promises. :)

And last but not least, you can explore by searching JIRA for topics you find interesting (compaction, repair, commitlog, messaging, etc): replace "ReplaceTextHere" with the topic on the following JIRA search: https://issues.apache.org/jira/issues/?jql=project%20%3D%20cassandra%20AND%20resolution%20!%3D%20unresolved%20AND%20assignee%20is%20EMPTY%20AND%20summary%20~%20%27ReplaceTextHere%27%20ORDER%20BY%20priority%20ASC. Or you can just YOLO it yourself with JQL if you like pain.

Here's our reference covering various types of contribution: https://cassandra.apache.org/_/community.html#how-to-contribute
An overview of the C* architecture: https://cassandra.apache.org/doc/latest/cassandra/architecture/overview.html
The getting started contributing guide: https://cassandra.apache.org/_/development/index.html
We hang out in #cassandra-dev on https://the-asf.slack.com and there's a @cassandra_mentors alias you can use to reach a bunch of us that have volunteered to help newcomers get situated.


[Dev list Digest]
https://lists.apache.org/list?dev@cassandra.apache.org:lte=14d:
Mick opened up a thread earlier today about 4.2 vs. 5.0; a bunch of folks have chimed in about work in flight that would necessitate a major revision as well as JDK support changes: https://lists.apache.org/thread/moy4f3lx4w5yvr1v33ok7ycgz6dcphyw

Branimir Lambov had the incredibly good manners of reaching out to the dev list about adding a new dependency - a first that I can recall! https://lists.apache.org/thread/ws3kgnqmgvblz4q7sxvvx1d3wnzw5ozm. He's looking at the Agrona libs nominally for use in the CASSANDRA-17240 Trie memtable impl work, but more broadly as a resource for us all to use on the project.

Stefan Miklosovic reached out about a new CEP-24 re: password validation and generation here: https://lists.apache.org/thread/1hs27lx2pw9lmp7rw499vn0m7vl2bgt1

Andrés de la Peña formally proposed CEP-20 Dynamic Data Masking for a vote which passed: https://lists.apache.org/thread/tt2n4jpgtzfnx47cdq2cyt5oo5v5bdvs

In the endless pursuit of us tidying up our project, Stefan's also introducing a checkstyle rule that'll fail on unused imports: https://lists.apache.org/thread/73wjj4nf9km0rg4dor97j8f8hrpq3psn. It'll be nice to have a backup check for that part of our workflow.

And last but not least, on the thread regarding improving the Commitlog write path, Branimir dropped a link to a paper that is no doubt excellent reading for anyone curious on the list. Email thread: https://lists.apache.org/thread/hw2x62165rrvvlqqd0qd0dd8z4v16cy2, paper: https://www.vldb.org/pvldb/vol15/p3359-lambov.pdf, "Trie Memtables in Cassandra". For anyone curious about this novel work and one potential future of our in-memory representation of data in Cassandra, definitely worth your while to check this out.


[CI Trends]
https://butler.cassandra.apache.org/#/

Here's our trends on our branches for the last two weeks:

3.0: 7 -> 10
3.11: 16 -> 17
4.0: 8 -> 5
4.1: 7 -> 11
trunk: 7 -> 11

Some new repair failures make up quite a few on the 3.0 and 3.11 branches, both unassigned:
- https://issues.apache.org/jira/browse/CASSANDRA-17701
- https://issues.apache.org/jira/browse/CASSANDRA-17702

And one other new failure on multiple branches: https://issues.apache.org/jira/browse/CASSANDRA-17329

Expect to see some new JIRAs open up for test failures on the 4.1 line regardless of how our discussion about circle vs. ASF for release gating turns out. Ultimately we're going to want to tidy up tests that are non-deterministic or otherwise exacerbate the ASF runtime env failures.


[Release progress]
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2175

In the interest of time I'm going to defer some of the digging to each of you if you're curious and point out highlights here:

4.1 alpha + beta + rc: 9 issues this round up from 8 last
- Variety of test fixes (3)
- fix for ccm logs not updating correctly in CI (CASSANDRA-17901)
- fix for a possible OOM during replacements (CASSANDRA-17895)
- fix for repair failure w/2 peers w/overlapping mismatching ranges (CASSANDRA-17900)
- ci env fix for test splits on resource intensive tests (CASSANDRA-17906)
- Tidy up deprecation of params in logs when redundant / unnecessary (CASSANDRA-17904)

4.X / Next: 9 issues
- Add replica specific exception passback to let a coordinator know something has timed out (CASSANDRA-17810)
- Fix a leak of non-standard Java types in exceptions (yay JMX!) (CASSANDRA-17668)
- Deprecate Throwables.propagate usage (thanks google!) (CASSANDRA-14218)
- Refactor python code from bin to pylib (CASSANDRA-17531)
- Allow disabling hotness persistence, or tuning of rate limiter (CASSANDRA-17868)
- Fix an infrequent edge case NPE on changing neverPurgeTombstones from true to false during runtime (CASSANDRA-17897)
- Add metrics around storage usage and compression to enable easier comparison between existing and new compression algorithms (CASSANDRA-17898)
- Allow reducing scope for L0 to L1 compactions when space is limited (CASSANDRA-17899)
- Remove usage of deprecated javax certificate class (CASSANDRA-17867)


Let's see how some of these discussions on the dev list evolve. I know it's been a bit of a grind on CI with 4.1 but we'll get there. Thanks everyone for your continued persistence - this is going to all pay off in future releases and for our end users!

~Josh

unsubscribe

[eu...@gmail.com]

CASSANDRA-14227 removing the 2038 limit

[Berenguer Blasi <be...@gmail.com>]

Hi all,

I have taken a stab in a PR you can find attached in the ticket. Mainly:

- I have moved deletion times, gc and nowInSec timestamps to long. That 
should get us past the 2038 limit.

- TTL is maxed now to 68y. Think CQL API compatibility and a sort of a 
'free' guardrail.

- A new NONE overflow policy is the default but everything is backwards 
compatible by keeping the previous ones in place. Think upgrade 
scenarios or apps relying on the previous behavior.

- The new limit is around year 292,471,208,677 which sounds ok given the 
Sun will start collapsing in 3 to 5 billion years :-)

- Please feel free to drop by the ticket and take a look at the PR even 
if it's cursory

Thx in advance.