Cassandra project biweekly status update 2022-05-03

[Josh McKenzie <jm...@apache.org>]

We have a cassandra-4.1 branch!
https://issues.apache.org/jira/browse/CASSANDRA-17589

Congratulations to everyone on the project for us hitting our one year branch and freeze commitment; this is a huge milestone for the project. Also, a special call out to thank Mick for all the hard (currently manual) work that's going into the release process.

As a refresher and reference, here's our current procedures surrounding our release lifecycles: https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle

By the numbers, here's what the past year has looked like:
Total Tickets:  498
Resolved Bugs:  216, 43.4%
Task:           67,  13.5%
Improvement:    167, 33.5%
New Feature:    28,   5.6%

Number of unique contributors: 89

High level scope of the changes:
git diff cassandra-4.1 cassandra-4.0.0 --stat
3169 files changed, 65128 insertions(+), 310117 deletions(-)

This is an immense amount of work by a huge community of people. Also, it warms my heart to see us deleting more code than we're adding =P (and yes, I know it's a fraught measure and vanity metric. Don't take this away from me.)

sloccount on 4.0.0 /test directory:
Totals grouped by language (dominant language first):
java:        201122 (99.91%)
sh:             185 (0.09%)

sloccount on 4.1.0 /test directory:
Totals grouped by language (dominant language first):
java:        259472 (99.93%)
sh:             185 (0.07%)

That is a 29% increase in lines of code testing our system in one year.

Expect to see some blog posts and emails about the details of why our users should be as excited about cassandra-4.1 as those of us that have been working on it are - this has truly been a critical year for us as we've used the incredibly stable foundation cassandra-4.0 provided us to modularize subsystems, add cutting edge new forms of testing, major improvements to LWT performance, guardrails, and more.

At the risk of coming across as cheerleading, Cassandra truly is a unique project (both the code and the people!) and I'm really excited to see how the community receives 4.1 as well as see the CEPs we worked through in this past year start to land in 2022!


[New contributor Getting Started]
You're joining at a great time. (I mean, it's always a great time here on Cassandra, but some nice clean delineation happens around a release).

We use JIRA for our work tracking, and here are a couple saved Kanban boards to visualize the outstanding workloads in question:
Unassigned failing tests (69 currently unassigned): https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252
Unassigned starter tickets (28 unassigned): https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2162&quickFilter=2160. The 4.0.x swim lane is for our latest stable GA release and the 4.x swim lane covers the upcoming 4.1 release we'll be freezing in May.

We're in the middle of a code freeze before the next major release, so I'd encourage anyone looking to contribute to look into test failures happening on the 4.1 branch at this time. You should be able to check the kanban board quick filter here to see these failures: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252&quickFilter=2456

Join us on slack at the apache server at https://the-asf.slack.com, channel #cassandra-dev, and don't hesitate to reach out to the @cassandra_mentors alias to get in touch with volunteers who are interested in helping new folks get up to speed.


[Dev list Digest]
https://lists.apache.org/list?dev@cassandra.apache.org:lte=2w:

The test artifact for cassandra-4.0.4 is out: https://lists.apache.org/thread/yvqwgznpmn23xnbcqm0gmjhygwpyqd9y

Chris Thornett has been a busy bee and reached out for feedback about the Liquibase case study blog post (https://lists.apache.org/thread/7w3s4zwhhx6vl1hqlmkpd678jso2y1jk), podcast interview for Feathercast (https://lists.apache.org/thread/pv9vgpym6wksq6ddtpo518xj7t6w7qcw), and a blog post around multi-value data types and gotchas to avoid (https://lists.apache.org/thread/vlv9yn59tbqrqb4rtm7jd8mfcyyokxt1)

Patrick reached out about a general marketing meeting about Apache Cassandra scheduled for May 5th at 0900 PST, 1700 GMT: https://lists.apache.org/thread/6nwft0xkrpz8wfk431yoy8jlcrh4jmhj

Ekaterina surfaced the change that new config parameters added on trunk will have some new format constraints: https://lists.apache.org/thread/9lo1hccvf93828k9rf1hkr2176kcvx8h

And last but not least, Gavin McDonald reached out about the travel assistance program for folks that are looking to go to ApacheCon - see more here: https://lists.apache.org/thread/8l1rd2f2s9w86qjf20o6tgsoxovdm3fq


[CI Trends]
Butler dashboard: https://butler.cassandra.apache.org/#/
We don't yet have cassandra-4.1 integrated into butler or CI running for it, but we can largely look at trunk to see the state of it.

Our last few runs are up over 300 failures; I believe there were some disk space issues and in general looks like we had some just unhealthy runs (https://ci-cassandra.apache.org/job/Cassandra-trunk/1125/). What we _can_ say, is that our last relatively clean run of trunk CI had 13 failures down from 15 the run prior. Hopefully we'll be able to get 4.1 out around the July 1st time frame which is our goal.

[Release progress]
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2175

cassandra-4.1 branch is cut.
We are distinguishing 4.1 blockers as "FixVersion = 4.1 and resolution = unresolved". There's a quick filter added to the kanban board here: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2455

So our body of work to get 4.1 out the door is the combination of tickets in the above board plus having a green test board.

4.1:
We had 25 tickets land in the past two weeks, and some were big ones. The pluggable memtable implementation landed in CASSANDRA-17034, Guardrails for disk usage in CASSANDRA-17150, CONTAINS and CONTAINS KEY support for LWT's in CASSANDRA-10537, Guardrails for ALLOW FILTERING and Min RF (CASSANDRA-17370 and CASSANDRA-17212), and a Guardrail to disallow DROP or TRUNCATE functionality on tables in CASSANDRA-17558 to name a few. Not surprised to see things squeaking in at the line, but glad to see there's been no overwhelming or unhealthy pressure on this front. I'd hoped that our commitment to yearly releases would help somewhat with the pressure we've seen in the past around major / minor releases and so far things seem promising.

4.0.4:
Test branch cut.

Ticket wise, some interesting close outs: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2175. Cleaning up mutation flushing on drain, some test timeouts, and the big one: BTree performance improvements in CASSANDRA-15510.

Again, very excited for all the progress we've made in the past year. Let's keep the pressure and energy up and burn through these last stragglers to try and get 4.1 out the door in July!


~Josh